Jun 16, 2018
Orphan Blocks — June 16, 2018
This time we revisit the topic of orphan blocks. An orphan block occurs when two wallets win a block reward at similar times and both publish blocks to the network. One block will be accepted and grow the blockchain, one block will become an orphan and will be rejected.
We look at two questions for orphan blocks:
1. Who got an orphan block?
2. Did my wallet get an orphan block, and what should I do about it?
We turn to a familiar friend to help answer these questions: the wallet’s debug.log file.
I’ll admit this is a specialized topic of interest to only a few people. If you are blockchain-curious please read on; if you think you have mined an orphan block you must read on; if you have a general interest in the blockchain and bad blockchain jokes, it’s up to you…
I am an independent researcher, occasional blogger, and social moderator, and appreciate the technical guidance from the Qtum Team and discussions in the Community. If you have any comments or corrections for this article, please reach out on the social medias.
Mainnet Performance
First, a brief review of recent Qtum Mainnet performance.
Network Weight
My alternate approach for calculating network weight relies on the percentage of block rewards raked in by big staking wallets with known balances and gives a more stable and accurate number than the calculated estimate of network weight by the wallets. In the last week, these big wallets with a balance of 1.75 million collected 9.96% of the block rewards giving a network weight of 17.6 million. The network weight of 17.6 million represents an annual staking return of 5.0%
Transaction Fees
Recently there have been some blocks with initial double-digit transaction fees, due to contract calls. A heavyweight was block 167,861 which collected 5.9 QTUM in transaction fees, including gas from a number of LSTR token transfers related to a Luna Stars promotion.
This block held 323 transactions, which paid initial transaction fees of 29.92 QTUM, after gas refunds leaving a payment of 9.9 QTUM for the block reward winners (4.0 QTUM block reward + 5.9 transaction fees split 10 ways). Note that for this block the transaction fees were greater than the block reward.
Orphan Blocks
TL;DR decentralized blockchain mining platforms will occasionally create orphan blocks when two nodes publish a new block at similar times. Orphan blocks are perfectly normal and perfectly healthy for decentralized networks. The orphan block rate for Qtum is about 1.6% or 10 orphan blocks per day. The Qtum network handles orphans compassionately (they are canceled out) but very occasional disruptions to this process cause orphan block stakes to be stranded, in which case some wallet command line tools can be used for recovery.
Introduction to Orphans
The Broadway musical “Little Orphan Annie” premiered in New York City in 1977 and told the story of the plucky (and musically gifted) Little Orphan Annie, her hard knock life in an orphanage, and eventual adoption by millionaire industrialist Daddy Warbucks. In the movie version of Annie, Miss Hannigan, the evil headmistress of the orphanage says:
.
“Why any kid would want to be an orphan is beyond me. You’ll stay up till this dump…shines like the top of the Chrysler Building.”
.
.
.
On the Qtum blockchain, orphan blocks are produced when two staking wallets produce a new block at a similar time. This is an occasional and completely natural result for decentralized nodes. The orphan blocks relation to the main blockchain looks like:
To analyze the number of orphan blocks produced, I processed the debug.log in Excel to identify when new blocks were received. For wallets that allow incoming connections each block is received from many different nodes, 30 or more, and the wallet will validate and select one to append to its local copy of the blockchain. All the 30 blocks received have identical data, including an identical transaction ID, since they come from the single block reward winning wallet, unless, well you know where this is going.
If an orphan block is mined, it will have the same block number, but a different transaction ID than the others. If you import a week’s worth of debug.log file lines into Excel, and run some string matching formulae, the orphan blocks will pop out.
How many orphan blocks are there? I had some estimates in my previous blog. The orphan rate for bitcoin is about 0.5% (perhaps due to a fast relay network) and for Ethereum uncle blocks currently about 23% (uncle blocks for Ethereum are incentivized and handled differently than Qtum orphan blocks).
My analysis shows the orphan block rate for Qtum is 1.6%. This means with 600 blocks per day, there will be about 10 orphans. Another way of looking at this: if a wallet mines 63 blocks (over some period of time) on average one will be an orphan.
Who Got an Orphan Block?
Finding an orphan block is a little like finding a needle in a haystack. Parsing the debug.log files looking for orphans means multiple Excel worksheets of 1,000,000+ lines. Fortunately, there is the Excel FINDNEEDLE function which is just right for finding those shy orphan blocks [Reference 2]:
For one week’s worth of data, I detected 67 orphans (there may have been a few more). Note orphan blocks 165,136 and 165,137 were back-to-back orphans, but the orphan chain height was a single block for both (orphan block 165,137’s parent was Mainchain block 165,136).
Big staking wallets are going to mint many orphan blocks. We can look at QNqKe…, which is currently the largest staking wallet, and I believe this is an exchange hot wallet that is also staking (I don’t know which exchange, but thanks for helping secure the blockchain!). We know that a wallet will hit about one orphan block every 63 block rewards (but with lots of random variation), and QNqKe… gets 63 block rewards in a little over 2 days. Here is orphan block 163,638 from QNqKe… :
Notice two things this block, the Explorer tags it as “Orphaned”, and the space where the Next Block should be listed is sadly blank. This orphan block has Previous Block 163,637 but there is no Next Block. There is a parent block but no child block [Reference 3]. Only explorer.qtum.org will show orphan blocks, the other Explorers can’t be bothered.
Mainnet block 163,638 was actually won by wallet QgVXC… and its Explorer page shows “Mainchain” and Next Block 163,639:
Those of you with eagle eyes will also notice the Mainchain block has a higher Difficulty than the Orphaned block, and this is how the other nodes chose the Mainchain block over the orphan block: to give the blockchain with the highest difficulty (the most security).
When Bad Things Happen to Good Wallets
A sure way to detect orphan blocks from your staking wallet is by looking at the debug.log file. If you are not familiar with the debug.log file please see this blog.
To see if your wallet has mined an orphan block, open the debug.log file and search upwards from the bottom for “conflicts with wallet transaction”. If your wallet has mined an orphan you will find this log line and a log line a few lines later showing “CT_DELETED” which I believe removes the orphan transaction from the table of transactions. If you are watching your desktop GUI wallet compulsively you will see that initial 0.4+ mined transaction appear and then mysteriously disappear.
How to read the log?
From the “conflicts with wallet transaction” log line take both transaction IDs and search for them in Explorer.qtum.org. You may see that neither of these transaction IDs can be found in the blockchain or they could show an orphan block. Either result means you definitely mined an orphan block.
Let’s review what your wallet does when it wins a block reward. The block reward winning wallet will 1) select one or more previous transaction amounts (UTXOs) to stake for 500 blocks, 2) grab as many waiting transactions as it can from the memory pool and assemble them into the new block, 3) execute all the smart contract calls in those transactions against its local copy of the blockchain, and 4) publish the new block to the network. When your wallet does all this it cannot know if it creating a Mainchain block or an orphan block, so it does all this and commits the stake.
The best thing to do when your wallet mines an orphan block is to do nothing and chill. Your wallet will figure out it mined an orphan and sooner or later cancel out all the orphan block activity, including returning the stake.
However, very rarely this orphan unwinding process can be disrupted. I’ll be honest, I don’t fully understand the orphan disruption process but can guess that if you are nervous in the service, are watching your wallet closely, and start restoring it and swapping wallet.dat files to get back the orphan stake — that could strand the orphan stake. In this case, the Explorer will show the correct balance for an address(es) but your wallet will show a balance at that address lower by the amount of the staked UTXO. The amount missing from your wallet probably matches exactly the amount of a previous unspent transaction, which your wallet selected to stake for the orphan block.
If you find yourself in this situation it is time for some command line tools. For recovering from orphan stakes “salvage wallet” is not effective since it can fix corrupted wallet.dat files, and that is not the issue.
Note we are looking at command line parameters used when launching the wallet, not debug console/qtum-cli commands entered after the wallet is running. These command line tools are used when you launch the wallet from the command line. For Mac OS X you will enter command lines in the Terminal, as described in this blog. For Windows, you will enter commands using the Windows Command Prompt as described in this blog. If you are running the server wallet qtumd you would add the command line parameters when you launch qtumd (do not use qtum-cli with these commands).
Use these steps ONLY IF YOU HAVE A GOOD BACKUP of the wallet.dat file and have completed the troubleshooting above to confirm an orphan block.
The Rescan Command
The rescan command will go through your local copy of the blockchain and rebuild the database from scratch. See how the rescan works in this post.
Enter the rescan command as:
qtum-qt.exe -rescan
When you enter the command for qtum-qt the splash screen will open and give status “Rescanning…XX%” for a minute or two, then the main wallet will open.
The Zapwallettxes Command
The zapwalletxes command will rescan the local blockchain and remove (“zap”) any transactions for your wallet that are not published on the blockchain. Read more about zapwallettxes in this wiki.
Enter the zapwallettxes command as
qtum-qt.exe –zapwallettxes=2
.
After entering the command, qtum-qt will launch and show “Rescanning…” in the lower right-hand corner. After a minute or two, the full wallet window will appear, hopefully with the orphan stake restored.
If you are using these commands with qtumd, give it a minute or two to complete the rescanning, it will not respond to info commands before that.
By the end of the musical and movie, little orphan Annie is adopted by the millionaire industrialist Daddy Warbucks and goes on to lead a life of luxury. Likewise, the Qtum orphan blocks return their stakes and go on to a fulfilling life staking and securing the Qtum blockchain.
Orphan blocks are a necessary part of decentralized mining, and I hope this blog gives you the understanding to treat these orphans with the recognition and respect they deserve.
Stay safe online,
Jackson
References
1. See my past blogs, with these topics:
QRC20 Token Update, setting the Gas Limit and Gas Price for sending QRC20 tokens, Published on May 28, 2018
Qtum Peer Connections, a detailed look at now Qtum nodes connect to other nodes, outgoing/incoming connections, and some tools for monitoring the traffic, published on May 21, 2018
Top 10 Qtum New User Questions, and answers for new Qtum users, Published on Apr 22, 2018
Network Hash Rates, a comparison of network hash rates for bitcoin, Ethereum and Qtum, the numbers may surprise you, published on Apr 6, 2018
Newbie Wallets, an analysis of block reward winning wallets and the new Chinese nodes, published on Apr 1, 2018
The Debug Log, the Virtual Machine Log, the debug.log (and the virtual machine log) is your friend, how to find and read the debug log, published on Mar 13, 2018
QTUM Capped Supply — The Halving, is the supply of QTUM capped, how many will there ever be? The cap for QTUM, how “halving” works, and the new Explorer, published on Mar 4, 2018
QRC20 Token Report, QRC20 tokens, BOT airdrop, published on Feb 25, 2018
A Qtum PoS Simulator, Published on Feb 16, 2018
Testnet, the Qtum Test Network, published on Jan 7, 2018
Virtual Private Networks (VPNs), using VPNs for staking, published on Dec 31, 2017
Transaction Fees, Published on Dec 24, 2017
Passphrases, passphrases and a passphrase guessing script, published on Dec 17, 2017
The Memory Pool (mempool), the memory pool and processing of unconfirmed transactions, published on Dec 10, 2017
Orphan Blocks, published on Dec 3, 2017
The QTUM.explorer.io Contracts Page, published on Nov 26, 2017
The SHA256 Hash Algorithm and Difficulty, published on Nov 20, 2017
Distribution of Wallets Winning Block Rewards, published on Nov 12, 2017
The SHA256 Hash Algorithm, Target and Difficulty, published on Nov 5, 2017
An Introduction to Qtum Proof of Stake Mining — A Racing Story, an ELI5 story about PoS mining, published on Oct 29, 2017
Foundation Wallets Leave the Field, published on Oct 29, 2017
Unofficial Staking FAQ, published on Oct 26, 2017
Network Weight, Time to Reward, Nodemap, published on Oct 22, 2017
Simulating Proof of Stake Mining for Qtum, published on Oct 21, 2017
Qtum Proof of Stake Mining, an early simulator for PoS mining, published on Oct 16, 2017
Qtum Mainnet Ignition Results October 9–15, introduction to Mainnet charts and graphs, published on Oct 15, 2017
2. Another bad Jackson joke. There is no FINDNEEDLE function in Excel, but these formula work nicely for debug.log data loaded with the block transaction IDs in column E and block numbers in column F. Use string functions to extract the transaction ID and block number:
=IF(FIND(“height”, F2),MID(F2, 8, 6)) — extract the block number=IF(FIND(“height”, F2),MID(E, 6, 64)) — extract the transaction ID
3. “Orphan” is not the best term for these blocks. An orphan block has a parent, and we know the parents of orphan blocks, but they have no children. I think a better term would be Freemartin. Apparently, Satoshi was not a word maven.
4. Bitcoin orphaned blocks are shown on blockchain.info
The Broadway musical Annie opening night was April 21, 1977, at the Alvin Theater in New York City. The production had 2,377 performances and closed on January 2, 1983.
5. Music for orphan research, Aileen Quinn singing “Tomorrow” from the 1982 movie Annie. The sun’ll come up tomorrow, and you’ll win a block reward tomorrow.
.
6. Spotify album link, Annie: The Broadway Musical 30th Anniversary Cast Recording
.
.
7. New York City Drone flight, watch in 4k
.
