Target and Difficulty
--
Hitting the Target (going under the Target)
This week a closer look at difficulty and the target, in a carryover from last weeks’ ELI5 explanation of PoS mining. There has been some discussion on the Slack and subreddit about the wide swings in network weight, so this week we look at that phenomena.
To summarize the code snippet from last week which makes all the block rewards:
hashProofOfStake = SHA256Hash(list of data)if hashProofOfStake > Target * myWeight:
you lose
else:
you won!
This is the fundamental formula by which bitcoin, many other PoW coins, Blackcoin and Qtum are mined, and it depends entirely on the randomness of the SHA-256 hash algorithm. It was hard for me to understand this formula, so I thought of a visualization, which is the limbo game, where you win by squeezing under the limbo bar.
To help visualize this comparison of the target and the limbo bar here is a video of Meng Jia, formerly of the K-pop group Miss A (미쓰에이孟佳) going under the “target” to win the limbo game, with the target set at 65 cm — just hold your hand 65 cm above the floor to appreciate how low this target is (a little under 26 inches for our metric system-challenged readers). In the limbo game, and in the SHA-256 random number mining game, the winner is the one that goes under the target (see more moves by Jia as a solo artist here).
Difficulty Bits
The target is expressed on the blockchain as “Difficulty Bits”, or just “Bits”. Bitcoin adjusts the target, changing the bits every 2,016 blocks (about two weeks) to give a target block spacing of 10 minutes. Qtum, using a nimbler PoS algorithm, adjusts the difficulty (changes the target) with every block. This would be like adjusting the limbo bar after each contestant in the limbo game.
The difficulty bits and related Difficulty are shown for every block on the Qtum Explorer. Here is an example for block 37,000, shown as “1a07ac05” on the right-hand side of the Block page.
I would have liked to present the whole derivation of the difficulty Bits, target and Difficulty here, but it was just too difficult. There are many good references for target difficulty online for bitcoin, and it works identically for Qtum.
Converting these Bits to decimal gives a target of:
which seems like a fairly high target, but we need to compare it to the maximum value that SHA-256 hash algorithm can produce, which is
To get our unweighted probability (for staking 1.0 QTUM) to win the next block simply divide the target by the maximum SHA-256 value:
We are dividing the target by the maximum to go mathematically under that limbo bar.
Note that every wallet has the same target because it is given by the difficulty bits in the previous block. To get the weighted probability (the actual probability number your wallet will enjoy) multiply this target times the number of mature coins staking in your wallet: more weight in your wallet means a higher target, easier to limbo under and win those block rewards!
Even so, this seems like an impossibly small number, even for the Queen of Limbo to go under, but we know the block reward winner for block number 37001 did just that!
The random number lottery in the kernel hash algorithm cares nothing about network weight and block reward payments. It only cares about maintaining the average spacing between blocks, and the Qtum blockchain couldn’t maintain an average 600 blocks a day without an invisible hand tweaking the spacing on every block (well not invisible, it’s open source).
Now let’s look at some charts. The first is for the target starting at block 15,000, when there were only a few Foundation wallets mining, with similar weights, even so, there are some big variations in the target:
The following charts show 2,000 blocks on the x-axis, about 3 days, and relative scaling on the y-axis (not corrected to any “network weight” value). The data source for these charts is the blockchain, using a simple Python script block ripper (thanks to @calbert65 for suggesting the RPC commands).
You can make a comparison to difficulty variation on other blockchains, such as Blackcoin, which shares the target ballistics and block-by-block retargeting with Qtum (but other parameters are different) and bitcoin which adjusts difficulty every 2016 blocks, about two weeks, the last time at bitcoin block 491,904.
Now let’s look at some more recent history from this week, starting with block 37000, and what is different about this picture is there are now hundreds of wallets with a wide range of weights (block reward winners with weights in the low 10s to almost 2 million).
These charts both show a lot of variability, and we don’t know the true network weight to see if it was really changing.
We can take a look at the target long-term trend by looking at the target since mainnet launch (with a caveat shown at the end of this report). This next chart looks at the target every 20 blocks, from block 5,220 to block 38,340, using a 120-block average:
As expected, this long-term chart shows a higher target during the early days of mainnet, when Foundation wallets with about 4 million network weight were the only miners, with difficulty increasing and the target decreasing (remember the limbo bar) as private wallets came online starting in the mid 23,000 blocks, approximately October 8th, 2017.
Even with this long-term chart, there is variation and maybe even ringing in the target that is probably unrelated to the true network weight. Unfortunately, to further explore this variation, we can’t setup a live blockchain with hundreds of real wallets the scale of mainnet and run a bunch of tests, even on Skynet (RIP) or testnet. It is much easier to do this with a Python script simulation, so I did.
The simulation can explore various wallet weights and even plug in the actual wallet weights for the biggest wallets on mainnet. The simulator uses the same SHA-256 hash algorithm and the same difficulty adjustment ballistics (on a block-by-block basis) and can try different scenarios in minutes that it would take weeks to observe on mainnet.
First, as a starting baseline, give 1,000 wallets the same weight with a total network weight of 15,000,000 coins:
For uniform wallet weights, we only see a little variation from the block timing ballistics, so the target curve is fairly smooth.
We can check something else with the uniform weight wallets to see what happens when we add some significant weight during the simulation run. Starting with 15,000,000 coins, at block 2000 we add another 5,000,000 just to see what happens (I would be happy to run this test on mainnet, just send me the 5 million)
Before we look at that chart, remember we are charting the moving average of the target, which gives the value the winning wallet must go under (remember Jia going under the limbo bar). As network weight increases, if there were no adjustment of the target, the bigger wallets would help win new blocks faster, more often, and the blocks would speed up. However, the block-by-block adjustment increasing difficulty will slow the blocks down by decreasing the target. The calculation and charts for “network weight” use the target in the divisor, so as the target goes down, the “network weight” goes up, and the peaks and valleys between target and network weight charts are inverted.
You can see how the target gets pulled down on the simulated chart with 5 million added at block 2,000:
Next, we load the 1,000 wallets with random weights between 100 and 30,000 coins:
When the wallets had random weights assigned, the target curve gets much more animated, and the patterns are like seeing animals in clouds. Are these patterns caused by miners turning off their staking computers at night to save energy? Is it big wallets beginning staking? Is it price action retracing Fibonacci ratios? No, no, and no. For this simulation run, it is only the randomness of the SHA-256 hash algorithm and the ballistics of the wallets adjusting the target to keep the blocks running uniformly because the true network weight was held constant during this simulation.
Finally, we will do one more simulation run to get closer to the mainnet configuration, with actual mainnet wallet weights for the 100 biggest wallets from QTUM Explorer.io Biggest Wallets (thank you @calbert65), and the other 900 filled in with a sampled distribution of small and medium wallets:
Again, we see some crazy variation from a simulation which held a constant true network weight throughout.
I had to drop in one more chart to show you the block pacing and target adjustment ballistics are the same for every block, unless they aren’t, and they weren’t for the genesis blocks. To bring the blockchain to life, both Satoshi and Earlz assigned a special target to the first (genesis) blocks. For mainnet this was an easy target to go under, and the blocks flew by at 4 to 5 a second, up to block 5,001, where the normal target and ballistics kicked in, and you can see some ringing around this step impulse. Here is the target chart for mainnet blocks 4,100 to 6,085:
When you think about the incredible randomness of SHA-256 hash algorithm, all the miners trying to go under the target to win, it helps to remember Jia with her pink hair going under that 65 cm limbo bar.
We will end today’s report by visiting the beautiful and serene Jogyesa Temple (조계사) in Seoul, with the wish that this exploration of randomness and the target will help give you understanding, calmness of mind and serenity, as you collect many block rewards.
JB395
