Qtum Peer Connections — May 21, 2018
--
This time the blog takes a deep dive into one of the most interesting issues running a Qtum full node: how does the wallet/node connect to other Qtum peers, how many connections does it need, and how do you troubleshoot when it is not connecting?
Home networking works easily for most applications, but when you hit a roadblock with your Qtum node, what are you going to do? Your router — firewall — antivirus have many built-in protections to keep you safe with normal Internet activities but running a cryptocurrency wallet may conflict with these built-in safeguards.
We will use the mental model of a “friend request” on social media to explore node connections, where you send a request to someone you want to chat with on Facebook, Telegram, etc. Likewise, your node sends a “friend request” to connect with other nodes, and they acknowledge requests to chat with your node. Your node may also be able to receive “friend requests” from other nodes.
We will look at what exactly what is a Qtum full node, how the Qtum network works with peer-to-peer connections, and how to override well-meaning firewalls and routers that can block these communications. We will spend some quality time with ports and networking protocols and get intimate with the “netstat” network status tool. I hope that by the time we are done you will understand how your Qtum node can be happily chatting with its peers worldwide.
I am an independent blockchain researcher, occasional blogger, and social media moderator who appreciates the technical guidance from the Qtum Team. I hope to chat with all of you soon on the social medias. Please look for me as “Jackson” or “JB395” on Qtum’s Telegram, Reddit, Discord, and Forum social channels.
Qtum Mainnet Performance
The Qtum Mainnet blockchain performance continues to run smoothly, recently passing 150,000 blocks and 7,000 fully decentralized Proof of Stake nodes.
My calculations give a network weight of 20.8 million because big staking wallets with a stable total balance of 1.85 million won 8.9% of the block rewards in the past week. This represents a 4.2% annual rate of return for Qtum staking nodes:
Anti AntiVirus
TL;DR For historical reasons, many antivirus programs will report a false positive virus error when downloading Qtum wallet installation files, and quarantine or delete those downloads. To work around this, add an exception to the antivirus program, and if have concerns, download from the Qtum GitHub site and verify the file with a checksum.
Before we can run our wallet and have it send out “friend requests” we have to install the wallet, and there can be some problems downloading the wallet installation file, caused by well-meaning antivirus programs.
Consumer antivirus is certainly not state-of-the-art for endpoint (desktops, etc.) malware protection, but it is what most of us are using. Antivirus software works by matching file signatures with known malware file signatures. This can be most frustrating because you start downloading the wallet installation file, the download finishes, and it’s gone. What is happening?
Back in some bitcoin prehistory, there was an incident of some malware in a bitcoin miner file, and ever since antivirus programs are tagging bitcoin mining files, which are a close match for Qtum nodes.
Here is the download warning message from McAfee antivirus, which is fairly well behaved in this regard, it presents an “Accept the risk” override to allow downloading of the install file. Other antiviruses are not so user-friendly and could quarantine or delete the file with no warning. If your wallet downloads are disappearing, look for a way to add an exception in the antivirus program (or worst case, turn off antivirus momentarily for the download). It is also possible that you will be able to download and install the wallet, only to have it disappear the next time your antivirus does a full scan of your computer. This can be fixed by adding an exception in the antivirus program.
So, what to do? Go to the Qtum GitHub site for the latest releases (https://github.com/qtumproject/qtum/releases) download the correct version for your wallet install and verify the SHA256 checksum.
Copy the appropriate checksum into your favorite checksum tool to validate the file:
Qtum Peer Connections
TL;DR the Qtum node (Core wallet) will connect with up to 125 peers. The first 8 connections are outbound only: the node reaches out to connect to 8 other nodes. If the router and home network have port 3888 open, the node will accept incoming connections for peers 9 up to 125. If the node is also used as a staking wallet, it will win block rewards equally well with 8 or 125 connections. Nodes with more than 8 peer connections support new nodes connecting to the network and will download blocks to sync those new nodes. Nodes with 8 outbound connections can’t do that.
Now that we got past the antivirus program and installed the wallet, we can launch the node and begin connecting to other peers.
The Full Node — 8 vs. 125 Connections
Before we look at more details about how Qtum peer connections work it is useful to review the role of full nodes, staking wallets, and the issue of outbound connections vs. inbound connections.
The Qtum full node (the qtumd or qtum-qt Core wallets) connects to the Qtum peer-to-peer network, syncs the entire blockchain into local storage, and validates and relays every block and transaction in real time. Qtum nodes help secure the network by this validation and then relaying the blocks/transactions to other nodes. A Qtum node doesn’t have to hold any QTUM or be staking to provide this important security role. Only the Qtum Core wallets (qtumd and qtum-qt) can be full nodes, although exchanges and lightweight wallet providers would run full nodes also, probably based on qtumd.
A Qtum full node may additionally hold QTUM and used for staking to win block rewards. These staking full nodes help secure the network and have the opportunity to win block rewards.
For Qtum, a full node is always a wallet, but a wallet is not always a full node, for example, the Android mobile wallet and the Web wallet are not full nodes and cannot be used for staking. In this report, I am writing about the “node” but keep in mind this is really the Qtum Core wallet, which may or may not be staking.
By design, the first 8 connections established by the node are outgoing connections only. This means your node will initiate the “friend request” reaching out to connect with other nodes. Initially, I found this “outgoing” concept to be confusing; the nodes are always in two-way communication (in and out) with their peers, but the explanation is that for the first 8 connections your node initiates the connection by reaching out to connect with other nodes.
Any peer connections above 8 will be incoming connections (unless you are using the addnode command, which is outgoing). This means your node will accept connections coming in from other nodes — the remote nodes are sending the “friend request.” Nodes with incoming connections have special powers on the Qtum network: they will allow new nodes to connect to the network and they will download their inventory of previous blocks to those new nodes. We should have a special appreciation for nodes with incoming connections, without them, the network could not grow by adding new nodes.
We will see below that the ability of a node to allow incoming connections on Mainnet depends on opening port 3888 through the router and home network so that those incoming friend requests can actually reach the node.
Note that for winning a block reward it makes no difference whether a staking wallet has 8 or up to 125 connections, in fact, a wallet can make transactions and even win a block reward with only a single peer connection (reference 1).
For qtumd (the “headless” server wallet) users, you can monitor connections using these commands:
“getconnectionscount” will give the current peer connections, such as 8 or 124.
“getpeerinfo” will give details about the current peer connections.
Routers and Home Networks
Most home networks connect to the Internet through a network access device provided by an Internet Service Provider (ISP), which could be cable modem, DSL modem, or other network interface device. The modem usually has a built-in router, and the router’s job is to provide address translation from the single ISP-provided public Internet address to multiple internal IP addresses on the home network. These internal IP addresses are usually automatically assigned by protocols such as UPnP (universal plug and play), and what is important to understand is that the router maps a single external IP address to multiple internal IP addresses for the devices connecting on the home network.
Let’s look at a simplified drawing for a Qtum node running on a home network:
There is a lot to unpack here, but let’s go through it. The public IP address provided by the ISP for this broadband customer is 123.24.67.89, and computers anywhere on the Internet can send messages to this address. The internal network IP addresses are 192.0.0.1 for Computer A, which is running the Qtum node, 192.0.0.2 for Computer B, which is doing some Web surfing, and so on for Computer C.
Based on the traffic going back and forth, the router knows how to send a Web page request back to Computer B, and a Qtum node outgoing node request back to Computer A. But what happens after the Qtum node connects with the first 8 nodes with those outgoing peer requests? The problem is that for incoming peer requests, the router may not be smart enough to automatically route those friend requests to the node in Computer A, and so the incoming peer requests are lost and the node can’t connect above 8 nodes.
To fix this and allow the node to receive incoming peer requests we can use Port Forwarding to open port 3888 for Computer A. To open port 3888 we configure the router to forward any traffic for port 3888 to IP address 192.0.0.1, to reach the node in Computer A. Depending on your network setup, you can also map port 3888 from the wallet, select Settings — Options — Network — Map port using UPnP.
The website portforward.com has procedures and screenshots to setup port forwarding on hundreds of routers (just click past their ads) and the procedure is basically to log into your router, find the section to set up the port forwarding, and fill in a table to make the assignment for TCP Input port 3888 to the local IP address for your node.
I’ll be honest, port mapping should work to allow incoming connections, but I was unable to make it work. If someone can give some pointers, I can update this blog.
The Netstat Utility
In this section, we add some networking utilities and diagnostic techniques. Usually, your home network allows the Qtum node to start syncing automatically, but if not, these tools may help troubleshoot.
For monitoring your home network, the netstat (Network Status) utility is a good place to start. Run this tool from a command prompt. It is built in for Mac and Windows, you may have to install it on Linux (#apt-get install net-tools). The list of netstat options is given in reference 2.
The “netstat -n” command will display network addresses and ports in numeric form, which shows a node happily connected to remote nodes on port 3888:
The Addnode Command
If your new node is not connecting with peers, one way to help is with the “addnode” command. You can tell the node to send a “friend request” to a specific IP. But what IPs should you use?
A good source of peer IP addresses for the addnode command is coinexchange.io, where they are list “getpeerinfo” data from their Qtum node. To use the IP addresses from this site, and enter them as
addnode 123.45.67.89:3888 add
(this is a made up IP address, you should use actual IP addresses from coinexchange.io)
The correct response to the addnode command is “null”, and then your node will try for a minute or two to connect to the peer at that IP address. You can try manually adding 5 or 10 peers.
Making Connections at Startup
In this section, we look at how the node establishes connections on startup and will use netstat to watch the network and see how the node connects. Our netstat command is “netstat -an 30” which means display all connections and listening ports in numeric format every 30 seconds.
When you launch the node, there are several ways for it to find peers for connections. If the node has been running previously, it will save IP addresses and time last seen in the peers.dat file. If the node is brand new or can’t find some good IP addresses in the peers.dat file, it will query a DNS server for a list of current IP addresses. Finally, you can manually enter IP addresses to try using the “addnode” command.
First, we set the node to start up as a brand-new installation without the peer history in the peers.dat file (simply rename the peers.dat file so the node can’t find it).
Here is a sequence of messages for a new node working to find some peers. This node does not have the benefit of a peers.dat file listing past “friends”. Over the course of a half hour, it will try a number of IP addresses, finding two peers to connect with:
Qtum Core wallets keep a file with all their “friend requests” which is the peers.dat file. Launching the same node with its peer.dat file available (change the previous file name back to “peers.dat”) allowed the wallet to connect with 10 peers in 12 minutes. The chart below shows the difference for this wallet when it had to find new friends (no peers.dat file — shown above) vs. just reaching out to previous friends (using the peers.dat file):
Debug.log File for Network Events
The Qtum node is now ready for its closeup, and we will do that by setting the debug.log to capture all the networking events. This is done by launching the wallet with the debug=net switch:
C:\Program Files\Qtum>qtum-qt.exe -debug=net
Adjust this command as appropriate for your operating system.
With this command, the node will log all the details about network activity. For a very simple list showing how a connection is established (a more complete sequence is given in reference 7 below) the log shows:
(1) After starting up, our node tries to connect to several IPs from the peers.dat file. It saw the node at 59.189.111.143 ten days ago but could not connect now.
(2) Our node connects to a DNS seed server to pick up some recent IP addresses.
(3) Our node tries to connect to address 35.226.31.206, the 6th node to be tried.
(4) Our node sends a version message (software version, block number and time reference) to the node at 35.226.31.206. This version message is the “friend request.” The remote node will respond with its own version message.
(5) The nodes exchange verack (version acknowledge) messages to confirm the connection. Now they are friends and can start chatting. Our node will request additional IP addresses from the remote node, to send out other “friend requests.”
(6) Our node sends its latest block (it has been offline for 4 blocks) to 35.226.31.206 and requests a download of blocks to catch up.
(7) Our node begins to receive blocks from node 6.
Note the time stamp, it took 45 seconds after the node launched to try the connection for node 6 (peers 1 through 5 didn’t connect), and then one second to complete the connection (the verack messages) and begin downloading blocks.
We end the blog in Moscow, in part because I got a friend request from Lisa L, admin in the Qtum Telegram Russia channel, to translate the previous blog for the Qtum Russian Community, because there are currently 17 Qtum nodes in Russia, and because the FIFA World Cup is coming to Russia in June. What better reasons to take a photographic stroll through the Kremlin and Red Square? This area is a cultural centerpiece for Russia, home of Lenin’s tomb, Saint Basil’s Cathedral (built 1561), numerous museums, and bone-chillingly cold when your correspondent visited in February.
Спасибо, что читаете и будьте бдительны в интернетах,
Джексон
Credits
Special thanks to Liza L for Russian translation.
References
1. I used a Testnet wallet to verify a node operating with a single peer connection, using the command:
qtum-qt.exe -testnet -noconnect -connect=35.197.132.10:13888
2. The netstat utility.
Qtum peer communications are based on bitcoin, here are some relevant bitcoin references:
3. BitcoinCore — Running a Full Node
4. Reddit bitcoin post: I’m running a full node and so should you.
5. Music for blockchain research. Tchaikovsky — Capriccio Italien — Igor Manasherov, Moscow Philharmonic Orchestra, Tchaikovsky Concert hall, June 2015, Moscow.
.
.
6. Drones over the Kremlin, watch on YouTube.
.
.
7. Here is a more complete log listing for node 6 connecting to 35.226.31.206:
