PolkaScan block explorer configured to show Kusama network

In this post we are goind to learn how to build your own Kusama (https://kusama.network) block explorer using PolkaScan software stack (https://github.com/polkascan/polkascan-pre). This is what decentralization it's about!

We'll use Ubuntu Server 18.04.3 LTS for this tutorial, althought you can use any other distro that support dockers.

Hardware requirements

For my tests i'd used a 4 vCPU / 24 GB RAM / 100 GB HDD VPS server.

Let's go

Login to your server and execute the following commands as root:

Upgrade!

Fisrt of all let's upgrade Ubuntu:

cd
apt update -y
apt upgrade -y

Install docker

apt install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
apt update -y
apt install -y docker-ce

Install docker compose

curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version

Build PolkaScan

git clone https://github.com/polkascan/polkascan-pre.git
cd polkascan-pre
git submodule update --init --recursive

Edit docker-compose.kusama.yml

nano docker-compose.kusama.yml

In explorer-gui section set API_URL: http://SERVER_IP_ADDRESS:8080/api/v1/, su sure to replace SERVER_IP_ADDRESS with the public ip address of your server.

Build & run docker

docker-compose -p dev -f docker-compose.kusama.yml up -d mysql
docker-compose -p dev -f docker-compose.kusama.yml up --build

Logs

When the build process ends the harvester will start to harvest blocks to extract all the information and place it in the MySQL database. In out test the process can take around 4 hours to sinchronize 45,000 blocks.

You'll see three different log entries:

Substrate node log entries:

substrate-node_1     | 2019-08-26 18:15:14 Idle (19 peers), best: #48278 (0x16f6…ec84), finalized #38322 (0xc883…7412), ⬇ 14.4kiB/s ⬆ 12.6kiB/s

Explorer GUI log entries:

explorer-gui_1       | 87.223.169.39 - - [26/Aug/2019:18:15:15 +0000] "OPTIONS /api/v1/networkstats/latest HTTP/1.1" 200 0 "http://kusamaexplorer.polkastats.io/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" "-"

Scheduler log entries:

harvester-beat_1     | [2019-08-26 18:15:19,730: INFO/MainProcess] Scheduler: Sending due task check-head-10-seconds (app.tasks.start_harvester)

Harvester workers log entries:

harvester-worker_1   | [2019-08-26 18:15:10,039: WARNING/ForkPoolWorker-2] + Added 0x9a04b6b7cc64f03d521306b6336b0ef95c922993f4dd086e28f5a9fa58cd049b

Open your awesome new kusama block explorer

Open a browser and point to URL http://SERVER_IP_ADDRESS:8080/ to see your new block explorer in action.

Useful commands

Check Substrate logs

docker container logs --follow --tail 10 $(docker container ls | grep "polkasource/substrate-client:kusama-latest" | cut -d " " -f1)

Output:

2019-08-26 18:28:44 Idle (18 peers), best: #48413 (0x3753…b971), finalized #38322 (0xc883…7412), ⬇ 20.7kiB/s ⬆ 18.6kiB/s
2019-08-26 18:28:48 Imported #48414 (0xdfc0…15e1)
2019-08-26 18:28:49 Idle (18 peers), best: #48414 (0xdfc0…15e1), finalized #38322 (0xc883…7412), ⬇ 19.0kiB/s ⬆ 18.7kiB/s
2019-08-26 18:28:54 Imported #48414 (0xc62e…2501)
2019-08-26 18:28:54 Idle (18 peers), best: #48414 (0xdfc0…15e1), finalized #38322 (0xc883…7412), ⬇ 14.8kiB/s ⬆ 15.2kiB/s
2019-08-26 18:28:58 Discovered new external address for our node: /ip4/10.0.1.203/tcp/30333/p2p/QmZ7cnG9RjWHBZ9iEDGz8NxM94EMfb1Btp11R7Yu2a5waW

Check GUI logs

docker container logs --follow --tail 10 $(docker container ls | grep "polkascan/pre-explorer-gui:latest" | cut -d " " -f1)

Output:

87.223.169.39 - - [26/Aug/2019:18:30:30 +0000] "OPTIONS /api/v1/balances/transfer HTTP/1.1" 200 0 "http://kusamaexplorer.polkastats.io/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" "-"
87.223.169.39 - - [26/Aug/2019:18:30:30 +0000] "GET /api/v1/networkstats/latest HTTP/1.1" 200 330 "http://kusamaexplorer.polkastats.io/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" "-"
87.223.169.39 - - [26/Aug/2019:18:30:30 +0000] "GET /api/v1/block HTTP/1.1" 200 36165 "http://kusamaexplorer.polkastats.io/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36" "-"

Check harvester worker logs

docker container logs --follow --tail 10 $(docker container ls | grep " polkascan/pre-harvester:latest" | grep "celery" | head -n1 | cut -d " " -f1)

Output:

[2019-08-26 18:33:59,813: INFO/MainProcess] Received task: app.tasks.accumulate_block_recursive[58acec04-a1fc-489f-b2d3-32077620129a]
[2019-08-26 18:33:59,963: WARNING/ForkPoolWorker-1] + Added 0xa567215867085a93b6b9c256ece22806887399dbe60e004240ee31ba7067025e
[2019-08-26 18:34:00,003: WARNING/ForkPoolWorker-1] . Skipped 0xdbc88b4a131d982f446b4bcaef76e7ebf6db4670866c4b8d401e7647cfa31d4d

Check scheduler logs

docker container logs --follow --tail 10 $(docker container ls | grep " polkascan/pre-harvester:latest" | grep "celery" | tail -n1 | cut -d " " -f1)

Output:

[2019-08-26 18:32:49,785: INFO/MainProcess] Scheduler: Sending due task check-head-10-seconds (app.tasks.start_harvester)
[2019-08-26 18:32:59,785: INFO/MainProcess] Scheduler: Sending due task check-head-10-seconds (app.tasks.start_harvester)
[2019-08-26 18:33:09,786: INFO/MainProcess] Scheduler: Sending due task check-head-10-seconds (app.tasks.start_harvester)

Connect to MySQL database

docker exec -it $(docker container ls | grep "mysql:latest" | cut -d " " -f1) mysql -u root -proot

Output:

mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 18638
Server version: 8.0.17 MySQL Community Server - GPL

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use polkascan;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select id, hash from data_block order by id as limit 1;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'as limit 1' at line 1
mysql> select id, hash from data_block order by id asc limit 1;
+----+--------------------------------------------------------------------+
| id | hash                                                               |
+----+--------------------------------------------------------------------+
|  0 | 0x3fd7b9eb6a00376e5be61f01abb429ffb0b104be05eaff4d458da48fcd425baf |
+----+--------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> select id, hash from data_block order by id desc limit 1;
+-------+--------------------------------------------------------------------+
| id    | hash                                                               |
+-------+--------------------------------------------------------------------+
| 47722 | 0x0002615730af83a6b4021fc707852ef605b57943f027246b3a318c9e450eebf7 |
+-------+--------------------------------------------------------------------+
1 row in set (0.01 sec)

References

https://kusama.network/

https://github.com/polkascan/polkascan-pre