Node operation and configuration

Start Node
Run docker-compose up.
1
$ docker-compose up -d
The docker ps command shows the list of running docker containers.
1
$ docker ps
2
CONTAINER ID   IMAGE                                                          COMMAND                CREATED              STATUS                          PORTS                                                                 NAMES
3
0de99e33cdc9     iconloop/prep-node:1910261021xc97f33    "/src/entrypoint.sh"      2 minutes ago        Up 2 minutes(healthy)    0.0.0.0:7100->7100/tcp, 0.0.0.0:9000->9000/tcp prep_prep_1
Corresponding columns,
Column
Description
CONTAINER ID
Container  ID
IMAGE
P-Rep Node's  image name
COMMAND
The script will be executed whenever a P-Rep Node container is run
STATUS
Healthcheck status. One of "starting" , "healthy", "unhealthy" or "none"
PORTS
Exposed ports on the running container
NAMES
Container name
You can read the container booting log from the log folder.
1
$ tail -f data/PREP-MainNet/log/booting_20190419.log
2
[2019-10-28 10:43:05.204] Your IP: xx.xx.xx.xx
3
[2019-10-28 10:43:05.209] RPC_PORT: 9000 / RPC_WORKER: 3
4
[2019-10-28 10:43:05.214] DEFAULT_PATH=/data/mainnet in Docker Container
5
[2019-10-28 10:43:05.219] DEFAULT_LOG_PATH=/data/mainnet/log
6
[2019-10-28 10:43:05.224] DEFAULT_STORAGE_PATH=/data/mainnet/.storage
7
[2019-10-28 10:43:05.229] scoreRootPath=/data/mainnet/.score_data/score
8
[2019-10-28 10:43:05.234] stateDbRootPath=/data/mainnet/.score_data/db
9
[2019-10-28 10:43:05.239] Time synchronization with NTP / NTP SERVER: time.google.com
10
[2019-10-28 10:43:12.022] P-REP package version info - _1910261021xc97f33
11
[2019-10-28 10:43:12.853] iconcommons             1.1.2
12
iconrpcserver           1.4.5
13
iconsdk                 1.2.0
14
iconservice             1.5.15
15
loopchain               2.4.16
Entrypoint.sh Diagram
​
Stop Node
1
$ docker-compose down
View Node Status
Check the current state and information of the prep-node
API Checks
The current state of a node can be confirmed by /api/v1/status/peer and /api/v1/avail/peer API functions. The response data are the same, but HTTP response codes are different.
1
$ curl localhost:9000/api/v1/status/peer
2
​
3
{
4
  "made_block_count": 0,
5
  "leader_made_block_count": 7,
6
  "status": "Service is online: 0",
7
  "state": "Watch",
8
  "service_available": true,
9
  "peer_type": "0",
10
  "audience_count": "0",
11
  "consensus": "siever",
12
  "peer_id": "hx72bff0f887ef183bde1391dc61375f096e75c74a",
13
  "block_height": 10106994,
14
  "round": 0,
15
  "epoch_height": 10106995,
16
  "unconfirmed_block_height": -1,
17
  "total_tx": 72003516,
18
  "unconfirmed_tx": 0,
19
  "peer_target": "xxx.xxx.xxx.xxx:7100",
20
  "leader_complaint": 1,
21
  "peer_count": 24,
22
  "leader": "hxc9e36a98a3fca0b636eb822ff5a96db658e4bb88",
23
  "epoch_leader": "hxc9e36a98a3fca0b636eb822ff5a96db658e4bb88",
24
  "versions": {
25
    "loopchain": "2.4.16",
26
    "iconservice": "1.5.15",
27
    "iconrpcserver": "1.4.5",
28
    "iconcommons": "1.1.2",
29
    "earlgrey": "0.0.4"
30
  },
31
  "mq": {
32
    "peer": {
33
      "message_count": 0
34
    },
35
    "channel": {
36
      "message_count": 0
37
    },
38
    "score": {
39
      "message_count": 0
40
    }
41
  }
42
}
/api/v1/avail/peer returns HTTP response 503 when the service is unavailable  
This is useful when performing a health check based on the HTTP response code of the load balancer.
1
#Return HTTP 503 Service Unavailable 
2
​
3
state : InitComponents, EvaluateNetwork, BlockSync, SubscribeNetwork
/api/v1/status/peer returns 200 OK at BlockSync
1
$ curl -i localhost:9000/api/v1/status/peer
2
HTTP/1.1 200 OK
3
Connection: close
4
Access-Control-Allow-Origin: *
5
Content-Length: 573
6
Content-Type: application/json
7
​
8
{
9
  "made_block_count": 0,
10
  "leader_made_block_count": 7,
11
  "status": "Service is online: 0",
12
  "state": "Watch",
13
  "service_available": true,
14
  "peer_type": "0",
15
  "audience_count": "0",
16
  "consensus": "siever",
17
  "peer_id": "hx72bff0f887ef183bde1391dc61375f096e75c74a",
18
  "block_height": 10106994,
19
  "round": 0,
20
  "epoch_height": 10106995,
21
  "unconfirmed_block_height": -1,
22
  "total_tx": 72003516,
23
  "unconfirmed_tx": 0,
24
  "peer_target": "xxx.xxx.xxx.xxx:7100",
25
  "leader_complaint": 1,
26
  "peer_count": 24,
27
  "leader": "hxc9e36a98a3fca0b636eb822ff5a96db658e4bb88",
28
  "epoch_leader": "hxc9e36a98a3fca0b636eb822ff5a96db658e4bb88",
29
  "versions": {
30
    "loopchain": "2.4.16",
31
    "iconservice": "1.5.15",
32
    "iconrpcserver": "1.4.5",
33
    "iconcommons": "1.1.2",
34
    "earlgrey": "0.0.4"
35
  },
36
  "mq": {
37
    "peer": {
38
      "message_count": 0
39
    },
40
    "channel": {
41
      "message_count": 0
42
    },
43
    "score": {
44
      "message_count": 0
45
    }
46
  }
47
}
/api/v1/avail/peer returns 503 OK at BlockSync
1
$ curl -i localhost:9000/api/v1/avail/peer
2
HTTP/1.1 503 Service Unavailable
3
Connection: close
4
Access-Control-Allow-Origin: *
5
Content-Length: 573
6
Content-Type: application/json
7
​
8
{
9
  "made_block_count": 0,
10
  "leader_made_block_count": 7,
11
  "status": "Service is online: 0",
12
  "state": "Watch",
13
  "service_available": true,
14
  "peer_type": "0",
15
  "audience_count": "0",
16
  "consensus": "siever",
17
  "peer_id": "hx72bff0f887ef183bde1391dc61375f096e75c74a",
18
  "block_height": 10106994,
19
  "round": 0,
20
  "epoch_height": 10106995,
21
  "unconfirmed_block_height": -1,
22
  "total_tx": 72003516,
23
  "unconfirmed_tx": 0,
24
  "peer_target": "xxx.xxx.xxx.xxx:7100",
25
  "leader_complaint": 1,
26
  "peer_count": 24,
27
  "leader": "hxc9e36a98a3fca0b636eb822ff5a96db658e4bb88",
28
  "epoch_leader": "hxc9e36a98a3fca0b636eb822ff5a96db658e4bb88",
29
  "versions": {
30
    "loopchain": "2.4.16",
31
    "iconservice": "1.5.15",
32
    "iconrpcserver": "1.4.5",
33
    "iconcommons": "1.1.2",
34
    "earlgrey": "0.0.4"
35
  },
36
  "mq": {
37
    "peer": {
38
      "message_count": 0
39
    },
40
    "channel": {
41
      "message_count": 0
42
    },
43
    "score": {
44
      "message_count": 0
45
    }
46
  }
47
}
Node Status Detail
value name
Description
Reason or allowed value
made_block_count
number of block generated by node
Reset when node became a leader after a rotation or reorerated.
status
service on / off status
"Service is online: 1" : Working (leader status)
 "Service is online: 0" : Working (not a leader status)
 "Service is offline: block height sync" : block sync condition
 "Service is offline: mq down" ": channel mq issue
state
node condition
detailed information (display at bottom)
service_available
service condition
[ true / false ]
peer_type
classify leader node and verifying node
"0" : verifying node
 "1" : leader node
audience_count
DEPRECATION, It will be remove
​
consensus
consensus algorithm
"siever" : current consensus algorithm
 "LFT" : consensus algorithm that will adapt in the future
peer_id
unique address of node
40 digit HEX string
block_height
Current block height of node
​
round
number of counts for current block consensus process
The count increases by one when leader node fails to process transaction and leader complaint is occured or consensus is failed. The count reset when block generation is complete
epoch_height
block heights of processing block. In case of citizen nodes, their block heights stop at SubscribeNetwork block heights because they don’t participate in consensus.
​
unconfirmed_block_height
block heights of unprocessed block
same as epoch_height
total_tx
Total number of tx until current block
​
unconfirmed_tx
number of unprocessed tx that hold by queue
 if it holds by leader more than one minute leader complaint will occur
​
peer_target
IP address and port of node
"IP:PORT"
leader_complaint
DEPRECATION, It will be remove
​
peer_count
total number of nodes in blockchain network
​
leader
Unique address of a leader node
40 digit HEX string
epoch_leader
Unique address of a leader node in consensus process
40 digit HEX string
mq.peer.message_count
Accumulated number of messages in pier MQ
Presents ‘-1’ when an error occurs and details can be found in "error"
mq.channel.message_count
Accumulated number of messages in channel MQ
Presents ‘-1’ when an error occurs and details can be found in "error"
mq.score.message_count
Accumulated number of messages in SCORE MQ
Presents ‘-1’ when an error occurs and details can be found in "error"
State Detail
State Value
Description
InitComponents
Channel Service initial state
Consensus
Loopchain Consensus begins. Convert to BlockHeightSync automatically
BlockHeightSync
Block height Sync state begins. Convert to EvaluateNetwork automatically
EvaluateNetwork
Evaluate the BlockSync state by checking the network status
BlockSync
Block Sync loop.
SubscribeNetwork
Determine the type depending on the node type. A citizen node requests the block generation message to a parent node.
Watch
Citizen node default state. Relay transaction and sync the blocks created by a parent node
Vote
Status of validating and voting blocks created by the leader
LeaderComplain
Status of requesting leader complain for current leader and wait for complain
BlockGenerate
Status of leader creating a block
GracefulShutdown
End Process
Docker Environment Variables
If you want to change the TimeZone, open docker-compose.yml in a text editor and add the following content:
1
version: "3"
2
services:
3
  prep-node:
4
     image: "iconloop/prep-node:1910261021xc97f33"
5
     container_name: "prep-mainnet"
6
     network_mode: host     
7
     restart: "always"
8
     environment:
9
        NETWORK_ENV: "mainnet"
10
        SWITCH_BH_VERSION3: "10324749"
11
        CERT_PATH: "/cert"
12
        LOOPCHAIN_LOG_LEVEL: "DEBUG"
13
        ICON_LOG_LEVEL: "DEBUG"        
14
        FASTEST_START: "yes" # Restore from lastest snapshot DB
15
        PRIVATE_KEY_FILENAME: "{YOUR_KEYSTORE or YOUR_CERTKEY FILENAME}" # only file name
16
        PRIVATE_PASSWORD: "{YOUR_KEY_PASSWORD}"
17
        TZ: "America/Los_Angeles" # Added content to change the TimeZone
18
     cap_add:
19
        - SYS_TIME      
20
     volumes:
21
        - ./data:/data # mount a data volumes
22
        - ./cert:/cert # Automatically generate cert key files here
23
     ports:
24
        - 9000:9000
25
        - 7100:7100
The P-Rep Node image supports the following environment variables:
Environment variable
Description
Default value
Allowed value
​
IPADDR
Setting the IP address
$EXT_IPADDR
​
​
LOCAL_TEST
false
false
​
​
TZ
Setting the TimeZone Environment
Asia/Seoul
​List of TZ name​
​
NETWORK_ENV
Network Environment name
PREP-TestNet
mainnet, PREP-TestNet
​
SERVICE
Service Name
default
​
​
SERVICE_API
SERVICE_API URI
​https://${SERVICE}.net.solidwallet.io/api/v3​
URI
​
ENDPOINT_URL
ENDPOINT API URI
​https://${SERVICE}.net.solidwallet.io​
URI
​
NTP_SERVER
NTP SERVER ADDRESS
time.google.com
​
​
NTP_REFRESH_TIME
NTP refresh time
21600
​
​
FIND_NEIGHBOR
Find fastest neighborhood PRrep
false
​
​
DEFAULT_PATH
Setting the Default Root PATH
/data/${NETWORK_ENV}
​
​
DEFAULT_LOG_PATH
Setting the logging path
${DEFAULT_PATH}/log
​
​
DEFAULT_STORAGE_PATH
block DB will be stored
${DEFAULT_PATH}/.storage
​
​
USE_NAT
if you want to use NAT Network
no
​
​
NETWORK_NAME
​
​
​
​
VIEW_CONFIG
for check deployment state
false
boolean (true/false)
​
USE_MQ_ADMIN
Enable RabbitMQ management Web interface.The management UI can be accessed using a Web browser at http://{node-hostname}:15672/​
false
boolean (true/false)
​
MQ_ADMIN
RabbitMQ management username
admin
​
​
MQ_PASSWORD
RabbitMQ management password
iamicon
​
​
LOOPCHAIN_LOG_LEVEL
loopchain log level
INFO
DEBUG, INFO, WARNING, ERROR
​
ICON_LOG_LEVEL
iconservice log level
INFO
DEBUG, INFO, WARNING, ERROR
​
LOG_OUTPUT_TYPE
loopchain's output log type
file
file, console
​
outputType
iconservice's output log type
$LOG_OUTPUT_TYPE
file, console
​
CONF_PATH
Setting the configure file path
/${APP_DIR}/conf
​
​
CERT_PATH
Setting the certificate key file path
/${APP_DIR}/cert
​
​
REDIRECT_PROTOCOL
http
http
​
​
SUBSCRIBE_USE_HTTPS
false
false
​
​
ICON_NID
Setting the ICON Network ID number
0x50
​
​
ALLOW_MAKE_EMPTY_BLOCK
false
false
​
​
score_fee
true
true
​
​
score_audit
false
false
​
​
scoreRootPath
${DEFAULT_PATH}/.score_data/score
${DEFAULT_PATH}/.score_data/score
​
​
stateDbRootPath
${DEFAULT_PATH}/.score_data/db
${DEFAULT_PATH}/.score_data/db
​
​
iissDbRootPath
${DEFAULT_PATH}/.iissDb
${DEFAULT_PATH}/.iissDb
​
​
CHANNEL_BUILTIN
boolean (true/false)
true
​
​
PEER_NAME
uname
uname
​
​
PUBLIC_PATH
public cert key location
${CERT_PATH}/${IPADDR}_public.der
​
​
PRIVATE_PATH
private cert key location
${CERT_PATH}/${IPADDR}_private.der
​
​
PRIVATE_PASSWORD
private cert key password
test
​
​
LOAD_PEERS_FROM_IISS
true
true
​
​
CHANNEL_MANAGE_DATA_PATH
${CONF_PATH}/channel_manange_data.json
${CONF_PATH}/channel_manange_data.json
​
​
CONFIG_API_SERVER
​https://download.solidwallet.io​
​https://download.solidwallet.io​
​
​
GENESIS_DATA_PATH
${CONF_PATH}/genesis.json
${CONF_PATH}/genesis.json
​
​
BLOCK_VERSIONS
​
​
​
​
NEXT_BLOCK_VERSION_HEIGHT
​
​
​
​
FORCE_RUN_MODE
Setting the loopchain running parameter e.g. if FORCE_RUN_MODE is
​
​
​
configure_json
${CONF_PATH}/configure.json
${CONF_PATH}/configure.json
​
​
iconservice_json
${CONF_PATH}/iconservice.json
${CONF_PATH}/iconservice.json
​
​
iconrpcserver_json
${CONF_PATH}/iconrpcserver.json
${CONF_PATH}/iconrpcserver.json
​
​
ICON_REVISION
4
4
​
​
ROLE_SWITCH_BLOCK_HEIGHT
1
1
​
​
mainPRepCount
22
22
​
​
mainAndSubPRepCount
100
100
​
​
decentralizeTrigger
0.002
0.002
​
​
RPC_PORT
Choose a RPC service port
9000
​
​
RPC_WORKER
Setting the number of RPC workers
3
​
​
RPC_GRACEFUL_TIMEOUT
rpc graceful timeout
0
​
​
PRIVATE_KEY_FILENAME
YOUR_KEYSTORE or YOUR_CERTKEY FILENAME
​
FileName
​
PRIVATE_PASSWORD
YOUR_KEY_PASSWORD
​
String
​
Troubleshooting
Q: How to check if a container is running or not?
The docker ps command shows the list of running docker containers.
1
$ docker ps
2
CONTAINER ID   IMAGE                                                          COMMAND                CREATED              STATUS                          PORTS                                                                 NAMES
3
0de99e33cdc9     iconloop/prep-node:1910261021xc97f33    "/src/entrypoint.sh"      2 minutes ago        Up 2 minutes(healthy)    0.0.0.0:7100->7100/tcp, 0.0.0.0:9000->9000/tcp prep_prep_1
You should look under the STATUS field to see if the container is up and in healthy state.
Inside the container, there is a healthcheck script running with the following configuration. It will return unhealthy when it fails.
Healthcheck option
value
retries
4
interval
30s
timeout
20s
start-period
60s
The container can have three states:
starting - container just starts
healthy - when the health check passes
unhealthy - when the health check fails
If the container does not start properly or went down unexpectedly, please check the booting.log. Below is the log messages on success.
1
$ cat data/PREP-MainNet/log/booting_${DATE}.log 
2
​
3
[2019-10-28 10:43:05.204] Your IP: xxx.xxx.xxx.xxx
4
[2019-10-28 10:43:05.209] RPC_PORT: 9000 / RPC_WORKER: 3
5
[2019-10-28 10:43:05.214] DEFAULT_PATH=/data/mainnet in Docker Container
6
[2019-10-28 10:43:05.219] DEFAULT_LOG_PATH=/data/mainnet/log
7
[2019-10-28 10:43:05.224] DEFAULT_STORAGE_PATH=/data/mainnet/.storage
8
[2019-10-28 10:43:05.229] scoreRootPath=/data/mainnet/.score_data/score
9
[2019-10-28 10:43:05.234] stateDbRootPath=/data/mainnet/.score_data/db
10
[2019-10-28 10:43:05.239] Time synchronization with NTP / NTP SERVER: time.google.com
11
[2019-10-28 10:43:12.022] P-REP package version info - _1910261021xc97f33
12
[2019-10-28 10:43:12.853] iconcommons             1.1.2
13
iconrpcserver           1.4.5
14
iconsdk                 1.2.0
15
iconservice             1.5.15
16
loopchain               2.4.16
Q: How to locate errors from logs?
Error log messages example
Grep the ERROR messages from the log files to find the possible cause of the failure.
1
$ cat data/PREP-MainNet/log/booting_${DATE}.log | grep ERROR
2
​
3
[2019-08-12 02:08:48.746] [ERROR] Download Failed - http://20.20.1.149:5000/cert/20.20.1.195_public.der status_code=000
4
​
5
[2019-08-12 01:58:46.439] [ERROR] Unauthorized IP address, Please contact our support team
Docker container generates these log files below
booting.log
The log file contains the errors that occurred when the docker container starts up.
iconrpcserver.log
The log file contains information about the request/response message handling going through the iconrpcserver. 
iconservice.log
The log file contains the internals of ICON Service
loopchain.channel-txcreator-icon_dex_broadcast.icon_dex.log
The log file contains information about TX broadcast from a node to other nodes
loopchain.channel-txcreator.icon_dex.log
The log file contains information about the process of confirming TXƒ
loopchain.channel-txreceiver.icon_dex.log
The log file contains information about receiving the broadcasted TX from a node.
loopchain.channel.icon_dex.log
The log file contains information about internals of loopchain engine
Q: How to monitor different resources?
We recommend the following tools for resource monitoring
1.
Network monitoring - iftop, nethogs, vnstat
2.
CPU/Memory monitoring - top, htop
3.
Disk I/O monitoring - iostat, iotop
4.
Docker monitoring - docker stats, ctop
ICON Node - Previous
Maintenance
Understanding log files
Copy link
Contents
Start Node
Entrypoint.sh Diagram
Docker Environment Variables
Troubleshooting