Advanced Operations

This guide covers advanced operational topics for running production-grade Shardeum nodes.

1. Production Deployment Best Practices

Using systemd Service

Create a systemd service file for automatic restarts and easier management:

sudo nano /etc/systemd/system/shardeumd.service

Example service file:

[Unit]
Description=Shardeum Node
After=network-online.target
 
[Service]
User=root
ExecStart=/root/go/bin/shardeumd start --home /root/.mainnet/node0
Restart=on-failure
RestartSec=3
LimitNOFILE=65535
 
[Install]
WantedBy=multi-user.target

Use the same node-specific start arguments and configuration described in the applicable node guide.

Using systemd With Cosmovisor

If the node was configured with Cosmovisor, use the following service configuration instead of the standard configuration above so the node continues to run through Cosmovisor.

Example service file:

[Unit]
Description=Shardeum Node
After=network-online.target
 
[Service]
User=root
Environment="DAEMON_NAME=shardeumd"
Environment="DAEMON_HOME=/root/.mainnet/node0"
Environment="DAEMON_ALLOW_DOWNLOAD_BINARIES=false"
Environment="DAEMON_RESTART_AFTER_UPGRADE=true"
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/go/bin"
ExecStart=/root/go/bin/cosmovisor run start --home /root/.mainnet/node0
Restart=on-failure
RestartSec=3
LimitNOFILE=65535
 
[Install]
WantedBy=multi-user.target

Use the same node-specific start arguments and configuration described in the applicable node guide.

After saving the applicable service file, enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable shardeumd
sudo systemctl start shardeumd
sudo systemctl status shardeumd

Firewall Configuration

Before enabling UFW, allow SSH access so you do not lock yourself out of the server:

sudo ufw allow OpenSSH

For Full Nodes / RPC Nodes:

sudo ufw allow 27656/tcp   # P2P
 
# Only open these ports if the node is intended to provide public RPC access
sudo ufw allow 8545/tcp    # JSON-RPC
sudo ufw allow 8546/tcp    # WebSocket
 
sudo ufw enable
sudo ufw status

For Validators:

sudo ufw allow 27656/tcp   # P2P
sudo ufw enable
sudo ufw status

2. Monitoring and Alerting

Enable Prometheus Metrics

Edit config.toml:

prometheus = true
prometheus_listen_addr = ":26660"

Monitoring Stack Setup

Recommended tools:

• Prometheus - Metrics collection
• Grafana - Visualization dashboards
• Alertmanager - Alert notifications

Key metrics to monitor:

• Sync status
• Block height
• Validator jail status
• Disk space usage
• Memory usage
• CPU usage
• Missed blocks
• Peer count
• Network latency

Alert Conditions

Set up alerts for:

• Node falls behind by more than 100 blocks
• Validator is jailed
• Disk usage exceeds 80%
• Memory usage exceeds 90%
• Peer count drops below 5
• Node stops producing blocks (for validators)

3. Security Best Practices

Sentry Node Architecture

A recommended production setup for validators:

Internet → Sentry Nodes (Public) → Validator (Private IP only)

Benefits:

• Hides validator's IP address
• Absorbs DDoS traffic
• Reduces attack surface
• Improves security

Configuration:

1. Run validator on private network
2. Connect validator only to sentry nodes
3. Configure sentry nodes with public IPs
4. Update persistent_peers to point validator at sentries

Key Management System (KMS)

For enhanced security, consider:

• Tendermint KMS for validator key management
• Hardware Security Modules (HSM) for key storage
• YubiHSM2 integration
• Remote signing capabilities

Security Checklist

• ✅ Use firewall rules to restrict access
• ✅ Disable SSH password authentication (use keys only)
• ✅ Keep system packages updated
• ✅ Use fail2ban or similar intrusion prevention
• ✅ Implement DDoS protection
• ✅ Regular security audits
• ✅ Monitor logs for suspicious activity
• ✅ Use VPN for administrative access

4. Backup and Recovery

Critical Files to Back Up

Validator-specific:

~/.mainnet/$NODE_ID/config/priv_validator_key.json
~/.mainnet/$NODE_ID/data/priv_validator_state.json

All nodes:

~/.mainnet/$NODE_ID/config/node_key.json
~/.mainnet/$NODE_ID/config/config.toml
~/.mainnet/$NODE_ID/config/app.toml

Wallet keys:

# Mnemonic phrase (keep offline and secure)

Backup Script Example

#!/bin/bash
NODE_ID="node0"
BACKUP_DIR="/secure/backup/location"
DATE=$(date +%Y%m%d_%H%M%S)
 
# Create backup directory
mkdir -p $BACKUP_DIR/$DATE
 
# Backup critical files
cp ~/.mainnet/$NODE_ID/config/priv_validator_key.json $BACKUP_DIR/$DATE/
cp ~/.mainnet/$NODE_ID/data/priv_validator_state.json $BACKUP_DIR/$DATE/
cp ~/.mainnet/$NODE_ID/config/node_key.json $BACKUP_DIR/$DATE/
cp ~/.mainnet/$NODE_ID/config/*.toml $BACKUP_DIR/$DATE/
 
# Create compressed archive
tar -czf $BACKUP_DIR/backup_$DATE.tar.gz -C $BACKUP_DIR $DATE
rm -rf $BACKUP_DIR/$DATE
 
echo "Backup completed: backup_$DATE.tar.gz"

Disaster Recovery

If validator key is compromised:

1. Immediately unbond and remove validator
2. Generate new keys
3. Create new validator
4. Report incident to network

If node fails:

1. Deploy new server with identical configuration
2. Restore backup files
3. Sync node to current block height
4. Unjail validator if necessary

Unjailing a validator

If your validator is jailed, first make sure the node is running correctly and using the currently supported Shardeum mainnet release. Once the underlying issue has been resolved, submit an unjail transaction using your operator key.

export SHARDEUM_NETWORK=mainnet
export SHARDEUM_CHAIN_ID=shardeum_8118-1
export SHARDEUM_CONFIG_DIR=<shardeum-config-dir>

shardeumd tx slashing unjail --from <operator-key> \
  --chain-id shardeum_8118-1 \
  --node https://rpc.shardeum.org:443 \
  --gas auto \
  --gas-adjustment 1.3 \
  --gas-prices 2048130280389041ashm \
  --yes

To find your operator key:

shardeumd keys list \
  --keyring-backend [test|os|file] \
  --home <home-path>

Note: <home-path> should be the same node home directory used when starting the validator. Use the same keyring backend that was used when the operator key was created.

5. Performance Optimization

Pruning Strategies

Full nodes (custom pruning):

pruning = "custom"
pruning-keep-recent = "10000"
pruning-interval = "50"

Archive nodes (no pruning):

--pruning nothing

Validators:

• Use minimal pruning or default settings
• Avoid aggressive pruning to maintain full state

Database Optimization

Enable state sync for faster initial sync:

Edit config.toml:

[statesync]
enable = true
rpc_servers = "rpc1.shardeum.org:26657,rpc2.shardeum.org:26657"
trust_height = <recent_height>
trust_hash = "<block_hash>"

Hardware Tuning

SSD optimization:

# Enable TRIM
sudo systemctl enable fstrim.timer
 
# Check I/O scheduler
cat /sys/block/nvme0n1/queue/scheduler

Network tuning:

# Increase network buffers
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728

6. Scaling RPC Infrastructure

Load Balancing

For high-traffic dApps:

• Use Nginx, HAProxy, or AWS ELB
• Run multiple RPC nodes behind a reverse proxy
• Implement rate limiting to avoid overload
• Separate "public RPC" from "private infra RPC"

Example Nginx configuration:

upstream rpc_backend {
    least_conn;
    server 10.0.1.10:8545;
    server 10.0.1.11:8545;
    server 10.0.1.12:8545;
}
 
server {
    listen 80;
    server_name rpc.example.com;
 
    location / {
        proxy_pass http://rpc_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
 
        # Rate limiting
        limit_req zone=rpc_limit burst=10 nodelay;
    }
}

Caching Strategies

• Cache common queries (latest block, chain ID)
• Use Redis for query caching
• Implement CDN for static responses

7. Logging and Debugging

Viewing Logs

If using systemd:

journalctl -u shardeumd -f
journalctl -u shardeumd --since "1 hour ago"

If running manually:

tail -f ~/.mainnet/$NODE_ID/node.log

Debug Mode

Enable verbose logging in config.toml:

log_level = "debug"

Common Debug Commands

# Check sync status
shardeumd status | jq '.SyncInfo'
 
# Check peer connections
curl -s http://localhost:26657/net_info | jq '.result.n_peers'
 
# Query consensus state
curl -s http://localhost:26657/consensus_state
 
# Check validator signing info
shardeumd query slashing signing-info $(shardeumd comet show-validator)

8. Upgrade Procedures

Coordinated Network Upgrades

Upgrade requirements may differ depending on whether the node is managed directly with shardeumd or through Cosmovisor. Always follow the release-specific instructions for the applicable network upgrade.

If your node uses Cosmovisor, add the required upgrade binary using the exact upgrade name specified for that release and continue running the node through Cosmovisor. Do not bypass the Cosmovisor-managed binary unless the release-specific instructions explicitly require it.

Preparation:

1. Monitor official Shardeum announcements for the upgrade schedule and any release-specific instructions.
2. Back up critical node files and configuration.
3. Review the release notes and any upgrade-specific requirements.
4. Confirm that your environment meets any updated requirements before proceeding.

Manual binary upgrade steps:

1. Confirm the currently supported Shardeum mainnet release from the official Shardeum EVM releases and review any release-specific instructions.
2. Stop the node.
3. Back up the existing binary and critical node files and configuration.
4. Download the new release or Docker image.
5. Verify the provided checksum/digest.
6. Replace the existing binary/image.
7. Verify the installed version.
8. Restart the node.
9. Confirm the node is syncing and operating normally.
10. If the validator was jailed, follow the unjail procedure.

Recovery and Rollback

Do not downgrade to an earlier binary unless the applicable release or network-upgrade instructions explicitly confirm that rollback is supported. Recovery requirements may vary between upgrades.

If an upgrade fails, first review the release-specific recovery instructions and official Shardeum announcements. If rollback is supported, restore the previous binary according to those instructions before restarting the node.

9. Troubleshooting Advanced Issues

High Memory Usage

# Check memory usage
free -h
htop
 
# Restart node to clear memory
sudo systemctl restart shardeumd

Database Corruption

Resetting node data requires the node to resync.

When a compatible snapshot is available, restore from a suitable post-upgrade snapshot.

If syncing from genesis, follow the fresh-sync instructions in the applicable node setup guide. After a state-breaking upgrade, syncing from genesis may require historical binaries and Cosmovisor to switch binaries at the appropriate upgrade heights.

shardeumd tendermint unsafe-reset-all --home ~/.mainnet/$NODE_ID

Network Connectivity Issues

# Test peer connectivity
telnet <peer-ip> 27656
 
# Check firewall
sudo ufw status verbose
 
# Monitor network traffic
sudo iftop -i eth0

10. Important Resources

• Chain ID: shardeum_8118-1 (mainnet)
• EVM Chain ID: 8118 (hex: 0x1fb6)
• Official Documentation: docs.shardeum.org
• GitHub: github.com/shardeum
• Discord: Community support and announcements