Advanced Operations

This guide covers advanced operational topics for running production-grade Shardeum nodes.

1. Production Deployment Best Practices

Using systemd Service

Create a systemd service file for automatic restarts and easier management:

sudo nano /etc/systemd/system/shardeumd.service

Example service file:

[Unit]
Description=Shardeum Node
After=network-online.target
 
[Service]
User=root
ExecStart=/usr/local/bin/shardeumd start --home /root/.mainnet/node0
Restart=on-failure
RestartSec=3
LimitNOFILE=65535
 
[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl enable shardeumd
sudo systemctl start shardeumd
sudo systemctl status shardeumd

Firewall Configuration

For Full Nodes / RPC Nodes:

sudo ufw allow 27656/tcp   # P2P
sudo ufw allow 8545/tcp    # JSON-RPC (optional)
sudo ufw allow 8546/tcp    # WebSocket (optional)
sudo ufw enable

For Validators:

sudo ufw allow 27656/tcp   # P2P
sudo ufw allow 8545/tcp    # JSON-RPC (optional)
sudo ufw allow 8546/tcp    # WebSocket (optional)
sudo ufw enable

2. Monitoring and Alerting

Enable Prometheus Metrics

Edit config.toml:

prometheus = true
prometheus_listen_addr = ":26660"

Monitoring Stack Setup

Recommended tools:

• Prometheus - Metrics collection
• Grafana - Visualization dashboards
• Alertmanager - Alert notifications

Key metrics to monitor:

• Sync status
• Block height
• Validator jail status
• Disk space usage
• Memory usage
• CPU usage
• Missed blocks
• Peer count
• Network latency

Alert Conditions

Set up alerts for:

• Node falls behind by more than 100 blocks
• Validator is jailed
• Disk usage exceeds 80%
• Memory usage exceeds 90%
• Peer count drops below 5
• Node stops producing blocks (for validators)

3. Security Best Practices

Sentry Node Architecture

A recommended production setup for validators:

Internet → Sentry Nodes (Public) → Validator (Private IP only)

Benefits:

• Hides validator's IP address
• Absorbs DDoS traffic
• Reduces attack surface
• Improves security

Configuration:

1. Run validator on private network
2. Connect validator only to sentry nodes
3. Configure sentry nodes with public IPs
4. Update persistent_peers to point validator at sentries

Key Management System (KMS)

For enhanced security, consider:

• Tendermint KMS for validator key management
• Hardware Security Modules (HSM) for key storage
• YubiHSM2 integration
• Remote signing capabilities

Security Checklist

• ✅ Use firewall rules to restrict access
• ✅ Disable SSH password authentication (use keys only)
• ✅ Keep system packages updated
• ✅ Use fail2ban or similar intrusion prevention
• ✅ Implement DDoS protection
• ✅ Regular security audits
• ✅ Monitor logs for suspicious activity
• ✅ Use VPN for administrative access

4. Backup and Recovery

Critical Files to Back Up

Validator-specific:

~/.mainnet/$NODE_ID/config/priv_validator_key.json
~/.mainnet/$NODE_ID/data/priv_validator_state.json

All nodes:

~/.mainnet/$NODE_ID/config/node_key.json
~/.mainnet/$NODE_ID/config/config.toml
~/.mainnet/$NODE_ID/config/app.toml

Wallet keys:

# Mnemonic phrase (keep offline and secure)

Backup Script Example

#!/bin/bash
NODE_ID="node0"
BACKUP_DIR="/secure/backup/location"
DATE=$(date +%Y%m%d_%H%M%S)
 
# Create backup directory
mkdir -p $BACKUP_DIR/$DATE
 
# Backup critical files
cp ~/.mainnet/$NODE_ID/config/priv_validator_key.json $BACKUP_DIR/$DATE/
cp ~/.mainnet/$NODE_ID/config/node_key.json $BACKUP_DIR/$DATE/
cp ~/.mainnet/$NODE_ID/config/*.toml $BACKUP_DIR/$DATE/
 
# Create encrypted archive
tar -czf $BACKUP_DIR/backup_$DATE.tar.gz -C $BACKUP_DIR $DATE
rm -rf $BACKUP_DIR/$DATE
 
echo "Backup completed: backup_$DATE.tar.gz"

Disaster Recovery

If validator key is compromised:

1. Immediately unbond and remove validator
2. Generate new keys
3. Create new validator
4. Report incident to network

If node fails:

1. Deploy new server with identical configuration
2. Restore backup files
3. Sync node to current block height
4. Unjail validator if necessary

Unjailing a validator

If your validator is jailed, first make sure the node is running correctly and using the currently supported Shardeum mainnet release. Once the underlying issue has been resolved, submit an unjail transaction using your operator key.

export SHARDEUM_NETWORK=mainnet
export SHARDEUM_CHAIN_ID=shardeum_8118-1
export SHARDEUM_CONFIG_DIR=<shardeum-config-dir>

shardeumd tx slashing unjail --from <operator-key> \
  --chain-id shardeum_8118-1 \
  --node https://rpc.shardeum.org:443 \
  --gas auto \
  --gas-adjustment 1.3 \
  --gas-prices 2048130280389041ashm \
  --yes

To find your operator key:

shardeumd keys list \
  --keyring-backend [test|os|file] \
  --home <home-path>

Note: <home-path> should be the same node home directory used when starting the validator. Use the same keyring backend that was used when the operator key was created.

5. Performance Optimization

Pruning Strategies

Full nodes (custom pruning):

pruning = "custom"
pruning-keep-recent = "10000"
pruning-interval = "50"

Archive nodes (no pruning):

--pruning nothing

Validators:

• Use minimal pruning or default settings
• Avoid aggressive pruning to maintain full state

Database Optimization

Enable state sync for faster initial sync:

Edit config.toml:

[statesync]
enable = true
rpc_servers = "rpc1.shardeum.org:26657,rpc2.shardeum.org:26657"
trust_height = <recent_height>
trust_hash = "<block_hash>"

Hardware Tuning

SSD optimization:

# Enable TRIM
sudo systemctl enable fstrim.timer
 
# Check I/O scheduler
cat /sys/block/nvme0n1/queue/scheduler

Network tuning:

# Increase network buffers
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728

6. Scaling RPC Infrastructure

Load Balancing

For high-traffic dApps:

• Use Nginx, HAProxy, or AWS ELB
• Run multiple RPC nodes behind a reverse proxy
• Implement rate limiting to avoid overload
• Separate "public RPC" from "private infra RPC"

Example Nginx configuration:

upstream rpc_backend {
    least_conn;
    server 10.0.1.10:8545;
    server 10.0.1.11:8545;
    server 10.0.1.12:8545;
}
 
server {
    listen 80;
    server_name rpc.example.com;
 
    location / {
        proxy_pass http://rpc_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
 
        # Rate limiting
        limit_req zone=rpc_limit burst=10 nodelay;
    }
}

Caching Strategies

• Cache common queries (latest block, chain ID)
• Use Redis for query caching
• Implement CDN for static responses

7. Logging and Debugging

Viewing Logs

If using systemd:

journalctl -u shardeumd -f
journalctl -u shardeumd --since "1 hour ago"

If running manually:

tail -f ~/.mainnet/$NODE_ID/node.log

Debug Mode

Enable verbose logging in config.toml:

log_level = "debug"

Common Debug Commands

# Check sync status
shardeumd status | jq '.SyncInfo'
 
# Check peer connections
curl -s http://localhost:26657/net_info | jq '.result.n_peers'
 
# Query consensus state
curl -s http://localhost:26657/consensus_state
 
# Check validator signing info
shardeumd query slashing signing-info $(shardeumd comet show-validator)

8. Upgrade Procedures

Coordinated Network Upgrades

Preparation:

1. Monitor official Shardeum announcements for the upgrade schedule and any release-specific instructions.
2. Back up critical validator files and configuration.
3. Review the release notes and any upgrade-specific requirements.
4. Confirm that your environment meets any updated requirements before proceeding.

Upgrade steps:

1. Confirm the currently supported Shardeum mainnet release from the official Shardeum EVM releases and review any release-specific instructions.
2. Stop the validator.
3. Back up the existing binary and critical validator files.
4. Download the new release or Docker image.
5. Verify the provided checksum/digest.
6. Replace the existing binary/image.
7. Verify the installed version.
8. Restart the node.
9. Confirm the node is syncing and the validator is operating normally.
10. If the validator was jailed, complete the unjail procedure below.

Recovery and Rollback

Do not downgrade to an earlier binary unless the applicable release or network-upgrade instructions explicitly confirm that rollback is supported. Recovery requirements may vary between upgrades.

If an upgrade fails, first review the release-specific recovery instructions and official Shardeum announcements. If rollback is supported, restore the previous binary according to those instructions before restarting the validator.

9. Troubleshooting Advanced Issues

High Memory Usage

# Check memory usage
free -h
htop
 
# Restart node to clear memory
sudo systemctl restart shardeumd

Database Corruption

# Reset data (will require full resync)
shardeumd tendermint unsafe-reset-all --home ~/.mainnet/$NODE_ID
 
# Restore from snapshot (if available)
# Download snapshot and extract to data directory

Network Connectivity Issues

# Test peer connectivity
telnet <peer-ip> 27656
 
# Check firewall
sudo ufw status verbose
 
# Monitor network traffic
sudo iftop -i eth0

10. Important Resources

• Chain ID: shardeum_8118-1 (mainnet)
• EVM Chain ID: 8118 (hex: 0x1fb6)
• Official Documentation: docs.shardeum.org
• GitHub: github.com/shardeum
• Discord: Community support and announcements