Consensus and Sharding Mechanism

Why are they so important?

Shardeum's approach to consensus and sharding involves complex mechanisms to ensure network robustness and efficiency. The consensus mechanism is deeply connected with the network's dynamic sharding system, allowing for effective node coordination and lifecycle management. Here you can explore what sharding is in-depth.

1. Consensus Mechanism:

• The Shardeum network utilizes a trustless consensus system where nodes must agree on the data before it becomes part of the next cycle.
• This process uses advanced algorithms to ensure that all nodes reach a consensus on the transaction records.
• Each node generates cycle records, akin to traditional blockchain blocks but are unique to Shardeum's structure.
• Here you can take a deep dive into the consensus algorithms used by Shardeum and Proof of Quorum, focusing on their design and implementation.

2. Node Lifecycle Management:

• Node Awareness: In Shardeum, every active validator is aware of all other active validators. This comprehensive awareness is crucial for maintaining deterministic sharding and ensuring accurate consensus.
• Lifecycle: Nodes in Shardeum transition from a single seed node to a network full of active and standby nodes. Active nodes handle transaction processing, while standby nodes are prepared to become active as needed, enhancing network resilience.
• Selection Process: The transition from standby to active nodes is determined by a deterministic lottery, ensuring randomness and reducing the risk of manipulation.
• Here you can delve deeply into the Shardeum Node Lifecycle.

3. Dynamic Sharding Approach:

• Shardeum employs dynamic sharding to distribute data and computational tasks across multiple nodes efficiently. This system adjusts the shard allocation based on the network's needs, optimizing both storage and processing resources.
• Shards are managed so that no single node has a complete set of data, enhancing security and performance. The network's sharding mechanism is designed to adapt to changes in node count and transaction volume, allowing for scalable and responsive network operations.
• These combined mechanisms enable Shardeum to manage a large network of nodes, handle high transaction volumes, and maintain security and consistency across the distributed ledger.
• You may want to see this page, if you want to see more technical details, regardless how data and transaction sharding are implemented, including the mathematical foundations.

Sharding

Sharding is a method of partitioning data and computation to achieve scalability, allowing a network to handle more transactions per second (TPS) as it grows. Shardeum's sharding mechanism dynamically distributes data and transaction processing across multiple nodes, ensuring efficient resource utilization and maintaining low fees.

2. Access Lists and Memory Control

• Access lists are used to manage and control memory access during transaction execution. In Shardeum, each transaction must specify the parts of memory it will read from or write to. This pre-determined list of memory locations, known as an access list, allows the system to predict and schedule transactions efficiently, avoiding conflicts and ensuring that transactions touching the same memory locations are handled correctly. This mechanism was introduced because the EVM lacks built-in concurrency primitives, requiring external control over memory access to manage concurrency effectively.
• Automatic Access List Generation: Shardeum employs an automatic access list generation mechanism to streamline this process. When a transaction is initiated, the system automatically determines which memory locations will be accessed. This includes both the data that will be read and the data that will be modified. This automatic generation helps to maintain consistency and predictability in transaction execution. The system can compute the access list before running the transaction, although this can be challenging due to the dynamic nature of certain operations, such as generating random numbers based on block data or gas prices.
• Concurrency Control: A major challenge in a sharded network like Shardeum's is ensuring that multiple transactions do not conflict when accessing the same memory locations. Shardeum addresses this by using access lists to control concurrency. If two transactions attempt to modify the same memory location simultaneously, the system schedules them in a way that ensures one completes before the other begins, thereby maintaining data integrity. This scheduling is crucial for preventing nondeterministic behavior in transaction execution.
• Retry and Recirculation Mechanisms: To enhance user experience, Shardeum includes retry and recirculation mechanisms for transactions that fail due to unexpected memory access issues. Instead of immediately rejecting such transactions, the system can retry them or recirculate them until they can be executed successfully.
• Gas Pricing and Optimization: Access lists also facilitate more sophisticated gas pricing mechanisms. By analyzing access patterns and memory usage, Shardeum can dynamically adjust gas prices to reflect the actual computational and storage costs. For example, operations that access multiple storage locations might incur higher fees than those that only access a few. This dynamic pricing helps to balance load across the network and prevent any single transaction type from disproportionately consuming resources. Also by knowing in advance which parts of memory will be accessed, gas estimation can be done more accurately.
• Real-World Example: Dynamic Recipient Transactions: Consider a transaction where the recipient is determined dynamically. For instance, a smart contract might generate a random number based on the block data and current gas price. If the random number is greater than 0.5, the funds are sent to Bob; if it is less than 0.5, the funds are sent to Sally. This transaction's access list must account for both possible recipients' addresses. The system pre-runs the transaction to generate the access list, but the actual execution might still encounter new memory requirements due to the dynamic nature of the recipient address.
• Surprise Memory Access: Shardeum supports a technical design for handling surprise memory access during transaction execution. If a transaction needs to access a memory location not included in its initial access list, it can request permission at the protocol level. This request is processed as a transaction where nodes in the network check if the new memory access can be granted without disrupting the order of existing transactions. If approved, the transaction can proceed; if not, it is retried or recirculated without user intervention to avoid a poor user experience.
• Security Considerations: Access lists also enhance security by preventing unauthorized memory access. Transactions are restricted to accessing only the memory locations specified in their access lists. If a transaction attempts to access a location not listed, it is immediately flagged and halted. This mechanism prevents potential exploits where a malicious transaction might try to read or modify data it should not have access to.

3. Access List Warm-Up Technology

• The access list warm-up process involves running a preliminary phase where hints about the memory locations that will be accessed are gathered. This phase is used to parallelize data fetching, significantly reducing the time required to generate access lists. By warming up memory slots, Shardeum can efficiently prepare for transaction execution, minimizing delays caused by serial data fetches.
• Performance Improvements: This technology allows for a significant reduction in access list generation times. During testing, complex transactions that previously took up to 50 seconds to generate an access list saw a drastic reduction in processing time to just a few seconds. This improvement is critical for maintaining high throughput and low latency in the network, especially for transactions that involve numerous memory accesses.
• Security Enhancements: Access list warm-up also enhances security by decoupling the process of access list generation from the actual transaction execution. This separation ensures that potential security risks associated with user-defined access lists are mitigated. By pre-fetching data in a controlled manner, the network can prevent malicious attempts to exploit memory access patterns.
• Compatibility and Flexibility: The warm-up process is designed to be flexible and compatible with various transaction types and memory access patterns. It supports both the predefined access lists and those that might change dynamically during execution. This flexibility ensures that even transactions with unpredictable memory usage can be handled efficiently.

4. Dynamic Sharding and Address Space Management

• Shardeum employs dynamic sharding, where nodes are responsible for specific ranges of the address space, ensuring balanced data distribution and efficient processing. Each node dynamically adjusts its range of responsibility as new nodes join or leave the network. This dynamic approach ensures that the network can scale smoothly without significant disruptions. This approach contrasts with vertical scaling, where a single node is upgraded to handle more load, which can be less efficient and more costly.
• Adaptation to Network Conditions: Dynamic sharding allows Shardeum to adapt to changing network conditions and workloads. As nodes join or leave, the address space is redistributed to maintain balance. This ensures that no single node becomes a bottleneck and that the network can continue to operate efficiently even as it scales. The auto-scaling feature detects when to scale up or down, allowing the network to grow and shrink efficiently.
• Shard Division: In a dynamic sharding system, the address space is divided into multiple shards, each managed by a different set of nodes. For example, in a network with 512 nodes and a consensus group size of 128, the network is divided into four shards, each responsible for a quarter of the total address space. As nodes are added or removed, the boundaries of these shards adjust to maintain balance and ensure that each node handles a proportional amount of data and transactions. If every node in the network had a single shard, then every node would store all the data. However, as more nodes are added to the network, the consensus group size determines how the data is divided among the nodes. This division allows each node to handle a portion of the workload, thus improving the network's overall efficiency and speed. For instance, if we have a small Shardeum network with a consensus group size of 128 and 128 nodes or fewer, we essentially have a single shard where every node stores all the data. However, as soon as we add one more node, the network dynamically shards, resulting in significant speedup. This is because the workload and data storage are distributed among more nodes, and each node handles only a fraction of the total transactions and data.
• Gas Pricing Optimization: Shardeum also optimizes gas prices by adjusting them based on the complexity and resource usage of transactions. This dynamic pricing model ensures that simple transactions remain inexpensive, while more complex transactions that require more resources are priced accordingly. This approach helps prevent abuse and ensures fair resource allocation across the network.