Transaction Lifcycle

1. Overview

The transaction lifecycle in the network encompasses both protocol-level transactions, which maintain network cycles and handle consensus, and user transactions, which include coin transfers and smart contract interactions. User transactions are particularly significant for measuring transactions per second (TPS), as these are the interactions that most users care about. Unlike some other networks that count internal transactions towards TPS, this network only counts user transactions to provide an honest metric.

2. Transaction Initiation

• User Intent: The lifecycle begins when a user expresses intent to perform a transaction, such as a coin transfer or interacting with a smart contract, via a wallet like MetaMask.
• RPC Server Interaction: The user's transaction request is sent to an RPC (Remote Procedure Call) server, which acts as the Ethereum standard interface to submit transactions to the network and handle various network queries. The RPC server connects the user's wallet to the network.

3. Transaction Submission

• Nonce and Gas Estimation: The RPC server fetches the nonce (a sequential number to keep transactions in order) and estimates the gas required for the transaction. The gas estimate involves simulating the transaction on a local data RPC server with all account data to predict the computational resources needed. For smart contracts, they are pre-run to tally up all the opcodes and their gas, which gives a detailed gas estimate. A feature called "nonce mode" ensures smoother transaction handling. If a user sends a transaction with a higher nonce than expected, instead of instant rejection, the network keeps these transactions until the nonces catch up. For example, if the account nonce is five, and the user sends a transaction with nonce ten, it is kept in the queue until nonces six to nine are processed. This ensures that once nonce nine is processed, nonce ten is automatically injected by the validator network, greatly improving user experience. More about the technical side of things can be seen at addTransactionToNonceQueue().
• Signing and Sending: Once the nonce and gas details are confirmed, the user signs the transaction with their private key in MetaMask. The signed transaction is then submitted from MetaMask to the RPC server, which injects the transaction into the validator network.

4. Transaction Processing

• Transaction Lifecycle States: A transaction goes through a lifecycle resembling a state machine. It starts in a waiting state after being injected into the network. This waiting state lasts about six seconds, allowing the transaction to mature. During this time, it is queued by multiple validators who need to process it. The six-second wait ensures that transactions are ordered correctly based on timestamps. Once the six seconds have passed, the transaction can be processed.
• Validator Network: The transaction is initially sent to one active validator. Then, this transaction is distributed to validators in the transaction group using the gossip protocol. Each validator in the group holds a part of the transaction data necessary for processing. For example, if a transaction requires access to account data for accounts A, B, and C, then each of these accounts will have its own consensus group. Let's say there is a consensus group for account A, a consensus group for account B, and a consensus group for account C. Together, these three consensus groups form a transaction group. Each validator within these consensus groups collaborates to process the transaction. These consensus groups can be perfectly or partially overlapped, ensuring flexibility and efficiency in transaction processing. The forwarding of a transaction is executed by the gossip protocol with the spread_tx_to_group message.
• Execution Group Selection: Before placing the transaction in a queue, an execution group is selected. The developers have determined that 2/3 of the shard size is enough to determine the value of a transaction. However, the shard executing the transaction may not have all necessary information for accounts B and C. The function that enqueues and choses the execution group is routeAndQueueAcceptedTransaction().

• Transaction Group Dynamics: Validators use a "corresponding tell" process to efficiently share data needed for transaction processing. For instance, if Account B's data is required by an execution group, nodes in Account B's consensus group will forward this data to the nodes in the execution group. This process is designed to be efficient and secure, preventing any single node from unduly influencing the transaction by ensuring data is only forwarded to a specific node as defined by sharding math. The sending of this message is done in tellCorrespondingNodes().
• Consensus Formation: Validators execute the transaction, apply it to their local state, and form a consensus on the transaction’s result. The process begins with execution group validators waiting to receive all necessary data for a given transaction, sometimes from non-execution group nodes. Note that no data can be sent or loaded while there are upstream transactions that could modify the state.
  
  Once execution group nodes have the required state, they apply the transaction to an in-memory copy of the data. The result of this transaction application includes outcomes such as applied/not-applied, along with hashes of the updated account values, forming the basis of the proposal. The hash is created by calculateVoteHash().
  
  Validators create a vote by hashing the proposal, signing it, and sharing this signed hash with other nodes. Each validator's vote is based on the proposal hash it created and signed. If a majority of the nodes agree and have the same hash, the transaction is confirmed. This hash is cryptographically signed, creating a provable receipt. The creation of this receipt is handled in the tryProduceReceipt() function.
  If a validator observes that a majority of other nodes have the same hash, it acknowledges the consensus and confirms the transaction receipt. The creation of a vote occurs in the createAndShareVote() function, and appending it to the list of votes happens in tryAppendMessage().
  
  The structure that contains the proposal, the proposal hash, and the array of signatures along with the IDs of the signers constitutes the signed receipt.
  Once a supermajority of votes is achieved, a valid signed receipt can be produced. After a node has this receipt, it can commit (save) any data it owns related to the transaction.
• Commit Phase: Once the receipt is created and the transaction is confirmed, validators proceed to commit the receipt. This involves reversing the data sharing process, where nodes that received account data for execution send updated data back to the original nodes. This ensures that all nodes in the transaction group have the correct and updated account states. This phase is expressed as code in commitConsensedTransaction().
• Gossip Protocol: The network employs a gossip protocol to ensure all nodes in the transaction group receive the transaction efficiently. This protocol optimizes communication by sending messages to a few nodes, which then propagate the message further, reducing the load on any single node. The function that sends the gossip message is sendGossip().
  
  There are two variations of gossip algorithms used in the network: "Burst" and "Linear." These algorithms differ in their deterministic calculations of how many messages are sent out at each level of gossip. "Burst" sends more messages and is typically used for gossiping to large groups, such as all nodes. In most cases, "Linear" gossip is preferred, note that while it also includes a burst aspect, it is less pronounced compared to the "Burst" algorithm.
• The main function the processes transactions is processTransactions().

5. Data Distribution and State Updates

• Transaction Receipt: The transaction receipt is sent to an archiver, a specialized node responsible for long-term storage and data integrity. The archiver ensures data integrity before storing it. The sending to the archiver is done in addReceiptToForward().
• Archiver Setup: Once the archiver approves the transaction receipt, it writes the receipt to a data log for inter-process communication and also stores it in a database. This database is not highly optimized for queries but is efficient for storage purposes. It is crucial that valid receipts make it to an Archive Server, as failing to do so can cause serious issues.
• Data Forwarding Mechanism: The archiver forwards the receipt to a distributor, which disseminates the transaction data to a collector. Distributors use WebSockets to send data efficiently to collectors, ensuring real-time updates across the network. The collector then sends the data back to the local data RPC server, which notifies MetaMask, which is polling, that the transaction has been completed. Once MetaMask receives the confirmation, it updates the user on the transaction status.
• Data Syncing: The archiver also forwards transaction data to an explorer collector, which updates the blockchain explorer. This allows users and developers to independently verify transaction status through the explorer interface.
• State Updates: Validators maintain only the latest state to optimize performance, while archivers store historical state data. This separation ensures efficient access to current data while preserving the ability to audit and review past transactions.
• Distributor and Collector Mechanism: The distributor, a microservice, reads the data log and continuously pulls in data. Collectors connected to distributors receive a constant stream of transactions and their updated states.
• Flexible Topologies: The system supports various topologies, allowing for operational flexibility. For instance, one archiver can connect to multiple collectors, or a collector can link to several distributors. This flexibility ensures that the network can scale effectively, addressing potential overloads in the main net.
• Efficiency of State Updates: Validators only store the latest state to ensure quick access and high performance, relying on archivers for historical data retrieval when necessary.