What is a Reorg?

A blockchain reorg occurs when a node discovers a new chain segment that is longer than the one previously considered the main chain. The node will switch to the longer chain, making it the new canonical chain. This process can lead to transactions that were considered finalized on the old chain being omitted from the new chain, necessitating a robust mechanism to handle such events. To learn more, check out this in-depth blog post: take a look at this in-depth blog post: Understanding Blockchain Reorgs: Why Block Numbers Don't Matter as Much as You Think

Reorg Handling with Streams

Streams is designed to deliver blockchain data to users in a sequential and continuous manner. That helps us and our users manage blockchain reorganizations effectively. Here’s how the reorg monitoring process looks:

1. Continuous flow: Our service streams blockchain data block by block in batches in a sequential order. This ensures that you receive data in the exact order it appears on the blockchain.
2. Monitoring blocks: As we stream data, we continually monitor each block to ensure its validity and position in the blockchain sequence.
3. Reorg identification: A blockchain reorg occurs when a block previously considered as part of the main chain gets replaced by a different block. We detect reorgs by observing discrepancies in the chain, specifically when a new block's parent hash does not match the hash of the previous block we streamed.
4. Immediate response: Upon detecting a reorg, our system immediately takes action to reconcile the data.

To reconcile any reorgs, the process is:

1. Reconciliation with blockchains: To reconcile the data during a reorg, we poll the blockchain directly to fetch the most accurate and up-to-date information about the blocks.
2. Connecting the chain: We work backwards from the point of discrepancy, comparing the streamed blocks with the data obtained from the blockchain. Our goal is to find the last block where both datasets match, ensuring we correctly identify the point of divergence in the chain.

Restream on Reorg option

Enterprise users may benefit from using “Restream on reorg” option for at-the-tip streams:

With this feature enabled, once the chain is reconciled and we've identified the correct sequence of blocks, we resume streaming. This involves re-streaming some blocks to correct any discrepancies caused by the reorg. In the Logs of your stream once we encounter a reorg, you may find entries telling you where reorg was detected and how the chain is connected once we identify the canonical hash order.

How to Handle Restreamed data

If you notice that we are streaming a block with a number that is not increased sequentially — it indicates that we are in the process of re-streaming to reconcile the chain post-reorg for you. This process ensures the integrity and accuracy of the data you receive, aligning it with the most current state of the blockchain.

Webhook Destinations

Utilizing webhooks allows users to directly monitor incoming stream messages. The deterministic "Restream on Reorg" method seamlessly integrates with webhook-based systems, ensuring users automatically receive updated data segments following a reorg.

For every incoming batch_start_range that is not sequentially increasing than previously processed batch_end_range consider overwriting the data you previously processed.

For S3 Destinations

Users storing data in S3 can leverage both approaches to enhance data integrity:

• Without Versioning: Files will be overwritten with the most recent data.
• With Versioning: Users should focus on the last file version but pay extra attention to file versions that mismatch in data size. A size discrepancy could indicate a reorg event, necessitating a review of using affected files for post-processing.

User Responsibility: Verifying Block Hash Continuity

Regardless of the chosen method or destination, implementing a robust monitoring system is essential. This system should verify the hash-connectedness of block data, ensuring the streamed data's integrity and relevance.

• Continuity Check: In addition to our internal mechanisms for handling reorgs and maintaining data integrity, it is crucial for users to independently verify the continuity of block hashes. This means ensuring that each received block's previous hash matches the hash of the block that was received immediately prior.
• Ensuring Chain Integrity: This step is essential for users to maintain the integrity of their own data representations of the blockchain. By verifying the hash continuity, users can independently confirm that they are working with a consistent and unbroken chain of blocks.
• Handling Discrepancies: In the rare event that you detect a mismatch in the hash continuity, it's advisable to pause your processes that depend on this data and verify against the blockchain. This could be indicative of a reorg that is being processed or an anomaly that needs attention.
• Proactive Data Management: While our service strives to manage reorgs and deliver accurate data seamlessly, this practice of user-side verification forms an additional layer of data integrity, ensuring that the users are always aligned with the true state of the blockchain.

Setting a Latest Block Delay

The "Latest Block Delay" feature offers a probabilistic way to mitigate the risks associated with reorgs. By setting a delay from the chain tip, users can stream data that is a specified number of blocks behind the most recent one.

This approach reduces the likelihood of processing blocks that may not be part of the hash-connected canonical chain. Although this method does not guarantee immunity from reorgs, it significantly decreases the chances of being affected by them, providing a safer yet near-real-time data stream.