Streams Destinations

Streams offer various destinations for sending blockchain data, tailored to different needs. For real-time applications, a Webhook destination is ideal. For archiving or managing large datasets, consider using an Object storage solution like S3. Functions serve well for scenarios requiring the execution of arbitrary code based on stream data.

Before setting up your destination, you must first configure your Stream settings. These include selecting the blockchain chain and network, determining batch size, specifying the date range, and setting up reorg handling, among others. Proper reorg handling is crucial for effective data management. For more information on managing reorgs, refer to the Reorg handling section.

Webhooks

The use of a Webhook for your Stream destination is ideal when you are using a lightweight application that requires real-time responses. The benefits of this include data availability in real-time and the flexibility to use any type of Webhook destination. This destination type is usually not ideal for storing large amounts of data. Other benefits include:

• Real-Time Data Handling: Webhooks are ideal for scenarios requiring real-time data processing. They enable immediate reaction to incoming data streams, which is crucial for applications needing instant updates.
• Direct Integration with Services: Webhooks provide a straightforward way to integrate Streams with various third-party services and custom applications. They can directly push data to services that can accept webhook payloads.
• Flexibility and Custom Workflows: Webhooks might offer more flexibility in handling data. They allow users to create custom workflows and processing logic tailored to their specific needs, which might not be as straightforward with predefined destinations like S3 or PostgreSQL.
• Simplicity and Ease of Use: For some users, setting up a webhook endpoint might be simpler than configuring integration with cloud storage or databases, especially if they already have a system in place to handle webhook calls.

To set up Webhooks as a destination, learn more here.

S3-Compatible Storage

The use of object storage destinations like S3 are suitable to use when processing and archiving large amounts of data in batches. The reliable and scalable nature of S3 storage enables data durability and integration with data lakes and other big data tools. Other benefits include:

• Large Data Storage: S3 offers virtually unlimited storage, making it suitable for handling massive amounts of data that webhooks might not efficiently process.
• Data Durability and Reliability: S3 provides high durability and secure storage options, ensuring data is safely stored and readily available for future analysis.
• Cost-Effective for Large Data: For substantial data volumes, S3 can be more cost-effective due to its pricing model based on storage and access.
• Ease of Data Analysis Integration: Data stored in S3 can be seamlessly integrated with various analytics tools, simplifying the data analysis process.
• Scalability: S3 scales automatically to accommodate data growth, which is beneficial for applications with increasing data streaming needs.

These factors make S3 a preferred choice for scenarios involving large-scale data storage, analytics, and applications requiring robust data backup and retrieval capabilities. To set up S3 storage as a destination, learn more here.

Functions

The use of Functions as a Stream destination enables developers to filter, transform and enhance the data they recieve from their Stream. Functions automatically takes care of deployment and scaling, ensuring your functions perform optimally at all times. Other benefits include:

• Data Transformation: Allows parsing and custom transformation of blockchain data according to your criteria.
• Scalability: Automatically scale your Function to meet your Streams' data requirements, ensuring optimal performance.
• Cost Efficiency: Ensures you only pay for the resources your Function uses, optimizing your expenses.
• Integration Flexibility: Facilitates easy integration with additional services like IPFS and Streams, enhancing functionality.

To set up Functions as a destination, learn more here.

PostgreSQL

PostgreSQL serves as a robust relational database destination for Streams, offering structured storage and powerful querying capabilities. It's particularly well-suited for applications requiring complex data relationships and SQL-based analysis. Benefits include:

• Structured Data Storage: Enables organizing blockchain data in a relational format with defined schemas and relationships.
• Advanced Querying: Powerful SQL querying capabilities for complex data analysis and reporting.
• ACID Compliance: Ensures data integrity and consistency through transactions.
• Performance: Optimized for both read and write operations with support for indexing and materialized views.
• Extensibility: Rich ecosystem of extensions and tools for enhanced functionality.

To set up PostgreSQL as a destination, learn more here.

Snowflake

Snowflake provides a cloud-native data warehouse solution that combines the power of traditional databases with modern cloud architecture. It's ideal for organizations requiring enterprise-scale analytics and data processing capabilities. Key advantages include:

• Elastic Scaling: Automatically scales compute and storage resources independently based on workload demands.
• Data Sharing: Easily share and collaborate on blockchain data across different organizations and departments.
• Multi-Cloud Support: Freedom to choose your preferred cloud provider while maintaining consistent functionality.
• Zero Management: Automated maintenance, updates, and optimization without manual intervention.
• Seamless Integration: Built-in connectors and tools for various data analytics and business intelligence platforms.

To set up Snowflake as a destination, learn more here.