10 min read

Streams

Streams is available starting from the Free plan. For teams with unique requirements, we offer tailored datasets, dedicated support, and custom integrations. Contact our team for more information.

Overview

Streaming blockchain data in real-time can often be daunting due to the complexity of infrastructure setup and the ongoing management required. QuickNode addresses this challenge with Streams, a blockchain data streaming solution that allows you to easily stream both historical and real-time data directly to your applications or services.

This guide will take you through the process of verifying HMAC signatures for incoming messages from Streams webhook services. Ensuring the integrity and authenticity of these messages is crucial for securing your applications against tampering and forgery. By the end of this guide, you'll be equipped with the knowledge to implement signature verification using Python effectively.

What You Will Do

• Implement webhook signature verification in Node.js, Python, or Go
• Set up a server to handle incoming webhook messages
• Verify signatures using HMAC SHA-256
• Process both compressed and uncompressed payloads
• Test your verification implementation
• Discuss best security practices to enhance the safety of your verification process

What You Will Need

• Your preferred language environment (Node.js, Python, or Go)
• A code editor (e.g., VSCode)
• ngrok installed
• Your Stream's security token from the QuickNode dashboard
• Language-specific requirements (see dependencies table below)

Node.js

Dependency	Version
node.js	>=16.x
express	^4.18.2
body-parser	^1.20.2
ngrok	^3.0.0

Python

Dependency	Version
python	>=3.7
flask	^2.0.0
ngrok	^3.0.0

Go

Dependency	Version
go	>=1.16
ngrok	^3.0.0

Understanding Webhook Signatures

When your server receives a webhook from Streams, it includes three critical headers that are used for verification:

• X-QN-Nonce: A unique string that prevents replay attacks
• X-QN-Signature: The HMAC signature you need to verify
• X-QN-Timestamp: The timestamp when the message was signed

The signature is created by combining these elements with the payload in a specific way, then applying an HMAC-SHA256 hash using your Stream's security token as the key. Now, we'll move onto the coding part of the guide and show you how to validate incoming webhook messages from Streams.

Stream Setup and Security Token

First, we'll need a security token for signature verification. You can either:

• If you have an existing Stream: Go to your Stream's Settings tab and keep your security token handy (we'll come back to this later)
• If you need to create a Stream:
  1. Visit TypedWebhook.tools and copy the provided webhook URL
  2. Go to the Streams section on your QuickNode dashboard
  3. Click Create Stream
  4. Configure your Stream:
     • Select Ethereum as the blockchain and Mainnet as your network
     • Choose Blocks as your dataset
     • Paste the TypedWebhook.tools URL in the webhook destination settings
  5. Create the Stream and copy your security token via the Settings tab

Note: This TypedWebhook.tools URL is temporary until we show you later on in the guide how to run your own local server and ngrok to validate the incoming message.

Setting Up Your Development Environment

First, create a new project directory and set up your environment based on your chosen language:

Node.js

mkdir webhook-verification
cd webhook-verification
npm init -y
npm install express body-parser

Python

mkdir webhook-verification
cd webhook-verification
pip install flask

Go

mkdir webhook-verification
cd webhook-verification
go mod init webhook-verification

Implementing the Verification Server

Now, let's implement the verification server. We'll create a server that:

• Listens for POST requests on a /webhook endpoint
• Extracts the necessary headers (e.g., nonce, signature, timestamp)
• Verifies the signature
• Returns HTTP responses

Choose your preferred language and create your server file within the appropriate directory (based on the language you chose). Remember to replace the your_security_token_here placeholder string with your actual security token.

Node.js

const express = require('express');
const crypto = require('crypto');
const bodyParser = require('body-parser');

const app = express();
const PORT = 9999;

app.use(bodyParser.raw({ 
    type: '*/*',
    limit: '50mb'
}));

app.use(bodyParser.json());

function verifySignature(secretKey, payload, nonce, timestamp, givenSignature) {
    // First concatenate as strings
    const signatureData = nonce + timestamp + payload;
    
    // Convert to bytes
    const signatureBytes = Buffer.from(signatureData);
    
    // Create HMAC with secret key converted to bytes
    const hmac = crypto.createHmac('sha256', Buffer.from(secretKey));
    hmac.update(signatureBytes);
    const computedSignature = hmac.digest('hex');

    console.log('\nSignature Debug:');
    console.log('Message components:');
    console.log('- Nonce:', nonce);
    console.log('- Timestamp:', timestamp);
    console.log('- Payload first 100 chars:', payload.substring(0, 100));
    console.log('\nSignatures:');
    console.log('- Computed:', computedSignature);
    console.log('- Given:', givenSignature);
    
    return crypto.timingSafeEqual(
        Buffer.from(computedSignature, 'hex'),
        Buffer.from(givenSignature, 'hex')
    );
}

app.post('/webhook', async (req, res) => {
    const secretKey = 'your_security_token_here';
    const nonce = req.headers['x-qn-nonce'];
    const timestamp = req.headers['x-qn-timestamp'];
    const givenSignature = req.headers['x-qn-signature'];

    if (!nonce || !timestamp || !givenSignature) {
        console.error('Missing required headers');
        return res.status(400).send('Missing required headers');
    }

    try {
        const payloadString = req.body.toString('utf8');
        const isValid = verifySignature(
            secretKey,
            payloadString,
            nonce,
            timestamp,
            givenSignature
        );

        if (isValid) {
            console.log('\n✅ Signature verified successfully');
            return res.status(200).send('Webhook received and verified');
        } else {
            console.log('\n❌ Signature verification failed');
            return res.status(401).send('Invalid signature');
        }
    } catch (error) {
        console.error('Error processing webhook:', error);
        return res.status(500).send('Error processing webhook');
    }
});

app.listen(PORT, () => {
    console.log(`Server running on port ${PORT}`);
});

Python

import hmac
import hashlib
import gzip
from flask import Flask, request, jsonify
import logging

app = Flask(__name__)

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Replace this with your actual security token
SECRET_KEY = "your_security_token_here"

def verify_signature(secret_key, payload, nonce, timestamp, given_signature):
    message = nonce + timestamp + payload
    computed_signature = hmac.new(
        secret_key.encode(),
        message.encode(),
        hashlib.sha256
    ).hexdigest()
    
    return hmac.compare_digest(computed_signature, given_signature)

@app.route('/webhook', methods=['POST'])
def handle_webhook():
    nonce = request.headers.get('X-QN-Nonce')
    timestamp = request.headers.get('X-QN-Timestamp')
    given_signature = request.headers.get('X-QN-Signature')

    if not all([nonce, timestamp, given_signature]):
        logger.error("Missing required headers")
        return jsonify({"error": "Missing required headers"}), 400

    # Get the raw payload
    raw_payload = request.get_data()

    # Check if the payload is gzip compressed
    if request.headers.get('Content-Encoding') == 'gzip':
        try:
            payload = gzip.decompress(raw_payload).decode('utf-8')
        except Exception as e:
            logger.error(f"Error decompressing payload: {str(e)}")
            return jsonify({"error": "Failed to decompress payload"}), 400
    else:
        payload = raw_payload.decode('utf-8')

    try:
        is_valid = verify_signature(SECRET_KEY, payload, nonce, timestamp, given_signature)
    except Exception as e:
        logger.error(f"Error verifying signature: {str(e)}")
        return jsonify({"error": "Failed to verify signature"}), 500

    if is_valid:
        logger.info("Received valid webhook")
        # Process the webhook payload here
        # For now, we'll just return a success message
        return jsonify({"message": "Webhook received and verified"}), 200
    else:
        logger.warning("Received invalid webhook")
        return jsonify({"error": "Invalid signature"}), 401

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Go

package main

import (
	"bytes"
	"compress/gzip"
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"fmt"
	"io"
	"log"
	"net/http"
)

// handler for POST requests
func postHandler(w http.ResponseWriter, r *http.Request) {
	// Ensure the method is POST
	if r.Method != http.MethodPost {
		http.Error(w, "Only POST method is allowed", http.StatusMethodNotAllowed)
		return
	}

	signature := r.Header.Get("X-QN-Signature")
	nonce := r.Header.Get("X-QN-Nonce")
	timestamp := r.Header.Get("X-QN-Timestamp")
	secretKey := "your_security_token_here"

	// Read the request body
	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "Failed to read body", http.StatusInternalServerError)
		return
	}

	defer r.Body.Close()

	// Check if the body is encoded with gzip
	if r.Header.Get("Content-Encoding") == "gzip" {
		gzipReader, err := gzip.NewReader(bytes.NewReader(body))
		if err != nil {
			http.Error(w, "Failed to create gzip reader", http.StatusInternalServerError)
			return
		}
		defer gzipReader.Close()

		decodedBody, err := io.ReadAll(gzipReader)
		if err != nil {
			http.Error(w, "Failed to read gzip body", http.StatusInternalServerError)
			return
		}
		body = decodedBody
	}

	err = VerifyHMAC(secretKey, nonce, timestamp, string(body), signature)
	if err != nil {
		fmt.Println(err)
		http.Error(w, "Invalid HMAC", http.StatusUnauthorized)
		return
	}

	fmt.Println("HMAC is valid")
	// Respond to the client
	w.WriteHeader(http.StatusOK)
	_, _ = w.Write([]byte("Received POST request"))
}

func main() {
	http.HandleFunc("/", postHandler)

	// Start the server on port 8080
	port := ":8080"
	fmt.Printf("Server is listening on port %s\n", port)
	log.Fatal(http.ListenAndServe(port, nil))
}

func VerifyHMAC(secretKey, nonce, timestamp, message, receivedHMAC string) error {
	// Combine nonce, timestamp, and message
	data := nonce + timestamp + message

	// Create a new HMAC using SHA-256 and the secret key
	h := hmac.New(sha256.New, []byte(secretKey))
	h.Write([]byte(data))

	// Compute the expected HMAC
	expectedHMAC := hex.EncodeToString(h.Sum(nil))

	// Compare the received HMAC with the expected HMAC
	if !hmac.Equal([]byte(expectedHMAC), []byte(receivedHMAC)) {
		return errors.New("invalid HMAC: message integrity or authenticity check failed")
	}

	// HMAC is valid
	return nil
}

Starting Your Server

Start your verification server based on your implementation:

Node.js

node server.js

Python

python verify_signature.py

Go

go run main.go

Setting Up ngrok

Then, create a tunnel to your local server (adjust the port based on your implementation):

Node.js

ngrok http 9999

Python

ngrok http 5000

Go

ngrok http 8080

Keep the terminal window handy as we'll need the URL when updating our Stream's webhook destination.

Validating Messages with Incoming Stream Data

1. First, copy the ngrok URL running in your terminal (e.g., https://abc123.ngrok.io)
2. Then, go to your Stream's settings in the QuickNode dashboard.
3. Pause your Stream and update the webhook URL to your ngrok URL + /webhook (e.g., https://abc123.ngrok.io/webhook)
4. Resume your Stream
5. Watch your server logs. For each block, you should see output similar to:

Node.js

Signature Debug:
Message components:
- Nonce: 02c6d2644296c8b830970891410825a3
- Timestamp: 1735613672
- Payload first 100 chars: {"data":[{"baseFeePerGas":"0xd557bf66","blobGasUsed":"0x60000","difficulty":"0x0","excessBlobGas":"0

Signatures:
- Computed: 049827ff010a1c24cd21594d72e490fd43a48d9e69a5c18628788063665134cc
- Given: 049827ff010a1c24cd21594d72e490fd43a48d9e69a5c18628788063665134cc

✅ Signature verified successfully

Python

INFO:__main__:Received valid webhook
INFO:werkzeug:127.0.0.1 - - [30/Dec/2024 21:53:01] "POST /webhook HTTP/1.1" 200 -

Go

Server running on port :8080
HMAC is valid

If you see successful verification messages, congrats! Your server is properly verifying webhook signatures.

Best Security Practices

While implementing the HMAC signature verification, ensure the following practices to secure your application:

• Always keep the security token confidential and securely stored, not hardcoded in publicly accessible areas of your code.
• Log all validation attempts, both successful and failed, to aid in auditing and troubleshooting.
• Implement additional checks like timestamp validation to prevent replay attacks.

Conclusion

Congratulations! You've successfully learned how to validate incoming Streams webhook messages by verifying HMAC signatures. This practice is vital for protecting your applications from external threats and ensuring data integrity.

If you have questions, please contact us directly. If you have any ideas or suggestions, such as new destinations, features, metrics, or datasets, you want us to support.

Also, stay up to date with the latest by following us on Twitter and joining our Discord and Telegram announcement channel.

We ❤️ Feedback!

Let us know if you have any feedback or requests for new topics. We'd love to hear from you.