Production-Grade Idempotency Middleware

A robust, transport-agnostic idempotency framework for Go applications.

This library guarantees that multiple identical requests to your API result in exactly one logical execution, while subsequent requests safely replay the recorded HTTP response without invoking your business logic again.

Features

• Exactly-Once Execution: Concurrent duplicate requests safely block and wait for the first request to complete, rather than causing race conditions.
• Deterministic Replays: Returns the exact HTTP Status Code, Headers, and Body from the original execution.
• Conflict Detection: Rejects requests (409 Conflict) that attempt to reuse an idempotency key with a different request payload.
• Panic Recovery: Automatically catches panics in your business logic, fails the execution gracefully, and notifies waiting requests.
• Bring Your Own Storage: Built on strict interfaces (Store and RuntimeRegistry). Ships with in-memory implementations, but designed to easily drop in Redis, Etcd, or Postgres.
• Transport Agnostic Core: The coordination engine is entirely independent of HTTP, cleanly separating business logic from infrastructure synchronization.

---

Installation

go get github.com/vectorshift/idempotency

---

Quick Start

Wrap any standard http.Handler with the idempotency middleware. By default, the middleware expects the idempotency key in the Idempotency-Key HTTP header.

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/vectorshift/idempotency/engine"
	"github.com/vectorshift/idempotency/memory"
	"github.com/vectorshift/idempotency/middleware"
)

func main() {
	// 1. Setup in-memory store and registry
	store := memory.NewMemoryStore()
	registry := memory.NewMemoryRegistry()

	// 2. Define execution policy (TTL & timeouts)
	policy := engine.Policy{
		TTL:         24 * time.Hour,
		WaitTimeout: 30 * time.Second,
	}

	// 3. Initialize the Coordinator and Middleware
	coordinator := engine.NewCoordinator(store, registry, policy)
	mw := middleware.NewIdempotencyMiddleware(coordinator)

	// 4. Start the background cleanup worker
	cleanup := memory.NewCleanupWorker(store, 5*time.Minute)
	cleanup.Start(context.Background())
	defer cleanup.Stop()

	// 5. Your business logic (only executes ONCE per Idempotency-Key)
	paymentHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		log.Println("Processing payment... (This only logs once!)")
		
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(http.StatusCreated)
		fmt.Fprintf(w, `{"payment_id":"pay_123","status":"completed"}`)
	})

	// Wrap the handler!
	http.Handle("/payments", mw.Wrap(paymentHandler))
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Testing the Quick Start

Request 1 (Executes):

curl -X POST http://localhost:8080/payments \
  -H 'Idempotency-Key: pay-001' \
  -d '{"amount": 100}'

Returns 201 Created and runs the business logic.

Request 2 (Replays):

curl -X POST http://localhost:8080/payments \
  -H 'Idempotency-Key: pay-001' \
  -d '{"amount": 100}'

Returns 201 Created immediately, with the header X-Idempotency-Replayed: true. Business logic is skipped.

Request 3 (Conflicts):

curl -X POST http://localhost:8080/payments \
  -H 'Idempotency-Key: pay-001' \
  -d '{"amount": 500}' # Different payload!

Returns 409 Conflict. The library detects payload tampering.

---

Customization

The middleware allows you to inject your own strategies for identity resolution and fingerprinting.

Custom Identity Resolution

By default, the framework looks for the Idempotency-Key header. If you are dealing with Webhooks where you cannot control the headers, you can extract the identity directly from the JSON body:

type WebhookIdentityResolver struct{}

func (r *WebhookIdentityResolver) Resolve(req *http.Request) (string, error) {
	var payload struct {
		EventID string `json:"event_id"`
	}
	
	// The middleware safely buffers the body for you, so reading here is safe.
	if err := json.NewDecoder(req.Body).Decode(&payload); err != nil {
		return "", err
	}
	return payload.EventID, nil
}

// Wire it up:
mw := middleware.NewIdempotencyMiddleware(coordinator,
	middleware.WithIdentityResolver(&WebhookIdentityResolver{}),
)

Custom Fingerprinting

By default, the fingerprint is a SHA-256 hash of HTTP Method + URL Path + Raw Body. If your requests include timestamps or random IDs that change but represent the same logical operation, you can implement a custom fingerprint strategy to hash only the business-relevant fields:

type PaymentFingerprintStrategy struct{}

func (s *PaymentFingerprintStrategy) Fingerprint(r *http.Request, body []byte) (string, error) {
	var payload struct {
		Amount float64 `json:"amount"`
		UserID string  `json:"user_id"`
		// Ignore timestamps!
	}
	json.Unmarshal(body, &payload)

	// Hash only the relevant fields
	h := sha256.New()
	h.Write([]byte(fmt.Sprintf("%f:%s", payload.Amount, payload.UserID)))
	return hex.EncodeToString(h.Sum(nil)), nil
}

// Wire it up:
mw := middleware.NewIdempotencyMiddleware(coordinator,
	middleware.WithFingerprintStrategy(&PaymentFingerprintStrategy{}),
)

---

Distributed Systems Support (Bring Your Own Storage)

The core engine relies on two tiny interfaces found in the engine package:

1. engine.Store — Manages persistent state (e.g., Postgres, MongoDB, DynamoDB).
2. engine.RuntimeRegistry — Manages runtime locks and active execution synchronization (e.g., Redis, Etcd).

If you are running across multiple servers, you can swap out the provided memory implementations with your own distributed implementations without changing a single line of your application logic or middleware configuration!