JSON Idempotency: API Deduplication & Request Keys

Overview

Idempotency allows clients to safely retry requests without causing duplicate side effects. This guide covers idempotency key design, deduplication storage strategies, request/response caching, timeout and cleanup policies, and implementing idempotent state machines for payment processing, money transfers, and order creation.

Key Patterns

• Idempotency Key: Client-generated unique identifier per logical request; server deduplicates based on key, not repeating side effects
• Request Deduplication: Store idempotency key plus response; if key recurs, return cached response without re-executing
• Idempotent Endpoints: POST requests should be idempotent with a key header; GET/PUT/DELETE are idempotent by design
• Timeout Policy: Keys expire after 24 hours (configurable) to free storage and allow legitimate retries with new keys
• State Machine: Record when request was received, processing started, completed; handle retries at each state
• Distributed Deduplication: Redis or database for deduplication across multiple API servers

Best Practices

1. Define a standard idempotency key header name (Idempotency-Key or Request-ID) and document it in API docs
2. Generate idempotency key on the client (UUID v4) or use a request ID service — never let the server generate it
3. Store idempotency key plus request timestamp plus response JSON in a deduplication table or cache (Redis)
4. For critical operations (payments), store the request JSON plus response in the database for audit trail
5. Set key expiration to 24 hours — balances storage cost with allowing legitimate retries
6. Return a header indicating if the response is fresh or cached (e.g., Idempotency-Replay: true)
7. For long-running operations, use a state machine: received, processing, completed; handle retries at each state
8. Implement exponential backoff on the client: retry after 1s, 2s, 4s, 8s with a max backoff
9. For money transfers, verify that the same idempotency key always results in the same outcome (no race conditions)
10. Log all idempotency key deduplication events for debugging retry storms

Idempotency Key Design

Clients generate a UUID v4 for each logical request. The key should be stable — the same request from the same client uses the same key. Use a combination of operation type and a client-assigned nonce (not auto-increment) to ensure stability across retries. Example: UUID like f47ac10b-58cc-4372-a567-0e02b2c3d479 is ideal because it is globally unique and collision-free.

Deduplication Storage

Store deduplication state in either Redis (for performance, auto-expiration) or a database (for durability and audit trail). For Redis: store key to response JSON with TTL. For database: add an idempotency_key column with a unique index and store request/response. For critical financial operations, store both Redis (fast reads) and database (durable audit trail). Clean up expired keys after 24 hours to manage storage.

Request/Response Lifecycle

When a request arrives with an idempotency key:

1. Check deduplication store for the key. If found, return the cached response with Idempotency-Replay header.
2. If not found, mark the key as processing and begin request handling.
3. After successful completion, store key to response in deduplication store with expiration.
4. If an error occurs, store the error response (same as successful).
5. On retry with the same key, return the stored response (success or error).

Timeout and Cleanup Policies

Idempotency keys expire after 24 hours (or a configured duration). This frees storage and allows a client to create a new request after a long delay. For request-processing idempotency, a shorter timeout (1 hour) may be appropriate. Use background jobs to clean up expired keys. If a client retries with an expired key, treat it as a new request.

State Machines for Long-Running Operations

For operations that take seconds or minutes (payment processing, order fulfillment), implement a state machine with these states:

1. Received: Key registered, request stored, no side effects yet.
2. Processing: Side effects in progress (charge card, reserve inventory).
3. Completed: Side effects done, response ready.

Retries at each state return the appropriate response. If a retry arrives during processing, return processing status. If retry arrives after completion, return the final response.

Handling Concurrent Retries

If the client retries before the first request completes (network timeout), both requests may execute concurrently. Use database row locks or Redis distributed locks to ensure only one execution per idempotency key. Example: lock the key during processing, release after completion.

Idempotent Endpoints

GET requests are naturally idempotent (reading does not change state). PUT requests are idempotent if you replace the entire resource (not merge). POST requests are only idempotent with an idempotency key. DELETE requests are idempotent (deleting a deleted resource is fine). For APIs, require idempotency keys on all mutating endpoints (POST/PUT/DELETE).

Client-Side Implementation

Clients should: generate a UUID v4 for each logical request, include it in the Idempotency-Key header, retry on network errors or 5xx responses with exponential backoff, and reuse the same key for retries of the same request.

Monitoring and Observability

Log idempotency key deduplication events: original request timestamp, cached response, retry count. Alert if a single key is retried more than N times (indicating a systemic issue). Track replay rate to understand network stability.