Idempotency in Payment Systems: How to Prevent a Customer from Being Charged Twice

Mohd Jeeshan

15 min read

·

Aug 9, 2026

--

Listen

Share

Imagine a scenario like: A customer clicks pay. The PaymentService receives the request, deducts the balance from customer’s account and successfully completes the transaction. But before response reaches to the client or browser, network connection fails. Customer sees the message on browser “Payment failed, please try again”.

So customer click Pay again. Now if backend is not designed correctly, customer will be charged twice. This simple problem leads to some important concepts in distributed systems.

• Idempotency
• Race conditions
• Concurrency
• Database constraints
• Transactions
• Distributed locks
• Retries
• Failure recovery
• Eventual consistency
• External payment providers
• Reconciliation
• Observability
• Webhook redelivery

For an engineer, understanding these concepts is particularly valuable because testing a payment system isn’t just about verifying the happy path. Now the question arise “What happens when everything goes wrong?”

Let’s deep dive into the Payment service how things looks like in real.

The Payment API Looks Simple

Let’s consider our payment request looks something like

POST /payments
Request:
{
  "amount": 100,
  "currency": "EUR",
  "fromAccount": "A",
  "toAccount": "B"
}

Backend may perform following steps

Receive request -> Validate payment -> Debit Account A -> Credit Account B -> Return SUCCESS

Here’s how it works

Suppose Account A has: €1,000

After the payment: €1,000 → €900

Everything looks fine. But now introduce the network.

The Real Problem: What If the Response Is Lost?

The Payment Service receives the request and fulfills it, debiting €100 from the account and updating the database balance to €900. After all this, it returns HTTP 200 SUCCESS. Right after this, a network failure occurs and the response is lost before reaching the client or browser. After all this it returns HTTP 200 SUCCESS.
From client perspective it may be a timeout. Client does not have any idea about the payment whether:

1. Payment is failed
2. Payment is succeeded but response is lost

Now in such case customer or client will try to retry to pay again which seems reasonable as well from the customer’s perspective. So now
Client -> RetryPayment -> Payment Service -> Debits €100 -> Account DB balance update. Now cycle will look like

€1,000(Initially)
   ↓
€900   ← first request
   ↓
€800   ← retry

Customer will be charged twice here. This is the fundamental problem that Idempotency solves. Now the question comes

What is actually Idempotency

It may be defined as. “Performing the same operation multiple times should produce the same business outcome as performing it once.”

Suppose if there are 4 same requests(e.g. Request 1, Request 1, Request 1, Request 1) made to the payment service, it should effectively produce 1 payment instead of 4 payments. This is different from saying that the server receives only one request. The server may receive the same request multiple times.The system must simply ensure that those requests don’t create multiple business effects(like deducting money from customer account 4 times instead of 1).

As a QA engineer, one idea that comes to mind is disabling the pay button after one click. This might prevent accidental double-clicks, but it doesn’t solve distributed-system failures. Why Frontend Safeguards Aren’t Enough Frontend protections like button disabling only prevent user error. They don’t protect against: Network timeouts where the response is lost, Application crashes during transaction processing, Multiple servers in a load-balanced environment, Retry storms from external systems This makes idempotency a primary responsibility of the backend, not an optional feature.

Let’s discuss how we can tackle this problem in real life while implementing such payment system.

Idempotency Key

The standard solution to this is to have a idempotency key attached to the request. The client generates a unique identifier for the payment operation and send it along with the request. For example PaymentService endpoints and request may look like this:
For example:

POST /payments
Idempotency-Key: 8f7a2c91-4e7b-4d8a-9f21-123456789abc

Request:

{
  "amount": 100,
  "currency": "EUR",
  "fromAccount": "A",
  "toAccount": "B"
}

The client may retry the HTTP request, but the idempotency key remains the same. Now what happens when user sees timeout after a successful payment and retry again with the same amount. The server needs somewhere to remember that it has already seen a particular idempotency key. There are several possible approaches:

• Relational database such as PostgreSQL or MySQL
• Redis
• DynamoDB or another distributed key-value store
• A dedicated idempotency store

Press enter or click to view image in full size

For payment systems: Always use relational DB as source of truth. Redis can sit in front as a read-through cache. For such system, the choice is important because we are not simply trying to remember:

“Have I seen this HTTP request before?”

We need to reliably associate the key with the payment operation and its state. A simplified model might look like:

Idempotency Key -> Payment ID -> Payment Status -> Original Response

Idempotency Key Generation Guidance

Use UUID v4: Most reliable (cryptographically unique)
Use domain-specific ID: paymentId = “USER-123–2024–08–08–1”
(But don’t re-use the same key for different users)
Use request hash: Hash(userId + amount + timestamp)
(Risk: same hash could collide on different requests)

Best practice: Use UUID v4 for financial operations.

What Happens During a Retry?

First request:

POST /payments
Idempotency-Key: ABC123

The server processes it:

ABC123 -> PAYMENT-A -> SUCCESS

Then the response is lost.

The client retries:

POST /payments
Idempotency-Key: ABC123

The payment service checks its idempotency store:

ABC123 → PAYMENT-A → SUCCESS

Instead of executing another debit, it returns the original result.

{
  "paymentId": "PAYMENT-A",
  "status": "SUCCESS"
}

No second payment is created. Let’s see how Idempotency table may look like. Again just creating a basic table, it may change according to specific use case

CREATE TABLE payment_requests (
    idempotency_key VARCHAR(100) PRIMARY KEY,
    payment_id      VARCHAR(100),
    request_hash    VARCHAR(128),
    status          VARCHAR(30),
    response        JSON,
    created_at      TIMESTAMP
);

Now there are two important pieces that need explanation:

Why do we store the request hash?

Why do we need both the idempotency state and the payment state?

Let’s start with first one

Why do we store the request hash?

Consider a scenario a when customer sends a request with key x, for second time by mistake or due to some malicious activity different request with same key x is sent to the server then what will happen in such case. Lets understand this with 2 requests.

First request:

POST /payments
Idempotency-Key: ABC123

{
  "amount": 100,
  "currency": "EUR"
}

The server processes it:

ABC123 -> PAY001 -> SUCCESS

Now suppose another request arrives:

POST /payments
Idempotency-Key: ABC123

but this time:

{
  "amount": 500,
  "currency": "EUR"
}

The server cannot simply say:

“ABC123 already exists, so I’ll return the previous result.”

It also needs to know whether this is actually the same request being retried. That’s where the request hash helps. Server creates a hash of the request e.g.

Original request -> SHA-256 -> HASH-XYZ

and store it with the idempotency key:

When a retry arrives: Same request has is calculated ABC123 requestHash = HASH-XYZ, then the server knows: Same key + Same request = Valid retry. But if it receives: ABC123 requestHash = HASH-ABC then Same key + Different request = Invalid reuse then API can reject the request, for example with 429 conflict. The exact response contract depends on the API design. Now consider real time scenario, there are chances of collision risk if 2 requests at same timestamp could collide. For example If you generate the idempotency key from a request hash

idempotencyKey = SHA256(userId + amount + timestamp);

Two requests at same timestamp could collide. A better approach would be

idempotencyKey = UUID.randomUUID(); // Primary key (guaranteed unique)
requestHash = SHA256(requestBody); // // For validation only

This separates concerns: UUID: Handles uniqueness (never collides) Hash:Validates request hasn’t mutated

In general different requests looks like

Press enter or click to view image in full size

When ABC123 arrives again:

SELECT *
FROM payment_requests
WHERE idempotency_key = 'ABC123';

The system finds the existing operation and does not execute the payment again. Instead, it returns the original result. This is where a race condition can occur. At this point, a naive implementation might look like:

if (!paymentExists(idempotencyKey)) {
    createPayment();
}

It looks safe but in real it is not safe under concurrency. Imagine two requests arrive at almost exactly the same time.

This results in double payment and customer may be charged twice for same payment. This is classical race condition and it happens because operation is not atomic. This introduce to another concept database constraints on key.

Database Constraints for Idempotency key

Unique constraint is quite important for idempotency key in our case. Let’s consider we create UNIQUE INDEX something like this

CREATE UNIQUE INDEX 
idx_payment_idempotency 
ON payment_requests(idempotency_key);

Now suppose two threads attempt:

INSERT INTO payment_requests 
(idempotency_key) 
VALUES ('ABC123');

The database guarantees that only one can successfully create the record.

Conceptually it may look like this:

Thread A ─────→ INSERT ABC123 ─────→ SUCCESS

Thread B ─────→ INSERT ABC123 ─────→ DUPLICATE KEY

This gives us an important concurrency guarantee. Now question may come what should Idempotency record contain?

What Should the Idempotency Record Contain?

Payment state overview:

Before diving deeper, here’s how states flow.

The payment status can be one of:

• PROCESSING — Payment initiated, not yet complete
• SUCCESS — Confirmed completed
• FAILED — Definitively failed
• UNKNOWN — Ambiguous (network failure or timeout)

This distinction is critical for reconciliation logic.

Press enter or click to view image in full size

Record may look like

idempotencyKey | requestHash | paymentId | status | response | createdAt

The payment status might be:

PROCESSING | SUCCESS | FAILED | UNKNOWN

This state model for payment status depends on the payment architecture. Now let’s see how this status may help. Suppose Request A starts: ABC123 The system creates:

ABC123 → PROCESSING

Then Request B arrives with the same key. It sees:

ABC123 → PROCESSING

It must not execute the payment independently. The API can handle this in multiple ways:

Option 1 — Wait: Wait for the original request to finish and return its result.

Option 2— Return 202 Accepted: with response

{
  "paymentId": "PAY001",
  "status": "PROCESSING"
}

Option 3 — Return current status:

The client can query — GET /payments/PAY001.

The important principle is

A duplicate request must never independently execute the payment.

The Idempotency Record Is More Than a Cache Entry

Sometimes an idempotency implementation can use a cache such as Redis to make lookups fast, rather than storing in database. For example
Client -> Payment API -> Redis → “ABC123 already processed”. But payment idempotency is more than response caching. The system needs to know:

Which operation was this?
        ↓
Which payment did it create?
        ↓
What is the current payment state?
        ↓
What request did the key originally represent?
        ↓
What response should a retry receive?

That’s why a payment system often maintains durable payment state rather than treating the idempotency key as simply a cached HTTP response.

Now let’s analyse what happens if network crashed

The hardest problem: The Network Crashes

Now things get interesting. Suppose

Request -> Create idempotency record -> PROCESSING -> Debit €100 -> Application crashes

Now database state may look ABC123 → PROCESSING, but account contains €900 instead of €1000. The system now has an ambiguity: Did the payment happen? If the service simply retries the debit, it may charge twice.If it refuses to retry forever, the payment may remain stuck. This is why idempotency cannot be treated as merely a cache problem.

Why Redis Alone Is Not Enough

A common implementation is: Redis ABC123 → PROCESSING. Redis can be extremely useful for:

• Distributed locking
• Request deduplication
• Short-lived state
• Fast lookups

For example: SET ABC123 PROCESSING NX EX 3600

NX means: Only set the key if it doesn’t already exist. This prevents two workers from simultaneously acquiring the same lock. But imagine

Redis → ABC123 = PROCESSING

Application ->Debit €100 -> Crash

The Redis state doesn’t necessarily tell us whether the financial operation succeeded. Therefore, for a payment system, the durable source of truth matters.

Redis can help coordinate payment processing, but it should not be blindly treated as the source of truth for the financial transaction.

This brings to our next topic Database transactions.

Database Transactions

If the payment record and account balance are in the same database, a transaction can provide a much stronger guarantee. Let’s consider this

BEGIN TRANSACTION 
  Create idempotency record -> PROCESSING
  Create payment 
  Debit account
  Update payment = SUCCESS 
COMMIT

If the application crashes after DEBIT but before COMMIT: ROLLBACK. The database can roll back all changes belonging to that transaction(idempotency record, debit).This gives us Idempotency record + Payment record + Balance update as one atomic unit. However, this only works directly when the operations participate in the same transactional boundary. What if two threads execute simultaneously? The UNIQUE(idempotency_key) constraint ensures only one INSERT succeeds. Let’s see how to implement this from code perspective. I will use code using SpringBoot

@Transactional 
public PaymentResponse processPayment( String idempotencyKey, PaymentRequest request) { 
   // Note: Even with @Transactional, this check-then-act is vulnerable to 
   // race conditions without a UNIQUE constraint on idempotency_key
  Payment existing = paymentRepository.findByIdempotencyKey(idempotencyKey); 
  if (existing != null) { 
    validateSameRequest(existing, request); 
    return existing.toResponse(); 
    }

    // If two threads reach here simultaneously with the same key,
    // the database UNIQUE constraint will catch the second one
     
   try {
   Payment payment = new Payment(); 
   payment.setIdempotencyKey(idempotencyKey); 
   payment.setStatus(PaymentStatus.PROCESSING); 
   // Perform Debit
   paymentRepository.save(payment); 
   accountService.debit( request.fromAccount(), request.amount() ); 
   payment.setStatus(PaymentStatus.SUCCESS); paymentRepository.save(payment); 
   return payment.toResponse(); 
} catch (DataIntegrityViolationException e) {
        // UNIQUE constraint violation = duplicate idempotency key
        // Retrieve the existing payment and return it
        Payment existing = paymentRepository.findByIdempotencyKey(idempotencyKey);
        if (existing != null) {
            validateSameRequest(existing, request);
            return existing.toResponse();
        }
        // If we can't find it, something is seriously wrong
        throw new PaymentException("Duplicate key but payment not found", e);
    }
}

But this code is not production ready. The database must also enforce: UNIQUE(idempotency_key) because two concurrent requests can both pass the initial lookup.

Idempotency Key Scope Matters

Another question may come to mind

How long should an idempotency key remain valid?

Suppose a key is stored forever. The database grows indefinitely. So systems generally define a retention period. For example:

Idempotency key
       ↓
stored for X hours/days
       ↓
expired

The correct duration depends on the business operation and retry behavior. For financial operations, this decision should be deliberate rather than arbitrary number that justifies business requirements. Let’s talk about the cleanup strategy. How to clean up expired keys ? A approach would be

Set retention based on retry window

• Payment systems: 24–48 hours
• API retries: Typically within minutes, so 1 hour is safe

Scheduled cleanup job (cron, every 6 hours)

DELETE FROM payment_requests 
WHERE created_at < NOW() - INTERVAL '48 hours'
AND status IN ('SUCCESS', 'FAILED');

What if a client retries after expiration?

• Option A: Treat as new payment (risky for user)
• Option B: Return “Request expired, cannot retry”
• Option C: Client generates a new idempotency key

Best practice: 48+ hours for financial operations.

As working with distributed system, as user grows number of server will grow. We will now understand what happens in such case.

What Happens With Multiple Application Instances?

In current times Payment services are rarely a single server. We have a load balancer and it will redirect the users to different instances of payments server running.

Two identical requests might arrive at different instances:

ABC123 → API-1

ABC123 → API-3

An in-memory structure like Set<String> processedKeys; doesn’t solve this because API-1 memory ≠ API-3 memory. And if API-1 restarts all in-memory state disappears. Therefore idempotency state must be shared and durable enough for the required guarantees.

External Payment Providers Make Things Harder

Real payment systems often look like:

Client -> Our Payment Service -> Payment Provider -> Bank / Card Network

Suppose our service sends €100to an external provider. The provider processes the transaction. Then Provider → SUCCESS. But the network connection fails before our service receives the response.Our service sees UNKNOWN. Now the dangerous thing to do is just send another €100 because the provider may already have processed the first transaction. That’s where we provide Provider-Level Idempotency.

Provider-Level Idempotency

This is why payment providers commonly support their own idempotency mechanism or unique transaction references. Our system might generate PAY-20260808-ABC123 and send that reference to the provider. If we retry PAY-20260808-ABC123 the provider can recognize as “This is the same payment operation.” This gives us multiple layers of protection

Now let’s understand UNKNOWN state.

UNKNOWN Is an Important Payment State

Many engineers think in terms of SUCCESS and FAILED as state mechanism. Payment service have different state SUCCESS, FAILED, PROCESSING, UNKNOWN. I guess all states are clear to us, let’s see what UNKNOWN means “We cannot safely determine the outcome yet.” This is fundamentally different from FAILED. For example:

Request sent to provider
       ↓
Connection timeout
       ↓
UNKNOWN

The correct next step may be:

Query provider
       ↓
What happened to PAYMENT-001?

rather than Create another payment.

Reconciliation

What if the provider and our database disagree? For example

Our DB: PAY001 → PROCESSING

Provider: PAY001 → CAPTURED

We need a reconciliation mechanism. For example

Scheduled reconciliation job -> Find old PROCESSING payments -> Query provider -> Update internal state

This is extremely important in financial systems because failures can occur at any boundary.

Idempotency Is Also Important for Webhooks

Imagine the payment provider sends PAYMENT_SUCCESS to our webhook.Our service processes it. Then the provider retries the webhook because it didn’t receive our acknowledgement.We receive

Webhook #1 → PAYMENT_SUCCESS

Webhook #2 → PAYMENT_SUCCESS

Webhook #3 → PAYMENT_SUCCESS

If webhook handling isn’t idempotent, we could accidentally:

Credit customer €100

Credit customer €100

Credit customer €100

Therefore:

Therefore Webhook consumers must also be idempotent.

A common pattern is to store a unique provider event ID event_id = evt_123 and enforce uniqueness.

Code example

//Store the provider's event ID and enforce uniqueness:

@RestController
public class WebhookController {
    
    @PostMapping("/webhooks/payment-success")
    public ResponseEntity<?> handlePaymentSuccess(@RequestBody WebhookPayload payload) {
        try {
            // Check if we've seen this event before
            WebhookEvent existing = webhookEventRepository
                .findByProviderEventId(payload.getEventId());
            
            if (existing != null && existing.isProcessed()) {
                // Already processed — return success to prevent provider retry storm
                return ResponseEntity.ok().build();
            }
            
            // Record as PROCESSING
            WebhookEvent event = new WebhookEvent(
                payload.getEventId(), 
                WebhookStatus.PROCESSING
            );
            webhookEventRepository.save(event);
            
            // Credit customer
            accountService.credit(payload.getCustomerId(), payload.getAmount());
            
            // Mark as complete
            event.setStatus(WebhookStatus.SUCCESS);
            webhookEventRepository.save(event);
            
            return ResponseEntity.ok().build();
            
        } catch (DataIntegrityViolationException e) {
            // Duplicate event_id — already processed
            // Return 200 OK anyway (idempotent response)
            return ResponseEntity.ok().build();
        }
    }
}

The Complete Picture

A reasonably robust payment architecture looks like

                         Client
                           |
                           |
                  Idempotency-Key
                           |
                           ↓
                    Payment API
                           |
                           ↓
              ┌─────────────────────┐
              │ Idempotency /       │
              │ Payment Database    │
              └─────────────────────┘
                           |
                           ↓
                  Payment State
                           |
                           ↓
                Payment Provider
                           |
                           ↓
                       Bank/Card
                           |
                           ↓
                       Webhook
                           |
                           ↓
                 Idempotent Consumer
                           |
                           ↓
                  Payment Database
                           |
                           ↓
                    Reconciliation

Every boundary introduces potential failure. Every retry introduces the possibility of duplication. Idempotency provides protection against those repeated operations.

Now comes the tricky part

What Should a QA Engineer Test?

With no UNIQUE constraint

// This looks safe but ISN'T
if (!paymentRepository.existsByIdempotencyKey(key)) {
    createPayment();
}

Test result

• 100 concurrent requests with same key
• Expected: 1 payment, €100 debit —
• Actual: 50–100 duplicate payments, €5,000-€10,000 debit

Why it fails Both threads can pass the existsByIdempotencyKey check before either one calls save().

This is where idempotency becomes particularly interesting from a QA perspective. Consider a case Click Pay Expected: SUCCESSis the easy part.The valuable tests are the failure scenarios.

Normal Payment

Request: Key = ABC123 Amount = €100

Expected: Payment count = 1 Debit = €100 Status = SUCCESS

Same Request Repeated

Send: POST /payments Key = ABC123 five times sequentially.

Expected: Payment count = 1

All responses should refer to the same payment.

Concurrent Requests

This is one of the most important tests. Send 100 concurrent requests with Idempotency-Key = ABC123 Amount = €100

Expected: Payments created = 1 Total debit = €100

Not: Payments created = 100
Total debit = €10,000

This test specifically targets the race condition.

Same Key, Different Amount

Request 1: ABC123 → €100

Request 2: ABC123 → €500

Expected: Request 2 rejected

The original operation must remain unchanged.

Different Keys

Request 1: ABC123 → €100 Request 2: XYZ789 → €100

Expected: 2 legitimate payments Total debit = €200

This ensures our deduplication isn’t too aggressive.

Response Lost After Payment

Simulate: Payment processed -> Response dropped -> Client retries

Expected:

One payment
One debit
Same payment ID

This is one of the most realistic idempotency scenarios.

Application Crash

Inject a failure after Payment created but before Payment completed.Then retry. Expected behavior depends on the transaction model, but the critical invariant is:The system must never create two financial effects for one logical payment.

Crash Around the External Provider Call

This is even more important.

Simulate:

Our service -> Provider request ->Provider processes €100 ->Our service crashes

Then restart and retry. Expected: One provider transaction not: Two provider transactions. This validates provider-level idempotency or transaction-reference handling.

Duplicate Webhook

Send: event_123 three times. Expected: Customer credited once not three times.

Multiple Application Instances

Run: Send three concurrent requests API-1 , API-2 , API-3.

Expected: One payment regardless of which instance receives the request. This catches implementations that incorrectly depend on local memory.

I can give an example how we can do this with Restassured and Java

ExecutorService executor =
    Executors.newFixedThreadPool(100);
List<Future<Response>> futures = new ArrayList<>();
for (int i = 0; i < 100; i++) {
    futures.add(
        executor.submit(() ->
            given()
                .header(
                    "Idempotency-Key",
                    "ABC123"
                )
                .body(paymentRequest)
            .when()
                .post("/payments")
        )
    );
}

Then verify: Database payment count = 1 and: Account balance decreased by exactly €100 . This is much more valuable than simply asserting HTTP 200.

A question may come to our mind PUT is used for idempotency the why we stick to POST.

PUT vs POST

HTTP methods can sometimes confuse this discussion. A PUT such as PUT /accounts/A may be designed to be idempotent, but POST /payments usually means Create/process a new payment.So simply changing the HTTP method isn’t enough.Payment idempotency is about business operations, not just HTTP verbs.