When I led payments for a restaurant POS platform, I built its payment processing infrastructure. When something goes wrong in payments, someone loses real money. These are the lessons I learned in production.
Every payment needs an idempotency key
Give every payment operation an idempotency key, even if you do not expect the system to retry requests. The network hiccups, the client double-taps, the load balancer times out and retries. Without idempotency on the endpoint, you will process duplicate charges.
We used a composite key: merchant ID plus a client-generated UUID plus the operation type. It was stored before we even talked to the payment processor. If we saw the same key twice, we returned the cached result from the first attempt. That avoided a second charge.
Race conditions will find you
Two concurrent requests could both read the same account balance, both verify sufficient funds, and both deduct from it. That let a merchant spend more than they had.
The fix was pessimistic locking on the balance check. SELECT FOR UPDATE on the account row, verify funds, deduct, commit. This serializes concurrent payments for the same account and is slightly slower. I am fine with that: I will take correctness over performance in a payment system every time.
But that was the easy race condition. The hard one was with concurrent card terminal transactions. Two POS devices, same merchant, both hitting the payment processor at the same time. The processor returned success for both, but our webhook handler processed them out of order and the second transaction overwrote the status of the first. We lost visibility into completed payments.
We moved to event sourcing for transaction state changes. Every status update became an append-only event, and we derived current state from the event log in the order of the processor timestamp, so a late webhook could no longer overwrite a newer status. Nothing got overwritten, and we kept the full audit trail.
Duplicate transactions
Cause: we had a webhook retry mechanism. When our processor sent a payment confirmation and we failed to acknowledge it (HTTP 200), they would retry. Standard stuff. Except our handler was not checking if the transaction had already been recorded. So every retry created a new transaction record in our system, and any window where the endpoint failed to acknowledge, a deployment included, turned queued retries into duplicate entries.
Detection: the balance reconciliation alerts caught it. We manually reversed every duplicate, contacted the affected merchants, and explained what happened.
What changed: before creating a new entry, the handler checked the processor transaction ID against our records. If it existed, it acknowledged and skipped. The fix was small, and from then on webhook handlers were tested with duplicate deliveries before they reached production.
Reconciliation catches what you missed
Every night, we ran a reconciliation job. It pulled every transaction from our system and every transaction from the payment processor for the past 24 hours and compared them. Any mismatch triggered an alert.
This caught bugs that nothing else would have found. Silent failures where we recorded a payment but the processor actually declined it. Edge cases where partial refunds got lost. Timezone bugs where a transaction showed up on different days in our system versus the processor.
Hardly anyone writes about reconciliation, but to me it is the most important piece of infrastructure in a payment system. Build it before you process your first real transaction.
Recalculate amounts on the server
The POS terminal sent the amount to charge. We did not trust it and recalculated on the server from the order items.
Server-side amount calculation from the source of truth (the order in the database) is non-negotiable. The terminal amount is for display purposes only.
The algorithms in payment systems are straightforward. The difficulty is that every edge case costs real money. A race condition in a blog platform means a duplicate post. A race condition in a payment system means someone loses money. The stakes change how you think about every line of code.
For the love of all that is holy, test your webhook handlers with duplicate deliveries before you go to production.