I have been building payment processing infrastructure for Yooga's restaurant POS systems for a while now. When something goes wrong, someone loses real money. These are the lessons I have learned in production.
Every payment needs an idempotency key
Give every payment operation an idempotency key, even if you do not expect the system to retry requests. The network will hiccup. The client will double-tap. The load balancer will timeout and retry. If your payment endpoint is not idempotent, you will process duplicate charges.
We use a composite key: merchant ID plus a client-generated UUID plus the operation type. This gets stored before we even talk to the payment processor. If we see the same key twice, we return the cached result from the first attempt. That avoids a second charge.
Race conditions will find you
We had a race condition where two concurrent requests could both read the same account balance, both verify sufficient funds, and both deduct from it. That let a merchant spend more than they had. We found this because a test restaurant managed to process negative balances during a lunch rush.
The fix was pessimistic locking on the balance check. SELECT FOR UPDATE on the account row, verify funds, deduct, commit. This serializes concurrent payments for the same account and is slightly slower. I am fine with that: I will take correctness over performance in a payment system every time.
But that was the easy race condition. The hard one was with concurrent card terminal transactions. Two POS devices, same merchant, both hitting the payment processor at the same time. The processor returns success for both, but our webhook handler processes them out of order and the second transaction overwrites the status of the first. We lost visibility into completed payments.
We moved to event sourcing for transaction state changes. Every status update is an append-only event with a sequence number. We reconstruct current state from the event log. Nothing gets overwritten, and we keep the full audit trail. If events arrive out of order, we resequence based on the processor timestamp.
Duplicate transactions
The most expensive bug I have ever shipped involved duplicate transactions.
We had a webhook retry mechanism. When our processor sent a payment confirmation and we failed to acknowledge it (HTTP 200), they would retry. Standard stuff. Except our handler was not checking if the transaction had already been recorded. So every retry created a new transaction record in our system.
One busy Friday night, our webhook endpoint went down for about 90 seconds due to a deployment. The processor queued up retries. When we came back online, we got hammered with retry webhooks. Each one created a duplicate transaction. That left duplicate entries across the affected merchants.
We caught it within two hours because of balance reconciliation alerts. But those two hours were the longest of my career. We had to manually reverse every duplicate, contact every affected merchant, and explain what happened.
The fix was embarrassingly simple. Check the processor transaction ID against our records before creating a new entry. If it exists, acknowledge and skip. A five-line fix to prevent duplicate entries.
Reconciliation catches what you missed
Every night at 3 AM, we run a reconciliation job. It pulls every transaction from our system and every transaction from the payment processor for the past 24 hours. It compares them. Any mismatch triggers an alert.
This has caught bugs that nothing else would have found. Silent failures where we recorded a payment but the processor actually declined it. Edge cases where partial refunds got lost. Timezone bugs where a transaction showed up on different days in our system versus the processor.
Reconciliation is not glamorous. Nobody puts it on their resume. But it is the single most important piece of infrastructure in a payment system. Build it before you process your first real transaction.
Recalculate amounts on the server
The POS terminal sends the amount to charge. Do not trust it. Recalculate on the server from the order items. We had an incident where a modified APK on a compromised terminal was sending lower amounts than the actual order total. The merchant was essentially giving discounts they did not intend to give.
Server-side amount calculation from the source of truth (the order in our database) is non-negotiable. The terminal amount is for display purposes only.
The algorithms in payment systems are straightforward. The difficulty is that every edge case costs real money. A race condition in a blog platform means a duplicate post. A race condition in a payment system means someone loses money. The stakes change how you think about every line of code.
Start with idempotency and use locks aggressively. Reconciliation still has to cover everything, and the client cannot be trusted with amounts. For the love of all that is holy, test your webhook handlers with duplicate deliveries before you go to production.