9 steps
A normal payment
You scan a shop’s code and press Pay. Here’s everything that happens in about 44 ms.
Step 1 of 9
The wallet picks a key first
Before the first attempt, the wallet chooses an idempotency key for this payment. It sends the request with your PIN approval, bound to that one key.
Step 2 of 9
The edge decides who you are
Envoy verifies your signed token, throws away any identity headers you claimed and sets its own. The service behind it trusts only that.
Step 3 of 9
Claim the key
A conditional write records this request. If the same key arrives again, the service finds this record and returns the original answer.
Step 4 of 9
Spend the QR code
Inserting the payment under a unique key on the QR code is what spends it. A second attempt on the same code fails on the key, so a cache can’t be wrong here.
Step 5 of 9
Reserve the money
The payment records AUTHORIZING, then asks the ledger to hold the amount. One conditional UPDATE does it. The row count is the verdict.
Step 6 of 9
Write balanced postings
On capture, ¥1,200 leaves the wallet, ¥1,164 is owed to the shop and ¥36 is fee revenue. Debits equal credits. The journal is append-only.
Step 7 of 9
State and event are one write
The result and a PaymentCaptured event are written in one transaction. The customer gets their answer now. Nothing has touched Kafka yet.
Step 8 of 9
Publish afterwards
A relay publishes the event later, carrying the original trace with it, so one payment is one trace from the edge to settlement.
Step 9 of 9
Settle at the end of the day
The settlement worker consumes the event. At payout, the ledger claims an exact set of postings. Three independent records must agree.
7 steps
The ledger’s answer is lost
This is the case the whole design is built around: the money may or may not have moved and nobody knows which.
Step 1 of 7
Write down the intent
The payment records CAPTURING before asking the ledger. This write-ahead state is what makes the next steps safe.
Step 2 of 7
Ask the ledger to capture
The request carries a key derived from the payment’s id, such as capture:{id}.
Step 3 of 7
The answer never arrives
The ledger did the work and committed it, but the response is lost on the network. The payment service sees a timeout.
Step 4 of 7
The API doesn’t guess
It answers 202 Accepted: “we don’t know yet”. Claiming success or failure here is how real systems lose money.
Step 5 of 7
Recovery re-drives the call
Every few seconds a recovery job finds work stuck in a write-ahead state and repeats the call with the same key.
Step 6 of 7
The ledger recognizes the repeat
A unique index finds the existing entry, so it returns what it already did instead of doing it again. The money moved exactly once.
Step 7 of 7
Finish normally
The payment completes. In the run where the ledger was killed twice, 4,005 payments ended like this. Every one resolved to exactly one outcome.
4 steps
A client retries
Your connection drops after you press Pay, so the wallet tries again. The customer must not be charged twice.
Step 1 of 4
The same request, again
The wallet resends with the same idempotency key. It doesn’t know the first attempt worked.
Step 2 of 4
Through the edge as usual
Nothing about the retry is special at this point.
Step 3 of 4
The key is already claimed
The conditional write fails, because this key has a stored answer.
Step 4 of 4
Same answer, no money moved
The original result is returned and the ledger is never called. I planted a bug that ignored these keys to prove the audit catches it: 200 of 200 wallets came out wrong.
5 steps
Kafka freezes
A dependency goes down. Each one has a decided failure mode. This one shouldn’t affect customers at all.
Step 1 of 5
Payments keep arriving
Nothing on the request path writes to Kafka, so customers can’t tell anything is wrong.
Step 2 of 5
Money still moves
The payment completes as normal. In the Kafka-frozen run, no request failed and p99 was 80 ms.
Step 3 of 5
Events wait in the outbox
Each event is safely in the database, in the same transaction as its payment. The alert watches how old the oldest unpublished event is.
Step 4 of 5
The relay can’t publish
Publishing fails and the rows stay put. Nothing is lost, because nothing has left the database.
Step 5 of 5
Kafka returns
The relay drains the backlog in order. Settlement catches up and the audit passes.
Component
Wallet
The client chooses an idempotency key when it shows “Pay ¥1,200”, before the first attempt. A double-tap or a retry sends the same key, so it can only ever be one payment.
Component
Envoy edge
The only way in. It verifies RS256 tokens and HMAC-signed shop requests, strips any identity a client claims and applies rate limits. No service publishes its own port.
Component
Payment service
Owns the life of every payment. It writes down what it’s about to ask the ledger before asking, so a crash or a lost answer leaves a record a recovery job can finish.
Component
Ledger
The only place a balance exists. It makes no outbound calls. Its database user can’t update or delete history. Every entry’s debits must equal its credits.
Component
DynamoDB
One record per client request, with the stored answer. Conditional writes make a retry recognizable. If this store is down, payments stop, because a retry could charge twice.
Component
Redis
Holds QR codes while they’re valid. It’s only a fast path: the code is really spent by a unique key in TiDB, not by deleting it here.
Component
TiDB
Everything that must be right. Each service has its own schema and database user. Unique keys and conditional updates here are what make the system correct.
Component
Outbox relay
Events are written in the same transaction as the change they describe, then published afterwards. There’s no moment when a payment is captured and its event doesn’t exist.
Component
Kafka
Everything downstream learns of changes here. A payment never waits for it. If Kafka freezes, payments keep succeeding and events queue in the outbox.
Component
Settlement
Pays merchants their net and reconciles three ways: events against ledger batches, payouts against entries and the bank statement against clearing accounts.
Component
Bank stub
A simulated bank with scripted latency, errors and timeouts, so failure paths can be tested on demand. All money in Maat is play money.