Webhooks will lie to you

The first time I integrated a mobile money gateway, I wrote the callback handler the way the documentation implied I should. The provider posts a payload, you look up the transfer, you mark it complete, you return 200. Four lines of real logic. It worked in staging, it worked in the pilot, and it worked in production for about three weeks.

Then a recipient in Douala was credited twice.

Three assumptions, all wrong

The handler I had written assumed three things that no payment provider actually guarantees.

That the callback arrives once. It does not. Providers retry on any response they cannot confirm - a timeout, a 502 from your load balancer, a connection reset while your process was restarting. From their side, an unacknowledged webhook is indistinguishable from a lost one, so they send it again. Correctly. The duplicate is not a bug in their system; it is the retry policy working.

That callbacks arrive in order. They do not. If a provider fires payment.pending and payment.completed within a few hundred milliseconds, those are two independent HTTP requests racing across the internet. The second one can land first. A handler that writes status directly from the payload will happily move a completed transfer back to pending.

That the payload is trustworthy. An endpoint that accepts a POST and credits an account is an endpoint anyone can POST to. Signature verification is not a hardening step for later.

The shape that survives

The fix is not defensive coding sprinkled through the handler. It is a different model of what a callback is. A webhook is not a command telling you to do something. It is a notification that state may have changed somewhere else, and the only correct response is to reconcile against a record you own.

Four properties, in order of how much they matter.

1. Verify before you parse

Check the signature against the raw request body, before deserialisation. Once the body has been parsed and re-serialised, byte-level equality is gone and the signature will not match - a genuinely irritating afternoon to debug.

// The raw body must be preserved. Parse after verifying, never before.
app.post(
  "/webhooks/payments",
  express.raw({ type: "application/json" }),
  (req, res) => {
    if (!verifySignature(req.body, req.header("x-provider-signature"))) {
      return res.sendStatus(401);
    }
    const event = JSON.parse(req.body.toString("utf8"));
    // ...
  },
);

2. Deduplicate on the provider's identifier

Every serious provider sends a stable event ID. Insert it into a table with a unique constraint before doing any work. If the insert conflicts, you have seen this event; acknowledge and stop.

The unique constraint is the important part. Checking for existence and then inserting is a race that two concurrent retries will win together.

const inserted = await events.insertOne({ _id: event.id, receivedAt: new Date() })
  .catch((e) => (e.code === 11000 ? null : Promise.reject(e)));

if (!inserted) return res.sendStatus(200); // already processed

3. Guard the transition, not the value

Never write the status the payload asked for. Write the transition, conditional on the current state being one you are allowed to move from. A completed transfer receiving a late pending event matches no valid transition, so nothing happens - which is exactly right.

const result = await transfers.updateOne(
  { _id: transferId, status: { $in: ["initiated", "funded"] } },
  { $set: { status: "settled", settledAt: new Date() } },
);

if (result.modifiedCount === 0) {
  // Not an error. Either a duplicate, or an out-of-order event we
  // correctly declined to apply. Log it and move on.
}

This single change eliminates the out-of-order problem completely, without any timestamp comparison or event ordering logic.

4. Acknowledge fast, work later

The provider is measuring your response time and will retry if you are slow. Do the verification, the dedupe insert and the state transition - all cheap - then return 200. Anything expensive triggered by the event, like sending a receipt or notifying a device, belongs on a queue.

A 200 means "I have durably recorded this event". It does not mean "I have finished reacting to it".

Reconciliation is not optional

Even with all four properties in place, some events never arrive. The provider had an outage, your DNS was briefly wrong, a message was dropped somewhere nobody will ever be able to prove.

So a scheduled job walks every transfer that has been sitting in a non-terminal state for longer than expected and asks the provider directly what happened. Webhooks make the system fast. The reconciliation job makes it correct. You need both, and if you had to lose one, lose the webhooks.

That job is unglamorous, it runs at three in the morning, and nobody will ever compliment you on it. It is also the reason the numbers add up.

The one-line version

If you take one thing from this: write the state transition as a conditional update, keyed on the provider's event ID, before you do anything else. Almost every duplicate-payment story I have heard reduces to a handler that skipped that.

The double credit in Douala took an hour to trace and a week of reconciliation to be confident about. The fix was about fifteen lines.