The 3 AM Page: A Postmortem I Actually Learned Something From

My phone buzzed at 2:47 AM with a PagerDuty alert that just said checkout error rate above threshold, and I remember lying in bed for about ten seconds negotiating with myself about whether it would resolve on its own before deciding, correctly, that it would not. This is the story of that incident, not because it was especially dramatic, but because the actual root cause taught me something about how bugs hide that I still think about constantly.

The first twenty minutes

I opened my laptop on the kitchen counter because waking up my partner by turning on a bedroom light felt like a worse outcome than a slightly stiff back. The dashboard showed checkout error rate climbing from a baseline of about 0.3 percent to somewhere north of 40 percent, which for a store processing real orders is about as bad as it gets without the site being fully down. The error logs were full of a message I had never seen before: PaymentIntentConfirmationError: resource_missing.

My first instinct, like most people's first instinct at 3 AM, was to check the obvious things. Was Stripe having an outage? I checked their status page, nothing reported. Had we deployed recently? Our deploy log showed the last release was eleven hours earlier, well before the errors started climbing, which ruled out the easiest explanation, a bad deploy causing an immediate spike. That eleven-hour gap between deploy and incident is the detail that should have told me something interesting was going on, but at 3 AM your brain is not exactly running at full analytical capacity, and I spent the next twenty minutes chasing a theory that our Stripe webhook signature verification had started failing due to a clock drift issue on one of our servers. It had not.

The wrong turn that actually helped

I want to be honest that the clock drift theory was wrong, but chasing it was not wasted time, because ruling it out forced me to actually read the full stack trace instead of just the error message, and stack traces are where the real information usually lives. Buried a few frames down was a reference to a payment intent ID, and when I looked that ID up directly through the Stripe dashboard, it did not exist. Not expired, not canceled, just gone, as if it had never been created.

That detail cracked the case open. We were creating payment intents through an idempotency key based on the cart ID, which is a completely reasonable pattern, you do not want a double-click on the checkout button to create two charges. But idempotency keys in Stripe expire after 24 hours. Our idempotency key generation used the cart's creation timestamp as part of the key, and it turned out that a batch job, one that had existed for eight months without a single reported issue, was periodically reactivating abandoned carts older than 24 hours as part of a win-back email flow, without updating their creation timestamp.

So the actual sequence was: a cart gets abandoned, sits for more than a day, our win-back job sends an email and marks the cart active again, the customer comes back and clicks checkout, our code reuses the idempotency key from the cart's original creation time, and Stripe rejects it because that key has expired and, worse, in a specific edge case tied to a Stripe API version we were on, returned a reference to a payment intent that had already been garbage collected on their end rather than a clean error about the key being invalid. The confusing resource_missing message was Stripe's side effect of an expired idempotency key colliding with an internal cleanup process, not a direct, obvious symptom of the actual bug.

Why it took an eight-month-old bug to surface that night

The part that stuck with me afterward was realizing this bug had been live in production since the win-back email feature shipped. It had presumably been causing a small trickle of failed checkouts the entire time, just below the volume needed to trip any alert threshold, because most carts do not sit abandoned for over 24 hours and then get revived. What changed that night was a marketing campaign that had gone out earlier in the evening specifically targeting a segment of month-old abandoned carts, which meant a much higher than usual fraction of checkout attempts that night were hitting exactly the cart-age condition that triggered the bug. The bug was not new. The traffic pattern that exposed it at scale was new.

This is the thing about production bugs that I think junior engineers, and to be fair a lot of senior engineers too, underestimate. Most nasty bugs are not introduced by the most recent deploy. They are latent, sitting quietly in a rarely-hit code path, waiting for a traffic pattern or a data shape that exposes them. Blaming the last deploy is the natural first move because it is the most recent change and recency bias is powerful, but it is often the wrong move, and ruling it out early, the way that eleven-hour gap should have told me, saves a lot of wasted investigation time.

The fix and the more important fix

The immediate fix was small: base the idempotency key on the current checkout attempt timestamp rather than the cart's original creation timestamp, with the cart ID still included to preserve the double-click protection. That shipped within the hour and error rates dropped back to baseline almost immediately.

The more important fix was process, not code. We added an alert specifically on payment intent creation failures broken out by Stripe error type, separate from the generic checkout error rate alert, because a generic aggregate metric had been masking a low, steady failure rate for eight months. We also added a lightweight rule to our incident retro process: any time a bug's actual introduction date is more than a week before its detection date, we write down explicitly what changed to cause it to surface now, not just what the code bug was. That distinction, between the bug's origin and its trigger, turned out to be useful in at least two later incidents where the same pattern showed up: old latent bug, new traffic shape.

What I would tell past me at 2:47 AM

Read the full stack trace before forming a theory, not after. Check the deploy timeline early and take the gap seriously if there is one. And when an error message sounds unfamiliar and vague, treat that vagueness as a clue rather than an obstacle, because vague error messages are often symptoms of your code hitting an edge case the original error message author never anticipated, which is exactly the kind of bug that has been quietly there the whole time, waiting for the right conditions to show up.

A few weeks after the incident I went looking for other places in the codebase where a similar pattern might be lurking, idempotency keys derived from a field that could theoretically become stale relative to the actual request timing, and found two more instances in less critical paths that had not caused a visible incident yet, purely by luck of traffic patterns. We fixed both proactively, and I now treat any incident retro as incomplete if it stops at the specific line that broke, rather than asking whether the same class of mistake exists anywhere else in the system. That habit alone, searching for siblings of a bug rather than just fixing the one that paged you, has caught real problems before they became real incidents at least three more times since that night, and it costs maybe twenty extra minutes during a retro that would otherwise end the moment the immediate fix ships.