The bug that taughtme to take walks

It was a Tuesday, which is important only because nothing interesting is supposed to happen on a Tuesday. A payment job that had run quietly for two years started failing — not always, just often enough to be rude.

I did what every engineer does. I added logs. Then more logs. Then I added logs to the logs. By Thursday I had a terminal full of timestamps and a head full of static, and I was no closer to knowing why.

The part where I gave up (briefly)

Around 4pm I closed the laptop, mostly out of spite. I walked to the park near our office, sat on a bench, and watched a man try to convince his dog that the ball was, in fact, worth returning.

"The bug wasn't in the code I was staring at. It was in the assumption I'd stopped noticing."

Somewhere between the dog's third refusal and the man's fourth sigh, it clicked: the job wasn't failing randomly. It failed whenever two retries landed in the same second — because we'd been using the timestamp as an idempotency key.

// what we had (oops)
const key = `pay_${Date.now()}`;

// what we needed
const key = `pay_${order.id}_${attempt}`;

What I do now

• If I've been stuck for more than an hour, I leave the desk. No phone.
• I write down the assumptions I'm making — especially the boring ones.
• I explain the bug out loud to someone. A rubber duck works. A dog works better.

The fix was one line. The lesson took three days. I think that's a fair trade.