I Blocked Double Execution - Then One Crash Locked It Forever

I added a guard to stop a background job from running twice. It checked status with a conditional update - textbook code. It passed review, and every test was green. But this guard had a hole that could lock the entire feature forever after a single deploy. This is the record of finding and fixing it a day later.

Why the guard existed

The feature in question is a batch audio-generation job in an admin tool. One job carries dozens of items, and each item calls an external speech-synthesis API. Every call is billed, and each takes minutes.

The scariest failure for a job like this is double execution. If an admin clicks the button twice, or runs the same job from two browser tabs, you generate the same audio twice and pay twice. So I put a guard at the entry point: flip the job to GENERATING only if it isn't GENERATING already - a one-line conditional update (compare-and-set).

const claimed = await prisma.job.updateMany({
  where: { id: jobId, status: { not: "GENERATING" } },
  data: { status: "GENERATING" },
});
if (claimed.count === 0) {
  throw new ConflictException("Job is already generating");
}

Even if two requests land at the same moment, the database lets exactly one through. So far, correct code. The problem is that this code is only half of the job.

Crashes are not caught

When generation finishes, the status becomes COMPLETED; if an exception occurs, it becomes FAILED. Everything is wrapped in try/catch, so whatever error happens, the status recovers - or so it's easy to think.

But there are ways a process dies without throwing anything. Two big ones:

• Deploys. A container orchestrator sends a termination signal to old tasks while spinning up the new version, and after the grace period it sends SIGKILL. SIGKILL cannot be caught. No catch, no finally, no shutdown hook ever runs.
• OOM. When memory runs out, the kernel kills the process on the spot. Again, no code gets a chance to run.

For a job that takes minutes, a deploy landing mid-generation is a matter of time. Walk through what happens next. The status stays GENERATING. The admin clicks generate again. The guard looks: status is GENERATING? Must be running. Rejected. An hour later, rejected. The next day, rejected. No process is actually running anywhere, yet one leftover status in the database blocks every retry.

The lock was installed to prevent double execution, but the process holding the key is dead. If this state needs a name, it's a crash-wedge: one crash drives the wedge in, and it never comes out.

How it was found: three sibling features had the missing piece

This hole wasn't discovered through an outage. It turned up while re-reading the code. Three other features in the same codebase used a similar guard (speech transcription, an asset pipeline, CDN warmup), and one piece of code present in all three was missing from this job: stale recovery, which flips long-stalled jobs back to failed.

I had copied half a pattern. The guard is the visible half. Duplicate execution is a concrete fear, the guard is one line, and it gives an instant answer when a reviewer asks "did you block double runs?" The unlock path, on the other hand, is invisible precisely because it's absent - nothing looks wrong until it fires. The moment locking code and unlocking code get treated as separate commits and separate concerns, this asymmetry creeps in.

The fix: a lock that releases itself

The direction is well known: give the lock a lifetime. Distributed-locking folks call it a lease. I added recovery logic to the read path. When a job is fetched and it's GENERATING but its last sign of progress is older than 30 minutes, flip the job to FAILED and its items to ERROR. Since FAILED passes the guard again, that transition is the unlock.

One sentence to describe, but three traps stood between it and a correct implementation.

Telling slow apart from dead

A job with many items can legitimately run past 30 minutes. Judge by the job's start time alone and you'll declare a hard-working large job dead and fail it.

Fortunately, a live job touches each item's update timestamp as it completes it. So liveness is judged by the most recent of the job's own update time and the newest item update time. A single item takes at worst about 3 minutes including retries, so 30 minutes with no updates at all is a signal no healthy run can produce. Put the timeout on the gap between progress signals, not on total job duration.

Racing other recoverers - and the real worker

Recovery lives on the read path, so several requests can attempt it at once. There's a nastier case too: the worker you assumed dead is actually alive, and right after you read "this looks stale," it finishes the job as COMPLETED. Blindly overwriting that with FAILED turns a perfectly completed job into a failure.

So the recovery itself is written as the same kind of conditional update as the guard: only touch rows whose status is still GENERATING and whose update time is still older than the threshold.

const recovered = await prisma.job.updateMany({
  where: {
    id: jobId,
    status: "GENERATING",
    updated_at: { lt: staleThreshold },
  },
  data: { status: "FAILED", error_message: "auto-recovered" },
});
// count 0 means a worker cleaned up in the meantime - do nothing

If zero rows were updated, someone else handled it between the read and the write, so back off quietly. The technique learned while adding the lock gets used again to release it.

On the read path, not a cron

A periodic cron could have run the recovery. I chose to check at fetch time instead. A wedged job only becomes a problem the moment someone comes to look at it. There's no need to diligently clean up jobs nobody is watching, and no reason to take on a cron - one more piece of infrastructure with failure modes of its own.

With all of that in place, five tests went in: a wedged job gets recovered; items wedged after the job itself finished get recovered separately; a freshly started job is left alone; an old job whose items are still updating stays alive; and the recovery backs off when it loses the race.

Three questions to ask whenever you add a lock

This pattern isn't specific to this job. A Redis lock acquired without a TTL, an is_processing boolean flag set outside a transaction, an advisory lock with no release queue - all the same family. Ask three questions every time you write code that locks something, and you'll avoid this trap.

1. If the locker dies without warning, who unlocks? catch and finally are not the answer. SIGKILL and OOM give your code no chance to run.
2. How do you tell alive from dead? Look at the last progress signal, not the start time.
3. Did you handle races in the release logic itself? Recovery is concurrency code too. The conditional update you used to lock is the one to use to unlock.

Compressed into one sentence: the commit that adds a lock must also carry the path by which it unlocks. Code with a guard and no release is only half finished, and one day, when a deploy lands at just the right moment, that half will demand the other.