SIGTERM Had Two Owners: Why Our Shutdown Hooks Ran Twice

Every deploy dropped the same error into Sentry: Transaction already closed. That's Prisma failing with P2028 when it tries to run a query after its transaction has already been closed. Users never saw any impact - BullMQ's retry immediately picked the job up on another task, and the completion logs never showed a gap. But it was a quiet, recurring failure mode any time a deploy overlapped with live traffic. Chasing it down, I discovered our server's shutdown hooks were actually running twice on every single shutdown.

Act One: the database dies before the job does

Lining up the error timestamps against CloudWatch logs, the timeline matched exactly. A rolling deploy sends SIGTERM to the old task. Six seconds later, app.close() starts cleanup, and the Prisma engine closes its transaction. Right after that, an updateMany from a still-running asset-processing job executes against a transaction that's already closed, and fails with P2028. A user happened to be uploading audio back-to-back, so there was an active job at the exact moment of the deploy, and that job's query died.

The root cause was shutdown ordering. NestJS calls lifecycle hooks in a defined order when it shuts down. The problem was that closing the BullMQ worker and disconnecting Prisma both lived in the same shutdown phase (onApplicationShutdown). Within a single phase, module registration order decides execution order - and in our case, Prisma happened to disconnect first, leaving the still-in-flight worker's query with nowhere to go.

The fix direction was obvious: close the worker in an earlier phase. NestJS's shutdown hooks have an ordering, and beforeApplicationShutdown finishes for every module before any module's onApplicationShutdown even starts. I wrote a service that finds every worker and drains it (waits for active jobs to finish) in this earlier phase.

@Injectable()
export class WorkerDrainService implements BeforeApplicationShutdown {
  constructor(private readonly discovery: DiscoveryService) {}

  async beforeApplicationShutdown() {
    // Find every registered WorkerHost and wait for its active jobs to finish.
    // This phase finishes before onApplicationShutdown, where Prisma disconnects.
    const workers = this.discovery
      .getProviders()
      .filter((p) => p.instance instanceof WorkerHost);
    await Promise.all(workers.map((w) => w.instance.worker.close()));
  }
}

Worker.close() returns the same promise on repeated calls, so it doesn't conflict with any existing worker-closing code, and it silently does nothing in environments without Redis. Now, even when a deploy overlaps with a job, the job finishes first and the DB closes afterward. I added a unit test and opened the PR. Up to this point, it was an ordinary shutdown-order bug fix.

Act Two: SIGTERM had two owners

The twist came in review. A reviewer going through the whole shutdown path asked a simple question: who, exactly, is handling SIGTERM?

It turned out there were two owners. main.ts had these two separate lines:

// ① Hand shutdown signals to Nest itself (calling with no args registers both SIGTERM and SIGINT)
app.enableShutdownHooks();

// ② Also hand shutdown to the http-graceful-shutdown library
setupGracefulShutdown(server, {
  onShutdown: async () => {
    await app.close(); // this is where the full set of lifecycle hooks runs
  },
});

Both were catching SIGTERM. When a signal arrives, Node calls every registered listener in order, without waiting for any of them to finish. So the moment a real SIGTERM arrives, here's what happens: the listener Nest registered runs the exact same shutdown sequence as app.close() immediately, without waiting for the HTTP drain. Separately, http-graceful-shutdown drains HTTP connections its own way, then calls app.close() a second time from onShutdown. The shutdown hooks run twice.

Reasoning alone wasn't enough to be sure, so I went back and read the production shutdown logs. The evidence was right there. Twenty milliseconds after the SIGTERM timestamp - before the HTTP drain had even finished - onModuleDestroy had already started, and the shutdown hook logs for SpeechService and QueueMonitorService each appeared twice. The only reason this never turned into an incident is that each service's hooks were written to be safe to call twice (idempotent). That defensive habit was the only thing keeping "runs twice" as a log smell instead of an incident - and that smell is what eventually caught the bug.

Deleting one line

The fix was deleting one line.

// main.ts
- app.enableShutdownHooks();

Remove enableShutdownHooks() and SIGTERM has a single owner: http-graceful-shutdown. Nothing is lost, because its onShutdown already calls app.close(), which runs the full set of lifecycle hooks anyway. The only thing that changes is that the hooks now run once instead of twice - and only after the HTTP drain has finished.

The smaller the fix, the more it needs to be verified against the real thing. I sent an actual SIGTERM to a built server and watched the sequence with my own eyes.

Received SIGTERM → health check 503 → HTTP connections closed
→ shutdown hooks run only once, after HTTP close
→ Drained 56/56 BullMQ workers in 211ms → shutdown completed → clean exit

What used to run twice before the HTTP drain in production now runs once, after the drain. Since this was verified with Act One's worker-drain fix already in place, it also confirmed the two fixes work correctly together.

I also wrote down a gotcha I hit along the way. Under NODE_ENV=development, http-graceful-shutdown switches to an immediate-exit mode and never takes the graceful path at all - so this verification can't be reproduced in development mode. I only got a real reading after setting NODE_ENV=staging with dummy secrets, so the process would take the same path as production. When you're verifying a shutdown path, the first thing to check is whether your verification environment actually exercises that path.

What the shutdown code taught me

The lesson here reaches well past the narrow stage of a shutdown sequence.

A single signal should have a single owner. If you've wired in a graceful-shutdown library, don't also leave the framework's automatic shutdown-hook registration turned on. The moment both catch the same SIGTERM, the shutdown sequence quietly runs twice. When you adopt a new tool, the first thing to check is which signal or event it's going to own - and whether something already owns it.

Shutdown has an order too. Code that cleans something up should only run after everything depending on it has finished. A database should close only after the jobs using it are done, which means those jobs need to be drained in an earlier phase. A framework's phased shutdown hooks aren't decoration - they exist precisely to express this kind of ordering.

Defensive code buys you time. If the shutdown hooks hadn't been written to tolerate being called twice, this bug would have shown up as a data incident, not a log smell. Because the hooks were idempotent, they absorbed the double execution, and we got to find the bug in a log instead of an incident report. If those hooks had been written assuming they'd only ever run once, this story would have ended very differently.