"Just Turn It Off in Staging" - What the Noisy Alert Was Actually Hiding

The same alert showed up at the same time every day: an ERROR saying the subscription reconciliation job had failed in staging. The cron ran at 3:17 AM, and three minutes later the alert landed. The stack trace always opened with the same line: Google Play Voided Purchases API returned 404. The job retried three times, died on 404 all three times, dropped into the DLQ, and left one more red alert piled up in our Slack.

This wasn't an incident. Nothing was broken. It was just noisy. And in the process of deciding what to do about that noise, I relearned something: turning a thing off and actually fixing it are not the same problem.

Why the 404 was happening

The Voided Purchases API is a Google Play endpoint that returns purchases which were refunded or canceled. We reconcile against it daily to catch people who get a refund quietly while continuing to use the app. But a 404 from this endpoint doesn't mean "no data." For voidedpurchases.list, a 404 means the application (package) can't be found, or the service account has no access to that package at all.

Our staging service account has no Play Console access to the production package - and there's no reason it should. Staging isn't an environment that actual store refund traffic flows through. So calling this API from staging produces a 404. Today, tomorrow, forever, unless the configuration changes. This isn't a blip that clears itself up; it's a permanent condition created by the environment.

That's where the real distinction lives. For a transient failure, retrying is the right call - if the network hiccups or the other side briefly returns 5xx, calling again a few seconds later can work. But a permanent condition doesn't improve with retries. Permissions the staging service account doesn't have aren't going to appear three seconds from now. So retrying this 404 three times was pure waste, and the ERROR alert at the end of it was pure noise.

Why the code was retrying in the first place

The root cause was in how the client classified responses. listVoidedPurchases was throwing every non-200 response through a single path.

if (!res.ok) {
  throw AppHttpException.serviceUnavailable(
    `Google Play Voided Purchases API returned ${res.status}`,
  );
}

serviceUnavailable is the error our queue reads as "retry this." So whether the status was 404, 500, or 429, everything that came out of here got retried three times and ended the same way: an alert. The problem is that these three statuses mean completely different things. 500 and 429 mean "not right now, but maybe soon." 404 means "not under this condition, ever." The moment they're merged into one exception, an error that retrying can never help climbs onto the retry pipeline anyway.

What's interesting is that the right answer already existed in the same file. getSubscriptionPurchase, on the same client, had long since split out 404 on its own.

if (res.status === 404) {
  throw new GooglePlayApiError(404, "purchase not found");
}

One method respected what 404 meant. The other flattened it away. The precedent was sitting right there in the codebase - the new method just didn't follow it. This kind of asymmetry usually shows up when methods get written on different days, with different things on the author's mind.

The fix: split it at the boundary

The fix landed on two layers, and both followed a pattern that already existed.

First, we split 404 out at the client boundary. A 404 now throws GooglePlayApiError(404), carrying the meaning "not found," while everything else - 5xx, 429, timeouts - still throws serviceUnavailable as before. Only the errors that genuinely warrant a retry stay on the retry path.

if (res.status === 404) {
  throw new GooglePlayApiError(404, "application not found");
}
if (!res.ok) {
  throw AppHttpException.serviceUnavailable(/* 5xx, 429, timeouts - genuinely retryable */);
}

Next, we decided how the reconciliation service should treat this 404. The service already had a flow for "no service account key, skip quietly" (isConfigured === false). A 404 is, at its core, the same situation: this is an API you can't call in this environment. So it got routed to the same place.

try {
  const voided = await client.listVoidedPurchases();
  // ... proceed with reconciliation
} catch (e) {
  if (e instanceof GooglePlayApiError && e.status === 404) {
    this.logger.warn("Voided purchases 404 - not reachable in this environment, skipping");
    return { skippedAppNotFound: true };
  }
  throw e; // anything other than 404 is re-thrown to retry normally
}

The catch block swallows only 404, logs a warning, and returns a skip count. Since it doesn't re-throw, there's no retry, no DLQ, no ERROR alert. Any error that isn't 404, on the other hand, gets re-thrown and the original retry logic still runs as before. Only the noisy part goes quiet; the real problems stay loud.

We backed this with tests: does the client throw GooglePlayApiError on 404, does reconciliation skip on 404, and does anything other than 404 still get re-thrown. That last case matters most. If you quiet a 404 and accidentally swallow a 500 along with it, you lose visibility into an actual outage.

"Can't we just turn this off in staging?"

After the PR went up, someone asked a good question: why run this job in staging at all? The instinct is sound - staging has no real refund traffic, so there's no reason to be calling this API in the first place. So we looked into it.

Turning it off was less trivial than it sounded. The natural switch for disabling this feature in staging would be whether a service account key is configured (isConfigured), but that key isn't dedicated to this one job. It also gates receipt validation for in-app purchases, handling Google's real-time developer notification (RTDN) webhook, and the subscription reconciliation processor - subscription verification as a whole hangs off the same key. To test payments and subscriptions in staging at all, that key has to be present. Pull it to kill this one job, and you kill the very things you'd want to test.

Fine - what about a dedicated toggle just for this job? No such toggle existed. Building one would mean introducing a new switch like GOOGLE_VOIDED_RECONCILE_ENABLED. In other words, even "just turn it off in staging" turns out to require touching code and environment variables. There was no free way to disable it.

Turning it off and fixing it are different problems

But the real point was never how hard the toggle would be to build. Even if adding it had been trivial, it still wouldn't have been a substitute for fixing the 404 - because the two approaches solve different problems.

• Taking 404 out of the retry path is a structural safeguard. In any environment - staging, a misconfigured production, some future case where somebody forgets to flip a toggle - it stops a 404 that retrying can never fix from turning into an alert storm.
• Disabling the job in staging is an operational optimization. "There's no reason to call this in this environment, so skip the call entirely" is a good thing to layer on top.

The order matters. Adding a toggle on top of the 404 fix is fine. But adding only the toggle while skipping the 404 fix means that someday, when production config drifts and a 404 shows up there, the alert comes back - and this time you can't wave it off as "just staging." Turning off the source erases the one alert you can see right now. Fixing the failure mode erases every future case where that same alert would fire again.

Faced with a noisy alert, the hand reaches for the source first: kill the job, delete the alert rule, exclude the environment. It's a natural impulse, and sometimes it's the right one. But before you flip that switch, it's worth asking one question: am I silencing the alert, or fixing the failure it was telling me about? The former makes this one alert go quiet. The latter keeps this same failure from coming back wearing a different face.

What to ask at the boundary

None of this is specific to the Google Play API. Any code that calls out to an external service and retries its failures through a queue faces the same fork in the road. Before touching retry policy in the queue configuration, ask this at the client boundary first.

Does this error get better if I retry it? 5xx, 429, and timeouts usually do - what fails now might succeed shortly after. 4xx usually doesn't. 404, 403, and 400 will keep giving you the same answer until the underlying condition changes. Merge the two into a single exception, and a failure that retrying can never fix ends up burning retry budget and filling the alert channel anyway. The place to draw that line isn't the queue - it's the client boundary, where the response code is first seen.

And before writing a fix, it's worth taking one pass through the codebase. In our case, the neighboring method in the same file was already handling 404 correctly. There was nothing to invent. More often than you'd expect, the answer is already sitting in the code - some other piece just never followed it.