GCS ifGenerationMatch autoRetry Misreads 412 as Already Sent

Conclusion

A write that uses a GCS object’s ifGenerationMatch: 0 to decide atomically “nothing sent today yet” can still be misread if the client’s autoRetry is on: when the first write succeeds server-side but only its response is lost, the retry’s 412 Precondition Failed gets treated as “already sent.” The fix was new Storage({ retryOptions: { autoRetry: false } }) — turn off autoRetry entirely, let the failure throw, and claim again on the next run.

Symptom

A Job sends a “✅ TV monitoring running” digest to Slack once per weekday morning. It runs every minute, and only the first successful run inside the window (08:30–09:29 JST on weekdays) is supposed to send. “Already sent today” was recorded as an empty GCS object at tv-liveness/digest/<YYYY-MM-DD>.

export async function decideDailyDigest(
  now: Date,
  marker: DigestMarkerPort | null,
): Promise<DigestDecision> {
  if (!isDailyDigestWindow(now)) return { send: false };
  const day = jstDay(now);
  if (marker === null) {
    return isDailyDigestMinute(now) ? { send: true, day, claimed: false } : { send: false };
  }
  try {
    return (await marker.claim(day)) ? { send: true, day, claimed: true } : { send: false };
  } catch (err) {
    console.warn(
      JSON.stringify({
        event: "tv.health_check.digest_marker_error",
        message: err instanceof Error ? err.message : "unknown",
      }),
    );
    return isDailyDigestMinute(now) ? { send: true, day, claimed: false } : { send: false };
  }
}

The record is created atomically with ifGenerationMatch: 0 (create only if the object doesn’t exist yet), so overlapping per-minute runs or the occasional retry should still only send once. But on days when the network was briefly unstable, the digest sometimes never arrived at all. GCS had the object recording “already sent,” while Slack had never actually received it.

Cause

claim() was written to swallow any 412 Precondition Failed as “already recorded today = already sent.”

export function createGcsDigestMarker(bucketName: string): DigestMarkerPort {
  const bucket = new Storage().bucket(bucketName);
  const objectFor = (day: string) => bucket.file(`tv-liveness/digest/${day}`);
  return {
    async claim(day) {
      try {
        await objectFor(day).save("", {
          resumable: false,
          contentType: "text/plain",
          preconditionOpts: { ifGenerationMatch: 0 },
        });
        return true;
      } catch (err) {
        // 412 Precondition Failed = already recorded today (already sent or in flight).
        if ((err as { code?: number }).code === 412) return false;
        throw err;
      }
    },
    async release(day) {
      await objectFor(day).delete({ ignoreNotFound: true });
    },
  };
}

new Storage() has client-side autoRetry on by default. If the first save() succeeds server-side but only its success response is lost to a network issue, the client sees no response and automatically resends the same write. That second write can no longer satisfy ifGenerationMatch: 0, because the object from the first attempt already exists — so it gets a 412.

claim() unconditionally reads that 412 as “already recorded today” and returns false. decideDailyDigest treats that false as “no need to send this run (some other run already did),” so send: false wins and nothing goes to Slack. But the object had already existed since the moment the first write succeeded server-side, so nobody can claim for that day ever again — the digest for that day is gone for good. A single transient lost response erased an entire day’s digest.

The fix

Client-side autoRetry on the Storage client was turned off, so a lost response now throws instead of being silently retried.

/** GCS implementation. Auth via Workload Identity (ADC); no key file is used (rule 5). */
export function createGcsDigestMarker(bucketName: string): DigestMarkerPort {
  // autoRetry is off: if the first attempt succeeds server-side and only its response is lost,
  // the client's own retry gets a 412 and misreads it as "already sent," losing that day's digest.
  // Letting the failure throw and claiming again on the next minute is safer.
  const bucket = new Storage({ retryOptions: { autoRetry: false } }).bucket(bucketName);
  const objectFor = (day: string) => bucket.file(`tv-liveness/digest/${day}`);
  return {
    async claim(day) {
      try {
        await objectFor(day).save("", {
          resumable: false,
          contentType: "text/plain",
          preconditionOpts: { ifGenerationMatch: 0 },
        });
        return true;
      } catch (err) {
        // 412 Precondition Failed = already recorded today (already sent or in flight).
        if ((err as { code?: number }).code === 412) return false;
        throw err;
      }
    },
    async release(day) {
      await objectFor(day).delete({ ignoreNotFound: true });
    },
  };
}

claim()’s own 412 handling was not changed. What was turned off is only the client library’s internal behavior of automatically resending the same write when a response is lost. Now a lost response propagates as a thrown exception all the way to decideDailyDigest’s catch, which already falls back to “this run failed to record, so send based on a single-run check if we’re inside the window.” Because the Job itself runs again every minute, it simply claims again next minute — retry responsibility sits with the outer scheduler, not inside the client library.

Preventing a repeat

The sources for this incident don’t record any additional mechanical check being added — what can be written here is only the structural reason this misread was hard to notice.

With autoRetry on, the path from a lost response to a 412 was fully absorbed inside claim()’s own catch, which just returns false and moves on. It never reached decideDailyDigest’s own catch (the logging path that does console.warn with tv.health_check.digest_marker_error), so the only thing visible in logs from the outside was an ordinary-looking send: false decision. A transient lost response and a genuinely normal case — some other run having already sent — were indistinguishable in the logs.

Turning off autoRetry means a lost response is no longer swallowed inside the client and instead throws, landing on the existing digest_marker_error log path. A transient lost response and a normal “already sent” can now be told apart inside logging that already existed, without adding new monitoring.