NoSuchBucket: Destroying Staging Broke Prod Ad Images

The short version

Give a code default a GCS bucket URL, and never override that default with an env var in production, and that default stops being an unused string in the code — it becomes real data baked into the production DB. When that default happened to point at a staging bucket, the moment staging was torn down with terraform destroy, production’s image delivery broke with GCS’s NoSuchBucket (The specified bucket does not exist.). The fix was to repoint the code’s default at the prod bucket and re-upload the images there — but preventing a repeat meant sweeping every build-config default that carried the same pattern, not just this one file.

What it looks like

Several production ad images 404’d at the same time. Signage had been shut down for summer break since 2026-07-28, so nothing rendered on-screen. Tracing the image URLs (media_url), all of them pointed at a GCS bucket that belonged to the staging environment. That bucket had been deleted along with the rest of the staging resources when the staging environment was torn down via terraform destroy on 2026-08-04.

Why

The seed script that writes ad-image URLs into the DB defined the base URL like this:

/** Public staging GCS bucket (Terraform: modules/ad_media, envs/staging). No trailing slash. */
export const AD_MEDIA_BASE =
  process.env.SEED_AD_MEDIA_BASE ??
  "https://storage.googleapis.com/<staging-bucket>/ads";

It was designed to be overridable via an env var, but the fallback baked into the code pointed at the staging bucket. During development the seed usually ran against staging, so this default never looked out of place.

The actual problem: the path that ran the seed against prod never passed that env var. With nothing passed, the right-hand side of ?? won every time, and each prod seed run wrote the code’s staging bucket URL straight into the production DB’s media_url column. Once a URL is written to the DB, nothing on the GCS side changes it automatically afterward.

Digging further, this wasn’t isolated to the seed script. The Cloud Build config for the web service also baked in staging-project values as the defaults for the auth domain, project ID, and image repo substitution variables — meaning a build run for prod without explicitly passing --substitutions would produce an image pointing at a Firebase project or registry that didn’t exist. The design of defaulting to staging was embedded in the same shape in more than one place.

Looking only at Terraform’s resource dependency graph for the staging environment can’t catch references left outside IaC’s view — a DB column, a build-config default. terraform destroying staging itself was the right call (it was costing over ¥10,000/month for little verification value), but whether anything torn down was still referenced elsewhere needed a separate sweep.

Fixing it

We repointed the defaults across the seed files that shared this pattern to the prod bucket.

/** Public prod GCS bucket (Terraform: modules/ad_media, envs/prod). No trailing slash.
 *  ⚠ 2026-08-04: changed from staging → prod after tearing down staging. Leaving it
 *  pointed at staging and destroying it 404'd five production ad images. Never default
 *  to anything but prod from here on. */
export const AD_MEDIA_BASE =
  process.env.SEED_AD_MEDIA_BASE ??
  "https://storage.googleapis.com/<prod-bucket>/ads";

The images that had been live were re-uploaded from source into the prod bucket, and media_url in the DB was refreshed by re-running the seed. We also changed the defaults in the web, jobs, and migrate build configs and Dockerfiles from the staging project to the prod project (including the project number), aligning where the auth domain, project ID, and image repo substitutions pointed.

Preventing a repeat

The fix commit left a comment reading exactly “never default to anything but prod from here on.” But that note only warns the next person writing the same kind of default — the underlying structure, “a default gets baked into real data,” hasn’t changed. A design that overrides via env var silently falls back to the code’s default the instant you forget to pass it. Part of why the missed override went unnoticed is that the seed run never logged which value it actually used. If you’re going to keep the default-value design, either log the value that got picked, or make production refuse to start at all when the env var is unset — either would stop the same shape of incident.