A staging Default in env Deleted Production Images

The short version

When you give an environment variable a fallback, and that fallback points at a resource that really exists in another environment (staging), forgetting the explicit override still works. And because it works, there is no way to notice the missed override. Here, a production seed script wrote that default straight into the production database, and the day staging was destroyed, the images went with the bucket production had been pointing at.

What it looks like

A signage application serves image assets (ad banners) from a public Cloud Storage bucket. The bucket’s public URL is stored in the DB’s media_url column, and devices GET that URL directly.

One day, after tearing down the staging environment for cost reasons (terraform destroy), five images displayed in production suddenly 404’d. Why deleting staging would delete production’s images was not obvious at first.

Why

The seed script that writes media_url read the bucket’s base URL from an environment variable while holding a default in case it was unset.

export const AD_MEDIA_BASE =
  process.env.SEED_AD_MEDIA_BASE ??
  "https://storage.googleapis.com/myapp-staging-media/assets";

The problem is that this default pointed at the staging bucket. The production deployment config never explicitly overrode SEED_AD_MEDIA_BASE, so the ?? default was used as-is.

Writing process.env.X ?? default is a common pattern for not halting when a variable is unset. But when that default points at “a resource in another environment that really exists”, a missed override does not error: production quietly starts running with a dependency on another environment’s resource. While the staging bucket’s objects exist the images display fine, so that dependency never surfaces unless you read the code.

It surfaced the moment the staging environment itself was terraform destroyed. The bucket disappeared, what the media_url baked into the production DB pointed at was gone, and five images 404’d at once.

Fixing it

We changed the default from the staging bucket to the production bucket, cutting the implicit cross-environment dependency.

export const AD_MEDIA_BASE =
  process.env.SEED_AD_MEDIA_BASE ??
  "https://storage.googleapis.com/myapp-prod-media/assets";

The media_url values already written to the DB are not fixed by re-running the seed. The objects still do not exist in the referenced bucket, so the lost images had to be re-uploaded to the production bucket from the originals.

Preventing a repeat

The commit fixing the default left a note in the code itself: never default to anything but prod from here on. We have not built a mechanism demanding the same check of every piece of code holding an env-var default, but at least in this file, the account of the incident sits next to the code so the next person does not set staging back as the default for the same reason.

We will keep designing env vars with defaults. What changed is how the default is chosen. The criterion is now a value that breaks immediately if the override is forgotten, not one that works anyway. Default to a real resource in another environment and a missed override hides inside normal operation, delaying discovery. A non-existent bucket name or an empty string errors on the spot and is never carried over until the day staging is deleted.

In the sense of getting caught in cross-environment configuration, this is close in root to the teardown trap in terraform enabled = false Cannot Tear Down an Env — “what an operation deleting one environment leaves behind in the other” is worth checking every time.