A Stale Branch Deploy Silently Reverted Features

The short version

A Cloud Run deploy is nothing more than rewriting the image tag written in terraform’s locals to a new value and applying; it never looks at branch history. Point the tag at an image built from a branch whose merge base is old, and features merged to main since then disappear from production without a single git revert commit.

What it looks like

Two bug reports of quite different character came in from production at the same time.

• One of the live screen templates, displaying fine moments earlier, stopped displaying
• Changing a template’s settings in the editor has no effect even after saving

They looked like bugs in two unrelated features. One on the delivery side, one on the editing side, in different parts of the code. But they coincided with a bump of the production image tag just beforehand.

# before the bump
web_image_tag = "e95d5ff" # public signage /signage shown on tablet/PC (≥900px) as a fixed-scale reduction
                          # of the real monitor (16:9, 1920×1080)…

# after the bump (this bump produced the symptoms)
web_image_tag = "9f73a56" # do not apply signage fit-stage on real devices (Android WebView/TV)…

The bump itself looked like exactly the intended change: “do not apply the scaled display on real devices”. The plan shows no destroy. The apply succeeds. The tag string changing was itself correct, and two features disappeared anyway.

Why

The image web_image_tag pointed to was built from a working branch, feat/signage-fit-to-monitor. That branch had diverged from main well before the commit production had been pointing at.

The problem was that after that divergence, two other features had been merged to main.

• A change adding one of the live screen templates
• Three changes fixing template application in the editor

The working branch does not contain those commits. Swapping the tag to an image built from it means bringing a state that is in the past relative to main’s tip straight into production.

image = "${module.artifact_registry.image_repo_url}/web:${local.web_image_tag}"

All the Cloud Run web service looks at on startup is the string this one line assembles. Which commits are inside the image local.web_image_tag points at is none of terraform’s business. Because the tag is being swapped as a string, “a newer tag” and “newer features” are not the same thing.

From git’s side, this operation does not look like a revert either. A revert leaves a new commit in the history; a tag swap never touches commit history at all. Looking at main’s history, the commits for the vanished features are still right there. What disappeared from production did so because the tag pointed at an image that did not include them — the code was not deleted or undone. That shape, “nothing happened in the history and only what is running changed”, is what made the two bugs look unrelated.

Fixing it

Correct recovery meant treating the tag production had deliberately pointed at before the bump as the base, and cherry-picking onto it only the two commits genuinely meant to be added.

web_image_tag = "f3fe951" # incident recovery: bumping prod from a stale branch
                           # (feat/signage-fit-to-monitor) had silently rolled back pattern4 and
                           # the editor's pattern resolution. f3fe951 = a12b7ef (contains the
                           # previous web:a12b7ef) with signage-fit and the real-device fit-stage
                           # exclusion cherry-picked. No schema change, no secret added or removed

Rebuilding from the latest main is an option, but that carries every other change merged into main into production at once, leaving room for unintended changes. Pinning the base to the previous intended tag, which contains the vanished features, and adding only the delta narrowed the change to the two commits for “do not apply the scaled display on real devices”.

Why it wasn’t noticed

CI verifies the contents of the branch it is told to build; how far behind main that branch is is not among the things it verifies. terraform’s plan shows the image tag string changing, but not which point of main the new tag’s image includes up to. A branch being stale was not something review was conscious of as a review item.

The result is that this kind of rollback is invisible to everyone until symptoms appear. The tag swap passes as a normal apply, and the vanished features’ commits remain in main’s history, so no trace of “what happened” is left on either the infrastructure side or the code history. The only clue was a report from a user who actually touched it: “it’s gone”.

Around Cloud Run image tags, Cloud Run Jobs Freeze Their Image deals with another form of “looking at a tag without really looking at it”. That one is about the same tag being re-pushed and never taking effect; this one is about changing the tag itself and going back to an unintended past.