The job succeeded but nothing changed. What do I check first?

The image digest the job actually runs. Read it with gcloud run jobs describe, then check whether your fix is an ancestor of that image with git merge-base --is-ancestor. If it is not, re-running will never help.

Is there a faster fix when production is down?

If the code reads the value from an environment variable, override it at execution time with --update-env-vars. No build, no apply. But it only affects that single execution, so the old default returns the next time you forget the flag.

Why does this rarely happen with Cloud Run Services?

Deploying a Service creates a new revision, so updating the image is already part of the normal workflow. Running a Job skips that step entirely, which is why a stale image goes unnoticed.

Cloud Run Jobs Freeze Their Image

※本記事にはアフィリエイトリンクを含む場合があります。内容は広告の有無に影響されません。

Check this first

Read the image the job actually runs.

gcloud run jobs describe <job> --region=asia-northeast1 \
  --format="value(spec.template.spec.template.spec.containers[0].image)"

Then check whether your fix is contained in it.

git merge-base --is-ancestor <fix-commit> <image-sha> && echo "contained" || echo "not contained"

If it is not contained, re-running changes nothing. Worse, the job may overwrite data again using the old behaviour.

What is actually happening

The image is pinned in the job definition. When the definition lives in infrastructure-as-code, the tag is declared as a variable, and the job keeps running the old code until that tag is bumped and applied.

local fix → commit → 【wall】 → job execution
                       ↑ image is still the old one

The awkward part is that the run succeeds. Nothing appears in the logs. All you are left with is “I ran it and it did not take effect”.

Two ways out

A. Override the environment variable at execution time

If the code reads process.env.X ?? default, you can swap the value without touching the image.

gcloud run jobs execute <job> --region=asia-northeast1 \
  --update-env-vars=KEY=value --wait

No build, no apply, so this is the fast path. But it applies to that execution only — the job definition is unchanged, and the old default comes back the next time the flag is omitted.

B. Update the image

Build, bump the tag, apply. The exact commands depend on your setup, but you cannot skip the tag bump. That is the part everyone skips.

Preventing a repeat

After this incident we moved the digest check to the very top of the runbook. The root cause was that running the job without checking was possible at all, so changing the order of the steps is what actually helps.

We also wrote down, in the runbook, that the environment-variable override is a stopgap. Because it is fast, it is easy to walk away believing the problem is fixed.