Cloud Run Jobs Freeze Their Image
This article may contain affiliate links. Its content is not affected by advertising.
In short
A Cloud Run Job runs the image pinned in its job definition, so committing a fix never changes what executes until the image tag is updated and applied.
Check this first
Read the image the job actually runs.
gcloud run jobs describe <job> --region=asia-northeast1 \
--format="value(spec.template.spec.template.spec.containers[0].image)"Then check whether your fix is contained in it.
git merge-base --is-ancestor <fix-commit> <image-sha> && echo "contained" || echo "not contained"If it is not contained, re-running changes nothing. Worse, the job may overwrite data again using the old behaviour.
What is actually happening
The image is pinned in the job definition. When the definition lives in infrastructure-as-code, the tag is declared as a variable, and the job keeps running the old code until that tag is bumped and applied.
local fix → commit → [ wall ] → job execution
↑ image is still the old oneThe awkward part is that the run succeeds. Nothing appears in the logs. All you are left with is “I ran it and it did not take effect”.
Two ways out
A. Override the environment variable at execution time
If the code reads process.env.X ?? default, you can swap the value without touching the image.
gcloud run jobs execute <job> --region=asia-northeast1 \
--update-env-vars=KEY=value --waitNo build, no apply, so this is the fast path. But it applies to that execution only — the job definition is unchanged, and the old default comes back the next time the flag is omitted.
B. Update the image (the real fix)
Build, bump the tag, apply. The exact commands depend on your setup, but you cannot skip the tag bump.
Preventing a repeat
After this incident we put the sha check at the very top of the runbook. The root cause was that running the job without checking was possible at all, so changing the order of the steps is what actually helps.
We also wrote down, in the runbook, that the environment-variable override is a stopgap. Because it is fast, it is easy to walk away believing the problem is fixed.
Frequently asked questions
Q1The job succeeded but nothing changed. What do I check first?
The commit sha the job's image was built from. Read the image with gcloud run jobs describe, then check whether your fix is an ancestor of that sha with git merge-base --is-ancestor. If it is not, re-running will never help.
Q2Is there a workaround when I am in a hurry?
If the code reads the value from an environment variable, override it at execution time with --update-env-vars. No build, no apply. But it only affects that single execution, so the old default returns the next time you forget the flag.
Q3Why does this rarely happen with Cloud Run Services?
Deploying a Service creates a new revision, so updating the image is already part of the normal workflow. Running a Job skips that step entirely, which is why a stale image goes unnoticed.
What this article is based on
- Markdown file lines 1-70
Every claim in this article comes from the records above. The repositories we operate are private so we cannot link to them, but which file, which lines, and at which commit we read them is recorded for every article. Nothing here is written from guesswork.