migrate_image_tag drifts from live after concurrent deploys

Starting with the conclusion

Terraform can only record a production action in one order: write the tag into locals, commit, then apply. When two sessions deploy to Cloud Run at the same time, the commit recording one session’s already-executed action (a migration Job run, a staging deploy) can land on main after the other session’s unrelated commit. Live itself was correct in both cases — only the ledger (Terraform’s locals) was left pointing at a stale tag for a while, an unrecorded drift.

Symptom

On 2026-06-16, after shipping a header-layout fix (#968), one session ran a staging deploy followed by a prod migration. Neither operation failed — both staging and prod responded with HTTP 200.

Meanwhile, a separate concurrent session had already committed two prod web bumps, #969 and #972. By the time this session went to commit a record of what it had just done, main already contained those other commits.

# Still on main when this session tried to record its own work
# (prod/main.tf:169, staging/main.tf:79)
migrate_image_tag = "6618708" # bumped for migration 20260615120000 (#954)

# (staging/main.tf:228)
web_image_tag = "6618708" # editor overhaul + academic_year removal. 200 OK

What this session had actually run in production was migration 20260615164918 (changing audit_log.occurred_at’s DEFAULT to clock_timestamp(), fixing a false-positive in the audit hash chain from #965), plus a staging deploy of both the web and migrate images. Both had already executed — but neither was reflected in main.tf’s locals yet.

Cause

Nothing was wrong with the actions themselves. The real problem was that “running something in production” and “committing that fact to Terraform’s locals” are not the same transaction.

• This session: ran the staging deploy → ran the prod migration Job → moved on before committing a record of either
• The concurrent session: #969 (bump prod web to 0a01b5c) → #972 (bump prod web to 8aefde1, which already included #968)

infrastructure/** was already documented as a serialized lane requiring a token (docs/parallel-lanes.md, line 54):

| **infra lane** | `infrastructure/**` (Terraform) | serial (tf state) | infra-token |

But that discipline exists to stop two sessions writing the same line at the same time. This incident touched two lines in staging and one in prod that never overlapped with the line the concurrent session touched. Nothing collided at the line level — the drift came from the order in which the recording commits landed, not from what they edited.

The practical effect: running terraform plan at that point would have shown a diff between the committed 6618708 and what was actually live — the committed value had fallen behind live, and nothing flagged it automatically.

Fix

Every action that had actually executed got reflected in locals, and terraform plan returning No changes on both environments confirmed committed == live again.

# infrastructure/terraform/envs/prod/main.tf:169
migrate_image_tag = "227a512" # migration 20260615164918: audit_log.occurred_at DEFAULT ->
                               # clock_timestamp() (fixes #965, DEFAULT-only, non-breaking).
                               # Verified on staging first, then run in prod.

# infrastructure/terraform/envs/staging/main.tf:79, 228
migrate_image_tag = "227a512" # same migration, run on staging
web_image_tag     = "227a512" # #968 header fix + #967 WYSIWYG direct editing + #965 fix.
                               # Note: prod had already moved ahead to 8aefde1 (via #970) concurrently

Prod’s web_image_tag wasn’t touched in this fix — #972 had already committed it as 8aefde1, and 8aefde1 included #968 as an ancestor, so committed and live already agreed on that field. The fix was scoped to exactly the actions this session had run but nobody had committed yet.

Why this wasn’t caught sooner

What makes this kind of drift dangerous is that apply never fails and never errors. terraform apply faithfully pushes whatever is currently committed to live — it has no way to check whether the committed value correctly represents the work that was actually done. The only way to notice is to actively run terraform plan and compare it against what you know you actually ran.

The infra lane discipline in docs/parallel-lanes.md is designed to stop two sessions writing the same line at the same time. It doesn’t by itself prevent this pattern — where the lines never overlap, but the timing of the recording commit does. In the end, the only real defense is operational: commit the locals update for whatever you just ran before moving on to anything else.