Enabling a Cloud Run Job Fails With SecretsAccessCheckFailed

Conclusion

When Terraform enables a Cloud Run Job for the first time and creates the Secret Manager accessor IAM bindings it needs in the same apply, the job can be tainted with SecretsAccessCheckFailed. The cause is not a missing permission but a race: Cloud Run’s creation-time validation runs before the IAM grant has propagated. Making the order explicit with depends_on — IAM first, then the job — stops it.

Symptom

A google_cloud_run_v2_job resource is enabled behind a flag, count = var.enabled ? 1 : 0. The job reads three secrets — the database URL, the Sentry DSN and a Slack webhook URL — each granted by a google_secret_manager_secret_iam_member.

We hit this in production on 2026-06-09, the first time it was applied with enabled = true. Creating the job fails and terraform apply leaves the resource tainted.

The IAM configuration itself is correct at that point. Because the cause is propagation delay rather than a missing grant, reading the IAM policy shows every binding present. It surfaces as the configuration being right while only the creation fails.

Cause

When google_cloud_run_v2_job and the google_secret_manager_secret_iam_member resources it depends on are both newly declared in the same apply with no explicit dependency between them, Terraform creates the two kinds of resource in parallel.

An IAM binding is not reflected in Cloud Run’s data plane the instant the API call returns; there is a short delay. On top of that, creating a Cloud Run Job makes Cloud Run validate access to the named secrets at creation time. If that validation lands while the binding has succeeded at the API level but not yet propagated, it is treated the same as having no permission: the job enters an error state with SecretsAccessCheckFailed and Terraform marks the resource tainted.

Terraform’s dependency graph only guarantees ordering where one resource references another (through var.xxx or a google_xxx.yyy.zzz expression). Here the job did not reference any attribute of the IAM bindings, so Terraform had no reason to order them and chose to run them in parallel.

The fix

Declare depends_on from the job to the three google_secret_manager_secret_iam_member resources it uses.

resource "google_cloud_run_v2_job" "tv_liveness" {
  count = var.enabled ? 1 : 0

  depends_on = [
    google_secret_manager_secret_iam_member.runtime_database_url,
    google_secret_manager_secret_iam_member.runtime_sentry_dsn,
    google_secret_manager_secret_iam_member.runtime_slack_webhook_url,
  ]

  project  = var.project_id
  location = var.region
  name     = var.job_name
  # ...
}

Terraform now finishes applying the IAM bindings before it moves on to creating the job. The propagation delay itself does not go away, but the creation-time validation is pushed back to a point where propagation has essentially finished, which in practice stops the race from reproducing.

A job that is already tainted recovers with terraform untaint followed by another apply. The IAM bindings exist and have propagated by then, so the second creation does not hit the same race.

Preventing a repeat

The same depends_on was applied to the staging copy of the module, not just production. The rule is fixed: whenever an apply creates both a secret accessor IAM binding and the Cloud Run resource that references it, add depends_on unless a reference expression already implies the order. A reference through var.xxx lets Terraform resolve the order itself; where the IAM resource name is not embedded in an expression, as here, the dependency is invisible to Terraform and has to be stated.

Cloud Run Jobs have another trap where the job runs but the contents are stale — see Cloud Run Job images freeze.