Rebounder Tech Blog

Written by the people who actually run these systems in production.

Three Errors You Actually Hit Connecting Actions to GCP

公開 読了時間 約4分執筆: Rebounder 開発チーム(当該システムの運用当事者)

※本記事にはアフィリエイトリンクを含む場合があります。内容は広告の有無に影響されません。

結論

Workload Identity Federation impersonation fails in one of three layers — the id-token permission, the pool's attribute_condition, or the workloadIdentityUser binding — and the error never says which.

Conclusion

Workload Identity Federation failures come from the configuration living in three layers. The error text does not tell you which one.

  • GitHub Actions side — does the job have id-token: write?
  • Pool side — does attribute_condition permit the calling repository?
  • Service account side — is the roles/iam.workloadIdentityUser binding still there?

Any of the three missing produces similar text. Without deciding the order of investigation up front you end up poking at permissions at random. Below are the three errors we actually hit, in the order you hit them.

Symptom

1. Unable to acquire impersonation credentials

The first one you meet. The authentication step itself does not pass. There are three candidate causes, and the higher the layer, the cheaper it is to check.

  • The job has no permissions with id-token: write. Entirely on GitHub’s side, so look here first
  • You are running from a different repository. The pool’s attribute_condition limits the caller via assertion.repository, so forks and other repositories are rejected
  • The roles/iam.workloadIdentityUser binding on the service account was lost to drift. Restore with terraform apply

The third is the awkward one. The code is correct and it still does not work, and it happens unrelated to any recent change. Someone touching IAM by hand puts Terraform’s state and the cloud out of sync, and it surfaces in this shape.

2. Permission 'iam.serviceAccounts.getAccessToken' denied

Here impersonation got as far as the entrance and the target service account refused. It looks like a missing grant; in practice it is usually the wrong target.

  • The service account email is wrong. Where plan and deploy accounts are separate, their permission scopes are separate too. Calling the deploy target with plan credentials is refused, correctly
  • The environment name does not match. If Terraform’s env_name and the GitHub Environment the workflow references diverge, you end up pointed at the wrong service account

The error only says “no permission”. It does not say “wrong target”, so start by confirming the email is the one you meant.

3. Provider or pool not found

The pool or provider simply does not exist. Check:

gcloud iam workload-identity-pools providers describe <provider-id> \
  --location=global \
  --workload-identity-pool=<pool-id> \
  --project=<project-id>

If it is absent, terraform apply has not been run. Apply in that environment’s directory.

Cause

All three share a root: the chain of trust is stored in three separate places.

GitHub issues an OIDC token per job. The pool receives it and decides, through attribute_condition, which repositories it accepts. What passes becomes a principal, and if that principal appears in the service account’s IAM policy, impersonation succeeds.

Those three are managed in different places: the workflow YAML, the Terraform pool definition, and the service account IAM policy. Break any one and the chain breaks, but the break is only ever observable as an authentication error.

The fix

Reading the service account IAM policy directly is the fastest way to see the real state.

gcloud iam service-accounts get-iam-policy \
  <sa-name>@<project-id>.iam.gserviceaccount.com \
  --project=<project-id>

The expected state has a principalSet of this shape in members:

principalSet://iam.googleapis.com/projects/<num>/locations/global/workloadIdentityPools/<pool-id>/attribute.repository/<owner>/<repo>

If the calling repository is not there, no amount of fixing the workflow will help. If it is there, the lower two layers are alive and you go and look at permissions on the GitHub side.

You can cut it off immediately

Because there is no JSON key in the setup, there is nothing to rotate. Instead you can stop impersonation from a repository directly.

gcloud iam service-accounts remove-iam-policy-binding \
  <sa-name>@<project-id>.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member='principalSet://iam.googleapis.com/projects/<num>/locations/global/workloadIdentityPools/<pool-id>/attribute.repository/<owner>/<repo>'

The moment the binding is removed, authentication from that repository stops. There is no key expiry to wait for.

But always update Terraform and open a PR afterwards. Leaving a hand-edited IAM policy drifts the configuration, and the binding you removed reappears the next time someone applies. Pair the stopgap with the code change.

And that drift, surfacing much later, is the first error in this article. A neglected emergency cut-off and an unexplained authentication failure are the same thing from two sides.

Making it less likely to recur

The three layers are not going away. What you can change is fixing the order in which you check them.

Cheapest first: permissions on the GitHub side, then get-iam-policy on the service account, then confirming the pool and provider exist. Following that order, the first is just opening a YAML file and the second is one command — enough to narrow down which of the three layers is broken.

For another case where the error text does not lead to the cause, see terraform’s enabled = false cannot tear down an environment, where the error appears in a module other than the one at fault.

よくある質問

Q1Where do I look for 'Unable to acquire impersonation credentials'?

Three places: whether the job has permissions with id-token: write, whether the pool's attribute_condition permits the calling repository, and whether the service account still has a workloadIdentityUser binding. Check in that order; the first is cheapest.

Q2Is 'getAccessToken denied' a missing grant?

More often it is the wrong destination. Where plan and deploy use separate service accounts, a mismatch between the environment the workflow references and the Terraform environment name points at the service account without the permission, and you get exactly this error. Check the email first.

Q3Does Workload Identity Federation need key rotation?

No. There is no JSON key in the setup, so there is nothing to rotate. What you need instead is a way to stop impersonation from a repository when something looks wrong: remove that principalSet from the service account's IAM policy and it is cut off immediately.

Q4What do I do after an emergency cut-off?

Update Terraform to match and open a PR. The cut-off touches the IAM policy directly through gcloud, so leaving it drifts the configuration and the binding you removed comes back the next time someone applies. Pair the stopgap with the code change.

この記事の根拠

  • ドキュメントファイル 91〜129行目
  • ドキュメントファイル 75〜87行目

本文の主張は、上の記録に書かれていることだけです。運用しているリポジトリは非公開のため リンクは張れませんが、どのファイルの何行目を、どのコミット時点で見て書いたかは 記事ごとに残しています。推測で書いた箇所はありません。