Rebounder Tech Blog

Written by the people who actually run these systems in production.

gcloud sql clone point-in-time: a Real Restore Drill

Published About 2 min readBy the Rebounder engineering team — the people who operate these systems

This article may contain affiliate links. Its content is not affected by advertising.

In short

A Cloud SQL point-in-time recovery clone took about 10 minutes to create per Cloud Audit Logs, and can't be reached by psql from a laptop since it only gets a private IP.

What we did

As a drill to confirm that production Cloud SQL (PostgreSQL) can actually be restored to an arbitrary point in time, we created a real clone of it.

gcloud sql instances clone <source-instance> <clone-name> \
  --point-in-time=<UTC timestamp, 10 minutes earlier>

The headline result: Cloud Audit Logs show about 10 minutes from the start of clone creation to completion (08:27:02 → 08:36:51 UTC in this run). We deleted the clone right after verifying it, so it was actually running for about 30 minutes total, at a cost of well under a dollar.

Why this needed to be tested on production

This instance runs with availability_type = "ZONAL".

module "cloud_sql" {
  # ...
  availability_type = "ZONAL"
  tier               = "db-custom-1-3840"
  deletion_protection = true
  # ...
}

It’s not an HA configuration — ZONAL was chosen to keep cost down, with the design assumption that availability during an outage is covered by backups plus point-in-time recovery (PITR) instead. If the fallback mechanism “exists as a documented procedure but has never actually been run,” the design is only a design on paper. So we ran it for real and kept a record.

The restored clone only has a private IP

A clone is created with the same network configuration as the source instance: private-IP-only, reachable through VPC Peering / Private Services Access. In other words, you cannot connect with psql directly from your own laptop to look inside it.

Inspecting the contents requires one of:

  • Connecting from a Cloud Run Job or VM running inside the VPC
  • Exporting just the tables you need and pulling those out

We used the second option here — gcloud sql export sql --table=<target-table> to a temporary Tokyo-region bucket (with public access blocked), then visually checked the 8 rows in the target table. Once verified, we deleted the temporary bucket, removed deletion protection on the clone, and deleted it. The part that’s easy to underestimate until you actually do it: a restore drill isn’t complete just because the restore succeeded — it’s only complete once you’ve confirmed there’s an actual path to access the restored data.

PITR’s constraint

For this instance (roughly 10GB), the PITR retention window was 7 days. Once that window passes, that point in time can no longer be restored. It isn’t something you can treat as “use it if you ever need it” — if you don’t exercise the procedure at least once inside the retention window, you lose the chance to find out whether the procedure still matches reality before you actually need it.

What we verified this time was the contents of a single table; consistency across all tables is still unverified.

What’s next

Building on this drill, the next step is setting up a job that runs an audit-log consistency check against the clone itself. Only once that’s done will we have actually verified consistency across all tables — a successful restore and whether the restored data is actually in a state usable for verification are still two separate questions.

Frequently asked questions

Q1Can I inspect the restored clone's contents directly from my own machine?

No. The clone keeps the same private-IP-only network config as the original, so it isn't reachable from the public internet. Connect from a job or VM inside the VPC, or export just the tables you need.

Q2How long does creating a clone take, and what does it cost?

Cloud Audit Logs showed about 10 minutes from start to completion. It was deleted right after verification, so it ran for roughly 30 minutes total, costing well under a dollar. Leaving it running keeps billing.

Q3Why isn't this production instance running in a high-availability (HA) configuration?

It runs ZONAL to keep cost down, relying on backups plus point-in-time recovery for availability instead — a tradeoff that only holds if the restore procedure is actually verified to work.

Environment verified

  • Cloud SQL for PostgreSQL (asia-northeast1 / db-custom-1-3840)
  • Via gcloud SDK. Performed on production on 2026-10-01

What this article is based on

  • Markdown filecommit a94d6fd
  • Terraform file lines 214-229commit 2b32556

Every claim in this article comes from the records above. The repositories we operate are private so we cannot link to them, but which file, which lines, and at which commit we read them is recorded for every article. Nothing here is written from guesswork.