Rebounder Tech Blog

Written by the people who actually run these systems in production.

01.dnsv.jp Full-Zone Replace Deletes Unrelated A Records

Published About 3 min readBy the Rebounder engineering team — the people who operate these systems

This article may contain affiliate links. Its content is not affected by advertising.

In short

Onamae.com DNS submits the whole zone, not just the edited record, so adding one subdomain can delete an existing A record that was missing from the stale copy on screen.

The short version

Onamae.com’s DNS record screen doesn’t just append the one record you edit – it submits the entire zone snapshot the screen currently holds. With several of our own small web services sharing the same registrar and the same zone, one person’s routine edit silently deleted another service’s subdomain A record. HTTP monitoring kept returning 200 for a few minutes because it was shielded by resolver caching, so nobody noticed at the time.

The symptom

On 2026-08-14, we were adding a new subdomain for one service to the rebounder.jp zone, which hosts multiple company services. The addition itself succeeded, and the new subdomain resolved correctly right away.

Shortly after, though, we noticed that the A record for a subdomain belonging to a completely different, unrelated service had disappeared. That subdomain had been in use for a while, and there’s no record of anyone intentionally touching it that day.

What made this tricky is that even after the A record was actually gone, HTTP uptime monitoring kept returning 200 for a while. The DNS resolvers used by browsers and CI runners keep serving the old A record from cache until its TTL expires, so unless you query the authoritative DNS server directly, you can’t detect that the record is already gone.

Why

Onamae.com’s DNS record screen looks like a screen for adding or deleting one record at a time, but what actually gets submitted is the entire zone the screen currently holds. In other words: whatever snapshot of the zone was loaded when the screen was opened, adding one record and submitting it replaces the whole existing zone with that snapshot.

This turns into an incident when two conditions line up:

  1. The same zone is edited by multiple people, or multiple services’ operators, at different times.
  2. One of them submits their own addition while still holding a stale snapshot from before the other’s change was reflected.

In that case, the later submission’s snapshot doesn’t include the record that was added earlier (or another service’s pre-existing record). Once submitted, that snapshot is treated as the new whole picture of the zone, so anything missing from it is deleted as a result. This isn’t malice or a fat-fingered edit – it’s behavior baked into the spec, and it can happen any time there’s a gap between when the screen was opened and when it was submitted.

What makes this hard to spot is the combination of HTTP monitoring and DNS caching. Even after an A record is gone from the authoritative DNS server, as long as resolvers elsewhere are still caching the old value within its TTL, HTTP uptime checks keep returning 200. “Monitoring stays green while only the authoritative DNS has changed” persists until the TTL expires.

Fixing it

We added a direct query to the authoritative DNS server ahead of HTTP monitoring, checking that it matches the expected A record.

A=$(dig +short <subdomain> @01.dnsv.jp | head -1)
[ "$A" = "<expected IP>" ] || echo "🔴 A record vanished or changed"

01.dnsv.jp is one of the authoritative DNS servers used by Onamae.com’s standard DNS. The key is pointing dig explicitly at this authoritative server, not a resolver. Querying the local machine’s or CI runner’s default resolver returns the stale cached value even right after the record disappears, delaying detection. Querying the authoritative server directly confirms the zone’s actual current state with no delay from resolver caching.

Lesson

The commit log for this check states the reason for adding it plainly: a record-loss incident from concurrent zone editing actually happened. What we chose as the durable fix wasn’t changing the registrar’s screen behavior – it was adding one line to our own monitoring: a direct check against the authoritative DNS. We didn’t change the underlying setup where multiple people and multiple services edit the same zone. We haven’t gone as far as splitting the zone or consolidating editing into one place, so the next time someone submits from a stale snapshot on the same screen, records will vanish the same way again until this monitoring catches it.

Frequently asked questions

Q1Why does an A record for an unrelated subdomain disappear too?

Onamae.com's DNS screen does not append just the record you edit -- it submits the entire zone the screen currently holds. If someone else has the same screen open with a stale copy and adds a record there, submitting it replaces the zone with that stale copy, deleting anything missing from it.

Q2We had HTTP monitoring running -- why did it take minutes to notice?

HTTP monitoring resolves through a DNS resolver used by the browser or CI runner, and that resolver keeps serving the old A record from cache until its TTL expires. Unless you query the authoritative DNS server directly, you cannot tell the record is already gone.

Q3What should we monitor to prevent the same incident?

Query the authoritative DNS server directly with dig +short and compare the returned IP against the expected value. Adding this one check ahead of HTTP monitoring catches the change while it is still hidden behind resolver caching.

Environment verified

  • Onamae.com's DNS record screen (behavior as of 2026)
  • GitHub Actions ubuntu-latest / daily scheduled run at 6:00 JST
  • Occurred 2026-08-14 on a subdomain in production use

What this article is based on

  • YAML file lines 1-20commit 698831e
  • YAML file lines 1-30commit 85c19ce

Every claim in this article comes from the records above. The repositories we operate are private so we cannot link to them, but which file, which lines, and at which commit we read them is recorded for every article. Nothing here is written from guesswork.