findUnmaskedPii False-Positives After Unmask

Running a PII check after unmasking flags dictionary-restored values as raw PII leaks and discards an entirely correct draft whole.

The short version

Run a PII check on output that has already been unmasked, and it flags even a teacher’s own phone number or email — the exact value restored from maskPII’s dictionary — as a “raw PII leak,” fail-closing and discarding an otherwise correct draft in its entirety. Moving the check to before unmasking (while the text is still in mask space) lets it catch only “raw PII the model generated that isn’t in the dictionary” — the real leaks — without losing any detection power.

What it looks like

The AI drafting feature is designed to tokenize a teacher’s typed PII (phone numbers, email addresses) with maskPII, pass that to the model, generate, then restore the original text with spec.unmask. Before the fix, assistant-actions.ts ran the following fail-closed check after unmasking:

// Re-check for residual PII in the unmasked output too, fail-closed (rule 4, all string fields).
for (const it of items) {
  for (const field of spec.strings(it)) {
    if (findUnmaskedPii(field, []).length > 0) {
      return { ok: false, reason: "pii_leak" };
    }
  }
}

items has already been unmasked via spec.unmask(it, dictionary) — phone numbers and emails are back in their original, raw form. Running findUnmaskedPii on this flags even the correct contact info the teacher originally typed as “PII still present,” and the whole draft was discarded as a pii_leak.

The same shape of check existed in the streaming delivery of notice drafts (notice-draft-sse.ts) too. It checked each unmasked element and silently dropped any element containing a dictionary-restored value as notice_redacted.

const unmasked = unmaskPII(el.text, dictionary);
// Re-check each unmasked element for residual PII too, fail-closed (drop only the leaked item, rule 4).
if (typeof unmasked !== "string" || unmasked.trim().length === 0) {
  send("notice_redacted", { index });
  index += 1;
  continue;
}
if (findUnmaskedPii(unmasked, []).length > 0) {
  send("notice_redacted", { index });
  index += 1;
  continue;
}

Why

maskPII replaces phone numbers and emails with tokens and records the original values in a dictionary. When the model outputs that token as-is (or repeats it) and unmask/unmaskPII restores the original text from the dictionary, that text naturally has “the shape of PII.” But this is the legitimate value in the dictionary — what the teacher wrote in the first place — not a raw PII leak generated by the model. findUnmaskedPii doesn’t distinguish whether a string is token-derived; it only checks whether it has the shape of PII, so running it after unmasking can’t tell the two apart.

Fixing it

We moved the check to before unmasking (in mask space).

// Fail-closed re-check in mask space, BEFORE unmasking (rule 4, all string fields). Dictionary-derived
// legitimate PII is still tokenized here and won't look like PII; only raw PII the model generated
// that isn't in the dictionary (= a real leak) gets caught.
for (const it of proposal) {
  for (const field of spec.strings(it)) {
    if (findUnmaskedPii(field, []).length > 0) {
      return { ok: false, reason: "pii_leak" };
    }
  }
}
// Unmask back to the original text (per-section target columns).
items = proposal.map((it) => spec.unmask(it, dictionary));

In mask space, dictionary-derived values stay as tokens (not shaped like PII), so they pass the check. If the model skipped masking or hallucinated raw PII outright, that raw PII still appears as-is at the mask-space stage, so it’s still caught. We only changed the target from “every string after unmasking” to “every string in mask space” — detection power didn’t change.

notice-draft-sse.ts was reordered the same way: check in mask space, then call unmaskPII.

Preventing a repeat

Fixing these two paths was a horizontal rollout that came after the same shape of false positive had already been found and fixed in a chat assistant path. The same design mistake — “check after unmasking” — had independently survived, unrelated to the chat fix, in two other draft paths: a general-purpose draft generator and the streaming delivery of notice drafts. Alongside the fix, tests were added to both paths pinning down “dictionary-derived restored values pass” and “raw PII not in the dictionary gets stopped or redacted.”