PII Masking in Patron Data Exports

Operating within the broader Patron Validation & Privacy Data Routing architecture, this guide covers the hard boundary every ILS integration must cross before circulation telemetry, collection-development metrics, or vendor feeds leave the trusted zone: rewriting personally identifiable fields into masked, non-reversible tokens without destroying the join keys analysts still depend on. Library-tech staff hit this the moment someone asks for a data extract — a consortium wants circulation counts by title, a grant report needs program-attendance figures, a discovery vendor wants usage logs — and the raw patron record cannot lawfully be the thing that ships. This page walks through the export contract, the field-classification table that drives every masking decision, a production-grade deterministic masking implementation, the compliance checkpoints that keep it defensible, and the quarantine, performance, and verification patterns that make it reliable at scale.

Specification & Contract

A masked export is a contract between two parties who are never allowed to see the same data. The upstream side holds validated patron records with full identity linkage; the downstream side — an analytics warehouse, a consortium partner, a reporting spreadsheet — is entitled to analytical utility but not to identity. The masking stage sits on that seam and its contract has two halves: which fields survive, and in what transformed shape.

The first half is field classification. Every field in the source payload is assigned exactly one disposition before any transform runs, because the class determines the technique. Misclassifying a single field — treating an email as a display string rather than a direct identifier, or a branch code as harmless rather than a quasi-identifier — is how re-identification happens.

The second half is the transform contract — the guarantee each technique makes and the property it protects.

Two rules govern the table. First, determinism is scoped to a salt epoch: the same patron_id produces the same token only while the salt is unchanged, which is exactly what lets an analyst join two nightly exports without ever seeing the underlying identifier. Rotate the salt and the linkage intentionally breaks — a feature for retention, a footgun for longitudinal counting, so rotation must be a deliberate, scheduled act. Second, tokenization is not encryption: there is no reverse-lookup table and no decryption path, so a masked export cannot be un-masked even by the team that produced it. For aggregate outputs where even a token is too much surface area, the mathematical calibration of noise budgets belongs to Implementing Differential Privacy for Patron Analytics, and student-record exports carry extra field-suppression obligations detailed in Automating FERPA Compliance in Student Patron Records.

Prerequisites & Environment Setup

The examples target Python 3.11+ and Pydantic v2 (the model_validator and EmailStr behaviour differs from v1; pin the major version so an upgrade does not silently change validation semantics). Cryptographic salts are never literals in the codebase — they are resolved at runtime from a secrets manager or KMS, injected as bytes, and rotated on the jurisdictional schedule. Hardcoding a salt fails public-sector security baselines and, worse, makes every export produced with it permanently linkable if the source ever leaks.

• Python >= 3.11 in the export worker image (python --version)
• pydantic[email] >= 2.5 pinned in requirements.txt (the v2 API is assumed throughout)
• PII_HMAC_SALT resolved from KMS/HSM at runtime (32+ random bytes), never committed
• A separate PII_HMAC_DOMAIN per field family so an email token and an address token of the same string never collide
• Structured (JSON) log handler configured for the patron_pii_audit logger, shipped to append-only storage
• Output sink is an access-controlled bucket or table with mutual TLS on the transport hop
• Field-classification map reviewed against your ILS’s actual patron schema — Sierra, Alma, Polaris, and Symphony name these fields differently, and the same normalization discipline described in ILS Schema Translation Patterns applies before masking begins

Core Implementation

The pipeline runs each record through four labeled stages: validate, resolve keys, transform by class, then assemble the masked record while emitting an audit line. Keeping the stages separate is what makes the transform auditable — a reviewer can point at any field in the output and name the function that produced it.

Step 1 — Validate the inbound payload

Nothing is masked until it is known to be well-formed. A strict Pydantic model rejects records that would otherwise slip an unclassified field into the export. Malformed records are not dropped; they raise a typed error that the caller routes to quarantine (Step covered below).

from __future__ import annotations

import hashlib
import hmac
import json
import logging
from typing import Optional

from pydantic import BaseModel, EmailStr, Field, ValidationError

audit_logger = logging.getLogger("patron_pii_audit")


class PatronPayload(BaseModel):
    """The validated contract every record must satisfy before masking."""

    patron_id: str = Field(min_length=1)
    email: Optional[EmailStr] = None
    address_line: Optional[str] = None
    postal_code: Optional[str] = None
    branch_code: Optional[str] = None
    demographic_bucket: Optional[str] = None

Pitfall: do not use EmailStr as a reason to keep the email. Validation confirms the field is well-formed; classification (Step 3) still decides it is a direct identifier and masks it.

Step 2 — Resolve the salt and domain keys

The salt and per-family domain strings are fetched once per batch from the secrets manager and threaded through as parameters, never read from module globals mid-loop. Domain separation means an email and an address that happen to share a string still tokenize differently.

class MaskingKeys:
    """Runtime-resolved cryptographic material. Never constructed from literals."""

    def __init__(self, salt: bytes, domains: dict[str, str]) -> None:
        if len(salt) < 32:
            raise ValueError("PII_HMAC_SALT must be at least 32 bytes")
        self._salt = salt
        self._domains = domains

    def token(self, family: str, raw_value: str) -> str:
        """Domain-separated HMAC-SHA-256 surrogate for `raw_value`."""
        domain = self._domains[family]
        msg = f"{domain}\x1f{raw_value}".encode("utf-8")
        return hmac.new(self._salt, msg, hashlib.sha256).hexdigest()


def load_keys() -> MaskingKeys:
    import os
    # Replace os.environ with a KMS/Vault client call in production.
    salt = os.environ["PII_HMAC_SALT"].encode("utf-8")
    domains = {
        "patron": os.environ.get("PII_DOMAIN_PATRON", "patron-id-v2"),
        "address": os.environ.get("PII_DOMAIN_ADDRESS", "address-v2"),
    }
    return MaskingKeys(salt=salt, domains=domains)

Pitfall: prefer hmac.new(..., hashlib.sha256) over hashlib.md5. MD5 is not a security primitive and is disabled outright on FIPS-mode hosts, so a pipeline that reaches for it will simply crash in a hardened deployment.

Step 3 — Transform each field by its class

Each class has exactly one function. Partial redaction preserves just enough structure to be human-scannable (the domain of an email, the first character of the local part) without leaking the address itself.

def mask_email(email: str) -> str:
    """Partial redaction preserving domain grain for routing triage."""
    local, _, domain = email.partition("@")
    head = local[0] if local else ""
    return f"{head}***@{domain[:3]}***"


def generalize_postal(postal_code: str) -> str:
    """Truncate to a regional grain that defeats point re-identification."""
    return postal_code[:3].upper()

Step 4 — Assemble the masked record and emit the audit line

The final stage composes the output purely from classified transforms and writes one structured audit entry per record. The correlation ID is itself a truncated HMAC token, so the audit trail is traceable end-to-end without the log becoming a second copy of the PII.

def process_patron_export(record: dict, keys: MaskingKeys) -> dict:
    """Validate one record, mask it by field class, and log the transform."""
    try:
        patron = PatronPayload.model_validate(record)
    except ValidationError as exc:
        audit_logger.warning(
            json.dumps({"event": "VALIDATION_QUARANTINE", "error": exc.errors()})
        )
        raise QuarantineError("schema_violation") from exc

    masked = {
        "patron_token": keys.token("patron", patron.patron_id),
        "email_masked": mask_email(patron.email) if patron.email else None,
        "address_token": (
            keys.token("address", patron.address_line)
            if patron.address_line else None
        ),
        "postal_region": (
            generalize_postal(patron.postal_code) if patron.postal_code else None
        ),
        "branch_code": patron.branch_code,
        "demographic_bucket": patron.demographic_bucket,
    }

    audit_logger.info(
        json.dumps({
            "event": "PII_MASKING_SUCCESS",
            "correlation_id": keys.token("patron", patron.patron_id)[:12],
            "schema_version": "2.1",
            "masking_rules_applied": [
                "hmac_sha256_tokenization",
                "email_partial_redaction",
                "postal_generalization",
            ],
        })
    )
    return masked

Pitfall: assemble the output dict by listing every field explicitly. A dict(patron) spread or a **record merge is how an un-classified field — added later by a vendor schema change — silently rides into the export in the clear.

PII & Compliance Checkpoints

Masking is a compliance control, not just a data transform, and it fails open in subtle ways. Three checkpoints keep it honest.

Minimization at the source. The export should request only the fields the downstream consumer is entitled to, following the data-minimization principle set out in Data Privacy Boundaries in Library Systems. A field that is never pulled cannot be leaked by a classification bug; masking is the second line of defense, not the first.

Retention alignment. A masked export is still governed by a retention clock. Tokens that persist in a warehouse indefinitely can, in aggregate with other releases, erode anonymity, so the export’s lifecycle must be mapped to the windows in Data Retention Policies for Public Libraries. Where an export feeds long-lived circulation analytics, the one-way anonymization boundary in Circulation History Routing & Anonymization should run upstream of the export so the token is already a surrogate, not a live identity.

Audit completeness. Every record that enters the stage must produce exactly one audit line — success or quarantine — carrying a correlation ID, the schema version, and the ordered list of rules applied. An export whose audit log has fewer entries than the export has rows is not defensible, because you cannot demonstrate that the missing rows were masked rather than skipped.

Error Handling & Quarantine Patterns

The stage distinguishes two failure classes and never lets either one contaminate the output. A schema violation means a single bad record; a key-resolution failure means the whole batch is unsafe and must stop.

class QuarantineError(Exception):
    """A single record could not be safely masked and must be diverted."""


def run_batch(records: list[dict], keys: MaskingKeys, quarantine_sink) -> list[dict]:
    masked_out: list[dict] = []
    for record in records:
        try:
            masked_out.append(process_patron_export(record, keys))
        except QuarantineError as exc:
            # Route the raw record to an isolated queue for human inspection;
            # it is never written to the export, and the batch keeps flowing.
            quarantine_sink.put({
                "reason": str(exc),
                "raw_ref": record.get("patron_id", "unknown"),
            })
    return masked_out

Per-record QuarantineErrors are caught and diverted so one malformed row cannot fail an otherwise valid nightly export — the same isolation discipline libraries use when they route bad ingest records to a schema validation quarantine queue. By contrast, a ValueError from MaskingKeys (salt too short, missing domain) or a KeyError from the secrets client is not caught here: if the cryptographic material is wrong, every token in the batch would be wrong, so the correct behaviour is to abort the whole run loudly before a single record ships. The quarantine record deliberately stores only a reference (patron_id), not the failed payload, so the quarantine queue does not become an unmasked shadow copy of the data.

Performance Considerations

HMAC-SHA-256 is cheap per call, so the bottleneck in a large export is almost never the cryptography — it is materializing the whole result set in memory. Pulling a full patron table into a list before masking is the classic cause of an OOM-killed export worker.

Stream instead. Consume the source query in bounded, keyset-paginated pages, mask each page, write it to the sink, and retain nothing across iterations:

def stream_export(cursor, keys: MaskingKeys, writer, page_size: int = 5_000) -> None:
    while True:
        page = cursor.fetchmany(page_size)
        if not page:
            break
        writer.write_page(process_patron_export(r, keys) for r in page)

For very large exports, offload paging to the broker rather than a single worker: the durable-topic, at-least-once consumer pattern developed for Async Batch Processing for Catalog Updates applies unchanged here, with masking as the consumer’s side effect. And because the source is usually a vendor API, respect the throttle discipline in ILS REST API Polling & Rate Limiting — an export that hammers the patron endpoint will be rate-limited into a stall long before masking becomes the limiting factor.

Verification & Testing

Two properties must be proven by test, not assumed: determinism (the same input yields the same token within a salt epoch) and non-leakage (no direct identifier survives in the output).

import pytest


@pytest.fixture
def keys() -> MaskingKeys:
    return MaskingKeys(salt=b"x" * 32, domains={"patron": "p", "address": "a"})


def test_tokenization_is_deterministic(keys):
    a = process_patron_export({"patron_id": "P1"}, keys)
    b = process_patron_export({"patron_id": "P1"}, keys)
    assert a["patron_token"] == b["patron_token"]


def test_salt_rotation_breaks_linkage():
    old = MaskingKeys(salt=b"x" * 32, domains={"patron": "p", "address": "a"})
    new = MaskingKeys(salt=b"y" * 32, domains={"patron": "p", "address": "a"})
    assert (
        process_patron_export({"patron_id": "P1"}, old)["patron_token"]
        != process_patron_export({"patron_id": "P1"}, new)["patron_token"]
    )


def test_no_raw_identifier_leaks(keys):
    out = process_patron_export(
        {"patron_id": "P1", "email": "[email protected]",
         "address_line": "42 Elm St"}, keys)
    flat = json.dumps(out)
    assert "[email protected]" not in flat
    assert "42 Elm St" not in flat
    assert out["patron_token"] != "P1"


def test_malformed_record_is_quarantined(keys):
    with pytest.raises(QuarantineError):
        process_patron_export({"patron_id": ""}, keys)

Run the same non-leakage assertion as a post-export gate over a sample of the actual output file, not just unit fixtures — a schema drift that adds a new PII field will pass every unit test written before the field existed, but the output-file scan catches the raw value the moment it appears.

Troubleshooting & FAQ

The same patron produces two different tokens in last night’s and tonight’s export. Why?

The salt was rotated between the runs, or the two runs used different PII_HMAC_DOMAIN values. Determinism holds only within a single salt epoch and a fixed domain. If analysts need tokens stable across a rotation for longitudinal counting, keep the previous salt for a grace window and re-key deliberately rather than treating the drift as a defect.

An analyst asked us to reverse a token back to the patron. Can we?

No, and that is the point. Tokenization keeps no reverse-lookup table and there is no decryption key — it is a one-way HMAC. If a genuine identity is needed, the correct path is to query the source ILS under proper authorization, not to un-mask an export.

A new demographic field started appearing unmasked in the export. What broke?

The output dict was assembled with a spread or merge (**record) instead of an explicit field list, so a vendor schema change added an unclassified field that rode straight through. Switch to explicit assembly (Step 4) and add the field to the classification table so it gets a deliberate disposition.

The export worker is OOM-killed on the full patron table. How do I fix it?

The query result is being materialized into a list before masking. Consume it in keyset-paginated pages with fetchmany, mask and write each page, and retain nothing across iterations, as shown under Performance Considerations. For tables too large for a single worker, move paging onto the broker.

The pipeline crashes at startup with an MD5 or salt-length error on a hardened host. Why?

FIPS-mode hosts disable MD5, and the MaskingKeys constructor rejects a salt under 32 bytes. Both are intentional guards: replace any hashlib.md5 call with the HMAC-SHA-256 tokenizer and resolve a full-length salt from KMS. A crash here is the system refusing to produce weakly-masked output.

Related

• Circulation History Routing & Anonymization — the one-way anonymization boundary that should run upstream of any long-lived export
• Data Retention Policies for Public Libraries — the retention clock a masked export is still governed by
• Implementing Differential Privacy for Patron Analytics — noise calibration for aggregate outputs where even a token is too much surface area
• Automating FERPA Compliance in Student Patron Records — the extra field-suppression rules that apply to student exports
• Data Privacy Boundaries in Library Systems — the minimization principle behind classifying fields before masking them