Generating State-Specific Retention Exports

This page answers one narrow operational question: why does a single circulation-record retention export satisfy one state’s confidentiality law while quietly violating another’s, and how do you parametrise the exporter by jurisdiction so each record is retained or purged under the rule that actually governs it? It sits under Compliance Reporting for Circulation Data, which covers the reporting and audit contract end to end, and within the broader Patron Validation & Privacy Data Routing architecture. If you run a multi-state consortium — a shared ILS whose branches sit in different states, or a state-library aggregation of county systems — a hardcoded export is a compliance gap waiting for the first audit that spans a state line.

Problem Framing

The symptom is not a crash. The retention export runs on schedule, produces a file the state library accepts, and no exception fires. The gap surfaces at audit. A Vermont auditor pulls the same export and finds circulation records that Vermont law required be non-retained sitting in the disclosable feed, while a California branch in the same consortium finds records purged eighteen months early — destroyed inside a window California still required them retained. One export config satisfied neither state; it satisfied whichever state its single hardcoded rule happened to encode.

The mismatch is visible the moment you diff the export’s behaviour against per-state law. A quick tally of what a fixed 12-month rule did to a multi-state batch makes it obvious:

$ python -m retention.audit --batch 2026-q2 --group-by governing_state
governing_state  records  over_retained  over_purged  status
VT               4,812    4,812          0            FAIL  <- VT allows no retention of loan history
CA               9,103    0              1,244        FAIL  <- CA requires 2y; purged at 12m
TX               6,540    0              0            OK    <- happens to match the hardcoded 12m-ish rule
NH               2,207    2,207          0            FAIL  <- NH window shorter than 12m

The single 12-month rule is correct only for Texas, which is the state whose window the original author happened to hardcode. Every other jurisdiction is either over-retained (a confidentiality violation) or over-purged (a records-retention violation). Because the two failure directions are opposites, no single fixed number can be moved to fix them all — raising the window worsens Vermont, lowering it worsens California.

Root Cause

The failure is a jurisdiction-blind export. US state library-record confidentiality and retention rules are not uniform: they differ in the retention window (how long circulation history may or must be kept) and in field disposition (which fields must be purged as confidential versus which remain disclosable). Vermont’s confidentiality statute effectively bars retention of identifiable loan history; California requires certain records be kept for a defined period; other states set windows in between and disagree on whether a patron identifier may persist at all.

A single hardcoded export encodes exactly one of these regimes. Retention policy is both jurisdiction-specific and time-based — a record’s disposition depends on which state governs it and how old it is — so a fixed rule is structurally incapable of being correct for more than one jurisdiction at once. It is the same default-open trap that field-level masking avoids in PII Masking in Patron Data Exports, transposed onto time: a rule that cannot represent “purge in VT, retain in CA” will silently pick one and violate the other.

The fix is to stop hardcoding the rule and start parametrising by jurisdiction. Each state must declare its own retention window and per-field disposition in data, and the exporter must resolve the matching policy for each record’s governing state before deciding to export or purge.

Solution

Replace the fixed rule with a policy-table-driven exporter. Each jurisdiction is declared as a typed policy — its retention window and the disposition of each confidential field — and a resolver selects the policy for a record’s governing_state. A record with no matching policy is routed to quarantine rather than exported under a guessed rule, so a state you have not yet encoded fails closed instead of leaking or purging under the wrong regime. This mirrors the disposition-per-field discipline used across circulation compliance reporting.

from __future__ import annotations

import enum
import logging
from dataclasses import dataclass, field
from datetime import date, timedelta

logger = logging.getLogger("retention.export")


class FieldDisposition(enum.Enum):
    KEEP = "keep"        # disclosable, retained verbatim
    HASH = "hash"        # retained only as a non-reversible surrogate
    PURGE = "purge"      # confidential, removed from any export


class UnknownJurisdictionError(KeyError):
    """Raised when no retention policy is declared for a record's state."""


@dataclass(frozen=True)
class RetentionPolicy:
    """One jurisdiction's declared retention rule."""

    state: str
    retention_days: int  # 0 = no identifiable retention permitted
    field_dispositions: dict[str, FieldDisposition] = field(default_factory=dict)

    def is_within_window(self, record_age: timedelta) -> bool:
        return record_age <= timedelta(days=self.retention_days)


# Jurisdiction-parametrised policy table. Each state declares its own rule;
# no single hardcoded window is assumed to apply across the consortium.
POLICY_TABLE: dict[str, RetentionPolicy] = {
    "VT": RetentionPolicy("VT", retention_days=0),
    "CA": RetentionPolicy(
        "CA",
        retention_days=730,
        field_dispositions={"patron_id": FieldDisposition.HASH},
    ),
    "TX": RetentionPolicy(
        "TX",
        retention_days=548,
        field_dispositions={"patron_id": FieldDisposition.KEEP},
    ),
    "NH": RetentionPolicy("NH", retention_days=90),
}


def resolve_policy(governing_state: str) -> RetentionPolicy:
    """Select the retention policy that governs a record's jurisdiction."""
    try:
        return POLICY_TABLE[governing_state]
    except KeyError as exc:
        raise UnknownJurisdictionError(governing_state) from exc

With the policy resolved per record, the exporter applies the matching window and field dispositions instead of a shared constant. Records past their state’s window go to a purge queue; records still inside it are emitted with confidential fields removed or hashed exactly as that state declared.

import hashlib
import hmac
import os


@dataclass(frozen=True)
class ExportOutcome:
    exported: list[dict[str, str]]
    to_purge: list[dict[str, str]]
    quarantined: list[dict[str, str]]


def _pseudonymize(value: str) -> str:
    """Non-reversible HMAC-SHA-256 surrogate for a field kept only in hashed form."""
    secret = os.environ["RETENTION_HMAC_SECRET"].encode()
    return hmac.new(secret, f"patron:{value}".encode(), hashlib.sha256).hexdigest()


def apply_disposition(
    record: dict[str, str], policy: RetentionPolicy
) -> dict[str, str]:
    """Project a record to its disclosable form under a jurisdiction's policy."""
    out: dict[str, str] = {}
    for key, value in record.items():
        disposition = policy.field_dispositions.get(key, FieldDisposition.KEEP)
        if disposition is FieldDisposition.PURGE:
            continue
        if disposition is FieldDisposition.HASH:
            out[key] = _pseudonymize(value)
        else:
            out[key] = value
    return out


def build_retention_export(
    records: list[dict[str, str]], *, as_of: date, batch_id: str
) -> ExportOutcome:
    """Route each record under the policy for its governing state.

    In-window records are exported with confidential fields removed; expired
    records are queued for purge; records from an undeclared state fail closed
    into quarantine rather than being exported under a substitute rule.
    """
    outcome = ExportOutcome(exported=[], to_purge=[], quarantined=[])

    for record in records:
        state = record.get("governing_state", "")
        try:
            policy = resolve_policy(state)
        except UnknownJurisdictionError:
            outcome.quarantined.append(record)
            logger.error(
                "unknown_jurisdiction",
                extra={"state": state, "batch_id": batch_id, "action": "quarantine"},
            )
            continue

        age = as_of - date.fromisoformat(record["checkout_date"])
        if policy.retention_days == 0 or not policy.is_within_window(age):
            outcome.to_purge.append(record)
            logger.info(
                "record_expired",
                extra={"state": state, "batch_id": batch_id, "action": "purge"},
            )
            continue

        outcome.exported.append(apply_disposition(record, policy))
        logger.info(
            "record_exported",
            extra={"state": state, "batch_id": batch_id, "action": "export"},
        )

    return outcome

The behavioural change is that the retention window is no longer a constant — it is looked up per record. Before, every record was measured against one hardcoded number, so Vermont over-retained and California over-purged. After, a Vermont record with retention_days=0 is always routed to purge, a California record is retained for two years with patron_id hashed, and a Texas record keeps its identifier for eighteen months, each under the rule that actually governs it. Adding a new state is a data change — one POLICY_TABLE row — not a code change, and until that row exists the state’s records quarantine rather than export under a neighbour’s law.

Compliance or Privacy Impact

Parametrising the export by jurisdiction changes the compliance posture in three concrete ways, and each has a downstream effect worth tracking.

• The retention window is now provably per-state. Because every record is logged with the state and action that governed it, you can answer an auditor’s “under which rule was this record retained?” with a log query rather than an assurance. Feed these record_exported, record_expired, and unknown_jurisdiction events into the same immutable audit sink defined for circulation reporting so the decision is durable.
• Purge is authoritative, not incidental. A retention_days=0 policy such as Vermont’s means the export never emits identifiable loan history for that state, and the purge queue is the enforcement point. Align the actual deletion job that drains that queue with the schedules described in Data Retention Policies for Public Libraries, and never let a purged record survive in an upstream table past its window.
• Field disposition composes with masking. The HASH and PURGE dispositions decide what leaves the export, but for any field that stays disclosable, apply the same field-classification discipline described in PII Masking in Patron Data Exports so that a “kept” field in one state is not silently a leak in another. A patron identifier kept verbatim under Texas law must still be pseudonymized before it enters any cross-state aggregate.

The change does not widen the disclosure surface — it narrows each state to its lawful minimum — but it shifts risk onto the correctness of POLICY_TABLE. A window set too long, or a field mistakenly marked KEEP, is a violation with the exporter’s full authority behind it, so every policy-row change deserves code review and the verification below.

Verification

Confirm the exporter is jurisdiction-correct with a test that asserts a zero-retention state never appears in the export, an in-window state is retained with its declared dispositions, and an undeclared state fails closed into quarantine.

from datetime import date


def test_export_is_jurisdiction_specific() -> None:
    records = [
        {"governing_state": "VT", "checkout_date": "2026-07-01", "patron_id": "P1"},
        {"governing_state": "CA", "checkout_date": "2026-01-01", "patron_id": "P2"},
        {"governing_state": "ZZ", "checkout_date": "2026-01-01", "patron_id": "P3"},
    ]
    outcome = build_retention_export(
        records, as_of=date(2026, 7, 16), batch_id="test"
    )

    exported_states = {r["governing_state"] for r in outcome.exported}

    # Vermont permits no identifiable retention: never in the export.
    assert "VT" not in exported_states
    assert any(r["governing_state"] == "VT" for r in outcome.to_purge)

    # California is in-window but must hash the patron identifier.
    ca = next(r for r in outcome.exported if r["governing_state"] == "CA")
    assert ca["patron_id"] != "P2"

    # An undeclared state fails closed rather than exporting under a guess.
    assert any(r["governing_state"] == "ZZ" for r in outcome.quarantined)
    assert "ZZ" not in exported_states

For a running pipeline, add a continuous invariant on the export itself: group the emitted rows by governing_state and assert that every state present has a POLICY_TABLE entry and that no row’s checkout_date is older than its policy window. If a batch ever produces a governing_state with no policy, halt the export and quarantine the batch — the same fail-closed reflex the resolver applies per record, applied to the batch as a whole. Track unknown_jurisdiction counts over time; a nonzero rate after a consortium onboards a new member is your early signal that a state’s policy row is still missing.

Related

• Compliance Reporting for Circulation Data — the parent guide to the circulation reporting and audit contract this export feeds.
• Patron Validation & Privacy Data Routing — the privacy and routing architecture that governs where circulation data may flow.
• Designing an Append-Only Audit Log Schema — the immutable sink that records every export, purge, and quarantine decision.
• Building Batch Reconciliation Reports for Circulation — reconciling exported and purged counts against source circulation batches.