Patron Consent and Right-to-Erasure Management

Operating within the broader Patron Validation & Privacy Data Routing architecture, this guide covers the problem that surfaces the day a patron says “delete everything you hold about me”: a circulation pipeline never keeps patron-linked data in one place. The same identity is scattered across a patron index, an append-only circulation log, a nightly-loaded analytics warehouse, and a rotation of encrypted backups — and a defensible erasure has to reach every one of them, prove it reached them, and reconcile the result against the append-only history that regulators also require you to keep. Library-tech staff hit this at the intersection of three obligations that pull in different directions: the GDPR Article 17 right to erasure, CCPA deletion rights, and the long-standing library duty to keep reading records confidential. This page walks through the consent and erasure state model, the fan-out propagation contract, a production-grade erasure orchestrator, the compliance checkpoints that keep the audit log and the erasure honest at the same time, and the quarantine, performance, and verification patterns that make erasure reliable across a distributed pipeline.

Specification & Contract

An erasure is a contract between a patron’s stated wish and a distributed system that never agreed to keep their data in one place. The request is singular — “erase me” — but honouring it is a fan-out: each store that holds patron-linked data must apply a store-specific deletion and report back, and the request is not complete until every one of them has. The contract therefore has two halves: the state you track for each request, and the propagation guarantee each downstream target makes.

The first half is the consent and erasure state model. Every request carries a small, explicit record that is itself PII-free — the patron is referenced only by a salted hash, never by a live identifier — so the erasure bookkeeping does not become a fresh copy of the thing being erased. The erasure_status field is the spine: it advances monotonically from pending to propagating to verified, and it never regresses, so a partially-completed erasure is always distinguishable from a finished one.

The second half is the propagation contract — what “deleted” actually means at each target, because it is not the same operation everywhere.

Two rules govern the contract. First, erasure and the audit log are not in conflict — they operate on different columns. The append-only circulation log and the compliance record must survive for their own statutory reasons, so the orchestrator erases the PII (name, contact, exact item title) while leaving a hashed tombstone that proves an event happened and was later erased, without revealing who or what. Retaining that minimized skeleton is exactly the discipline described in Data Retention Policies for Public Libraries, and the tombstone itself is a masked surrogate produced by the same technique used in PII Masking in Patron Data Exports. Second, backups are tombstoned, not rewritten: you cannot cheaply delete one patron from an immutable snapshot, so the accepted practice is to record a suppression marker that the restore path honours, guaranteeing the erased record never re-materializes even after a disaster recovery. Student records carry an additional retention-schedule overlay, detailed in Enforcing FERPA Retention Schedules in Patron Records.

Prerequisites & Environment Setup

The examples target Python 3.11+ and lean on dataclasses, enum, and the standard logging module; no framework is assumed, because the orchestrator has to sit above whatever driver each store speaks. The hashing salt is never a literal — it is resolved at runtime from a secrets manager, so the patron_id_hashed written into the erasure ledger cannot be reversed even by someone holding the ledger. Every downstream target is addressed through a small adapter with a uniform erase(hashed_id) method, so the orchestrator never encodes store-specific SQL and can be tested against fakes.

• Python >= 3.11 in the erasure-worker image (python --version)
• PATRON_HASH_SALT resolved from KMS/HSM at runtime (32+ random bytes), never committed
• One adapter per store, each exposing erase(hashed_id) -> TargetResult, registered with the orchestrator
• Structured (JSON) log handler configured for the patron_erasure_audit logger, shipped to append-only storage
• A durable retry queue (broker or table) for targets that fail transiently, so a warehouse timeout does not abandon the request
• The append-only circulation log schema exposes nullable PII columns plus a tombstone_hash column — see Designing an Append-Only Audit Log Schema
• Field naming reconciled against your ILS’s patron schema — Sierra, Alma, Polaris, and Symphony differ, and the same normalization discipline from ILS Schema Translation Patterns applies before erasure begins

Core Implementation

The orchestrator runs each request through four labeled stages: hash the identifier, record the request in the pending state, fan the deletion out to every registered target while tracking per-target results, then verify. Keeping the stages separate is what makes an erasure auditable — a reviewer can point at any target and name the result that store returned.

Step 1 — Model the request and target results

The state model is expressed as typed dataclasses so the erasure_status transitions are explicit and the propagation_targets map is never a loose dict. Nothing here holds a live identifier.

from __future__ import annotations

import enum
import hashlib
import hmac
import json
import logging
from dataclasses import dataclass, field
from datetime import datetime, timezone

audit_logger = logging.getLogger("patron_erasure_audit")


class ErasureStatus(enum.Enum):
    PENDING = "pending"
    PROPAGATING = "propagating"
    VERIFIED = "verified"


class TargetStatus(enum.Enum):
    PENDING = "pending"
    TOMBSTONED = "tombstoned"
    VERIFIED = "verified"
    FAILED = "failed"


@dataclass(frozen=True)
class TargetResult:
    """Outcome reported by one downstream store's adapter."""

    target: str
    status: TargetStatus
    detail: str = ""


@dataclass
class ErasureRequest:
    """PII-free state record for one right-to-erasure request."""

    patron_id_hashed: str
    consent_scope: frozenset[str]
    consent_granted_at: datetime
    erasure_requested_at: datetime
    erasure_status: ErasureStatus = ErasureStatus.PENDING
    propagation_targets: dict[str, TargetStatus] = field(default_factory=dict)

Pitfall: make erasure_status an enum, not a string. A stray "complete" typo compared against "verified" is how a half-propagated request gets reported as done; an enum makes the illegal state unrepresentable.

Step 2 — Hash the identifier once, at the boundary

The live patron id enters the orchestrator exactly once and is immediately reduced to a salted HMAC. Every store, every log line, and every ledger entry downstream references only the hash, so no component below this line can leak the identity.

class PatronHasher:
    """Runtime-resolved salt. Never constructed from literals."""

    def __init__(self, salt: bytes) -> None:
        if len(salt) < 32:
            raise ValueError("PATRON_HASH_SALT must be at least 32 bytes")
        self._salt = salt

    def hash_id(self, patron_id: str) -> str:
        msg = patron_id.encode("utf-8")
        return hmac.new(self._salt, msg, hashlib.sha256).hexdigest()


def load_hasher() -> PatronHasher:
    import os
    # Replace os.environ with a KMS/Vault client call in production.
    salt = os.environ["PATRON_HASH_SALT"].encode("utf-8")
    return PatronHasher(salt=salt)

Pitfall: hash with hmac.new(..., hashlib.sha256), not a bare hashlib.sha256(patron_id). An unkeyed hash of a low-entropy identifier (a six-digit barcode) is trivially reversed by brute force; the keyed HMAC defeats that as long as the salt stays secret.

Step 3 — Define the store adapter contract

Each downstream store is wrapped in an adapter with one job: apply its own deletion semantics for a hashed id and return a TargetResult. The orchestrator never sees SQL. The circulation-log adapter is the interesting one — it nulls PII columns and writes a tombstone rather than deleting the row.

from typing import Protocol


class EraseTarget(Protocol):
    name: str

    def erase(self, hashed_id: str) -> TargetResult: ...


class CirculationLogTarget:
    """Append-only log: erase PII in place, keep a hashed tombstone row."""

    name = "circ_log"

    def __init__(self, conn) -> None:
        self._conn = conn

    def erase(self, hashed_id: str) -> TargetResult:
        with self._conn.begin() as tx:
            tx.execute(
                "UPDATE circ_log "
                "SET patron_name = NULL, patron_contact = NULL, "
                "    item_title = NULL, tombstone_hash = :h "
                "WHERE patron_id_hashed = :h",
                {"h": hashed_id},
            )
        return TargetResult(self.name, TargetStatus.TOMBSTONED,
                            "pii nulled; tombstone retained")

Pitfall: do not DELETE FROM circ_log. The append-only history is a statutory record; removing rows destroys the very audit trail Compliance Reporting for Circulation Data depends on. Erase the PII columns, keep the tombstone.

Step 4 — Fan out, track per-target results, and verify

The orchestrator moves the request to propagating, invokes each adapter inside a context manager that guarantees an audit line, records each TargetResult, and promotes the request to verified only when every target reports a terminal success. A single failed target keeps the whole request in propagating.

import contextlib


class ErasureOrchestrator:
    def __init__(self, targets: list[EraseTarget], retry_sink) -> None:
        self._targets = targets
        self._retry_sink = retry_sink

    def run(self, request: ErasureRequest) -> ErasureRequest:
        request.erasure_status = ErasureStatus.PROPAGATING
        for target in self._targets:
            with self._audited(request, target.name):
                try:
                    result = target.erase(request.patron_id_hashed)
                except Exception as exc:  # adapter-level failure is isolated
                    request.propagation_targets[target.name] = TargetStatus.FAILED
                    self._retry_sink.put({
                        "hashed_id": request.patron_id_hashed,
                        "target": target.name,
                        "reason": repr(exc),
                    })
                    raise TargetErasureError(target.name) from exc
                request.propagation_targets[target.name] = result.status

        terminal = {TargetStatus.VERIFIED, TargetStatus.TOMBSTONED}
        if all(s in terminal for s in request.propagation_targets.values()):
            request.erasure_status = ErasureStatus.VERIFIED
        return request

    @contextlib.contextmanager
    def _audited(self, request: ErasureRequest, target: str):
        try:
            yield
        finally:
            audit_logger.info(json.dumps({
                "event": "ERASURE_TARGET",
                "patron_id_hashed": request.patron_id_hashed[:12],
                "target": target,
                "target_status": request.propagation_targets.get(
                    target, TargetStatus.PENDING).value,
                "erasure_status": request.erasure_status.value,
                "ts": datetime.now(timezone.utc).isoformat(),
            }))

Pitfall: promote to verified only after checking every target against a terminal set. An early return inside the loop, or an any() where you meant all(), marks a request done while the warehouse deletion is still pending — the single most dangerous bug in an erasure pipeline, because it produces a compliant-looking record over live data.

PII & Compliance Checkpoints

Erasure is a compliance control, not just a set of deletes, and it fails open in subtle ways. Three checkpoints keep it honest.

Completeness over the whole surface. An erasure is only lawful if it reaches every store that holds patron-linked data — including derived analytics that were computed from circulation history and may outlive the source rows. Map the full data surface first, following the boundary discipline in Data Privacy Boundaries in Library Systems; a target that is never registered with the orchestrator cannot be erased, and its omission will not show up in any per-target status.

Erase PII, retain the minimized proof. The tension between Article 17 and the append-only audit log resolves at the column level, not the row level. The compliance record must show that an event occurred and was later erased, which the hashed tombstone provides, while the identifying content is gone. The masking used to produce that tombstone is the same one-way tokenization documented in PII Masking in Patron Data Exports, and the retention windows that decide how long even the tombstone lives come from Data Retention Policies for Public Libraries.

Verifiable evidence. Every request must produce a per-target audit trail and a final verified transition, so that when a regulator or the patron asks you to demonstrate the erasure, you can show which store was cleared, when, and by what operation. An erasure whose ledger stops at propagating is not evidence of completion — it is evidence of an unfinished obligation, which is exactly what the reconciliation reporting under Compliance Reporting for Circulation Data is built to surface.

Error Handling & Quarantine Patterns

The orchestrator distinguishes two failure classes and never lets either one falsely report success. A single target failure means that store must be retried; a hasher or configuration failure means the whole run is unsafe and must stop before any target is touched.

class TargetErasureError(Exception):
    """One downstream store could not be erased and must be retried."""


def run_with_isolation(orchestrator: ErasureOrchestrator,
                       request: ErasureRequest) -> ErasureRequest:
    try:
        return orchestrator.run(request)
    except TargetErasureError as exc:
        # The failed target is already on the retry queue and marked FAILED;
        # the request stays in `propagating` so it is never reported complete.
        audit_logger.warning(json.dumps({
            "event": "ERASURE_INCOMPLETE",
            "patron_id_hashed": request.patron_id_hashed[:12],
            "failed_target": str(exc),
            "erasure_status": request.erasure_status.value,
        }))
        return request

A TargetErasureError is caught and the failing store is diverted to the retry queue so one flaky warehouse connection cannot abandon an otherwise-progressing erasure — the same isolation discipline libraries use when they route bad ingest records to a schema validation quarantine queue. By contrast, a ValueError from PatronHasher (salt too short) or a KeyError from the secrets client is not caught here: if the hashing material is wrong, the patron_id_hashed would be wrong at every target, so the correct behaviour is to abort before a single store is touched. The retry queue stores only the hashed id and the target name, never the live patron record, so the queue never becomes an un-erased shadow copy of the data being erased.

Performance Considerations

An erasure is I/O-bound across independent stores, so the cost is dominated by the slowest target — usually the analytics warehouse, where deleting a token’s derived rows can touch several large fact tables. Because the four targets share no state, they can be fanned out concurrently rather than serially; a ThreadPoolExecutor (or async gather) collapses the wall-clock time to the slowest single target instead of their sum, while each adapter still reports its own TargetResult.

Two constraints bound the concurrency. First, the warehouse deletion should be keyed on the patron token so it hits an index rather than scanning; the token comes pre-computed from the anonymization boundary in Circulation History Routing & Anonymization, so the warehouse never needs the live id to find the rows. Second, when requests arrive in bulk — a consortium offboarding, a data-subject campaign — batch them onto the broker rather than spawning a thread per request: the durable, at-least-once consumer pattern from Async Batch Processing for Catalog Updates applies unchanged, with erasure as the side effect, and its backpressure keeps a surge from exhausting warehouse connections.

Verification & Testing

Two properties must be proven by test, not assumed: completeness (a request reaches verified only when every target is terminal) and non-regression (a failing target keeps the request in propagating and never falsely completes).

import pytest


class _FakeTarget:
    def __init__(self, name: str, status: TargetStatus) -> None:
        self.name = name
        self._status = status

    def erase(self, hashed_id: str) -> TargetResult:
        if self._status is TargetStatus.FAILED:
            raise RuntimeError("store unreachable")
        return TargetResult(self.name, self._status)


def _request() -> ErasureRequest:
    now = datetime.now(timezone.utc)
    return ErasureRequest(
        patron_id_hashed="a" * 64,
        consent_scope=frozenset({"circulation"}),
        consent_granted_at=now,
        erasure_requested_at=now,
    )


def test_all_targets_terminal_reaches_verified():
    targets = [_FakeTarget("index", TargetStatus.VERIFIED),
               _FakeTarget("circ_log", TargetStatus.TOMBSTONED)]
    out = ErasureOrchestrator(targets, retry_sink=[]).run(_request())
    assert out.erasure_status is ErasureStatus.VERIFIED


def test_failed_target_stays_propagating():
    sink: list[dict] = []
    targets = [_FakeTarget("index", TargetStatus.VERIFIED),
               _FakeTarget("warehouse", TargetStatus.FAILED)]
    orch = ErasureOrchestrator(targets, retry_sink=sink)
    run_with_isolation(orch, _request())
    assert sink and sink[0]["target"] == "warehouse"


def test_status_never_regresses():
    req = _request()
    req.erasure_status = ErasureStatus.VERIFIED
    # A later no-op run must not pull a verified request backwards.
    assert req.erasure_status is ErasureStatus.VERIFIED

Run the completeness assertion as a post-erasure gate over the actual stores, not just fakes: read back the hashed id from the patron index, count warehouse rows for the token, and confirm the tombstone exists in the circulation log. A registration bug that quietly drops a target from the orchestrator will pass every unit test written against the fakes, but the read-back gate catches the store that was never cleared.

Troubleshooting

A request shows verified but the analytics warehouse still returns rows for the patron. What happened?

The orchestrator promoted the request before checking every target — almost always an any() where all() was meant, or an early return inside the fan-out loop. Verification must require every entry in propagation_targets to be in the terminal set. Re-run the request; the read-back gate under Verification & Testing should be wired as a hard check so this can never surface as a false verified again.

The circulation log lost rows after an erasure and now compliance reports don’t reconcile. Why?

An adapter ran DELETE FROM circ_log instead of nulling the PII columns and writing a tombstone. The append-only history is a statutory record; erasure operates on the PII columns, not the row. Restore the tombstone rows from backup and switch the adapter to the in-place update shown in Step 3.

A patron was erased, then reappeared after a database restore. How do we prevent that?

The backup set was not tombstoned. Immutable snapshots cannot be edited per-patron, so the restore path must consult a suppression list and re-apply the erasure to any restored record. Record a tombstone against every in-window snapshot at erasure time, and make the restore job honour it before the data is served.

Erasure requests are timing out under a bulk offboarding. What’s the fix?

The requests are being processed inline, one thread per request, exhausting warehouse connections. Move them onto the broker and process them as at-least-once consumer messages with bounded concurrency, as covered under Performance Considerations, so backpressure caps the concurrent load on the slowest target.

Two runs produced different patron_id_hashed values for the same patron. Why?

The hashing salt changed between runs, or a bare unkeyed hash was used in one path and the HMAC in another. The hash must be a keyed HMAC-SHA-256 with a stable, secret salt for the identity to correlate across the index, log, and warehouse. Rotating the salt intentionally breaks that linkage, so treat rotation as a scheduled event, not an incidental config change.

Related

• Implementing GDPR Right-to-Erasure in Circulation Data — the Article 17 workflow applied end-to-end across a circulation pipeline
• Enforcing FERPA Retention Schedules in Patron Records — the student-record retention overlay that constrains what an erasure may remove
• Compliance Reporting for Circulation Data — the reconciliation reports that surface any erasure stuck short of verified
• PII Masking in Patron Data Exports — the one-way tokenization that produces the hashed tombstone left behind after erasure
• Data Retention Policies for Public Libraries — the retention clock that decides how long even a minimized tombstone survives