Library Catalog Data Sync Pipelines

Implementing Differential Privacy for Patron Analytics

You published a branch-level circulation dashboard and a privacy reviewer flagged that a “world-language DVDs” cell showing a count of 3 — cross-referenced against last quarter’s release — narrows to a single identifiable borrower. Or the analytics job that adds statistical noise is getting OOM-killed on the full patron_activity table, or its privacy budget is exhausted before the month ends. This page answers one narrow question: how do you release aggregate patron analytics that are provably non-attributable and still usable, without the noise going unbounded on sparse bins? It sits under the PII Masking in Patron Data Exports cluster and the broader Patron Validation & Privacy Data Routing architecture, and picks up exactly where token-based masking runs out of protection: aggregate outputs where even an HMAC surrogate token is too much surface area.

Differential-privacy aggregation path from raw circulation telemetry to a published aggregate A horizontal pipeline. Raw circulation telemetry flows into a per-patron contribution clip that caps each patron at C, bounding the query's global sensitivity. The clipped stream is aggregated in bounded chunks into counts by branch. A Laplace mechanism then adds noise drawn from Lap(C over epsilon); an epsilon-budget accountant sits below it, tracks the remaining per-tenant budget under sequential composition, and calibrates the noise scale. The noised output passes through a non-negative clamp, which is pure post-processing and spends no budget, before the published aggregate is released to the dashboard. The clip stage is annotated as bounding sensitivity, the noise stage as consuming epsilon, and the clamp stage as free post-processing. INGEST BOUND · AGGREGATE PRIVATIZE RELEASE Raw circulation telemetry Per-patron clip min(c, C) bounds global sensitivity to C Chunked aggregate counts by branch consumes ε Laplace mechanism + Lap(C / ε) Clamp max(0, x) post-processing · free Published aggregate → dashboard ε-BUDGET ACCOUNTANT per-tenant · sequential composition ε_remaining before ceiling sets scale

Problem Framing

The symptom shows up in one of three forms, all of which trace back to the same mishandled quantity — query sensitivity — or to the same mishandled resource — the privacy budget.

Form 1: a small cell in a published aggregate is effectively attributable. A count of 1–3 in a rare category (a specialized program, a low-demand language collection) survives to the release, and differencing it against a prior release identifies a person. Noise was either not applied or applied with too small a scale.

Form 2: the released numbers are unusable. Branch totals swing by hundreds between runs, or come back negative, because the noise scale was calibrated against the observed dataset variance during a checkout spike rather than the true sensitivity of the query.

Form 3: the job dies. Either an out-of-memory kill while materializing the circulation window, or a hard stop from the budget accountant:

text
2026-06-30T02:14:05Z ERROR dp.aggregate tenant=lib-west event=budget_exhausted
  epsilon_spent=1.03 epsilon_ceiling=1.00 query=branch_checkout_counts
  msg="composition over 41 daily releases exceeded monthly ceiling; refusing to emit"

That last line is the honest failure — the pipeline refused to publish rather than silently overspend the budget. The other two forms are the dangerous ones, because they publish successfully and look fine.

Root Cause

Differential privacy gives a formal guarantee — the presence or absence of any one patron changes the output distribution by at most a factor of exp(epsilon) — but only if two numbers are correct: the global sensitivity of the query and the cumulative epsilon spent across all releases.

Unbounded sensitivity on sparse bins. For a plain count, one patron’s contribution is 1, so sensitivity is 1. But circulation analytics are almost never plain counts — they are sums (checkouts per branch, holds per format) where a single heavy borrower can contribute dozens. If you do not cap each patron’s contribution, the global sensitivity is unbounded, so the correct noise scale is unbounded, so no finite noise makes the release safe. Worse, teams often estimate sensitivity from the data itself; a temporal spike (a school assignment driving 400 checkouts of one title) inflates the estimate and either destroys utility or, if ignored, leaves the true sensitivity understated.

Budget spent without composition accounting. Each query that touches the raw data spends epsilon. The sequential composition theorem says these add up: 41 daily releases at epsilon 0.025 each is a real spend of ~1.0, and if you were treating 0.025 as “the budget” per release without tracking the running total, you have already blown a monthly ceiling of 1.0 without noticing. Composition must be tracked per tenant, not globally, or a busy consortium member silently borrows privacy budget from a quiet one.

Full materialization of the window. Pulling the entire patron_activity window into a DataFrame to aggregate it is what triggers the OOM kill during high-concurrency sync windows — the same materialization anti-pattern that shows up in large PII masking exports, applied here to the aggregation input.

Solution

Fix the two numbers, then stream the input. Three changes, in order: clip each patron’s contribution to bound sensitivity, aggregate in bounded chunks, then add Laplace noise calibrated to the clipped sensitivity — not the observed variance.

The Laplace mechanism is the right primitive for count and clipped-sum queries because it offers pure differential privacy and its scale depends only on sensitivity / epsilon:

python
import numpy as np


def laplace_mechanism(true_value: float, sensitivity: float, epsilon: float) -> float:
    """Add Laplace noise calibrated to query sensitivity and privacy budget.

    Args:
        true_value: The real query result (e.g. checkout count for a branch).
        sensitivity: L1 sensitivity of the query — for a sum clipped to C, this is C.
        epsilon: Privacy budget for this release (smaller = stronger privacy).
    Returns:
        The differentially private noisy result.
    """
    if epsilon <= 0:
        raise ValueError("epsilon must be positive")
    scale = sensitivity / epsilon
    return true_value + float(np.random.laplace(loc=0.0, scale=scale))

The key move is before, not inside, the noise call: bound each patron’s contribution so sensitivity is a known constant, and stream the events so nothing is materialized. The clip threshold comes from a historical 99th-percentile checkout rate, never from the current batch (which a spike would poison).

python
import logging
from collections import defaultdict
from typing import Iterator

logger = logging.getLogger("patron_dp_aggregate")


def aggregate_with_dp(
    events: Iterator[dict],
    epsilon: float,
    clip_threshold: int = 10,
    chunk_size: int = 1_000,
) -> dict[str, float]:
    """Stream circulation events into DP-noised per-branch counts.

    clip_threshold bounds each patron's contribution to any branch total, which
    fixes the query's global sensitivity at clip_threshold (not 1, and not
    unbounded). It is derived from a historical p99 checkout rate, so a temporal
    spike in the current batch cannot inflate it.
    """
    counts: dict[str, float] = defaultdict(float)
    chunk: list[dict] = []

    def drain(rows: list[dict]) -> None:
        for row in rows:
            branch = row.get("branch_code", "UNKNOWN")
            counts[branch] += min(row.get("checkout_count", 1), clip_threshold)

    for event in events:
        chunk.append(event)
        if len(chunk) >= chunk_size:
            drain(chunk)
            chunk.clear()
    drain(chunk)  # remaining tail

    noisy = {
        branch: laplace_mechanism(count, sensitivity=clip_threshold, epsilon=epsilon)
        for branch, count in counts.items()
    }
    logger.info(
        "dp_release branches=%d epsilon=%.4f sensitivity=%d noise_scale=%.2f",
        len(noisy), epsilon, clip_threshold, clip_threshold / epsilon,
    )
    return noisy

Before this fix a heavy borrower could shift a branch total by 40, so no finite noise scale protected them; after it, one patron moves any total by at most clip_threshold, and Lap(clip_threshold / epsilon) provably masks that shift. Rare-category cells (Form 1) are still handled, but their negative-noise artifacts need one more step — post-processing.

Applying DP to zero- and low-count categories produces negative counts, which are nonsensical for public reporting. Clamp them at the boundary:

python
def clamp_nonnegative(noisy_counts: dict[str, float]) -> dict[str, float]:
    """Clamp DP output to non-negative values.

    This is post-processing: a deterministic function of an already-privatized
    output. It consumes NO additional privacy budget and cannot weaken the
    guarantee — provided it acts on the noised output, never on the raw input.
    """
    return {k: max(0.0, v) for k, v in noisy_counts.items()}

For full context on where this aggregation stage sits relative to field-level tokenization and the export contract, see the parent PII Masking in Patron Data Exports cluster — noise injection is the row in its transform table reserved for aggregate demographic outputs.

Compliance or Privacy Impact

The reason the clamp is safe is worth stating precisely, because getting it backwards breaks the guarantee. Post-processing a DP output is free — you can clamp, round, or reformat the noised numbers as much as you like and the epsilon spend does not change. What you must never do is clamp or re-derive from the raw data after noising, because that reintroduces a data-dependent path that the privacy analysis did not account for.

Two boundary alignments keep the release defensible:

The epsilon budget is itself a retention-adjacent control: tokens and aggregates that persist indefinitely erode anonymity as releases accumulate, so the composition ceiling must be scoped to the same windows described in Data Retention Policies for Public Libraries.

Verification

Three properties must be proven by test, not assumed: sensitivity is bounded, the budget decrements correctly under composition, and the job streams instead of materializing.

python
import numpy as np
import pytest


def test_sensitivity_is_bounded_by_clip_threshold():
    """One heavy borrower may not move a branch total by more than clip_threshold."""
    base = [{"branch_code": "MAIN", "checkout_count": 1} for _ in range(50)]
    spiked = base + [{"branch_code": "MAIN", "checkout_count": 400}]
    np.random.seed(0)
    a = aggregate_with_dp(iter(base), epsilon=0.5, clip_threshold=10)["MAIN"]
    np.random.seed(0)
    b = aggregate_with_dp(iter(spiked), epsilon=0.5, clip_threshold=10)["MAIN"]
    # The extra patron's raw contribution is 400, but the clipped delta is <= 10.
    assert abs(b - a) <= 10 + 1e-9


def test_noise_scale_matches_sensitivity_over_epsilon():
    """Empirical std of the mechanism must match the Laplace scale sqrt(2)*b."""
    draws = [laplace_mechanism(0.0, sensitivity=10, epsilon=0.5) for _ in range(20_000)]
    expected_std = np.sqrt(2) * (10 / 0.5)
    assert np.std(draws) == pytest.approx(expected_std, rel=0.05)


def test_clamp_is_pure_postprocessing():
    """Clamping never raises the values and never touches the raw counts."""
    assert clamp_nonnegative({"RARE": -3.2, "MAIN": 118.0}) == {"RARE": 0.0, "MAIN": 118.0}

To confirm the OOM fix, profile peak resident memory while streaming a large window and assert it stays flat regardless of window size:

python
import tracemalloc


def peak_kib(events, **kw) -> float:
    tracemalloc.start()
    aggregate_with_dp(iter(events), epsilon=0.5, **kw)
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return peak / 1024

Peak memory should be governed by chunk_size and the number of distinct branches, not by the total event count — if it scales with the input size, an accidental list(events) has crept back in. In production, run the noise-scale check as a monitored assertion on each release: log raw_aggregate, noisy_aggregate, and noise_scale, and alert when the observed delta exceeds roughly 3× the scale, which flags a sensitivity or clipping regression before the numbers reach a dashboard.