Designing Zero-Trust Architecture for Library APIs
This page answers one narrow operational question: when you put a catalog and circulation sync worker behind continuous verification — mutual TLS, short-lived tokens, and per-record payload validation — why do long-running async batch jobs die mid-stream with 401 Unauthorized, why do workers get OOM-killed on the exact same feed they parsed fine last month, and how do you enforce zero-trust without watering either problem down. It sits under ILS Schema Translation Patterns, which defines the canonical envelope and mapping registry these workers move records through, and within the broader Core Architecture & Catalog Standards architecture. If you are hardening the credentials and payloads flowing between an integrated library system (ILS) and its downstream consumers, start here before you turn on enforcement in production.
Problem Framing
Two failure modes surface together the moment a perimeter-trusted worker is placed behind continuous verification. They look unrelated but share a small root-cause set.
1. Long batches fail mid-stream on token rotation. A worker that authenticated cleanly at job start dies forty minutes in, partway through a nightly feed, when the identity provider rotates the signing key:
$ journalctl -u catalog-sync --since "-1h" | grep -E "401|expired"
14:22:07 catalog-sync[4417] WARNING push_to_ils status=401 reason="token expired" record_id=b1993847
14:22:07 catalog-sync[4417] ERROR batch aborted after 6120/48000 records; last_commit=b1993846
The static service-account token the worker was initialized with expired against a rotation window it never checked, so every request after the boundary is rejected.
2. Workers are OOM-killed by the validation gate. A container that handled a 5,000-record test file dies against the full feed once strict payload validation is switched on:
$ dmesg | grep -i "killed process"
[ 8123.44] Out of memory: Killed process 4419 (catalog-sync) total-vm:6291456kB, anon-rss:5872140kB
Enforcing zero-trust means nothing enters the translation layer unvalidated — but a naive validator that materializes each MARCXML batch into a DOM to inspect it doubles peak memory over the old trusting parser, and a malformed authority record with an unbounded repeated field tips the worker over.
These are not two bugs to triage separately. They are two symptoms of one design gap: the worker treats authentication and payload trust as one-time facts established at startup, when zero-trust requires them to be re-established continuously and cheaply per request.
Root Cause
Token rotation is a lifecycle, not a constant. A worker initialized with a single bearer token assumes the token outlives the job. Under continuous verification the identity provider issues short-lived credentials (typically 15–60 minutes) and rotates the signing kid. Any request whose token crosses the exp boundary — or whose kid no longer matches a published key — is rejected with 401. Because a large ingest runs longer than one token lifetime, the failure is not intermittent; it is guaranteed the instant the job outlives its first token. Retrying with the same expired token just multiplies the rejections.
Strict validation without streaming materializes the whole payload. The trusting parser you are replacing likely read records lazily. A zero-trust gate that deserializes the entire batch to validate it — json.loads on a multi-hundred-megabyte body, or lxml.etree.fromstring building a full DOM — holds every record resident at once. Peak resident set scales with feed size, not with a bounded window, so the job that fit in memory at test scale exhausts the container at production scale. A single pathological record (an authority record with thousands of repeated 4xx see-also fields) can spike allocation on its own.
The shared fix is to make both trust decisions per request and bounded: refresh credentials reactively when a 401 proves the token stale, and validate payloads through a streaming parser that never holds more than one record plus a fixed buffer.
Solution
Enforce the three gates — transport, identity, payload — as cheap, repeatable checks rather than startup assumptions. The full mapping and quarantine context these workers feed into lives in ILS Schema Translation Patterns; this page covers only the verification layer in front of it.
Credential refresh middleware. Wrap ILS calls in a client that treats a 401 as a signal to rotate, not to fail. Pre-fetch a replacement token at 85% of its lifetime so the common path never blocks, and carry an idempotency key so a retry after refresh cannot double-write a circulation update or overwrite a bib record:
from __future__ import annotations
import threading
import time
import logging
import uuid
import httpx
logger = logging.getLogger("ils.zero_trust.auth")
class TokenProvider:
"""Fetches and caches a short-lived OAuth2 token, refreshing proactively.
Refreshes at 85% of token lifetime so long batches never cross an
expiry boundary mid-request. Thread-safe for a shared worker pool.
"""
def __init__(self, token_url: str, client_id: str, client_secret: str,
refresh_ratio: float = 0.85) -> None:
self._token_url = token_url
self._client_id = client_id
self._client_secret = client_secret
self._refresh_ratio = refresh_ratio
self._lock = threading.Lock()
self._token: str | None = None
self._refresh_after: float = 0.0
def _fetch(self) -> None:
resp = httpx.post(
self._token_url,
data={"grant_type": "client_credentials"},
auth=(self._client_id, self._client_secret),
timeout=10.0,
)
resp.raise_for_status()
body = resp.json()
lifetime = int(body["expires_in"])
self._token = body["access_token"]
self._refresh_after = time.monotonic() + lifetime * self._refresh_ratio
logger.info("token refreshed lifetime=%ss refresh_after=%.0fs",
lifetime, lifetime * self._refresh_ratio)
def token(self) -> str:
with self._lock:
if self._token is None or time.monotonic() >= self._refresh_after:
self._fetch()
return self._token # type: ignore[return-value]
def push_to_ils(client: httpx.Client, tokens: TokenProvider,
record_id: str, payload: dict) -> httpx.Response:
"""POST one record, refreshing the token once if the ILS rejects it.
The idempotency key makes the post-refresh retry a no-op if the first
attempt actually committed before the 401 was raised.
"""
idem_key = f"{record_id}:{uuid.uuid5(uuid.NAMESPACE_URL, record_id)}"
for attempt in (1, 2):
resp = client.post(
"/bibs",
json=payload,
headers={
"Authorization": f"Bearer {tokens.token()}",
"Idempotency-Key": idem_key,
"X-Request-ID": idem_key,
},
)
if resp.status_code != 401:
return resp
logger.warning("401 on record_id=%s attempt=%s; forcing token refresh",
record_id, attempt)
tokens._refresh_after = 0.0 # invalidate so next token() re-fetches
resp.raise_for_status()
return resp
The key discipline: the Idempotency-Key is derived deterministically from the record’s control number, so if the first attempt committed to the ILS before the 401 was raised, the retry after refresh collapses to a no-op instead of a duplicate write.
Bounded payload validation. Replace whole-batch deserialization with an incremental parser so the validation gate holds one record at a time — the same memory-bounded streaming discipline detailed in Optimizing pymarc Performance for Large Record Sets, applied here as a trust boundary rather than a throughput one. Enforce a hard size ceiling at the boundary and route anything oversized or malformed to a schema validation quarantine queue rather than into the worker:
from collections.abc import Iterator
from lxml import etree
MAX_RECORD_BYTES = 1 * 1024 * 1024 # reject any single record over 1 MiB
MARC_NS = "{http://www.loc.gov/MARC21/slim}"
def validated_records(source_path: str) -> Iterator[etree._Element]:
"""Stream MARCXML records one at a time, freeing each after yield.
iterparse never materializes the full document, so peak memory stays
flat regardless of feed size. Oversized records are rejected, not loaded.
"""
context = etree.iterparse(source_path, events=("end",),
tag=f"{MARC_NS}record", huge_tree=False)
for _event, record in context:
approx_bytes = len(etree.tostring(record))
if approx_bytes > MAX_RECORD_BYTES:
logger.warning("record over size ceiling bytes=%s; quarantining",
approx_bytes)
_clear(record)
continue # caller routes rejected records to quarantine
yield record
_clear(record) # release the element and its now-processed siblings
def _clear(record: etree._Element) -> None:
record.clear()
parent = record.getparent()
if parent is not None:
while record.getprevious() is not None:
del parent[0]
The _clear step is what keeps the streaming guarantee real: without deleting processed siblings, iterparse still accumulates the parent’s child list and quietly reintroduces the OOM.
Compliance or Privacy Impact
Tightening the verification layer generally shrinks the exposure surface, but two checkpoints must not be skipped. First, the X-Request-ID / idempotency key is derived from a bibliographic control number, never from a patron identifier — deriving it from a barcode or borrower ID would leak an identity-bearing value into API gateway logs and the audit trail. Any patron-adjacent field must already be masked before a record reaches this layer, following the boundary model in Data Privacy Boundaries in Library Systems and the masking rules in PII Masking in Patron Data Exports.
Second, the refresh middleware logs only the record ID, attempt number, and status — never the token, the Authorization header, or the record body — so a credential rotation can never spill a bearer secret or a patron record into a log aggregator. Records rejected at the size or schema gate are diverted to quarantine with their control number and rejection reason only, keeping the audit log complete without carrying the offending payload forward. Never disable mTLS or certificate validation to “unblock” a batch: even a temporary bypass widens the attack surface and can violate state-level and public-sector data-governance requirements, and it must never happen without a documented incident record.
Verification
Confirm each fix independently before enabling enforcement in production.
Bounded memory. Profile a worker against the full feed with tracemalloc and assert peak stays flat rather than scaling with record count:
import tracemalloc
tracemalloc.start()
count = sum(1 for _ in validated_records("nightly_full.xml"))
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
assert peak < 256 * 1024 * 1024, f"peak {peak / 1e6:.0f} MB exceeds worker budget"
assert count > 0
Survives token rotation. Drive push_to_ils against a mock ILS that returns 401 on the first call and 200 on the second, and assert exactly one committed write despite the refresh retry:
def test_refresh_on_401(monkeypatch, mock_ils):
mock_ils.queue_responses([401, 200]) # rotation boundary, then success
resp = push_to_ils(mock_ils.client, tokens, "b1993847", {"title": "…"})
assert resp.status_code == 200
assert mock_ils.committed_writes("b1993847") == 1 # idempotency held
Rejects malformed payloads at the gate. Feed the validator a record that exceeds the size ceiling and confirm it is skipped, not loaded, and that peak memory does not spike on the pathological input. A clean run leaves no oversized record in the worker and one entry in the quarantine sink.
If a completed batch still shows 401s in the logs, the proactive refresh ratio is not taking effect — verify the expires_in the IdP actually returns and that _refresh_after is being set, then reconcile X-Request-ID values across the gateway and worker logs to separate a genuine rotation gap from a clock-skew exp drift.
Related
- ILS Schema Translation Patterns — parent guide: the canonical envelope and mapping registry these verified requests feed into.
- Implementing Circuit Breakers for ILS API Timeouts — the sibling resilience pattern that isolates a degraded ILS from cascading into these workers.
- Schema Validation for Ingested Records — where records rejected at the payload gate go for quarantine and replay.
- Optimizing pymarc Performance for Large Record Sets — the streaming, memory-bounded parsing technique the payload gate reuses to stay flat under the full feed.
- Data Privacy Boundaries in Library Systems — the masking and audit boundary that keeps patron data out of tokens, logs, and canonical records.