Implementing Circuit Breakers for ILS API Timeouts

This page answers one narrow operational failure: your nightly catalog sync workers hang and then get OOM-killed whenever a vendor ILS endpoint (Sierra, Alma, Koha, FOLIO) slows down under peak circulation load. It sits beneath ILS Schema Translation Patterns and the broader Core Architecture & Catalog Standards architecture, and it covers the specific fix: wrapping every outbound ILS call in a deterministic circuit breaker so one degraded endpoint isolates cleanly instead of taking the whole pipeline down. If you are here because a worker’s resident set climbs steadily while an ILS API is slow — and it never recovers on its own — this is the exact symptom the breaker resolves.

Problem Framing

The symptom is unmistakable once you have seen it. An ILS endpoint degrades — a backend database lock, an index rebuild, a maintenance window — and instead of failing fast, your translation workers pile up. Connection pool slots fill with sockets stuck in a half-open TCP state, new requests queue behind them, and every retry loop allocates another in-flight response buffer that never gets freed. Resident set size climbs monotonically until the kernel’s OOM killer reaps the worker.

The diagnostic signature in the logs is a storm of ReadTimeout interleaved with rising retry counts, all against a single host:

2026-07-02T02:14:07Z WARN  ils.sync retry=1 host=sierra.example.org op=fetch_bib err=ReadTimeout
2026-07-02T02:14:19Z WARN  ils.sync retry=2 host=sierra.example.org op=fetch_bib err=ReadTimeout
2026-07-02T02:14:41Z WARN  ils.sync retry=3 host=sierra.example.org op=fetch_bib err=ReadTimeout
2026-07-02T02:14:41Z ERROR ils.sync pool_exhausted host=sierra.example.org active=100 idle=0
2026-07-02T02:15:03Z FATAL ils.sync worker rss=1974MB oom_killed

A quick way to confirm the failure domain is to watch pooled connections and worker RSS side by side while the batch runs:

watch -n 2 'ss -tan state established "( dport = :443 )" | wc -l; \
  ps -o rss= -p "$(pgrep -f ils_sync_worker)" | awk "{print \$1/1024 \"MB\"}"'

If the established-connection count pins at the pool maximum while RSS only ever rises, the workers are not shedding load — they are absorbing a stalled upstream, which is precisely what a circuit breaker exists to prevent.

Root Cause

Three compounding causes turn a transient slowdown into a fatal one.

Unbounded retry amplification. A naive retry wrapper treats every timeout as retryable, so a slow endpoint receives more traffic exactly when it can least handle it. Each attempt holds a connection and a partially-read response buffer for the full read_timeout duration; with no ceiling on concurrent retries, memory and pool pressure grow without bound.

Silent success masquerading as failure. Many legacy ILS gateways return HTTP 200 with a truncated or malformed body when a backend lock occurs — Sierra’s {"code": "-1"} error envelope is the canonical example. Code that only counts 5xx responses never registers these as failures, so the breaker (or retry policy) never trips, and the malformed payloads flow downstream to poison schema translation. Failure detection must inspect payload integrity, not just the status line.

No failure isolation. Without a breaker, there is no shared circuit state across workers. Each one independently rediscovers that the endpoint is down, and each one pays the full timeout cost per record. The breaker centralizes that knowledge: once the failure rate crosses a threshold, every subsequent call fast-fails locally without touching the network, which is what lets the pool drain and RSS flatten.

Solution

Wrap every outbound ILS call in a state machine with vendor-specific thresholds, and make failure detection payload-aware. pybreaker provides synchronous circuit state tracking; pair it with per-request httpx timeouts so a single call can never block longer than the read budget. The full canonical-record contract these calls feed is defined in ILS Schema Translation Patterns; here we only guard the transport.

Configure the breaker against timeouts, not just status codes. For a typical ILS REST endpoint, trip after roughly a 40% failure rate in a 60-second window, keep connect_timeout under 3 seconds and read_timeout under 10, and route the silent-error envelope through the same failure path as a hard timeout:

import logging

import httpx
import pybreaker

logger = logging.getLogger("ils.circuit_breaker")


class ILSSilentError(RuntimeError):
    """ILS returned HTTP 200 with a malformed or error-envelope body."""


# Trip after 5 failures inside the reset window; probe again after 60s.
ils_breaker = pybreaker.CircuitBreaker(
    fail_max=5,
    reset_timeout=60,
    name="ils_api",
    exclude=[httpx.HTTPStatusError],  # 4xx client errors are not breaker failures
)

# Fast connect budget, bounded read budget: no call blocks past ~10s.
_TIMEOUT = httpx.Timeout(connect=3.0, read=10.0, write=5.0, pool=3.0)


@ils_breaker
def fetch_bib_record(control_number: str, base_url: str, token: str) -> dict:
    """Fetch one bib record through the breaker. Raises on timeout or silent error."""
    with httpx.Client(timeout=_TIMEOUT) as client:
        response = client.get(
            f"{base_url}/bibs/{control_number}",
            headers={"Authorization": f"Bearer {token}"},
        )
        response.raise_for_status()
        data = response.json()

        # Sierra's silent error envelope: HTTP 200 body of {"code": "-1"}.
        if list(data.keys()) == ["code"] and data.get("code") == "-1":
            logger.warning("ils silent error control_number=%s", control_number)
            raise ILSSilentError(control_number)  # counts toward fail_max

        return data

Share the circuit across every worker process. The breaker above is in-process: it only isolates the worker that owns it. With a pool of sync workers, each one independently rediscovers the outage and pays the full timeout budget before its own circuit trips — the very “no failure isolation” cause from the previous section, reintroduced. Back the breaker with CircuitRedisStorage so all workers read and write one shared circuit state; the first worker to trip the endpoint fast-fails every other worker’s calls immediately, which is what actually lets the connection pool drain fleet-wide:

import httpx
import pybreaker
import redis

_redis = redis.Redis(host="localhost", port=6379, db=0)

# Same breaker, shared state: ONE circuit across every worker process.
ils_breaker = pybreaker.CircuitBreaker(
    fail_max=5,
    reset_timeout=60,
    name="ils_api",
    state_storage=pybreaker.CircuitRedisStorage(
        pybreaker.STATE_CLOSED, _redis, namespace="ils_api"
    ),
    exclude=[httpx.HTTPStatusError],
)

Keep the storage namespace endpoint-specific (one circuit per ILS host), so a degraded Sierra instance never fast-fails healthy Alma or FOLIO calls sharing the same worker pool.

Fall back to streaming, never to buffering. When the breaker is OPEN and a fallback path runs, it must not materialize whole record sets in memory — that reintroduces the exact OOM you are fixing. Parse MARCXML with lxml.etree.iterparse (or JSON with orjson chunked decoding) and feed a bounded asyncio.Queue so backpressure holds resident set size flat. The same streaming discipline is detailed in Optimizing pymarc Performance for Large Record Sets:

import asyncio

WORK_QUEUE: asyncio.Queue[dict] = asyncio.Queue(maxsize=50)  # bounded: backpressure


async def drain_records(source, sink) -> None:
    """Move records through a bounded queue so a slow sink cannot inflate RSS."""
    async for record in source:              # generator, not a materialized list
        await WORK_QUEUE.put(record)         # blocks when full -> upstream slows
        await sink(await WORK_QUEUE.get())
        WORK_QUEUE.task_done()

Probe carefully during recovery. After reset_timeout, the breaker enters HALF_OPEN and allows a single probe. Limit that probe to one dedicated executor so it cannot restart pool starvation, and pair the breaker with jittered exponential backoff on the retryable calls it still permits — the same backoff curve used in Configuring Exponential Backoff for Sierra API Calls, governed by the limits in ILS REST API Polling & Rate Limiting.

When the breaker does trip, follow a deterministic recovery sequence rather than force-killing work:

Halt ingestion. Pause new batch submissions and mark the endpoint OPEN.
Drain in-flight work. Let existing workers finish their bounded generators; do not terminate active translation tasks.
Emit telemetry. Fire a CIRCUIT_OPEN event carrying the endpoint, failure rate, and last successful checkpoint ID.
Wait the recovery window. Hold OPEN for reset_timeout before allowing a probe.
Probe once. Issue a single lightweight health check (GET /status) on the dedicated executor.
Transition. On success, reset counters and go CLOSED; on failure, return to OPEN, double the window up to a 10-minute cap, and escalate.

For safe rollback, keep a persistent snapshot of the last successful checkpoint in a small key-value store (Redis or SQLite). If the endpoint stays OPEN long enough to indicate vendor-side degradation, revert to that checkpoint, drop pending transaction logs, and serve a read-only catalog cache. Never reconcile partial writes without idempotency verification — which is where the compliance impact begins.

Compliance or Privacy Impact

The dangerous edge case for circulation data is a partial timeout cascade: the ILS backend commits a transaction but the gateway drops the TCP FIN, so httpx raises ReadTimeout even though the circulation update already happened. If the breaker’s fallback simply retries the write, you double-count a checkout or a fine — a data-integrity and audit-accuracy problem, not just a performance one.

The fix is to make every state-changing call (POST/PUT) idempotent on a stable key. Carry an idempotency key derived from the record’s control number so a retried write updates one row instead of creating a duplicate, exactly as the write-back path in the parent ILS Schema Translation Patterns requires. Log the gateway’s X-Request-ID on both send and timeout so a reconciliation job can prove whether the original write landed.

Two privacy constraints apply to the breaker’s own telemetry. First, CIRCUIT_OPEN events and retry logs must carry only endpoint metadata, correlation IDs, and failure reasons — never raw bibliographic bodies or patron-adjacent fields, per the boundary rules in Data Privacy Boundaries in Library Systems. Second, the read-only cache you fail over to must honor the same retention and access controls as the live catalog; a fallback store is not an exemption. The credential and transport hardening that keeps those probes verified is covered in Designing Zero-Trust Architecture for Library APIs.

Verification

Confirm three things: the breaker trips, memory stays flat under a stalled upstream, and idempotent retries do not duplicate writes.

Assert the breaker opens. Drive the wrapped call against a stub that always times out and assert it stops calling the network after fail_max and raises CircuitBreakerError instead:

import pybreaker
import pytest


def test_breaker_opens_after_threshold():
    breaker = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=60, name="test")
    calls = {"n": 0}

    @breaker
    def always_times_out() -> None:
        calls["n"] += 1
        raise TimeoutError("upstream stalled")

    for _ in range(3):
        with pytest.raises(TimeoutError):
            always_times_out()

    # Circuit is now OPEN: further calls fast-fail WITHOUT invoking the target.
    with pytest.raises(pybreaker.CircuitBreakerError):
        always_times_out()
    assert calls["n"] == 3  # no 4th network call was made

Prove RSS stays flat. Run a representative stalled batch under tracemalloc and assert the high-water mark stays inside your per-worker budget (e.g. 256 MB). A flat peak across 10k simulated timeouts confirms the fallback streams instead of buffering:

import tracemalloc

tracemalloc.start()
run_batch_against_stalled_ils(record_count=10_000)  # breaker trips early
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
assert peak < 256 * 1024 * 1024, f"RSS peak {peak / 1e6:.0f}MB exceeds budget"

Test idempotency. Replay the same write twice through the breaker against a mock ILS and assert one create and one update keyed on the control number, never two creates — the same non-duplicating guarantee the schema validation quarantine queue relies on when it re-runs quarantined records.

For very large migrations, split ingress, breaker-guarded reads, and write-back into independent tasks so a tripped breaker on one endpoint never stalls the others, following the distributed pattern in Async Batch Processing for Catalog Updates.

ILS Schema Translation Patterns — the parent translation layer whose write-back these breakers protect.
Designing Zero-Trust Architecture for Library APIs — the sibling deep dive on credential rotation and transport hardening.
Configuring Exponential Backoff for Sierra API Calls — the jittered backoff curve the breaker’s retryable calls should use.
ILS REST API Polling & Rate Limiting — the rate limits that govern probe and retry pacing.