Library Catalog Data Sync Pipelines

Configuring Exponential Backoff for Sierra API Calls

This page answers one narrow operational question: your poller against the Innovative Sierra REST API keeps getting throttled into a widening cascade of 429 Too Many Requests responses, and adding a longer fixed sleep only made throughput worse. It sits under ILS REST API Polling & Rate Limiting within the broader Catalog Ingestion & ILS Sync Pipelines architecture, and it covers the exact backoff parameters — base delay, cap, and jitter distribution — that keep Sierra’s per-key token-bucket throttle from tripping under variable load. The fix is not “retry less”; it is to make retries decorrelated so that a fleet of workers stops re-colliding on every rate-limit reset boundary.

Fixed-interval retries versus full-jitter exponential backoff Two stacked timelines contrast retry strategies for three Sierra workers. In the top track, all three workers sleep a fixed five seconds and retry in the same instant on every token-bucket reset boundary, so each boundary is a stacked column of 429 collisions that empties the refilled quota immediately. In the bottom track, full-jitter backoff draws each delay from a uniform window that doubles each attempt — U(0,1s), U(0,2s), U(0,4s), U(0,8s), capped at 30 seconds — scattering the three workers so they no longer collide; by the fourth, widened window their requests succeed with 200 OK. A circuit-breaker state chip shows the breaker staying closed while retries flow, then flipping to open after N consecutive failures to halt calls and drain pending payloads to a durable queue. RETRY DISPERSION · FIXED INTERVAL vs FULL JITTER 429 collision dispersed retry 200 OK ①  FIXED sleep(5s) — synchronized wakeups W1 W2 W3 429 429 429 429 t₀ +5s +10s +15s time → Every worker wakes on the same reset boundary → the refilled bucket drains instantly → the 429 storm repeats. ②  FULL-JITTER backoff — decorrelated delays attempt 1 U(0,1s) attempt 2 U(0,2s) attempt 3 U(0,4s) attempt 4 — succeeds U(0,8s) · cap 30s time → Each delay is drawn from a window that doubles per attempt, so the three workers spread out and stop re-colliding. breaker CLOSED · retries flow N fails breaker OPEN · halt calls, drain to durable queue Persistent failure past the retry budget trips the circuit breaker instead of retrying forever.

Problem Framing

The symptom is a self-reinforcing throttle. A batch of Sierra requests trips the rate limit, every worker retries on a fixed schedule, those retries land together, and the endpoint returns 429 again — the retries themselves are now the load. In the structured retry log the signature is unmistakable: escalating attempt counts clustered on the same endpoint, all within a narrow time band.

bash
jq 'select(.status_code == 429 and .attempt > 3)' sync.log

If that filter returns rows, you have a retry storm rather than an isolated throttle. Sierra makes this easy to fall into because its token-bucket rate limiting operates independently per API key and enforces strict concurrency caps on the /bibs, /items, and /patrons endpoints. A naive time.sleep(5) between attempts does nothing to spread simultaneous retries across a distributed worker pool; it just synchronizes them at a coarser interval.

Root Cause

Fixed-interval retries fail for three compounding reasons specific to Sierra:

  1. Synchronized wakeups. When N workers all back off by the same amount, they retry at the same instant. The reset boundary of the token bucket becomes a thundering-herd trigger — quota refills, every worker fires, the bucket empties immediately, and everyone is throttled again.
  2. No decorrelation between attempts. Plain exponential backoff (base * 2 ** attempt) grows the mean delay but, without jitter, keeps all workers perfectly in phase. The delay gets longer; the collisions do not stop.
  3. Missing authoritative wait signal. Sierra does not consistently return a Retry-After header, so clients that rely on it fall back to guesswork. The quota state has to be reconstructed from X-RateLimit-Remaining and X-RateLimit-Reset instead, and a client that ignores those headers keeps probing a bucket it already knows is empty.

The cure for the first two is full-jitter backoff: draw each delay from a uniform random window that grows exponentially, so two workers almost never wake at the same moment. The cure for the third is to read the quota headers and, when remaining quota is near zero, sleep to the reset boundary instead of retrying blindly.

Solution

The baseline delay formula is delay = min(base * (2 ** attempt) + jitter, max_delay). For Sierra-scale library automation, a base of 1.0 s with a max_delay cap of 30–60 s is a reliable envelope. Full jitter replaces the additive term entirely — each delay is drawn from U(0, base * 2 ** attempt), capped — which is what actually decorrelates a worker fleet. The implementation below wraps Sierra fetches in that policy and also honors the quota headers when they are present. For the full stateful polling loop this retry policy plugs into, see the parent guide on ILS REST API Polling & Rate Limiting.

python
import asyncio
import logging
import random

import httpx

logger = logging.getLogger("ils.sierra.backoff")

SIERRA_BASE = "https://sierra.example.edu/iii/sierra-api/v6"


async def fetch_with_backoff(
    client: httpx.AsyncClient,
    path: str,
    headers: dict[str, str],
    *,
    base: float = 1.0,
    max_delay: float = 30.0,
    max_attempts: int = 5,
) -> httpx.Response:
    """Async Sierra fetch with full-jitter exponential backoff.

    Retries only on 429/5xx, decorrelating attempts across workers, and
    yields to X-RateLimit-Reset when the advertised quota is nearly gone.
    """
    for attempt in range(max_attempts):
        response = await client.get(f"{SIERRA_BASE}{path}", headers=headers)

        # Proactively honor near-exhausted quota before it becomes a 429.
        remaining = response.headers.get("X-RateLimit-Remaining")
        if remaining is not None and int(remaining) < 5:
            reset = float(response.headers.get("X-RateLimit-Reset", 10))
            await asyncio.sleep(reset + random.uniform(0, 2))

        if response.status_code == 429 or response.status_code >= 500:
            if attempt == max_attempts - 1:
                response.raise_for_status()
            # Full jitter: delay drawn from [0, base * 2**attempt], capped.
            cap = min(max_delay, base * (2 ** attempt))
            delay = random.uniform(0, cap)
            logger.warning(
                "sierra_retry",
                extra={"attempt": attempt, "delay": round(delay, 3),
                       "status_code": response.status_code, "endpoint": path,
                       "rate_limit_remaining": remaining},
            )
            await asyncio.sleep(delay)
            continue

        response.raise_for_status()
        return response

    raise RuntimeError(f"All {max_attempts} attempts exhausted for {path}")

Two Sierra-specific edge cases defeat a generic retry policy, so the fix has to account for them explicitly.

Silent error envelopes. Sierra v6 can return 200 OK with malformed JSON or an empty {"code":"-1"} body when the underlying Oracle middleware hiccups — a transient failure disguised as success. Retry logic must validate the payload shape before treating a request as done, or the poison response advances the cursor over records that were never delivered. Normalize this quirk once at the boundary through the ILS Schema Translation Patterns layer so the rest of the pipeline only ever sees the canonical model.

python
def is_valid_sierra_response(data: dict) -> bool:
    """Reject Sierra's silent error envelope disguised as a 200 OK."""
    return not (len(data) == 1 and data.get("code") == "-1")

Session-token expiry mid-batch. Sierra session tokens typically expire after 60 minutes (the exact window is site-configurable). Reacting to a 401 Unauthorized in the middle of a backoff queue compounds retry latency with re-authentication latency. Refresh proactively at 80% of token lifetime in a background task, and when a runtime 401 does slip through, bypass the standard backoff queue and negotiate a new session immediately rather than retrying the doomed request. Scope that integration key as narrowly as the vendor allows — a polling key that can also mutate patron records widens the blast radius of a leak far beyond the catalog, as developed in Designing Zero-Trust Architecture for Library APIs.

Finally, wrap the whole thing in a circuit breaker so a persistently degraded endpoint gets time to recover instead of absorbing retries forever. Trip after a configurable count of consecutive failures (for example, 10); when open, halt all outbound Sierra calls, persist pending payloads to a durable local queue, emit a critical alert carrying the triggering correlation_id, and require a health-check pass before closing again. The breaker mechanics themselves — the closed/open/half-open transitions — are covered in Implementing Circuit Breakers for ILS API Timeouts.

Compliance or Privacy Impact

Backoff tuning looks purely operational, but two of its side-effects touch regulated data directly.

First, retries create duplicate-write risk on mutating calls. For POST and PATCH operations against /patrons or /bibs, a retry that fires after the original request actually succeeded (but whose response was lost to a timeout) can create a second record. Derive a client-side idempotency key from a hash of the request payload plus the record control number, so a replayed write during a backoff window collapses to a no-op instead of forking a patron identity. This is the same idempotency discipline the broker layer applies in Async Batch Processing for Catalog Updates.

Second, the durable queue behind the circuit breaker holds real payloads. When the breaker trips and pending MARCXML is persisted to SQLite or Redis, those records can carry patron-linked data in local 9xx fields, circulation notes, or hold-shelf slips. Anything paused there inherits the same retention and masking obligations as the live feed; route it through the controls in the Patron Validation & Privacy Data Routing domain rather than treating a retry buffer as exempt. Retry logs are the other leak surface: emit masked request metadata (endpoint, status, quota consumed, correlation_id) but never raw response bodies, so a backoff trace cannot become an unaudited copy of patron data. Records that fail validation on retry belong in the schema validation quarantine queue, not silently dropped.

Verification

Confirm the fix along three axes: that retries decorrelate, that quota headers are honored, and that the change did not open a duplicate-write path.

Reproduce the throttle deterministically. Never tune against the live vendor — mock the transport so a 429 is reproducible and you are not burning real quota. Assert that the delays actually spread rather than stacking on a fixed interval:

python
import httpx
import pytest
import respx


@pytest.mark.asyncio
@respx.mock
async def test_backoff_retries_then_succeeds() -> None:
    route = respx.get(f"{SIERRA_BASE}/bibs").mock(
        side_effect=[
            httpx.Response(429, headers={"X-RateLimit-Remaining": "0",
                                         "X-RateLimit-Reset": "0"}),
            httpx.Response(429),
            httpx.Response(200, json={"entries": []}),
        ]
    )
    async with httpx.AsyncClient() as client:
        resp = await fetch_with_backoff(client, "/bibs", headers={}, base=0.01)

    assert resp.status_code == 200
    assert route.call_count == 3  # two throttles absorbed, third succeeds

Watch the retry log flatten. After deploying, the earlier jq filter should stop returning clustered high-attempt rows — decorrelated retries no longer pile up past attempt 3 on the same endpoint. Cross-reference correlation_id values against database transaction logs to confirm idempotency keys prevented any duplicate catalog updates during a backoff window.

Bound retry-queue memory. Under sustained throttling, an unbounded retry queue can accumulate large MARCXML payloads in memory when tasks are not offloaded to the durable store. Snapshot allocations with Python’s tracemalloc across a high-concurrency backoff period and diff two snapshots; a monotonically growing MARCXML allocation is the signal that payloads are being held in the retry path instead of persisted behind the circuit breaker.

Roll the parameters out safely: keep base, max_delay, jitter, and concurrency_limit in a versioned config, deploy new values to a single worker node, validate against a staging Sierra tenant for a full quota cycle, and promote only after telemetry stays stable. To recover a pipeline already in a stall, drain the affected workers, clear local rate-limit caches, force a fresh token, and re-engage gradually with an elevated base (e.g. 2.0 s) and reduced concurrency until X-RateLimit-Remaining holds above 30% and the 429 rate drops below 1%.