Library Catalog Data Sync Pipelines

Handling HTTP 429 Retry-After Headers from ILS APIs

This page answers one narrow operational question: your poller against an ILS REST API — Innovative Sierra, Ex Libris Alma, or FOLIO — starts returning 429 Too Many Requests with a Retry-After header, but the client ignores that header and keeps firing at its fixed interval, so the server throttles it harder or drops it into an IP ban. It sits under ILS REST API Polling & Rate Limiting within the broader Catalog Ingestion & ILS Sync Pipelines architecture. The Retry-After header is the server telling you exactly when to resume; the fix is to read it, sleep precisely that long, and only fall back to jittered backoff when the header is absent. This is the header-honoring complement to the blind-backoff tuning covered in Configuring Exponential Backoff for Sierra API Calls.

Ignoring versus honoring a 429 Retry-After header A poller sends a GET request to an ILS REST API, which returns 429 Too Many Requests carrying a Retry-After of thirty seconds. Two branches follow. In the top, non-compliant branch the client ignores the header and retries immediately, and the server responds by throttling harder or IP-banning the client. In the bottom, compliant branch the client parses the header, waits exactly thirty seconds, and its retry then succeeds with a 200 OK. IGNORED · non-compliant HONORED · compliant poller GET /bibs ILS API 429 Too Many Requests Retry-After: 30 ignore header retry immediately throttled harder or IP-banned wait exactly 30s parse Retry-After retry → 200 OK quota restored

Problem Framing

The symptom is a poller that will not recover. A batch of requests trips the rate limit, the ILS returns 429 with a Retry-After header naming the exact moment quota resumes, and the client — which never reads the header — retries on its next fixed tick regardless. Each premature retry is another request against an empty bucket, and many gateways escalate: a repeated 429 within the cool-down window is treated as abusive traffic and answered with a longer penalty or a temporary IP block. The retry log makes the pattern unmistakable — a Retry-After value present on every response, and a waited value that is either zero or the client’s fixed interval, never the header:

text
$ jq -c 'select(.status_code == 429) | {ts, endpoint, retry_after, waited}' sync.log
{"ts":"2026-07-15T02:14:01Z","endpoint":"/bibs","retry_after":"30","waited":0}
{"ts":"2026-07-15T02:14:06Z","endpoint":"/bibs","retry_after":"30","waited":0}
{"ts":"2026-07-15T02:14:11Z","endpoint":"/bibs","retry_after":"60","waited":0}
{"ts":"2026-07-15T02:14:16Z","endpoint":"/bibs","retry_after":"120","waited":0}
{"ts":"2026-07-15T02:14:21Z","endpoint":"/bibs","retry_after":"Wed, 15 Jul 2026 02:20:00 GMT","waited":0}

Two things stand out. The server keeps raising retry_after — 30, then 60, then 120 seconds — because the client keeps ignoring it, and the last row switches from an integer to an HTTP-date. A client that only knows how to parse one of those two forms silently mishandles the other, which is how “we honor Retry-After” becomes “we honor it except when Alma sends a date.”

Root Cause

Fixed-interval polling has no feedback loop. The poller decides when to fire based purely on its own clock, so a 429 carries no information back into the scheduler — the next request goes out on schedule whether the quota reset one second ago or is two minutes away. The server, meanwhile, has already answered the timing question precisely: RFC 9110 defines Retry-After in two interchangeable forms, and different ILS platforms use different ones. Sierra and FOLIO typically emit delta-seconds (a non-negative integer count of seconds to wait); Alma’s gateway may emit an HTTP-date (an RFC 1123 timestamp naming the wall-clock instant to resume). A client that assumes one form treats the other as garbage and falls back to guessing.

The two grammars, and which platforms emit which, are worth pinning down before writing any parser:

Form Grammar Example value Seen on
delta-seconds non-negative integer seconds to wait Retry-After: 30 Sierra, FOLIO
HTTP-date RFC 1123 timestamp of the resume instant Retry-After: Wed, 15 Jul 2026 02:20:00 GMT Alma gateway

A parser that handles only the first column returns nothing useful for an Alma date and falls through to a guess; a parser that assumes the second crashes on Sierra’s bare integer. Both forms must resolve to the same thing — a non-negative number of seconds from now.

The deeper error is treating this as the same problem the sibling page on exponential backoff solves. Blind exponential backoff is the right tool when the server gives you no wait signal — it decorrelates a worker fleet so retries stop colliding. But when a Retry-After header is present, backoff is simultaneously too slow and non-compliant: too slow because a doubling delay routinely overshoots a 30-second reset and idles the pipeline for minutes, and non-compliant because the server explicitly told you the answer and you overrode it with a guess. Worse, backing off less than the header asks is what triggers the escalating penalty in the log above. The header is authoritative; backoff is only the fallback for when it is missing.

Solution

Honor the header first, back off second. Parse Retry-After in both its integer and HTTP-date forms, sleep for exactly the interval it names (clamped to a sane ceiling so a hostile or buggy header cannot pin a worker for an hour), and reserve jittered exponential backoff for the case where the header is absent. The parser is the load-bearing part — it must accept both grammars and reject anything it cannot interpret rather than silently returning zero.

python
from __future__ import annotations

import asyncio
import logging
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

import httpx

logger = logging.getLogger("ils.retry_after")


class RetryAfterError(RuntimeError):
    """Raised when retries are exhausted against a persistently throttled ILS."""


def parse_retry_after(raw: str | None, *, now: datetime | None = None) -> float | None:
    """Parse an RFC 9110 Retry-After header into a delay in seconds.

    Accepts both forms: an integer count of delta-seconds, or an HTTP-date.
    Returns None when the header is absent or unparseable so the caller can
    fall back to backoff; never returns a negative delay.
    """
    if raw is None:
        return None
    raw = raw.strip()
    if raw.isdigit():
        return float(raw)
    try:
        when = parsedate_to_datetime(raw)
    except (TypeError, ValueError):
        logger.warning("retry_after_unparseable", extra={"raw": raw})
        return None
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    now = now or datetime.now(timezone.utc)
    return max(0.0, (when - now).total_seconds())

With the parser in place, the retry handler makes the header the primary signal and backoff the fallback. The ceiling clamp bounds any single wait; the small positive jitter added to the exact interval keeps a fleet of workers from all waking on the identical reset boundary even when they honor the same header.

python
async def fetch_honoring_retry_after(
    client: httpx.AsyncClient,
    url: str,
    *,
    headers: dict[str, str],
    max_attempts: int = 5,
    base: float = 1.0,
    ceiling: float = 300.0,
) -> httpx.Response:
    """GET a throttled ILS endpoint, honoring Retry-After when present.

    Waits exactly the header interval (plus small jitter, clamped to `ceiling`)
    on a 429; falls back to full-jitter exponential backoff when the header is
    absent. Retries only 429 and 5xx; raises RetryAfterError when exhausted.
    """
    for attempt in range(max_attempts):
        response = await client.get(url, headers=headers)
        if response.status_code != 429 and response.status_code < 500:
            response.raise_for_status()
            return response

        retry_after = parse_retry_after(response.headers.get("Retry-After"))
        if retry_after is not None:
            delay = min(retry_after, ceiling) + random.uniform(0, 1.0)
            source = "retry_after"
        else:
            # No authoritative signal: full-jitter backoff as the fallback.
            delay = random.uniform(0, min(ceiling, base * (2 ** attempt)))
            source = "backoff"

        if attempt == max_attempts - 1:
            raise RetryAfterError(
                f"exhausted {max_attempts} attempts for {response.request.url.path}"
            )
        logger.warning(
            "throttled_retry",
            extra={
                "attempt": attempt,
                "status_code": response.status_code,
                "endpoint": response.request.url.path,
                "delay": round(delay, 3),
                "wait_source": source,
            },
        )
        await asyncio.sleep(delay)

    raise RetryAfterError(f"unreachable retry loop for {url}")

The behavioural change is the addition of the feedback loop. Before, the delay came from a fixed clock that no response could influence; after, a 429 with a Retry-After of 30 produces a wait of almost exactly 30 seconds, an HTTP-date resolves to the seconds remaining until that instant, and only a 429 with no header falls through to the decorrelated backoff the exponential-backoff guide tunes. The two strategies are complementary, not competing: this handler honors the server when it speaks and guesses well only when it stays silent. For the stateful polling loop this retry policy plugs into, see the parent guide on ILS REST API Polling & Rate Limiting.

Compliance or Privacy Impact

Retry logic looks purely operational, but the retry log is a data-egress surface, and a throttle-recovery path is a tempting place to get sloppy. The extra={...} payloads above are deliberately narrow — attempt number, status code, endpoint path, delay, and the wait source — and that discipline is not incidental. The request headers passed into the handler carry the ILS bearer token or API key, and a throttled patron-lookup call carries a barcode or patron identifier in its query string. Never let either into a log line: logging response.request.headers to “debug the 429” writes the integration credential into a file with a longer retention window than the token itself, and logging the full request URL of a /patrons call copies patron PII into an unaudited trace. Emit only masked metadata, exactly as the export-masking contract requires for every field that leaves the trusted zone — the same field-level discipline described in PII Masking in Patron Data Exports.

The RetryAfterError boundary matters here too. When retries are exhausted, the exception should carry the endpoint path and attempt count for triage but never the payload or credentials — an error surfaced to an alerting channel or a crash reporter is one more place raw patron data can leak. Route the un-fetched work item to a durable queue for later replay, and treat anything buffered there as inheriting the same masking and retention obligations as the live feed rather than as an exempt scratch space.

Verification

Confirm the fix along two axes: that both header forms parse to the right delay, and that the handler waits the header interval rather than its own guess. Test the parser directly with a frozen clock so the HTTP-date branch is deterministic.

python
from datetime import datetime, timezone


def test_parse_retry_after_both_forms() -> None:
    now = datetime(2026, 7, 15, 2, 14, 0, tzinfo=timezone.utc)

    # delta-seconds form
    assert parse_retry_after("30", now=now) == 30.0

    # HTTP-date form resolves to seconds until that instant
    assert parse_retry_after("Wed, 15 Jul 2026 02:16:00 GMT", now=now) == 120.0

    # a date already in the past clamps to zero, never negative
    assert parse_retry_after("Wed, 15 Jul 2026 02:13:00 GMT", now=now) == 0.0

    # absent or garbage falls through to backoff (None), not a silent zero
    assert parse_retry_after(None) is None
    assert parse_retry_after("soon") is None

Then assert end to end that a mocked 429-then-200 sequence honors the header. Capturing the slept delay confirms the wait came from Retry-After and not the fixed interval:

python
import httpx
import pytest
import respx


@pytest.mark.asyncio
@respx.mock
async def test_waits_for_retry_after_then_succeeds(monkeypatch) -> None:
    slept: list[float] = []

    async def _record(delay: float) -> None:
        slept.append(delay)

    monkeypatch.setattr(asyncio, "sleep", _record)
    respx.get("https://alma.example.edu/almaws/v1/bibs").mock(
        side_effect=[
            httpx.Response(429, headers={"Retry-After": "30"}),
            httpx.Response(200, json={"bib": []}),
        ]
    )
    async with httpx.AsyncClient() as client:
        resp = await fetch_honoring_retry_after(
            client, "https://alma.example.edu/almaws/v1/bibs", headers={}
        )

    assert resp.status_code == 200
    assert 30.0 <= slept[0] <= 31.0  # honored the header, not a fixed guess

After deploying, re-run the jq filter from the Problem Framing section: the waited field should now track retry_after on every 429, and the escalating penalty ladder should flatten because the server no longer sees premature retries. A wait_source of backoff in the logs is your signal that a given endpoint is not sending the header — expected for Sierra under some conditions — and confirms the fallback is doing its job rather than the primary path silently failing.