ILS REST API Polling & Rate Limiting

Operating within the broader Catalog Ingestion & ILS Sync Pipelines architecture, this guide covers how to pull incremental catalog and circulation changes from a live Integrated Library System (ILS) over its REST API without exhausting the vendor’s request quota or drifting out of sync. Library technical staff hit this problem the moment a nightly SFTP drop is no longer fresh enough — self-check availability, hold queues, and item status need near-real-time reconciliation, but the same vendor endpoints that expose that data enforce strict per-key rate limits and return partial, cursor-paged result sets. The failure modes are specific: fixed-interval cron jobs either poll too aggressively and get throttled into a retry storm, or too slowly and let the discovery layer fall behind the shelf. A production poller must be stateful and adaptive — tracking incremental cursors, honoring rate-limit headers, and pacing itself against observed quota consumption rather than a hard-coded sleep.

This page specifies the endpoint contract most ILS platforms expose, walks through a runnable adaptive polling loop, and details the compliance checkpoints and quarantine patterns that keep a rate-limited feed both safe and durable.

Specification & Contract

Most modern ILS REST APIs — Ex Libris Alma, Innovative Sierra, FOLIO, and Koha’s REST layer — share a common incremental-read contract even though their URL shapes differ. The pipeline never fetches the full corpus on each cycle; it requests only records changed since a persisted watermark, then pages forward through a server-supplied cursor. The vendor communicates its quota state through response headers, and the poller’s entire pacing strategy is derived from them.

The rate-limit signalling headers you must handle:

The incremental-read query contract, expressed against a generic /bibs collection:

Two status codes drive control flow. A 304 Not Modified on a conditional request means “nothing changed” — advance nothing, cost almost nothing. A 429 Too Many Requests (or a 503 under load) means back off immediately and do not advance the cursor, so the same page is safely re-fetched on recovery. Because the same vendor payload quirks surface on every poll, normalize them once at the boundary using the ILS Schema Translation Patterns layer so the rest of the pipeline only ever sees the canonical model.

Prerequisites & Environment Setup

The reference implementation targets a modern async HTTP stack. Pin versions so backoff and connection-pool behavior stay reproducible across worker hosts.

• Python 3.11+ (for asyncio.TaskGroup and exception groups)
• httpx>=0.27 for the async client and connection pooling
• tenacity>=8.2 for declarative retry/backoff policies
• pydantic>=2.6 for response-envelope validation
• respx>=0.21 (dev only) for mocking ILS responses in tests
• ILS API credentials provisioned as a dedicated integration key, scoped read-only to the bib/item/holdings endpoints
• A persistent store for the sync watermark (a single row in Postgres, or a key in Redis) — never hold the cursor only in process memory
• Secrets injected via environment, never committed

Resolve configuration from the environment at startup and fail fast if anything is missing:

import os
from dataclasses import dataclass


@dataclass(frozen=True)
class ILSConfig:
    """Immutable runtime configuration for the ILS poller."""

    base_url: str
    api_key: str
    page_size: int
    rate_limit_floor: float  # fraction of quota below which we throttle

    @classmethod
    def from_env(cls) -> "ILSConfig":
        try:
            return cls(
                base_url=os.environ["ILS_BASE_URL"].rstrip("/"),
                api_key=os.environ["ILS_API_KEY"],
                page_size=int(os.environ.get("ILS_PAGE_SIZE", "100")),
                rate_limit_floor=float(os.environ.get("ILS_RATE_FLOOR", "0.10")),
            )
        except KeyError as exc:
            raise RuntimeError(f"Missing required ILS env var: {exc.args[0]}") from exc

Provision the integration key with the narrowest scope the vendor allows. A polling key that can also mutate patron records widens the blast radius of a leaked credential far beyond the catalog; the reasoning is developed in Designing Zero-Trust Architecture for Library APIs.

Core Implementation

The poller is built in three layers: a token-bucket pacer that never lets request volume exceed the vendor’s window, an adaptive loop that reads rate-limit headers and adjusts, and a cursor that is only advanced after a page is durably handed off to the broker.

Step 1 — A token-bucket pacer

Rather than sleeping a fixed interval, gate every outbound request through a token bucket sized from X-RateLimit-Limit. This smooths bursts and keeps concurrent workers collectively under quota.

import asyncio
import time


class TokenBucket:
    """Async token bucket for pacing outbound ILS requests.

    Refills `rate` tokens per second up to `capacity`. `acquire()` blocks
    until a token is available, so callers can never exceed the vendor window.
    """

    def __init__(self, rate: float, capacity: float) -> None:
        self._rate = rate
        self._capacity = capacity
        self._tokens = capacity
        self._updated = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self) -> None:
        async with self._lock:
            while True:
                now = time.monotonic()
                elapsed = now - self._updated
                self._tokens = min(self._capacity, self._tokens + elapsed * self._rate)
                self._updated = now
                if self._tokens >= 1.0:
                    self._tokens -= 1.0
                    return
                deficit = (1.0 - self._tokens) / self._rate
                await asyncio.sleep(deficit)

    def resize(self, rate: float, capacity: float) -> None:
        """Re-tune from observed X-RateLimit-Limit without dropping tokens."""
        self._rate = rate
        self._capacity = capacity
        self._tokens = min(self._tokens, capacity)

Pitfall: if you run more than one worker per API key, each process holds its own bucket and the sum can still exceed quota. Either run a single poller per key, or centralize the bucket in Redis. Distributing the poll across a worker pool is a job for the broker layer described in Async Batch Processing for Catalog Updates, not for parallel raw pollers.

Step 2 — Parse rate-limit headers into a control signal

Translate the vendor’s headers into a small, typed decision object so the loop logic stays readable.

import logging
from dataclasses import dataclass
from datetime import datetime, timezone

import httpx

logger = logging.getLogger("ils.poller")


@dataclass(frozen=True)
class RateState:
    remaining_fraction: float  # 1.0 == full quota, 0.0 == exhausted
    reset_in: float            # seconds until the window refills
    retry_after: float | None  # authoritative wait from a 429/503, if present


def read_rate_state(response: httpx.Response) -> RateState:
    """Derive a pacing decision from ILS rate-limit headers, defensively."""
    headers = response.headers

    def _int(name: str) -> int | None:
        raw = headers.get(name)
        try:
            return int(raw) if raw is not None else None
        except ValueError:
            logger.warning("Unparseable %s header: %r", name, raw)
            return None

    limit = _int("X-RateLimit-Limit")
    remaining = _int("X-RateLimit-Remaining")
    reset_epoch = _int("X-RateLimit-Reset")

    fraction = 1.0
    if limit and remaining is not None and limit > 0:
        fraction = max(0.0, remaining / limit)

    reset_in = 0.0
    if reset_epoch is not None:
        now = datetime.now(timezone.utc).timestamp()
        reset_in = max(0.0, reset_epoch - now)

    retry_after = None
    if (raw := headers.get("Retry-After")) is not None:
        try:
            retry_after = float(raw)  # seconds form
        except ValueError:
            retry_after = 30.0  # HTTP-date form: fall back to a safe floor

    return RateState(remaining_fraction=fraction, reset_in=reset_in, retry_after=retry_after)

Step 3 — The adaptive polling loop

The loop reads the watermark, fetches a page, hands validated deltas to the broker, and only then persists the new cursor. When quota runs low it sleeps to the reset boundary instead of hammering the endpoint. Vendor-specific retry tuning — the backoff multipliers and jitter windows that keep Sierra’s per-key throttle from tripping — is delegated to Configuring Exponential Backoff for Sierra API Calls.

import asyncio
from typing import AsyncIterator, Protocol


class Watermark(Protocol):
    async def load(self) -> str: ...
    async def store(self, cursor: str) -> None: ...


class ILSPoller:
    def __init__(
        self,
        config: ILSConfig,
        client: httpx.AsyncClient,
        bucket: TokenBucket,
        watermark: Watermark,
    ) -> None:
        self._cfg = config
        self._client = client
        self._bucket = bucket
        self._watermark = watermark

    async def _fetch_page(self, cursor: str) -> httpx.Response:
        await self._bucket.acquire()
        return await self._client.get(
            "/bibs",
            params={
                "updatedAfter": cursor,
                "limit": self._cfg.page_size,
                "fields": "id,updatedDate,marc",
            },
            headers={"Authorization": f"Bearer {self._cfg.api_key}"},
        )

    async def poll_forever(self, handoff) -> None:
        """Continuously reconcile changes, pacing against vendor quota.

        `handoff` is an awaitable that durably enqueues the page's deltas
        (e.g. publishes to the message broker) before the cursor advances.
        """
        cursor = await self._watermark.load()
        while True:
            try:
                response = await self._fetch_page(cursor)
            except httpx.TransportError as exc:
                logger.warning("Transport error, holding cursor: %s", exc)
                await asyncio.sleep(5.0)
                continue

            rate = read_rate_state(response)

            if response.status_code == 304:
                await self._sleep_for_quota(rate, idle=True)
                continue
            if response.status_code == 429 or response.status_code >= 500:
                wait = rate.retry_after or max(rate.reset_in, 1.0)
                logger.warning("Throttled (%s); backing off %.1fs", response.status_code, wait)
                await asyncio.sleep(wait)
                continue  # cursor deliberately NOT advanced
            response.raise_for_status()

            page = response.json()
            records = page.get("entries", [])
            if records:
                await handoff(records)          # durable enqueue first
                cursor = records[-1]["updatedDate"]
                await self._watermark.store(cursor)  # then advance the watermark

            # Re-tune the bucket from the vendor's advertised limit.
            if (limit := response.headers.get("X-RateLimit-Limit")):
                self._bucket.resize(rate=int(limit) / 60.0, capacity=int(limit))

            await self._sleep_for_quota(rate, idle=not records)

    async def _sleep_for_quota(self, rate: RateState, idle: bool) -> None:
        if rate.remaining_fraction <= self._cfg.rate_limit_floor:
            logger.info("Quota low (%.0f%%); sleeping %.1fs to reset",
                        rate.remaining_fraction * 100, rate.reset_in)
            await asyncio.sleep(max(rate.reset_in, 1.0))
        elif idle:
            await asyncio.sleep(5.0)  # nothing new; gentle idle cadence

The ordering in the if records: block is the load-bearing invariant: the page is enqueued to the broker before the watermark moves. If the process dies between the two, the next start re-fetches the same page and the downstream stage — which applies idempotent upserts keyed on record id and modification timestamp — absorbs the duplicate harmlessly.

PII & Compliance Checkpoints

Even a bib-focused feed leaks patron data more often than teams expect: local 9xx fields, circulation notes, and hold-shelf slips carry barcodes and names, and a poller that logs raw request/response bodies for debugging will write those straight into log aggregation. Mask at the ingestion boundary, before anything reaches persistent storage, and route the masking policy through the same controls as the rest of the Patron Validation & Privacy Data Routing domain. The full export-side treatment lives in PII Masking in Patron Data Exports; the poller only needs a logging filter that redacts before emission.

import logging
import re
from typing import Any

PII_PATTERNS: tuple[re.Pattern[str], ...] = (
    re.compile(r"\b\d{14}\b"),                    # 14-digit patron barcode
    re.compile(r"[\w.%+-]+@[\w.-]+\.\w{2,}"),     # email address
    re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),         # SSN
)
REDACTION = "***REDACTED***"


class PIIMaskingFilter(logging.Filter):
    """Redact patron identifiers from every log record before it is emitted."""

    def filter(self, record: logging.LogRecord) -> bool:
        if isinstance(record.msg, dict):
            record.msg = self._redact_mapping(record.msg)
        elif isinstance(record.msg, str):
            for pattern in PII_PATTERNS:
                record.msg = pattern.sub(REDACTION, record.msg)
        return True

    @staticmethod
    def _redact_mapping(data: dict[str, Any]) -> dict[str, Any]:
        return {
            key: (REDACTION
                  if isinstance(value, str) and any(p.search(value) for p in PII_PATTERNS)
                  else value)
            for key, value in data.items()
        }


audit_logger = logging.getLogger("ils.audit")
audit_logger.addFilter(PIIMaskingFilter())
audit_logger.setLevel(logging.INFO)

Two retention flags matter for a polling feed specifically. First, the immutable audit trail should record masked request metadata (endpoint, status, quota consumed) but must never persist raw response bodies. Second, the sync watermark itself is operational metadata, not patron data — keep it out of any circulation-retention purge so a retention job cannot silently reset your cursor. The boundary rules that separate operational state from regulated patron data are set out in Data Privacy Boundaries in Library Systems.

Error Handling & Quarantine Patterns

Distinguish three error classes, because each demands a different response. Transport and throttle errors are retryable and cursor-preserving — the loop above already holds the cursor and backs off. Malformed payloads are poisonous — retrying re-fetches the same broken record forever, so they must be diverted. Authentication failures are fatal — retrying a revoked key wastes quota and hides the real problem.

Validate the response envelope with a strict model and route validation failures to a durable quarantine rather than crashing the batch. The same quarantine surface backs the schema validation quarantine queue used elsewhere in ingestion.

import logging
from pydantic import BaseModel, Field, ValidationError

logger = logging.getLogger("ils.poller")


class BibEntry(BaseModel):
    record_id: str = Field(alias="id", min_length=1)
    updated_date: str = Field(alias="updatedDate")
    marc: dict = Field(default_factory=dict)


async def validate_and_route(entries: list[dict], quarantine) -> list[BibEntry]:
    """Validate each entry; divert poison records without stalling the page."""
    clean: list[BibEntry] = []
    for raw in entries:
        try:
            clean.append(BibEntry.model_validate(raw))
        except ValidationError as exc:
            record_id = raw.get("id", "<unknown>")
            logger.error("Quarantining malformed entry %s: %s", record_id, exc.errors())
            await quarantine.put({"record_id": record_id, "errors": exc.errors(), "payload": raw})
    return clean


class FatalAuthError(RuntimeError):
    """Raised on 401/403 so the supervisor can page an operator, not retry."""

Wire the auth case into the fetch path so a revoked or expired token stops the poller loudly instead of burning quota on doomed retries:

if response.status_code in (401, 403):
    raise FatalAuthError(f"ILS rejected credentials: {response.status_code}")

Downstream corruption caused by silently accepting a truncated MARC payload is exactly what deterministic parsing exists to prevent — records that survive quarantine still pass through the leader-byte checks in Parsing MARC Records with pymarc before they are trusted.

Performance Considerations

Polling is I/O-bound, so throughput comes from concurrency under a strict bound, not from more processes. Use a single httpx.AsyncClient with a tuned connection pool and an asyncio.Semaphore to cap in-flight requests — this prevents socket exhaustion during a burst of hold or checkout events while keeping the token bucket authoritative on rate.

import asyncio
import httpx


async def build_client(base_url: str) -> httpx.AsyncClient:
    limits = httpx.Limits(max_connections=20, max_keepalive_connections=10)
    timeout = httpx.Timeout(10.0, connect=5.0)
    return httpx.AsyncClient(base_url=base_url, limits=limits, timeout=timeout, http2=True)


SEMAPHORE = asyncio.Semaphore(8)  # cap concurrent ILS requests independent of quota

Keep memory flat by streaming pages to the broker rather than accumulating the full change set in a list — the page size bounds resident set size, and each page is released before the next is fetched. Prefer fields projections and conditional ETag requests to shrink payloads; a feed that earns mostly 304 responses costs almost nothing. When change volume grows past what a single ordered poller can drain in real time, move fan-out to a distributed worker pool as described in Using Celery for Distributed Catalog Ingestion rather than spinning up competing raw pollers against the same key.

Verification & Testing

Never test a poller against the live vendor endpoint — you will burn quota and cannot deterministically reproduce a 429. Mock the transport with respx and assert on the behaviors that matter: that a 429 holds the cursor, that a 304 advances nothing, and that malformed entries are quarantined.

import httpx
import pytest
import respx


@pytest.mark.asyncio
@respx.mock
async def test_429_preserves_cursor(poller_factory) -> None:
    route = respx.get("https://ils.test/bibs").mock(
        side_effect=[
            httpx.Response(429, headers={"Retry-After": "0"}),
            httpx.Response(200, json={"entries": []},
                           headers={"X-RateLimit-Limit": "60", "X-RateLimit-Remaining": "59"}),
        ]
    )
    poller, watermark = poller_factory(start_cursor="2026-07-01T00:00:00Z")
    await poller.poll_once()  # a single-iteration variant of poll_forever

    assert route.call_count == 2
    # Cursor unchanged after the throttled attempt.
    assert await watermark.load() == "2026-07-01T00:00:00Z"


@pytest.mark.asyncio
async def test_malformed_entry_is_quarantined(quarantine) -> None:
    entries = [{"id": "b1", "updatedDate": "2026-07-01T00:00:00Z"}, {"updatedDate": "bad"}]
    clean = await validate_and_route(entries, quarantine)

    assert [c.record_id for c in clean] == ["b1"]
    assert quarantine.qsize() == 1

Complement unit tests with a soak test that replays a recorded window of vendor responses (including at least one throttle event and one malformed record) and asserts the watermark advances monotonically and the quarantine depth matches the number of bad records injected.

FAQ: Troubleshooting Polling & Rate-Limit Failures

Why does my poller keep getting 429s even though I sleep between requests?

A fixed sleep does not track the vendor’s window. If multiple workers share one API key, or the window is shorter than your sleep assumes, aggregate volume still crosses the limit. Gate every request through a shared token bucket sized from X-RateLimit-Limit, and when X-RateLimit-Remaining drops below your floor, sleep to X-RateLimit-Reset instead of a constant. Run exactly one poller per key unless the bucket is centralized in Redis.

The catalog is missing recent updates but no errors are logged — what happened?

Almost always a cursor that advanced past records it never delivered. This happens when the watermark is stored before the page is durably handed off. Reorder so the broker enqueue completes first and the watermark is persisted only after. Confirm the store is not in process memory — a restart with an in-memory cursor silently resets to a stale (or empty) position.

A single bad record halts the whole batch. How do I isolate it?

Do not let a ValidationError propagate out of the page loop. Validate each entry individually and route failures to the quarantine queue, as in the error-handling section. The page continues with the clean records, and the poisoned entry is inspected out of band instead of blocking every healthy update behind it.

The ILS returns 304 Not Modified constantly and I never see updates — is that a bug?

No, that is the conditional-request path working. A 304 means the entity version behind your ETag / If-Modified-Since has not changed. If you genuinely expect changes and still see only 304, verify you are sending the change-window parameter (updatedAfter/since) with the correct timezone — a naive local timestamp against a UTC endpoint can mask a whole window of updates.

My retries make throttling worse during peak circulation hours. Why?

Synchronized retries across workers create a thundering herd that re-collides on every reset boundary. Add multiplicative jitter to the backoff so retries spread across the window, cap the maximum delay, and open a circuit breaker after repeated failures so a struggling endpoint gets time to recover. The vendor-tuned parameters are worked through in Configuring Exponential Backoff for Sierra API Calls and the breaker mechanics in Implementing Circuit Breakers for ILS API Timeouts.

Related

• Configuring Exponential Backoff for Sierra API Calls — vendor-specific retry, jitter, and quota-header tuning for the Sierra REST API.
• Async Batch Processing for Catalog Updates — the broker seam that consumes the deltas this poller emits, with idempotent upserts.
• Schema Validation for Ingested Records — the quarantine and contract-validation surface for records that survive polling.
• Implementing Circuit Breakers for ILS API Timeouts — the breaker that protects the poller from a degraded endpoint.
• ILS Schema Translation Patterns — normalizing vendor payload quirks into the canonical model at the boundary.