Circulation Protocols & Interoperability for Library Data Sync Pipelines

Every checkout, checkin, renewal, hold, and patron-status lookup that a library performs on a self-service device is a machine-to-machine transaction that has to cross a protocol boundary and commit against the integrated library system (ILS) exactly once. The Circulation Protocols & Interoperability domain defines the architectural standards for the transports that carry those events — chiefly SIP2 (the 3M Standard Interchange Protocol 2), NCIP (the NISO Circulation Interchange Protocol, Z39.83), and Z39.50 with its modern SRU successor for search retrieval — between the ILS core and the fleet of kiosks, RFID pads, discovery layers, and consortial partners that depend on it. Where the Catalog Ingestion & ILS Sync Pipelines domain moves bibliographic records in bulk and the Patron Validation & Privacy Data Routing domain resolves patron identity before it crosses a network boundary, this domain moves the circulation event — the smallest, most time-sensitive, and most frequently replayed unit of work in the entire platform. It is written for library technical staff, ILS administrators, public-sector developers, and Python automation engineers who own the reliability of the self-check fleet and the correctness of the loan record it produces.

Circulation protocols look deceptively simple — a SIP2 checkout is a single line of text terminated by a carriage return — but they are unforgiving in production. A dropped socket mid-transaction can leave an item checked out on the kiosk display and available in the ILS; a replayed NCIP CheckOutItem can bill a patron twice; a self-check vendor that logs the raw patron barcode turns a routine transaction into a confidentiality incident. The engineering consequence is that framing, idempotency, and privacy are not afterthoughts layered on top of the protocol — they are properties the integration must guarantee at the boundary, because the protocols themselves guarantee almost nothing. The sections below establish the transport topology and directional flows, the schema contracts that SIP2 fixed-field framing and NCIP XML must satisfy, the idempotent routing implementation that makes replayed messages safe, the compliance architecture that keeps patron identifiers from leaking to third-party devices, and the failure-recovery patterns that keep circulation working when a socket dies mid-transaction.

Domain Isolation & Data Flow Topology

At the center of the domain sits a protocol gateway that terminates two fundamentally different transports and presents one internal contract to the rest of the platform. On one side it runs a long-lived TCP socket server that speaks SIP2 to self-check kiosks and RFID pads; on the other it exposes an HTTP endpoint that accepts NCIP XML from consortial partners and, in reverse, initiates NCIP requests to them. The gateway is the only place in the platform where raw protocol bytes exist. Once a message is framed and normalized, everything downstream operates on a canonical circulation event, so the ILS core never has to know whether a checkout originated at a SIP2 kiosk or arrived as an NCIP CheckOutItem from a partner library.

The socket/HTTP transport boundary

The defining topological split in this domain is the transport boundary between the connection-oriented socket world and the request/response HTTP world. SIP2 is a synchronous line protocol over a raw TCP socket: the kiosk opens a connection, sends a login, and then exchanges one request line and one response line per transaction, often keeping the socket open for the duration of a patron’s session. There is no HTTP status code, no content type, and no framing beyond a carriage return — the transport gives you a byte stream and nothing else, which is why the SIP2 Protocol Integration for Self-Service Circulation layer has to impose framing, timeouts, and connection health-checking itself. NCIP, by contrast, is a stateless XML message exchanged over HTTP(S): each request is self-describing, carries its own envelope, and benefits from the entire HTTP toolchain — TLS termination, load balancing, and standard timeout semantics. The gateway normalizes both into the same internal event so that the transport choice becomes an implementation detail rather than a fork in the business logic.

Unidirectional vs bidirectional flows

Not every protocol flow runs in both directions, and modeling directionality precisely is what makes the topology auditable. SIP2 from a self-check kiosk is effectively unidirectional at the domain level: the kiosk initiates, the ILS answers, and the ILS never pushes an unsolicited message back to the kiosk. Z39.50 and SRU search retrieval is strictly outbound query and inbound result — the discovery layer asks the Z39.50 and SRU Catalog Search Integration endpoint a question and receives MARC or Dublin Core records, but no circulation state ever changes. NCIP resource sharing is the one genuinely bidirectional flow: in an interlibrary loan, your ILS both sends RequestItem and CheckOutItem to a partner and receives their reciprocal messages, so the NCIP Resource-Sharing Workflows layer must handle initiator and responder roles on the same connection. As in the patron domain, a bidirectional concern is modeled internally as two independent unidirectional streams, so that a stalled inbound partner update can never back-pressure an outbound checkout a patron is waiting on.

Message-broker patterns

Behind the router, circulation events flow onto a durable message broker with one topic per commit target. Partitioning by a stable hash of the item barcode preserves per-item ordering — so that a checkout and its subsequent checkin are never reordered — while still allowing parallel consumption across items. Consumers commit their broker offset only after the ILS acknowledges the underlying transaction, which turns the broker into the backbone of the commit-once guarantee developed below. This is the same broker discipline established for bulk record movement in Async Batch Processing for Catalog Updates, reused here for the far more latency-sensitive circulation stream, where a patron is physically standing at the kiosk waiting for the transaction to complete.

Schema Interoperability & Transformation Standards

Circulation messages arrive in two radically different serializations — SIP2’s fixed-field line format and NCIP’s XML — and the normalizer’s job is to collapse both into a single canonical event before any routing logic runs. This is the circulation analogue of the bibliographic normalization work in ILS Schema Translation Patterns: one deterministic translation layer, not a web of point-to-point mappings between every device model and every ILS.

SIP2 fixed-field framing

A SIP2 message is a single line of ASCII beginning with a two-digit message identifier, followed by a run of fixed-length positional fields, then variable-length fields each introduced by a two-character field code and terminated by a pipe (|) delimiter. A 11 Checkout request, for example, opens with two boolean flags and an 18-character transaction-date field, then carries variable fields for the institution id (AO), patron identifier (AA), item identifier (AB), and terminal password (AC). Two optional trailing fields carry a sequence number (AY) and a checksum (AZ) — the only integrity controls the protocol offers. Parsing SIP2 is unforgiving: the positional fields have no delimiters, so an off-by-one in the fixed header silently misreads every field after it. The table below shows a representative slice of the Checkout (11) message contract that the normalizer must satisfy.

NCIP XML and the canonical event

NCIP carries the same logical operations in verbose, namespaced XML: a CheckOutItem element wraps UserId, ItemId, and RequestId children, each self-describing and order-independent. The XML is far easier to parse correctly than SIP2’s positional fields, but it is heavier on the wire and demands schema validation against the NCIP XSD before trust — a partner can send a structurally valid document with a semantically impossible combination of elements. Whichever serialization arrives, the normalizer emits the same canonical event: a typed record carrying the operation, the item barcode, a masked patron reference, the originating channel, the client timestamp, and the integrity metadata (sequence and transaction id). Because the RFID self-check path adds a tag-to-barcode reconciliation step ahead of framing, the RFID and Self-Check Circulation Sync layer produces the same canonical event as a bare SIP2 kiosk once the tag scan has been resolved to an item barcode. Choosing which protocol to standardize a given integration on — and when the XML overhead of NCIP is worth its richer semantics — is the trade-off analyzed in SIP2 vs NCIP: Choosing the Right Circulation Protocol for Your ILS.

Encoding and diacritic hazards

Circulation messages inherit the same encoding hazards as legacy bibliographic data. SIP2 predates Unicode and older kiosks still emit Latin-1 or even MARC-8 in the patron-name fields of a Patron Information response, so a naive UTF-8 decode either raises or silently mojibakes a name on the receipt. The normalizer applies the same defensive decoding discipline documented for MARC21 Field Mapping for Modern Pipelines — detect the source encoding, transcode to canonical UTF-8, and normalize to NFC — before the event is routed, so that receipt text and hold-shelf slips render correctly regardless of the device’s vintage.

Idempotent Sync Patterns & Production Implementation

Circulation protocols were designed for reliable point-to-point links and offer only the thinnest integrity guarantees, so the burden of making a replayed message safe falls entirely on the integration. A dropped socket after the ILS commits but before the kiosk reads the response will cause the kiosk to retry the identical checkout; a partner whose NCIP client times out will resend CheckOutItem; an operator will click “resync” twice. Without idempotency each of these produces a duplicate loan. The domain achieves commit-once side effects through three cooperating mechanisms, all keyed off the integrity metadata the protocols already carry.

1. Deterministic transaction keys. Each circulation event is assigned a key derived from a stable hash of the item barcode, the operation type, the originating terminal, and — where the client supplies one — the protocol sequence number or NCIP RequestId. The same logical transaction always produces the same key, so a replay is recognizable before any ILS call executes.
2. Sequence and checksum verification. SIP2’s AY sequence number and AZ checksum are verified at the gateway, and the last-seen sequence per terminal is persisted in the ledger. A message whose sequence has already been recorded is a confirmed duplicate; a message whose checksum fails is corrupt and is nacked so the kiosk resends. These edge cases are worked through in detail in Handling SIP2 Checksum and Sequence Number Errors.
3. Conditional commit. The ILS-facing consumer applies each event under the transaction key, so a replayed key is a no-op at the commit layer rather than a second checkout, and the deduplicating router short-circuits any key it has already seen.

Reference implementation

The following module is a production-shaped gateway core. It frames an incoming SIP2 line, verifies the checksum and sequence number, normalizes the message into a canonical event, derives a deterministic transaction key, and routes it through a deduplicating worker so that a replayed checkout commits exactly once. The in-memory stores stand in for Redis and a durable broker; in production the ledger is a database table and secrets are loaded from a KMS.

from __future__ import annotations

import asyncio
import hashlib
import logging
import time
from dataclasses import dataclass, field
from typing import Mapping, Optional

logger = logging.getLogger("ils.circulation_gateway")

# SIP2 field codes we extract from a Checkout (11) message.
_FIELD_CODES = {"AO": "institution", "AA": "patron_ref", "AB": "item_barcode", "AC": "terminal_pw"}


class ChecksumError(ValueError):
    """Raised when a SIP2 AZ checksum does not match the computed value."""


@dataclass(frozen=True)
class CirculationEvent:
    type: str
    item_barcode: str
    patron_ref: str  # already masked; raw barcode never stored on the event
    terminal: str
    seq: Optional[int]
    client_ts: str
    txn_key: str
    metadata: Mapping[str, str] = field(default_factory=dict)


def sip2_checksum(message: str) -> str:
    """Compute the SIP2 AZ checksum: two's complement of the summed bytes, 4 hex chars."""
    total = sum(message.encode("ascii", errors="replace"))
    return f"{(-total) & 0xFFFF:04X}"


def verify_and_split(line: str) -> str:
    """Strip and validate the trailing AY sequence and AZ checksum, return the body.

    The checksum covers every byte up to and including 'AZ', so we recompute over
    that prefix and compare against the four hex characters that follow.
    """
    marker = line.rfind("AZ")
    if marker == -1:
        return line  # terminal not configured for error detection
    covered = line[: marker + 2]
    supplied = line[marker + 2 : marker + 6]
    computed = sip2_checksum(covered)
    if supplied.upper() != computed:
        raise ChecksumError(f"checksum mismatch: got {supplied!r}, computed {computed!r}")
    return line[:marker].rstrip("|")


class LedgerStore:
    """Stand-in for the persisted sequence + transaction ledger. Replace in production."""

    def __init__(self) -> None:
        self._last_seq: dict[str, int] = {}
        self._committed: set[str] = set()

    def is_replay(self, terminal: str, seq: Optional[int], txn_key: str) -> bool:
        if txn_key in self._committed:
            return True
        if seq is not None and self._last_seq.get(terminal) == seq:
            return True
        return False

    def record(self, terminal: str, seq: Optional[int], txn_key: str) -> None:
        self._committed.add(txn_key)
        if seq is not None:
            self._last_seq[terminal] = seq


def _mask_barcode(raw: str) -> str:
    """Keep only the last four digits; the raw patron barcode never leaves the gateway."""
    return f"****{raw[-4:]}" if len(raw) >= 4 else "****"


class CirculationGateway:
    def __init__(self, key_salt: bytes) -> None:
        self._salt = key_salt
        self._ledger = LedgerStore()
        self._queue: asyncio.Queue[CirculationEvent] = asyncio.Queue()

    def _txn_key(self, item_barcode: str, op: str, terminal: str, seq: Optional[int]) -> str:
        raw = f"{item_barcode}:{op}:{terminal}:{seq}".encode("utf-8")
        return hashlib.blake2b(raw, key=self._salt, digest_size=20).hexdigest()

    def _parse_sip2(self, body: str, terminal: str, seq: Optional[int]) -> CirculationEvent:
        fixed, _, tail = body.partition("|")
        fields: dict[str, str] = {}
        for chunk in (body[len(fixed) + 1 :]).split("|"):
            if len(chunk) >= 2 and chunk[:2] in _FIELD_CODES:
                fields[_FIELD_CODES[chunk[:2]]] = chunk[2:]
        item = fields.get("item_barcode", "")
        txn_key = self._txn_key(item, "checkout", terminal, seq)
        return CirculationEvent(
            type="checkout",
            item_barcode=item,
            patron_ref=_mask_barcode(fields.get("patron_ref", "")),
            terminal=terminal,
            seq=seq,
            client_ts=fixed[3:21],
            txn_key=txn_key,
            metadata={"institution": fields.get("institution", "")},
        )

    async def ingest(self, line: str, terminal: str) -> Optional[CirculationEvent]:
        try:
            body = verify_and_split(line.rstrip("\r\n"))
        except ChecksumError as exc:
            # Corrupt on the wire: nack so the kiosk resends; do not touch the ILS.
            logger.warning("sip2_checksum_reject", extra={"terminal": terminal, "err": str(exc)})
            return None

        seq = int(line[line.rfind("AY") + 2]) if "AY" in line else None
        event = self._parse_sip2(body, terminal, seq)

        if self._ledger.is_replay(terminal, seq, event.txn_key):
            logger.info(
                "circulation_replay_ignored",
                extra={"terminal": terminal, "txn": event.txn_key[:12], "item": event.item_barcode},
            )
            return None

        await self._queue.put(event)
        logger.info(
            "circulation_routed",
            extra={"terminal": terminal, "txn": event.txn_key[:12], "op": event.type},
        )
        return event

    async def run_committer(self) -> None:
        while True:
            event = await self._queue.get()
            try:
                await asyncio.sleep(0.03)  # placeholder for the durable ILS checkout call
                self._ledger.record(event.terminal, event.seq, event.txn_key)
                logger.info("ils_commit_ok", extra={"txn": event.txn_key[:12]})
            except asyncio.CancelledError:
                raise
            except Exception:
                # Never record on failure: leaving the key unrecorded keeps the retry safe.
                logger.exception("ils_commit_failed_requeue", extra={"txn": event.txn_key[:12]})
                await asyncio.sleep(1)
                self._queue.put_nowait(event)
            finally:
                self._queue.task_done()

The transaction key deliberately incorporates the protocol sequence number so that two genuinely distinct checkouts of the same item — a checkout, a return, and a second checkout — produce different keys, while a byte-for-byte replay of one checkout produces an identical key. The ledger records a key only after the ILS confirms the commit, so a crash between commit and record leaves the key unrecorded and the retry is safely deduplicated at the ledger’s is_replay check on the way back in. Tune the sequence-tracking granularity to the terminal’s behavior; a single-digit SIP2 AY counter wraps every ten messages, so the transaction key, not the raw sequence, is the authoritative dedupe token.

Compliance & Privacy Architecture

The circulation boundary is where patron confidentiality is most easily lost, because the protocol places a raw patron barcode on the wire and hands it to a device the library often does not fully control. A self-check kiosk is frequently a third-party appliance running vendor firmware; a consortial partner’s ILS is operated by another institution entirely. The governing regimes are the same as elsewhere on the platform — FERPA for academic libraries serving students, GDPR and its US state analogues for institutions serving covered residents, and the state library-confidentiality statutes that protect circulation records in most US states — but the enforcement point is specific to this domain: the patron identifier must never persist in cleartext beyond the moment of authorization, and it must never reach a third-party device’s logs.

Masking the patron identifier at the boundary

The single most important control is that the gateway masks the patron barcode the instant it has been used to authorize a transaction. The canonical event carries only a masked reference (****1234), and the raw AA field is discarded before the event is routed or logged. This enforces least privilege structurally: a self-check vendor that captures gateway traffic, or an over-eager debug log, cannot leak a barcode the event never carried. When a downstream export genuinely needs attribute-level patron data — for a hold-notice run or a consortial reciprocity report — it passes through the field-level obfuscation specified in PII Masking in Patron Data Exports, never a raw dump of the circulation stream.

What must never cross to a third party

Consortial NCIP flows and third-party self-check firmware are outside the library’s trust boundary, so the gateway treats them as untrusted egress. A partner library in an interlibrary-loan transaction needs to know that a patron in good standing requested an item; it does not need that patron’s home barcode, name, or contact details. The gateway therefore substitutes a per-transaction pseudonymous reference for the patron identifier on every outbound NCIP message and resolves it back only on the return leg, so the partner never receives a durable identifier it could correlate across transactions. Terminal passwords (AC) and institution secrets are stripped from the event immediately after the gateway authenticates the device — they are transport credentials, not circulation data, and they have no place in the routed event or the audit log.

The circulation audit log

The audit log proves that circulation happened without becoming a shadow copy of the patron records it governs. It records the masked patron reference, the item barcode, the operation, the originating terminal, the transaction key, the protocol sequence, and the commit outcome (committed / replay-ignored / checksum-rejected / dead-lettered) — never the raw patron barcode, name, or terminal password. Because the log keys on the transaction key rather than on patron identity, an on-call engineer can trace one item’s checkout end to end, and an auditor can reconcile the ILS loan record against the gateway’s ledger, without either of them ever reading a patron identifier. This is the same minimization principle the platform enforces everywhere, applied at the one boundary where the protocol itself would otherwise leak it.

Operational Failure Modes & Recovery

Circulation failures are uniquely visible: they happen while a patron is standing at the kiosk, and a wrong outcome — a double charge, an item that shows as checked out but reads as available — is immediately felt. The recovery architecture is built around four named failure classes.

Dropped sockets and the ambiguous commit

The signature SIP2 failure is a socket that dies after the ILS commits the checkout but before the kiosk reads the response. The kiosk, seeing no reply, retries; without idempotency this double-checks-out the item. The gateway defends against this with the transaction key and the ledger: the retry carries the same item, operation, terminal, and sequence, produces the same key, and is recognized as a replay before it reaches the ILS. Socket health is monitored with an idle timeout and a periodic SIP2 SC Status heartbeat, so a half-open connection is detected and torn down rather than left to hang a patron’s session. Deduplicating these replayed checkouts is the specific concern of Deduplicating Replayed SIP2 Checkout Messages.

NCIP timeouts and duplicate resource-sharing requests

NCIP over HTTP fails differently: a partner’s request can time out at the HTTP layer while the operation actually completes on your side, so the partner resends and, again, risks a duplicate. Because the transaction key incorporates the NCIP RequestId, a resent request maps to the same key and commits once. When your ILS is the initiator and the partner times out, the gateway retries with bounded exponential backoff and jitter, and after the retry budget is exhausted it routes the message to the dead-letter queue rather than blocking the interlibrary-loan workflow indefinitely. Reconstructing state after one of these stalls is covered in Recovering from NCIP CheckOutItem Timeouts.

Poison messages and the dead-letter queue

A message that repeatedly fails to commit — a malformed NCIP document, an item barcode the ILS rejects, a checksum that never validates — must not block the partition behind it. After a bounded number of retries, the gateway moves it to a durable dead-letter queue where the framed message is stored (with the patron identifier already masked) alongside the failure reason, and an operator is alerted with the transaction key needed to replay it once the root cause is fixed. Because every operation is keyed and idempotent, replay from the dead-letter queue is always safe — the same discipline the ingestion domain applies to dead-letter recovery in catalog ingestion.

Alerting patterns

Effective alerting watches rates and ratios, not raw counts. The high-value circulation signals are: checksum-rejection rate above a rolling baseline per terminal (a failing kiosk or a noisy link), replay-ignored rate spiking (a stuck kiosk retrying, or a partner whose HTTP timeouts are too aggressive), commit p99 latency on the synchronous path that patrons feel directly, dead-letter arrival rate (poison messages or an ILS outage), and socket-reconnect churn per terminal (flaky hardware). Correlation identifiers stitched from the gateway through the normalizer to the ILS commit let an engineer trace one transaction end to end, and because the identifier keys on the item and transaction rather than the patron, that tracing never requires reading a patron barcode.

Reference Implementation Checklist

Use this as the acceptance gate for a circulation integration before it touches production traffic. Each item maps to an architectural layer described above.

• Protocol gateway terminates the SIP2 socket and the NCIP HTTP endpoint in one place; no raw protocol bytes exist downstream
• SIP2 messages are framed on carriage-return boundaries with an idle timeout and a periodic status heartbeat
• AZ checksums are verified before any ILS call; a failed checksum is nacked for resend, never committed
• AY sequence numbers are persisted per terminal and used as a replay signal
• NCIP XML is validated against the NCIP XSD before it is trusted
• Both serializations normalize to one canonical circulation event before routing
• Legacy patron-name fields are transcoded to canonical UTF-8 and NFC-normalized before receipt rendering
• Deterministic transaction keys incorporate item barcode, operation, terminal, and protocol sequence / RequestId
• The ILS-facing consumer applies each event as a conditional commit-once operation keyed to the transaction key
• The sequence-and-checksum ledger records a key only after the ILS confirms the commit
• Raw patron barcodes are masked at the gateway and never routed, logged, or persisted in cleartext
• Outbound consortial NCIP substitutes a per-transaction pseudonym for the patron identifier
• Terminal passwords and institution secrets are stripped from the event immediately after device authentication
• Bidirectional NCIP is modeled as two independent unidirectional streams so inbound cannot back-pressure outbound
• Retries use bounded exponential backoff with jitter; exhausted messages route to a durable dead-letter queue
• Dead-lettered messages store the masked, framed payload plus the failure reason and are replay-safe by transaction key
• The audit log records masked references, item barcodes, and commit outcomes only — no raw patron identifiers
• Rate-and-ratio alerting is wired for checksum rejection, replay spikes, commit latency, dead-letter arrival, and socket churn

Related

• SIP2 Protocol Integration for Self-Service Circulation — framing, checksums, and socket handling for self-check kiosks
• NCIP Resource-Sharing Workflows — bidirectional interlibrary-loan messaging over HTTP with consortial partners
• SIP2 vs NCIP: Choosing the Right Circulation Protocol for Your ILS — the trade-offs between fixed-field SIP2 and XML NCIP
• Z39.50 and SRU Catalog Search Integration — outbound search retrieval feeding the discovery layer
• RFID and Self-Check Circulation Sync — reconciling tag scans to item barcodes ahead of framing
• Patron Validation & Privacy Data Routing — the identity and privacy domain the masked patron reference resolves against