Deduplicating Replayed SIP2 Checkout Messages

This page answers one narrow operational question: why does a self-check station occasionally loan or charge a patron for the same item twice, when the staff swear the item was scanned only once — and how do you make the ILS treat a retransmitted Checkout the same way twice in a row? It sits under SIP2 Protocol Integration for Self-Service Circulation, which covers the request/response contract end to end, and within the broader Circulation Protocols & Interoperability architecture. If you run a self-check fleet or a SIP2 gateway in front of Sierra, Polaris, Koha, or FOLIO, this double-loan is one you will eventually see on a busy branch’s flaky wireless.

Problem Framing

The symptom is not a crash and not an error dialog on the kiosk. A patron checks out one copy of a book, the station prints one receipt, and the item leaves the building once. Days later the patron calls about two loans on the same barcode with staggered due dates, or a fines report shows the title billed twice. The circulation desk cannot reproduce it, because the second loan was never triggered by a second scan — it was triggered by the network.

The evidence lives in the SIP2 gateway log, where the two messages are byte-for-byte identical and only milliseconds to seconds apart:

11:02:14.882  recv  11YN20260716    110214AOMAIN-BR|AAB1029384756|ABI30012000029894|ACterm7|AY3AZF1C2
11:02:14.883  proc  checkout OK item=I30012000029894 patron=B1029384756  -> loan 88214431
11:02:17.401  recv  11YN20260716    110214AOMAIN-BR|AAB1029384756|ABI30012000029894|ACterm7|AY3AZF1C2
11:02:17.402  proc  checkout OK item=I30012000029894 patron=B1029384756  -> loan 88214452

Two Checkout (11) requests, the same AA patron barcode, the same AB item barcode, the same AY3 sequence number, and the same AZF1C2 checksum. The gateway processed both and the ILS opened two loans (88214431 and 88214452). The AZ checksum passed on both — because the bytes were never corrupted. The AY sequence number matched on both — because the station genuinely intended them to be the same transaction. Nothing at the protocol layer flagged the second message as a repeat, so the application committed it as new work.

Root Cause

SIP2 is a stateless, line-oriented protocol carried over a plain, unreliable TCP (often socket-over-serial or a terminal server) connection. Each request is self-contained; the server keeps no per-transaction memory between messages. The station appends two integrity fields to every message: AY, an error-detection sequence number that lets the endpoints notice a missed or reordered reply, and AZ, a checksum over the message bytes. Those fields — covered in depth in Handling SIP2 Checksum and Sequence Number Errors — solve transmission integrity. They do not solve application-level replay.

The replay happens above the wire. The station sends Checkout (11), the ILS commits the loan and sends the Checkout Response (12), but the response is slow or dropped before the station’s read timeout fires. To the station the transaction looks unacknowledged, so its retry logic does exactly what a resilient client should do: it retransmits the identical request, sequence number and all. From the server’s point of view the second message is indistinguishable from a fresh checkout, because the only thing that would make it distinguishable — a record that this exact transaction was already applied — does not exist. The AY sequence number cannot fill that gap: a correct retransmission reuses the same AY, so equal sequence numbers are ambiguous between “you missed my reply, here it is again” and “process this now.”

The missing invariant is an idempotency key. Without a deduplication key on the tuple of (institution, patron AA, item AB, transaction identity), the server has no way to recognise that the second message is a repeat of committed work, so it commits a second loan.

Solution

Put a deduplication gate in front of the checkout handler that computes a deterministic key from the message’s identifying fields and refuses to run the loan twice for the same key. A SETNX-style insert (set-if-not-exists) against Redis, or equivalently a UNIQUE constraint on a dedup table, makes the first attempt for a key win and every later attempt within the window a no-op. A short TTL bounds the window to the station’s realistic retry horizon — long enough to absorb a slow ACK and a couple of retransmissions, short enough that a genuine, deliberate second checkout of the same item minutes later is not swallowed.

The relevant SIP2 Checkout (11) fields that identify a transaction:

from __future__ import annotations

import hashlib
import hmac
import logging
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Iterator, Protocol

logger = logging.getLogger("sip2.dedup")

# The station's realistic retransmission horizon. Long enough to cover a slow
# ACK and a couple of retries; short enough that a deliberate re-checkout later
# is treated as new work.
DEDUP_TTL_SECONDS = 90


class DuplicateCheckout(Exception):
    """Raised when a Checkout is a replay of one already applied in-window."""


class DedupBackend(Protocol):
    def set_if_absent(self, key: str, value: str, ttl_seconds: int) -> bool:
        """Atomically insert key only if absent. True if we won the insert."""


@dataclass(frozen=True)
class CheckoutTxn:
    institution: str  # AO
    patron_barcode: str  # AA
    item_barcode: str  # AB
    sequence: str  # AY


def idempotency_key(txn: CheckoutTxn, *, secret: bytes) -> str:
    """Deterministic dedup key over the transaction's identifying fields.

    The patron barcode is folded in via HMAC-SHA-256, never as cleartext, so
    the dedup store holds no raw PII even if it is dumped or replicated.
    """
    patron_digest = hmac.new(
        secret, f"patron:{txn.patron_barcode}".encode(), hashlib.sha256
    ).hexdigest()
    material = "|".join(
        (txn.institution, patron_digest, txn.item_barcode, txn.sequence)
    )
    return "sip2:checkout:" + hashlib.sha256(material.encode()).hexdigest()


@contextmanager
def dedup_guard(
    backend: DedupBackend,
    txn: CheckoutTxn,
    *,
    secret: bytes,
) -> Iterator[None]:
    """Admit the first Checkout for a key; reject replays within the TTL window.

    Raises DuplicateCheckout if another message already claimed this key, so the
    caller can return the prior Checkout Response instead of committing a loan.
    """
    key = idempotency_key(txn, secret=secret)
    won = backend.set_if_absent(key, value="1", ttl_seconds=DEDUP_TTL_SECONDS)
    if not won:
        logger.warning(
            "duplicate_checkout_suppressed",
            extra={
                "dedup_key": key,
                "institution": txn.institution,
                "item": txn.item_barcode,
                "sequence": txn.sequence,
            },
        )
        raise DuplicateCheckout(key)
    logger.info(
        "checkout_admitted",
        extra={"dedup_key": key, "institution": txn.institution},
    )
    yield


def handle_checkout(
    backend: DedupBackend,
    txn: CheckoutTxn,
    *,
    secret: bytes,
    commit_loan,  # Callable[[CheckoutTxn], str]
) -> str:
    """Commit exactly one loan per transaction, even under replay."""
    try:
        with dedup_guard(backend, txn, secret=secret):
            loan_id = commit_loan(txn)
            logger.info("loan_committed", extra={"loan_id": loan_id})
            return loan_id
    except DuplicateCheckout:
        # The first message already opened the loan; replay is a safe no-op.
        # Re-send the cached Checkout Response (12) rather than open a new loan.
        return _cached_response_for(txn)

The behavioural change is the inversion of the default. Before, the checkout handler ran unconditionally for every well-formed message, so two identical requests produced two loans. After, the first request claims the key with an atomic set_if_absent and commits; the second loses the race, raises DuplicateCheckout, and returns the cached Checkout Response (12) instead of touching the loan table. Because the guard is a context manager, the loan commit and the key claim share one code path — there is no window where a crash between “claim key” and “commit loan” leaves the key held but no loan created without also failing loudly.

One subtlety worth calling out: fold AY into the key only if your stations reliably reuse the same sequence number on a retransmission. Many do; some increment it. If yours increment, drop AY from the key so a genuine retransmission with a bumped sequence still collides with the original — the (institution, patron, item) tuple within a short TTL is the durable identity of the checkout.

Compliance or Privacy Impact

The dedup store is a new place where patron identifiers could accumulate, so it must not become a shadow copy of the barcode table. The idempotency_key function above never stores the AA barcode in the clear — it folds it through HMAC-SHA-256 with a server-side secret before it ever reaches Redis, so the key is opaque and cannot be reversed to a patron even if the dedup store is dumped, replicated to a read replica, or captured in a memory snapshot. This is the same tokenize-before-store discipline the export pipeline applies in PII Masking in Patron Data Exports; reuse the same secret-management and rotation posture so the dedup keys and the export surrogates stay consistent.

Two further consequences follow. First, the short TTL is itself a data-minimisation control: keys expire in seconds, so the store holds only the working set of in-flight transactions, not a durable log of who borrowed what. Second, if you back the gate with a persistent UNIQUE table instead of a TTL cache, treat that table as circulation data with a real retention window rather than a cache — do not let it outlive its operational need. Because the stored value is a hash and not the barcode, the gate narrows the PII surface rather than widening it, but the secret used for the HMAC now guards patron linkability and deserves the same handling as any other identity key.

Verification

Confirm idempotency with a test that replays the identical transaction and asserts the loan handler runs exactly once, while a distinct transaction still passes through.

class FakeBackend:
    """In-memory set_if_absent, ignoring TTL for the test."""

    def __init__(self) -> None:
        self._keys: set[str] = set()

    def set_if_absent(self, key: str, value: str, ttl_seconds: int) -> bool:
        if key in self._keys:
            return False
        self._keys.add(key)
        return True


SECRET = b"unit-test-secret-not-for-prod"


def test_replayed_checkout_commits_one_loan() -> None:
    backend = FakeBackend()
    txn = CheckoutTxn("MAIN-BR", "B1029384756", "I30012000029894", "3")
    calls: list[CheckoutTxn] = []

    def commit_loan(t: CheckoutTxn) -> str:
        calls.append(t)
        return f"loan-{len(calls)}"

    first = handle_checkout(backend, txn, secret=SECRET, commit_loan=commit_loan)
    second = handle_checkout(backend, txn, secret=SECRET, commit_loan=commit_loan)

    assert first == "loan-1"          # first message opened the loan
    assert second != "loan-2"         # replay did NOT open a second loan
    assert len(calls) == 1            # commit_loan ran exactly once


def test_key_holds_no_raw_barcode() -> None:
    txn = CheckoutTxn("MAIN-BR", "B1029384756", "I30012000029894", "3")
    key = idempotency_key(txn, secret=SECRET)
    assert "B1029384756" not in key   # barcode never appears in cleartext
    assert key.startswith("sip2:checkout:")

For a running gateway, watch the duplicate_checkout_suppressed log rate as a health signal. A steady trickle is normal — it is the gate doing its job on a lossy link. A sudden spike points at a network or ACK-timeout regression upstream: the stations are retransmitting far more than usual, and the gate is the only thing standing between that and a wave of double loans. Alert on the rate, not on individual events, and confirm end to end by scripting two identical Checkout (11) frames into the gateway socket back to back and asserting the ILS loan count for that item rises by exactly one.

Related

• SIP2 Protocol Integration for Self-Service Circulation — the parent guide to the SIP2 request/response contract this dedup gate sits in front of.
• Circulation Protocols & Interoperability — the wider architecture for self-service circulation and resource-sharing protocols.
• Handling SIP2 Checksum and Sequence Number Errors — the sibling AY/AZ integrity checks that detect corruption but not replay.
• PII Masking in Patron Data Exports — the tokenize-before-store discipline the dedup key reuses so no raw barcode is persisted.