Recovering from NCIP CheckOutItem Timeouts
This page answers one narrow operational question: when an NCIP CheckOutItem request to a responder agency times out at the HTTP layer, how do you find out whether the loan actually committed on the responder side before you decide to retry? Retrying blindly risks a double checkout on the same item; giving up risks a loan that exists on the responder but not on your books. It sits under NCIP Resource-Sharing Workflows, which covers the initiator/responder message exchange end to end, and within the broader Circulation Protocols & Interoperability architecture. If you run resource sharing between consortium members over NCIP, this ambiguous timeout is one you will eventually hit on a slow responder night.
Problem Framing
The symptom is a stalled request, not a rejection. The initiator sends a CheckOutItem to the responder agency, the socket stays open past the read deadline, and the HTTP client aborts. There is no NCIP response document to parse — no CheckOutItemResponse, no Problem element — only a transport-layer exception. The loan is now in a state your database cannot describe: neither confirmed nor refused.
An httpx timeout on this call surfaces exactly like this:
Traceback (most recent call last):
File "resource_sharing/ncip_client.py", line 88, in check_out_item
response = client.post(responder_url, content=request_xml)
File ".venv/lib/python3.12/site-packages/httpx/_client.py", line 1146, in post
return self.request("POST", url, ...)
...
httpx.ReadTimeout: The read operation timed out
resource_sharing.ncip WARNING checkout_timeout request_id=RS-2026-0417-118 \
responder=NORTHBRANCH item=31234000567890 elapsed_ms=30014 outcome=unknown
The outcome=unknown is the whole problem. The responder may have received the request, committed the loan to its circulation database, and only then failed to return the response — a lost reply, not a lost request. Or it may never have processed the request at all. From the initiator’s side these two cases are indistinguishable, and the naive reflex — catch the timeout and re-send CheckOutItem — turns the lost-reply case into a double checkout, with the same physical item now recorded on loan twice on the responder side.
Root Cause
The failure is a category error: treating an HTTP timeout as a transactional outcome. It is not one. A timeout tells you only that a response did not arrive within the deadline. It says nothing about whether the request was received, whether it was processed, or whether its side effects committed. The responder’s circulation transaction and your HTTP client’s read deadline are wholly independent clocks, and the responder can easily commit the loan at t=29s while your client gives up at t=30s.
Compounding this, NCIP CheckOutItem is not inherently idempotent. Each call, as most responders implement it, is an unconditional “create a loan for this item to this user” command. The protocol’s RequestId element exists to correlate a request with its response, but the NCIP standard does not require responders to deduplicate on it — a compliant responder is free to honour a second CheckOutItem for the same item as a fresh transaction. So the initiator cannot lean on the protocol to make a blind retry safe; two identical CheckOutItem messages can produce two loans.
The fix is to stop guessing and go find out. Before any retry, the initiator must query the responder for the item’s actual circulation state and let that observed state — not the timeout — drive the decision. This is the same reconcile-before-write discipline the exchange relies on elsewhere in NCIP Resource-Sharing Workflows: never mutate on an assumption when a read can give you ground truth.
Solution
Replace the blind retry with a reconcile-then-retry pattern. On timeout, the initiator issues a read-only LookupItem (or a LookupUser loan-list query) to the responder to establish the item’s true state. If the item already shows as on loan to this patron, the loan committed and you simply accept it — record it locally and stop. If the item shows as available, the request never committed, so you re-send CheckOutItem once, reusing the original RequestId so a responder that does deduplicate treats it as the same logical transaction.
from __future__ import annotations
import logging
from dataclasses import dataclass
from enum import Enum
import httpx
logger = logging.getLogger("ncip.checkout")
TIMEOUT = httpx.Timeout(connect=5.0, read=30.0, write=5.0, pool=5.0)
MAX_RECONCILE_ATTEMPTS = 3
class LoanState(Enum):
ON_LOAN = "on_loan" # responder already committed the checkout
AVAILABLE = "available" # responder never committed it
UNKNOWN = "unknown" # reconcile itself could not resolve state
class NcipReconcileError(RuntimeError):
"""Raised when the responder's true state cannot be determined."""
@dataclass(frozen=True)
class CheckoutOutcome:
request_id: str
state: LoanState
committed: bool
def _post_ncip(client: httpx.Client, url: str, xml: str, *, request_id: str) -> httpx.Response:
"""POST an NCIP document, letting httpx.TimeoutException propagate."""
response = client.post(url, content=xml.encode("utf-8"))
response.raise_for_status()
return response
def lookup_item_state(
client: httpx.Client,
url: str,
*,
item_id: str,
request_id: str,
) -> LoanState:
"""Read-only LookupItem: ask the responder for the item's true state.
Carries no patron PII in the request or the log line — only the opaque
item barcode and the correlation RequestId.
"""
xml = build_lookup_item(item_id=item_id, request_id=request_id)
try:
response = _post_ncip(client, url, xml, request_id=request_id)
except httpx.TimeoutException:
logger.warning("reconcile_timeout", extra={"request_id": request_id})
return LoanState.UNKNOWN
return parse_item_circulation_state(response.text)
def checkout_with_reconcile(
client: httpx.Client,
url: str,
*,
item_id: str,
user_id: str,
request_id: str,
) -> CheckoutOutcome:
"""CheckOutItem that resolves timeout ambiguity by reconciling first.
On an HTTP timeout the outcome is unknown, so we query the responder for
the item's real state and only retry when it is provably safe to do so.
"""
checkout_xml = build_check_out_item(
item_id=item_id, user_id=user_id, request_id=request_id
)
for attempt in range(1, MAX_RECONCILE_ATTEMPTS + 1):
try:
_post_ncip(client, url, checkout_xml, request_id=request_id)
logger.info(
"checkout_committed",
extra={"request_id": request_id, "attempt": attempt},
)
return CheckoutOutcome(request_id, LoanState.ON_LOAN, committed=True)
except httpx.TimeoutException:
logger.warning(
"checkout_timeout_reconciling",
extra={"request_id": request_id, "attempt": attempt},
)
state = lookup_item_state(
client, url, item_id=item_id, request_id=request_id
)
if state is LoanState.ON_LOAN:
# Lost reply, not lost request: the loan already committed.
logger.info(
"checkout_reconciled_committed",
extra={"request_id": request_id, "attempt": attempt},
)
return CheckoutOutcome(request_id, LoanState.ON_LOAN, committed=True)
if state is LoanState.UNKNOWN:
continue # responder unreachable; try to reconcile again
# AVAILABLE: nothing committed, safe to re-send the same RequestId.
logger.info(
"checkout_retry_safe",
extra={"request_id": request_id, "attempt": attempt},
)
raise NcipReconcileError(
f"could not resolve checkout state for request {request_id}"
)
The behavioural change is that the timeout no longer decides anything. Before, a caught ReadTimeout led straight to a re-send and a potential double loan. After, the timeout only triggers a LookupItem, and the item’s observed state decides the next move: ON_LOAN means accept, AVAILABLE means retry with the same RequestId, and an unresolved reconcile is bounded by MAX_RECONCILE_ATTEMPTS so the pattern can never spin forever. Reusing the original RequestId on the retry is what keeps a deduplicating responder from creating a second transaction, and it keeps your local correlation stable across attempts.
Compliance or Privacy Impact
Reconciliation adds a second network call, and that call must not become a privacy leak. Three points matter.
- Reconcile with the item, not the patron.
LookupItemkeys on the item barcode alone, so prefer it overLookupUserwhen the item identifier is enough to resolve state. When you must useLookupUserto read a loan list, treat the responder’s returned patron block as untrusted PII in transit — parse only the loan you care about and discard the rest. - Keep PII out of the logs. The recovery path is exactly where hurried logging leaks names and card numbers. Every log line above carries only the opaque
request_idand item barcode — neveruser_id, patron name, or address. Apply the same field-level redaction the export path uses, described in PII Masking in Patron Data Exports, to any responder document you persist for debugging. - The correlation key is not PII, but the join is. A
RequestIdis a safe surrogate to log and retry on. The mapping from that request to a real patron lives in your own records; keep it there, and follow the resolution rules in Mapping NCIP LookupUser Responses to ILS Patrons when you do need to bind a responder reply back to a local borrower.
The pattern narrows risk rather than widening it — it prevents a duplicate loan that would attach a second circulation event to a patron — but it shifts weight onto the correctness of parse_item_circulation_state. A parser that misreads AVAILABLE for ON_LOAN re-sends a checkout it should have accepted, so that function deserves the verification below.
Verification
Confirm the recovery logic with a test that simulates a lost reply — the responder committed the loan, then the read timed out — and asserts the initiator reconciles to a committed outcome without ever sending a second checkout.
import httpx
import pytest
def test_lost_reply_reconciles_to_committed(monkeypatch) -> None:
calls: list[str] = []
def fake_post(self, url, content): # noqa: ANN001
body = content.decode()
if "CheckOutItem" in body:
calls.append("checkout")
raise httpx.ReadTimeout("read timed out")
calls.append("lookup")
return httpx.Response(200, text=ITEM_ON_LOAN_XML)
monkeypatch.setattr(httpx.Client, "post", fake_post)
with httpx.Client(timeout=TIMEOUT) as client:
outcome = checkout_with_reconcile(
client, "https://responder.example/ncip",
item_id="31234000567890", user_id="P-99", request_id="RS-1",
)
assert outcome.committed is True
assert outcome.state is LoanState.ON_LOAN
# Exactly one checkout attempt: the lost reply was NOT re-sent blindly.
assert calls.count("checkout") == 1
assert calls.count("lookup") == 1
def test_available_item_triggers_bounded_retry(monkeypatch) -> None:
with httpx.Client(timeout=TIMEOUT) as client:
# Item stays AVAILABLE and every checkout times out: retries are bounded.
with pytest.raises(NcipReconcileError):
checkout_with_reconcile(
client, "https://responder.example/ncip",
item_id="31234000000001", user_id="P-1", request_id="RS-2",
)
For a running exchange, emit a metric on the checkout_reconciled_committed event and alert if its rate climbs — a rising count of lost replies points at responder-side latency creeping toward your read deadline, which you fix by raising read in the httpx.Timeout rather than by retrying harder. Track any NcipReconcileError as a hard failure that pages a human: it means the responder’s state stayed unknown across every bounded attempt, and that loan needs manual settlement before the patron is charged for an item they may never have received.
Related
- NCIP Resource-Sharing Workflows — the parent guide to the initiator/responder message exchange this recovery pattern plugs into.
- Circulation Protocols & Interoperability — the wider architecture covering NCIP, SIP2, and self-check circulation sync.
- Mapping NCIP LookupUser Responses to ILS Patrons — resolving a responder’s patron block back to a local borrower during reconciliation.
- PII Masking in Patron Data Exports — the redaction contract that keeps patron identifiers out of reconciliation logs.