NCIP Resource-Sharing Workflows

Operating within the broader Circulation Protocols & Interoperability architecture, this guide covers the protocol that lets one library’s system act on a circulation transaction inside a different library’s ILS: the NISO Circulation Interchange Protocol (NCIP, standardized as ANSI/NISO Z39.83). Resource sharing is the reason the protocol exists — a patron discovers a title their home branch does not own, and a borrowing system has to look that patron up, request the item, check it out, and later check it back in, all against an owning institution’s catalog it does not control and whose patron database it must never see in full. NCIP frames each of those actions as a named XML message exchanged over HTTP between an initiator (the borrowing or discovery system) and a responder (the owning ILS), across an agency boundary that is also a privacy boundary. This page walks through the message contract and the elements that cross that boundary, a production-grade Python implementation that builds and POSTs a LookupUser and CheckOutItem request and parses the response safely, the identity-minimization checkpoints that keep it defensible, and the error-handling, performance, and verification patterns that make it reliable in a live consortium.

Specification & Contract

NCIP is a request/response protocol: the initiator builds one XML document per action, POSTs it to the responder’s NCIP endpoint, and reads back a single XML response. Every message shares an envelope — the root NCIPMessage element carrying a version attribute, then exactly one service element (the request or its matching response) inside it. There is no session and no sequence counter at the protocol level; each message is self-contained and must carry, in its own body, the agency identifiers that say who is asking and on whose behalf.

The contract has two halves that matter for an implementation: the set of service messages you will actually exchange for resource sharing, and the response envelope that tells you whether the action succeeded. The initiator names the service (LookupUser, RequestItem, CheckOutItem, CheckInItem); the responder either returns the requested data inside a <ServiceResponse> or replaces it with a <Problem> structure. Getting the element names and their nesting exactly right is the whole game — NCIP responders are strict, and a mis-cased <UserIdentifierValue> will draw an unhelpful parse error rather than a useful Problem.

Every request carries an InitiationHeader and every response an equivalent ResponseHeader. The header is where the agency identities live, and it is the machinery that lets a single responder endpoint serve many borrowing partners without confusing them.

Two rules govern this contract. First, a NCIP response is either a service response or a Problem, never a partial mix: the presence of a <Problem> element is the single authoritative failure signal, and code that only checks the HTTP status will treat an application-level Unknown Item (delivered as HTTP 200 with a Problem body) as a success. Second, the RequestId is the correlation spine of a multi-message workflow: the value the responder mints in its RequestItem response is the value your CheckOutItem must echo, and losing it means you cannot tie a checkout to the hold that authorized it. Mapping a LookupUser response back to a local patron record is subtle enough to have its own treatment in Mapping NCIP LookupUser Responses to ILS Patrons, and choosing NCIP over a lighter self-service protocol is weighed in SIP2 vs NCIP: Choosing the Right Circulation Protocol.

Prerequisites & Environment Setup

The examples target Python 3.11+, httpx for the HTTP transport (its explicit timeout model and connection pooling matter for a protocol with no session), and defusedxml wrapping lxml for parsing. Never parse a NCIP response with the standard-library xml.etree defaults or a bare lxml parser — a responder is a remote system on another network, and an untrusted XML document is an XXE and billion-laughs vector. defusedxml disables entity expansion and external DTD loading, which is exactly the guard a cross-agency protocol needs.

Agency symbols and endpoint URLs are configuration, resolved at runtime — never literals compiled into the client, because the same code serves a dozen consortium partners whose symbols and endpoints differ.

• Python >= 3.11 in the resource-sharing worker image (python --version)
• httpx >= 0.27 pinned in requirements.txt for the transport layer
• defusedxml >= 0.7 and lxml >= 5.0 pinned — parsing goes through defusedxml.lxml, never bare lxml.etree
• NCIP_ENDPOINT_URL per partner resolved from config, not hardcoded
• NCIP_FROM_AGENCY (your borrowing symbol) and the partner’s NCIP_TO_AGENCY supplied per request
• mTLS client cert or a bearer token configured if the responder requires transport authentication
• Structured (JSON) log handler configured for the ncip_workflow logger, shipped to append-only storage
• The local patron-schema mapping reviewed against each responder’s UserElementType output — Sierra, Alma, Polaris, and Symphony populate these differently, and the same normalization discipline described in ILS Schema Translation Patterns applies before a NCIP identity is trusted

Core Implementation

The workflow runs each action through four labeled stages: build the request XML, POST it under an explicit timeout, parse the response safely, then branch on Problem versus service data. Keeping the stages separate is what makes a NCIP exchange auditable — a reviewer can point at any field in the outbound document and name the code that produced it, and at any decision and name the response element that drove it.

Step 1 — Model the exchange with typed dataclasses

Nothing is built from loose dicts. Typed dataclasses pin the agency identity and the item/user identifiers so a missing ToAgencyId is a construction error, not a silent empty element the responder rejects.

from __future__ import annotations

import logging
from dataclasses import dataclass

logger = logging.getLogger("ncip_workflow")

NCIP_NS = "http://www.niso.org/2008/ncip"


@dataclass(frozen=True, slots=True)
class AgencyContext:
    """Immutable agency identity for one NCIP exchange."""

    from_agency: str
    to_agency: str


@dataclass(frozen=True, slots=True)
class LookupUserRequest:
    agency: AgencyContext
    user_id: str


@dataclass(frozen=True, slots=True)
class CheckOutItemRequest:
    agency: AgencyContext
    user_id: str
    item_id: str
    request_id: str

Step 2 — Build the request XML

The document is assembled with lxml.etree element construction rather than string templating, so values are escaped and the namespace is applied once at the root. String-formatting a patron identifier straight into XML is how an ampersand in a name produces a malformed document the responder silently drops.

from lxml import etree


def _header(agency: AgencyContext) -> etree._Element:
    """Build the InitiationHeader carrying From/To agency symbols."""
    header = etree.Element(f"{{{NCIP_NS}}}InitiationHeader")
    for tag, value in (("FromAgencyId", agency.from_agency),
                       ("ToAgencyId", agency.to_agency)):
        agency_el = etree.SubElement(header, f"{{{NCIP_NS}}}{tag}")
        inner = etree.SubElement(agency_el, f"{{{NCIP_NS}}}AgencyId")
        inner.text = value
    return header


def build_lookup_user(req: LookupUserRequest) -> bytes:
    """Serialize a LookupUser request document to UTF-8 bytes."""
    root = etree.Element(f"{{{NCIP_NS}}}NCIPMessage", version="2.02")
    service = etree.SubElement(root, f"{{{NCIP_NS}}}LookupUser")
    service.append(_header(req.agency))
    user = etree.SubElement(service, f"{{{NCIP_NS}}}UserId")
    value = etree.SubElement(user, f"{{{NCIP_NS}}}UserIdentifierValue")
    value.text = req.user_id
    # Ask only for the loanability signals the borrow decision needs.
    for element_type in ("Block Or Trap", "User Privilege"):
        desired = etree.SubElement(service, f"{{{NCIP_NS}}}UserElementType")
        desired.text = element_type
    return etree.tostring(root, xml_declaration=True, encoding="UTF-8")

Pitfall: request only the UserElementType values the borrow decision actually needs. Asking for Name Information or User Address Information pulls patron PII across the agency boundary you are trying to keep thin — the minimization argument is in the compliance section below.

Step 3 — POST the request under an explicit timeout and parse safely

httpx.Client is used as a context manager so the connection pool is always closed, and an explicit Timeout is set because a NCIP responder is a remote ILS that can and will hang. The response is parsed through defusedxml.lxml, never bare lxml.

import httpx
from defusedxml.lxml import fromstring as safe_fromstring
from defusedxml.common import DefusedXmlException


class NCIPTransportError(Exception):
    """The NCIP request never produced a usable HTTP response."""


class NCIPParseError(Exception):
    """The response body was not well-formed or safe NCIP XML."""


def post_ncip(endpoint: str, body: bytes, *, timeout_s: float = 10.0) -> etree._Element:
    """POST a NCIP document and return the parsed, XXE-guarded response root."""
    try:
        with httpx.Client(timeout=httpx.Timeout(timeout_s)) as client:
            resp = client.post(
                endpoint,
                content=body,
                headers={"Content-Type": "application/xml; charset=utf-8"},
            )
            resp.raise_for_status()
    except httpx.TimeoutException as exc:
        logger.warning("ncip request timed out", extra={"endpoint": endpoint})
        raise NCIPTransportError("timeout") from exc
    except httpx.HTTPError as exc:
        raise NCIPTransportError(str(exc)) from exc

    try:
        return safe_fromstring(resp.content)
    except (DefusedXmlException, etree.XMLSyntaxError) as exc:
        raise NCIPParseError("unsafe or malformed NCIP response") from exc

Pitfall: a NCIP responder frequently returns HTTP 200 while the body carries a <Problem>. raise_for_status() therefore does not mean success — the application-level check in Step 4 is mandatory, not optional.

Step 4 — Branch on Problem versus service data

The response is only trustworthy after the ResponseHeader is checked for a Problem. This is the single decision that separates the success path from quarantine.

@dataclass(frozen=True, slots=True)
class NCIPProblem:
    problem_type: str
    detail: str | None


def _q(local_name: str) -> str:
    return f"{{{NCIP_NS}}}{local_name}"


def extract_problem(root: etree._Element) -> NCIPProblem | None:
    """Return a structured Problem if the responder reported one, else None."""
    problem = root.find(f".//{_q('Problem')}")
    if problem is None:
        return None
    ptype = problem.findtext(_q("ProblemType"), default="Unknown")
    detail = problem.findtext(_q("ProblemDetail"))
    return NCIPProblem(problem_type=ptype.strip(), detail=detail)


def lookup_user(endpoint: str, req: LookupUserRequest) -> etree._Element:
    """Run a LookupUser exchange, raising on an application-level Problem."""
    root = post_ncip(endpoint, build_lookup_user(req))
    problem = extract_problem(root)
    if problem is not None:
        logger.warning(
            "ncip lookup returned a problem",
            extra={"problem_type": problem.problem_type,
                   "to_agency": req.agency.to_agency},
        )
        raise NCIPProblemError(problem)
    return root

Pitfall: search for Problem with a namespaced, descendant-scoped find (.//{ns}Problem), not a bare root.find("Problem"). NCIP nests the problem inside the ResponseHeader, and a non-namespaced lookup will never match a correctly namespaced document — making every failure look like a success.

PII & Compliance Checkpoints

A resource-sharing exchange is a controlled disclosure of a patron’s identity to an institution that is not their home library, so identity minimization is the governing constraint, not an afterthought. Three checkpoints keep it defensible.

Send the least identity that resolves the patron. The borrowing agency should transmit an AgencyId-scoped UserIdentifierValue — ideally an opaque, agency-issued surrogate rather than a barcode or a name — and request only the UserElementType block-and-privilege flags the loan decision needs. The full patron record must never cross the boundary; the same tokenization and field-suppression discipline documented in PII Masking in Patron Data Exports applies to the outbound UserId so the responder learns that this patron may borrow, not who this patron is.

Minimize what the responder returns, too. A LookupUser response can carry name and address blocks if you ask for them. Do not — and if a responder volunteers them anyway, drop them before the value is persisted, following the minimization principle in Data Privacy Boundaries in Library Systems. What you retain locally about a cross-agency loan is itself circulation history and inherits the one-way anonymization boundary in Circulation History Routing & Anonymization.

Log the transaction, not the person. Every NCIP exchange must produce exactly one structured audit line carrying the agency symbols, the service name, the RequestId, and the outcome (success or ProblemType) — but never the raw UserIdentifierValue, the patron name, or a ProblemDetail that quotes them. An audit trail that reproduces the patron identity has just created a second copy of the data the boundary exists to protect.

Error Handling & Quarantine Patterns

The workflow distinguishes three failure classes and never lets one contaminate the others. A transport failure means the message may not have been delivered; a parse failure means the response is unusable; an application Problem means the responder understood the request and refused it. Only the first is a candidate for a blind retry.

class NCIPProblemError(Exception):
    """The responder returned a NCIP Problem for this request."""

    def __init__(self, problem: NCIPProblem) -> None:
        super().__init__(problem.problem_type)
        self.problem = problem


def run_checkout(endpoint: str, req: CheckOutItemRequest, quarantine_sink) -> bool:
    """Attempt one CheckOutItem; quarantine anything that is not a clean success."""
    try:
        root = post_ncip(endpoint, build_check_out_item(req))
        problem = extract_problem(root)
        if problem is not None:
            raise NCIPProblemError(problem)
        return True
    except NCIPTransportError:
        # May be a lost request: safe to retry with backoff and the same RequestId,
        # because RequestId makes a replayed CheckOutItem idempotent at the responder.
        raise
    except (NCIPParseError, NCIPProblemError) as exc:
        # Responder answered but the exchange cannot complete: park it for a human.
        quarantine_sink.put({
            "service": "CheckOutItem",
            "request_id": req.request_id,
            "to_agency": req.agency.to_agency,
            "reason": type(exc).__name__,
        })
        logger.warning("checkout quarantined",
                       extra={"request_id": req.request_id,
                              "reason": type(exc).__name__})
        return False

A NCIPTransportError is re-raised so an outer retry loop can back off and resend — and because the CheckOutItem echoes the responder-minted RequestId, a resend that duplicates a checkout the responder already recorded is idempotent rather than a double loan. A NCIPProblemError is not retried blindly: a User Blocked or Unknown Item will fail identically on every attempt, so it goes to quarantine for human resolution. The quarantine record stores the RequestId and the ProblemType, never the patron identifier, so the quarantine queue does not become an unminimized shadow copy of the exchange. Tuning the backoff on the transport branch, including how long to keep retrying a specific CheckOutItem, is worked through in Recovering from NCIP CheckOutItem Timeouts; the circuit-breaker pattern that stops a dead responder from stalling the whole queue is in Implementing Circuit Breakers for ILS API Timeouts.

Performance Considerations

NCIP has no session, so every action pays a full connection setup unless you pool. The dominant cost in a busy consortium is not XML construction — it is round-trips to a remote ILS whose latency you do not control. Two disciplines keep throughput sane.

Reuse the connection pool. Construct one httpx.Client per worker (or per partner endpoint) and keep it open across many exchanges rather than opening a fresh client per message, so keep-alive and TLS session resumption amortize the setup cost:

def make_pooled_client(timeout_s: float = 10.0) -> httpx.Client:
    """A long-lived, connection-pooled client shared across NCIP exchanges."""
    limits = httpx.Limits(max_keepalive_connections=20, max_connections=40)
    return httpx.Client(timeout=httpx.Timeout(timeout_s), limits=limits)

Parse lazily and bound the document. A LookupUser response with unrequested name and address blocks is larger and slower to walk; requesting only the block-and-privilege element types keeps the response small and the find cheap. For high-volume resource sharing where many partners are polled, move each exchange onto a broker rather than a synchronous request thread — the durable, at-least-once consumer pattern from Async Batch Processing for Catalog Updates applies unchanged, with the NCIP exchange as the consumer’s side effect, and the throttle discipline in ILS REST API Polling & Rate Limiting keeps a single partner endpoint from being hammered into a rate-limited stall.

Verification & Testing

Two properties must be proven by test, not assumed: that a Problem response is never mistaken for success, and that outbound documents are well-formed and namespaced. Both are cheap to assert against fixture XML and expensive to discover in production.

import pytest

PROBLEM_XML = b"""<?xml version="1.0" encoding="UTF-8"?>
<NCIPMessage xmlns="http://www.niso.org/2008/ncip" version="2.02">
  <LookupUserResponse>
    <ResponseHeader>
      <Problem>
        <ProblemType>User Authentication Failed</ProblemType>
        <ProblemDetail>barcode not recognized</ProblemDetail>
      </Problem>
    </ResponseHeader>
  </LookupUserResponse>
</NCIPMessage>"""


def test_problem_response_is_detected():
    root = safe_fromstring(PROBLEM_XML)
    problem = extract_problem(root)
    assert problem is not None
    assert problem.problem_type == "User Authentication Failed"


def test_lookup_user_document_is_namespaced_and_wellformed():
    body = build_lookup_user(
        LookupUserRequest(AgencyContext("BORROW", "OWNER"), user_id="U1"))
    root = safe_fromstring(body)
    assert root.tag == f"{{{NCIP_NS}}}NCIPMessage"
    assert root.find(f".//{_q('UserIdentifierValue')}").text == "U1"


def test_from_and_to_agency_are_present():
    body = build_lookup_user(
        LookupUserRequest(AgencyContext("BORROW", "OWNER"), user_id="U1"))
    root = safe_fromstring(body)
    values = [e.text for e in root.iter(_q("AgencyId"))]
    assert values == ["BORROW", "OWNER"]


def test_xxe_payload_is_refused():
    xxe = b"""<?xml version="1.0"?>
    <!DOCTYPE r [<!ENTITY x SYSTEM "file:///etc/passwd">]>
    <NCIPMessage xmlns="http://www.niso.org/2008/ncip"><r>&x;</r></NCIPMessage>"""
    with pytest.raises(Exception):
        safe_fromstring(xxe)

Run the Problem-detection assertion against a captured sample of each partner’s actual responses, not just the synthetic fixture — responders vary in exactly how they nest and spell ProblemType, and a real capture catches a partner whose casing your descendant find would otherwise miss.

Troubleshooting

LookupUser returns HTTP 200 but the checkout never happens. Why?

The 200 body almost certainly contains a <Problem> — commonly User Blocked or User Authentication Failed — and code that trusts the status code alone skips it. NCIP delivers application failures inside an HTTP-success envelope. Run extract_problem on every response before touching the service data, as in Step 4.

My find("Problem") never matches even though the response clearly has one. What is wrong?

The document is namespaced and your lookup is not. NCIP nests Problem inside ResponseHeader under the http://www.niso.org/2008/ncip namespace, so a bare root.find("Problem") returns None on every real response. Use a namespaced, descendant-scoped query: root.find(".//{http://www.niso.org/2008/ncip}Problem").

The responder rejects my document as malformed when a patron name contains an ampersand. How do I fix it?

You are building the XML with string formatting instead of element construction, so the raw & breaks the document. Build with lxml.etree element and SubElement calls and set .text, which escapes reserved characters automatically, as shown in Step 2. Never f"..."-interpolate a value into an XML string.

CheckOutItem times out intermittently and I am afraid a retry will double-loan the item. What is safe?

Because CheckOutItem carries the responder-minted RequestId, a resend of the same request is idempotent at a conformant responder — it recognizes the RequestId and does not create a second loan. Retry the transport failure with backoff and the identical RequestId; the detailed timeout-recovery policy is in Recovering from NCIP CheckOutItem Timeouts.

One dead partner endpoint is stalling my whole resource-sharing queue. What do I do?

A synchronous client blocked on a hung responder holds a worker until timeout, and enough of them drain the pool. Wrap per-partner calls in a circuit breaker that trips after repeated timeouts and fast-fails until the endpoint recovers, following Implementing Circuit Breakers for ILS API Timeouts, and move the exchanges onto a broker so a slow partner backs up only its own lane.

Related

• Mapping NCIP LookupUser Responses to ILS Patrons — reconciling an agency-scoped UserId to a local patron record across vendor schemas
• Recovering from NCIP CheckOutItem Timeouts — idempotent retry and RequestId replay when a checkout hangs
• SIP2 vs NCIP: Choosing the Right Circulation Protocol — when a lighter self-service protocol beats full resource-sharing messaging
• SIP2 Protocol Integration for Self-Service Circulation — the field-delimited alternative for single-institution self-check
• PII Masking in Patron Data Exports — the tokenization discipline that keeps the outbound UserId from carrying identity across the agency boundary