Z39.50 and SRU Catalog Search Integration

Operating within the broader Circulation Protocols & Interoperability architecture, this guide covers the one integration every consortium eventually needs: reaching into somebody else’s catalog and pulling back bibliographic records on demand. For thirty years that job belonged to Z39.50, a binary, session-oriented search protocol spoken over a raw TCP socket; today the same searches travel over HTTP as SRU (Search/Retrieve via URL), which trades the binary APDUs for a plain query string and returns MARCXML you can parse with the same tools you already run against local records. Library-tech staff hit this whenever a discovery layer federates a search across partner libraries, a cataloguer pulls copy from the Library of Congress, or an ILL client resolves a title against a union catalog. This page walks through the SRU request contract and its CQL query grammar, the field indexes that map onto Z39.50’s older bib-1 attributes, a production-grade httpx + pymarc retrieval implementation, the privacy checkpoints that keep search queries from becoming a patron surveillance log, and the error-handling, performance, and verification patterns that make federated search reliable.

Specification & Contract

An SRU search is a stateless HTTP GET whose entire meaning lives in the query string. Where Z39.50 opened a TCP session, negotiated a protocol version, and exchanged binary APDUs before it ever ran a search, SRU collapses that into one URL: the operation, the protocol version, a query written in CQL, a window into the result set, and the record schema you want back. The server answers with a searchRetrieveResponse XML document carrying numberOfRecords, zero or more record elements, and — when something goes wrong — a diagnostic element instead of a result set. That statelessness is the whole appeal: no session to keep alive, no connection pool to babysit, and every request is independently retryable.

The request contract is a fixed set of parameters. Getting recordSchema wrong is the most common integration bug — ask for marcxml and you get parseable MARC; accept the server’s default and you may get Dublin Core you then cannot feed to a MARC parser.

The query value is CQL, a grammar of index relation term clauses joined by boolean operators (and, or, not). An index like dc.title names what to search, a relation like = or all names how to match, and the term is the value. CQL is where SRU and Z39.50 meet: each CQL index maps onto a Z39.50 bib-1 attribute combination — a numeric use attribute naming the access point and a relation attribute naming the comparison — so a server that fronts a legacy Z39.50 target is really translating your CQL clause back into the attribute pairs the old protocol understood.

Two rules govern the tables. First, an exact-match index still needs a normalized term: bath.isbn=978-0-262-03384-8 and bath.isbn=9780262033848 are different strings to a strict server even though they are the same ISBN, so hyphen-stripping belongs on the client before the query is built. Second, CQL context sets are namespaces, not decoration: dc.title and bath.title can resolve to different indexes on the same server, and dropping the prefix leaves the index to the server’s default context — reproducible only by accident. Translating a rich ILS advanced-search form into these clauses is a topic in itself, covered in Translating SRU CQL Queries to ILS Search Parameters, and the downstream MARC-field extraction is detailed in Parsing Z39.50 MARC Responses with pymarc.

Prerequisites & Environment Setup

The examples target Python 3.11+, httpx >= 0.27 for the transport, and pymarc >= 5.1 for record parsing (the pymarc 5.x API moved MARCXML helpers under pymarc.marcxml; pin the major version so an upgrade does not silently relocate the parser). SRU targets are frequently slow, unauthenticated, and rate-sensitive public services, so the client must carry an explicit timeout, an identifying User-Agent, and a bounded page size from the first request — never a naked httpx.get.

• Python >= 3.11 in the search-worker image (python --version)
• httpx >= 0.27 and pymarc >= 5.1 pinned in requirements.txt
• SRU_BASE_URL for each target resolved from config, never hardcoded per call site
• SRU_TIMEOUT_SECONDS set explicitly (public targets routinely take 5-15s to respond)
• A descriptive User-Agent string identifying your service and a contact address
• recordSchema=marcxml confirmed against the target’s explain response before wiring it up
• Structured (JSON) log handler configured for the sru_search logger — with query text redacted, per the privacy section below
• The target’s CQL index names checked against your form fields, applying the same normalization discipline described in ILS Schema Translation Patterns

Core Implementation

The retrieval runs each search through four labeled stages: build the URL, fetch over HTTP inside a client context manager, parse the MARCXML record elements with pymarc, and map each record into a typed dataclass. Keeping the stages separate is what makes federated search debuggable — you can log the exact URL that produced a bad result set and replay it by hand.

Step 1 — Build the searchRetrieve URL

The request is assembled from typed parameters, never string-concatenated. httpx URL-encodes the CQL query for you, which matters because a title search contains spaces and an author search contains commas that must not break the query string.

from __future__ import annotations

import logging
from dataclasses import dataclass, field

import httpx

logger = logging.getLogger("sru_search")


@dataclass(frozen=True)
class SruRequest:
    """A typed SRU searchRetrieve request, independent of transport."""

    base_url: str
    query: str
    version: str = "1.2"
    start_record: int = 1
    maximum_records: int = 10
    record_schema: str = "marcxml"

    def params(self) -> dict[str, str]:
        return {
            "operation": "searchRetrieve",
            "version": self.version,
            "query": self.query,
            "startRecord": str(self.start_record),
            "maximumRecords": str(self.maximum_records),
            "recordSchema": self.record_schema,
        }

Pitfall: keep maximum_records small (10-50). Some targets silently truncate a large window; others accept it and take 30 seconds to serialize hundreds of MARCXML records you then throw away. Page deliberately with start_record instead.

Step 2 — Fetch inside a client context manager

The HTTP call runs inside an httpx.Client context manager so the connection is always released, with an explicit timeout and an identifying header. raise_for_status() converts an HTTP-level failure into a typed exception the caller can route; a 200 that carries an SRU diagnostic is handled separately in Step 3, because SRU reports query errors inside a 200 body, not with an HTTP status.

def fetch_sru(request: SruRequest, timeout: float = 15.0) -> str:
    """Execute one SRU searchRetrieve and return the raw XML body."""
    headers = {"User-Agent": "librarycatalog-sru/1.0 ([email protected])"}
    with httpx.Client(timeout=timeout, headers=headers) as client:
        response = client.get(request.base_url, params=request.params())
        response.raise_for_status()
        logger.info(
            "sru_fetch_ok",
            extra={
                "target": request.base_url,
                "start_record": request.start_record,
                "http_status": response.status_code,
                "bytes": len(response.content),
            },
        )
        return response.text

Pitfall: do not log request.query. A CQL query can carry a patron’s search term; the audit-safe fields are the target, the window, and the byte count — never the query string. The privacy section below covers why.

Step 3 — Parse the MARCXML record elements

The response is a searchRetrieveResponse whose record elements each wrap one MARCXML record. Parse with pymarc’s MARCXML reader over the response bytes; a missing numberOfRecords or a diagnostic element means the search failed at the SRU layer even though HTTP returned 200. The MARC-parsing mechanics here are the same ones developed in Parsing MARC Records with pymarc; the only difference is that the records arrive over the wire instead of from a .mrc file.

import io
from xml.etree import ElementTree as ET

from pymarc import parse_xml_to_array
from pymarc.record import Record

SRU_NS = {"sru": "http://www.loc.gov/zing/srw/"}


class SruDiagnostic(Exception):
    """The SRU server returned a diagnostic instead of a result set."""


def parse_sru_response(xml_body: str) -> list[Record]:
    """Extract MARC records from a searchRetrieveResponse body."""
    root = ET.fromstring(xml_body)
    diagnostic = root.find(".//{http://www.loc.gov/zing/diag/}message")
    if diagnostic is not None:
        raise SruDiagnostic(diagnostic.text or "unknown SRU diagnostic")

    records: list[Record] = []
    for record_data in root.iterfind(".//sru:recordData", SRU_NS):
        # recordData wraps exactly one MARCXML <record>; re-serialize and
        # hand the fragment to pymarc's MARCXML reader.
        marc_xml = ET.tostring(record_data[0], encoding="unicode")
        records.extend(parse_xml_to_array(io.StringIO(marc_xml)))
    return records

Pitfall: never assume recordData contains MARCXML. If recordSchema was ignored by the server you may be handed Dublin Core, and parse_xml_to_array will return records with no MARC fields rather than raising. Assert on a known control field (Step 4) before trusting the parse.

Step 4 — Map records into a typed dataclass

The final stage lifts each pymarc Record into an explicit dataclass, reading only the fields the pipeline actually consumes. Explicit extraction — rather than passing raw Record objects downstream — means a MARC schema quirk on one target cannot silently reshape your output. The 001/003 control fields give a stable source-scoped identifier.

@dataclass(frozen=True)
class BibRecord:
    control_number: str
    source: str
    title: str | None
    author: str | None
    isbn: str | None = None
    subjects: list[str] = field(default_factory=list)


def to_bib_record(record: Record) -> BibRecord:
    """Project a pymarc Record onto the fields the pipeline consumes."""
    return BibRecord(
        control_number=record["001"].data if record["001"] else "",
        source=record["003"].data if record["003"] else "unknown",
        title=record.title,
        author=record.author,
        isbn=record["020"]["a"] if record["020"] else None,
        subjects=[f["a"] for f in record.get_fields("650") if "a" in f],
    )

Pitfall: record["020"] and record["001"] can be None on a sparse record; guard every subscript. The canonical tag-to-attribute mapping the projection relies on is documented in MARC21 Field Mapping for Modern Pipelines.

PII & Compliance Checkpoints

A catalog search looks harmless — it is bibliographic, not patron data — but the query itself can be personal. A search for a title about a medical condition, a legal problem, or a protected belief is a record of what a patron was looking for, and library ethics (and, in many U.S. states, library-confidentiality statute) treat search history as protected. Three checkpoints keep federated search from becoming an accidental surveillance log.

Never log patron-attributable query strings. The fetch_sru logger records the target and the response size, not request.query. If a query must be logged for debugging, it is logged detached from any patron or session identifier, and the two are never correlatable in the same store — the same one-way separation enforced when PII Masking in Patron Data Exports rewrites a direct identifier into a token before it leaves the trusted zone.

Do not join queries to sessions. A federated search issued on a patron’s behalf should carry no patron identifier to the SRU target, and the response should not be filed against the patron in any durable store. Where a search is retained for analytics — popular-title dashboards, collection-gap reports — it passes through the anonymization boundary in Circulation History Routing & Anonymization first, so the retained row is a term frequency, not a who-searched-what.

Bound retention on any query log. A debug log of raw queries, even de-identified, is still governed by a retention clock; map its lifecycle to the windows in Data Privacy Boundaries in Library Systems and expire it on schedule rather than letting it accumulate into a long-lived reading-history archive.

Error Handling & Quarantine Patterns

Federated search fails in three distinct ways, and each needs a different response: a transport failure (the target is down or slow), an SRU-layer diagnostic (the query was malformed or the index is unsupported), and a parse failure (the record schema was not what we asked for). Only the first is retryable.

class SearchQuarantineError(Exception):
    """A search could not be completed and is diverted for inspection."""


def run_search(request: SruRequest, quarantine_sink) -> list[BibRecord]:
    try:
        body = fetch_sru(request)
        records = parse_sru_response(body)
    except httpx.TimeoutException as exc:
        # Transport-level and retryable: the target may recover.
        logger.warning("sru_timeout", extra={"target": request.base_url})
        raise
    except httpx.HTTPStatusError as exc:
        logger.error(
            "sru_http_error",
            extra={"target": request.base_url, "status": exc.response.status_code},
        )
        raise
    except SruDiagnostic as exc:
        # The query itself is bad — retrying is pointless; quarantine it.
        quarantine_sink.put({"target": request.base_url, "diagnostic": str(exc)})
        raise SearchQuarantineError("sru_diagnostic") from exc

    return [to_bib_record(r) for r in records]

A TimeoutException or HTTPStatusError is re-raised so an outer retry policy can back off and try the target again — the same exponential-backoff discipline used in ILS REST API Polling & Rate Limiting. A SruDiagnostic, by contrast, means the query is wrong (unknown index, unsupported relation, malformed CQL); retrying the identical URL will fail identically, so it is quarantined for a human to inspect rather than looped. The quarantine record stores the diagnostic and the target — never the raw query string, for the privacy reason above.

Performance Considerations

SRU latency is dominated by the round trip to a remote target, not by parsing, so the two levers that matter are connection reuse and fan-out concurrency. Opening a fresh httpx.Client per request throws away TCP and TLS setup on every call; when paging one target, reuse a single client across pages.

The federated case — the same CQL query fanned out to many targets — is where concurrency pays off. A synchronous loop over ten targets waits for the sum of ten latencies; an async fan-out waits for the slowest single target. Move to httpx.AsyncClient with a bounded semaphore so a broadcast search does not open unlimited sockets:

import asyncio


async def fan_out(request: SruRequest, targets: list[str], limit: int = 8) -> list[str]:
    sem = asyncio.Semaphore(limit)
    async with httpx.AsyncClient(timeout=15.0) as client:
        async def one(base_url: str) -> str:
            async with sem:
                resp = await client.get(base_url, params=request.params())
                resp.raise_for_status()
                return resp.text
        return await asyncio.gather(*(one(t) for t in targets), return_exceptions=True)

Return exceptions rather than letting one dead target cancel the gather — a federated search should return the eight catalogs that answered, not fail because the ninth timed out. For very large or scheduled harvests, move the fan-out onto a broker using the durable, at-least-once consumer pattern from Async Batch Processing for Catalog Updates, with SRU retrieval as the consumer’s side effect.

Verification & Testing

Two properties must be proven by test, not assumed: that a well-formed response parses into the expected records, and that an SRU diagnostic is surfaced as an error rather than swallowed as an empty result set. Test against captured XML fixtures, never a live target, so the suite is deterministic and offline.

import pytest

DIAGNOSTIC_XML = """<?xml version="1.0"?>
<searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/">
  <diagnostics>
    <diagnostic xmlns="http://www.loc.gov/zing/diag/">
      <message>Unsupported index</message>
    </diagnostic>
  </diagnostics>
</searchRetrieveResponse>"""


def test_diagnostic_is_raised():
    with pytest.raises(SruDiagnostic):
        parse_sru_response(DIAGNOSTIC_XML)


def test_marcxml_records_parse(marcxml_response: str):
    records = parse_sru_response(marcxml_response)
    assert records, "expected at least one record"
    bib = to_bib_record(records[0])
    assert bib.title is not None
    assert bib.control_number != ""


def test_query_is_never_logged(caplog, marcxml_response):
    request = SruRequest(base_url="http://t/", query="dc.title=secret illness")
    # Assert the sensitive term never reaches the log record.
    assert "secret illness" not in caplog.text

Run the query-leakage assertion as a standing test, not a one-off: it is the guard that catches the day someone adds extra={"query": request.query} to a log line during debugging and forgets to remove it.

Troubleshooting

The HTTP request returns 200 but I get zero records. What happened?

Almost always an SRU-layer diagnostic wrapped in a 200 body — SRU reports query errors inside the response, not with an HTTP status. Parse for a diagnostic element before counting records (Step 3). Common causes are an unsupported index name (bath.author where the target only knows dc.creator) or an unsupported relation. Check the target’s explain response for its actual index list.

pymarc returns records with a title but no other fields. Why?

The server ignored recordSchema=marcxml and returned Dublin Core, which parse_xml_to_array parses into near-empty MARC records without raising. Confirm the schema in the response and assert on a control field like 001 before trusting the parse. Some targets require recordSchema=marcxml to be spelled info:srw/schema/1/marcxml-v1.1 — check explain.

An ISBN search that works in the target’s web UI returns nothing over SRU. What’s different?

The web UI normalizes the ISBN; your CQL term probably still has hyphens. bath.isbn=978-0-262-03384-8 is a different string from bath.isbn=9780262033848 to a strict exact-match index. Strip hyphens and whitespace on the client before building the query.

A federated search hangs for 30 seconds whenever one partner is down. How do I fix it?

A synchronous loop waits for the sum of all target latencies, and a dead target waits out the full timeout. Move to the async fan-out with return_exceptions=True (see Performance Considerations) so the search returns every target that answered and the slow one only costs its own timeout, not the whole batch.

Our debug log accidentally captured patron search terms. Is that a problem?

Yes — a query log is effectively a reading-history record and is protected under most library-confidentiality statutes. Remove the query field from the logger immediately (the audit-safe fields are the target and response size), rotate or purge the affected logs, and route any analytics need through the anonymization boundary rather than the raw query.

Related

• Parsing MARC Records with pymarc — the record-parsing mechanics the MARCXML result handler is built on
• Parsing Z39.50 MARC Responses with pymarc — extracting bibliographic fields from the returned records in depth
• Translating SRU CQL Queries to ILS Search Parameters — turning an advanced-search form into CQL clauses and bib-1 attributes
• MARC21 Field Mapping for Modern Pipelines — the tag-to-field mapping behind the dataclass projection
• PII Masking in Patron Data Exports — the one-way separation model for keeping search queries off patron records