Parsing Z39.50 MARC Responses with pymarc

This page answers one narrow operational question: why do MARC records returned in a Z39.50 or SRU result set fail to parse in pymarc — raising RecordLengthInvalid, blowing up with a UnicodeDecodeError, or emerging with mojibake in the title — and how do you make the reader choose the right decoder for what a server actually sent? It sits under Z39.50 and SRU Catalog Search Integration, which covers the search-and-retrieve contract end to end, and within the broader Circulation Protocols & Interoperability architecture. If you federate searches across a handful of consortium catalogs, you will eventually hit a server whose result set your parser cannot read, and the traceback will not tell you why.

Problem Framing

The symptom arrives as a traceback, not a warning, and it fires on the second record as often as the first — which is the tell. A federated search fans a query out to several targets, and one of them returns a result set that pymarc refuses to iterate:

Traceback (most recent call last):
  File "search/federate.py", line 88, in harvest
    for record in MARCReader(result_bytes):
  File ".../pymarc/reader.py", line 84, in __next__
    raise RecordLengthInvalid
pymarc.exceptions.RecordLengthInvalid

...or, on a different target in the same fan-out:

  File ".../pymarc/record.py", line 180, in decode_marc
    field_data = field_data.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 12:
    invalid continuation byte

Three distinct failures hide behind those two tracebacks. RecordLengthInvalid means MARCReader read the first five bytes of the leader as a record length and got something that is not a valid ISO2709 length — almost always because the payload is actually MARCXML (<?xml ...><collection>...) or SRU envelope XML, not binary MARC. The UnicodeDecodeError means the record is ISO2709 but is encoded in MARC-8 (or Latin-1), and the reader was told to assume UTF-8. A third, quieter variant produces no exception at all: the record parses, but a diacritic in a 245 title comes out as é instead of é because bytes were decoded with the wrong codec. All three trace back to the same mistake — handing pymarc a byte stream without first establishing what the server actually sent.

Root Cause

Z39.50 and its web-facing sibling SRU do not define a single wire format for records; they carry whatever record syntax the client negotiated or the SRU recordSchema parameter requested. A single consortium of a dozen targets will, in practice, hand you all of the following: ISO2709 binary MARC21 in UTF-8, ISO2709 binary in MARC-8, and MARCXML wrapped in an SRU <searchRetrieveResponse> envelope. The client code that works against your primary catalog assumes that catalog’s format and breaks the moment a different target answers.

Two independent axes have to be resolved before pymarc can read anything:

• Syntax — is each record ISO2709 binary or MARCXML? pymarc.MARCReader parses only ISO2709; MARCXML must go through pymarc.parse_xml_to_array. Feeding XML to MARCReader is exactly what raises RecordLengthInvalid, because the reader interprets <?x as a record-length header.
• Encoding — for ISO2709 records, is the character data MARC-8 or UTF-8? This is not a guess. Leader position 09 (byte index 9) is the encoding flag: a value of a means UTF-8 (Unicode), and a blank means MARC-8. pymarc can translate MARC-8 to Unicode for you, but only if you tell it to with to_unicode=True; and it will misread UTF-8 data as MARC-8 unless you set force_utf8=True when the flag says a.

The framing errors compound the encoding ones. Z39.50 hands back a batch — many records concatenated in one octet string — and each ISO2709 record self-delimits by its own leader length. If any record in the batch has a wrong length (a truncated or MARC-8-mislabelled record), the reader loses stream alignment and the next record fails, which is why the traceback so often points at a record that is itself valid. The same class of MARC-8-versus-UTF-8 mismatch is dissected for legacy ingest in handling UTF-8 encoding in legacy MARC records; the Z39.50 case is that problem arriving over the wire instead of from a file.

Solution

Resolve syntax and encoding before choosing a reader, then dispatch. Detect MARCXML by sniffing the leading non-whitespace bytes for an XML prolog or a MARC element; treat everything else as ISO2709 and let the per-record leader byte 9 decide the codec. Stream the batch record-by-record so one malformed record can be quarantined without discarding the rest of the result set. This is the same reader discipline documented for file-based ingest in parsing MARC records with pymarc, extended to cope with a mixed-syntax network source.

from __future__ import annotations

import logging
from collections.abc import Iterator
from io import BytesIO

from pymarc import MARCReader, Record, parse_xml_to_array
from pymarc.exceptions import RecordLengthInvalid, BaseAddressInvalid

logger = logging.getLogger("z3950.parse")

_XML_MARKERS = (b"<?xml", b"<collection", b"<record", b"<searchRetrieveResponse")


def _looks_like_marcxml(payload: bytes) -> bool:
    """True when the result set is MARCXML rather than ISO2709 binary."""
    head = payload.lstrip()[:64].lower()
    return any(head.startswith(marker) for marker in _XML_MARKERS)


def _uses_utf8(record_bytes: bytes) -> bool:
    """Leader position 09 is the MARC encoding flag: 'a' == Unicode/UTF-8."""
    return len(record_bytes) > 9 and record_bytes[9:10] == b"a"


def parse_result_set(payload: bytes, *, target_id: str) -> Iterator[Record]:
    """Yield MARC records from a Z39.50/SRU result set of unknown syntax.

    MARCXML is dispatched to parse_xml_to_array; ISO2709 is streamed through
    MARCReader with the codec chosen per-record from the leader flag. A record
    that fails to frame is logged and skipped so the batch keeps flowing.
    """
    if _looks_like_marcxml(payload):
        for record in parse_xml_to_array(BytesIO(payload)):
            if record is not None:
                yield record
        return

    force_utf8 = _uses_utf8(payload)
    reader = MARCReader(
        payload,
        to_unicode=True,       # translate MARC-8 into Python str
        force_utf8=force_utf8,  # honour the leader flag; do not guess
        utf8_handling="replace",
    )
    with reader:
        while True:
            try:
                record = next(reader)
            except StopIteration:
                break
            except (RecordLengthInvalid, BaseAddressInvalid) as exc:
                logger.warning(
                    "record_framing_failed",
                    extra={"target_id": target_id, "error": type(exc).__name__},
                )
                continue
            if record is None:
                logger.warning(
                    "record_decode_error",
                    extra={"target_id": target_id, "reason": str(reader.current_exception)},
                )
                continue
            yield record

Three decisions carry the fix. First, _looks_like_marcxml inspects the stream before instantiating a reader, so XML never reaches MARCReader and RecordLengthInvalid stops being a syntax-confusion symptom and becomes a genuine framing signal. Second, force_utf8 is derived from the leader flag rather than hard-coded, so a MARC-8 target and a UTF-8 target in the same fan-out are each decoded correctly. Third, the read loop catches RecordLengthInvalid and BaseAddressInvalid per record and continues, so one corrupt record in a batch of fifty costs you one record, not the whole result set. When a target advertises its recordSchema in the SRU response, prefer that over byte-sniffing — but keep the sniff as a fallback, because plenty of servers mislabel.

A note on the encoding flag: some targets set leader byte 9 to blank while actually emitting UTF-8, or vice versa. utf8_handling="replace" keeps a single mislabelled record from aborting the batch by substituting U+FFFD for bytes that will not decode, at the cost of a silently damaged character. Log those events; a rising record_decode_error rate against one target_id is your evidence that a specific server lies about its encoding and needs a per-target override.

Compliance or Privacy Impact

Bibliographic MARC records are not patron data — a 245 title and an 020 ISBN carry no personally identifiable information, so the parsed records themselves fall outside FERPA and GDPR scope. The privacy exposure in a Z39.50 integration lives one layer up, in the query. What a patron searched for, tied to their session or account, is patron activity, and it is easy to leak it into logs while debugging exactly the parse failures above. The extra={...} fields in the code deliberately carry target_id and error types, never the CQL query string or the user’s identifier.

• Keep query context out of parser logs. When you log a framing or decode failure, log the target and the record ordinal, not the search terms that produced the result set. If you must capture a failing query for reproduction, route it through the redaction discipline in PII Masking in Patron Data Exports before it lands in a durable log.
• Separate the two retention clocks. Cached bibliographic records can persist indefinitely; the association between a patron and the query that retrieved them cannot. Store parsed records and query-audit rows in different sinks so the search history can expire on its own schedule without deleting the catalog cache.

The parser is a good place to enforce this boundary because it is the seam where a network response becomes stored data — anything the parser does not log or persist cannot leak downstream.

Verification

Confirm the dispatcher reads all three shapes correctly with a test that feeds it a UTF-8 ISO2709 batch, a MARC-8 record, and a MARCXML payload, and asserts the decoded title is byte-clean in every case.

import pytest
from pymarc import Record, Field, Subfield

SECRET_TITLE = "Café society"  # é must survive the round trip


def _iso2709_utf8() -> bytes:
    record = Record(force_utf8=True)
    record.add_field(
        Field(tag="245", indicators=[" ", " "],
              subfields=[Subfield(code="a", value=SECRET_TITLE)])
    )
    return record.as_marc()  # leader byte 9 == 'a'


def test_iso2709_utf8_round_trips() -> None:
    records = list(parse_result_set(_iso2709_utf8(), target_id="test"))
    assert len(records) == 1
    assert records[0].title == SECRET_TITLE  # no mojibake


def test_marcxml_dispatches_without_recordlength_error() -> None:
    xml = (
        b'<?xml version="1.0"?>'
        b'<collection xmlns="http://www.loc.gov/MARC21/slim">'
        b'<record><datafield tag="245" ind1=" " ind2=" ">'
        b'<subfield code="a">Plain title</subfield>'
        b"</datafield></record></collection>"
    )
    records = list(parse_result_set(xml, target_id="test"))
    assert records[0].title == "Plain title"  # never raises RecordLengthInvalid


def test_malformed_record_is_skipped_not_fatal() -> None:
    good = _iso2709_utf8()
    payload = good + b"00072garbage-length-header" + good
    records = list(parse_result_set(payload, target_id="test"))
    assert len(records) >= 2  # the two valid records survive the bad one

For a running federation, add a per-target invariant: after each harvest, assert that the count of yielded records plus logged framing failures equals the numberOfRecords the SRU envelope promised. A persistent gap against one target_id means records are silently vanishing — usually a mislabelled encoding flag — and tells you which server needs a per-target override before the next term’s federated search.

Related

• Z39.50 and SRU Catalog Search Integration — the parent guide to the search-and-retrieve contract this parser sits at the end of.
• Circulation Protocols & Interoperability — the wider protocol architecture covering how catalogs exchange records and circulation state.
• Translating SRU CQL Queries to ILS Search Parameters — the query side of the same integration, which produces the result set this page parses.
• Parsing MARC Records with pymarc — the file-based reader discipline this network parser extends.