Library Catalog Data Sync Pipelines

Handling UTF-8 Encoding in Legacy MARC Records

This page answers one narrow operational question: why does a legacy MARC21 record ingest without an exception yet land in your catalog with garbled diacritics — Beyoncé becoming Beyoncé, or an author’s name silently losing its accents — and how do you make the decode step deterministic instead of lossy? It sits under MARC21 Field Mapping for Modern Pipelines, which covers the full mapping contract from raw bytes to canonical object, and within the broader Core Architecture & Catalog Standards architecture. If you ingest pre-2000s ILS exports, vendor batch feeds, or union-catalog dumps, the mismatch between what the MARC leader declares and what the variable fields actually contain is the failure you are chasing.

Encoding recovery decision matrix: Leader/09 declaration crossed with actual payload bytes A two-by-two matrix. The rows are what Leader byte 09 declares: 0x20 space declares MARC-8, or 0x61 lowercase a declares UTF-8. The columns are what a scan of the variable-field payload actually finds: MARC-8 bytes with no 0xC2 to 0xF4 lead bytes, or UTF-8 lead bytes in the 0xC2 to 0xF4 range. The four cells give the decode outcome. Declared MARC-8 with MARC-8 bytes is clean and routes to marc8_to_unicode. Declared MARC-8 with UTF-8 bytes is recoverable drift and routes to a forced strict UTF-8 decode with a logged warning. Declared UTF-8 with MARC-8 bytes is unrecoverable drift that raises and is sent to quarantine. Declared UTF-8 with UTF-8 bytes is clean and routes to a strict UTF-8 decode. Below the matrix, the unrecoverable cell flows down into a quarantine dot-send box carrying only the leader hex and no field bytes, which then flows into a durable, replayable dead-letter queue. ENCODING RECOVERY MATRIX · LEADER/09 × PAYLOAD BYTES input · ISO 2709 record → read leader[9], scan payload for UTF-8 lead bytes clean recoverable drift unrecoverable → quarantine the leader is a claim verify it against the bytes PAYLOAD SCAN MARC-8 bytes · no 0xC2–0xF4 PAYLOAD SCAN UTF-8 lead bytes 0xC2–0xF4 LEADER/09 0x20 ␣ declares MARC-8 LEADER/09 0x61 'a' declares UTF-8 ✓ CLEAN marc8_to_unicode(raw) Leader and payload agree: MARC-8. Deterministic table translation. RECOVERABLE DRIFT raw.decode('utf-8') Leader claims MARC-8, payload is UTF-8. force_utf8 · log drift · keep record. UNRECOVERABLE DRIFT raise → quarantine Leader claims UTF-8, payload is MARC-8. No safe auto-fix — needs source policy. ✓ CLEAN raw.decode('utf-8','strict') Leader and payload agree: UTF-8. Strict decode; raises on invalid byte. quarantine.send() leader hex only · no field bytes dead-letter queue durable · replayable

Problem Framing

The corruption is silent because a permissive decoder never raises. A record whose leader says MARC-8 but whose 245 $a carries UTF-8 bytes will decode “successfully” under a latin-1-style fallback and produce mojibake that only a human cataloger notices weeks later. The strict variant is louder — a record that declares Unicode but carries MARC-8 diacritics throws a UnicodeDecodeError mid-batch:

text
$ python -m ingest.worker --source vendor_2019.mrc
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 14:
  invalid start byte
  record 004112  field 245  offset 0x1e3f

Byte 0x92 here is a MARC-8 curly apostrophe that was never valid UTF-8. Before you touch code, quantify the blast radius. Pipe the ingestion log through a structured query to surface which vendors and which fields concentrate the failures:

bash
grep -E "UnicodeDecodeError|leader_mismatch|byte_offset" /var/log/marc-ingest/worker.log \
  | awk '{print $3, $7, $12}' | sort | uniq -c | sort -nr

Recurring (vendor, field, byte) tuples tell you whether you have one bad export or a systemic source-ILS misconfiguration. The two signatures to separate are strict failures (UnicodeDecodeError, batch halts) and permissive fallbacks (a replacement character U+FFFD or mojibake injected, batch continues). The second is more dangerous precisely because nothing alerts.

Root Cause

MARC21 predates Unicode. The historical character set is MARC-8, a shift-based multi-byte scheme, and the transition to UTF-8 was never atomic across the installed base of integrated library systems. The single source of truth for which scheme a record uses is Leader byte 09 (leader[9], the character coding scheme position):

Leader/09 value Byte Declares Correct decoder
space 0x20 MARC-8 marc8_to_unicode translation
a 0x61 UCS/Unicode (UTF-8) strict UTF-8

Encoding drift is the state where leader[9] disagrees with the actual bytes. It arises three ways: a source ILS was migrated to UTF-8 storage but its export routine never rewrote the leader; a vendor re-encoded records in transit without updating byte 09; or a file was concatenated from mixed-vintage sources so the declaration is right for some records and wrong for others. Because the leader is only a claim, a robust pipeline treats it as a hypothesis to verify against the payload — never as ground truth. This is the same leader-as-contract discipline enforced downstream when validating MARC leader fields before database insert; the decode stage and the validation stage share one view of what the leader means.

Solution

The fix is to route decoding on the leader, verify against the actual byte pattern, and never let a permissive fallback silently mangle data. Use the pymarc MARCReader rather than the standard-library codecs module — Python ships no marc8 codec, so MARC-8 to Unicode translation is a pymarc-specific capability.

Before — a single permissive path that hides drift:

python
# Anti-pattern: latin-1 never raises, so MARC-8/UTF-8 drift becomes mojibake
text = raw_bytes.decode("latin-1")

After — leader-routed decoding with an explicit drift branch:

python
import logging
from pymarc import MARCReader
from pymarc.marc8 import marc8_to_unicode

logger = logging.getLogger("marc.decode")

_UTF8_LEAD_BYTES = bytes(range(0xC2, 0xF5))  # 0xC2-0xF4 start a UTF-8 sequence


def declares_utf8(leader: str) -> bool:
    """Leader/09 == 'a' means UCS/Unicode; space means MARC-8."""
    return len(leader) > 9 and leader[9] == "a"


def looks_like_utf8(raw: bytes) -> bool:
    """Heuristic: does the payload carry multi-byte UTF-8 lead bytes?"""
    return any(b in _UTF8_LEAD_BYTES for b in raw)


def decode_field(raw: bytes, leader: str, record_id: str, tag: str) -> str:
    """Deterministically decode one field's bytes, flagging drift."""
    declared_utf8 = declares_utf8(leader)
    payload_utf8 = looks_like_utf8(raw)

    if declared_utf8 and payload_utf8:
        return raw.decode("utf-8", errors="strict")
    if not declared_utf8 and not payload_utf8:
        return marc8_to_unicode(raw.decode("latin-1"))
    if not declared_utf8 and payload_utf8:
        # declares MARC-8 but carries UTF-8: recoverable drift
        logger.warning("encoding_drift record=%s tag=%s fix=force_utf8", record_id, tag)
        return raw.decode("utf-8", errors="strict")
    # declares UTF-8 but carries MARC-8: unrecoverable without source policy
    raise UnicodeDecodeError("marc-drift", raw, 0, len(raw), "declared UTF-8, found MARC-8")

At the stream level, let MARCReader apply the same policy across a whole file. Hand it the open file handle rather than reading the record into a BytesIO buffer — MARC records are variable-length, and slicing fixed chunks would split a record mid-leader. This is also the streaming boundary that keeps multi-gigabyte .mrc dumps from triggering OOM kills, covered in depth in optimizing pymarc performance for large record sets:

python
from collections.abc import Iterator
from pymarc import MARCReader, Record


def stream_decoded_records(filepath: str, *, force_utf8: bool = False) -> Iterator[Record]:
    """Stream records with pymarc's built-in MARC-8 -> Unicode conversion.

    to_unicode=True enables translation; utf8_handling='strict' makes drift
    loud instead of silent so corrupt records reach the quarantine branch.
    """
    with open(filepath, "rb") as fh:
        reader = MARCReader(
            fh,
            to_unicode=True,
            force_utf8=force_utf8,
            utf8_handling="strict",  # 'strict' | 'replace' | 'ignore'
        )
        for record in reader:
            if record is None:
                logger.error("unparseable_record file=%s", filepath)
                continue
            yield record

Two byte-level defects survive leader routing and must be normalized before the parser state machine sees them. First, records mangled in transit sometimes carry a spurious UTF-8 lead byte glued to a MARC structural control character — a 0xC2 prepended to the 0x1E field terminator or 0x1F subfield delimiter — which corrupts field boundaries. Second, a stray UTF-8 BOM (0xEF 0xBB 0xBF) or zero-width no-break space (U+FEFF) at the head of a field shifts indexing downstream:

python
def sanitize_control_bytes(raw: bytes) -> bytes:
    """Strip spurious lead bytes from MARC control chars and drop a leading BOM."""
    cleaned = raw.replace(b"\xc2\x1e", b"\x1e").replace(b"\xc2\x1f", b"\x1f")
    if cleaned.startswith(b"\xef\xbb\xbf"):
        cleaned = cleaned[3:]
    return cleaned

Choose utf8_handling deliberately. strict is correct for ingestion: it converts silent corruption into a catchable exception you can route to a schema validation quarantine queue. Reserve replace (which injects U+FFFD) and ignore (which deletes bytes) for read-only reporting paths where a partially legible record beats no record — never for the write path into the catalog of record.

Compliance or Privacy Impact

Encoding choices are a data-fidelity control, and fidelity is a compliance concern the moment a corrupted field is patron-adjacent. A 100 $a personal-name heading or a 500 note referencing a patron that decodes to mojibake, or loses bytes under utf8_handling="ignore", is a record whose accuracy you can no longer attest to — which matters for the retention and export guarantees described in the data privacy boundaries in library systems reference. Two rules follow. First, ignore must never touch a field that feeds a downstream export subject to patron PII masking: silently deleting bytes can defeat a masking regex that expects a well-formed string, leaking the raw value. Second, every drift event you log must record the record control number (001) and field tag but not the raw field contents, so the diagnostic trail itself does not become an unmasked copy of patron data. Log the fact of corruption, its location, and the decoder applied — hash the bytes if you need a collision guard, never store them in plaintext.

Verification

Confirm the fix two ways: prove that recovered records are byte-faithful, and prove that unrecoverable drift is quarantined rather than written.

A round-trip assertion catches lossy decoding. Decode a known-good MARC-8 fixture, re-encode, and assert the Unicode form matches the expected reference — any U+FFFD in the output means a fallback leaked into the write path:

python
def test_marc8_roundtrip_is_lossless():
    raw = b"Beyonc\x92"  # MARC-8 apostrophe after "Beyonc"
    decoded = marc8_to_unicode(raw.decode("latin-1"))
    assert "�" not in decoded          # no replacement char injected
    assert decoded == "Beyoncʼ"             # exact expected Unicode form


def test_declared_utf8_with_marc8_is_quarantined():
    raw = b"caf\x92"                         # MARC-8 bytes, leader claims UTF-8
    leader = "00000nam a2200000 a 4500"      # position 09 == 'a'
    try:
        decode_field(raw, leader, record_id="004112", tag="245")
    except UnicodeDecodeError:
        return                               # correct: routed to quarantine
    raise AssertionError("drift decoded silently instead of raising")

In an integration run, count outcomes against the vendor manifest: records_in == records_written + records_quarantined, with zero silent U+FFFD substitutions in the written set. If a systemic mismatch is confirmed across a whole feed, halt the worker, revert the checkpoint offset to the last known-good commit, purge uncommitted index writes, and re-run with force_utf8=True set for that source. Because writes are keyed on the 001 control number with UPSERT semantics, the replay is idempotent — no duplicate holdings, no double-counted circulation deltas.