Validating MARC Leader Fields Before Database Insert

This page answers one narrow operational question: a record whose 24-byte MARC leader is subtly malformed sails through parsing, then blows up at insert time — a directory-offset IndexError, a UnicodeDecodeError deep in field extraction, or worse, a silent commit of a truncated record that surfaces as mojibake days later. It sits under Schema Validation for Ingested Records within the broader Catalog Ingestion & ILS Sync Pipelines architecture, and it covers the byte-level leader gate that the parent guide’s structural-validation tier delegates to. The leader dictates record length, base address, and character-coding scheme, so validating it before the record acquires a database connection is the cheapest place to stop a corrupt payload — reject 24 bytes, or debug a poisoned catalog.

Problem Framing

The symptom is an insert-stage failure whose stack trace points nowhere near the real fault. A record parses cleanly enough to produce field structures, then the persistence layer raises — or, more dangerously, commits garbage without raising at all. In the structured ingest log the signature is a burst of exceptions clustered on records from a single vendor drop, all after the parse stage reported success:

jq 'select(.stage == "insert" and .exc_type | test("IndexError|UnicodeDecodeError|DataError"))' ingest.log

If that filter returns rows that the parse stage never flagged, the leader is the prime suspect. A leader that declares a record length of 01234 while the payload is 900 bytes, a base-address-of-data pointer smaller than the leader itself, or a coding-scheme byte claiming Unicode over MARC-8 content will each pass a permissive parser and then fail — or silently corrupt — at materialization. The most expensive variant does not raise anything: a truncated record with a padded length field commits a partial bibliographic record, and the gap only becomes visible when a patron searches for a title that was cut off mid-245.

Root Cause

The leader occupies positions 00–23 as a fixed-width ASCII block, and three of its fields are load-bearing for everything downstream:

1. Record length (00–04) drives truncation detection. ISO 2709 stores the total record length as a 5-digit number at the front. Vendors that pad this field to the expected size while shipping a truncated body produce a record that looks whole to a length-agnostic parser. Nothing downstream re-checks int(leader[0:5]) against the actual byte count, so the truncation is invisible until the missing bytes matter.
2. Base address of data (12–16) drives directory arithmetic. The base address tells the parser where the variable fields begin. If it is non-numeric or smaller than 24 (the minimum possible, since the leader alone is 24 bytes), directory-offset math underflows and field extraction indexes past the buffer — the IndexError you see at insert time is a leader fault surfacing three stages late.
3. Character coding scheme (09) drives the encoding contract. Position 09 is a for UCS/Unicode or a space for MARC-8. A record flagged Unicode but carrying MARC-8 bytes (or the reverse) decodes into mojibake. Modern PostgreSQL/MySQL columns expect UTF-8, so a mislabeled leader either raises a UnicodeDecodeError or, when errors are coerced, commits corrupted diacritics — the same class of fault dissected in Handling UTF-8 Encoding in Legacy MARC Records.

Because these fields are checked (if at all) only when their values are used, the failure always lands downstream of its cause. Validating the leader as a contiguous block at the ingestion boundary collapses that distance to zero and keeps a malformed record from ever consuming a connection-pool slot.

Solution

Validate the leader against the byte-level contract below before any field extraction or ORM hydration occurs. Slice the raw bytes directly rather than routing them through a high-level MARC parser at this stage — the whole point is to reject before the parser runs. The full record-shape contract that surrounds this gate (mandatory tags, control-vs-data fields, cardinality) lives in the parent Schema Validation for Ingested Records guide; this page owns only positions 00–23.

from dataclasses import dataclass
from typing import Optional

VALID_RECORD_STATUSES = frozenset("acdnp")
VALID_ENCODING_SCHEMES = frozenset((" ", "a"))


@dataclass(frozen=True)
class LeaderValidationResult:
    valid: bool
    error_code: Optional[str] = None
    detail: Optional[str] = None


def validate_marc_leader(raw_bytes: bytes) -> LeaderValidationResult:
    """Validate the 24-byte MARC leader before database insert.

    Operates on raw bytes at the ingestion boundary — no parser, no
    connection. Returns a typed result so the caller can route rejects
    to quarantine without exception-driven control flow.
    """
    if len(raw_bytes) < 24:
        return LeaderValidationResult(False, "ERR_LEADER_TOO_SHORT",
                                      f"Payload only {len(raw_bytes)} bytes")

    leader = raw_bytes[:24].decode("ascii", errors="replace")

    # 00-04: declared length must equal the real payload length.
    try:
        declared_length = int(leader[0:5])
    except ValueError:
        return LeaderValidationResult(False, "ERR_LEADER_LEN_NAN",
                                      f"Leader[0:5] not numeric: {leader[0:5]!r}")
    if declared_length != len(raw_bytes):
        return LeaderValidationResult(False, "ERR_LEADER_LEN_MISMATCH",
                                      f"Declared {declared_length}, actual {len(raw_bytes)}")

    # 05: record status.
    if leader[5] not in VALID_RECORD_STATUSES:
        return LeaderValidationResult(False, "ERR_INVALID_RECORD_STATUS",
                                      f"Status {leader[5]!r} not in {sorted(VALID_RECORD_STATUSES)}")

    # 09: character coding scheme (space = MARC-8, 'a' = Unicode).
    if leader[9] not in VALID_ENCODING_SCHEMES:
        return LeaderValidationResult(False, "ERR_INVALID_ENCODING_FLAG",
                                      f"Encoding flag {leader[9]!r}")

    # 10-11: indicator + subfield code counts are fixed at "22".
    if leader[10:12] != "22":
        return LeaderValidationResult(False, "ERR_INDICATOR_SUBFIELD_COUNTS",
                                      f"Expected '22', got {leader[10:12]!r}")

    # 12-16: base address of data; < 24 is structurally impossible.
    try:
        base_addr = int(leader[12:17])
    except ValueError:
        return LeaderValidationResult(False, "ERR_BASE_ADDR_NAN",
                                      f"Leader[12:17] not numeric: {leader[12:17]!r}")
    if base_addr < 24:
        return LeaderValidationResult(False, "ERR_BASE_ADDR_TOO_SMALL",
                                      f"Base address {base_addr} < 24")

    # 20-23: standard MARC21 entry map.
    if leader[20:24] != "4500":
        return LeaderValidationResult(False, "ERR_ENTRY_MAP_NONSTANDARD",
                                      f"Entry map {leader[20:24]!r}, expected '4500'")

    return LeaderValidationResult(True)

Wiring it into the write path is the other half of the fix. Before: the record is constructed and inserted, and only the database driver decides whether it was well-formed. After: validation runs inside the same transaction scope as the INSERT, and a failure raises before commit so the transaction rolls back cleanly:

import structlog

log = structlog.get_logger()


def insert_record(conn, raw_bytes: bytes, quarantine, correlation_id: str) -> bool:
    """Validate the leader, then insert inside one transaction. Returns
    True on commit, False when the record was quarantined."""
    result = validate_marc_leader(raw_bytes)
    if not result.valid:
        quarantine.send(original=raw_bytes, error_code=result.error_code,
                        detail=result.detail, correlation_id=correlation_id)
        log.warning("leader.reject", error_code=result.error_code,
                    correlation_id=correlation_id)
        return False

    with conn.transaction():  # commits on exit, rolls back on exception
        conn.execute(
            "INSERT INTO bib_record (control_number, raw) VALUES (%s, %s)"
            " ON CONFLICT (control_number) DO UPDATE SET raw = EXCLUDED.raw",
            (extract_001(raw_bytes), raw_bytes),
        )
    return True

Two details make this safe under retries. The ON CONFLICT upsert keeps reprocessing idempotent, so a replayed batch after a transient broker fault collapses to a no-op instead of forking a bibliographic record. And the primary key is derived from the 001 control field only after leader validation passes — deriving it earlier would trust bytes the gate has not yet cleared. Records that fail here are durable and replayable, not dropped; the broker mechanics behind that quarantine bucket are governed by Async Batch Processing for Catalog Updates.

Compliance or Privacy Impact

Leader validation looks purely structural, but its diagnostics touch regulated data. When a record is rejected, the instinct is to log a full hex dump for forensic and vendor-dispute purposes — and a MARC record’s variable fields routinely carry patron-linked data in local 9xx fields, circulation notes, and hold-shelf slips. A rejection log that dumps the whole payload becomes an unaudited copy of that data outside the retention controls. Bound the dump to the 24-byte leader, which contains no bibliographic or patron content, and log a salted fingerprint of the remainder rather than the bytes themselves:

import hashlib


def leader_diagnostic(raw_bytes: bytes, salt: bytes) -> dict:
    """Forensic-safe rejection payload: leader hex only, body fingerprinted."""
    return {
        "leader_hex": raw_bytes[:24].hex(),        # no PII in positions 00-23
        "body_fingerprint": hashlib.sha256(salt + raw_bytes[24:]).hexdigest()[:12],
        "byte_length": len(raw_bytes),
    }

The quarantine bucket itself inherits the live feed’s obligations: a rejected payload paused there still carries whatever patron-linked fields the record held, so it is subject to the same masking and retention rules as an accepted record, per the boundaries set out in Data Privacy Boundaries in Library Systems. Treating a reject buffer as exempt because “it never got inserted” is exactly the gap auditors flag. Keep the fingerprint salt in a secret, and the leader gate stays a compliance asset rather than a leak surface.

Verification

Confirm the gate along two axes: that it rejects each malformed-leader class deterministically, and that a rejection actually rolls the transaction back instead of leaving a partial write. Drive it with hand-built leaders that violate exactly one rule so the asserted error_code pins the fault precisely:

import pytest


def _leader(length: int, body: bytes = b"") -> bytes:
    # Valid template: status 'n', Unicode, counts "22", base 00033, map 4500.
    leader = f"{length:05d}nam a2200033 a 4500".encode("ascii")
    assert len(leader) == 24
    return leader + body


def test_accepts_conforming_leader():
    body = b"x" * 9  # 24 + 9 = 33 bytes, matching declared length
    assert validate_marc_leader(_leader(33, body)).valid


def test_rejects_length_mismatch():
    # Declared 99999 but real payload is 24 bytes: padded-length truncation.
    result = validate_marc_leader(_leader(99999))
    assert result.error_code == "ERR_LEADER_LEN_MISMATCH"


def test_rejects_short_payload():
    assert validate_marc_leader(b"0001").error_code == "ERR_LEADER_TOO_SHORT"


def test_rejects_nonstandard_entry_map():
    corrupt = bytearray(_leader(24))
    corrupt[20:24] = b"0000"
    result = validate_marc_leader(bytes(corrupt))
    assert result.error_code == "ERR_ENTRY_MAP_NONSTANDARD"

Run the suite against a corpus of real anonymized vendor records, not only synthetic fixtures — the failure modes that matter (padded lengths over truncated bodies, MARC-8 masquerading as Unicode, legacy exports with non-22 counts) come from real drops and rarely appear in hand-written cases. For a live sanity check, pipe rejected payloads through python -c "import binascii,sys; print(binascii.hexlify(sys.stdin.buffer.read(24)))" to eyeball the leader bytes against the spec table, then confirm the database transaction log shows zero rows written for any correlation_id the gate rejected. If the parse stage upstream is the real bottleneck rather than validation, the streaming techniques in Optimizing pymarc Performance for Large Record Sets keep the leader gate from becoming a memory sink on large files.

Related

• Schema Validation for Ingested Records — the parent guide whose structural tier delegates the byte-level leader contract to this gate.
• Parsing MARC Records with pymarc — the parse stage that runs after a leader clears this validator.
• Handling UTF-8 Encoding in Legacy MARC Records — the transcoding path for records whose position-09 flag routes them to MARC-8 conversion.
• Optimizing pymarc Performance for Large Record Sets — streaming discipline that keeps byte-level validation from materializing whole batches.