Library Catalog Data Sync Pipelines

Optimizing pymarc Performance for Large Record Sets

This page answers one narrow operational question: why does a pymarc-based ingest worker climb steadily in memory and get OOM-killed partway through a multi-gigabyte MARC21 batch, and how do you make it run at flat resident set size instead? It sits under Parsing MARC Records with pymarc, which covers the parse stage end to end, and within the broader Catalog Ingestion & ILS Sync Pipelines architecture. If you are ingesting nightly vendor feeds, legacy union-catalog exports, or high-frequency circulation deltas, the symptoms and fixes below apply directly to the worker that decodes ISO 2709 bytes into Record objects.

Problem Framing

The failure shows up as a container that dies mid-batch with no application-level traceback. The worker’s resident set size grows monotonically with the number of records processed, then the kernel OOM killer reaps it. In practice you see the process vanish and a kernel log line like this:

text
$ dmesg -T | grep -i oom
[Wed Jul  1 02:41:17 2026] Out of memory: Killed process 4812 (python) total-vm:9438208kB, anon-rss:8912344kB, file-rss:0kB
[Wed Jul  1 02:41:17 2026] oom_reaper: reaped process 4812 (python), now anon-rss:0kB

A quick tracemalloc snapshot taken at two points in the batch confirms that allocation scales with records seen rather than staying constant:

text
[ Top allocations at record 50,000 ]
pymarc/record.py:212      size=612 MiB    count=1,048,102
[ Top allocations at record 100,000 ]
pymarc/record.py:212      size=1224 MiB   count=2,096,204

The size roughly doubles as the record count doubles. That linear growth is the signature of retaining every parsed record instead of releasing it after emit. The same pattern appears with MARCXML sources, except the memory is consumed by the XML DOM rather than by Record objects.

Resident set size versus records processed: list accumulation versus generator streaming A line chart with records processed on the horizontal axis from zero to one hundred thousand and worker memory on the vertical axis from zero to eight gigabytes. A dashed horizontal line marks the eight-gigabyte OOM-killer cutoff. The list-accumulation curve rises linearly with record count and crosses the cutoff near eighty-five thousand records, where the worker is killed. The generator-streaming curve stays flat just above the baseline at roughly a quarter gigabyte for the entire batch, because only one record is resident at a time. WORKER MEMORY OVER A LARGE BATCH 0 2 4 6 8 resident set size (GB) 0 25k 50k 75k 100k records processed OOM-killer cutoff (8 GB) process killed records = [r for r in MARCReader(fh)] yield record — one record resident (~0.25 GB, flat) list accumulation generator streaming

Root Cause

There are three distinct causes, and a given worker usually has one of them.

Unbounded list accumulation. The most common cause is collecting parsed records into a Python list — often hidden inside an innocent-looking comprehension such as records = [r for r in MARCReader(fh)] or a records.append(record) inside the read loop. Each Record retains its full field graph (leader, directory, control fields, and every data field with its subfields), so a multi-gigabyte source expands to several times its on-disk size once resident. Nothing is freed until the list goes out of scope, which is after the whole file is read — exactly when the heap is already exhausted.

MARCXML DOM materialization. For MARCXML feeds, pymarc.parse_xml_to_array builds the entire document tree in memory before returning a list of records. A 4 GB MARCXML export becomes a DOM many times that size. The function is convenient for small files and catastrophic for large ones.

Per-record decoder instantiation. A subtler cost, visible as CPU rather than memory, is constructing a MARC-8 to Unicode converter (or compiling a regex) inside the read loop. On MARC-8-heavy feeds this dominates parse time. It does not cause the OOM, but it is why an otherwise-fixed streaming worker is still slow. MARC-8 versus UTF-8 handling itself is covered in handling UTF-8 encoding in legacy MARC records; here the concern is only where the converter is built.

The unifying principle: pymarc.MARCReader is already an incremental parser. It reads one record at a time from the file object. Any memory growth is something your code adds on top of it by holding references.

Solution

Consume the reader as a generator and emit each record downstream immediately, retaining nothing across iterations. The parse stage in Parsing MARC Records with pymarc shows the full extraction logic; the change here is purely about lifetime.

Before — accumulates every record, grows without bound:

python
from pymarc import MARCReader

def load_records(filepath: str) -> list:
    with open(filepath, "rb") as fh:
        # Every Record is retained until the list is returned — RSS grows
        # linearly with file size and OOM-kills the worker on large batches.
        return [record for record in MARCReader(fh) if record is not None]

After — streams at flat RSS, one record resident at a time:

python
import logging
from typing import Iterator
from pymarc import MARCReader, Record

logger = logging.getLogger("marc_ingest.stream")


def stream_marc_records(filepath: str) -> Iterator[Record]:
    """Yield MARC records one at a time from an ISO 2709 file.

    MARCReader is incremental, so we hand it the open binary file object and
    never build a list. Each record is garbage-collected once the caller has
    emitted it downstream, keeping resident set size flat regardless of the
    input file's size. A None value marks an unrecoverable parse error for the
    current record; the reader resynchronizes on the next record terminator.
    """
    with open(filepath, "rb") as fh:
        reader = MARCReader(fh, to_unicode=True, force_utf8=False,
                            utf8_handling="replace")
        for record in reader:
            if record is None:
                logger.warning(
                    "unrecoverable parse error; record skipped",
                    extra={"reader_error": str(reader.current_exception)},
                )
                continue
            yield record

The caller must not defeat the streaming by re-collecting: iterate and emit inside the loop (to a message broker, a bulk-insert batch of bounded size, or a checkpoint), then drop the reference.

For MARCXML, replace parse_xml_to_array with lxml.etree.iterparse, which surfaces one <record> element at a time. Crucially, call elem.clear() after each record and also delete already-processed siblings so lxml does not retain the growing prefix of the tree:

python
from typing import Iterator
from io import BytesIO
from lxml import etree
from pymarc import parse_xml_to_array, Record

_MARC_NS = "http://www.loc.gov/MARC21/slim"


def stream_marcxml(filepath: str) -> Iterator[Record]:
    """Constant-memory streaming of MARCXML via lxml iterparse.

    parse_xml_to_array materializes the whole DOM; iterparse yields each
    <record> as it closes. We clear the element and its consumed predecessors
    so lxml releases the parsed prefix instead of holding the entire tree.
    """
    context = etree.iterparse(filepath, events=("end",),
                              tag=f"{{{_MARC_NS}}}record")
    for _event, elem in context:
        xml_bytes = etree.tostring(elem, encoding="unicode").encode("utf-8")
        records = parse_xml_to_array(BytesIO(xml_bytes))
        if records and records[0] is not None:
            yield records[0]
        elem.clear()
        while elem.getprevious() is not None:
            del elem.getparent()[0]

Finally, hoist any per-record cost to module scope. pymarc performs MARC-8 to Unicode conversion internally when to_unicode=True, so the win here is compiling your own inspection regexes and caching tag-profile lookup tables once at import rather than rebuilding them per record — worth 40 to 60 percent CPU on MARC-8-heavy feeds:

python
import re

# Compiled once at import, reused for every record.
_LOCAL_TAG = re.compile(r"^(59\d|9\d\d)$")
_OVERSIZE_LIMIT = 10_240  # bytes; truncate before ORM hydration

To keep a long-running batch resumable without accumulating state, checkpoint a byte offset periodically rather than holding processed records. A lightweight SQLite offset tracker records the last committed position and batch sequence, so a killed worker restarts from the saved offset instead of the top of a multi-gigabyte file. Records that fail to parse still route to the schema validation quarantine queue rather than aborting the run, and downstream retry-and-backoff remains owned by async batch processing for catalog updates.

Compliance or Privacy Impact

Switching from list accumulation to streaming changes when records exist in memory, not what is in them, so it does not by itself expand or shrink PII surface area. But two compliance properties must be preserved through the change.

First, audit-log completeness. If a batch previously logged one summary record after building the full list, streaming removes that natural join point. Emit the audit event per record inside the loop instead — record length, field count, and a SHA-256 checksum of the raw byte payload — so the audit trail stays complete even though no record is retained. The retention rules governing how long each record class may persist live in data privacy boundaries in library systems.

Second, masking idempotency. The patron PII masking pass must run per record before the record crosses the parse boundary, exactly as it did in the batch version; streaming must not create a shortcut path that emits an unmasked record ahead of the mask. Because masking is deterministic and structure-preserving, running it inside the generator loop yields byte-identical output to the batch form — verify that equivalence explicitly (below) so the optimization cannot silently regress a privacy control.

Verification

Confirm the fix three ways: memory stays flat, the streamed output equals the batched output, and a killed worker resumes correctly.

Flat memory under load. Snapshot with tracemalloc at intervals and assert the peak does not scale with records processed:

python
import tracemalloc


def peak_mib_over_batch(filepath: str, sample_every: int = 25_000) -> float:
    """Return peak traced memory (MiB) while streaming a whole file."""
    tracemalloc.start()
    for i, _record in enumerate(stream_marc_records(filepath), start=1):
        if i % sample_every == 0:
            current, peak = tracemalloc.get_traced_memory()
            # Peak must stay bounded; it should NOT track record count.
            assert peak < 256 * 1024 * 1024, f"RSS regressed at record {i}"
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return peak / (1024 * 1024)

For a live worker, tracemalloc usage and snapshot comparison are documented in the standard-library reference; the assertion above is the CI-friendly form.

Streamed output equals batched output. Guard the privacy and correctness invariant that streaming changed nothing but lifetime:

python
def test_stream_matches_batch(tmp_path):
    fixture = _write_fixture(tmp_path, n=500)  # small known-good .mrc
    batched = [r.as_marc() for r in load_records(str(fixture))]
    streamed = [r.as_marc() for r in stream_marc_records(str(fixture))]
    assert streamed == batched  # identical records, identical order

Resumability. Simulate a mid-batch kill by stopping at a checkpoint offset, then restart the iterator from the saved offset and confirm the control-number (001) range of the second pass begins exactly where the first stopped, with no gap or overlap against the vendor manifest. Full leader-level validation of each resumed record is covered in validating MARC leader fields before database insert.