← MarkRadar

Parsing a 30,000-Page Government PDF Without OOM-Killing a 2GB Server

USPTO's weekly Trademark Official Gazette is a single 527MB, 30,822-page PDF, freely downloadable, no API key. Getting it into a nightly job on a small box surfaced a memory bug that only shows up at this exact scale.

MarkRadar's data source for new trademark filings is USPTO's weekly Trademark Official Gazette (TMOG) — the one unauthenticated, non-WAF-blocked publication format USPTO ships (their data.uspto.gov bulk API sits behind an AWS WAF bot-challenge that blocks scripted access entirely). The Gazette itself is a single PDF per week, and recent issues run to 527MB across 30,822 pages.

The first working version used pypdf, the obvious stdlib-adjacent choice, wrapped in a small test fixture of a few dozen pages. It passed. Against the real file, memory climbed past what I'd budgeted before the parser had even finished opening it.

The numbers

The box running the nightly job has 1.9GB of RAM total, shared with an always-on daemon process. pypdf's real-file trajectory would have OOM-killed the job before it produced any output — and because this was caught in a dry run before any real subscriber depended on it, it never actually happened in production. It would have on the first real Thursday run.

Why the small fixture didn't catch it

Memory-per-page for pypdf isn't flat — on this file it compounds as the in-memory document tree grows, so a 40-page fixture and a 30,822-page real file aren't the same test at different sizes; they're qualitatively different regimes. A parser that's fine on a fixture 1/700th the size of production input tells you almost nothing about production behavior. The only fixture that would have caught this ahead of time is the real file, or something within an order of magnitude of it.

The fix

Two changes, both cheap:

1. Switch the PDF library from pypdf to pymupdf (import name fitz) for this extraction path. Same output, ~7x lower peak memory on this file. pymupdf is the one non-stdlib dependency in this build for exactly this reason.
2. Add a hard ceiling as defense-in-depth, not as the primary fix: an RLIMIT_AS cap (900MB) on every subprocess this job spawns. If a future change to the Gazette's format or a different library regresses this, the process gets killed cleanly at a known ceiling instead of taking the whole box down with it.

Neither change required touching the matching logic downstream — the bug was entirely in how the raw file got opened, not in anything that reads its content.

The general version of this

Before letting any unattended job process a real-sized input, test it against something close to real-sized, not a fixture built for correctness alone. Fast, correct, and untested-at-scale is exactly the shape of bug that a nightly cron job discovers on a Thursday no one is watching — test the memory profile before the schedule finds it for you.

This is the same pipeline behind MarkRadar: a $15/month weekly check of new USPTO trademark filings and newly registered domains against your brand name, no sales call, no enterprise contract.