pdf2epub-calibration

PDF→EPUB Classifier Calibration

When to Use

• A converted EPUB has citations in body text, page numbers as phantom endnotes, or broken/duplicate footnote links
• Chapter headings resolve to wrong PDF pages
• The cover image comes out as a texture fragment or a naked photo without typography

Span-merge reality (check the dump, not the flag)

• page.get_text("dict") gives lines whose spans merge text at different font sizes. A body line can contain a 5pt footnote-ref span inside 12pt text — text extraction sees ONE line at the max span size.
• Superscript flags are unreliable: several authoring tools (QuarkXPress-style CJK exports included) emit footnote refs as STANDALONE 6pt lines with the sup flag UNSET, and emit the note's own number as a leading 5pt span merged into the note's first line. Gate ref detection on size + position, never on the sup flag alone.
• Before changing any threshold, dump per-page lines with text/size/flags/y-ratio. Most classification misses are span-merge artifacts, not wrong thresholds.

Classification order and thresholds (validated values)

1. FOLIO first: bare 1-4 digit (or roman i-vi) number, font < 0.85*body, y > 0.90. Rules run in order; if the footnote rule runs before the folio rule, page numbers become phantom notes.
2. FNREF: whole line is a bare digit, font < 0.75*body, any y.
3. FOOTNOTE: font < 0.92*body, y > 0.60, not a list marker. Long note blocks climb to 2/3 page height — a fixed 'footnotes below 80%' y-cut loses note starts. Use the largest threshold that still excludes body text.
4. BODY: everything else.
5. Body font size is LEARNED (2-char-mode of line sizes), never hard-coded. Books legitimately differ.

Footnote block grouping

• A note starts at a line whose first token is a bare digit followed by body text; bare-digit strays are dropped. Continuations attach unconditionally to the previous block.
• Per-chapter note numbering restarts are normal in books — scope anchor ids by chapter or duplicate ids fail the link check.
• Glued CJK note numbers need a monotonic matcher (see Endnotes-section books below), not a whitespace-required regex.

Printed-TOC page resolution (probes)

• Never probe with a fixed title[:16] slice against clean() page text: CJK chapter titles break across lines (the body page inserts a space/newline mid-title), the TOC may spell 章一 (Chinese numerals) while the body prints 20 (digits), and 2-char titles like 注释 fall under any ≥4-char filter. Probe instead against whitespace-COLLAPSED page text with title variants: full title de-spaced, title without the leading 第N章/部分/Part marker, and accept probes ≥2 chars.
• The probe's own match window (printed-3 … printed+40) REACHES THE TOC PAGES for early chapters — a chapter title matches its own TOC line and the section resolves to the TOC page itself. Pass the TOC-run page indexes as an exclusion set to the resolver.
• Trust the actual matched page over majority-offset arithmetic per entry; same-page sections (part title + chapter 1 sharing a start page) are LEGITIMATE output — a seen-set on page alone (or a strictly-increasing idx guard) silently drops chapter entries whose printed page equals their part title's.
• Probe order matters: try the FULL stripped title as a standalone short line (≤34 chars, per-line whitespace-collapsed) across the whole window FIRST, then loose variants (title minus chapter marker, prefix cut at '·' for parenthetical insertions), then whole-page substring. A mid-sentence body fragment of a 2-3 word title matches earlier pages than the real heading and wins the substring pass — probe the full title first to starve that false positive. Lines keep internal spaces ('第十三章 预防性管教' does not contain '第十三章预防性管教') — collapse whitespace INSIDE each line, not just page-wide.

Verification loops

• Link check: for every xhtml, the set of href="#fn..." targets must equal the set of id="fn..." anchors; no duplicates. With endnote chapters the check must also handle secNNN.xhtml#anchor cross-file forms (resolve the referenced file from the zip and match the id inside it). Re-run after EVERY classifier change.
• Chapter-count sanity against the printed TOC: sections == number of non-roman TOC entries. A silent drop (probe miss, dedupe) shows up as a missing chapter in nav — diff nav titles against the entry list before accepting a build.
• Whole-section figures: a chapter can render with 0 <figure> refs while the zip contains the images — images extracted into imgs_by_page must have EVERY consuming render path (front-matter section included) append them; the front-matter render path bypasses the chapter render_pages helper.
• One threshold change per iteration, against the span dump. Debug script shape: classify → group → report dups/missing → dump the suspect page's raw lines. Run the resolution stage standalone (parse_printed_toc_run → resolve_printed_toc) and eyeball the (title, idx) tuples when nav looks wrong.

Cover extraction

• Extracting the largest page-1 image is safe ONLY when it is the SOLE image and covers ≥ 85% of the page area. Composite covers (full-bleed texture + framed photo + paint-op typography) exist only as the rendered composite — render the page at 200 DPI in every other case.

Inline illustration crops

• Extract body figures as RENDERED page-surface crops (page.get_pixmap(matrix=Matrix(dpi/72), clip=bbox) from get_image_info() geometry), never raw xref XObjects — composed/vector art and CMYK/mask oddities yield fragments or color-broken extractions; the rendered surface always matches what the reader saw. Geometry from PDF structure keeps the pipeline deterministic; no AI layout model needed.
• Filter ≥ 3% page area (drops icons/logos), drop < 2 KB crops (blank/noise), dedup by pixel sha256 (books reuse identical art), skip the cover region, normalize RGBA→RGB on white, LANCZOS cap 1600 px, JPEG q85.
• Page-margin caption text sitting on a figure's top edge bleeds into the crop and appears clipped (its line bbox overlaps the clip's top strip). Fix: shrink the clip top below the y1 of any text line overlapping the top strip — and guard for pages with NO figure in the same loop, or the fix crashes on plain-text pages (image_info() returns an empty list, indexing [0] raises IndexError).
• Caption text that appears CLIPPED at a figure crop's top edge (mid-letter cuts, typeset italic unlike the art) is usually part of the publisher's raster asset, not page bleed. Verify against PDF text-line intersection with the image bbox: no overlap = caption lives inside the image = ship it. Do not raise the crop line further — you will amputate real art.
• Verify by vision-checking the LARGEST and one mid-book crop per book, checking panel borders intact and no clipped margin text at edges.

Common Pitfalls

• ebooklib's default epub3_pages: True builds a nav page-list from EVERY element carrying both epub:type and an id — noteref anchors qualify, so a notes-heavy book grows a hidden "Pages" nav listing 50+ raw anchor ids (refc2-4, reffront-1). Some readers surface that hidden nav as a second TOC page. Fix: epub.write_epub(path, book, {"epub3_pages": False}) — pdf2epub never emits a real page map, so the list is garbage in every build.
• Spine + cross-file href defects validated in the reader only: a lenient reader and the zip-level link check both passed while Kobo still failed per-chapter. Strict-reader validation (Kobo device or epubcheck) belongs in the verification loop.
• Fixing a bleed by shrinking the clip top needs the no-figure-page guard — plain-text pages crash on image_info()[0] indexing otherwise.
• Do not remove clipped captions by cutting higher: verify against PDF text-line intersection first; captions inside the raster asset ship with the art.
• Resolution regressions cascade: dropping one TOC entry (e.g. 注释) also drops the endnote-section detection, which reclassifies all notes to BODY and kills every anchor — a nav-title diff against the TOC entry list catches it in one step, a link check alone shows only the dangling-ref symptom.

Endnotes-section books (notes on their own pages)

• Two footnote architectures exist. Bottom-of-page footnotes: per-chapter numbering, the classic y > 0.60 path. Global endnotes: ONE 注释/Notes chapter near the book's end, numbering GLOBAL and continuous (e.g. 1..127 across all chapters), notes at small font from the very TOP of dedicated pages — the y > 0.60 cut classifies them BODY, so nearly all note blocks vanish and nearly all refs dangle.
• Fix: resolve sections BEFORE line classification, pass the endnote section's page range into the classifier, and drop the y-test on those pages. TOC parsing needs no classified lines, so the reorder is safe.
• TOC parse must read the WHOLE TOC RUN (continue while consecutive pages yield dot-leader entries). A single-page parse loses trailing entries (appendices, notes) and dumps every trailing PDF page into the last chapter.
• Glued note numbers: endnote entries may merge the number into the text with no space (1查尔斯…). Group blocks with a monotonic expected-number matcher: a block starts only when its leading number equals prev+1 or 1; anything else (dates like 1996, page cites) attaches as continuation. A regex that accepts any leading digits as a start re-splits on dates.
• Cross-file anchors: with the notes chapter on its own spine file, a same-document #id href does NOT navigate between files. Forward refs emit sec{NNN}.xhtml#fn...; the notes chapter's [n] back-links emit sec{NNN}.xhtml#refc{N}-{num} for the citing section (first citation wins; no citing ref → plain text). Link check must cover both directions and both forms.
• Cross-file hrefs must NOT get another '#' prepended: href="#sec030.xhtml#fnc30-4" is a same-document fragment pointing at an id that exists nowhere — a href-template #{tgt} blind-prepends it. Route every inline anchor through one helper that prepends '#' only when the target lacks .xhtml#. (epubcheck-style same-file link checks can miss this: the checker resolves #{file}#{id} as an internal fragment and fails only on 'fragment id not found'.)
• Spine entries must be MANIFEST IDS, never href strings: book.spine = [...] + [c.file_name] writes <itemref idref="sec002.xhtml"/> — an idref pointing at no manifest id. ebooklib accepts it silently, nav still works in lenient readers, but Kobo errors per chapter ('EPUB/sec002.xhtml does not exist in this book') and other strict readers drop the whole book. Spine by c.id ('chapter_N'), then verify every <itemref idref> exists in the manifest — nav working in one reader is NOT evidence the spine is valid.

Language of generated labels

• Never hard-code a language's label in the conversion path — the pre-chapter front-matter title must follow the --lang flag (前言 for zh, Front Matter for en), and the section ships only when its rendered part list is non-empty (an otherwise-empty front section renders as a bare h1).

Verification Checklist

• Link check PASS (script in pdf2epub SKILL.md): 0 missing targets, 0 duplicate ids, both href forms (same-file + secNNN.xhtml#)
• <itemref idref> spine check: every idref exists in the manifest (Kobo-style strict-reader failure mode)
• nav.xhtml: no page-list block, no raw refc/reffront ids in TOC entries
• nav titles diff against the printed-TOC entry list: every entry present, no stubs (<2000-char non-divider chapters)
• 127-note-style anchors: note-block count == anchor count == expected total
• Done when: all above green AND at least one strict-reader target opens the book (Kobo device, or epubcheck clean)