fidelity-scoring

Fidelity Scoring — Source Fidelity Scoring

How fidelity-qa determines "does this pack actually resemble the original."

diversity and fidelity measure different things

B = baseline (팩 없는 기본 출력)
O = 원본 레퍼런스
T = 선언된 변환 계약 (preserve/adapt/substitute/drop + page_mirror_plan)
R = 팩 렌더

diversity = distance(R, B)        ← "밋밋하지 않은가"
fidelity  = correspondence(R, T(O))  ← "원본을 닮았는가"

The two are not summed together. High diversity must not offset low fidelity. The publication condition is diversity_pass AND fidelity_pass다.

Why diversity cannot substitute for fidelity — the diversity rubric's full-score definition is "clearly different from the baseline + claimed style match," but the person who wrote that 'claimed' (meta.yamlfamily·axes) is the same person who wrote the pack. It is a structure where one scores oneself by criteria one declared oneself, and the original is not in the input. Counterexamples:

1. The original is a conservative navy document, but the result is a neon poster → diversity full score, fidelity 0.
2. If the original is already similar to the baseline, a result that reproduces it exactly is rejected on diversity.
3. Even if the color, typography, and layout values match, the signature motif, page order, chart meaning, and repetition rules can be wrong.
4. pHash, color histograms, and edge density cannot judge "what is where and why."

Applicability

reference_modeThe scoring method differs depending on .

mode	Scoring
exact_document	All 7 axes below
official_system	F1·F3·F4·F6·F7 (F2 signature → replaced by component coverage, F5 exempt)
historical_canon	F2·F3·F4·F7 (against the principle inventory)
style_synthesis	fidelity is not scored. Only diversity + craft need to pass

Input (validator firewall)

Strictly separate what is given to fidelity-qa from what is not.

Given: original page renders (pages_src/), pack renders (pages/), transform_contract, page_mirror_plan, signature list, this rubric.

Not given: The reasoning, self-evaluation, andprompt.md source text of the curator who wrote the pack. If the validator reads the author's explanation of intent, they believe things exist that are not in the render. Judge only by what is visible.

7-axis rubric (0–4 points each, 28 points total)

must_preserve: true If an item is not reproduced per the contract, regardless of score, handle it with this one line without reject다. hedging with weights — there is no need to punish three times over by combining triple weighting, forced F2 full score, and automatic rejection for omission.

축	What is measured	0점	2점	4점
F1 contract fulfillment	transform_contractfulfillment rate of preserve/adapt/substitute/drop obligations, page_mirror_plan mapping completeness	No mapping or less than 40%	60~79%	95%+, every original page is mapped or has a reason for non-selection
F2 signature reproduction	must_preserve motif reproduction rate	Most core motifs missing	Partially reproduced	95%+, all signature pages reproduced
F3 Evidence precision	Among the render's main elements, the proportion of the original observedthat is evidenced in (= hallucination suppression)	Invented elements dominate	Only 60–79% evidenced	95%+, zero new major motifs absent from the original
F4 Relationship grammar	Card anatomy, repetition rules, element co-occurrence relationships, placement positions	Relationship system differs	Individual elements similar but rules unstable	Rules 95%+ match
F5 Attention hierarchy	Top-3 focal elements and reading order, relative area and emphasis ratio	Main focal absent or inverted	Only dominant matches	Top-3 order matches, relative area ±10%
F6 Semantic integrity	Page role, chart type, values, units, order	Manipulation, invention, mistranslation	No major errors but multiple minor ones	100% match
F7 Rights boundary	Substitution of trademarks, logos, proprietary photos, icons, official copy	Direct copying of original assets	Brand similarity remains, human judgment required	All protected assets substituted, only visual roles preserved

F3은 Reverse checkYes — while the other axes look at "is the original's in the render" (recall), F3 looks at "was the render's in the original" (precision). Inventing plausible elements that were not in the original was this project's v1 failure mode.

Pass criteria — by mode

Since the number of active axes differs by mode, the pass threshold differs too. Applying 23/28 to historical_canon, which has 4 active axes (max 16 points), makes passing mathematically impossible.

exact_document   : F1~F7 전 7축, 총점 ≥ 23 / 28
official_system  : F1·F2(컴포넌트 커버리지로 대체)·F3·F4·F6·F7 6축, 총점 ≥ 20 / 24
historical_canon : F2·F3·F4·F7 4축, 총점 ≥ 13 / 16
style_synthesis  : 채점 대상 아님 (qa.fidelity = not_applicable)

공통 조건:
  모든 활성 축 ≥ 2
  AND 활성 축 중 F1·F2·F3·F7에 해당하는 것 ≥ 3
  AND page_mirror_plan 매핑률 ≥ 80% (exact_document 전용)

Automatic rejection (immediate reject regardless of score)

• Original sha256, page count,reference_mode — any one missing
• Some pages were not viewed directly yet pass is judged — the most common disqualifier
• (The canvas,spec_version, and hash items below are first judged by craft-qa Group A. Here they are used only as a precondition— if that result is fail, scoring does not begin — and are not scored twice.)
• Direct copying of protected assets (logos, emblems, proprietary photos, paid fonts)
• One or more errors in numbers, units, or chart direction
• Missing signature page
• prompt.md·tokens.json·meta.yaml의 spec_version mismatch
• Canvas aspect ratio declaration vs. actual render error exceeds 0.5%
• style_synthesisbut source_depth: full or claims "original text reproduction"

Judgment vocabulary — do not promote to pass

unreviewed · unverifiable · evidence_missing은 passDo not convert to . If required pages could not be viewed directly or verification input is missing, keep it as needs_reviewand block publication.

Quantitative metrics may assistqualitative judgment, but cannot replaceit. Insufficient image viewing quota or time is not a reason to pass, but a reason for needs_review .

Cross-validation with heterogeneous models

Set up a model from a different family than the author (Claude) as the secondary verifier. If the same model grades its own output, false consensus arises.

It is a sample, not an exhaustive review. Image re-review is expensive, so run it only when the following applies:

• The batch's 첫 팩 (an anchor that calibrates that batch's scoring criteria)
• Primary total score is at the pass threshold boundary(22–24 points under exact_document)
• F7 ≤ 2 (suspected rights boundary)
• 그 외 20% sample

Record packs not run as cross_check: {status: not_run, reason: policy}— do not pretend they were run.

codex exec -m gpt-5.6-sol -c model_reasoning_effort=high \
  --sandbox read-only -o _workspace/{batch}/fidelity_qa/{code}_cross.md \
  -i 원본페이지.png -i 팩렌더.png \
  "두 이미지를 대조하라. 첫째는 원본, 둘째는 재현본이다. 아래 7축 루브릭으로 채점하고,
   재현본에 있으나 원본에 없는 요소(환각)를 전부 지목하라. 관대하게 채점하지 마라."

Handling of the two verifiers' results:

• Total score difference ≤ 3 points → adopt the lower one
• Total score difference ≥ 4 points → do not average; needs_human
• Total score 22–24 points (pass threshold boundary) → needs_human
• F7 is 2 points or less on either side → needs_human

Cost discipline

• Run deterministic checks (hash match, aspect ratio, page count,spec_version)를 first via script . If they fail here, do not spend cost on visual verification.
• Get an overview of the full original with a low-resolution contact sheet. However, for pack renders, view every page directly at original resolution.
• High-resolution detailed review only for signature pages and suspected failure areas.
• Original sha256is the same, reuse the previous analysis. prompt/tokenschanges, invalidate only after render; if the original changes, invalidate everything.

Deliverables

_workspace/{batch}/fidelity_qa/{slug}.json

{ "slug": "...", "reference_mode": "exact_document",
  "source_sha256": "...", "pages_reviewed": [1,2,3,4,5,6,7,8,9,10],
  "pages_not_reviewed": [],
  "axes": { "F1": 4, "F2": 4, "F3": 3, "F4": 3, "F5": 3, "F6": 4, "F7": 4 },
  "total": 25, "verdict": "pass",
  "hallucinated_elements": [],
  "missing_signatures": [],
  "cross_check": { "verifier": "gpt-5.6-sol", "total": 24, "delta": 1 },
  "repair_requests": [] }

Verifiers do not modify the pack

fidelity-qa는 repair_requestsonly. Modifications are made by the responsible curator, and after modification, re-run from render onward and re-verify with the new hash. If the verifier fixes it directly, it becomes co-authoring, not verification.