dedupe-rank

Dedupe + Rank

Triggers & routing

• Trigger: dedupe, rank, core set, deduplication, ranking, selected papers, core set.
• Use when: After retrieval, the broad-coverage set needs to be narrowed down into a manageable core set (used for taxonomy/outline/mapping).
• Skip if: Someone has already manually curated a stable papers/core_set.csv(no need to churn again).
• Network: none.
• Guardrail: Leaning deterministic; output should be reproducible (stable paper_id, field specifications).

Turns a raw candidate pool into a deduped pool and a stable core set.

Input

• papers/papers_raw.jsonl

Outputs

• papers/papers_dedup.jsonl
• papers/core_set.csv

Script boundary

scripts/run.py should own only:

• title/year deduplication
• deterministic ranking
• stable paper_id generation

Use shared domain packs or pipeline contract metadata for topic-specific or product-specific behavior.

Contract-driven behavior

The script should prefer pipeline contract metadata over profile-name branching.

Current important field:

• quality_contract.candidate_pool_policy.keep_full_deduped_pool
• quality_contract.candidate_pool_policy.include_domain_pins
• quality_contract.candidate_pool_policy.minimum_domain_surveys
• quality_contract.candidate_pool_policy.survey_title_bonus

If true, the script keeps the full deduped pool in papers/core_set.csv unless the user explicitly overrides core size. Focused briefing Workflows can disable unconditional domain pins, reserve a smaller survey floor, and remove the generic survey-title bonus without changing broad Survey defaults.

Acceptance

• deduped JSONL exists
• core-set CSV exists
• reruns are stable for the same inputs

Non-goals

• retrieval
• screening
• manual topic authoring inside the script