source-ingest

Source Ingest

Triggers & routing

• Trigger: source ingest, ingest sources, normalize tutorial sources, web extraction, material normalization.
• Use when: source-tutorial for C1, it is necessary to turn the sources/manifest.yml web pages/PDF/repo/docs into traceable text.
• Skip if: the source manifest has not been decided yet, or the sources have not been confirmed.
• Network: required for remote URLs.
• Guardrail: only successfully extracted content is treated as a valid source; failed sources must be recorded to disk and must not be silently ignored.

Goal: normalize mixed source inputs into local tutorial-ready text while preserving provenance.

Inputs

• sources/manifest.yml

Outputs

• sources/index.jsonl
• sources/provenance.jsonl

Supported kinds

• webpage
• pdf
• markdown
• repo
• docs_site
• video

Behavior

• Accept local file paths or remote URLs from sources/manifest.yml.
• For video, use transcript-first ingestion:
  • provided transcript_locator
  • Bilibili subtitles when available
  • otherwise fail clearly instead of pretending the watch page is usable text
• Plain YouTube/Bilibili watch pages should not be modeled as kind: webpage.
• Continue past optional-source failures and record warnings.
• Fail closed when any source marked required: true cannot be ingested.

Script

Quick Start

• uv run python .codex/skills/source-ingest/scripts/run.py --workspace <workspace>

All Options

• --workspace <dir> (required)
• --unit-id <U###>
• --inputs <semicolon-separated>
• --outputs <semicolon-separated>
• --checkpoint <C#>

Examples

• Ingest all listed sources:
  • uv run python .codex/skills/source-ingest/scripts/run.py --workspace <workspace>

Troubleshooting

Issue: repo or docs site ingests too much noise

Fix:

• Narrow the manifest to the most relevant docs root or replace the source with a specific docs page.