wiki

Upstream hierarchy:

Canonical source: Andrej Karpathy's LLM Wiki pattern — the core methodology this skill implements.

Conceptual enhancements: nashsu/llm_wiki — methodology niceties adopted conceptually (not code): purpose.md, Two-Step Chain-of-Thought Ingest, overview.md, 4-signal relevance model, graph insights, dedup pipeline, cohesion scoring. The upstream desktop app, API, MCP server, and vector search are not used — this skill is a standalone agent-curated markdown wiki with no external dependencies.

v3.0 changelog: Adopted the nashsu enhancements above as conceptual improvements to the Karpathy pattern.

Karpathy's LLM Wiki

Build and maintain a persistent, compounding knowledge base as interlinked markdown files. Based on Andrej Karpathy's LLM Wiki pattern.

Unlike traditional RAG (which rediscovers knowledge from scratch per query), the wiki compiles knowledge once and keeps it current. Cross-references are already there. Contradictions have already been flagged. Synthesis reflects everything ingested.

Division of labor: The human curates sources and directs analysis. The agent summarizes, cross-references, files, and maintains consistency.

When This Skill Activates

Use this skill when the user:

Asks to create, build, or start a wiki or knowledge base
Asks to ingest, add, or process a source into their wiki
Asks a question and an existing wiki is present at the configured path
Asks to lint, audit, or health-check their wiki
References their wiki, knowledge base, or "notes" in a research context

Wiki Location

Location: Plugin config first, then environment fallback.

paths.wiki_path in the Lumen plugin config (shared/config/lumen.yaml, or --config)
$WIKI_PATH environment variable
Default ~/wiki

# Resolved by config_loader.py. Equivalent fallback:
WIKI="${WIKI_PATH:-$HOME/wiki}"

The wiki is just a directory of markdown files — open it in Obsidian, VS Code, or any editor. No database, no special tooling required.

Architecture: Three Layers + Purpose

wiki/
├── SCHEMA.md           # Conventions, structure rules, domain config
├── purpose.md          # The wiki's soul — goals, key questions, scope (read on every ingest/query)
├── index.md            # Sectioned content catalog with one-line summaries
├── overview.md         # Auto-regenerated global summary of the entire wiki
├── log.md              # Chronological action log (append-only, rotated yearly)
├── raw/                # Layer 1: Immutable source material
│   ├── articles/       # Web articles, clippings
│   ├── papers/         # PDFs, arxiv papers
│   ├── transcripts/    # Meeting notes, interviews
│   └── assets/         # Images, diagrams referenced by sources
├── entities/           # Layer 2: Entity pages (people, orgs, products, models)
├── concepts/           # Layer 2: Concept/topic pages
├── comparisons/        # Layer 2: Side-by-side analyses
└── queries/            # Layer 2: Filed query results worth keeping

Layer 1 — Raw Sources: Immutable. The agent reads but never modifies these. Layer 2 — The Wiki: Agent-owned markdown files. Created, updated, and cross-referenced by the agent. Layer 3 — The Schema: SCHEMA.md defines structure, conventions, and tag taxonomy. Purpose — purpose.md: Defines goals, key questions, research scope, and evolving thesis. The LLM reads it during every ingest and query to stay aligned with the wiki's intent. This is different from schema — schema is structural rules, purpose is directional intent.

Resuming an Existing Wiki (CRITICAL — do this every session)

When the user has an existing wiki, always orient yourself before doing anything:

① Read purpose.md — understand the wiki's goals, scope, and key questions. ② Read SCHEMA.md — understand the domain, conventions, and tag taxonomy. ③ Read index.md — learn what pages exist and their summaries. ④ Scan recent log.md — read the last 20-30 entries to understand recent activity.

WIKI="${WIKI_PATH:-$HOME/wiki}"
# Orientation reads at session start
read_file "$WIKI/purpose.md"
read_file "$WIKI/SCHEMA.md"
read_file "$WIKI/index.md"
read_file "$WIKI/log.md" offset=<last 30 lines>

Only after orientation should you ingest, query, or lint. This prevents:

Creating duplicate pages for entities that already exist
Missing cross-references to existing content
Contradicting the schema's conventions
Repeating work already logged

For large wikis (100+ pages), also run a quick search_files for the topic at hand before creating anything new.

Initializing a New Wiki

When the user asks to create or start a wiki:

Determine the wiki path (from plugin config paths.wiki_path, else $WIKI_PATH, else ask the user; default ~/wiki)
Create the directory structure above
Ask the user what domain the wiki covers — be specific
Write purpose.md — goals, key questions, scope, evolving thesis (see template below)
Write SCHEMA.md customized to the domain (see template below)
Write initial index.md with sectioned header
Write initial overview.md with a brief summary
Write initial log.md with creation entry
Confirm the wiki is ready and suggest first sources to ingest

purpose.md Template

The wiki's soul — directional intent, not structural rules. Read on every ingest and query.

# Wiki Purpose

## Goal
[What this wiki is for — e.g., "Track AI/ML research and synthesize learnings into a compounding knowledge base"]

## Key Questions
- [What questions should this wiki help answer?]

## Scope
- **In scope:** [What earns a page]
- **Out of scope:** [What doesn't belong here]

## Evolving Thesis
[Your current understanding of the domain — updated as the wiki grows]

SCHEMA.md Template

Adapt to the user's domain. The schema constrains agent behavior and ensures consistency:

# Wiki Schema

## Domain
[What this wiki covers — e.g., "AI/ML research", "personal health", "startup intelligence"]

## OKF Conformance

This wiki is conformant with the [Open Knowledge Format (OKF) v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
— Google's open specification for agent-readable knowledge bundles.

**Conformance requirements (all met):**
- Every non-reserved `.md` file has parseable YAML frontmatter ✓
- Every frontmatter block has a non-empty `type` field ✓
- Reserved filenames (`index.md`, `log.md`) follow OKF structure ✓

**OKF fields used:**
- `type` (required) — concept type, also used for our richer taxonomy
- `title` (recommended) — display name
- `description` (recommended) — one-line summary, used by index generators and search
- `resource` (optional) — canonical URI for external assets
- `tags` (optional) — cross-cutting categorization
- `updated` (maps to OKF `timestamp`) — last-modified date

**Extensions beyond OKF (allowed by spec):**
- `created` — creation date (producer-defined extension)
- `sources` — provenance links to raw source files
- `confidence`, `contested`, `contradictions` — quality signals
- `purpose.md`, `SCHEMA.md`, `overview.md` — wiki-specific reserved files
- `[[wikilinks]]` — Obsidian-style cross-links (OKF uses standard markdown links;
  both coexist — `[[wikilinks]]` for Obsidian, standard links for OKF consumers)
- `raw/` layer — immutable source material (OKF has no equivalent)

**Root index.md declares OKF version:**
```yaml
okf_version: "0.2"

Conventions

File names: lowercase, hyphens, no spaces (e.g., transformer-architecture.md)
Every wiki page starts with YAML frontmatter (see below)
Use [[wikilinks]] to link between pages (minimum 2 outbound links per page)
When updating a page, always bump the updated date
Every new page must be added to index.md under the correct section
Every action must be appended to log.md
Provenance markers: On pages that synthesize 3+ sources, append ^[raw/articles/source-file.md] at the end of paragraphs whose claims come from a specific source. This lets a reader trace each claim back without re-reading the whole raw file. Optional on single-source pages where the sources: frontmatter is enough.

Frontmatter

---
type: entity | concept | comparison | query | summary   # REQUIRED (OKF)
title: Page Title
description: One-line summary of this concept             # OKF recommended
created: YYYY-MM-DD
updated: YYYY-MM-DD                                      # maps to OKF timestamp
tags: [from taxonomy below]
sources: [raw/articles/source-name.md]
resource: <Optional canonical URI for the underlying asset>  # OKF optional
# Optional quality signals:
confidence: high | medium | low        # how well-supported the claims are
contested: true                        # set when the page has unresolved contradictions
contradictions: [other-page-slug]      # pages this one conflicts with
---

OKF conformance: type is the only required field. All other fields are extensions that OKF consumers must tolerate. description enables index generation and search snippets in OKF consumers. resource links to external assets (APIs, dashboards, repos). updated serves as OKF's timestamp.

confidence and contested are optional but recommended for opinion-heavy or fast-moving topics. Lint surfaces contested: true and confidence: low pages for review so weak claims don't silently harden into accepted wiki fact.

raw/ Frontmatter

Raw sources ALSO get a small frontmatter block so re-ingests can detect drift:

---
source_url: https://example.com/article   # original URL, if applicable
ingested: YYYY-MM-DD
sha256: <hex digest of the raw content below the frontmatter>
---

The sha256: lets a future re-ingest of the same URL skip processing when content is unchanged, and flag drift when it has changed. Compute over the body only (everything after the closing ---), not the frontmatter itself.

Tag Taxonomy

[Define 10-20 top-level tags for the domain. Add new tags here BEFORE using them.]

Example for AI/ML:

Models: model, architecture, benchmark, training
People/Orgs: person, company, lab, open-source
Techniques: optimization, fine-tuning, inference, alignment, data
Meta: comparison, timeline, controversy, prediction

Rule: every tag on a page must appear in this taxonomy. If a new tag is needed, add it here first, then use it. This prevents tag sprawl.

Page Thresholds

Create a page when an entity/concept appears in 2+ sources OR is central to one source
Add to existing page when a source mentions something already covered
DON'T create a page for passing mentions, minor details, or things outside the domain
Split a page when it exceeds ~200 lines — break into sub-topics with cross-links
Archive a page when its content is fully superseded — move to _archive/, remove from index

Entity Pages

One page per notable entity. Include:

Overview / what it is
Key facts and dates
Relationships to other entities ([[wikilinks]])
Source references

Concept Pages

One page per concept or topic. Include:

Definition / explanation
Current state of knowledge
Open questions or debates
Related concepts ([[wikilinks]])

Comparison Pages

Side-by-side analyses. Include:

What is being compared and why
Dimensions of comparison (table format preferred)
Verdict or synthesis
Sources

Update Policy

When new information conflicts with existing content:

Check the dates — newer sources generally supersede older ones
If genuinely contradictory, note both positions with dates and sources
Mark the contradiction in frontmatter: contradictions: [page-name]
Flag for user review in the lint report


### index.md Template

The index is sectioned by type. Each entry is one line: wikilink + summary.

```markdown
# Wiki Index

---
okf_version: "0.2"
---

> Content catalog. Every wiki page listed under its type with a one-line summary.
> Read this first to find relevant pages for any query.
> Last updated: YYYY-MM-DD | Total pages: N

## Entities
<!-- Alphabetical within section -->

## Concepts

## Comparisons

## Queries

Curator-managed block: when lumen:curator runs on the wiki, it regenerates only the text between  and  in index.md and overview.md (appending that block if a file has none). Keep hand-maintained sections, like the ones above, outside the markers; never edit inside them, since the next run replaces it.

Scaling rule: When any section exceeds 50 entries, split it into sub-sections by first letter or sub-domain. When the index exceeds 200 entries total, create a _meta/topic-map.md that groups pages by theme for faster navigation.

log.md Template

# Wiki Log

> Chronological record of all wiki actions. Append-only.
> Format: `## [YYYY-MM-DD] action | subject`
> Actions: ingest, update, query, lint, create, archive, delete
> When this file exceeds 500 entries, rotate: rename to log-YYYY.md, start fresh.

## [YYYY-MM-DD] create | Wiki initialized
- Domain: [domain]
- Structure created with purpose.md, SCHEMA.md, index.md, overview.md, log.md

overview.md Template

Auto-regenerated after every ingest. The wiki's "elevator pitch" for fast orientation.

# Wiki Overview

> Auto-generated. Last updated: YYYY-MM-DD | Total pages: N

## Summary
[2-3 paragraph summary of what the wiki covers and its current state]

## Main Topics
- [Topic 1] — [N pages] — [brief description]
- [Topic 2] — [N pages] — [brief description]

## Recent Changes
- [YYYY-MM-DD] [action] — [what changed]
- [YYYY-MM-DD] [action] — [what changed]

## Key Themes
[Emerging patterns and connections across the wiki]

Core Operations

1. Ingest (Two-Step Chain-of-Thought)

When the user provides a source (URL, file, paste), integrate it into the wiki in two sequential phases for better quality — Analysis, then Generation.

Step 1: Capture & Analysis

① Capture the raw source:

URL → use web_extract to get markdown, save to raw/articles/
PDF → use web_extract (handles PDFs), save to raw/papers/
Pasted text → save to appropriate raw/ subdirectory
Name the file descriptively: raw/articles/karpathy-llm-wiki-2026.md
Add raw frontmatter (source_url, ingested, sha256 of the body). On re-ingest of the same URL: recompute the sha256, compare to the stored value — skip if identical, flag drift and update if different. This is cheap enough to do on every re-ingest and catches silent source changes.

② Read purpose.md — align with the wiki's goals and scope before analyzing.

③ Discuss takeaways with the user — what's interesting, what matters for the domain. (Skip this in automated/cron contexts — proceed directly.)

④ Check what already exists — search index.md and use search_files to find existing pages for mentioned entities/concepts. This is the difference between a growing wiki and a pile of duplicates.

⑤ Structured Analysis — before writing any pages, produce an analysis:

Key entities, concepts, arguments in the source
Connections to existing wiki content (which pages relate, how)
Contradictions & tensions with existing knowledge
Recommendations for wiki structure (new pages vs updates vs merges)

In conversation context, share this analysis with the user. In automated/cron contexts, proceed directly to Step 2.

Step 2: Generation

⑥ Write or update wiki pages based on the analysis:

New entities/concepts: Create pages only if they meet the Page Thresholds in SCHEMA.md (2+ source mentions, or central to one source)
Existing pages: Add new information, update facts, bump updated date. When new info contradicts existing content, follow the Update Policy.
Cross-reference: Every new or updated page must link to at least 2 other pages via [[wikilinks]]. Check that existing pages link back.
Tags: Only use tags from the taxonomy in SCHEMA.md
Provenance: On pages synthesizing 3+ sources, append ^[raw/articles/source.md] markers to paragraphs whose claims trace to a specific source.
Confidence: For opinion-heavy, fast-moving, or single-source claims, set confidence: medium or low in frontmatter. Don't mark high unless the claim is well-supported across multiple sources.

⑦ Update navigation & overview:

Add new pages to index.md under the correct section, alphabetically
Update the "Total pages" count and "Last updated" date in index header
Edit hand-maintained content only outside the  …  block (the curator owns that block)
Regenerate overview.md — a global summary of the entire wiki, updated after every ingest. Include: total pages, main topics, recent changes, key themes. This is the wiki's "elevator pitch" and helps the LLM orient quickly on future sessions.
Append to log.md: ## [YYYY-MM-DD] ingest | Source Title
List every file created or updated in the log entry

⑧ Report what changed — list every file created or updated to the user.

A single source can trigger updates across 5-15 wiki pages. This is normal and desired — it's the compounding effect.

2. Query (4-Signal Relevance Model)

When the user asks a question about the wiki's domain:

① Read purpose.md — align with the wiki's scope. ② Read index.md to identify relevant pages. ③ For wikis with 100+ pages, also search_files across all .md files for key terms — the index alone may miss relevant content. ④ Read the relevant pages using read_file. ⑤ Rank results using the 4-signal relevance model:

Signal	Weight	Description
Direct link	×3.0	Pages linked via `[[wikilinks]]` from seed pages
Source overlap	×4.0	Pages sharing the same raw source (via frontmatter `sources[]`)
Adamic-Adar	×1.5	Pages sharing common neighbors (weighted by neighbor degree)
Type affinity	×1.0	Bonus for same page type (concept→concept, entity→entity)

For each candidate page, compute a composite score. Start with keyword-matched seed pages, then expand 2 hops via [[wikilinks]] and source overlap. Rank by composite score. Read only the top pages (budget control: ~60% bodies / 20% chat context / 5% index / 15% system prompt).

⑥ Synthesize an answer from the compiled knowledge. Cite the wiki pages you drew from: "Based on [[page-a]] and [[page-b]]..." ⑦ File valuable answers back — if the answer is a substantial comparison, deep dive, or novel synthesis, create a page in queries/ or comparisons/. Don't file trivial lookups — only answers that would be painful to re-derive. ⑧ Update log.md with the query and whether it was filed.

3. Lint

When the user asks to lint, health-check, or audit the wiki:

① Orphan pages: Find pages with no inbound [[wikilinks]] from other pages.

# Use execute_code for this — programmatic scan across all wiki pages
import os, re
from collections import defaultdict
wiki = "<WIKI_PATH>"
# Scan all .md files in entities/, concepts/, comparisons/, queries/
# Extract all [[wikilinks]] — build inbound link map
# Pages with zero inbound links are orphans

② Broken wikilinks: Find [[links]] that point to pages that don't exist.

③ Index completeness: Every wiki page should appear in index.md. Compare the filesystem against index entries.

④ Frontmatter validation: Every wiki page must have all required fields (title, created, updated, type, tags, sources). Tags must be in the taxonomy.

⑤ Stale content: Pages whose updated date is >90 days older than the most recent source that mentions the same entities.

⑥ Contradictions: Pages on the same topic with conflicting claims. Look for pages that share tags/entities but state different facts. Surface all pages with contested: true or contradictions: frontmatter for user review.

⑦ Quality signals: List pages with confidence: low and any page that cites only a single source but has no confidence field set — these are candidates for either finding corroboration or demoting to confidence: medium.

⑧ Source drift: For each file in raw/ with a sha256: frontmatter, recompute the hash and flag mismatches. Mismatches indicate the raw file was edited (shouldn't happen — raw/ is immutable) or ingested from a URL that has since changed. Not a hard error, but worth reporting.

⑨ Page size: Flag pages over 200 lines — candidates for splitting.

⑩ Tag audit: List all tags in use, flag any not in the SCHEMA.md taxonomy.

⑪ Log rotation: If log.md exceeds 500 entries, rotate it.

⑫ Graph insights — surprising connections: Find cross-community edges (pages in different Louvain clusters linked via [[wikilinks]]), cross-type links (entity↔concept), and peripheral↔hub couplings. These are often the most interesting parts of the knowledge base — flag them for the user to review.

⑬ Graph insights — knowledge gaps:

Isolated pages: degree ≤ 1 (few or no connections — candidates for expansion)
Sparse communities: Louvain clusters with cohesion < 0.15 and ≥ 3 pages
Bridge nodes: pages connecting 3+ clusters (high leverage but fragile)

⑭ Deduplication check: Look for pages with similar titles or overlapping content. If two pages cover the same concept with >70% content overlap, flag for merge. Use LLM judgment: extract summaries, compare, recommend merge or keep-separate. Do NOT auto-merge — surface as a review item.

⑮ Report findings with specific file paths and suggested actions, grouped by severity (broken links > orphans > source drift > contested pages > stale content > style issues > graph insights > dedup candidates).

⑯ Append to log.md: ## [YYYY-MM-DD] lint | N issues found

Working with the Wiki

Searching

# Find pages by content
search_files "transformer" path="$WIKI" file_glob="*.md"

# Find pages by filename
search_files "*.md" target="files" path="$WIKI"

# Find pages by tag
search_files "tags:.*alignment" path="$WIKI" file_glob="*.md"

# Recent activity
read_file "$WIKI/log.md" offset=<last 20 lines>

Bulk Ingest

When ingesting multiple sources at once, batch the updates:

Read all sources first
Identify all entities and concepts across all sources
Check existing pages for all of them (one search pass, not N)
Create/update pages in one pass (avoids redundant updates)
Update index.md once at the end
Write a single log entry covering the batch

Archiving

When content is fully superseded or the domain scope changes:

Create _archive/ directory if it doesn't exist
Move the page to _archive/ with its original path (e.g., _archive/entities/old-page.md)
Remove from index.md
Update any pages that linked to it — replace wikilink with plain text + "(archived)"
Log the archive action

Obsidian Integration

The wiki directory works as an Obsidian vault out of the box:

[[wikilinks]] render as clickable links
Graph View visualizes the knowledge network
YAML frontmatter powers Dataview queries
The raw/assets/ folder holds images referenced via ![[image.png]]

For best results:

Set Obsidian's attachment folder to raw/assets/
Enable "Wikilinks" in Obsidian settings (usually on by default)
Install Dataview plugin for queries like TABLE tags FROM "entities" WHERE contains(tags, "company")

If using the Obsidian skill alongside this one, set OBSIDIAN_VAULT_PATH to the same directory as the wiki path.

Obsidian Headless (servers and headless machines)

On machines without a display, use obsidian-headless instead of the desktop app. It syncs vaults via Obsidian Sync without a GUI — perfect for agents running on servers that write to the wiki while Obsidian desktop reads it on another device.

Setup:

# Requires Node.js 22+
npm install -g obsidian-headless

# Login (requires Obsidian account with Sync subscription)
ob login --email <email> --password '<password>'

# Create a remote vault for the wiki
ob sync-create-remote --name "LLM Wiki"

# Connect the wiki directory to the vault
cd ~/wiki
ob sync-setup --vault "<vault-id>"

# Initial sync
ob sync

# Continuous sync (foreground — use systemd for background)
ob sync --continuous

Continuous background sync via systemd:

# ~/.config/systemd/user/obsidian-wiki-sync.service
[Unit]
Description=Obsidian LLM Wiki Sync
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/path/to/ob sync --continuous
WorkingDirectory=/home/user/wiki
Restart=on-failure
RestartSec=10

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now obsidian-wiki-sync
# Enable linger so sync survives logout:
sudo loginctl enable-linger $USER

This lets the agent write to ~/wiki on a server while you browse the same vault in Obsidian on your laptop/phone — changes appear within seconds.

Pitfalls

Never modify files in raw/ — sources are immutable. Corrections go in wiki pages.
Always orient first — read SCHEMA + index + recent log before any operation in a new session. Skipping this causes duplicates and missed cross-references.
Always update index.md and log.md — skipping this makes the wiki degrade. These are the navigational backbone.
Don't create pages for passing mentions — follow the Page Thresholds in SCHEMA.md. A name appearing once in a footnote doesn't warrant an entity page.
Don't create pages without cross-references — isolated pages are invisible. Every page must link to at least 2 other pages.
Frontmatter is required — it enables search, filtering, and staleness detection.
Tags must come from the taxonomy — freeform tags decay into noise. Add new tags to SCHEMA.md first, then use them.
Keep pages scannable — a wiki page should be readable in 30 seconds. Split pages over 200 lines. Move detailed analysis to dedicated deep-dive pages.
Ask before mass-updating — if an ingest would touch 10+ existing pages, confirm the scope with the user first.
Rotate the log — when log.md exceeds 500 entries, rename it log-YYYY.md and start fresh. The agent should check log size during lint.
Handle contradictions explicitly — don't silently overwrite. Note both claims with dates, mark in frontmatter, flag for user review.
Read purpose.md on every ingest and query — skipping this causes drift from the wiki's intended scope. Purpose is the compass; schema is the rulebook.
Regenerate overview.md after every ingest — a stale overview is worse than no overview because it misleads orientation. It should reflect the current state of the wiki.
Page merge on re-ingest — when re-ingesting a source that was already processed, use a 3-layer merge: (1) array union for lists/links, (2) LLM body merge with a 70% length sanity check (if merged body is <70% or >130% of the longer original, flag for manual review), (3) locked frontmatter fields (title, created — never overwritten). This prevents both information loss and uncontrolled bloat from repeated re-ingestion.
Dedup is advisory, not automatic — the dedup check surfaces candidates but never auto-merges. Two pages with similar titles may cover genuinely different aspects. Always present to the user.

Upstream & Related

Canonical source: Andrej Karpathy's LLM Wiki gist — the pattern this skill implements.

Conceptual enhancements adopted from: nashsu/llm_wiki — methodology niceties (purpose.md, two-step CoT ingest, overview.md, 4-signal relevance, graph insights, dedup, cohesion scoring). We adopt the concepts, not the code. The upstream desktop app, API, MCP server, vector search, and Chrome clipper are separate products and are not used by this skill. Upstream app is GPL-3.0; this skill remains MIT (original implementation, no upstream code/text imported).

Related CLI tool: llm-wiki-compiler — a Node.js CLI that compiles sources into a concept wiki with the same Karpathy inspiration. Obsidian-compatible. Use this skill when you want agent-in-the-loop curation; use llm-wiki-compiler when you want batch compile of a source directory.