SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsresearch-library

research-library

Writing & research

View repository

Updated 10-16 09:35

About this skill

Research library

$RESEARCH_HOME (default ~/research) is a shared, cross-project store for reference material that shouldn't live inside any single git repo: vendor repo clones, PDFs/epubs, mirrored docs-site snapshots. It exists to avoid two things: (1) agents reading unvetted third-party content ambiently just because it happens to sit inside a repo's working tree, and (2) every project re-cloning the same reference material into its own gitignored folder.

This skill assumes a library exists. It is a plain directory — nothing installs it and nothing but the conventions below depends on the layout. If $RESEARCH_HOME is unset and ~/research does not exist, say so and offer to create it rather than silently falling back to fetching from the web; setting RESEARCH_HOME in a shell profile is the only setup step.

What this skill reads, runs and writes

  • Reads: $RESEARCH_HOME and the entries in it; a repo's own AGENTS.md for pointers. size --ungreppable opens every file in the store and reads its first 8 KB.
  • Runs: git clone/fetch/reset/sparse-checkout/tag -d/reflog expire/gc inside library clones only, and gh api for package health (repo stats, commits, issues, the releases list and the latest release) and for a repo's reported size before add clones it. package_health.py also runs node --version when node is on PATH, to compare a floor, and for its apt source apt-cache policy and apt-cache show, which only read this machine's apt cache.
  • Writes: only inside $RESEARCH_HOME — add clones and writes a provenance file, provenance writes that file, update and deepen and reshallow change clones and their depth: field. name, check and size write nothing, and --dry-run prints what would run. Never a symlink or a copy into a project repo, and nothing outside the store is ever touched.
  • Reads from the environment: RESEARCH_HOME for the store, and RESEARCH_TEXT_ONLY for whether add clones text-only by default.
  • Network: the clone URL you give; PyPI, the npm registry, crates.io, Debian's madison API (api.ftp-master.debian.org), Launchpad (api.launchpad.net) and GitHub for package_health.py, and GitHub for the pre-clone size question, GitHub through your own gh login. Only public read-only endpoints; nothing is uploaded.

Before fetching anything from the web

Check $RESEARCH_HOME/repos/, $RESEARCH_HOME/docs/, and $RESEARCH_HOME/pages/ for existing material on the topic before reaching for WebFetch/WebSearch or cloning a fresh copy. A project's own AGENTS.md may already point at the specific entry relevant to that project.

Layout

$RESEARCH_HOME/
  repos/<host>--<owner>--<repo>/   # shallow git clones
  docs/<file>.pdf|.epub            # downloaded reference docs
  pages/<slug>/                    # mirrored/llms.txt-derived doc site snapshots
  README.md                        # full conventions + rationale

Naming, adding, and checking entries

The convention is <host>--<owner>--<repo> for every repo on every host, with no exceptions and no GitHub special case, and every entry carries a provenance file. Do not derive either by hand — scripts/library.py is that derivation, and the reason it exists is that both failures are silent: a name derived from the URL you were handed rather than from the clone's own origin looks completely normal (self-hosted instances resemble the popular host, and a renamed repo redirects without saying so), and a store nothing version-controls has nothing else that would notice.

python3 $S/scripts/library.py name <url>            # the entry name, from any spelling of a remote
python3 $S/scripts/library.py add <url>             # clone, canonical name, SOURCE.md written
python3 $S/scripts/library.py add <url> --dry-run   # the clone and the file it would write
python3 $S/scripts/library.py check                 # every entry against these conventions

S=~/.agents/skills/research-library. add re-derives the name from the clone's own remote after cloning and renames the entry when they disagree; it records ref as the branch and short sha it actually got, and it clones text-only unless told otherwise — see below for what that leaves out and why it costs a search nothing. For an entry that is not a git clone — a downloaded PDF, a mirrored docs site — fetch it however the material requires, then library.py provenance <path> --url <url> --kind <kind> --ref <ref> writes the metadata half, which is the deterministic part and the part that gets skipped. --json on everything; name and check are read-only, add and provenance write only inside $RESEARCH_HOME.

The provenance file's shape, since check enforces it and a reader may need to fix one by hand: url, kind (repo-clone, llms-txt-mirror or site-mirror), ref (branch/tag/commit for a clone, fetch date for a mirror), fetched, depth when the history was deliberately deepened (see below), text-only when the checkout is narrowed (see below), and note only when non-obvious — e.g. docs publishing from a different branch or repo than the one cloned.

add asks before cloning something the host calls large, exiting 3 with the reported figure rather than prompting — an interactive prompt inside an agent's Bash call hangs with nothing to type into. --yes proceeds, --min <MB> moves the line (default 250 MB), --depth N or --full clones more than one commit and records it. The check is GitHub's API, so a host it cannot ask is never a host it refuses to clone from.

The reported size is a trigger, never a prediction, and the warning deliberately quotes no estimate. Measured 2026-09-07 against five real entries at depth 1: on-disk cost ran from 0.23× the reported size (cpython, 851 MB reported and 192 MB on disk) to 1.32× (Roo-Code, 359 reported and 473 on disk). A 5.7× spread, and not even an upper bound — a warning naming a predicted figure would have been wrong by 4× in the reassuring direction.

Clones hold text, because that is all anyone searches

add clones text-only by default: images, video, audio, fonts, archives, compiled objects and model weights are left on the server, and everything a text search can read is checked out. --all-files opts one entry out; RESEARCH_TEXT_ONLY=0 opts a whole machine out.

python3 $S/scripts/library.py add <url>               # text-only, the default
python3 $S/scripts/library.py add <url> --all-files   # every file, images and binaries included
python3 $S/scripts/library.py add <url> --dry-run     # the two commands and the file it would write

The exclusion cannot cost a search a single hit, and that is the only reason it is a default. Every format it skips is one ripgrep and grep already refuse to search — both treat a file with a NUL byte in its first block as binary and skip it — so those bytes buy a search nothing. This is also why it is not the same decision as excluding a directory, which makes a grep silently blind and stays unadopted here for exactly that reason.

Verified end to end 2026-09-08 on intellectronica/ruler, 97% binary, the same commit cloned both ways: 257 searchable files in both trees, the identical set, grep -rIl ruler returning 190 files in both, and one file absent from the text-only clone — a 69 MB .gif. On disk, 141 MB became 3.0 MB, and .git 70 MB became 644 KB.

That last number is the part worth knowing, because it comes from combining two things that each do nothing alone. --filter=blob:none defers nothing on a --depth 1 clone, since the checkout materialises every blob at HEAD anyway; the sparse set alone shrinks the working tree while .git stays exactly as large. Together, blobs the sparse set never wants are never fetched. A clone carrying one without the other saves a fraction of what it appears to.

Documents are kept, and PDFs are the reason. A PDF is binary to a grep and readable by an agent, so the line is document-versus-demo-asset rather than greppable-versus-not. Nine PDFs sit inside repo clones today — 6.6 MB, 0.5% of the excluded weight, including a 4.6 MB flameshot-documentation.pdf. Half a percent of the saving removes the only case where this could lose something a reader wanted. Ebook and office formats are kept on the same judgement. .svg needs no exemption at all: it is text, so the NUL rule keeps it regardless.

An entry whose checkout is narrower than its HEAD has to say so, or a later grep that finds nothing reads as an answer rather than as a question about what was checked out. add records text-only: yes, and check reports both directions of drift — an entry claiming it over a full checkout, and a sparse checkout nothing records, which also catches a path-based sparse-checkout someone applied by hand.

update needs no special case: a shallow fetch and hard reset leave core.sparseCheckout and the pattern list intact, confirmed on a real refresh.

What the library costs

python3 $S/scripts/library.py size                  # entries at or above 250 MB, biggest first
python3 $S/scripts/library.py size --min 0          # everything
python3 $S/scripts/library.py size --ungreppable    # and how much of it no text search can read

--ungreppable is what makes re-cloning an existing entry text-only a decision with a number attached rather than a principle. It classifies by content, not by the exclusion list, so the report can contradict the patterns — which is how the extensionless residue stays visible: 620 files and 16 MB library-wide, 1.25% of the ungreppable weight and reachable by no extension pattern. Small enough to be the answer rather than a reason for a second filter. It reads every file in the store (184,279 files in 2.9s warm on a 3 GB library), which is why it is a flag.

Measured 2026-09-08 over 86 entries: 1,286 MB of 3,274 MB of working tree, 39%, is ungreppable. It concentrates — block/goose is 310 MB of 343, RooCodeInc/Roo-Code 276 of 293 — and the biggest entry in the library is the counterexample that keeps path exclusion a separate question: nodejs/node is 15 MB of 669, because its 669 MB working tree is vendored V8 and OpenSSL source, greppable to the last byte.

Per entry it splits .git from the working tree, because they have different remedies and the ratio says which applies: a large .git at one commit is big blobs, where the only lever is not keeping the clone; a large working tree at one commit is vendored directories. Measured on the worst entry in a real 4.8 GB library — 940 MB at one commit, of which a single vendored deps/ directory was 675 MB, while the src, lib and doc a reference clone exists for totalled 34 MB. Depth was not the problem there and no amount of re-shallowing would have moved it.

Updating, deepening, and getting the disk back

python3 $S/scripts/library.py update                      # every clone, each at its intended depth
python3 $S/scripts/library.py update <entry>              # just this one
python3 $S/scripts/library.py deepen <entry> --depth 500  # more history, recorded as deliberate
python3 $S/scripts/library.py reshallow <entry>           # back to a depth-1 footprint, disk included

update refreshes each clone to its remote's latest — a shallow fetch and hard reset, since these are disposable reference clones, not working copies with local commits to preserve. It and library.py check answer different questions and neither replaces the other: one moves every clone forward, the other says whether moving it forward can possibly do anything.

A clone deeper than one commit with nothing recorded is skipped, not truncated. Nothing distinguishes a deliberate deepening from an accident, and truncating is the answer that cannot be undone by reading. Confirmed 2026-09-07 in a 71-entry library: exactly one entry was deep, it had been deepened on purpose to read a dependency's constraint history, and a loop refreshing everything with fetch --depth 1 would have destroyed that in silence. Use deepen rather than a hand-typed git fetch, so the intent lands in the entry's depth: field and update can see it.

reshallow exists because re-shallowing by hand does not reclaim the disk. Measured 2026-09-07 on encode/httpx, .git in KB: a fresh --depth 1 clone is 2,492; deepened by 400 commits and re-shallowed with fetch --depth 1 and reset --hard it reports one commit again while the disk does not move at all; adding reflog expire, gc --prune=now, repack -a -d, prune and gc --aggressive reaches 4,852 and stops there. The cause is tags — deepening brings them, each pins a commit deep in history, and every object below stays reachable. Delete them and the same clone lands at 2,472, below where it started. The widely-published sequence omits that step, so run verbatim it leaves a clone permanently 95% larger and reports success. That is five commands in order, one of which appears in no reference material: exactly the shape that belongs in code.

It refuses on a detached HEAD, where the tags may be the thing the clone exists to read; --force overrides.

An entry that looks suspiciously stale after a refresh is usually pinned, and check names it: a clone made with an explicit --branch <tag> leaves HEAD detached and keeps tracking that one ref forever, so git fetch origin re-fetches the same thing and the refresh reports "up to date" on something years old. Confirmed 2026-09-02, the first run of check over a 52-entry library: one such clone, and 51 entries that a cruder rule would have flagged as pinned when they were cloned exactly as this skill says. check --remote adds the one question that needs the network — whether the tracked branch is still the remote's default — and is off by default so the rest stays instant.

Grep the real source, don't trust docs/README prose

Once a repo's cloned, prefer grepping its actual source for ground truth over trusting its README or a docs site's prose — both can be stale or wrong relative to the installed version. This has caught real bugs before: a docs page describing a GNOME keybinding schema that didn't actually exist in the installed GNOME version (only found by reading gnome-shell source directly), and a docs-site build tool whose real mount behavior only matched its actual minified JS bundle, not its rendered docs page.

Both of those are staleness. There is a second, worse shape: a README advertising a feature that was never implemented at all. Confirmed 2026-08-27 while surveying medication trackers for ingesta — a Home Assistant integration's README listed NIH RxNorm ingredient lookup among its features, and neither rxnorm nor ingredient appears anywhere in the repo outside that README. A feature-list comparison scores it as present; one grep settles it. When the question is "does this project do X", the cheap decisive check is whether X appears in the code at all, not whether the docs claim it.

This is also why source beats a hands-on trial for an open-source candidate, where both are available: a trial exercises the path you happened to walk, source shows every path there is, and it answers questions no UI exposes — whether a dose amount is a number or free text, whether a permission is stripped from the manifest.

Judging a candidate dependency

Picking a library or a tool is the same activity as the rule above, one step earlier: the question is what the source says, not what the summary claims. Judge a package from its own registry metadata and file list rather than from a search summary — this is what to look at once you are there.

Judge each candidate against an absolute bar, on its own, before any head-to-head. Popularity is a weak signal past a threshold and is never a tiebreaker — a less popular project that clears the bar beats a more popular one that does not. Stated 2026-08-30 while choosing between two Telegram libraries, and it is the reason the script prints stars under not scored.

python3 $S/scripts/package_health.py pypi <name>
python3 $S/scripts/package_health.py pypi anyio --clone $RESEARCH_HOME/repos/github.com--agronholm--anyio
python3 $S/scripts/package_health.py pypi shellcheck-py --upstream koalaman/shellcheck
python3 $S/scripts/package_health.py npm @biomejs/biome
python3 $S/scripts/package_health.py crates ripgrep
python3 $S/scripts/package_health.py apt ripgrep
python3 $S/scripts/package_health.py github BurntSushi/ripgrep

Stdlib only, S=~/.agents/skills/research-library. The source is a required subcommand — pypi, npm, crates, apt or github, with no default, so name it every time; the retired package_health.py <name> <owner/repo> form exits 2 naming its replacement. When one tool is offered through several channels — a PyPI or npm wrapper, a crate, an apt package, a release binary — run each channel's source and compare what they ship and cost. The package's own GitHub repo is read from its metadata and the report's repo from line says where; pass --repo <owner/repo> when that lookup is wrong or missing. Registries over HTTPS, GitHub through gh api so it uses your own token and rate limit. --json for the whole answer on every source.

The axes every source reports where they apply, and what each is for:

  • Maintenance — releases on the stable line and their median gap, last push against last release (which separates "actively developed, slow to release" from "stalled"), human contributor count and bus factor over the last year, time to close an issue, archived flag, licence, yanked or deprecated releases.
  • Typing, for dynamically typed ecosystems only — PyPI's py.typed (with --clone) and npm's own types against a separate @types/… against neither. For Python, also the project's own type-checker config and its strictness, which predicts what leaks. Then measure: run your checker in your mode over a small real usage sample. Nothing else tells you what your gate will say.
  • Battle-tested — with pypi --clone: test-to-source ratio against hand-written source (--generated <glob> marks a mechanical layer), whether coverage is enforced in CI or only reported, and what the CI workflows actually cover.
  • Fit — runtime dependency count and names (never the raw requires_dist, which is mostly extras), version ceilings and whether they bind the distributed artifact or only the dev lockfile, licence compatibility, and whether the thing can be exercised offline.
  • Ships — what installing the latest stable release actually downloads, and its size: the adoption cost. What that means differs per source; see below.
  • Floors — every minimum version the metadata states, never inferred, each compared with this machine where that is cheap: requires_python against this python3, the glibc floor a manylinux tag or a libc6 dependency states against this machine's glibc, engines.node against this node, a crate's MSRV, the libc family an asset name states. NOT MET HERE is the finding; not compared says why it was not.
  • Upstream, with --upstream <owner/repo> on pypi and npm — for a wrapper, name the project it repackages, which is not the wrapper's own repo. It reports upstream's latest stable GitHub release, the wrapper version spelled to match it and the lag in days, and that release's Linux x86_64 assets with any checksum or signature file beside them. NO MATCHING WRAPPER RELEASE means the spelling differs or the wrapper is behind: compare the two versions printed. GitHub digest is GitHub's hash of the upload, not a publisher's checksum.

What each source adds:

  • pypi <name> — the wheels and sdist, their tags, sizes and upload dates. Read the linux x86_64 line before adopting a wrapper around a binary: a wheel present means the binary is inside it; SDIST ONLY or NO WHEEL for this machine means pip builds it, or a build hook downloads it, on every install. --clone <path> adds what no API answers — py.typed, the test-to-source ratio, the CI inventory, the licence files actually present.
  • npm <name> — judged on dist-tags.latest, never the newest publish. install scripts (RUNS AT INSTALL means code executes on your machine at install time), provenance, and the platform package that actually carries a binary — resolved by default, because a wrapper like @biomejs/biome is 779 kB and its Linux package 64.7 MB. install size is the two together.
  • crates <name> — crates.io's own stable version, the crate's size and bin_names, and the MSRV (undeclared when null). A crate is source that cargo install compiles; when it has binaries, a prebuilt binary section lists the latest GitHub release's Linux assets.
  • apt <name> — a different question, and the report says so: an apt package is maintained by the distro. It reports this machine's candidate and who maintains it (main is Canonical-supported; universe gets Canonical security fixes only with Ubuntu Pro; a third-party repo is its publisher's), sizes, the lag behind upstream's stable GitHub releases (upstream read from the package's Homepage, or --upstream), and a cross-release view of which Ubuntu and Debian releases carry which version — for choosing between apt and a release binary, and for what the next LTS brings.
  • github <owner/repo> — for a tool with no registry at all, a Go or Rust binary shipped only as release assets: the maintenance axis, the stable-release cadence from the releases list, and the latest stable release's Linux x86_64 assets.

Four of the report's lines are traps wearing the shape of an answer, so read them as written: open issues+PRs is GitHub's field and counts both; a PRE-RELEASE ONLY or pre-releases line means a dev version is moving while the stable one may not be; a shallow clone line means any history question needs git fetch --deepen first; an npm tarball size is the wrapper alone, and install size is the number that counts. Confirmed 2026-09-02 on httpx: read naively, PyPI says 4 releases in the last year with the newest yesterday. On the stable line it is zero in the last year, the last one 634 days ago — the opposite answer to "is this maintained for me".

Everything the numbers hide — the bot that dominates a bus factor, the dual-licensed project the API reports as GPL, why a version cap is not a cost until its historical lag says so, how to prove offline-testability, and each registry's own traps — is in references/dependency-health.md. Read it before writing a recommendation, not before running the script.

No symlinks into project repos

Never symlink $RESEARCH_HOME or any entry in it into a project's working tree. That would put this content back in the ambient read path of anything scoped to that repo — the entire reason it lives outside every repo. Reach it by its $RESEARCH_HOME path directly, only when a task actually calls for it.

Per-project pointers

This skill can't know which entries matter to a given project. Each project's own AGENTS.md should name the specific paths relevant to it ("for GNOME Shell extension behaviour, check $RESEARCH_HOME/repos/gitlab.gnome.org--GNOME--gnome-shell before reading anything online"), as a standing rule rather than a suggestion. That is also what makes the library reachable by agents that read AGENTS.md but have no skill-discovery mechanism.

Full design rationale

references/rationale.md — why this exists rather than a per-repo reference/ directory, why naming has no special case for the popular host, the pinned-tag refresh trap, the docs-site mirroring research (llms.txt and its fallbacks), and why RAG/embeddings were researched and deliberately not adopted.