SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsevaluate-shortform

evaluate-shortform

Marketing & growth

View repository

Updated 10-15 21:43

About this skill

Short-Form Eval — Orchestrator

Feedback-loop skill — closes the brief → publish → score → pattern-log loop for a published short-form video. Scores hook + hold + save/share + comment-quality fidelity against the brief-shortform hypothesis. Gap-gate consumes its outputs to decide what the stack should learn next.

Core Question: "Did the published video survive contact with the platform — and what's the signal-bearing pattern this cycle adds to the log?"

Why, methodology, refutability, rubric-revision discipline, when NOT to use: references/playbook.md [PLAYBOOK].

Critical Gates

  1. Provisional rubric, not locked. references/rubric.md is v0.1, provisional. Mandatory revision after cycle 2-3 against real variance. Encode per-cycle drift in the artifact, never silently in the rubric file.
  2. Cycle 1 = 70% observation / 30% scoring. First cycle leans toward describing what you saw; later cycles harden scoring as variance accumulates.
  3. Brief + reference catalog must exist. No platform-intel reference → BLOCKED. No brief-shortform hypothesis → BLOCKED. The eval scores a fidelity claim against known patterns; missing either side reduces the run to vibes.
  4. No fabricated metrics. Every engagement number, completion rate, hold-curve point, save/share count, and sample-size claim cites the URL or panel screenshot it came from. Critic rubric #1 fails the artifact otherwise.
  5. Pattern-log entries are atomic. One cycle = one pattern-log entry block (claim · evidence · refutability · expiry). Free-form prose patterns are unusable downstream.

Quality Gate

Critic verifies before delivery (all four binary PASS required, max 2 rewrite cycles):

  • Every metric and observation has a source URL or panel/screenshot citation
  • All four rubric dimensions scored against v0.1 rubric in references/rubric.md; each score has a one-sentence falsifiable justification
  • At least one pattern-log entry exists in canonical block shape (claim / evidence / refutability / expiry)
  • Cycle 1 weighting (70 obs / 30 score) honored — prose-to-score ratio is observation-heavy on cycle 1

Before Starting

Apply the before-starting-check [PLAYBOOK]:

  1. Mode resolution per references/_shared/mode-resolver.md [PROCEDURE]. Skill is budget: standard; --fast collapses to single-pass eval-runner (skip hook-strength parallel + skip pattern-extractor as separate Layer-2 agent), but Critical Gates STILL enforced. Cold Start fires under --fast if catalog/brief/loop unresolvable. Session execution profile (single-vs-multi): inherit per references/_shared/execution-policy.md.
  2. Read implementation-roadmap/canonical-paths.md if present — verify output paths match canonical inventory (.forsvn/loops/[slug]/evals/ + results.tsv).
  3. Read .forsvn/index/manifest.json — find matching research-shortform catalog by topic+market; check freshness.
  4. Run Pre-Dispatch per references/procedures/pre-dispatch.md [PROCEDURE] — read order, Cold/Warm prompts, hard-block conditions, catalog-freshness handling.

Artifact Contract

  • Path: .forsvn/loops/[slug]/evals/[YYYY-MM-DD]-cycle-N.md (one file per cycle; single platform, single brief — loop-internal artifacts retain the loop tree; flat v2 grammar applies only to docs/forsvn/artifacts/)
  • Lifecycle: evaluation (lives inside the marketing eval-loop workspace; decision_state: not_required)
  • Frontmatter fields (17): skill, type, status, date, stack (=research), review_surface (=none), cycle, loop, post_url, brief_path, catalog_path, catalog_freshness, topic, market, platform, rubric_version, rubric_status, weighting, scores. Cross-stack v2 contract: references/_shared/artifact-contract-template.md.
  • Required sections (7, in order): TL;DR · Observation · Brief vs Observed · Rubric Scores (v0.1, provisional) · Pattern-Log Entry · Open Risks & Caveats · Recommendations for next cycle / catalog
  • Side effects: append row to .forsvn/loops/[slug]/results.tsv via bun scripts/append-loop-result.ts; call bun scripts/manifest-sync.ts after artifact write (both mandatory per procedures/dispatch-mechanics.md)
  • Cross-stack INPUT contract: reads research-shortform catalog at catalog_path — frontmatter (sample_size_per_platform, mechanics_sources_verified, trend_signals_date, platform_mechanics_date drive catalog_freshness), §3 Per-Platform Findings (hook archetypes feed hook-strength-agent), §6 Recommendations (the reference patterns the brief-shortform hypothesis should have followed, feed eval-runner-agent). Schema drift breaks Layer-1 dispatch.
  • Cross-stack OUTPUT contract: consumed by future research-shortform re-runs (mine pattern-log to update catalogs) + gap-gate analysis + operator audit. Schema changes require atomic update of downstream consumers (per anti-patterns.md "Cross-stack contract drift").

Agent Manifest + Dispatch

4 sub-agents: Layer 1 parallel (hook-strength-agent + eval-runner-agent) → Layer 2 sequential (pattern-extractor → critic). Critic FAIL → re-dispatch named agent(s); max 2 cycles; after 2 → ship done_with_concerns. Skipping hook check OR rubric pass produces hollow report. Full per-agent focus + spawn tables + critic routing + post-write side effects + chain position: references/procedures/dispatch-mechanics.md [PROCEDURE]. Per-agent files in agents/{hook-strength,eval-runner,pattern-extractor,critic}-agent.md.

Anti-Patterns

references/anti-patterns.md [ANTI-PATTERN] — 15 failure modes. Re-read before any output ships. Covers premature rubric lock, treating cycle 1 as graded test, fabricated metrics, free-form pattern claims, skipping refutability, author-discretion dominating, missing post-write side effects, cross-stack contract drift.

Worked Example

Short-form hold-rate cycle (retention curve vs platform-intelligence catalog benchmark, critic PASS): references/examples/shortform-eval-cycle-walkthrough.md.

Durable Rules (protected)

Completion Status

  • DONE — all 4 critic rubrics PASS within ≤2 cycles. Pattern-log entry exists in canonical shape. Cycle-1 weighting honored where applicable.
  • DONE_WITH_CONCERNS — critic loop cap reached; remaining failures surfaceable as warnings. Concerns pinned at top of artifact. Also returned when cycle 2 or 3 and rubric not yet revised — flag the mandatory revision.
  • BLOCKED — post URL unfetchable, brief path missing, catalog cannot be resolved. State what's needed.
  • NEEDS_CONTEXT — cold-start abandoned, or matching catalog truly missing for the post's topic+market. Defer to research-shortform.

References

  • references/{playbook, anti-patterns, format-conventions, rubric}.md
  • references/procedures/{pre-dispatch, dispatch-mechanics}.md
  • references/_shared/{before-starting-check, mode-resolver, pre-dispatch-protocol, artifact-contract-template, platform-intelligence/} (canonical at top-level references/, synced)
  • Sibling coordination: brief-shortform (upstream — produces the hypothesis this skill scores), research-shortform (upstream catalog + downstream pattern-log consumer), run-pipeline (owns loop scaffolding + ledger schema)
  • research-skills/CLAUDE.md §"Pre-Dispatch Protocol" + §"Complexity Routing" + §"Multi-Agent Skills" — stack-level conventions inherited