SuperAPI AI Directory

signal-first-plan

Coding & development

View repository

Updated 10-26 17:48

About this skill

Signal-first plan

Before you build, decide how you will see that the feature works and how you will see that it broke. A feature breaks at its exits: the places where its data or behaviour leaves the process. Name every exit, put one signal on each, and make every signal checkable before you write the feature.

A small change can have one exit. Then the plan is one row.

Step 1: List the exits

An exit is any place where the feature's data or behaviour leaves the process:

  • responses: rendered pages, page data, API bodies, redirects, status codes
  • messages: emails, webhooks, notifications, feeds, sitemaps, badges, embeds
  • outbound calls: HTTP, DNS, queues, third-party APIs
  • writes: database rows, files, caches, logs

Trace exits from the data, not from the files you plan to change. Start at each table, column, flag or config value the feature reads or changes. Follow every query, controller, route and job that reads it, out to the place where it leaves the process. A reader you did not plan to touch is still an exit.

Done when every reader of the touched data is in the list as an exit, or is marked internal with a reason.

Step 2: Put one signal on each exit

A signal is a number that a broken version changes: DNS lookups per request, rows written per job, payment API calls per checkout, emails sent per signup.

  • Write the expected value for every case the feature splits on: each type of viewer, each input class, each config value.
  • On a response exit, count over the whole body the client receives (the serialized page data or JSON), not over the one field you expect to carry the data. A leak sits in the field nobody planned for.
  • A number that the working and the broken version share is noise. Pick another.
  • Aggregates such as p99 and error rate are weak, because many causes move them. Prefer a count tied to one exit.
  • Read .claude/debug-layers.md at the project root, when it exists, for signals that past bugs needed. Each one that touches an exit of this feature becomes a check in Step 3.

Done when every exit has a signal and an expected value for every case.

Step 3: Make every signal checkable

  • Prefer a test that reaches the exit the way a client does and asserts the signal. One test can loop over many exits and cases.
  • When only production can show the signal (latency, call counts at real scale), emit it where the stack already logs or measures. A later debug session then finds it at layer 1.
  • Emit each signal with the fields that let a later question be asked without a new deploy: the request or job id, the input class, the exit, the count. An aggregate alone answers only the question you planned.
  • Run the checks on main. A check for new behaviour fails there. A check for behaviour the feature must keep passes there. If a keep check fails on main, the bug already exists: report it to the user before you build on top of it.

Done when every signal has a test or an emitted metric, and you know which checks pass on main.

Show the user the plan as a table, then build: exit, signal, expected value per case, check (test name or metric), result on main.

Step 4: Compare before merge

  1. After you build, run every check on main and on the branch.
  2. List each signal that changed. Every change must be one the plan expected. Fix any other change before merge.
  3. For a signal that only production or pre-production can show, compare the canary population against the control population at the same time, broken down by population, when the rollout gives you one. A before-and-after comparison of one population against its previous successful run is the fallback; time of day and load move it, so say so in the PR. Check the control against its own baseline too, because a bad canary can hurt the control it is compared with. Never an absolute threshold alone, and not only pass or fail. This applies to every signal in the plan, including the ones taken from .claude/debug-layers.md. The copyconstruct post's DNS lookup count differed from the previous successful run even at pre-production scale, so this check would have caught that bug before production.
  4. Before you commit, list every file in the diff. Generated files, such as test screenshots and attachments, stay out unless the plan named them.
  5. In the PR, give the exit table and the comparison.