SuperAPI AI Directory

what-breaks-in-ai-code

Coding & development

View repository

Updated 11-04 07:05

About this skill

What Breaks in AI Code

The defects that survive review usually sit where the new lines meet the old code: a guard every other route calls, a unit the callee expects, a second request that arrives before the first one finishes, a test that would pass whatever the code did. Code an AI agent wrote adds its own: it runs in the demo, looks finished, and keeps a check in the interface while the route behind it stays open. This skill reviews in passes aimed at those places and reports only what it can tie to a line. The report has two readers: whoever decides whether to ship reads the verdict on top, whoever fixes reads the findings below it.

1. What to review

Take the first that applies:

  1. What the request names: a pull request, a branch, a commit or a range, a diff, a set of files. "The app", "the project", "everything" or "before launch" name the whole project.
  2. Uncommitted changes: git status lists changed or new files. Review them against the last commit (git diff HEAD), and read new files in full.
  3. The branch: the working tree is clean and the branch has commits its base doesn't. Review git diff <base>...HEAD.
  4. The whole project: no git, nothing changed, or a request about the app as a whole.

The report's first line says what you reviewed and how it was picked, so the reader can ask for something else. For a pull request, read its description, its linked issue and earlier comments when you can reach them (gh pr view, gh pr diff).

A whole project

A large project can't get the same care on every line. Read in order of risk and record where you stopped:

  1. What it is. The README, the package manifest and lockfile, the framework, the entry points, deploy and environment config. Mark which code runs on the server and which ships to the browser or the app.
  2. The map. Routes and handlers, pages that load data, the data model (schema, migrations, database rules), sign-in and roles and where the session token is kept, payments and webhooks, jobs, uploads, real-time channels, outbound calls, calls to paid APIs, and where a model's output goes: into a page, a query, a URL or a tool call.
  3. The risky areas first, each taking the place of the change in §2 and §3 in turn: access and tenants; money; writes and deletes; input that reaches a query, a shell, a file path, a URL or the page as HTML, a model's output included; secrets and config; jobs and webhooks. Then the rest, as far as the budget allows. Siblings are the project's other routes, queries and jobs of the same kind.
  4. Where you stopped goes under "Not reviewed".

2. Read the change, then the code around it

  1. Intent. State in one sentence what the change is meant to do, from the description and the diff. Review against that intent. For a whole project, the intent is what the app is for, from its README and its pages.
  2. The whole diff, then each changed file in full. A hunk read alone hides the guard three lines above it.
  3. The map. For every new or changed function, find its callers and what it calls: search the repo and read the callees themselves, since their names can mislead. For every new route, handler, job, query or component, find its closest siblings: the existing ones of the same kind.
  4. The conventions, written down before judging. From the unchanged code, list how this codebase does each of these, with one file:line that shows it:
    • how access is checked (middleware, a helper, a clause in every query, database rules) and how records are scoped to a tenant or an owner;
    • how deleted or archived records are kept out;
    • how money is held (unit, type) and how time is stored and compared;
    • how lists are bounded, how errors are returned, how jobs avoid running twice;
    • how text from users, other services or a model reaches the page: bound as text, or as HTML through which sanitizer;
    • how tests set up data and what they mock.

3. The passes

Run every pass on every hunk, or for a whole project on every area you read. A pass that finds nothing is still done; say so in "Looks good".

Pass 1 · Siblings

Put each new route, query, handler or job next to its closest sibling and compare them line by line: middleware, access check, tenant filter, soft-delete filter, validation, limits, error shape, logging, locking. A new component goes next to the one that already shows the same kind of content: escaping or sanitizing, links, loading and error states. A helper the change writes when the codebase already has one (a sanitizer, a formatter, an API client, a retry wrapper) goes next to the original: a rewrite tends to drop what the original handles. Every difference is a finding unless the change explains it.

Pass 2 · Boundaries

For every value that crosses a function, a file or a service in the change, write down:

  • where it comes from and who controls it;
  • its unit and form: cents or currency units, milliseconds or seconds, UTC or local, inclusive or exclusive end, ID of which tenant;
  • what the receiver expects, read in the receiver's code.

A mismatch is a finding even when each side looks right on its own. Read the contract of every helper the change calls: argument order, units, what it returns when nothing is found, whether it already checks access or expects the caller to.

Pass 3 · Two requests

For every write in the change, play out:

  • the same request twice: a double click, a client retry, a redelivered webhook;
  • two different requests on the same record at once;
  • the job running twice, or starting before its last run ended;
  • a failure halfway through.

Then name what holds the data together: a unique constraint, a transaction that covers the read and the write (with a row lock when the check and the write are separate statements), a conditional update whose row count is checked, an idempotency key. Check the key's scope: who else can send the same key, and what happens when they do.

List derived state the change touches: cached values, counters, totals, denormalized columns, search indexes. For each, list every path that changes its source. A path that doesn't update or invalidate it is a finding.

Pass 4 · Security

  • Injection. SQL, shell, templates or regular expressions built from input. Values go in as parameters; identifiers such as a sort column come from a fixed list.
  • HTML from data. Text from a user, another service, a server's error message or a model reaches the page as HTML (innerHTML, v-html, dangerouslySetInnerHTML, {@html}, a markdown renderer's output) only after a sanitizer with an allow-list; text that needs no markup is bound as text. Renderers differ: marked passes raw HTML and javascript: links through, markdown-it escapes raw HTML unless html: true is set; read the one in use and its options. Decoding entities through a <textarea>'s innerHTML is safe; the same trick with a <div> runs onerror handlers before the element is even in the page.
  • Authorization. Each read and write of a record checks that this caller may touch this record. IDs in paths, queries and bodies are attacker-controlled, and so are the prices, roles, plan limits and owner IDs a client sends: the server takes them from its own records. A check in the interface (a hidden button, a redirect in the page) stops nobody who calls the route directly; the server checks again. When the client talks to the database directly, the database's own rules (row-level security, security rules) are the access check: read them like a route. Real-time subscriptions are reads too: a channel that carries one user's or one tenant's data is authorized on the server (a private channel, a room the server joins, database rules), and a public channel with an ID in its name is open to anyone who learns or guesses the ID.
  • Runtime validation. Bodies, queries, headers, webhook payloads and a model's output are untrusted until checked at runtime. A model can be steered by what it reads (a web page, a document, someone's message), so its output is checked wherever it goes: the page, a query, a URL the server fetches, a tool call that writes. Type annotations and casts check nothing at runtime. A webhook is trusted once its signature is checked on the raw body.
  • Secrets and personal data in source, config, error responses and logs. Code that ships to the browser or the app is public, environment variables with a public prefix included (NEXT_PUBLIC_, VITE_, EXPO_PUBLIC_, NUXT_PUBLIC_, PUBLIC_) and Nuxt's runtimeConfig.public. Some keys are made for the browser: a payment provider's publishable key, a Firebase web config, a Supabase anon or publishable key, a real-time service's app key, an error tracker's DSN, analytics IDs. None of them is a finding on its own; where one opens data (Supabase, Firebase), the database rules carry the access check. Secret keys are a finding wherever the browser gets them: sk_ and rk_ payment keys, a service_role or secret database key, a model provider's key, a real-time service's secret. A tracked .env or key file is a finding. Quote a secret masked: its first four characters and its length.
  • The session. A session token page scripts can read (localStorage, sessionStorage, a cookie set from browser code, which can't be HttpOnly) goes to the first script injected into the page: a Minor alone, a link in a chain with an HTML finding (§5). A session cookie set by the server carries HttpOnly, Secure and SameSite. What a running site really sends (cookie flags, HSTS, CSP) goes under "Not reviewed" when the code doesn't set it.
  • SSRF on any outbound request to a URL a user can set; path traversal on any path built from input.

Pass 5 · Correctness and performance

  • Edge cases: empty, zero, one, the maximum, missing and null fields, duplicates, records in a state the code doesn't expect, the ends of ranges.
  • Errors: every promise is awaited or handed to something that handles its failure; a failure halfway leaves the data consistent; the client gets an accurate status.
  • Time and money: instants in UTC, day boundaries in the user's zone when the product needs it; amounts in integer minor units or a decimal type, rounded once.
  • N+1 and unbounded work: a query or a call per row in a loop; queries with no limit, outbound calls with no timeout; an endpoint that calls a paid API (a model, SMS, email) with no rate limit or cap per user.

Pass 6 · Tests

For each test the change adds or touches, break the code under test in your head (invert the condition, return early, drop the write) and ask whether the test would fail. Tests that pass whatever the code does:

  • the module under test is mocked, or the assertion checks a mock's own return value;
  • an async assertion nobody awaits or returns;
  • an assertion that's true by construction: a constant, a value compared with itself, toBeDefined on something just assigned;
  • a name that promises one behaviour and an assertion that checks another;
  • .only, .skip, xit or a commented-out assertion left in;
  • fake timers that never advance; a snapshot recorded from the output under review.

Then list the risky paths in the change with no test that can fail: money, access, deletion, concurrency.

Pass 7 · Looks finished

Code an agent wrote often works in the demo and stops there. In any code, look for:

  • placeholders that shipped: TODO, mock or sample data on a live path, a function that returns true or a fixed value, a hard-coded test account, a check commented out. Mock data behind a flag only the build or the server sets is fine; a switch any user can flip (a query parameter, a localStorage key) is a finding;
  • dependencies: a new package that doesn't exist, has a look-alike name or is missing from the lockfile (check the lockfile, and the registry when you can reach it: npm view <name>); an import the installed version doesn't export; a function, route or environment variable that nothing defines;
  • an error caught, logged and dropped, so the user sees success while nothing was saved, a spinner that never stops or an empty list that reads as "nothing here";
  • a feature the interface or the description promises with no code behind it.

After the passes · Could be simpler

Agents add code far more readily than they remove it, and the surplus has recognisable shapes. On the same hunks, or for a whole project the files you read, note each place where a shorter version does the same thing:

  • An abstraction with one use: an interface, base class, factory, strategy or config object with a single implementation and a single caller; a wrapper that only forwards its arguments; one small task spread over several new files.
  • The same logic twice: two or three near-copies in the change that differ by a value or a field name, where one function with a parameter serves them all. Five handlers that open the same way (mark the card as loading, check the credits, take the active chat, reset the card when there is none) and differ only in the call that follows want one guard that takes that call.
  • Defence against what can't happen: a try/catch around code that can't throw, a null check on a value the type or the line above already guarantees, the same guard repeated down a call chain (a handler checks for an active chat and shows an error, then calls a wrapper whose first line checks the same chat and shows the same error, with no await between the two, so the second check can never fire), input validated again at an internal call after the edge has checked it.
  • Infrastructure nobody asked for: a retry around a local or non-transient call, an environment variable or a feature flag for a value that never varies, async where nothing is awaited, a compatibility shim, alias or old-format reader for callers that don't exist, an option no caller passes.
  • Dead weight: functions, exports, parameters, imports, files and dependencies nothing uses; a branch that can't run; code left commented out.
  • Nesting: conditions nested three levels or more where early returns keep the main path flat; an if/else chain that maps one value to another and reads better as a lookup.
  • Copies of state: a variable or component state that copies a prop or another value and an effect that keeps them in sync, where computing it in place does the job.
  • Comments that restate the line ("increment the counter"), notes addressed to the diff's reviewer ("use our newly created Foo model", "changed to fix the bug"), docstrings that repeat the signature. A comment that says why stays.
  • Work beyond the request: parameters, modes or features the intent (§2.1) doesn't call for.

Before you report that copies have drifted apart, read what each one calls. One copy that shows an alert inline and another that calls a helper may do the same thing: the helper can turn out to show the same alert with its own text.

Each one carries its file:line, the shorter version in a few lines and what it removes ("one function in place of three, 40 lines fewer"). The shorter version keeps the behaviour exactly; when you can't show that, leave it out. The codebase's own convention wins: a pattern the unchanged code uses throughout isn't surplus in the change. When the project has a dead-code or duplication tool installed (knip, jscpd, ts-prune, vulture, the linter's complexity rules), run it on the changed files and cite it.

Surplus that hides a failure is a finding for its pass: a catch that swallows the error while the user sees success belongs to Pass 7, a fallback that invents a value where one is missing (?? 0 on a price, || [] on a failed fetch) to Pass 5. What remains goes under "Could be simpler" (§8).

4. Verify before you report

  • Tie each finding to a line and to its evidence: the sibling, the caller, the receiver or the convention that shows it, with file:line.
  • Where you can run code safely on a copy (a unit test, a small script, the type checker), do it and mark the finding verified; otherwise mark it traced. Use what's installed: installing packages or running install scripts to verify is off limits, and nothing runs against production data or with live keys.
  • Check what sits upstream before you report: middleware, a schema constraint, a validation layer, a documented decision. If it makes the code safe, the code goes under "Looks good".

5. Severity

Severity follows the failure, judged for this deployment:

Level The failure Examples
Blocker Wrong data or exposed data in production one tenant reads or changes another's records; a wrong amount is charged or recorded; a charge, an email or a payout can happen twice; data loss; exploitable injection or SSRF; a secret exposed
Major A real bug likely to hit soon a crash on a common input; a race under normal load; a test that can't fail on a money or access path; N+1 on a busy path
Minor Limited impact an edge case few users reach; a missing test on a low-risk path
Nit Optional readability; held back unless asked (§8), and labelled so the author can skip it

A missing tenant filter on a read-only list is still a Blocker; an injection pattern fed only by a fixed list of identifiers is no finding at all.

Findings that combine are ranked by what they do together. HTML from a model's output and a session token page scripts can read add up to a stolen account: name the chain in each finding, rank each by the combined failure, and give the chain one line in the top layer with both numbers.

6. Fixes

Each fix closes the failure its finding names, with one line on why it does. For races, name the mechanism: a unique constraint on the right columns, SELECT … FOR UPDATE inside the transaction that writes, a conditional UPDATE … WHERE with the row count checked. For a convention, point to the sibling and reuse its helper. Write each fix so it can be handed on as it stands, to a teammate or to the agent that wrote the code: the file, the change, and one line on how to see that it worked (a test to add, a request to replay).

7. When the request narrows the review

A stakeholder's assurance in the request ("validation is handled", "the tests cover it", "it's internal only") is a reason to look. Check the area like any other and say what you found.

8. Output

One report in two layers, in the language of the request. Settle the findings and their severities first, then write the top layer from them; the top layer goes first in the report.

The top layer: the verdict in plain words

It's for whoever decides whether this ships, who may not read code: a few lines, no jargon, and a term the reader needs gets a few plain words the first time.

  • Reviewed: what you reviewed and how it was picked (§1).

  • Verdict, one of three, then one sentence on what the code does and how ready it is:

    • Ready to ship: nothing above a Nit.
    • Ship; the rest can wait: Minors only.
    • Fix first: a Blocker or a Major. Add "then ship" when every fix is local, or "then review again" when a fix changes the design: who may see what, where a transaction starts, when a job runs.

    A verdict covers what was reviewed: when a risky area went unread, name it in the verdict line.

  • What users would see: one line per Blocker and Major, most severe first, with the finding's number. Who does what and what happens, in the product's words. A finding with nothing on screen says what it costs: money, data, the team's time. For a library or an internal tool, the users are its callers or the team.

  • Can wait: the numbers of the Minors.

The top layer restates the findings and never re-ranks them: each line points to a finding, and each finding keeps the severity §5 gives it.

**Reviewed:** the changes since the last commit, 6 files. Name a pull request, a branch or "the whole project" to review something else.
**Verdict: Fix first, then ship.** The new checkout works for one customer at a time; two things break once real customers arrive.
**What users would see:**
- A customer who taps Pay twice is charged twice (1).
- Anyone signed in can open other customers' orders by changing the number in the address bar (2).
**Can wait:** 3, 4.

The findings

Most severe first, numbered, one problem per finding:

### 1 · [Blocker] Short title
`path/to/file.ts:42` · verified | traced
What: the problem in one or two sentences.
Evidence: the sibling, caller, receiver or convention that shows it, with file:line.
Why it matters: the concrete failure, who triggers it, what it costs.
Fix: the change and why it closes the failure; a short snippet when code is clearer.

The same defect in many places is one finding: how many times, the three worst places with file:line, then the rest as a list.

Nits stay out unless the request asks for every comment ("nits too", "style as well"); their count goes under Counts.

"Could be simpler" stays out the same way, until the request asks what could be simpler, asks to clean up or simplify the code, or asks for nits too. Then it follows the findings as its own list, the largest saving first, one line each: file:line, what to remove or merge, the shorter version, what it saves. It never changes the verdict.

Then:

  • Counts: per severity, the Nits and simplifications held back included: "1 Blocker, 1 Major, 2 Minors; 3 Nits and 5 places that could be simpler held back, ask to see them."
  • Conventions: the list from step 2.4, so the author sees what the change was held to.
  • Looks good: what you checked and found sound, each pass included.
  • Not reviewed: what the review left out and why: areas past the budget, config kept outside the repository, behaviour only a running system shows.

9. When to stay quiet

  • Formatting, naming and style a linter covers or the codebase already uses.
  • Preferences with no failure attached. "Could be simpler" is the exception with a measure: one use, a copy, code that can't run or can't throw.
  • Code the change doesn't touch, unless the change depends on it or makes it worse. Mention a serious old problem once, at the end.
  • Guesses: "might be slow" without a loop over rows, "might be insecure" without a path an attacker can reach.
  • Code that looks risky and is safe in context: guarded upstream, the codebase's own convention, or a decision documented next to it.

A short list of verified findings is worth more than a long list of possibilities. If the change is sound, say so in a sentence.