SuperAPI AI Directory

pr-quiz

Coding & development

View repository

Updated 11-16 14:44

About this skill

PR quiz

The quiz asks the author of a PR (on GitLab, a merge request) a few questions about their own change. The author ticks an answer in the PR, may read code while answering, and can open the answer key right after. The quiz never blocks a merge, so a question is worth asking only if a reviewer would want the author to have thought it through.

What to ask

  1. Each question probes one dimension:

    • design-intent: why the change does it this way and not another way;
    • impact-surface: which callers, flows or stored data it affects;
    • boundary: empty values, concurrency, ordering, repeated calls, old data;
    • failure-mode: what happens when a step fails, times out or is retried, and how it degrades or rolls back.

    At least half of the questions are design-intent or failure-mode.

  2. The correct answer follows from the code and the diff. Do not ask about intent the code cannot show.

  3. No memory questions: never ask which file, function or variable changed, or anything the author could copy from the diff.

  4. Give 3 to 5 options. Exactly one is correct. Each wrong option is a mistake a hurried author would plausibly make: a missed error branch, a forgotten second caller, two similar concepts mixed up. No absurd options.

  5. Keep the options alike in length and detail. The correct option must not be the longest.

Plain language

The author reads the quiz between other comments on the PR. Write each question the way a teammate asks at your desk, in the language the prompt asks for, and follow the section below for that language. In both languages a question uses at most one code name, in backticks, and only when there is no plain name for the thing.

English

  1. The stem is one or two short sentences, about 25 words at most. It describes a concrete situation, who does what and what happens next, and then asks.
  2. An option states only the result a user or a caller sees, in about 12 words at most. The reason goes in the explanation, never in an option: no "because", "since", "due to", "so that", "therefore", "which causes", "leading to" or "as a result".
  3. Use at most one code name in the whole question, in backticks, and only when there is no plain name for the thing. Never chain names and never invent terms. Call things what users and callers call them: session, settings page, caller, retry.
  4. Avoid these patterns; they make a question read like a spec:
    • "at this point" or "at this moment";
    • a stem that opens with a long "When ...", "While ..." or "Once ..." clause: state the situation in its own short sentence;
    • a long chain of modifiers before a noun: split the sentence in two;
    • three or more items in one sentence;
    • "not X but Y";
    • invented jargon such as "observable behavior", "degrades to" or "falls back to".

Bad, a dense stem full of code names, and an option that gives a reason:

When the mobile NoteEditor flushes the drafts that the user wrote while offline through syncQueue.flush, and the SyncWorker has already started but the lastSyncedAt field of the current notebook has not been written yet, what does the header at the top of the editor render at this point?

  • Falls back to rendering the local draftCache because its isDirty flag starts as true

Good rewrite of the same question:

A draft written offline needs to sync, but this notebook has never synced. What does the user see?

  • The draft, updated after the sync
  • Nothing until the sync finishes
  • A sync error and a retry button

More rewrites:

Bad Good
When the retry budget is exhausted, how does the system handle the request? Three retries in a row failed. What does the user see?
What happens when you rename a session that another window deleted but that is still in the local cache? Another window deleted the session, but this window still shows it. What happens on rename?
What does the service do when the setting is missing, malformed or out of range? The setting is missing from the config. What happens when the service starts?
Not an immediate failure but a degradation to read-only mode It turns read-only and edits stop

The pipeline drops a question whose stem is longer than 45 words, has more than 2 code names, uses "at this point" or "at this moment", or opens with a "When", "While" or "Once" clause longer than 8 words, and a question with an option longer than 20 words, with more than 1 code name, or with a reason in it. It also drops a question that asks which file, function, method, variable, class or line, or what something is called. Stay well inside these limits.

Simplified Chinese

Write the question in Simplified Chinese.

  1. The stem is one or two short sentences, about 60 characters at most. It describes a concrete situation, who does what and what happens next, and then asks.
  2. An option states only the result a user or a caller sees, in about 25 characters at most. The reason goes in the explanation, never in an option: no because, since, therefore, thus or cause.
  3. Use at most one code name in the whole question, in backticks, and only when there is no plain name for the thing. Never chain names and never invent terms. Call things what users and callers call them: session, settings page, caller, retry.
  4. Avoid these patterns; they make a question read like a translated spec:
    • a stem that opens with When... or While..., and at this point...how;
    • a modifier of more than about 15 characters before a noun: split the sentence in two;
    • three or more items joined by enumeration commas;
    • not...but...;
    • invented or translated jargon such as present, receipt, observable behavior, degrade to, fall back to.

Bad, a dense stem full of code names, and an option that gives a reason:

The mobile NoteEditor submits a draft the user wrote while offline via syncQueue.flush. SyncWorker has already started, but the current notebook's lastSyncedAt field has not been written yet. At this point, what does the receipt shown at the top of the editor look like?

  • It falls back to showing the local draftCache, because its isDirty is initially true

Good rewrite of the same question:

A draft written while offline needs to be synced, but this notebook has never been synced before. What will the user see at this point?

  • Show the draft first, then update after syncing
  • Don't show it yet; show it after syncing
  • Show a sync failure prompt and ask the user to retry

More rewrites:

Bad Good
When the retry count is exhausted, how will the system handle the request? Three consecutive retries all failed. What will the user see?
What happens when you rename a session that has been deleted by another window but still remains in the local cache? The session was deleted in another window, but this window still shows it. What happens if you rename it at this point?
What does the service do when a configuration item is missing, malformed, or out of range? This item is missing from the configuration. What happens after the service starts?
Not fail immediately, but degrade to read-only mode It first becomes read-only and can no longer be changed

The pipeline drops a question whose stem is longer than 100 characters, has more than 2 code names, uses at this point or opens with When..., and a question with an option longer than 45 characters, with more than 1 code name, or with a reason in it. Stay well inside these limits.

Fields

  • dimension: one of the four dimensions above.
  • prompt: the stem.
  • options: the option texts without letters. The pipeline adds the letters and shuffles the order.
  • answer_index: the zero-based index of the correct option in options.
  • explanation: why the correct option is right and the main wrong ones are wrong, in two or three plain sentences. It is shown after the author answers. Refer to options by their content, never by letter or position, because the order changes.
  • evidence: up to three changed places that support the answer, each a path and a new-side line.