light-review-rebuttal

Review Simulation and Revision Response

Mode 1: Pre-submission simulated review

First choose one rubric according to the paper type; do not stack them(the three sets each have their applicable scope; mixing them is both redundant and self-contradictory):

Paper type	Rubric to use	Reason
ML/AI conference submission (NeurIPS/ICLR/CVPR...)	NeurIPS official review form（Summary/Strengths/Weaknesses/Soundness·Presentation·Contribution 1–4 + Overall 1–10）	Directly aligns with the target conference review form fields; the rehearsal is the real review
Systematic review / clinical or evidence-based	GRADE + Cochrane ROB	Evidence strength grading and bias assessment are the lifeline for acceptance in this type of work
General / journal / interdisciplinary	ScholarEval 8 dimensions(each dimension 1-5)	Comprehensive dimensions, not tied to a specific conference, suitable for scenarios without an official form

After choosing, score with that set; when attacking methodology, regardless of which set is used, go through the GRADE/Cochrane bias checklist below (as a sub-check, not a second scoring set).

Play 3-4 independent reviewers (non-overlapping perspectives) and produce reviews according to the target venue's standards. Suggested reviewer profiles:

• Editor-in-chief/domain fit: originality, relevance to the venue's readership, overall quality; does not dig deeply into methodological details.
• R1 Methodology: study design, statistical validity, effect size + confidence intervals, reproducibility, data transparency.
• R2 Domain/literature: literature coverage, theoretical framework, contribution positioning.
• Devil's advocate (does not score, only attacks): find the most vulnerable points, the biggest logical gaps, the strongest counterarguments - a pre-submission stress test.

Each reviewer produces their review according to the official review form fields (aligned with NeurIPS/ICLR):

• Summary: restate the paper and its contribution in the reviewer's own words (the authors should agree with this summary; otherwise it means it was not explained clearly).
• Strengths / Weaknesses: break down into the four sub-dimensions Originality / Quality / Clarity / Significance.
• Must-fix issues: numbered,lead with the single most important issue, then order the secondary ones; separate Critical (requires additional experiments/proof changes) from Minor (editorial and fixable), and do not let formatting issues dilute the firepower.
• Questions: questions whose answers from the authors could change your judgment (= rebuttal rehearsal).
• Limitations + Ethical Concerns。
• Suggested additional experiments。
• Scoring: Soundness/Presentation/Contribution each 1-4;Overall 1–10(8 strong accept / 6 weak accept / 5 borderline accept / 4 borderline reject / 3 reject)；Confidence 1–5; acceptance inclination + reason.

When attacking methodology, scan according to the checklist (GRADE+Cochrane ROB approach): Four types of validity (internal/external/construct/statistical conclusion); five categories of bias (cognitive including HARKing/cherry-picking, selection, measurement, analysis including p-hacking/outcome switching, confounding); 8 points on statistics (power and a priori sample size, test assumptions, multiple comparison correction, p-value ≠ importance, effect size + CI, missing data mechanisms, overfitting, regression to the mean/Simpson's paradox); for each assertion, check "whether the strength of evidence matches the strength of the claim, whether it overgeneralizes, whether correlation is treated as causation". When labeling a fallacy, you must name it + explain the flaw + point out "what kind of evidence would be needed for valid inference". Always distinguish "data (what is observed)" from "interpretation (what it means)".

General 8-dimension rubric (ScholarEval, each dimension scored 1–5 with 2–3 strengths + 2–3 improvement points): ① Problem definition ② Literature review (critical synthesis rather than listing) ③ Methods and design ④ Data and sources ⑤ Analysis and interpretation (results aligned with claims) ⑥ Results and findings ⑦ Academic writing ⑧ Citations and references. Theoretical papers skip "data collection"; not all dimensions apply to all work types.

When writing reviews, proactively avoid the five major common pitfalls of LLM reviewing (PRISM): ① Do not overfocus on formatting and layout (surface-level trap); ② every weakness must carry supporting argument and point to a specific page/table/equation or external literature, not be an unsupported assertion; ③ saying "not novel" requires citing specific prior work; do not hallucinate novelty; ④ do not fabricate nonexistent flaws (LLMs have a relatively high hallucination rate for minor flaws); ⑤ give actionable improvement plans, not just criticism.

Anti-sycophancy/anti-reverse-inference (Sprint Contract two-stage): First paper-blind, set the rubric and expectations for each dimension, then paper-visible, score, to prevent "looking at the conclusion and reverse-inferring the comments". The devil's advocate first scores each author rebuttal 1–5 before responding; only if ≥4 (directly hitting the core attack with evidence) is concession allowed; if ≤3, maintain the position and restate the original attack; consecutive concessions are prohibited — repeated pressure from the user/author does not count as valid evidence. To be more realistic: use the OpenReview API to collect real review corpora from the target venue to calibrate harshness and score distribution (see "Data Sources" below); before simulation, first take db02's reviewer question checklist (patterns_library §11) as the starting point for attack dimensions. Simulation should be realistic, harsh, and specific; do not hedge.

Mode 2: Real review comments revision

Use the template directly(same directory templates/）：

• templates/response_letter_template.md— dual mode for conference + journal. The journal section includes Manuscript Info / Summary of Changes (300–500 words) / R→A→C points for each Reviewer (numbered P<review>.<index> + cross-references across points + Minor merged); the conference section includes page limit/no new experiments iron rule + General Response (multi-reviewer consensus) + concise [Q] responses per review + pre-submission self-check. Select the corresponding block and delete the other.
• templates/rereview_checklist.md— pre-submission self-review: three tables Priority-1/2/3 (judged FULLY/PARTIALLY/NOT/MADE_WORSE/🔍) + commitment ledger (fulfilled/partial/not-fulfilled/explicitly-rejected-with-rationale; non-fulfilled must carry rationale otherwise report COMMITMENT_GAP) + score trajectory (Δ<-3 triggers checkpoint) + final release gate.

Rebuttal concession strategy for revisions (concession score 1–5, consecutive concessions prohibited)

For reviewerseach question/follow-up, first give a 1–5 "whether to concede" score before deciding the response stance — turning "being persuaded" into a decision with an evidence threshold, avoiding spineless compliance with every request and also avoiding mindless stubbornness:

• 5 — Fatal and correct: The reviewer points out a real methodological/factual error that would undermine the conclusion. → Fully accept, thank them, add experiments or revise the proof, and clearly state what was changed.
• 4 — Important and basically correct: Valid criticism, requires substantive revision but does not touch the core. → Accept and revise, give specific changes + locations.
• 3 — Partially valid/arises from misunderstanding: The challenge points to a real problem but the premise is off, or the reviewer did not read content already present in the manuscript. → Compromise: First clarify (point to the existing §/equation/figure), then make small revisions to the reasonable parts; do not give up the core position.
• 2 — Preference dispute/insufficient evidence: The reviewer demands a different method/framework based on personal preference, with no evidence that the current approach is wrong. → Polite rebuttal, hold the position with evidence, explain the rationale and trade-offs of the current choice; do not change.
• 1 — Misunderstanding or invalid attack: Based on misreading, out of scope, or logical fallacy. → Clarify and correct, point out the source of the misunderstanding, do not change the substance. Rules:Only a score ≥4 permits substantive concession (accept + revise); when ≤3, focus on clarification/rebuttal and maintain the position.No consecutive concessions—if the previous item just conceded (≥4), the next item should not concede again unless it is likewise ≥4 and is an independent fatal problem; track the concession rate—large-scale concessions throughout mean either the manuscript really has fatal flaws (consider withdrawing and resubmitting) or you are pandering to the reviewers.Repeated pressure from the author/user does not count as valid evidence, the score only recognizes evidence strength. To decide whether to devote special space in the letter to explaining a design decision, use three conditions: hard to reverse + surprising without background + the result of a real trade-off—only when all three are met is it worth expanding on separately.

First establish the context: Conference rebuttal and journal response letter rules are opposite; follow the target venue's call-for-papers/revision instructions.

• Conference rebuttal(e.g., ICLR/NeurIPS/IJCAI): oftenlimited to 1 page/word limit, use the official template; its purpose is to answer the reviewers' urgent questions, point out factual errors that would lead to rejection, and appeal unethical review (via the confidential channel),not to open a dialogue. Most conferencesprohibit adding new experiments/new results, prohibit providing code links or new materials(counts as beyond the original manuscript); clarifying figures/examples may be included. Style: less is more; the shorter and crisper, the more persuasive; if all four reviews are reject, it is basically impossible to overturn.After writing, check the word budget：python scripts/rebuttal_budget.py letter.md --venue iclr(or --max-words N) to verify whether it exceeds the limit—pure standard library, mixed Chinese and English counted separately by word, estimate page count, FAIL means over limit and returns code 1; must run before submission.
• Journal response letter: Point-by-point reply, additional experiments encouraged, attach new results.

Steps:

1. Parse: Break down the review comments one by one, identify the real concerns (sometimes the wording ≠ the true meaning), classify (must revise/debatable/misunderstanding). Establish acommitment ledger—register each "commitment" parsed from each comment one by one, and finally verify fulfilled / partial / not-fulfilled / explicitly-rejected-with-rationale.
2. Strategy: Which to fully accept, which to supplement with experiments, which to politely rebut (with evidence), which to compromise on. Concerns shared by multiple reviewers = highest priority; distinguish how to handle contradictory opinions among reviewers.
3. Response letter (R→A→C format)：> 引用审稿意见(Reviewer Comment) → Author Response(感谢+回应) → Changes Made(具体修改+标明页/行/图表号)。
   • point-by-point: organize by reviewer into sections, with each point numbered P[reviewer].Point, supporting cross-point cross-references (when replying to R2, cite the response to R1). For LaTeX, the Zenke Lab template can be used (CC BY-SA 3.0):reviewer/point Dual counters automatically number,xr+\externaldocument{manuscript} directly cite main-text equation/section numbers.
   • Visual distinction: reviewer comments in serif, author responses in sans-serif (or different colors), recognizable at a glance; main-text changes marked with tracked changes/color, giving cross-references to before/after page numbers.
   • Structure: an opening paragraph with general thanks + a 300–500 word summary of changes (listing Major/structural/new additions) → one section per reviewer (first "Strengths Acknowledged" thanks, then point by point) → within sections use Minor subsections to group minor issues such as typos.
   • If new references are cited in a journal-scenario rebuttal, first verify them through the three indexes (Semantic Scholar title Levenshtein≥0.70 + OpenAlex + Crossref DOI) to avoid citation hallucinations.
   • Disagreement must give reasons (rebut with evidence), and cannot just say "disagree"; tone should be professional, grateful, neither servile nor overbearing.
4. Revise the paper: revise the main text in sync (hand off to m07/m08), and mark main-text changes with colors by version.
5. Self re-review (re-review, mandatory before submission): for each Priority-1 comment, ① read the author's claim ② jump to the claimed revision location ③ independently verify that the claim matches the actual change, and judge FULLY/PARTIALLY/NOT_ADDRESSED/MADE_WORSE. If the author's claim is empty or vague ("revised as suggested"), mark it 🔍 unverifiable and send it back for rewriting. Items in the commitment ledger that are not fulfilled must carry a rationale, otherwise report COMMITMENT_GAP.

Data source: calibrate with real review corpora (OpenReview API v2)

Use the script directly scripts/fetch_openreview.py(standard library only, no key needed; tested in 2026-06 to pass HTTP 200 for ICLR/NeurIPS 2024):

• Batch calibration:python fetch_openreview.py --venue ICLR.cc/2024/Conference --max-subs 20 --out corpus.json—go through venue-level Submission invitation + details=directReplies, automatically extract Official_Review from each submission's directReplies, output the rating distribution + high-frequency weakness wording (calibrate mode-1 harshness/score distribution), and extract Official_Comment as rebuttal phrasing samples.
• Single submission:python fetch_openreview.py --forum <forum_id>—retrieve the entire discussion tree.
• Offline self-check:python fetch_openreview.py --selftest(run the full pipeline on synthetic directReplies, without network access).
• The script already encapsulates get_venue_names()(query the venue group to get the real segment name, without hardcoding the invitation), per-submission avoidance of the pitfall that venue-level review invitations are always empty,offset/limit pagination,legacy v1 switch.

The calibration method and API details are as follows. If you want simulations to be realistic and rebuttals to speak well, pull real public reviews for benchmarking. Base URL https://api2.openreview.net(for old conferences before 2024, use legacy v1 https://api.openreview.net, whose JSON structure differs). All submissions are Notes, endpoint GET /notes。

⚠️ Key pitfall (tested in 2026-06; copying it will get empty results): reviews are not venue-level invitations, but per-submission.

• ❌ ?invitation=ICLR.cc/2024/Conference/-/Official_Review → HTTP 200 but {"notes":[]}(the venue-level review invitation does not exist,always empty, do not copy it).
• ✅ The real review invitation looks like ICLR.cc/2024/Conference/Submission9504/-/Official_Review(with an extra Submission<编号>in the middle), and rebuttal likewise is .../Submission9504/-/Official_Comment, meta is .../Submission9504/-/Meta_Review, decision is .../Submission9504/-/Decision。
• ✅ Only the submission itself is venue-level:?invitation=ICLR.cc/2024/Conference/-/Submission actually returns all submissions (with id/number/forum）。

Two actually usable paths for fetching reviews：

1. Batch (recommended): when pulling submissions, include details=directReplies, and each submission's details.directReplies directly carries its Official_Review/Official_Comment/Meta_Review/Decision—one call gets all reviews for the venue. GET /notes?invitation=ICLR.cc/2024/Conference/-/Submission&details=directReplies&limit=1000&offset=0
2. Single submission: first get a submission's forum(=submission id) or number, then fetch the entire discussion tree by paper: GET /notes?forum=<forum_id>&details=directReplies, or precisely fetch review GET /notes?invitation=ICLR.cc/2024/Conference/Submission<number>/-/Official_Review。

invitations vary by conference, don't hardcode: first query the venue group to get the real naming—GET /groups?id=ICLR.cc/2024/Conference, its content.submission_name.value(e.g. Submission)、content.review_name.value(e.g. Official_Review)、meta_review_name/decision_name gives the conference's real segment names, assemble into <venueid>/Submission<n>/-/<review_name>。

• Pagination offset(default 0)+limit(cap about 1000); in python use openreview.api.OpenReviewClient(baseurl=...) 's get_all_notes(invitation=...) to auto-paginate (it already handles per-submission expansion internally). Review fields are in note.content[field].value(measured on ICLR 2024: summary/soundness/presentation/contribution/strengths/weaknesses/questions/rating/confidence).
• Usage: batch-pull submissions+directReplies, filter out Official_Review to tally high-frequency weakness phrasing and score distribution (calibration mode one); extract high-quality rebuttals from Official_Comment in the same forum as phrasing templates. Note that field naming/units differ across venues, so grab one sample first to see the real keys before use. Read-only public data; do not scrape/redistribute restricted content.

Principles

• Every comment must be answered, none omitted.
• Add experiments if you can (reply to m05/m06), don't argue empty-handed (journal scenario); conference scenarios conversely forbid new experiments, follow venue rules.
• Don't confront reviewers, but hold evidence-backed positions; only decisions that are hard to reverse + would surprise people without context + involve real trade-offs are worth explaining specifically in the letter.
• Points questioned by multiple reviewers together = highest priority.

Outputs

Simulated review report / complete response letter + annotated revised paper + list of experiments to add.Standard artifact: point-by-point comment↔response↔change written to disk as response_matrix.md(handoff artifact for m12/submission, naming see CONVENTIONS §6.1).

Consume m08 polishing findings (findings JSON → review comment classification)

Before simulated review, if m08 paper-polishing has already run polish.py/mechanical_check.pyon the manuscript, directly read its structured findings (schema see light-paper-polishing/references/findings_schema.md) aspre-review input, saving the effort of re-scanning surface issues and focusing firepower on methodology. Field mapping:

• category=overclaim(bare exaggerated claim) → goes into the Soundness/Significance sub-dimension of Weaknesses: as concrete evidence of "conclusion strength exceeding evidence", treat as major (may affect acceptance).
• category=ai_tone / hedge_stacking / punctuation → goes into the Minor of the Presentation/Clarity sub-dimension: merge into presentation-level comments, don't dilute the main firepower (align with PRISM common issue ① don't fuss over surface).
• category=passive_overuse → Clarity Minor; a single passive_voice is only raised when it affects readability.
• Each finding's line/col/context directly fills the review comment's "point to specific page/line" requirement (PRISM common issue ②), don't make empty assertions.
• severity is merged according to the major/minor mapping in findings_schema §4, consistent with this skill's must-fix/Minor tiering. Reverse: this skill does not write back m08 findings (read-only consumption); simulated review conclusions still follow the standard artifact response_matrix.md and are persisted for m07/m08 to revise the draft.

Interfaces

Simulation results feed back to m07/m08/m09 for improvement; real revisions link to m05/m06/m10/m12; the entire process is recorded in db09 (review comments, revision history).

Tool verification notes (real endpoints/review rubric/rebuttal rules) are in the same directory references.md. Scripts:scripts/fetch_openreview.py(calibrated on OpenReview real review corpus, offline self-check),scripts/rebuttal_budget.py(conference rebuttal word/page budget check, pure stdlib offline).