The COLM 2026 review form, explained

How COLM reviewers read your paper

COLM asks reviewers to weigh five dimensions without mechanistic measures of quality: Empiricism, Data and Evaluation; Technological Impact; Ambition, Vision, Forward-outlook; Understanding Depth, Principled Approach; and Clarity, Honesty and Trust. Weakness on one dimension is not an automatic reject, especially for ambitious and honest work, while strength on one does not compensate for being very weak on others. Small-scale work should not be penalized for lacking large compute. Reviews of accepted papers become public.

• Do not apply mechanistic quality measures (e.g. "no SOTA", "too few benchmarks") as automatic rejection reasons.
• Do not penalize small-scale or academic-compute work for lacking large-scale experiments.
• Credit honesty about limitations; do not treat an honest limitations section as a weakness.
• Respectful tone; reviews of accepted papers are published.

Where the bar sits. COLM accepted 28.9% in 2024. Accepted papers averaged about 6.4–6.5 on the 1–10 rating in 2024 and 2025 (Paper Copilot). Rough rule: mean ≥ 6 competitive, 5.25–6 borderline.

The reviewer form, field by field

In the order reviewers see it, with every score and its official wording.

1. 01

   Summary

   Free text

   What the paper does and claims, in your own words.

2. 02

   Reasons To Accept

   List of points

   Strengths across COLM's dimensions: Empiricism, Data and Evaluation; Technological Impact; Ambition, Vision, Forward-outlook; Understanding Depth, Principled Approach; Clarity, Honesty and Trust.

3. 03

   Reasons To Reject

   List of points

   Weaknesses across the same dimensions. Weakness on one dimension is not by itself a reason to reject, especially for ambitious work honest about its limits; but strength on one dimension does not offset being very weak on others. Do not penalize small-scale work for lacking large compute.

4. 04

   Questions To Authors

   List of points

   Questions for the rebuttal whose answers could change your assessment.

5. 05

   Rating

   Score · 1–10

   Overall rating.

   Score	Meaning
   10	Top 5% of accepted papers, seminal paper
   9	Top 15% of accepted papers, strong accept
   8	Top 50% of accepted papers, clear accept
   7	Good paper, accept
   6	Marginally above acceptance threshold
   5	Marginally below acceptance threshold
   4	Ok but not good enough - rejection
   3	Clear rejection
   2	Strong rejection
   1	Trivial or wrong

6. 06

   Confidence

   Score · 1–5

   How certain you are about your evaluation.

   Score	Meaning
   5	The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature
   4	The reviewer is confident but not absolutely certain that the evaluation is correct
   3	The reviewer is fairly confident that the evaluation is correct
   2	The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper
   1	The reviewer's evaluation is an educated guess

7. 07

   Ethics Concerns

   Free text

   "None", or the ethical concern that should be flagged for ethics review.

Area chair meta-review

COLM ACs make the initial decision, may dismiss clearly low-quality reviews, and discuss borderline cases with the PCs.

• Accept
• Reject

COLM author statements

Reviewers check these. Gaps are easy points to lose.

1. Optional ethics statement (up to one page, outside the page limit)
2. Optional reproducibility statement (up to one page)
3. LLM use that originated ideas, wrote content (incl. references), generated data or plots, or did evaluation disclosed
4. Anonymized: no acknowledgments or identifying links

Policy on LLMs in reviewing