The problem with rubber-stamp LGTM reviews

When a pull request touches, say, 15 files and 600 lines of code, human reviewers rarely trace every error path or connection-pool boundary. Conversely, a 10-line change can attract a pile of bikeshedding comments on variable naming.

An AI reviewer that answers in free prose has the same weakness with better grammar. One review says "looks good", the next buries one real bug among style notes, and nothing is comparable from one pull request to the next. CodeOtter puts a typed layer under the prose: every review is scored on the same six 0–100 gauges, and an optional System One model adds yes/no gates, including one for your own AGENTS.md or CLAUDE.md, that each return a pass or a fail.

Two engines, one contract

CodeOtter has two review engines. The language model writes the prose: a summary, a walkthrough of the changed files, and findings. The optional System One model gives typed answers instead. CodeOtter sends it a state payload built from the pull request plus a set of questions, to POST /v1/systemone, and it returns one typed answer per question.

When System One is configured it owns every score and every gate, and the language model is not asked for scores at all. The two engines run in parallel, and either works alone.

Six gauges and nine gates

Every review reports six gauges. Four read higher-is-better. Risk and Blast Radius read the other way, so 100 is the worst answer. With System One, each gauge has a five-level rubric, and these are its two ends:

GaugeKey0 means100 means
QualityqualityBroken or wrongExemplary
Riskcorrectness_riskNo realistic way to break anythingVery likely to break production
Teststest_coverageNo tests for the changeThorough coverage
ReadabilityreadabilityHard to followVery clear
PR Hygienepr_hygieneUnclear title and description, mixed concernsExemplary title, description and scope
Blast Radiusblast_radiusIsolated change with no downstream effectSystem-wide or critical-path impact (auth, payments, migrations, deploy)

The model picks a level from 0 to 4 and CodeOtter maps it to 0, 25, 50, 75 or 100. There is no 67 to argue about: a 3 is a 75, every time. The result is clamped to 0–100, so a malformed index of 9 becomes 100, not 225.

System One also answers nine yes/no gates: title, description, security, complexity, tests, documentation, scope, repository guidelines and issue requirements. Each gate is one plain sentence, such as "The pull request does one focused thing." The model answers with a number and CodeOtter cuts it at 0.5.

Security, complexity and guidelines are phrased as the bad outcome ("The change introduces a security risk…"), so a yes fails them. For the other six, a yes passes. Two gates are conditional: guidelines is asked only when the repository has an AGENTS.md or CLAUDE.md, and issue requirements only when the pull request links issues. Neither is asked when System One runs on a local model.

An answer set looks like this:

{
  "answers": {
    "score_quality": { "score": 3 },
    "gate_security": { "noul": 0.9 }
  }
}

The score of 3 is a Quality of 75. The noul of 0.9 on the security gate means yes, the change introduces a security risk, so that gate fails.

Read the gauges together. A small diff with a Blast Radius of 100 deserves more scrutiny than a large one at 25, and a high Risk beside low Tests is a combination worth a second look.

How judge() normalizes untrusted model output before storage

A core engineering rule in CodeOtter is that model output is untrusted. The product's AGENTS.md says judge() normalizes scores, verdict, findings and walkthrough before storage, and that raw model fields are never rendered. In core/server.mjs, judge() handles the language model's reply like this:

  • It takes everything between the first { and the last } as the JSON object, and rejects a reply that does not parse.
  • When scores are requested, it requires all six, rounds each to an integer and clamps it to 0–100. A missing one fails the review with returned no usable scores for: ….
  • It limits the verdict to approve, comment or request_changes, with comment as the default.
  • It passes every finding through deriveSuggestionFromDetail, which maps severity to high, medium, low or nit: blocker becomes high, major medium, minor low, and anything unknown becomes low.
  • It keeps at most 20 findings and 100 walkthrough entries, and caps the stored raw output at 20,000 characters.
  • It strips HTML comments from the text it keeps, because they could forge CodeOtter's own comment markers.

Findings get one more filter. With the Self-check setting on (3, 5 or 7, with 5 the default), a second pass at temperature 0 scores each finding from 0 to 10 as a skeptical reviewer, and findings below the threshold are dropped. The review summary says what happened, for example "Self-check kept 3 of 5 finding(s)." If that second pass cannot score every finding, the review fails instead of keeping findings nobody checked.

Normalized results are stored in PocketBase when PB_URL is set (the Docker setup sets it) and in JSON files otherwise.

When an engine is missing or wrong

Typed output only helps if the failure path is honest.

  • No System One model. judge() is asked for the six scores itself and applies the checks above. The nine gates belong to System One, so they do not appear. CodeReviewer, the local comment-only model, cannot score, so it needs a System One model beside it.
  • No language model. There is no prose, so the review reads "Scored by CodeOtter." with no findings or walkthrough, and the verdict is computed from the typed answers: request_changes if the security gate fails or Quality is below 40, approve if Quality is at least 70 and every gate passes, and comment otherwise.
  • A model that does not answer. If a System One answer is missing, CodeOtter throws System One model <provider>/<model> did not answer: … and names the missing questions. It never fills the gap with a default.

The last case is a written rule in AGENTS.md: scores, gates and findings come from a model, or the review fails with the model's error on screen and nothing is posted. Heuristics never stand in for a model. A review that fails loudly is a smaller problem than a green gauge built on a guess.

Where to read the code

The rubrics (S1_SCORES, S1_GATES), scoreWithSystemOne and judge() all live in core/server.mjs. CodeOtter is self-hosted and licensed under AGPL-3.0, and you bring your own model, hosted or local. The source is on GitHub.

Related reading: