The problem with rubber-stamp LGTM reviews
When a pull request touches, say, 15 files and 600 lines of code, human reviewers rarely trace every error path or connection-pool boundary. Conversely, a 10-line change can attract a pile of bikeshedding comments on variable naming.
An AI reviewer that answers in free prose has the same weakness with better grammar. One review says "looks good", the next buries one real bug among style notes, and nothing is comparable from one pull request to the next. CodeOtter puts a typed layer under the prose: every review is scored on the same six 0–100 gauges, and an optional System One model adds yes/no gates, including one for your own AGENTS.md or CLAUDE.md, that each return a pass or a fail.
Two engines, one contract
CodeOtter has two review engines. The language model writes the prose: a summary, a walkthrough of the changed files, and findings. The optional System One model gives typed answers instead. CodeOtter sends it a state payload built from the pull request plus a set of questions, to POST /v1/systemone, and it returns one typed answer per question.
When System One is configured it owns every score and every gate, and the language model is not asked for scores at all. The two engines run in parallel, and either works alone.
Six gauges and nine gates
Every review reports six gauges. Four read higher-is-better. Risk and Blast Radius read the other way, so 100 is the worst answer. With System One, each gauge has a five-level rubric, and these are its two ends:
| Gauge | Key | 0 means | 100 means |
|---|---|---|---|
| Quality | quality | Broken or wrong | Exemplary |
| Risk | correctness_risk | No realistic way to break anything | Very likely to break production |
| Tests | test_coverage | No tests for the change | Thorough coverage |
| Readability | readability | Hard to follow | Very clear |
| PR Hygiene | pr_hygiene | Unclear title and description, mixed concerns | Exemplary title, description and scope |
| Blast Radius | blast_radius | Isolated change with no downstream effect | System-wide or critical-path impact (auth, payments, migrations, deploy) |
The model picks a level from 0 to 4 and CodeOtter maps it to 0, 25, 50, 75 or 100. There is no 67 to argue about: a 3 is a 75, every time. The result is clamped to 0–100, so a malformed index of 9 becomes 100, not 225.
System One also answers nine yes/no gates: title, description, security, complexity, tests, documentation, scope, repository guidelines and issue requirements. Each gate is one plain sentence, such as "The pull request does one focused thing." The model answers with a number and CodeOtter cuts it at 0.5.
Security, complexity and guidelines are phrased as the bad outcome ("The change introduces a security risk…"), so a yes fails them. For the other six, a yes passes. Two gates are conditional: guidelines is asked only when the repository has an AGENTS.md or CLAUDE.md, and issue requirements only when the pull request links issues. Neither is asked when System One runs on a local model.
An answer set looks like this:
{
"answers": {
"score_quality": { "score": 3 },
"gate_security": { "noul": 0.9 }
}
}
The score of 3 is a Quality of 75. The noul of 0.9 on the security gate means yes, the change introduces a security risk, so that gate fails.
Read the gauges together. A small diff with a Blast Radius of 100 deserves more scrutiny than a large one at 25, and a high Risk beside low Tests is a combination worth a second look.
How judge() normalizes untrusted model output before storage
A core engineering rule in CodeOtter is that model output is untrusted. The product's AGENTS.md says judge() normalizes scores, verdict, findings and walkthrough before storage, and that raw model fields are never rendered. In core/server.mjs, judge() handles the language model's reply like this:
- It takes everything between the first
{and the last}as the JSON object, and rejects a reply that does not parse. - When scores are requested, it requires all six, rounds each to an integer and clamps it to 0–100. A missing one fails the review with
returned no usable scores for: …. - It limits the verdict to
approve,commentorrequest_changes, withcommentas the default. - It passes every finding through
deriveSuggestionFromDetail, which maps severity tohigh,medium,lowornit:blockerbecomes high,majormedium,minorlow, and anything unknown becomes low. - It keeps at most 20 findings and 100 walkthrough entries, and caps the stored raw output at 20,000 characters.
- It strips HTML comments from the text it keeps, because they could forge CodeOtter's own comment markers.
Findings get one more filter. With the Self-check setting on (3, 5 or 7, with 5 the default), a second pass at temperature 0 scores each finding from 0 to 10 as a skeptical reviewer, and findings below the threshold are dropped. The review summary says what happened, for example "Self-check kept 3 of 5 finding(s)." If that second pass cannot score every finding, the review fails instead of keeping findings nobody checked.
Normalized results are stored in PocketBase when PB_URL is set (the Docker setup sets it) and in JSON files otherwise.
When an engine is missing or wrong
Typed output only helps if the failure path is honest.
- No System One model.
judge()is asked for the six scores itself and applies the checks above. The nine gates belong to System One, so they do not appear. CodeReviewer, the local comment-only model, cannot score, so it needs a System One model beside it. - No language model. There is no prose, so the review reads "Scored by CodeOtter." with no findings or walkthrough, and the verdict is computed from the typed answers:
request_changesif the security gate fails or Quality is below 40,approveif Quality is at least 70 and every gate passes, andcommentotherwise. - A model that does not answer. If a System One answer is missing, CodeOtter throws
System One model <provider>/<model> did not answer: …and names the missing questions. It never fills the gap with a default.
The last case is a written rule in AGENTS.md: scores, gates and findings come from a model, or the review fails with the model's error on screen and nothing is posted. Heuristics never stand in for a model. A review that fails loudly is a smaller problem than a green gauge built on a guess.
Where to read the code
The rubrics (S1_SCORES, S1_GATES), scoreWithSystemOne and judge() all live in core/server.mjs. CodeOtter is self-hosted and licensed under AGPL-3.0, and you bring your own model, hosted or local. The source is on GitHub.
Related reading:



