Why single-LLM PR reviewers fail as CI/CD merge gates
Most AI code review bots ask a single chat model to do two incompatible jobs in one prompt: write conversational feedback for the author AND emit numeric scores or pass/fail verdicts for branch protection.
When a single generative prompt is responsible for both prose and numerical evaluation, temperature variance, prompt length, and diff formatting cause the same pull request to score 84/100 on one run and 68/100 on the next. Engineering teams quickly disable blocking gates when scores cannot be trusted.
// Traditional single-LLM failure mode: scores coupled to prose generation // Run #1 -> Quality: 85, Tests: 78 (PASS) // Run #2 -> Quality: 72, Tests: 61 (BLOCKED - identical diff!)
How CodeOtter’s Dual-Engine pipeline splits typed rubrics from prose
In CodeOtter, when a System One model is configured (either hosted via S1_PROVIDERS or locally via the ggmlc laya runtime), it owns every score in S1_SCORES and every boolean gate in S1_GATES. The language model is not asked for numeric scores at all.
Both engines execute concurrently against the pull request diff and repository guidelines (AGENTS.md and CLAUDE.md). Because the local laya runtime processes typed questions in deterministic chunks (at most maxQuestions per request: 8 for the local System One models, 2 when partial answers are streamed), rubric evaluation runs without hallucinated JSON schemas.
POST /v1/systemone
{
"model": "<System One model>",
"state": "<normalized PR diff + AGENTS.md + CLAUDE.md context>",
"questions": {
"score_blast_radius": {
"type": "score",
"instructions": "Rate the pull request's blast radius.",
"criteria": [
"Isolated change with no downstream effect",
"Small local impact",
"Moderate impact on nearby modules",
"Wide impact across several areas",
"System-wide or critical-path impact (auth, payments, migrations, deploy)"
]
},
"gate_security": {
"type": "noul",
"instructions": "The change introduces a security risk (injection, auth bypass, secrets, unsafe deserialisation, SSRF, XSS)."
}
}
}
The six 0–100 gauges and binary merge gates
Every pull request evaluated by CodeOtter produces a standardized, organization-scoped scorecard stored in PocketBase:
• Quality (0–100): Rates the overall quality of the change, on a rubric that runs from "Broken or wrong" to "Exemplary". • Blast Radius (0–100): Rates how far the change reaches, on a rubric that runs from "Isolated change with no downstream effect" to "System-wide or critical-path impact (auth, payments, migrations, deploy)". • Risk (0–100): Rates how likely the change is to break production, on a rubric that runs from "No realistic way to break anything" to "Very likely to break production". • Tests (0–100): Rates how well tests cover the change, on a rubric that runs from "No tests for the change" to "Thorough coverage". • Readability (0–100): Rates how readable the change is, on a rubric that runs from "Hard to follow" to "Very clear". • PR Hygiene (0–100): Rates the pull request's title, description and scope, on a rubric that runs from "Unclear title and description, mixed concerns" to "Exemplary title, description and scope".
Four of the six gauges run higher-is-better: Quality, Tests, Readability and PR Hygiene reach their best rubric level at 100. Risk and Blast Radius run inverted — they measure how bad the change is, so 100 is their worst level, with Risk reaching "Very likely to break production" and Blast Radius reaching "System-wide or critical-path impact". A Risk score of 85 is therefore a bad result, not a good one.
Security is not one of the six gauges. It is a binary merge gate: a yes/no question asking whether the change introduces a security risk (injection, auth bypass, secrets, unsafe deserialisation, SSRF, XSS). A yes fails the gate: it shows as a Warning row in the pull request comment, and codeotter ci exits with code 1 when any gate fails, so a pipeline can stop on it. Injection vectors and auth-boundary concerns are therefore a pass/fail check rather than a score.
Enforcing untrusted model normalization with judge()
CodeOtter treats language-model output as untrusted input. The server-side judge() parser clamps each score to [0, 100], strips HTML comments from the summary and the file walkthrough (so a model cannot forge CodeOtter's own comment markers), caps findings at 20 and the walkthrough at 100 entries, and falls back to a "comment" verdict when the model returns anything other than approve, comment or request_changes.


