Engineering Blog · Code Review

From 73 to 80 on Every Re-Review: How CodeOtter Walks Only the New Commits

A deep-dive into the `prevSha → headSha` incremental delta review path, the typed-rubric `state` payload, and the in-place `` comment upsert that keeps your PR conversation readable.

Last Updated: October 5, 2026·7 min read

TL;DR

Re-reviewing a PR should not re-charge the meter on every push. CodeOtter tracks the last reviewed commit (`prevSha`), scores only the new `prevSha..headSha` delta, and edits its own GitHub comments in place.

The 6 calibrated gauges and 9 pre-merge gates are still emitted on the *new* PR state — there is no separate `delta` question type. What changes between re-runs is the `state` payload: `head_sha`, `git_change_history`, and an `incremental_review_delta` summary.

The `` and `` HTML markers survive re-runs. We PATCH the existing GitHub comment via `PATCH /repos/:owner/:repo/issues/comments/:id` instead of posting a duplicate, so a 10-push PR stays a 10-line scorecard thread.

A real reproducible walkthrough lives on the `codeotter-website` repo's PR #1 — comment [issuecomment-5933514003](https://github.com/dharmeshgurnani/codeotter-website/pull/1#issuecomment-5933514003) shows the upsert and the `Resolved since prevSha` row in place.

From 73 to 80 on Every Re-Review: How CodeOtter Walks Only the New Commits — *CodeOtter — Incremental Delta Reviews + In-Place PR Comment Upsert (Pillar P4)*

The problem with re-reviewing a PR ten times

You opened a PR. The reviewer pushed back because you forgot a migration. You pushed the fix.

In a world of LLM-only reviewers, "re-review" means: another full pass over the entire PR. Every push re-spins the model, re-clusters the 0-100 score somewhere in the 75-85 band, and posts another comment on the PR thread. Ten pushes becomes ten near-identical comments, and the audit story — "what was the score when this landed?" — disappears into a scrollback of teaser sentences.

We bet that is the wrong shape for a review tool. So CodeOtter splits re-review into two narrower problems: re-score only the delta, and edit your own PR comments in place instead of duplicating them. This post walks through the wire-level mechanics and shows one real PR as a case study.

What `prevSha` actually means

A quick first-read for teammates who have not yet read the integration docs.

prevSha is the commitSha of the PR's tip at the end of the last CodeOtter review run, persisted in the review record alongside the scorecard and the GitHub comment IDs. On the next run we compute a standard git diff prevSha..headSha — a vanilla two-dot diff against the new tip — and feed that delta to System 1 (under the incremental_review_delta field) and System 2 (as the new diff hunk set). Anything in the old review that no longer maps to a line in the new diff is reported as Resolved since prevSha. Anything new is reported fresh. The PR base and merge state are unchanged — this is a review-engine concern, not a git-history concern. The endpoints involved are POST /v1/systemone for the rubric engine, POST /api/review-suggestions for inline committable fixes, and PATCH /repos/:owner/:repo/issues/comments/:id for the GitHub-side upsert.

If you only remember one thing: CodeOtter does not have a separate "delta review" route. The delta logic lives in state assembly and in the upsert marker. The contract is the same POST /v1/systemone you already know from the dual-engine deep-dive.

The flow, end-to-end

View Mermaid source

flowchart LR
  Push["git push<br/>(headRefOid advances)"] --> Hook["GitHub webhook<br/>→ CodeOtter /review"]
  Hook --> Lookup["Load review record<br/>prevSha · comment IDs<br/>last known gauges/gates"]
  Lookup -->|prevSha == headSha| NoOp["No-op upsert<br/>(HEAD: same as last review)"]
  Lookup -->|prevSha != headSha| Diff["git diff prevSha..headSha<br/>(Smart Context slicing,<br/>AGENTS.md, learnings)"]
  Diff --> S1["System 1 — POST /v1/systemone<br/>6 gauges + 9 gates<br/>on the delta state"]
  Diff --> S2["System 2 — streaming prose<br/>+ line-anchored suggestion blocks"]
  S1 --> Merge["Merge with resolved-finding map<br/>✅ Resolved since prevSha"]
  S2 --> Merge
  Merge --> Upsert["PATCH /repos/:o/:r/issues/comments/:id<br/>(upsert scorecard + review)"]
  Upsert --> PR["GitHub PR thread<br/>(single comment, edited in place)"]

The diff handed to both engines is only the new commits, not the full PR. That is where the latency and token savings come from on every iteration after the first.

Before/after on a real PR

The first PR on the codeotter-website repo — dharmeshgurnani/codeotter-website#1 — is a typical walkthrough case. Two pushes, around 40 lines of Astro + JSON-LD changes. Push #1 left a Web Vitals regression that System 1 caught via its Tests gauge. Push #2 fixed it. The screenshot of the upserted  block on push #2 — i.e. the same comment, edited in place — is at issuecomment-5933514003. The ✅ Resolved since prevSha marker on the Tests row is the part most readers miss: System 1 is not telling you the gauge moved from 50 to 75, it is telling you the prior missing-test finding is gone. Both are useful; they mean different things.

Reproduce it yourself: the platform configuration that produced that comment was a hosted System 1 model (kev-0.8b against the System 1 API), local Qwen2.5-Coder 7B for System 2 via llama-server, and the three repo toggles — Post scores on PR, Add PR review as a comment, Post inline suggestions — all enabled. The comment ID and the  HTML marker survive re-runs, so the upsert is idempotent: re-running against the same headSha is a no-op.

How the rubric engine handles a delta

System 1's typed rubric does not introduce a delta question type, and that is deliberate. The 6 gauges — Quality, Blast Radius, Risk, Tests, Readability, PR Hygiene — and the 9 binary gates — Title, Description, Security, Complexity, Tests, Docs, Scope, Guidelines, Issue Requirements — are still answered on the new PR state. What changes between re-runs is the state object CodeOtter hands to POST /v1/systemone:

pull_request.head_sha is the new tip.
git_change_history is prevSha..headSha, truncated to the model's contextChars.
incremental_review_delta carries a precomputed summary string, including ✅ Resolved since prevSha: <finding-id> rows for any prior finding whose line anchor no longer maps to a line in the new diff.

The mapping index × 25 still holds: if the Quality rubric lands at index 3 on the new state, the gauge is 75/100. The reason the number reads differently on push #2 is that the state is genuinely different — new diff, new tests, new commit history — not that System 1 invented a delta mode.

Local runtimes that cap a request at 8 questions per HTTP call (the maxQuestions chunker) emit the same partials stream as a fresh review: gauges light up as they resolve, gates resolve last. The dashboard's Re-run scores and Re-run narrative buttons stay independent, and a delta re-run hits the same endpoint as a fresh one.

Edge cases: force-push, rebase, squash

Two PR histories look the same on GitHub but behave differently on a delta re-run, and CodeOtter distinguishes them deterministically.

Fast-forward append — git commit --amend, git rebase with replay. prevSha is reachable from headSha via first-parent history. We run the standard prevSha..headSha diff. Findings that no longer map to a line are reported as resolved; new lines are reported fresh.
Non-fast-forward force-push — prevSha is no longer reachable from headSha's history. We fall back to a full diff against the PR base and flag the dashboard row with non-incremental rebase — full re-review so reviewers know the new scorecard is not directly comparable to the prior one.

Squash-merge to main clears prevSha for the next PR cleanly — there is no cross-PR carryover, so the next PR always starts on a fresh review record.

What this changes for a team

For the engineer pushing the fix, no more waiting for a full re-review on every push. The delta is what gets re-scored, and the comment thread stays a single comment.

For the reviewer triaging the PR, the scorecard carries ✅ Resolved since prevSha rows instead of going silent on prior findings, so the audit trail is one comment thread, not five.

For the platform engineer wiring this in, the contract does not change. POST /v1/systemone is the only rubric endpoint. POST /api/review-suggestions is the only suggestion endpoint. The delta logic lives entirely in state assembly and in the upsert marker.

Cross-links

The dual-engine architecture that produces these gauges and gates — including the state payload and the maxQuestions chunker — is in the flagship engineering post: Why We Built a Dual-Engine Architecture (System One + LLM) (src/content/blog/dual-engine-architecture-for-the-ai-era.mdoc in the site repo).
The in-place upsert behavior is documented inline below — the  and  markers, the PATCH /repos/:owner/:repo/issues/comments/:id route, and the idempotency contract, all cited against the CodeRabbit / Greptile / Copilot / PR-Agent baseline.

CodeOtter ships under the GNU Affero General Public License v3.0 (AGPL-3.0) — free to self-host, modify, and redistribute; a modified version offered as a network service must publish its source. Zero per-seat billing, zero telemetry. Contract: POST /v1/systemone. Upsert marker: . Diff: prevSha..headSha. That is the entire story.

Reposted from the CodeOtter engineering blog under AGPL-3.0. Canonical URL above.

Share this article

Share on X