How CodeOtter Runs Five Open-Weight Models Locally: A Walk Through `models.json`, Three Runtimes, and the `Admin → Local Models` Flow

Why we chose 5 specific open-weight models, not "whatever's on Hugging Face"

"Run a local model" is easy to say and brutal in production. The model has to (a) be small enough to load on a developer laptop without a 24 GB GPU, (b) be fast enough that a CI step finishes inside its timeout, (c) produce structured output (typed rubrics, not chatty prose), and (d) not phone home. The roster in models.json is the answer we shipped after testing 11 candidates in late September:

Two are not in the list and we are honest about why:

• Nous Hermes / Yi / DeepSeek-Coder — strong general code models, but their chat-shaped output does not collapse cleanly into typed rubric answers without a wrapper prompt that doubled our latency budget.
• Code Llama 7B / 13B / 34B — great at code completion, mediocre at judging a diff. We benchmarked Code Llama 7B Instruct on 20 PRs and it clustered every System 1 rubric between 78 and 84, the same problem we built System 1 to avoid.

The 5-model roster is small on purpose: every model in models.json is a tool, not a checkbox. We have written test harnesses for each (in models.test.mjs inside the repo) and a dev workflow that benchmarks new candidates against the existing 5 before they're added.

The three runtimes and what each one is

models.json declares a runtimes block that pins exact versions of every sidecar binary. Pinning matters because the runtimes are not just model-hosting wrappers — they are the local HTTP servers CodeOtter's server.mjs talks to on 127.0.0.1. The block looks like this (paraphrased from /workspace/codeotter/models.json):

{
  "runtimes": {
    "laya":   { "version": "0.9.7",  "source": "https://github.com/monatis/ggmlc/releases/download/v0.9.7/",
                "license": "MIT",     "bin": "laya",          "assets": { …vulkan/Metal per OS… } },
    "llama":  { "version": "b11215", "source": "https://github.com/ggml-org/llama.cpp/releases/download/b11215/",
                "license": "MIT",     "bin": "llama-server",  "assets": { …vulkan/Metal/CPU per OS… } },
    "codereviewer": { "version": "1", "license": "Apache-2.0", "kind": "python",
                       "script": "runtimes/codereviewer/serve.py",
                       "pip":   ["torch", "transformers>=4.40", "sentencepiece", "protobuf"],
                       "pipIndex": "https://download.pytorch.org/whl/cpu" }
  }
}

The three roles:

server.mjs reads models.json at boot, builds the per-runtime asset list for the host OS, and resolves the sidecar command. The 15-minute idle shutdown is critical for laptop users: a developer who opens a PR, walks away for lunch, and comes back gets a fast first review because the runtime is still warm. A developer who closes the lid overnight gets a cold start at 5 minutes on the first review of the morning.

The Laya 0.9.7 bump is worth a note: 0.9.5 stopped with Illegal instruction on CPUs without AVX-512 (most laptops, including a lot of CI runners). 0.9.7 fixed the codepath; the bump is in CHANGELOG.md under "Unreleased."

The model roster as a models.json row

Every checkpoint is one typed object. The shape — taken from the actual models.json for the Qwen2.5-Coder 7B row — is what an admin sees in Admin → Local models:

{
  "id": "qwen-coder-7b",
  "kind": "llm",
  "runtime": "llama",
  "label": "Qwen2.5-Coder 7B",
  "maker": "Alibaba Qwen",
  "repo": "Qwen/Qwen2.5-Coder-7B-Instruct-GGUF",
  "icon": "https://cdn-avatars.huggingface.co/v1/production/uploads/6215ca5692c0ecfba9186921/hrRM50-6XcdWgg2AKpENG.jpeg",
  "file": "qwen2.5-coder-7b-instruct-q4_k_m.gguf",
  "sizeMB": 4683,
  "modelId": "qwen2.5-coder-7b-instruct",
  "contextTokens": 16384,
  "contextChars": 60000,
  "license": "Apache-2.0",
  "notes": "Best local review quality, Q4_K_M. Wants a GPU with 6 GB or 8 GB of RAM on CPU."
}

The fields that matter for an admin's mental model:

• id — internal slug. local_qwen-coder-7b is the provider id in the dropdown.
• kind — "s1" for System 1, "llm" for language. Drives which sidecar runtime is selected.
• runtime — which of the 3 runtimes hosts the model.
• file (single-file GGUF) or files + dir (multi-file checkpoint like CodeReviewer's 8 files) — what to download.
• sizeMB — pre-flight display in the UI. Admins see "4.7 GB" before they click.
• contextTokens / contextChars — what the chunker trims the diff to. The 5-model roster's contextChars range is 2,500 → 90,000; small-context System 1 is the floor.
• maxQuestions — for kind: "s1", how many typed questions to send per POST /v1/systemone request. Laya and Kev both use 8 (the local-runtime soft limit). Hosted System 1 calls the platform's /v1/systemone with the full question set in one call.
• license — surfaced in the admin UI. Apache-2.0 across the board; no GPL, no commercial-only.

The row that drives the most admin questions is CodeReviewer:

{
  "id": "codereviewer",
  "kind": "llm",
  "runtime": "codereviewer",
  "api": "codereviewer",
  "label": "CodeReviewer (Microsoft)",
  "files": ["config.json","generation_config.json","added_tokens.json","merges.txt",
            "special_tokens_map.json","tokenizer_config.json","vocab.json","pytorch_model.bin"],
  "dir": "microsoft__codereviewer",
  "sizeMB": 895,
  "contextTokens": 512,
  "contextChars": 90000,
  "license": "Apache-2.0",
  "notes": "CodeT5-based, pre-trained on code review data (arXiv:2203.09095). Writes a comment per hunk, no summary or scores: needs a System One model alongside. CPU is fine."
}

The 8-file files array is what makes modelReady(m) non-trivial. CodeReviewer is a CodeT5 checkpoint (8 sharded files + a tokenizer), not a single-file GGUF. The readiness check has to verify every shard.

The modelReady lifecycle: how a row becomes "ready"

The readiness check is the most important 6 lines of server.mjs. Single-file GGUF and multi-file checkpoints use the same shape:

const modelReady = (m) =>
  Array.isArray(m.files)
    ? m.files.every((f) => fs.existsSync(path.join(modelPath(m), f)))
    : fs.existsSync(modelPath(m));

That is the whole predicate. For a single-file GGUF, it checks one path. For a multi-file checkpoint, it checks every shard under PR_SCORER_DATA/<dir>/<file>. The admin UI uses this same function to render the row's button: a row is Download when !modelReady(m), Use when modelReady(m) && selected !== m.id, and Selected when modelReady(m) && selected === m.id.

modelPath(m) resolves to either PR_SCORER_DATA/<file> for single-file GGUF or PR_SCORER_DATA/<dir> for multi-file. The download path is symmetrical:

if (!m.files) return downloadFile(`https://huggingface.co/${m.repo}/resolve/main/${m.file}`, modelPath(m), m.id);
// else for each f in m.files:
await downloadFile(`https://huggingface.co/${m.repo}/resolve/main/${f}`, dest, sub);

Every download is a plain GET to huggingface.co. No auth, no token, no parallel fetcher. The download progress is reported back to the admin UI on a per-file basis. Nothing downloads on startup — the platform explicitly does not auto-fetch. The download is an explicit admin action from the Admin → Local models page.

The sidecar lifecycle is then gated on modelReady:

if (rd?.active && modelReady(m)) return { state: "downloading", text: rd.label || "Installing runtime" };

A model is "usable" only when the runtime asset for the host OS is downloaded AND the model checkpoint is modelReady. If the runtime asset is missing, the admin gets a runtime-download row first; if the model file is missing, they get a model-download row.

The PR_SCORER_DATA storage layout

Everything local lives under one directory. The default is the platform's working directory; on Docker it is the volume mounted at /app/pb_data (or the sidecar data path that the entrypoint exports). The layout on disk:

$PR_SCORER_DATA/
  laya_typed_decisions_q8_0.gguf          # 455 MB — Laya System 1
  kev_0.8b_q8_0.gguf                       # 828 MB — Kev 0.8B System 1
  qwen2.5-coder-1.5b-instruct-q4_k_m.gguf # 1.1 GB — Qwen local LLM
  qwen2.5-coder-7b-instruct-q4_k_m.gguf    # 4.7 GB — Qwen local LLM (optional)
  microsoft__codereviewer/                 # 895 MB — CodeReviewer (multi-file)
    config.json
    generation_config.json
    added_tokens.json
    merges.txt
    special_tokens_map.json
    tokenizer_config.json
    vocab.json
    pytorch_model.bin
  runtimes/                                # Sidecar binaries
    laya-0.9.7-linux-x86_64-vulkan         # per-OS asset
    llama-server-b11215-linux-x86_64-vulkan
    codereviewer-venv/                     # Python venv (private)

A few decisions to call out:

• GGUF, not safetensors. Every language-model checkpoint is the GGUF build, served by llama.cpp. We do not download full-precision safetensors and re-quantize at install time — the Q4_K_M and Q8_0 quants are already in the Hugging Face repos, so a Q4_K_M 7B is 4.7 GB, not 14 GB. For an air-gapped box with a tight disk budget, that delta is the difference between "fits on a 1 TB workstation" and "needs a NAS."
• The runtime assets are downloaded once per host OS. When you upgrade models.json (say, to a new llama.cpp build), server.mjs diffs the runtime version against the on-disk copy and re-downloads only the changed asset. An admin does not re-pull the GGUF when a sidecar bumps.
• PR_SCORER_DATA persists across Watchtower upgrades. On Docker, mount it as a named volume (codeotter-data:/app/pb_data) or a bind mount (./data:/app/pb_data). Watchtower pulls the new image and restarts the container; the model and runtime assets are untouched. The first review on the new image pays a 5-minute cold start because the runtime warm cache is in-memory only, but the downloads are not repeated.
• No telemetry, no analytics, no opt-in banner. The platform does not phone home to report which models you have downloaded. models.json has no analyticsId field. The admin UI does not load any third-party scripts. A network capture during a review shows traffic only to 127.0.0.1 (sidecar) and huggingface.co (admin-time download).

Discrete GPU detection: Vulkan, Metal, CPU

A CI runner is a Linux x86_64 VM. A MacBook is darwin-arm64. A Windows workstation is win32-x64. A Raspberry Pi is linux-arm64. Each gets a different binary.

The platform's server.mjs selects the OS-specific asset by detecting process.platform and process.arch, then resolving the asset list under runtimes.<id>.assets[<os>]. An admin does not pick the binary — the platform picks for them based on the host. If the Vulkan binary crashes on load (e.g. an old nvidia driver), the admin can delete it from PR_SCORER_DATA/runtimes/ and the platform re-fetches the CPU build on the next review.

The "no AVX-512" gotcha is worth repeating: if you run CodeOtter on a host without AVX-512 (most laptops, a lot of CI sandboxes), Laya 0.9.5 will crash with Illegal instruction on first review. 0.9.7 fixes it; an admin on a stale 0.9.5 should delete the runtime asset and re-download. The CHANGELOG entry is one line: "Laya runtime 0.9.7. The 0.9.5 Linux build stopped with 'Illegal instruction' on CPUs without AVX-512 (Docker on most laptops)."

Small-context System 1: maxQuestions: 8 and the diff digest

A chat LLM has a 100k–200k context window. A local 0.8B System 1 model has 1,024–2,048 tokens. A 50-file PR with 4,000 lines of diff does not fit. Two things have to give.

First, the question chunking. Laya and Kev both declare maxQuestions: 8 in models.json. server.mjs splits a POST /v1/systemone call into batches of 8 typed questions and runs them sequentially. The 8-question batch is the soft limit where the local ggmlc laya runtime stops being latency-friendly — beyond 8, the time per question degrades non-linearly.

Second, the diff digest. For a small-context System 1 model, CodeOtter does not send the raw diff. It sends a structured digest sized to the model's contextChars (Laya: 2,500 chars; Kev: 5,500). The digest is built by server.mjs like this (paraphrased from the actual implementation):

function buildSmallContextDigest(files, ctx) {
  const header = files.map(f => `${f.path} +${f.additions} -${f.deletions}`).join("\n");
  const tests  = files.filter(f => /\.(test|spec)\./.test(f.path));
  const hunks  = files.flatMap(f => f.hunks).sort((a,b) => b.lines - a.lines).slice(0, 4);
  return [
    `Files: ${files.length} (tests: ${tests.length})`,
    `Hotspots: ${hotspotsFromFiles(files).join(", ")}`,
    `Per-file diff sizes:\n${header}`,
    `Largest hunks:\n${hunks.map(h => `--- ${h.path} @@ ${h.header}\n${h.body}`).join("\n\n")}`
  ].join("\n\n").slice(0, ctx);
}

The digest is a lossy summary: file paths + line counts, a test count, hotspot directory names, and the 4 largest hunks in full. It is what makes small-context System 1 useful at all on a 50-file PR. The trade-off is real: the System 1 model is judging the diff at a higher level of abstraction than the language model is. That is acceptable for gates (the rubric questions are coarse-grained — "does the title match?", "is security preserved?") but not acceptable for findings (which need line-anchored critique from the language model).

The honest disclosure in onboarding: CodeOtter shows an info icon next to Laya and Kev in Admin → Local models and in Settings → Model provider, and the message is "on larger pull requests they see only part of the change." Onboarding now preselects Kev 0.8B for new admins because the larger context window makes the digest loss less painful.

This is also why the 8-questions-per-call cap matters even on a model that could accept them. A 30-question rubric call with a small-context model would force the chunker to send 4 sequential calls; a 60-question rubric call would force 8. The total wall-clock is roughly proportional to the number of chunks, and small-context System 1 is not a low-latency model to begin with — it's ~80 seconds on a triage call with Kev. Chunks multiply that budget linearly.

CodeReviewer is not standalone

Microsoft CodeReviewer is a CodeT5 model pre-trained on the arXiv:2203.09095 review-comment dataset. It writes one review comment per diff hunk. It does not produce numeric rubric scores and it does not produce an executive summary. It is a hunk annotator, not a reviewer.

CodeOtter treats it as such:

1. It must be paired with a System 1 model. A review that selects local_codereviewer as the language engine and no System 1 model is refused at review time. The bin/codeotter.mjs CLI's runReview checks for the System 1 provider and surfaces the error to the user. The platform UI shows the same check in Settings → Model provider.
2. System 1 re-validates every CodeReviewer comment. Each CodeReviewer candidate comment is sent to System 1 with the hunk, and only comments System 1 judges (with at least 0.7 confidence) to name a concrete problem become findings. The default maxFindings cap is 25. The summary on the review page states how many were kept and how many were dropped.
3. It uses 512 tokens of context. The contextTokens: 512 field in models.json is the upper bound for a hunk the model will annotate. Hunks larger than 512 tokens are chunked along the diff boundary; hunks smaller than 512 tokens get the full hunk in the prompt.
4. It runs entirely on CPU. The Python venv installs the cpu-indexed PyTorch wheel (https://download.pytorch.org/whl/cpu) so the install footprint stays at ~250 MB and a GPU is not required. An air-gapped box can use it.

The CHANGELOG entry that landed with v0.4.0 is precise about the contract:

"A System One model that fails to start, errors or skips a question now fails the review with the model's error on the review page, and nothing is posted to the PR. A language model that owns the scores must return all of them, including blast radius, or the review fails."

CodeReviewer does not own the scores, so it cannot run alone. A review with local_codereviewer and local_laya works. A review with local_codereviewer and no System 1 fails closed.

End-to-end: what happens when an admin clicks "Download Qwen2.5-Coder 7B"

This is the literal sequence an admin walks through in Admin → Local models. The steps are the same for any of the 5 models.

1. Open Admin → Local models / Language models. The page lists the 4 language rows in models.json (Qwen 1.5B, Qwen 7B, CodeReviewer) — though CodeReviewer is filed under a "CodeReviewer" sub-section because of the Python venv install. The admin sees the 5-model roster with size, license, and a per-row Download / Use / Selected button.
2. Click Download on Qwen2.5-Coder 7B. The platform checks modelReady(qwen-coder-7b) — it returns false because the GGUF is not on disk. The platform starts a downloadFile('https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/main/qwen2.5-coder-7b-instruct-q4_k_m.gguf', <PR_SCORER_DATA>/qwen2.5-coder-7b-instruct-q4_k_m.gguf, 'qwen-coder-7b') call. Progress is reported to the UI on a 1-second poll. A 4.7 GB download on a 100 Mbps link takes ~6 minutes.
3. At the same time, the runtime asset is verified. If PR_SCORER_DATA/runtimes/llama-server-b11215-linux-x86_64-vulkan is missing, the platform starts a parallel download from https://github.com/ggml-org/llama.cpp/releases/download/b11215/llama-b11215-bin-ubuntu-vulkan-x64.zip. The runtime is ~70 MB and finishes in seconds.
4. The runtime asset is unpacked into PR_SCORER_DATA/runtimes/llama-server-b11215-linux-x86_64-vulkan. The platform then sets the OS permissions (chmod +x) and verifies the binary launches with --help (this is the smoke test — if Vulkan is missing on the host, the Vulkan binary will fail the smoke test, and the platform will fall back to the CPU build).
5. The model file lands. modelReady(qwen-coder-7b) now returns true. The row's button changes from Download to Use.
6. The admin clicks Use. The platform writes selected = "qwen-coder-7b" to the org's model-provider settings. The next review call sends a POST /api/review with languageModel: "local_qwen-coder-7b".
7. First review on the new model spins up the sidecar. server.mjs shells out to llama-server -m <PR_SCORER_DATA>/qwen2.5-coder-7b-instruct-q4_k_m.gguf --port <auto> --host 127.0.0.1. The runtime loads the GGUF into RAM (5 minutes on a CPU laptop, ~30 seconds on a CUDA-capable GPU). The review POST hits the sidecar, returns scores + walkthrough + findings, and server.mjs writes the review row.
8. 15 minutes of idle and the sidecar shuts down. The next review pays a ~5-second warm-up cost, not a 5-minute cold start.

The 6-minute download is the longest single admin-time cost. After that, every review is local. No outbound traffic during a review except to 127.0.0.1.

What this changes for a team

For the platform engineer wiring CodeOtter into a self-hosted infra, the contract is one volume mount, two env vars, and a list of model ids to install at admin time. docker-compose.yml ships with codeotter-data:/app/pb_data already wired. The env vars are GH_TOKEN and ANTHROPIC_API_KEY (or none, if you intend to run only local models). There is no SaaS webhook, no hosted control plane, no per-seat tax.

For the security lead writing the audit story, the network story is boring on purpose. During a full PR review the only outbound traffic is the huggingface.co fetch that happened at admin time, possibly still cached. A network capture during a review shows 127.0.0.1 → llama-server and 127.0.0.1 → laya. That is the entire surface.

For the AI coding agent (Claude Code, Cursor, Windsurf, etc.) running locally, bin/codeotter.mjs runs the same dual-engine against the org's selected local model from the platform's settings; no separate config.

For the CFO comparing this to CodeRabbit or Greptile, the dollar math is the same as the rest of P2: $0 forever on self-hosted infra vs $24–$30/dev/month on SaaS. The download time is a one-time cost; the per-review cost is electricity.

Where this still loses, honestly

• First-review cold start is 5 minutes on a CPU laptop. A Qwen 7B GGUF takes 4–5 minutes to load into RAM on CPU. The first review of the morning is the slow one. Subsequent reviews are 30–60 seconds. A GPU with 6 GB cuts the cold start to 30 seconds. Mitigation: pair with a hosted API key for the first review of the day, or warm the runtime on a long-lived runner via pnpm start.
• Small-context System 1 is approximate. Laya sees a 2,500-char digest; Kev sees 5,500. A 50-file PR with 4,000 lines of diff cannot fit. System 1 still answers the 8 typed rubric questions on the digest, but the answers are about the summary of the change, not the full diff. Mitigation: pair with a hosted System 1 model (via OpenRouter) for large PRs, or run two reviews — one with local System 1 (fast), one with hosted System 1 (precise) — and compare.
• CodeReviewer is 512 tokens of hunk context. A 1,000-line diff hunk gets chunked. The chunking is along diff boundaries (the @@ markers), so the model sees whole logical units, but a single chunk may be too narrow to understand the cross-file implication. Mitigation: use Qwen 7B as the language engine for the executive walkthrough, and use CodeReviewer as a second language pass for the per-hunk comments. The anti-hallucination critique pass on the CLI and the platform review will deduplicate.
• linux-arm64 is CPU-only. The llama.cpp Vulkan build does not target linux-arm64. A Raspberry-Pi 5 install runs Qwen 1.5B on CPU at ~3 tokens/sec — usable for the System 1 digest, painful for the language walkthrough. Mitigation: run System 1 only on a Pi (Laya is small enough to be fast), and use a hosted language engine.
• No automatic model upgrades. When models.json bumps a runtime version (Laya 0.9.7 replacing 0.9.5), the platform re-downloads the runtime on the next review but does not re-download the model checkpoint. That is correct: a new laya version does not invalidate a laya_typed_decisions_q8_0.gguf. But a new Qwen 3 (if/when it lands in models.json) requires a fresh ~4.7 GB download. The admin gets a notification badge in Admin → Local models; the model is not pulled automatically.
• AGPL-3.0 is OSI-approved copyleft. The license is GNU Affero General Public License v3.0 (AGPL-3.0). Free to self-host, modify, and redistribute; a modified version offered as a network service must publish its source. The models in models.json are all Apache-2.0 (or MIT for the runtimes), but the platform itself is AGPL-3.0. The audit reads the license file, not the badge.

Where to go next

• Walk the dual-engine architecture in the flagship deep-dive — System 1 + System 2 rubric contract and the 9-gate verdict in §3.1.
• The 9th gate (Issue Requirements, gate_issue_requirements) is enforced server-side via POST /v1/systemone with full forge context — see the AGENTS.md-as-merge-gate blog.
• The incremental delta review (prevSha → headSha) is documented in the incremental-delta engineering blog.
• The CI integration (codeotter ci --fail-on-gate) is in the CLI-in-CI blog — the same 8 CLI-enforced gates and the 9th server-side gate.
• The model roster source of truth is dharmeshgurnani/CodeOtter/models.json — 5 rows, 3 runtimes, exact pinned versions.
• The runtime download paths are the GitHub Releases pages for ggml-org/llama.cpp and monatis/ggmlc.
• The CodeReviewer sidecar is at dharmeshgurnani/CodeOtter/runtimes/codereviewer/serve.py and the arXiv:2203.09095 paper that pre-trained it.

If you want a self-hosted AI code reviewer on your own hardware, vendor dharmeshgurnani/CodeOtter (or run the Docker image), open Admin → Local models, click Download on Qwen 7B (or Laya, or Kev — your call), and the next PR review runs entirely on 127.0.0.1. No bytes to the cloud. $0 per seat.

CodeOtter ships under the GNU Affero General Public License v3.0 (AGPL-3.0) — free to self-host, modify, and redistribute; a modified version offered as a network service must publish its source. Zero per-seat billing, zero telemetry. The 5 open-weight models in models.json are all Apache-2.0; the 3 runtimes (laya, llama-server, runtimes/codereviewer/serve.py) are MIT, MIT, and Apache-2.0. You own the binary; you own the disk; the network only sees 127.0.0.1 and huggingface.co.

Reposted from the CodeOtter engineering blog under AGPL-3.0.