Local PR scoring for regulated engineering teams

Many engineering organizations want automated PR scoring and inline diff walkthroughs, but security policies strictly forbid transmitting proprietary source diffs to external model providers.

CodeOtter was architected from day one so that local models are first-class citizens alongside hosted providers. Every local model is defined in models.json and managed directly from Settings → System → Local models.

Three specialized local runtimes orchestrated by server.mjs

Depending on whether a local checkpoint is a System One rubric engine or a language/comment generator, CodeOtter boots the matching sidecar process on demand:

  1. System One Sidecar (kind: "s1"): Served by the ggmlc laya runtime speaking POST /v1/systemone. Requests are automatically chunked to maxQuestions (8) and diffs are sized to contextChars.
  2. Local GGUF Language Model Sidecar (kind: "llm"): Served by llama.cpp's llama-server for full prose summaries, file-by-file walkthroughs, and structured findings.
  3. Microsoft CodeReviewer Sidecar (api: "codereviewer"): Served by the Python venv runtime (runtimes/codereviewer/serve.py) for specialized per-hunk review comments, paired with a local System One model for scoring.
// core/models.json (excerpt)
{
  "id": "laya",
  "kind": "s1",
  "runtime": "laya",
  "label": "Laya typed-decisions",
  "repo": "mys/laya-typed-decisions-GGUF",
  "file": "laya_typed_decisions_q8_0.gguf",
  "sizeMB": 455,
  "modelId": "typed-decisions",
  "contextTokens": 1024,
  "contextChars": 2500,
  "maxQuestions": 8,
  "license": "Apache-2.0"
}

Runtime install, device choice and multi-file checks

On Linux, macOS, and Windows, the local-models manager downloads the first runtime build listed for your platform that installs: a Vulkan build on Windows x64 and Linux x64 (the language model's llama.cpp runtime also lists a CPU build second) and a macOS arm64 build on Apple silicon. When a model starts, CodeOtter looks for a discrete GPU (NVIDIA, AMD Radeon, or Intel Arc) among the Vulkan devices and runs the model on it; with none found, the runtime picks its own default device. Set S1_DEVICE (System One) or LLM_DEVICE (language model) to cpu to force the CPU. If an archive fails to download, unpack or contain the binary, CodeOtter tries the next one listed; if none installs, the download fails with a Could not install the … runtime error. The runtime builds are Vulkan, Metal and CPU; there is no CUDA build.

A multi-file model such as Microsoft CodeReviewer counts as downloaded only when every file listed in its catalog entry is on disk (modelReady is an existence check, not a hash check). CodeOtter will not start a model that is not fully downloaded; it reports that the model is not downloaded and points to Settings / Local models.