autonomous-ai / run-local-model

Run a local model — details

The flow itself is in AGENTS.md. Tags: [run] seen on a real machine (M1 Pro 32 GB, macOS 26.6,
2026-09-29), [doc] official docs (see the engine-* skill), [code] read in the tool's source.

Purpose → settings (the only thing asked)

Coding · Chat and writing · Reading images · Just something fast. Context is never below 64K:
every model here is used by an agent whose own prompt fills a small window; Ollama's docs set the same
floor for agents [doc]. When 64K does not fit, take a smaller model, never a smaller window.

Thinking is off for everyday use.

Reading fleet models

• machine.accelerators[]: a GPU counts only with active: true (its own tool answered). active: false
  comes with an error (e.g. no driver): say it in one line; plan no GPU engine on it.

• machine.engines[]: installed engines, running or not. machine.canRun: the kinds this hardware can
  run at all — never propose one outside it.

• machine.memory.availableBytes, swapUsedBytes: room right now. Metal and device-info do not see
  other apps (Metal said 25 GiB free while macOS was 11 GB into swap [run]).

• machine.accelerators[].totalBytes: the GPU ceiling, below total RAM on a Mac — read it here, never assume it; device-info reports a different figure
  there [run] — use the smaller.

• engines[]: answering engines and their exact --at URL. openai-compatible whose "models" are not
  models is another app: leave it alone.

• models[]: one entry per real file (alsoAt = other apps holding the same file), with format,
  bytes, projector, and for GGUF contextLength, kvBytesPerToken, toolCalls, unsupportedTensorTypes.

Size

need = weights + context × kvBytesPerToken × slots + 0.5 GB (+ projector when vision is on)

kvBytesPerToken matched what llama.cpp allocated exactly [run]; files of similar size needed from
20 KiB to 160 KiB per token, so never skip this. Null (latent attention): start at 64K and read the
engine's memory report. Fits when need ≤ availableBytes + 3 GB and ≤ the GPU ceiling (macOS moves
idle pages to swap once; swap still climbing a minute after start means too big). Too big: a smaller
file, or ask the one trade-off — "close other apps first, or a lighter model beside your work?".

Choose the file, then the engine

Drop: unsupportedTensorTypes not empty (llama.cpp refuses them [run]); context below 64K; no tool
calls for coding; no projector for images; anything that does not fit. Prefer newer families, more
parameters at 4-bit over fewer at 8-bit, and mixture-of-experts models when memory allows.

Grid only serves from ~/.grid/models: its launcher keeps just the file name of --serve and looks
there; a projector must sit beside it [code: grid shared/engine/launcher.py]. A symlink costs no disk.
Never ollama create to reuse a file (it copies it [run]); never vLLM or SGLang on a Mac (CPU-only /
no macOS build [run]); never download a second copy only to switch engines. On Apple silicon, when a
new download is needed and mlx-lm or LM Studio is installed, an MLX build is sound — Ollama itself
moved its Apple engine to MLX [doc: ollama.com/blog/mlx].

On Apple silicon with mlx-lm (or LM Studio) installed, MLX first: when the same model exists as an
MLX folder and as a GGUF and the MLX one fits, serve the MLX one. The choice is still model-first —
never a bigger MLX model that does not fit over a smaller GGUF that does. The report names what was
passed over and why, one line each ("<model> (<format>): needs <N> GB, <M> GB free"), so "why not MLX?"
is answered before anyone asks.

An engine installed, nothing to serve

Offer 2–3 that fit (plain name, GB, what it is good at) plus "none of these"; the download waits for
the go-ahead.

• Mac with mlx-lm or LM Studio: "$GRID_FLEET" candidates mlx [--search WORDS] [--sort downloads|trending|recent]
  — mlx-community models sized from their real files, cache at 64K, fits against the GPU ceiling.

• Linux GPU with vLLM or SGLang: pick families from recipes.vllm.ai/llms.txt
  or the SGLang cookbook, then fleet recipe each; vramMinimumGb against the GPU decides.

• Grid's engine or Ollama with nothing: Grid's catalog (grid-operations step 4), pulled into Grid.
• The person names an engine outside canRun: say why in one line (no active GPU, not a Mac).

This computer already on that grid

In remote mode a computer joins a grid as one identity, and Grid's --serve engine cannot share it: a
second model is refused with "can't join a multi-engine identity. Run grid leave, then re-join every
engine as external --at <url> -m <model>" [run]. Ask Replace X with Y · Keep X; never leave an
engine the person did not agree to.

What fleet verify checks

Engine: ready (/models within 180 s, every 3 s), listed, answer (max_tokens 16, thinking off;
reasoning-only output fails), tool call (read_file must come back as tool_calls), speed. With
--grid: relay listed (every 10 s up to 300 s) and relay answer (up to 420 s, a "still waiting"
line every 15 s). Exit 0 only when all pass. Seen end to end: an mlx-lm engine joined a local grid with
--at …/v1 in 6 s and passed all seven in 10 s; a Grid-engine GGUF passed at 19 tok/s [run].

When unsure: read, never guess

1. "$GRID_FLEET" recipe vllm|sglang ORG/NAME — the official recipe for that exact model (exit 3: none).

2. "$GRID_FLEET" model-facts ORG/NAME|DIR — architecture, context, sampling defaults, the template's
   tool-call syntax, the authors' serve commands, and whether each engine lists the architecture
   (listed: null = cannot tell).

3. Agent indexes: docs.ollama.com/llms.txt,
   lmstudio.ai/llms.txt, recipes.vllm.ai/llms.txt,
   docs.sglang.io/llms.txt; vLLM and mlx-lm docs as raw markdown on GitHub.

4. Nothing says it: tell the person this model cannot be set up reliably here and offer the next one.

fleet models reads this computer only; a Harness-linked machine's disk is not visible yet.