autonomous-ai / engine-vllm

vLLM (Linux + NVIDIA CUDA or AMD ROCm)

Official sources, read 2026-09-29. Agent index: recipes.vllm.ai/llms.txt.
Reference docs as raw markdown under https://raw.githubusercontent.com/vllm-project/vllm/main/docs/:
OpenAI-compatible server (serving/online_serving/openai_compatible_server.md) ·
Tool calling (features/tool_calling.md) ·
Reasoning outputs (features/reasoning_outputs.md) ·
Supported models (models/supported_models.md) ·
GPU install · Docker ·
Optimization · Conserving memory
Tested: only vLLM 0.11.0 on an Apple-silicon Mac (CPU backend), 2026-09-29 — no GPU run yet.
Tags: [doc] official source above, [run] seen on the tested machine, [?] unverified.

When to use it

• Linux with an NVIDIA GPU of compute capability 7.5 or newer [doc], or an AMD GPU with ROCm [doc], and a
  Hugging Face model (safetensors, FP8, AWQ…). vLLM is built for many requests at once and for models split
  across several GPUs.

• A GGUF file for one person on one GPU: Grid's own engine serves it with nothing to install.

• Not on a Mac. There it runs on the CPU only, slowly (26 tokens in 2.3 s for a 0.5B model), and needed
  pinned transformers<5, --dtype float32 for JSON output and a batched-token limit to start [run].

1. The recipe — "$GRID_FLEET" recipe vllm ORG/NAME

Reads recipes.vllm.ai live (models.json, then <ORG>/<NAME>.json) and prints:

The recipe beats the generic docs: a family's generic parser entry and a model's recipe can name different
parsers [doc]. Exit code 3 means there is no recipe: go to step 2.

2. No recipe — read the model: "$GRID_FLEET" model-facts ORG/NAME

1. support.vllm.listed must be true (its architecture is in vLLM's supported-models list). false:
   the Transformers backend (--model-impl transformers) may still run it [doc] — a test, not a promise.

2. modelCard.serveCommands / toolCallParsers / reasoningParsers: the model authors' own vLLM command.
   Use it when present.

3. Otherwise match chatTemplate.toolSyntax against the parser descriptions in features/tool_calling.md
   (each parser documents the output format it reads) and, when chatTemplate.thinking, pick the reasoning
   parser from the table in features/reasoning_outputs.md [doc]. Quote the doc line you matched.

4. contextLength must be ≥ 65536. Sampling defaults come from the model's generation_config.json
   automatically [doc]; leave them.

5. No match in the docs: say so and stop — do not cycle through parsers until one passes.

Install and version

• Slow step, ask first. Venv: uv venv && uv pip install -U vllm --torch-backend=auto [doc]; pin with
  "vllm==X.Y.Z" at or above the recipe's minVersion. Wheels target CUDA 12.9 by default, builds for 12.8
  and 13.0 exist, Blackwell needs CUDA ≥ 12.8 [doc]. AMD: --extra-index-url https://wheels.vllm.ai/rocm
  (Python 3.12, ROCm 7.0, glibc ≥ 2.35) [doc]. Check the GPU and driver with nvidia-smi.

• Docker: vllm/vllm-openai (NVIDIA), vllm/vllm-openai-rocm (AMD) [doc], with
  --gpus all --ipc=host -p 127.0.0.1:P:8000 -v ~/.cache/huggingface:/root/.cache/huggingface [doc].

Common configs (the recipe's flags win)

Each adds the recipe's (or step 2's) tool and reasoning flags, --host 127.0.0.1 --port P and a
--max-model-len of at least 65536.

Start, ready, join

"$GRIDFLEET" serve vllm-P --env HFHUB_OFFLINE=1 -- ~/.grid/envs/vllm/bin/vllm serve <model id or snapshot dir> \
--served-model-name <id> <flags>

(Install into ~/.grid/envs/vllm so fleet models finds it; fleet serve keeps it alive after your shell
returns; log run/vllm-P.log.)

• Verify: "$GRID_FLEET" verify --at http://127.0.0.1:P/v1 --model <id> --kind vllm (bounded, narrated).

• Ready: PID alive; log shows Application startup complete. [run] (minutes for a big model); GET /health
  → 200 and GET /v1/models lists <id> [run]; one bounded answer; one tool call.

• Join: "$GRID_FLEET" run -- join GRID --at http://127.0.0.1:P/v1 -m <id> --advertise-as ALIAS. /v1 is required —
  without it /models and /chat/completions answer 404 [run]. --api-key guards only /v1 routes [doc].

• Thinking off for everyday use: the recipe's guide names the flag when there is one
  (--default-chat-template-kwargs '{"enable_thinking": false}' in the recipes that document it [doc]).

Knobs

--gpu-memory-utilization (share of GPU memory pre-allocated for weights plus cache), --max-model-len,
--max-num-seqs, --max-num-batched-tokens, --kv-cache-dtype fp8, --enforce-eager (fastest start,
slower serving), --tensor-parallel-size [doc].

Stop

"$GRID_FLEET" stop vllm-P (or docker stop the container you started); confirm the port is free.

Known failures → what to do