Local models

Olyvr can download an open-weight model and run it on your computer with llama.cpp. Nothing is sent to a cloud provider.

Preview

Once a local model is running, chat uses it when you haven’t set up a subscription or a custom endpoint. If you have, pick the local model from the model menu in the composer, or turn on run every model call on the local server to keep everything on your computer.

Available models

Model	Variant	Download	Recommended RAM
gpt-oss 20B	F16	13.7 GB	24 GB
Qwen3.6 35B-A3B	UD-IQ3_XXS	14.1 GB	24 GB
Qwen3.6 35B-A3B	UD-Q4_K_M	22.7 GB	32 GB

If you don’t choose a variant, Olyvr picks one that fits your machine’s memory. Qwen3.6 35B-A3B can also read images.

Install and run a model

1. Open Settings → Model access → Local models.

2. Choose a model and click install. Olyvr downloads the llama.cpp runtime from its GitHub releases and the model from Hugging Face. Interrupted downloads resume where they left off.

3. When the download finishes, the model server starts automatically. Starting can take a few minutes for large models.

Models and the runtime are stored in your data folder, under state/local. Model files are large, so check your free disk space before installing. To free the space again, click the trash button next to a downloaded model; Olyvr stops it first if it is running. Models in Hermes’s folder are never deleted from Olyvr.

Hermes compatibility

If you already use Hermes to manage llama.cpp models, Olyvr detects a running Hermes llama.cpp server and its downloaded models (in ~/.hermes, or $HERMES_HOME) and can reuse them instead of downloading again.

Advanced settings

These environment variables are read when the model server starts:

Variable	Default	Effect
OLYVR_LOCAL_CTX	131072	Context length in tokens
OLYVR_LOCAL_SLOTS	2	Parallel request slots
OLYVR_LOCAL_VISION	off	Set to 1 to load the vision projector
OLYVR_LOCAL_EXTRA_FLAGS	none	Extra llama-server arguments

Platform notes

• macOS: the downloaded runtime is built for Apple Silicon.
• Windows: local models have had less testing than on macOS. If the server does not start, check server.log in state/local and report the problem.

If startup takes longer than 4 minutes, Olyvr stops waiting and points you to the log file.