LocalAI is a self hosted OpenAI compatible inference server that fronts llama.cpp, vLLM, whisper.cpp, and related backends. mudler/LocalAI put 143 commits on master in the past week. Most of that is nightly pin churn. The operator visible work is a credentials file for private galleries and registries, a new local-ai benchmark command, and a gallery restore after a batch apply overwrote seven model entries.
Private registry auth from a credentials file ¶
Private GHCR, Harbor, and Artifactory pulls used to depend on docker login or one off tokens. The credentials file change adds --credentials-file and LOCALAI_CREDENTIALS_FILE. If neither is set, local-ai run loads credentials.yaml from LOCALAI_DATA_PATH when that file exists. The same load path covers worker, backends install, and models install, not only the HTTP server.
The file is a YAML list of match rules. Secrets live in env vars or mounted files, not in the gallery index. LocalAI reads _env and _file values at request time, so a rotated Kubernetes secret is picked up without a restart. Invalid YAML or an unknown key stops the process at startup. Parse errors redact values.
- match: ghcr.io/acme
username: bot
password_env: GHCR_TOKEN
Matching is prefix based and per hop. A redirect to a CDN that no rule matches gets no credential. Plain http:// only gets credentials when allow_insecure is true. Registry names on private IPv4 ranges, localhost, or loopback are matched as http://, so a rule written as https://192.168.1.5:5000 never applies to those pulls. GHCR, Docker Hub, and Quay want basic auth with the token as the password, not a bearer entry.
If no rule matches, docker config is still tried. ollama:// manifests stay anonymous; only blob downloads use the file. In distributed mode the controller needs credentials for model files, and every worker needs the same file for private backend images. Credentials are not sent over NATS. The CLI reference now documents the flag.
CLI benchmark of configured text models ¶
The benchmark command talks to a running server. It does not install, discover, or unload models. Pass configured aliases, not backend names. Each model gets warmup requests, then sequential /v1/chat/completions calls with temperature 0 and streaming off.
Reported latency is the full request, from send through parse, including queueing and prompt processing. It is not time to first token, and it is not a substitute for llama-bench. Throughput is server reported completion tokens divided by those full durations. Missing usage fields become N/A in the table and null in JSON. A failed run stops the command before any results are written.
--endpoint defaults to http://127.0.0.1:8080. Credentials in the URL, query strings, and fragments are rejected. Redirects are refused. The API key can come from --api-key, LOCALAI_API_KEY, or API_KEY, and is stripped from the JSON report. --json dumps settings plus every sample, which is the form worth keeping next to the model YAML.
Gallery clobber, Ministral tools, and Orukeet ¶
gallery/index.yaml was the busiest file, touched in 34 commits. A restore put back seven entries that a batch of gallery: apply PR commits had overwritten. Each new top of file entry replaced the one above it instead of inserting. Lost names included qwen3.8-27b-uncensored-q4, the spark-x2.5-4b Q4/Q6/Q8 set, and deepseek-v4-flash-vision-exp, whose name line had been rewritten into a duplicate. The restore matches the YAML in the original PR heads. Unique name lint caught the last collision.
Separately, the Ministral 3 14B Reasoning gallery entry stopped inheriting the Mistral 0.3 prompt and JSON parser. Name first tool calls such as pick_tool{...} were reaching clients as plain text. The entry now points at virtual.yaml, sets backend: llama-cpp, enables use_tokenizer_template and use_jinja:true, and uses llama.cpp native tool parsing with grammar disabled. Gallery updates do not rewrite installed model YAML. Existing installs have to edit template and function, drop the inherited stopwords list, and reload.
Orukeet landed as a 0.6B, 25 language fine tune of Parakeet TDT v3 on the nemo-speech-cpp backend. Q8 GGUF, CPU first, GPU optional through the backend gpu option.
Nightly C++ backend pins ¶
The rest of the window is the bump bot. Pins moved for ggml llama.cpp, whisper.cpp, audio.cpp, stable-diffusion.cpp, vllm.cpp, and the Darwin vllm-metal wheel to v0.29.0. The ROCm CTranslate2 wheel is at v4.8.2. backend/cpp/llama-cpp/Makefile is the usual pin site.
The TurboQuant bump is the one that can break a custom build. Upstream renamed the RPC server target from rpc-server to ggml-rpc-server. LocalAI now builds that target and still packages the binary as turboquant-rpc-server. Released images keep the old packaged name. A Makefile that copies rpc-server out of the llama.cpp build directory will fail at the copy step.
What to watch ¶
Mount a credentials file and set LOCALAI_CREDENTIALS_FILE before pointing galleries at private GHCR or Harbor. Give workers the same file. The controller copy is not enough for backend image pulls.
If Ministral 3 14B Reasoning is already installed, edit the model YAML. A gallery pull will not fix the old Mistral 0.3 prompt.
The LocalAGI 0.6.5 bump changed state.NewAgentPool and broke the Go build until pool limits were passed through. Defaults now cap conversation dumps at 200 per agent and 30 days, scheduler runs at 20 per task, and self scheduled tasks at 100 per agent. Agent nodes that were accumulating dumps on disk will start pruning.