LocalAI's C++ Ports Beat Python Stacks — and the Reason Isn't What You Think

LocalAI's default is to wrap someone else's engine — llama.cpp, vLLM, whisper.cpp, and the rest are maintained by people who know those models better. But eighteen of its backends are hand-written C/C++ ports, and each exists because wrapping the upstream would mean shipping something unusable: a multi-gigabyte Python install, a CUDA-only stack, or a model with no C++ implementation at all.

The headline number is vllm.cpp, a C++20 port of vLLM's V1 serving architecture. Installing vLLM produces a 9.1 GiB virtualenv; vllm.cpp is a 66 MiB binary. It implements paged KV cache, continuous batching, prefix caching, the scheduler, and the sampler — no Python, no PyTorch, no ggml at inference. On an NVIDIA GB10 with Qwen3.6-27B, it ties vLLM's production graphed configuration across concurrency 1–32, with single-stream 4.5% ahead and the rest within noise. Peak host memory drops from 28.18 GiB to 24.88 GiB.

Sometimes the port is simply faster. depth-anything.cpp, a port of Depth Anything 3, runs 1.31x faster than PyTorch on CPU with 27% of the memory — not because of better matmul kernels, but because two positional embeddings were being recomputed on every forward pass with single-threaded scalar loops. Caching them removed ~95 ms of host-side overhead per forward. That's the general shape: GEMMs are a wash, the difference is in host-side work a Python reference never optimized.

Parity is the gate, speed is the follow-up. face-detect.cpp and voice-detect.cpp ship even without CPU speed wins, because for biometric pipelines an embedding that differs in the fourth decimal place changes verification decisions. face-detect.cpp matches insightface to cosine 1.000000; voice-detect.cpp uses ~62 MB vs ~334 MB for the Python path.

The method is consistent: convert weights to one GGUF first, port the graph second with per-component parity gates, optimize third with a profiler, and expose a flat C ABI last. The cost is maintenance — each engine has its own CI, benchmarks, and converter — and GPU kernels are the weak spot, with ggml's generic kernels trailing cuDNN on conv-heavy models.