Cactus Hybrid: On-Device Models That Know When They're Wrong
Cactus Hybrid post-trains small models to output a confidence score with every answer, enabling reliable cloud handoff only when needed. The first release, Gemma 4 E2B Hybrid, routes just 15–35% of queries to a larger model while matching Gemini 3.1 Flash-Lite on benchmarks.
Cactus Hybrid is a pragmatic answer to a perennial problem in edge AI: small models are fast and private, but they hallucinate or fail silently. The Cactus team post-trains compact checkpoints to emit a structured confidence score (0 to 1) alongside every answer — not parsed from text, but as a first-class output from the model itself. The logic is brutally simple:
if confidence < 0.85:
answer = ask_a_bigger_model(prompt)
The first model in the lineup, Gemma 4 E2B Hybrid, is available on Hugging Face. According to Cactus, it matches Gemini 3.1 Flash-Lite on most benchmarks while routing only 15–35% of queries to that larger model — the rest is handled on-device.
How It Works
The confidence probe is baked into the checkpoint during post-training. It reads a modality-independent correctness signal from the model's hidden states, not from surface-level patterns. The evidence: the probe achieves 0.79–0.88 AUROC on four audio benchmarks despite having been trained on zero audio data. That rules out memorization and suggests the probe is genuinely measuring whether the model knows its answer is sound.
Cactus provides copy-paste quickstarts for four inference engines:
- Cactus (native Python bindings)
- MLX (Apple Silicon)
- Transformers (Hugging Face, with a caveat: avoid
device_map="auto"because the probe runs outside the module forward path) - llama.cpp (requires a one-time patch to the engine)
All code snippets are MIT-licensed; the Gemma model itself is subject to Google's terms.
Benchmarks: Handoff Rates
The table below shows what fraction of queries must be handed off to Gemini 3.1 Flash-Lite to match its accuracy at different quantization levels:
| Benchmark | Handoff to match Flash-Lite (FP16) | At 4-bit | At 3-bit |
|---|---|---|---|
| ChartQA | 15–20% | 25–30% | 40–50% |
| MMBench | 30–35% | 40–45% | 50–55% |
| LibriSpeech | 25–30% | 35–40% | 55–65% |
| GigaSpeech | 30–35% | 40–45% | 50–55% |
| MMAU | 30–35% | 35–40% | 50–55% |
| MMLU-Pro | 45–55% | ~90% | n/a |
At 3-bit quantization, handoff rates climb sharply — especially on ChartQA and MMLU-Pro — suggesting the probe's confidence signal degrades with aggressive compression.
The probe was trained on zero audio data, yet achieves 0.79–0.88 AUROC on four audio benchmarks. This rules out surface-level explanations — the probe is reading a modality-independent correctness signal from the hidden state.
Source: GitHub