Packing Ternary Weights at 1.6 Bits per Trit

The BitNet b1.58 paper uses ternary weights (-1, 0, 1), but storing them naively wastes space. The theoretical minimum is log₂(3) ≈ 1.585 bits per trit. A practical scheme needs to get close while allowing fast parallel unpacking on modern hardware.

Why 5 trits per byte

5 ternary digits represent 3⁵ = 243 states, which fits snugly into 8 bits (256 states). That's 1.6 bits per trit — 99.06% of the theoretical limit. The next best block sizes (3 trits in 5 bits, 4 trits in 7 bits) are less efficient or awkward for byte alignment.

The fixed-point trick

The core insight: treat the packed byte as a fixed-point number in [0, 1). Multiply by 3 and grab the integer part to extract the most significant trit, then repeat with the fractional part. No divisions or modulo needed — just multiplication and bit shifts, which SIMD loves.

Packing requires a ceiling division: b = ((b * 256) + 242) / 243. Unpacking is a tight loop of b = b * 3; trit = b >> 8; b = b & 0xFF.

The author implemented this in llama.cpp for TriLM and BitNet b1.58 inference, with AVX2 and NEON SIMD paths. The pull request is here.

Why it matters for LLMs

Ternary quantization cuts model size dramatically — BitNet b1.58 uses 1.58 bits per weight on average. Efficient packing and unpacking directly impacts inference throughput, especially on CPU. This technique is a neat example of trading a small amount of density for a big win in decode speed.