Intel has developed a new compression format, BITCOS, that reduces the storage requirements of ternary language models to 1.485 bits per weight, down from the theoretical minimum of 1.58 bits. This is achieved without altering the model’s weights or accuracy.
The innovation lies in how BITCOS stores the weights. It separates them into two streams: one for recording the presence of a weight (zero or non-zero) and another for the sign of non-zero weights. This approach leverages the fact that ternary models often have a higher proportion of zero weights than the standard 1.58-bit calculation assumes.
Breaking the 1.58-Bit Barrier
Ternary models, which use only -1, 0, and +1 weight values, are typically stored using a method called five-trit packing. This method fits five ternary values into an eight-bit byte, resulting in an average of 1.6 bits per weight. However, when weights are stored in blocks of 128, the final byte is often partially unused, pushing the actual storage cost to 1.625 bits per weight.
Read Also: Zed Launches Delta to Replace GitHub
Intel’s research revealed that zeros account for a significant portion of weights in ternary models, ranging from 29.7% to 51.5% across 29 checkpoints from seven model families. BITCOS exploits this zero-heavy distribution by storing zeros more efficiently, requiring only a single bit to indicate their presence.
Efficiency Gains and Performance Boosts
BITCOS becomes more efficient than five-trit packing when more than 37.5% of a model’s weights are zero. This threshold was met in 26 of the 29 checkpoints Intel examined. The most significant reduction was achieved with a ternary version of Qwen3-1.7B, where 51.48% of weights were zero, resulting in a storage cost of just 1.485 bits per weight.
Beyond storage savings, BITCOS also improves decoding speed. Intel developed specialized unpacking kernels for AVX-512 and AVX2 CPUs, as well as Xe2 GPUs. These kernels reduce the amount of weight data moving through memory during token-by-token decoding, leading to performance gains.
Read Also: Meta simplifies WhatsApp Business setup with AI
Performance Varies Across Hardware
On AVX-512 hardware, BITCOS demonstrated 10% to 18% faster decoding on a 64-core Xeon server and 2% to 15% on a 24-core Core Ultra 9. GPU performance also saw improvements, with 9% to 22% faster decoding on the integrated Arc 140V and 2% to 27% on the discrete Arc Pro B70.
Limitations and Future Directions
The BITCOS paper has not undergone peer review, and all tests were conducted on Intel hardware. Further evaluation on Nvidia, AMD, and Arm systems is needed to assess its broader applicability. Additionally, the benchmarks focused on seven models with a batch size of one, leaving room for further exploration with larger batch sizes and diverse model architectures.
Hardware-Specific Optimization Insights
The format’s effectiveness varies depending on whether compute, memory, or bandwidth is the bottleneck. Intel developed separate unpacking kernels for AVX-512, AVX2 CPUs, and Xe2 GPUs to address these differences. On AVX-512 hardware, the kernel uses a presence bitmap and pdep instruction to scatter sign bits efficiently, while Xe2 GPUs rely on a lookup table for the same operation.
