Google Research publishes TurboQuant, a compression algorithm for large language models

Google Research presented TurboQuant in March 2026, a compression algorithm targeting the Key-Value (KV) cache of large language models. During the inference phase, the moment when a model actually generates responses, that cache grows rapidly and represents one of the primary bottlenecks in deploying LLMs at scale.

According to Google researchers Amir Zandieh and Vahab Mirrokni, TurboQuant compresses that cache to an average of 3 bits per value. On Nvidia H100 GPUs, tests with open-source models such as Gemma, Mistral and Llama-3-8B yielded six times lower memory usage and eight times higher processing speed, with no reduction in model quality. The algorithm requires no retraining or fine-tuning of the model.

The announcement immediately drew attention beyond the research community: shares in memory chip manufacturers Samsung, SK Hynix, Micron and SanDisk fell temporarily following the publication. Analysts quickly put that reaction into perspective, but it illustrates how sensitive hardware markets are to news about AI efficiency.

How TurboQuant compresses the KV cache

TurboQuant works in two steps. The first step is called PolarQuant: data vectors in the KV cache are randomly rotated and then compressed. That rotation spreads information more evenly across the vector components, making compression more accurate.

The second step applies the Quantized Johnson-Lindenstrauss method (QJL). This eliminates the remaining rounding errors that persist after PolarQuant, at a cost of just one additional bit of memory per value. The combined effect brings the average bit width of the cache to approximately 3 bits, whereas standard inference typically uses 16 or 32 bits per value.

An important distinction from some other compression techniques is that TurboQuant exclusively compresses the ephemeral KV cache, not the permanent model weights. A model therefore does not need to be reloaded or modified; compression takes place at runtime.

What this means for those deploying or building AI

The KV cache grows linearly with the length of a conversation or document. With long contexts, multiple concurrent users, or models with large numbers of parameters, memory requirements escalate quickly. This currently forces companies to choose between costly GPU clusters or shorter contexts. A sixfold reduction in memory usage offers tangible headroom here.

In concrete terms, this opens the door to applications that have until now been too expensive or technically too demanding, such as AI assistants that maintain a long-running conversational memory, models running locally on hardware with limited GPU memory, and vector search over large datasets without a proportional increase in hardware.

For AI builders looking to cut costs today, the barrier-free implementation, no retraining required, is a practical advantage. Whether and when TurboQuant will become available as a production component in Google's own infrastructure or as an open library is not yet clear from the current announcement.

The reaction in memory chip markets was premature

The share price declines at chip companies following the announcement reflect a line of reasoning that analysts consider incomplete. TurboQuant compresses working memory during inference, not the storage of model weights or the training phase, which by far consumes the most GPU capacity.

Furthermore, economists point to the so-called Jevons paradox effect: when a technology becomes more efficient, the cost per unit of output falls, which can increase total demand rather than reduce it. If AI inference becomes six times cheaper, companies are likely to deploy it at greater scale, keeping overall demand for hardware stable or even pushing it higher.

The market reaction does illustrate how strongly sentiment is tied to hardware-intensive AI: a single research result on efficiency gains is enough to temporarily wipe billions off the market capitalisation of chip manufacturers.