How a Memory-Saving Algorithm Tanked Memory Stocks

Opening

Subscriber, something curious happened in Wednesday’s U.S. markets. While the Nasdaq 100 climbed, memory semiconductor stocks moved in the opposite direction and fell. SanDisk -5.7%, Western Digital -4.7%, Seagate -4%, Micron -3%. The trigger was a compression algorithm called TurboQuant, published by Google Research. The effect reached the Korean stock market almost immediately too. As of 6 PM, when I’m writing this newsletter, semiconductor-related companies like Samsung Electronics, SK hynix, Hanmi Semiconductor, Leeno Industrial, and Wonik IPS were mostly down.

“A technology that lets AI use less memory” — read just the headline, and it sounds like bad news for memory makers. But what this technology actually shrinks is the temporary working memory on the GPU (the KV cache), not the HBM¹​ or DRAM modules plugged into servers. There’s a gap between the signal the market read and the signal the technology is actually sending — and behind that gap sits a bigger question spanning the entire AI hardware stack. Today I want to walk through both the gap and the question. (Honestly, if you don’t care much about the mechanics, feel free to skip straight to “Why Did the Market React?”)

What TurboQuant Actually Does

Let’s start with the technology. When an AI keeps a conversation going, it needs to remember what was said earlier — and to do that, it stores information in a temporary memory called the KV cache²​. The longer the conversation runs, the more exponentially this memory grows, making it one of the main drivers of AI service costs. TurboQuant is an algorithm that compresses this memory as small as possible while preserving nearly all of its content. No additional training or fine-tuning required.

Here’s an analogy for the core idea. The data an AI normally stores has wildly varying magnitudes — some values are large, some small, and the distribution is uneven. Data like this is hard to compress efficiently. The first stage of TurboQuant, called PolarQuant, applies a random rotation to this data, flattening the distribution of all values into something uniform. It’s like shuffling a pile of unevenly sized luggage until everything comes out roughly the same size — which then packs neatly and efficiently into identically sized boxes. In the paper’s mathematical terms: after rotation, each coordinate follows a beta distribution and becomes nearly independent of the others, which allows optimal scalar quantization³​ to be applied to each coordinate individually.

But there’s another problem here. AI doesn’t just store information — it constantly performs comparisons of similarity between stored pieces of information (inner product operations). The paper proves that a quantizer optimized for MSE (mean squared error) introduces a systematic bias into these comparison values. That’s why a second stage is needed: QJL (Quantized Johnson-Lindenstrauss). After the first stage compresses the bulk, QJL corrects the remaining residual error with just 1 additional bit. This two-stage approach eliminates the comparison bias entirely.

The paper’s experiments show that at 3.5 bits, quality is essentially identical to the original model, and even at a more aggressive 2.5 bits, quality loss is minimal. On Llama-3.1-8B-Instruct, the average LongBench benchmark score was nearly unchanged from the uncompressed version (50.06 vs 50.06), and the model maintained 100% accuracy on a Needle in a Haystack test at 104,000 tokens. The compression ratio exceeds 4.5x.

What’s especially striking is the speed. Traditional product quantization (PQ) took about 240 seconds to index 1,536-dimensional data; TurboQuant took just 0.0013 seconds — roughly a 180,000x difference. That’s because TurboQuant is a data-oblivious, online method that doesn’t need to learn a codebook tailored to the data.

One thing worth flagging, though: the “up to 8x speedup” figure that Google’s blog emphasized was measured for a specific step — attention logit computation — against a JAX baseline. It is not 8x end-to-end inference throughput. And the “6x memory reduction” figure differs subtly between the blog and the paper — the paper is more conservative, stating “4.5x or more.” Numbers get packaged differently depending on the announcement channel, and that’s always worth watching when reading tech news.