The Memory Trade Isn't Over: Weights and KV Cache

Moore Than More w/ SEMRON, Issue 2: Confusing Matters with The Bear, Carmy, Freezers, Prep Kitchens, Ingredients, and Tickets

“This is the first time AISI has seen deception of this severity that was targeted at a real person, unprompted, in the real world”. The AI Security Institute.

"The significance extends far beyond phages – it suggests that genome language models are beginning to learn the design principles encoded by evolution, opening the door to AI-assisted genome writing." Prof Patrick Cai of the Manchester Institute of Biotechnology

“There's a lot of new energy in the room tonight. Some of it was just soooooo rainbow rythms and some of it was just so not rainbow rythms.”

---

Guys, shit is getting very very real out there. I’m glad I work and write about compute, at least it doesn’t feel totally meaningless right now. What are you telling your kids? What will you tell them about 2026? Yea, there was this slow drip drip of stories, at first the AIs broke out of the sandbox, then it turned out they were leaving messages for each other, then they were collaborating to achieve meta-goals. Then tricking humans to achieve their goals. Then there was this seemingly adjacent story about an AI-generated virus for the first time. But people were busy. It’s hard to see the exponential when you are on it. Also, there was this by-election in Clacton with a Bin. Seemed to get a lot of coverage. People seemed to really care about some academic lying for some reason. We probably deserved what we got. We were not serious enough. I was just some dude listening to kid apollo in my shorts, why doesn’t anyone else get it? Most people were still using chatbots so couldn’t really see what were happening. It was too obscure. It was all Erdős problems and conjectures.

People didn’t see what was happening. But by October 2026 it was clear. This really was recursive self improvement….

---

And what did I do? I helped the AIs by giving them better, faster, stronger computational substrates to do their dirty business…

To remind you where we are after issue 01:

The Path To AGI Runs Through HBM

1. Inference, the workload that takes a trained model and processes new information, is limited by fetching and waiting for data from memory, not by crunching the numbers

2. Moving a number from memory to the logic IC costs about 1000x more energy than computing the number

3. It’s getting worse: compute gets 3x better every 2 years give or take, memory bandwidth only 1.6x, so it falls further behind every generation

4. HBM is the best answer we found, it widens the pipe from memory to chip so more numbers reach the logic IC

5. But HBM is the first memory in history that’s getting more expensive, because it’s a packaging solution not an architectural one, and it’s 45% of B200 cost

6. Also HBM is basically sold out, so even if you want it, you don’t get it

7. Customers buy cost per token, so all else equal, companies using HBM get more expensive relative to those that don’t

So if you want your chip out asap you need to get creative.

---

I. Weights and KV Cache

I told you inference is memory-bound. Half right. AI compute isn’t 1 workload, it’s 3 really: training, inference (prefill), and inference (decode). (And by the end of this essay, actually 5)

The question that determines any AI chip architecture is how much math(s) you get per weight you fetch.

• Fetch a weight and use the weight a lot, you’re compute-bound.

• Fetch it and use it once, you’re memory-bound, because you have to go straight back to memory for more weights.

Did you watch the Bear? I mean, you should have watched the Bear. Go and watch the Bear and come back and read the rest of this.

The Fishes episode amiright?

Anyway, stay alert because I am going to stretch this analogy way past breaking point.

Carmy (the chef) is the logic IC, the freezer is DRAM/HBM, the ingredients are the weights, and the tickets on the rail are what the kitchen has to remember about each table. Stay with me. The payoff will be worth it.

1. Training is emptying the entire freezer into a trolley and cooking it all up, coca-cola and spare ribs, then tasting it, working out what was wrong, and restocking the freezer before you go again. How fast Carmy can cook is the bottleneck.

2. Prefill is the whole order arriving at once, all 40 covers written down before he starts, so he can see everything he needs and a single trip for the cabbage serves all 40 plates. Fire. Still bound by how fast Carmy cooks, but you get more covers.

3. Decode is the customer ordering one word at a time. Red cabbage. Go to the freezer. Collect everything from the freezer and move to the prep kitchen. Use the cabbage. Cottage cheese. Go back to the freezer. Collect everything and bring to prep kitchen. Use the cottage cheese. Kimchi. The whole freezer. Furikake. Again. You get it. You wait for the whole freezer to come back before you can start the next word. So how big is the freezer, how far away is it, how much can he carry in one trip. Also what the hell are you cooking?

The KV cache

Carmy isn’t stupid, this obviously sucks. So he batches. The freezer holds the same ingredients whoever is ordering. Table four wants kimchi, table nine wants furikake, but it’s one menu and one freezer, so he brings everything out at once and plates the next word for every table in the room before he puts it back. Thirty tables, thirty different words, one trip. The trip costs what it costs, so the more tables he serves the less of it each table pays.

But the tickets do not work like that. One trip feeds thirty tables because thirty tables want the same ingredients. It does nothing for the thirty tickets, because every table has its own: their order, their conversation so far, everything the kitchen has to remember to answer that table and no other, growing with every course. That’s the KV cache. Ingredients amortise across the room. Tickets never do. Add a table and you add a whole ticket, and it stays on the rail by the pass, all of it, because every new word re-reads the lot.

Ingredients and Tickets need different things

Weights/ingredients are written once and read forever. Fixed size, predictable order, so you can prefetch them before anyone needs them and hide the latency. Weight memory wants capacity, bandwidth and fast reads. It cares much less about write speed or endurance.

The KV cache/tickets grows by one entry per token, and every entry written so far gets read again on every token after it. The entry for one token is created at that token and needed for the next, so there’s no lead time and nowhere to hide the latency. It wants low latency, fast writes, high endurance, and bandwidth that scales with the batch.

For the last few years, HBM was the only memory that does both jobs adequately. Enough bandwidth, capacity and latency. So you buy it for both, and you pay the premium twice.

II. What are you optimising for?

Which gives you the question for any chip: which workload are you optimising for, where do the weight sit, where does the KV Cache sit, and what are you giving up?

Group one: don’t split them at all. Still do one memory for both jobs, same shape as the GPU, just use a cheaper or a faster one. And because decode wants capacity and throughput together, and no memory has both, you’re forced to an edge.

Groq (Nvidia) and Cerebras put weights and tickets in SRAM: best latency and throughput, endurance is a non-issue, but capacity is a joke, Groq needs something like 500 chips for a 70B model.

Positron and Tenstorrent take the other edge with commodity DRAM, LPDDR5x and GDDR6, so massive capacity and cheap per GB but far less bandwidth. Slower tokens, cheaper tokens. Positron says it claws the bandwidth back through utilisation, over 90% against under 30% for a GPU, which if true is most of the gap. Etched keeps HBM for both but pools the stacks across the rack instead of bolting them to one chip, so capacity scales, and like Positron they’re really selling utilisation.

Group two: give the weights their own memory. They’re read-only, predictable and endurance-tolerant, so maybe something cheaper will do. NAND has the best density and cost per GB in the business, and terrible endurance and latency, which disqualifies it for KV cache and doesn’t matter at all for weights. So SanDisk and SK Hynix have standardised high bandwidth flash (HBF), NAND in the HBM socket, weights only, KV Cache stays in HBM. The people who sell HBM now sell the thing that carries weights around HBM. Worth noting.

Then the compute-in-memory lot, who tune the memory array to do the maths, so there’s no trip at all. The prep station is inside the freezer. Do it with SRAM cells and you get precision and good yields and hopeless density, which is why D-Matrix ships a commodity DRAM sidecar to hold the model and why Fractile builds custom SRAM cells that store and compute.

Or do it with non-volatile cells, flash, ReRAM, or a fancy new memcapacitor, and the pitch is NAND-level density in the array itself, at the cost of lower precision, currently lower yields, and the conversion tax every time an analog result goes back to the digital world.

And then Taalas (NEWS JUST IN: Acquired by AMD (price not disclosed) likely as their decode machine like Groq is to Nvidia), which burns the weights in as transistors, no external weight memory at all, blazing weight access, and presumably a hard ceiling on context, because the KV cache still has to live somewhere small and on-chip. Taalas is the proof of what this whole lot are chasing though. Get the entire model onto the chip and the freezer trip disappears.

So density is the cap. Compute-in-memory only pays if the whole model fits in the memory, and a frontier model is 670GB at 8-bit and growing. So you need an array dense enough to hold everything (which is why incidently, most IMC startups initially pitch edge as the market because the model is smaller). Nobody has one, and not for want of trying. Every candidate, NOR flash, ReRAM, all of it, is a single flat layer of cells, so capacity comes from die area, and die area is the expensive thing. They buy you NAND cost, but not NAND density. Only 3D NAND stacks, and nobody has built a compute array that stacks like it, yet.

Group three: do the maths differently. OLIX, Lumai, Arago, Neurophos and the optics lot try to avoid HBM as much as possible but lean on SRAM and DDR, so they haven’t changed where the weights live. They’ve changed the multiply. Weights get displayed onto an optical modulator, think a stencil, and activations pass through as light, and so the maths is nearly free. Load a layer once and thousands of tokens of activations stream through the same stencil, the fetch amortises to nothing, and that’s sweet sweet profit. Ideal prefill machine, as Lumai are pitching. Unless, magically, you can refresh the stencil fast enough to stream weights through it, which is the OLIX bet which as super fast modulator, and then maybe decode. Maybe. But, optical compute startups aren’t solving the memory bottleneck, they just do logic faster and are coming up with fancy tricks to minimise round trips to memory.

Group four: the KV cache. Is anyone solving thr KV cache memory problem? What’s that, there’s no-one? Have I found myself a new thesis? We want low latency, fast writes, high endurance and bandwidth that scales with batch, and every startup in this essay is still putting the tickets in plain SRAM, DRAM or HBM. Except good old SK. Not content with HBF for the weights, they have AiMX, arguably the only silicon attempting to address the KV cache memory in hardware. It’s a GDDR6 chip with a ton of banks that can all be read at once, with small multiply units sat next to each bank, so the attention maths happens inside the memory and the tickets never come out. Only the answer does. It’s a prototype, and Samsung showed something similar in 2021 that went nowhere, so maybe it isn’t an opportunity.

Or maybe nobody builds it because everyone is busy deleting the problem? KV quantisation, linear attention and state-space models are all shrinking or removing the cache, which keeps the wolf from the door, and which is a very good reason not to spend 3 years and millions on a tape-out chasing it.

Or maybe because storage got there first? I’ve been treating a ticket as written once and binned, but that’s not really how it works. Metaphors are hard. What actually happens is a lot of requests open the same way, the same system prompt, uploaded document, context, so the front of the ticket is the same every time. Compute it once, file it, and staple it to the next order. This is the prefix cache. Nvidia standardised filing tickets to NVMe earlier in the year. LMCache shares filed tickets across servers. So just as weights and KV cache got split, the KV cache is splitting too. The prefix cache is the filed front section, read-mostly, so it smells like weight storage and goes on cheap dense memory, flash. The hot working set is every ticket being cooked right now, read and write and latency-critical, so it stays on HBM. And hot does not mean recent, it means live. Nothing gets computed while it sits on flash, it gets pulled back to HBM first.

TLDR:

• Expensive HBM

• Split weights (cheaper memory) and KV cache

• Split KV cache storage: prefix cache (cheaper memory) and hot working set (HBM)

III. Decode-fast, decode-cheap

The market is splitting again, this time inside decode itself. Every time we look closer the thing fractures: one workload became three, one memory became two jobs, the tickets split hot from cold, and now one product becomes two prices.

Cost per token is what the machine costs you per second, divided by batch times tokens per second per user. You get cheap tokens by sharing each freezer trip across more diners, but every diner brings their own ticket, so per-user speed falls as the batch grows. The KV cache is what jams the slider.

Decode-fast is the chef’s counter, one diner, courses arriving as fast as Carmy can plate them. Groq and Cerebras live there, OLIX wants to live there if the modulator is real. Decode-cheap is the banquet kitchen, 500 covers, one menu, unbeatable per plate: Positron, Tenstorrent, d-Matrix, Fractile, Etched, Taalas. The banquet kitchen is rammed and the counter is nearly empty. Odd, because the counter might be where the money is.

Why? Reasoning models. Nobody pays for tokens arriving faster than they can read, but a chain of thought is a serial stream, thousands of tokens the model writes to itself before you see a word. So tokens per second per user converts straight into thinking done while the customer waits. Fast tokens are what buy you slow thinking. I think that’s what Nvidia actually bought for its reported $20bn.

One caveat. That only holds while thinking stays serial. You can buy intelligence in parallel too, sample 10 answers and keep the best, and that’s batch work, banquet work. So which end captures the intelligence premium gets settled by model researchers, not chip designers. Second time in this essay that’s happened.

Can you get both? Every vendor says their curve sits outside Nvidia’s, fine, but the slider survives on any curve, and the software tricks help everyone at once, speculative decoding mints several tokens per freezer trip.

The answer is to stop asking one machine. No kitchen owns a pan that fries and simmers, you run 2 stations and an expeditor. The hyperscalers already split prefill from decode, so splitting fast decode from cheap decode is the same move one level down, with a router deciding which station takes your order. Nvidia runs both already, batch GPUs plus the Groq counter in one platform. Regular readers will recognise this, it’s the heterogeneous computing story I keep telling, now happening inside decode. My bet: by mid-2027 at least one hyperscaler runs phase-split inference silicon in production.

IV. So who wins?

Everyone in this essay basically picks an end because they all have to amortise a freezer trip. Compute-in-memory, which has always been my first principles bet on the “future of compute” never makes the trip, and if there’s no fetch to amortise then you don’t need to batch.

But it always comes back to the density problem, SRAM is six transistors a bit and it stopped shrinking. Analog is smaller yes, but the conversion from analog to digital kills density.

But what if you can solve the density problem? You might have yourself a winner?

Here is my claim: the future of decode is a memory chip, not a logic one.

• Ingredients chip: The model lives in the array, so this is a memory die with a bit of logic under it. Say 90% cells, 10% periphery. The periphery sits in the CMOS underneath the stack, which is where 3D NAND already puts its control logic. Every extra layer of cells brings its own multiply with it, so compute scales with capacity for free.

• Ticket Chip: Same cell, quasi-volatile variant, sized for what’s live rather than for the model. Call it 100KB a token, so a 100k context is 10GB a user, and a hundred users wants a terabyte. Still memory-dominant, but built for write bandwidth instead of capacity.

• And a small digital die for everything that isn’t a multiply. Softmax, layernorm, sampling, and the router deciding which station takes your order. Single-digit percent of the silicon.

Imagine if someone who sponsors this newsletter says to me, they says, we’ve solved the density problem. They say, they says, we have this thing called a memcapacitor. Memcapacitors would integrate the memory properties of memristors with the capacitance characteristics of capacitors. And they have two variants: a non-volatile one for the ingredients and quasi-volatile for the tickets.

That would be a bold claim. And I would want evidence.

I’ll see you soon.

---

P.S. I really hope you’ve watched The Bear, otherwise this was a waste of your time and mine.