It's an old paper (from 2025, so, a decade ago in AI years), but the concept is still fascinating. And I'm yet to see it show up in any production models.
If multiple models can use cache representations for this kind of enrichment, the KV cache representations of different models must be somewhat compatible.
What stops us then from going a step further, and producing a model family where all models are "KV aligned", and each model can utilize the KV cache of other models directly?
So, an "expensive" reasoning model can use its full faculties to plan, but "delegate" simple subgoals to a smaller model. That smaller model can access the large model's intent directly, as rich KV cache representations - with no prefill recompute and no associated "handover" latency. Or, likewise, a "cheap" small model can generate a diminished but highly compact KV cache that the "expensive" model can then operate on - for example, for skimming a large file for shallow patterns.
A few months ago I asked why semantic representation rather than text wasn't used, since natural language seems quite a lossy representation for semantic concepts:
I wouldn't have thought to use it for LLM-to-LLM communication, though
foota 12 hours ago [-]
I feel like multimodal models that can read images should work differently than they do. My understanding is that multimodal models basically first generate an image embedding and then the model is trained to interpret that embedding, but in the same way that text is lossy, it seems like the embedding would be as well. Why don't multimodal models learn to interpret images themselves without an embedding? Or e.g., by passing some "prompt" to the embedding model?
thfuran 11 hours ago [-]
What does interpreting images mean in practice if you exclude the possibility of feature extraction or any other sort of implicit embedding?
foota 10 hours ago [-]
I'm not an ML expert, but I was thinking of a sort of "guided" embedding. E.g., give the image model some prompt for what it's trying to do? I don't understand why multimodal models generate an embedding that doesn't understand what the model is trying to "figure out".
I think this is similar to how Gemma 4 12B is implemented, but even then I don't think the single layer image embedding is "aware" of the context.
cubefox 12 hours ago [-]
So the models will not only be using more and more Neuralese in their CoT (like GPT-6), but different agents will also be able to communicate with each other in Neuralese. It's not looking good for monitorability.
sparky_twofort 11 hours ago [-]
Is Neuralese in no way decodable into a human-interpretable system? Genuine question -- I don't know the answer.
11 hours ago [-]
Y_Y 11 hours ago [-]
Definitely decodable, that's what's being done now
cubefox 9 hours ago [-]
It's not.
jephs 11 hours ago [-]
This is from like a year ago.
A1Lab-tech 58 minutes ago [-]
[flagged]
paidx 7 hours ago [-]
[flagged]
Transformanshen 10 hours ago [-]
[dead]
lemonleks 6 hours ago [-]
[dead]
Rendered at 07:52:20 GMT+0000 (Coordinated Universal Time) with Vercel.
If multiple models can use cache representations for this kind of enrichment, the KV cache representations of different models must be somewhat compatible.
What stops us then from going a step further, and producing a model family where all models are "KV aligned", and each model can utilize the KV cache of other models directly?
So, an "expensive" reasoning model can use its full faculties to plan, but "delegate" simple subgoals to a smaller model. That smaller model can access the large model's intent directly, as rich KV cache representations - with no prefill recompute and no associated "handover" latency. Or, likewise, a "cheap" small model can generate a diminished but highly compact KV cache that the "expensive" model can then operate on - for example, for skimming a large file for shallow patterns.
Curious to know if anyone is aware of research trying what parent suggested?
https://x.com/RampLabs/status/2042672773747589588
https://news.ycombinator.com/item?id=47195212
I wouldn't have thought to use it for LLM-to-LLM communication, though
I think this is similar to how Gemma 4 12B is implemented, but even then I don't think the single layer image embedding is "aware" of the context.