The really interesting thing to me was that they didn’t need to train this model from scratch they just used their existing MOE checkpoint:
“To convert a decoder-only model (Gemma 4 26B A4B) into a denoiser, we can make use of something it is not directly using when generating tokens, namely the logits of all tokens!”
What makes me hopeful about this release is that possibly this same conversion can be applied to other open models and we might see a bunch of diffusion versions of existing local models. It’s exciting stuff!
jermaustin1 22 minutes ago [-]
I'm very interested in Diffusion text models. The concept of taking noise and adding words starting randomly all over the response, and filling in the noise from there on breaks my brain.
I'm sure I have a fundamental misunderstanding of the technology, though.
Diffusion text models for me is the more interesting type of LLMs for local usage, as it really makes good use of single GPUs for single responses, rather than auto-regressive ones, and is a lot faster! Probably the fastest model I've been able to run so far, ending up doing ~670 tok/s (depending on the type of text) on a Pro 6000
keel-control 20 minutes ago [-]
there's still JEPA to be integrated before AGI.
Would DiffusionGemma be suitable candidate for DFlash 2?
discobot2 24 seconds ago [-]
its more of a competitive approach to improve compute utilisation at lower batch sizes
Rendered at 14:32:47 GMT+0000 (Coordinated Universal Time) with Vercel.
The really interesting thing to me was that they didn’t need to train this model from scratch they just used their existing MOE checkpoint:
“To convert a decoder-only model (Gemma 4 26B A4B) into a denoiser, we can make use of something it is not directly using when generating tokens, namely the logits of all tokens!”
What makes me hopeful about this release is that possibly this same conversion can be applied to other open models and we might see a bunch of diffusion versions of existing local models. It’s exciting stuff!
I'm sure I have a fundamental misunderstanding of the technology, though.
Diffusion text models for me is the more interesting type of LLMs for local usage, as it really makes good use of single GPUs for single responses, rather than auto-regressive ones, and is a lot faster! Probably the fastest model I've been able to run so far, ending up doing ~670 tok/s (depending on the type of text) on a Pro 6000
Would DiffusionGemma be suitable candidate for DFlash 2?