Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically
throwa356262 34 minutes ago [-]
"Can't" is not really correct.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
ReptileMan 30 minutes ago [-]
>The two numbers that decide everything
Do you have LLM paranoia? The moment you read something in this style suddenly your brain starts doubting whether you are consuming human generated content.
cheevly 23 minutes ago [-]
Maybe like… focus on the actual content instead of perceived writing patterns? Crazy I know.
glimshe 21 minutes ago [-]
This. I don't particularly like the LLM writing style, but we've read a ton of very poorly written texts over the year with no complaints. LLMs are average writers with an annoying style, but not bad writers. If the content is good, I don't care if a LLM wrote it.
jazzyjackson 18 minutes ago [-]
It’s a flag that no care went into it. Web surfing involves lots of little decisions following cues of “is this worth my time”
lukeinator42 18 minutes ago [-]
The writing style is this staccato LLM-like style that is difficult to read as it has zero flow and meaningless sentences.
Like what does the second sentence even mean? Is it even a sentence? "The roofline math, the prompt-processing catch, the NPU red herring, and the owner-measured speeds."
LoganDark 1 hours ago [-]
Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
mountainriver 56 minutes ago [-]
Yeah this is such a funny thing going around. Try to run or train a small/medium sized model on a MacBook. It doesn’t go very good compared to a dedicated gpu
This is likely the right path in the future but it isn’t there yet today
bahmboo 55 minutes ago [-]
To be fair it's "only" half the throughput of a 4090 and a third of an RTX 6000. Significant but not an order of magnitude.
OutOfHere 58 minutes ago [-]
Let's also ensure the SSD doesn't age prematurely.
craftkiller 53 minutes ago [-]
I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.
giantrobot 7 minutes ago [-]
It is and it doesn't. You only get into disk writes if the system starts paging out to disk.
Rendered at 15:47:04 GMT+0000 (Coordinated Universal Time) with Vercel.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
Do you have LLM paranoia? The moment you read something in this style suddenly your brain starts doubting whether you are consuming human generated content.
Like what does the second sentence even mean? Is it even a sentence? "The roofline math, the prompt-processing catch, the NPU red herring, and the owner-measured speeds."
This is likely the right path in the future but it isn’t there yet today