> The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.

What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right

Extreme speeds are actually interesting if you think about it. We are apparently getting some interesting results from LLMs nowadays. If they would put 100x effort in 10% the time it takes, or imagine 1% where how long tool calls take become the bottleneck, that would actually probably feel like a really special kind of "super intelligence".

Ok I'm sold, where to buy? :D

At 20 kW average consumption per WSE-3T wafer this gives roughly 1.18 MW for the whole system. Assuming 1500 tokens per second and 190 concurrent requests this gives 4.1 J/token. Which is good I suppose for the size of the model?

[dead]