It's a reasonable summary, although I feel there's some fundamentals missing. I wonder how much it will make sense to a platform engineer who doesn't already have a very solid understanding of transformer architecture internals such as the Attention mechanism and its more recent variants. I don't know if you can really get this stuff without having some underlying model internals picture in your head of what the KV cache actually is and how it falls out of repeated attention computations on a continuously growing prefix, to say nothing of understanding KV cache optimizations like PagedAttention which depend very much on the details of how you compute the attention weights with matrix multiplications, softmax, etc. I certainly couldn't make sense of something like "Rather than recompute that history, the engine keeps the attention keys and values for every token in flight" without a pretty solid understanding of how Attention works, which I had to gain from other sources.
This is also a shortcoming of Philip Kiely’s otherwise fantastic book Inference Engineering; it kind of glosses over the transformer architecture details, which you need to get elsewhere. Sebastian Raschka's books are very good for that but you have to then link it all back to how to do efficient inference, which he doesn't talk about so much.
d-ramdass 25 minutes ago [-]
That's fair. We tried to balance level of detail to make it approachable for newcomers to the space as well as practical tips for experienced practitioners. That being said, we'd love to incorporate your feedback in the guide - feel free to contribute to the repo and we'll also add those topics to our list to cover
d-ramdass 3 hours ago [-]
This guide is for platform engineers new to inference. If you need to understand how models and GPUs work together and how to run inference in production, this guide will help. You’ll also learn about the tools to use and how to grow into an inference engineering role. This guide was written by the maintainers of the open source CNCF sandbox project Modelplane
clilog_d 27 minutes ago [-]
[flagged]
lcampbell 1 hours ago [-]
[dead]
Rendered at 21:09:31 GMT+0000 (Coordinated Universal Time) with Vercel.
This is also a shortcoming of Philip Kiely’s otherwise fantastic book Inference Engineering; it kind of glosses over the transformer architecture details, which you need to get elsewhere. Sebastian Raschka's books are very good for that but you have to then link it all back to how to do efficient inference, which he doesn't talk about so much.