NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
▲Inkling-2-Mini in 5.4 GB Memory (github.com)
syntaxing 12 minutes ago [-]
I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.
vineyardmike 9 minutes ago [-]
According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.
syntaxing 8 minutes ago [-]
This has been observed by the broad community for llama cpp.
nicce 7 minutes ago [-]
llama.cpp is one of the slowest these days
saghm 1 minutes ago [-]
What alternatives do people use instead? I hadn't heard of any others (except I guess ollama, which I thought no one used anymore)
plasticchris 10 minutes ago [-]
It’s just not optimized in the same way. A little tweaking on my vllm fork and it runs faster on my 7900xtxes.
woadwarrior01 7 minutes ago [-]
The NV in NVFP4 stands for NVIDIA. :)
manlymuppet 42 minutes ago [-]
My Quantitive!
22 minutes ago [-]
an0malous 56 minutes ago [-]
Our future hinges on the capability gap between frontier lab models and open source local models
singpolyma3 29 minutes ago [-]
What gap?
rs_rs_rs_rs_rs 8 minutes ago [-]
The gap is so big you can't see it.
agentmaxxing 35 minutes ago [-]
yep came here to say this
zulux 15 minutes ago [-]
Weirdly, I use lab models to make the local models useful. Nothing like getting Claude to spin up the system that will put it out of work.
Dwedit 2 hours ago [-]
How does this method compare to the use of an importance matrix?
aktenlage 1 hours ago [-]
Probably better but more effort
woadwarrior01 60 minutes ago [-]
From the model card:

> The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.

What's the point in quantizing a QAT model? Just for social media karma?

gavmor 47 minutes ago [-]
Just to reiterate the question: QAT approximates quantization-errors in full-size fine-tuning, so that the model itself is insensitive to (or accommodating of, I'm not sure) them when quantized to the corresponding format.

Quantization to a non-corresponding format promises none of those accommodations.

asaddhamani 20 minutes ago [-]
QAT models are trained to quantize well? I don’t get the concern here.
woadwarrior01 9 minutes ago [-]
Please read their blog post. This isn't a run of the mill QAT post trained model (like Gemma 4 QAT). It was natively trained in NVFP4, through and through.

> Most of Inkling 2 Mini’s weights were trained and are deployed natively in NVFP4, with the last four layers in MXFP8.

https://thinkingmachines.ai/news/inkling-2-mini/

Prospernwo 21 minutes ago [-]
[flagged]
cheapcheerful 54 minutes ago [-]
"Look at my quant 2"

/igotthatreference.jpg

nico 50 minutes ago [-]
Yeah, I enjoyed the reference, though I didn't really understand it until I saw the picture on https://lookatmyquant.pages.dev
hugmynutus 18 minutes ago [-]
*TL;DR* Promising initial results but shoddy research. OP's quant did not get 1st or 2nd prize in that math competition.

----

Gave it a quick browse to verify what my local clanker told me, it seemed to check out

I believe it is Dynamically allocating weights based on fisher information of conversation traces & For some K-quant look up table stuff it can store in an importance matrix to get better compression.

The results are complete nonsense and really don't back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth's UD-IQ1_S: 0.0895 vs 0.0808 on English/chat/code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt'd model on sample inputs (AFAIT).

According to anthropomorphic-clud their results for GSM8K & HumanEval are within the noise regime and vastly over stated.

IronWolve 1 hours ago [-]
Only mx version?
Prospernwo 21 minutes ago [-]
MX/Apple Silicon only for the released Inkling files, yes. The ht2/ht3 codes are hashed block-vector trellis codes meant to fit 4 KB of SRAM, decoded by custom Metal kernels, and the drafter is MLX-only too - none of that runs in llama.cpp today.

The GGUF route is in the repo (gguf_compiler.py emits plans over llama.cpp's own tensor types and the docs say the result runs in llama.cpp as-is), so porting these plans is clearly intended, but right now a 4090 gets you the Qwen files to read about rather than run.

Pragmata 9 minutes ago [-]
> Don't post generated text or AI-edited text. HN is for conversation between humans.

Please [review the commenting guidelines](https://news.ycombinator.com/newsguidelines.html)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 20:06:34 GMT+0000 (Coordinated Universal Time) with Vercel.