I just did a run with a benchmark I just used to test other models against. (It's about detecting irony in german soccer tweets). On my M5 Pro with 48GB it took over 30min to decide on just 100 tweets, the thinking definitely takes long.
It performed quite below Jev, but above other open decision models I tested (68 correct vs 79 correct for Jev - see [1]). I'm running it for the moderation benchmark as well, but that will probably take a few hours on my machine.
What is the point of this, if it is p90 17 seconds? Might as well use an LLM. The beauty of Jev is that it is dirt cheap and insanely fast.
zihotki 37 minutes ago [-]
I would hold your horses to paint it as dirt cheap.. In my cases for spam detection Luna was 20% cheaper due to prompt caching, although not as fast.
olgava 8 minutes ago [-]
[dead]
esafak 34 minutes ago [-]
Jev ought to offer a flex mode that uses their spare capacity for a discount.
mabini 31 minutes ago [-]
[dead]
thm 1 hours ago [-]
Ask Jeeves - Only took us 30 years to come full circle.
tmnstr85 1 hours ago [-]
this was the comment i came here for
onaclov2000 41 minutes ago [-]
My bots are all named Jeeves lol. I have a CLI tool I use that connects up to a LLM I made and I call it Jeeves too ...so funny. I really didn't use Jeeves all that much I tended to use...I think it was called Web crawler pre-google era
RamblingCTO 1 hours ago [-]
Super dope. If it would ship as prod ready code supporting mps as well that would be even doper.
But funny that jev is getting its lunch eaten apparently in under two weeks?
pavlov 1 hours ago [-]
It’s ok, one week of AI hype is now enough to close a billion-dollar term sheet with VCs.
swader999 2 hours ago [-]
Seems like this is the way, a hybrid approach where some of the pipeline will be jev like and some traditional LLM depending on the nature of the work.
zerop 2 hours ago [-]
Are there "good" Open source Decision models built on Gemma-4 and also trainiable on own data?
what about benchmark against smaller or bigger models? 9B looks too small for llm-level decisions.
woadwarrior01 2 hours ago [-]
This isn't really surprising. LLM reasoning and before that, chain of thought prompting are essentially forms of test-time compute scaling.
alienbaby 2 hours ago [-]
Just curious, where has this term 'noul' come from for yes/no ansers?
/a bit more digging and..
A Noul performs a Bernoulli trial—an experiment with exactly two outcomes (yes or no)—but instead of picking one, it returns the calibrated probability (ranging from 0.0 to 1.0) that the statement is true.
I hate it :)
LudwigNagasena 33 minutes ago [-]
In Bayesian statistics that’s called credence. Weird that they felt the need to invent a new term.
doginasuit 1 hours ago [-]
I like it. It is short and distinct which is a good fit for a primitive. It describes its fundamental meaning and draws a connotation with Boolean.
k__ 1 hours ago [-]
The whole "no hallucinations" premise is based on that.
Like, yeah, you don't hallucinate, but only because you force the user to decide in the end.
kjs3 1 hours ago [-]
force the user to decide in the end
And that's...bad?
doginasuit 1 hours ago [-]
That seems like the only possible way to eliminate hallucination, short of a model that is never wrong.
rusk 1 hours ago [-]
Wait til you hear about how digital circuits work at die level
keepitwiel 2 hours ago [-]
Bernoulli
user3939382 2 hours ago [-]
If you want to get super pedantic about what’s happening in a transistor every digital Boolean is actually this
kevindamm 2 hours ago [-]
Not quite.. that boolean is about whether the voltage exceeds some threshold. It's not about how close the voltage is to the circuit's maximum possible threshold, or how much it exceeds the threshold.
In an analog circuit, maybe.
mxkuzn 1 hours ago [-]
interesting bench list, what about benchmark against smaller or bigger models? 9B looks too huge for small like laya, and too small for llm-level decisions.
captainbland 1 hours ago [-]
See if it can beat Jev's Pokémon benchmark
mxkuzn 30 minutes ago [-]
what benchmark this one or it's just fun?
raverbashing 2 hours ago [-]
Jeeves, that's a name I haven't heard in a long time...
gizajob 1 hours ago [-]
Personally I’m happy that after a 30 year effort and hundreds of billions spent, AskJeeves finally works as intended.
fishfasell 2 hours ago [-]
If Jeeves returned as an AI chat bot it would be the most brilliant resurgence of nostalgia
grokkedit 2 hours ago [-]
jeeves is currently the name of my local hosted assistant, in its context there are rules that tell it to behave like good old jeeves.
soon I'll make sure that my home assistant pod answers to "Hey jeeves"
kjs3 1 hours ago [-]
We locked him in the basement with Clippy, Bob and BonziBuddy. Who opened the damn basement door???
lherron 45 minutes ago [-]
…a long time.
esafak 29 minutes ago [-]
Jev-like models give calibrated decision probabilities, but at low accuracy.
So why didn't they show both??
AnodicElegy 56 minutes ago [-]
I'm surprised we haven't seen a "Jehovah" yet.
jadar 53 minutes ago [-]
With the amount of talk about "inventing god", I'm surprised too.
phplovesong 1 hours ago [-]
So "askjeeves" has been resurrected?
hjun1052 2 hours ago [-]
[dead]
Rendered at 13:52:50 GMT+0000 (Coordinated Universal Time) with Vercel.
It performed quite below Jev, but above other open decision models I tested (68 correct vs 79 correct for Jev - see [1]). I'm running it for the moderation benchmark as well, but that will probably take a few hours on my machine.
[1] https://tn1ck.com/blog/jevdit
But funny that jev is getting its lunch eaten apparently in under two weeks?
/a bit more digging and..
A Noul performs a Bernoulli trial—an experiment with exactly two outcomes (yes or no)—but instead of picking one, it returns the calibrated probability (ranging from 0.0 to 1.0) that the statement is true.
I hate it :)
Like, yeah, you don't hallucinate, but only because you force the user to decide in the end.
And that's...bad?
In an analog circuit, maybe.
soon I'll make sure that my home assistant pod answers to "Hey jeeves"
So why didn't they show both??