the decays are just more capable other models entering the population, making all prior models lose more frequently
cherioo 21 minutes ago [-]
The interesting thing I find is how Anthropic has been more consistently improving over time in the last few years, that allows it to catchup and surpass OpenAI and Google. The latter two have pretty much plateau over the last year or so. GPT 5.5 is somehow not moving the needle at all.
I hope to see the other labs can bring back competition soon!
XCSme 25 seconds ago [-]
Gpt 5.5 is quite a big leap, it's a lot better than opus 4.7 for agentic coding
Thomashuet 19 minutes ago [-]
It seems to be a USA only thing, Chinese models and Mistral don't show any downward trend.
tedsanders 1 hours ago [-]
For what it's worth, I work at OpenAI and I can guarantee you that we don't switch to heavily quantized models or otherwise nerf them when we're under high load. It's true that the product experience can change over time - we're frequently tweaking ChatGPT & Codex with the intention of making them better - but we don't pull any nefarious time-of-day shenanigans or similar. You should get what you pay for.
selcuka 1 hours ago [-]
> we don't switch to heavily quantized models
That sounded like a press bulletin, so just to let you clarify yourself: Does that mean you may switch to lightly quantized models?
jychang 1 hours ago [-]
There's almost 0% chance that OpenAI doesn't quantize the model right off the bat.
I am willing to bet large amounts of money that OpenAI would never release a model served as fully BF16 in the year of our lord 2026. That would be insane operationally. They're almost certainly doing QAT to FP4 for FFN, and a similar or slightly larger quant for attention tensors.
selcuka 1 hours ago [-]
It's ok if they never release a BF16 model, but it's less ok if they release it, win the benchmarks, then quantise it after a few weeks.
Ciph 1 hours ago [-]
Thank you for your answer. I have a similar question as OP, but in regards of the GPT models in MS copilot. My experience is that the response quality is much better when calling the API directly or through the webUI.
I know this might be a question that's impossible for you to answer, but if you can shed any light to this matter, I'd be grateful as I am doing an analysis over what AI solutions that can be suitable for my organisation.
eis 2 hours ago [-]
The Elo rating system measures relative performance to the other models. As the other models improve or rather newer better models enter the list, the Elo score of a given existing model will tend to decrease even though there might be no changes whatsoever to the model or its system prompt.
You can't use Elo scores to measure decay of a models performance in absolute terms. For that you need a fixed harness running over a fixed set of tests.
tedsanders 1 hours ago [-]
FYI, Elo isn't an acronym - it's a person's name. No need to capitalize it as ELO.
andrewshadura 41 seconds ago [-]
Unless you've just missed your last train to London.
Thank you, I just looked at the chart and said to myself: ELO? YOLO!
That Elo ranking is also called chess ranking
refulgentis 1 hours ago [-]
Is this slop? It has wildly aggressive language that agrees with a subset of pop sentiment, re: models being “nerfed”. It promises to reveal this nerfing. Then, it goes on to…provide an innocuous mapping of LM Arena scores that always go up?
Rendered at 07:10:52 GMT+0000 (Coordinated Universal Time) with Vercel.
the decays are just more capable other models entering the population, making all prior models lose more frequently
I hope to see the other labs can bring back competition soon!
That sounded like a press bulletin, so just to let you clarify yourself: Does that mean you may switch to lightly quantized models?
I am willing to bet large amounts of money that OpenAI would never release a model served as fully BF16 in the year of our lord 2026. That would be insane operationally. They're almost certainly doing QAT to FP4 for FFN, and a similar or slightly larger quant for attention tensors.
I know this might be a question that's impossible for you to answer, but if you can shed any light to this matter, I'd be grateful as I am doing an analysis over what AI solutions that can be suitable for my organisation.
You can't use Elo scores to measure decay of a models performance in absolute terms. For that you need a fixed harness running over a fixed set of tests.
Thank you, I just looked at the chart and said to myself: ELO? YOLO!
That Elo ranking is also called chess ranking