One month I could use Github Copilot fully with no disruptions. The next month, after pricing changes, I’ve run out of tokens in two days.
Such drastic changes tell me that pricing of tokens is arbitrary, and AI business is running out of money fast.
lucaspiller 4 hours ago [-]
I think it's more a consequence of pushing for the biggest valuation/IPO. Rumoured profits on inference are north of 70%.
Taking SpaceX as an example, they have increased prices across all their consumer products over the past six months. But they definitely aren't short on money with Alphabet and Anthropic combined paying them over $2 billion per month.
Microsoft/GitHub lost out here as they were just repacking other people's products.
jurgenburgen 2 hours ago [-]
> Rumoured profits on inference are north of 70%.
Rumors are worth squat when they’re most likely put in motion by the people with a vested interest in this industry.
Let’s talk about profits when there’s real data from the IPO documentation.
NitpickLawyer 1 hours ago [-]
> Rumors are worth squat
You can make some educated guesses and find out some limits on inferencing cost by looking at 3rd party providers on platforms like openrouter. You can get some median cost /tok for a given model size. Then make some educated guesses on SotA model sizes, and you can get an estimate on pure cost of serving a model. Error bars and all that, of course. But still a range, with some limits.
lefra 3 hours ago [-]
Inference can only happen after having invested in training and datacenter construction. Arguing about "inference profitability" sounds a lot to me like ignoring large cost centers of these comanies.
4 hours ago [-]
bob1029 1 hours ago [-]
> Furthermore, we observe that input tokens consistently constitute the largest share of consumption for an average of 53.9%
I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code.
I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with the agent or the codebase is new / empty.
zozbot234 47 minutes ago [-]
If input tokens dominate the cost to that extent, this implies that major gains are possible by making better use of caching. You could basically ask the model to do a one-time "compaction" step including a dump of the relevant portions of the code, and use that as the cached prefix for a large amount of "swarm" subagent calls.
kolinko 38 minutes ago [-]
Did you experiment with giving agent better tools to navigate and document the codebase? Asts, language servers and so on?
A million tokens (not cached) sounds like a lot.
bob1029 29 minutes ago [-]
The target codebase is very large. A million tokens is a drop in the proverbial bucket.
I still don't understand how caching helps me very much. I must be misunderstanding it because I thought the user's prompt (which is the biggest variable) necessarily sits prior to all of these token intensive tool calls. How can we cache the reading of codebase if the prefix is always moving?
gmerc 3 hours ago [-]
It’s just like Airline reward miles and offers no benefit to companies over just renting bare metal GPU time
emsign 3 hours ago [-]
I hope this horrible time will soon be over when cheaper NPUs come available from more hardware companies, and also when model size get optimized down further.
I wonder what hyperscaled compute farms and models will be good for at that running cost when most AI needs can be fulfilled by on-prem and on-device hardware and models. Probably only customer left are big governments. So in the end the tax payer has to pay for those billions of investments by the AI cartel.
zozbot234 3 hours ago [-]
The typical NPU is only marginally helpful for on-prem inference. A GPU can read quantized data from main memory and dequantize/pad it locally (making effective use of memory throughput); a NPU often needs to read padded data directly from memory, which is wasteful. So it only helps a little bit wrt. prefill.
Also, smaller models can obviously be used but a smaller model will be a lot weaker in real-world knowledge and this tends to limit their smarts in a way that can't be compensated by more thinking.
SubiculumCode 6 hours ago [-]
Reminded me of this paper from last year trying to optimize efficient token usage providing budget guidance information. [1]
One thing I've noticed using agents for coding is that they really like to write thousands of unit tests but not dynamically test.
drivebyhooting 7 hours ago [-]
And they like to burn a ton of tokens writing and debugging tests that are semantically corrupt.
esperent 4 hours ago [-]
Unit tests are a type of dynamic testing. As opposed to static testing which is linting/typechecking etc.
If you want a difference kind of dynamic testing besides unit tests, have you tried writing it in as a requirement during the planning/PRD phase?
gib444 6 hours ago [-]
And AWS heavily pushes a complex lambda solution stringing together as many chargeable AWS services as possible for a simple requirement
Their interests are often not your interests. In this case they want you to unnecessary money on useless work (let's stop the euphemism of "tokens" btw)
make3 7 hours ago [-]
you can just tell them to do more dynamic testing. I think dynamic testing is partly frowned upon because it slows things down & can take down software where you wouldn't expect
becomevocal 3 hours ago [-]
First thought was "only 30 tasks" however the findings map to what I've seen personally: code review consumes majority of tokens
zozbot234 3 hours ago [-]
Code review could also be run as an unattended/batched task though, possibly with at least some use of on-prem inference (which excels at this). That would be a major saving compared to the usual cloud inference scenario.
drivebyhooting 7 hours ago [-]
In the past Google et al would hire engineers based on how well they could optimize the infrastructure.
Maybe soon companies will look at how engineers can optimize the token efficiency of AI.
Retric 7 hours ago [-]
That assumes Tokens will remain a meaningful expense. I’m not sure developers will find uses for ever more tokens nearly as quickly as the prices fall.
ares623 5 hours ago [-]
How are we so confident that prices will fall? Isn't the exact opposite happening, right now, during arguably the most critical part of this whole saga (pre-IPO to make things appear as beautiful and as not-obviously-illegal as possible)? And the only reason they were "falling" previously was for hyper growth.
jpatt 4 hours ago [-]
The Growth aspect mentioned is that VCs are subsidizing the bill right now, so it is hard to know if at the current moment the demand curve would promote as much usage without it, but assuming demand remained constant (not even growing), you could expect token prices to be competed down. It is a commodity without a moat.
Now that we have pretty decent open source models, anyone can create a new business to supply more tokens. Sure there’s short term scarcity: energy, GPUs, cooling, but this is a scale up problem. More token demand = more data center build = more energy plant build. This downward pressure will also keep frontier private model prices in check.
Differentiation seems to be happening at the harness level, whereby we can expect token spend to be a metric to compete on and drive down for the customer (at least hoping tools in the application space don’t continue token based billing as their primary revenue stream).
These are not short term hyper growth forces, but a fundamental alignment of incentives.
fc417fc802 3 hours ago [-]
In the one direction the hardware continues to improve, new buildouts continue to come online, and methods for improving the parameter efficiency of models continue to be discovered.
In the other direction models continue to grow larger, new customers continue to arrive, and existing customers continue to find ever more creative ways to burn large quantities of tokens as the prices fall.
I doubt anyone can say with certainty where the equilibrium will be 1 or 5 years from now largely because (among many other things) it's impossible to predict how much of the current economy AI will end up eating. In general though the third party providers of open weights models are probably the most reliable data source available since they have little to no incentive to subsidize usage.
oersted 2 hours ago [-]
Prices have fallen dramatically over the last few years. It’s just that our standards have increased because we are using AI in ways that were not possible with worse models. But for the same level of “intelligence” as we had a couple years ago, the prices are so much lower.
Retric 4 hours ago [-]
I don’t think we can extrapolate from current API pricing, but dramatically improving hardware in terms of cost:performance is the underlying reality.
Betting against that you need to assume exponentially more expensive models every year.
mobelkh 4 hours ago [-]
it is falling if you look elsewhere, deepseek made their 75% discount on their V4 models permanent, on one hand there's LLM improvements that make inference cheaper (e.i. MoE, hybrid attention), on the other hand we're getting more inference focused chips that break the nvidia monopoly.
i don't think a lot of people know this, but a cluster of GPUs can serve multiple clients without much of a drop in performance, e.i. worst case scenario you band together with 6-16 people to run a 2-3 H100 server to host deepseek V4 Flash or 4-6 to run Pro, and you're getting the same performance as if you ran it alone, this means a lot of companies can afford throwing 50-100k into their own LLM server cluster.
We're at a price point where if you push it further people will move, there's no real vendor lock in, your agent config, skills, MCP servers etc are all reusable with other models and harnesses, so unless you get all providers to collude on a price hike, you risk an exodus of customers
dnlosx 6 hours ago [-]
[dead]
deadbabe 56 minutes ago [-]
I know how to drop a company’s token costs to zero: treat tokens as a utility same as internet and make engineers pay for it.
senectus1 6 hours ago [-]
amusing side note:
Was in a meeting reviewing a potential new product, it was going well until they showed us that they had added AI to it (of course they have). It was pretty obviously just shoehorned in, and one part of that obviousness was that they had a column that showed how many tokens it took to make each query.
I asked who is paying for the tokens, they said its included in the license. I said, so is there a budget or is it all you can eat.
they said good question they didnt know and would get back to me. I said the reason i asked was just one query there had a 250k token burn on it. and it was a fairly simple query about one device.
then, one of the execs on their side was heard saying out loud "Why are we even showing this to the customers?"
it have us quite a chuckle.
But lesson learned... the cost of adding AI to anything isnt really being accounted for let alone the true cost of actually running the AI.
all things AI are going to get more expensive. even if you dont want the AI aspect.
prymitive 3 hours ago [-]
AIshittification
satvikpendem 6 hours ago [-]
Tokenomics is already a word used to describe cryptocurrency economics, not sure why they'd try to redefine it for AI even if a different sort of token is used.
2 hours ago [-]
NewJazz 6 hours ago [-]
New fad. Forget about the old fad. This one will be old soon, you better get on board before its too late!
dkersten 2 hours ago [-]
Crypto was already a term before cryptocurrencies made it about them. Web 3.0 was already a thing before crypto bros made web 3 about cryptocurrencies.
So what? Terms are reused in different contexts all the time. And most people have moved on from cryptocurrencies anyway, so there’s little chance it’ll confuse anyone.
emsign 3 hours ago [-]
At its current iteration the AI tech market is not economically sustainable, not for the other markets outside the AI economy, and most deadly not even for the main target customers or AI tech companies themselves. There have been several news of companies having overspent their token budget month after month. The hardware monopolist and his network of buddy companies can determine the token price as freely as they want, there are no competitors, their only "competitor" is when people stop using AI alltogether.
eddysir 2 hours ago [-]
[flagged]
winphoto 6 hours ago [-]
[flagged]
baarse 6 hours ago [-]
[dead]
andrewvu0203 6 hours ago [-]
[dead]
bonigv 6 hours ago [-]
[dead]
Waffle2180 7 hours ago [-]
[flagged]
4 hours ago [-]
Rendered at 10:07:34 GMT+0000 (Coordinated Universal Time) with Vercel.
Such drastic changes tell me that pricing of tokens is arbitrary, and AI business is running out of money fast.
Taking SpaceX as an example, they have increased prices across all their consumer products over the past six months. But they definitely aren't short on money with Alphabet and Anthropic combined paying them over $2 billion per month.
Microsoft/GitHub lost out here as they were just repacking other people's products.
Rumors are worth squat when they’re most likely put in motion by the people with a vested interest in this industry.
Let’s talk about profits when there’s real data from the IPO documentation.
You can make some educated guesses and find out some limits on inferencing cost by looking at 3rd party providers on platforms like openrouter. You can get some median cost /tok for a given model size. Then make some educated guesses on SotA model sizes, and you can get an estimate on pure cost of serving a model. Error bars and all that, of course. But still a range, with some limits.
I'm seeing a ratio of around 10:1 in my usage. A vast majority of the tokens consumed are on the input side. The agent will often read a million tokens just to patch one line of code.
I think if you are seeing something closer to 1:1 or more on the output side, there is either a problem with the agent or the codebase is new / empty.
A million tokens (not cached) sounds like a lot.
I still don't understand how caching helps me very much. I must be misunderstanding it because I thought the user's prompt (which is the biggest variable) necessarily sits prior to all of these token intensive tool calls. How can we cache the reading of codebase if the prefix is always moving?
I wonder what hyperscaled compute farms and models will be good for at that running cost when most AI needs can be fulfilled by on-prem and on-device hardware and models. Probably only customer left are big governments. So in the end the tax payer has to pay for those billions of investments by the AI cartel.
Also, smaller models can obviously be used but a smaller model will be a lot weaker in real-world knowledge and this tends to limit their smarts in a way that can't be compensated by more thinking.
[1] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Stee...
If you want a difference kind of dynamic testing besides unit tests, have you tried writing it in as a requirement during the planning/PRD phase?
Their interests are often not your interests. In this case they want you to unnecessary money on useless work (let's stop the euphemism of "tokens" btw)
Maybe soon companies will look at how engineers can optimize the token efficiency of AI.
Now that we have pretty decent open source models, anyone can create a new business to supply more tokens. Sure there’s short term scarcity: energy, GPUs, cooling, but this is a scale up problem. More token demand = more data center build = more energy plant build. This downward pressure will also keep frontier private model prices in check.
Differentiation seems to be happening at the harness level, whereby we can expect token spend to be a metric to compete on and drive down for the customer (at least hoping tools in the application space don’t continue token based billing as their primary revenue stream).
These are not short term hyper growth forces, but a fundamental alignment of incentives.
In the other direction models continue to grow larger, new customers continue to arrive, and existing customers continue to find ever more creative ways to burn large quantities of tokens as the prices fall.
I doubt anyone can say with certainty where the equilibrium will be 1 or 5 years from now largely because (among many other things) it's impossible to predict how much of the current economy AI will end up eating. In general though the third party providers of open weights models are probably the most reliable data source available since they have little to no incentive to subsidize usage.
Betting against that you need to assume exponentially more expensive models every year.
i don't think a lot of people know this, but a cluster of GPUs can serve multiple clients without much of a drop in performance, e.i. worst case scenario you band together with 6-16 people to run a 2-3 H100 server to host deepseek V4 Flash or 4-6 to run Pro, and you're getting the same performance as if you ran it alone, this means a lot of companies can afford throwing 50-100k into their own LLM server cluster.
We're at a price point where if you push it further people will move, there's no real vendor lock in, your agent config, skills, MCP servers etc are all reusable with other models and harnesses, so unless you get all providers to collude on a price hike, you risk an exodus of customers
Was in a meeting reviewing a potential new product, it was going well until they showed us that they had added AI to it (of course they have). It was pretty obviously just shoehorned in, and one part of that obviousness was that they had a column that showed how many tokens it took to make each query.
I asked who is paying for the tokens, they said its included in the license. I said, so is there a budget or is it all you can eat. they said good question they didnt know and would get back to me. I said the reason i asked was just one query there had a 250k token burn on it. and it was a fairly simple query about one device.
then, one of the execs on their side was heard saying out loud "Why are we even showing this to the customers?"
it have us quite a chuckle. But lesson learned... the cost of adding AI to anything isnt really being accounted for let alone the true cost of actually running the AI.
all things AI are going to get more expensive. even if you dont want the AI aspect.
So what? Terms are reused in different contexts all the time. And most people have moved on from cryptocurrencies anyway, so there’s little chance it’ll confuse anyone.