11 quotes from AI researchers about benchmarks, models, and evaluation
"Engineers, they are competing to consume the most AI measured by tokens. It is almost like a sport at this point. Jensen Huang, CEO of Nvidia, he said that he would be alarmed if a top engineer was not burning 250K a year in AI compute. Shopify told me that they use it as a performance signal. And Meta employees, reportedly they blew through an estimated 900 million tokens in a month."
"Look at two of the biggest AI labs. OpenAI is making AI cheaper, easier to use, so more people consume it. It needs the usage numbers to justify spending. Anthropic, meanwhile, putting limits on how much and making people pay for it, maybe because it wants to know the demand it is seeing is real."
"Across Ramp data, token and AI spend has grown by 13 times over the past year, 50% a quarter. And what is very clear is no one knows how to budget for this."
"There was a lot of discussion even on X a few days ago about incentives at Meta. And I think people are pointing out this idea of Goodhart law which says a measure of performance that becomes a goal ceases to become a good measure of performance. Once you incentivize use as many tokens, you will see engineers go and count all of the numbers of prime or all the digits of pi and use these tokens and it goes on and on."
"We separated out across over 50,000 businesses the bottom quartile of spenders on AI and the top quartile. The bottom quartile over the past 3 years their revenue grew by about 12%. The top quartile more than doubled. And the rate of growth has grown by each year successively."
"The frontier models have gone to I think over 20% of the share of tokens used to 4%. I think that is a harbinger of what is going to come. It is very interesting seeing this development coming out of China. It is a culture that is very attuned to efficiency."
"The interesting question comes down to the classic question of business, the innovator dilemma. If your business model is predicated on extracting the maximum amount of spend, do you want to do it? And are you willing to make the sacrifice? Maybe they will or maybe they will not. I think it sets the stage and opening for a great third party to go in and keep things in check."
"OpenAI is cutting prices to try to get customers to them. Meanwhile, Anthropic is actually raising prices and cutting people off because they have so much demand that they are trying to keep people from using too much of it. And to me, that shows a company that is feeling the pressure between Anthropic and Google."
"If you look at OpenRouter, in the two months prior to the end of January, token growth was up about 20% or so. In the two months after agentic AI got formalized, that token growth has grown about 130%."
"When you go to agents and you have got to do a whole bunch of different things, all of a sudden you go from 7 gigawatts of GPUs to just one gigawatt of CPUs. That ratio goes to maybe like 4 to 1."