Companies need to set up instrumentation to measure all the aspects of AI-related consumption, he says, starting with the number of tokens used, who’s using them, and how it correlates to work output.

“That measurement is fundamentally lacking,” he says.

At least when humans are using AI chatbots, there’s a limit to how many questions they’re physically able to ask, combined with predictable subscription costs. And when business processes are AI-enabled via RAG, the API calls to LLMs are being made by predictable, traditionally-scripted business systems.

But now, agentic AI is making everything worse because the agents can act unpredictably, and the number of API calls can quickly spiral out of control. In a report by the Boston Consulting Group, two-thirds of companies are reporting uncontrollable AI scaling expenses.

Another cost some companies might not anticipate well, or not track because it’s part of a different budget, is data-related cost. Whether preparing data for training or fine-tuning, using RAG embeddings, or setting up direct MCP access via agents, these costs can quickly add up when AI comes into the picture.

“Egress fees are one of the big ones,” says Tom Coughlin, IEEE fellow and president of consulting firm Coughlin Associates. “If you have to bring data out of the cloud, those egress fees could be considerable.”

Then there are all the human costs of deploying AI, he adds.

“There’ll be a lot of value that people get out of AI in the long run, but they need to know how to use it properly,” he says. “If they don’t have those skills, you’ll be at a disadvantage.”

Solutions and mixed messages

Then there’s fixing problems. A majority of companies have had at least one AI-related incident in the last 18 months, with most resulting in financial loss, some over $500,000. Then there’s the AI that’s being embedded in everything.

“We know our direct costs,” says Andrew Johnson, CIO at Brownstein Hyatt Farber Schreck, a leading national law firm. “But where it becomes more difficult to measure is with platforms we already have in place, and SaaS applications that didn’t have AI capabilities,” he says. “They’re asking for extraordinary increases and attribute them to new capabilities due to AI. How much should be ascribed to AI? That’s a little wishy-washy.”

Even when AI saves money, there are often extra costs associated with that. For example, the firm was spending about $70,000 a year on a contract management platform. Building their own version with AI took about $40,000 in labor costs and another $3,000 a year for hosting. Ongoing maintenance will be minor for that particular application, he adds, totaling another couple of thousand a year.

But there are also other indirect costs that come with running your own applications, including security audits, vulnerability assessments, penetration tests, and code review.

“The more complex and riskier the platform, the less appetite there is for trying to create an in-house solution,” he says.

Still, the software development team is now dramatically more productive as a result of AI, with four or five developers able to do the work of 20 or 30.

But the productivity improvements don’t translate to labor savings, since there’s plenty of new work for the developers to do. “We have an enormous backlog of opportunities to develop solutions,” he says.

The tendency of work to expand to fill the time available isn’t just true for software development, says Carnegie Mellon’s Rao.

Say for example, AI is expected to lead to a 20% improvement in productivity, he says. “There were a hundred people doing it, and now we only need 80.” But at the end of the year, headcount hasn’t changed. “The tasks they were doing, there’s improvement,” he adds “But humans will add tasks to supplement or complement that 20%. It’s not that they’re going home an hour early, but they’re finding other value-generating activities.”

In fact, in some cases, increased productivity at a company can actually hurt the bottom line. Lawyers, for example, bill by the hour.

“Efficiency runs counter to our traditional ways of making money,” says Brownstein’s Johnson. “We have to think past that. It’s not detrimental to our long-term interest, but it’s a challenge in the short term. If we don’t do this, though, it’s likely we won’t be competitive in the mid- to long-term.”

So if a new AI tool helps an attorney with due diligence, there’s no straight line between the investment in that tool and increased revenues.

“It’s a given that it’s directionally right,” Johnson says. “But we can’t say it’s going to lead to a particular return.”