Over the last year, I’ve watched organisations rush to measure AI adoption. As with every major technology shift, the instinct is understandable: leaders want evidence that the investment is working.
The problem is that many teams are measuring the easiest things to count rather than the things that matter.
Tokens consumed. Prompts submitted. Agent executions. AI-generated lines of code.
These metrics tell us that AI is being used. They tell us almost nothing about whether outcomes are improving.
The recent DORA discussion around “tokenmaxxing” (which you should definitely open in a new tab) resonated with me because we’ve seen this movie before…
Our industry has a long history of becoming obsessed with activity metrics. First it was lines of code. Then story points. Then velocity. Now, in some organisations, it’s token consumption.
The metric changes. The mistake stays the same.
AI Changes How We Build; Not What Success Looks Like
One of the most common mistakes I see is the assumption that AI somehow changes the definition of successful product delivery.
It doesn’t.
Customers still don’t care how many prompts were written, or how many agents were invoked, or how many tokens were burned to produce the solution (unless they really do!).
They care that products solve problems, arrive quickly, and work reliably.
The outcomes haven’t changed. The constraints have.
For decades, software teams have been limited by the speed at which they could create, test, and refine ideas. AI has dramatically changed that equation. Many of the tasks that once consumed hours or days can now be completed in minutes. Work that previously required specialist expertise can often be accelerated through someone with a passing knowledge, collaborating with AI and cross-checking outputs - if the guardrails are good enough!
But faster execution is not the goal.
Faster learning is.
If AI allows a team to write software twice as fast, but they continue building the wrong thing, the organisation hasn’t become twice as effective. It’s simply become more efficient at producing waste.
The Four Metrics I Care About
Rather than proposing an entirely new set of AI metrics, I believe we should double down on the metrics that have always mattered:
1. Lead Time
How long does it take to move an idea from brain-fart or idea to Production?
This remains one of the clearest measures of delivery effectiveness.
If AI is genuinely improving the product development lifecycle, lead times should fall. Teams should be able to move from idea to customer feedback more quickly, with fewer bottlenecks and less waiting.
Not because they wrote more code.
Because they removed friction.
SIDE NOTE: If this metric isn’t already being measured in your organisation, you should start. It’s one of the most powerful indicators of delivery health, and it can be measured without any AI-specific instrumentation. If you don’t have a way to measure lead time, you can’t know whether AI is helping or hurting! …and if this measure IS being measured, but it’s “not good” (for whatever value of “not good” applies to you), then AI isn’t going to help; fix your lead-time problems first!
2. Change Failure Rate
Are we shipping safely?
One of the risks in the current AI conversation is that speed becomes the headline metric while quality quietly deteriorates in the background. A team that doubles its release frequency while tripling its production incidents has not improved.
AI should help teams move faster without increasing operational risk.
If it doesn’t, we should be asking some really hard questions.
3. Defect Escape Rate
Where are defects being discovered?
The real promise of AI isn’t that it helps developers generate code. It’s that it can assist throughout the entire delivery lifecycle, from requirements and design, through development and testing, including documentation and validation/attestation, and even operations.
If those capabilities are working, fewer defects should reach customers.
If defect escape rates remain unchanged while AI usage skyrockets, I’d wager that you’re simply accelerating existing delivery patterns rather than improving them. Playing “single-player-mode” if you will…
4. Time-to-Learning
This is the one metric I think deserves far more attention in our AI era.
How quickly can a team validate or invalidate an idea?
Historically, product development has been constrained by the cost of experimenting. Building prototypes took time. Research took time. Analysis took time. AI reduces so many of those costs!
The organisations that benefit most won’t necessarily be the ones generating the most software. They’ll be the ones learning, iterating, and adapting the fastest.
The ability to rapidly test assumptions, gather feedback, and adjust direction may ultimately prove more valuable than any efficiency gain measured in hours or burnt tokens.
Measure Outcomes, Not Activity
None of this means that AI “activity metrics” are useless.
Token consumption, prompt counts, agent usage, and similar signals can be useful diagnostic indicators. They can help us understand adoption patterns and identify where teams may need additional support. They might be able to help us highlight the “single-players” inside our organisations, who are using AI to accelerate their own work but not necessarily contributing to team learning or product improvement. They can also help us identify where AI is being used in ways that are not aligned with organisational goals - including where it may be introducing risk or inefficiency. But they should not become the definition of success!
The goal isn’t AI adoption. The goal is better product delivery.
AI is a capability, a tool, an enabler; not an outcome.
As organisations continue investing in AI-accelerated product development, I think we need to resist the temptation to create a new generation of vanity metrics. We’ve already learned that activity is a poor proxy for impact.
The most successful teams won’t be the ones consuming the most tokens. They’ll be the ones delivering value faster, learning faster, and maintaining quality while doing both, so that’s what I’d measure.
…And, just as importantly, that’s what I’d optimise for: getting solutions into customers’ hands faster, learning from their feedback, and iterating to improve outcomes.