Andi Smith

Technical Leader Product Focused AI Consultant

Measure Shipped Work, Not Tokens

  • By Andi Smith
  • 3 minute read

If everyone on your team already uses AI every day, what does your AI adoption metric actually tell you?

Nothing, is the honest answer. And yet usage is still the number most companies are chasing. Earlier this year, Meta ranked its 85,000 employees by token consumption on an internal leaderboard. The winner earned the title of Token Legend after burning 281 billion tokens in a month. Uber blew through its annual Claude Code budget by spring. Amazon engineers reportedly delegated pointless tasks to agents just to climb the internal charts.

There's a name for this now: tokenmaxxing. And it's like the time where we tried to measure productivity through lines of code all over again. Tokens measure an input. They tell you the machine is running, not that it's producing anything you'd want to keep.

The Metric That Matters

There's one number worth tracking instead:

The percentage of shipped work completed end to end by agents.

End to end means from a human intent through build, test, deploy and verify, with humans reviewing by exception rather than gating every step. It measures what the AI delivers, not how often you touch it.

The real value is the question it forces. What class of work do you not yet trust agents with, and what would it take to trust them? That turns AI strategy from a spend conversation into an engineering conversation.

Verification Is the Gap, Not Throughput

Throughput is largely solved. Trust is not. Research across 22,000 developers found task completion up 34% with heavy AI use, but bugs per developer up 54% and code churn up 861%. Throughput measures what shipped. It doesn't measure what survived.

The companies furthest ahead understood this early. Spotify's background agent Honk merges around 1,000 PRs every 10 days, and it works because every change is verified against builds and tests before a PR is even opened, on a codebase standardised over 15 years.

There's also a structural risk to design against. When AI writes the ticket, the code and the review, errors correlate. The fix is independent checks the authoring model never touches. Tests generated from real incidents, security scanning on every PR, canary rollouts to a small slice of users.

As your confidence in verification grows, the class of work you are confident you can allow the AI to do end-to-end also increases. You can move from menial low risk tasks to more complex areas of the system.

Foundations plus verification.

That's the whole playbook. Get the foundations right so you can build the right verification layer and have confidence in the output - then tokens are just the electricity bill.

Andi Smith

By Andi Smith

Andi Smith is a passionate technical leader who excels at building and scaling high-performing product engineering teams with a focus on business value. He has successfully helped businesses of all sizes from start up, scale up to enterprise build value-driven solutions.

Related Blog Posts