Sooner or later, someone asks whether the AI coding tools are worth it. The honest answer at most companies is “people like them”. That is worth something, but it is not a measurement, and it can be wrong in ways that matter.
This is a practical guide for engineering leads: what not to measure, what to measure instead, and how to set things up so the answer means something.
Why “it feels faster” isn't enough
In a randomized study published in July 2025, METR had experienced open-source developers work on real tasks in their own repositories, with and without AI tools. With AI, tasks took about 19% longer. Afterwards, the same developers estimated that AI had made them about 20% faster.
That result comes from one setting, with the tools of early 2025, and METR's 2026 follow-up saw small speedups while calling its own evidence weak. The lasting lesson is not that AI makes developers slower. It is that people's sense of their own speed is not a reliable instrument. If you want to know, you have to measure.
The team level tells a similar story. The 2024 DORA report found that as AI adoption increased, delivery throughput and stability went slightly down, even as documentation quality, code quality and review speed went up. Individual gains don't automatically turn into team delivery.
What not to measure
- Lines of code. Agents can produce a lot of it. More code is often a cost, not a benefit.
- Suggestion acceptance rate. It measures how often people press Tab, not whether the result was good or saved time.
- Number of pull requests. Easy to inflate, and it rewards splitting work rather than finishing it.
- Percentage of code written by AI. Interesting as a curiosity, and says nothing about whether the team ships better software faster.
What to measure instead
Measure the delivery system, and the quality of what it delivers. Six signals cover most of it:
- Lead time for changes. From first commit to running in production. The headline number for “are we faster?”.
- Pull request cycle time, split into parts. Time to first review, time in review, and time from approval to merge. Agents often shrink coding time and grow review time. You only see that if you split it.
- Change failure rate. The share of deployments that cause an incident, a rollback or a hotfix.
- Rework rate. How much recently written code gets rewritten or reverted within a few weeks. A rising rework rate means speed now is being paid for later.
- Pull request size. Bigger pull requests are reviewed less carefully. If agents push size up, quality usually follows down a few weeks later.
- Repeated review comments. How often reviewers flag the same issue again: the missing auth check, the wrong package manager, the edited migration. This one is rarely tracked, and it is the clearest sign of whether your agents are learning from your team or starting from zero every time.
The first and third are part of DORA's well-known delivery metrics, and most of these can be pulled from your Git host and deployment tool without new tooling.
Set a baseline first
Without a baseline, every number after the rollout is a story. Before you roll out a new tool, or now, if you already have, record at least a month of these metrics. Then compare like with like:
- Same team, before and after, keeping in mind that other things change too: hiring, product phase, the season.
- Similar teams, with and without, if you roll out in stages. This is the closest most companies get to a controlled experiment.
- Give it time. Teams are usually slower for the first weeks while they learn what works. Judge the trend over a quarter, not the first sprint.
Ask people, carefully
Surveys are not useless. They are just measuring something different: satisfaction, focus, frustration and where the tools help. Ask specific questions, regularly, and compare them over time:
- Which tasks did AI help with most this month? Which did it make worse?
- How much time did you spend reviewing or fixing AI-written code?
- How often did you have to explain the same thing to an agent again?
The answers tell you where to invest: better rule files, better tests, smaller tasks, or a different tool for a certain kind of work.
Watch where the cost moves
The most common pattern we see is not “AI made us faster” or “AI made us slower”. It is that the work moved. Writing got cheaper, and reviewing, debugging and re-explaining got more expensive. A good measurement setup makes that visible, so you can fix the bottleneck instead of arguing about the tool.
That is why repeated review comments are the number Memmo is built to bring down. Every comment that has to be written twice is time spent teaching an agent something the team already knew. Cutting that is one of the most direct ways to turn individual speed into team speed.