What the Dashboard Cannot See

We recently introduced an AI-powered developer metrics system across all of our engineering teams.

Ten weeks in, and it is already changing how I think about performance visibility. For better and for worse.

The system tracks the usual signals: PRs, commits, reviews, merges, ticket activity, documentation, etc. Each week, everyone gets an individual summary with AI commentary. What went well, and one actionable step to improve. As a manager, I also get a team-wide view. I am included in that too; I get the same metrics everyone else does.

When scores are high, people feel seen. When scores are low, people feel judged.

That second part matters. Because the system does not know about the engineer who spent three days tracking down a flaky test pipeline. It does not see the manual testing or the cross-team dependencies that quietly ate a sprint. And when tickets are blocked or poorly ordered, that is not always the engineer. Correctly ordering tickets and flagging dependencies are my responsibility, along with my principal engineers. The metrics will reflect that, and right now, they are telling me I have work to do there.

AI commentary can surface patterns. It cannot supply context.

That is not an argument against the system. It is genuinely useful. But ten weeks in, my biggest takeaway is this: the metrics are a starting point for a conversation, not a conclusion. I am still learning how to use them well.

Measure what you can. Stay curious about what you cannot.

Seeing this in your org too? I would love to hear how other engineering leaders are thinking about it.