When a Great AI KPI Tells an Incomplete Story

There is a lot of pressure right now to demonstrate value from AI. Understandably, we tend to gravitate toward things we can count: time saved, cost savings, adoption rates, productivity gains.

Those things matter, but they don't always tell the whole story.

I saw this firsthand in an AI project I led a couple of years ago. We built a job-posting generator to solve a problem we'd wrestled with for years: creating and maintaining high-quality, consistent job postings at a global scale.

We had tried creating a global library of job postings for recruiters to reuse. In theory, it made perfect sense. In practice, roles and messaging changed faster than we could maintain it.

We knew the issues:

• Too much time spent creating postings: Recruiters started from old postings or a blank page and spent valuable time creating and refining content.
• Outdated and inconsistent content: Roles and messaging changed faster than we could maintain the library. Plus, postings varied across geographies, job levels, and recruiters.
• A disconnect from job architecture: What we advertised externally didn't always align with the role the organization had actually designed.
• Missed brand and employee value proposition (EVP) opportunities: The messaging Marketing had carefully developed wasn't consistently making its way into what candidates saw.
• A weak starting point for recruiters: Recruiters needed a stronger foundation for job-launch conversations with hiring managers.

AI gave us a different way to solve the problem. Instead of trying to maintain a static library, we could generate a current posting when we needed one, in nine languages, using a consistent framework.

And it worked. We estimated the solution reduced the time required to create a posting by a whopping 80%.

80%! That's a fantastic outcome. It's exactly the kind of clear, compelling KPI we want when demonstrating the value of AI.

And time savings wasn't the only outcome. The solution also:

• Created global consistency and stronger alignment with job architecture by anchoring postings to the role the organization had actually designed, so the same role was represented consistently across countries and regions.
• Addressed risk by replacing outdated, inconsistent content with a more standardized approach.
• Strengthened alignment with Marketing by consistently bringing our brand, tone of voice, and EVP into candidate-facing content.
• Empowered recruiters by giving them a strong starting point for job-launch conversations. If a hiring manager wanted to substantially change the posting, the recruiter had a basis for asking whether we were still hiring for the role that had actually been defined.

That's the point. The 80% time savings was a great KPI, and only one part of a much broader value story.

Your baseline shapes the value you can demonstrate

We intentionally designed the solution to deliver all of those outcomes, but the problem was that we hadn't established equally strong baselines for all of them.

Time was relatively easy to figure out. We could ask recruiters how long creating a posting used to take and compare that with the new experience. We knew the other problems existed too, but much of our evidence was anecdotal. That made it much harder to demonstrate increased consistency, improved brand alignment, reduced risk, or empowered recruiters.

An 80% reduction in posting creation time also makes for a much cleaner KPI than “recruiters were better equipped to challenge role misalignment.”

But harder to measure doesn't make something less valuable.

Going into an AI project today, I’d spend more time understanding where we’re starting: what’s happening now, what we want to improve, and what evidence we’ll need to show we’ve made a difference.

Three questions I would ask now

If I were starting that project again, I’d build the same solution and spend more time upfront answering three questions:

1. What problem are we actually trying to solve? Understand the process, the data, and the experience of the people doing the work. Then decide whether AI is the right solution, whether it needs to be paired with other changes, or whether another approach would work better.

2. So what? If we solve it, what gets better? Is it faster? Cheaper? Higher quality? A better experience? Less risk? And, importantly, why does that matter to the business or the people it serves?

3. How will we know it worked? What do we know about the current state, and what evidence will tell us later that it improved? Some of that evidence will be quantitative, and some won't be. Candidate and employee surveys, stakeholder feedback, open-text comments, interviews, and focus groups reveal things that aren't visible in operational data alone.

The numbers may tell you where to look, while the narrative may tell you what to fix.

You can start experimenting with AI while you work out how to measure its value. Thinking upfront about the benefits you expect, and how you’ll recognize them, puts you in a better position to show what improved, even when those benefits don’t have a dollar amount attached.

Understanding where we are, where we want to go, and why it matters gives us a stronger foundation for showing AI’s value. When the results come in, we’re ready to tell the whole story.