I
Infopro Learning
Guest
- Thread Author
- #1
When I started at Infopro 15 years ago, I was pretty naive. I’d look at something like the Kirkpatrick Model and think, why aren’t we just measuring business impact? Isn’t that all the business actually cares about? Nobody in the boardroom wants to hear that people liked the training or passed the quiz. They want to know if it moved the number.
So that’s what I chased. I wrote a lot of articles about it. I talked to a lot of people about it. The importance of measuring learning through business metrics, not course completions, became something close to a personal crusade. I was sure I’d found the thing everyone else was missing.
Then I actually got my hands dirty, and the whole thing got a lot more complicated.
Here’s what surprised me. It wasn’t that we couldn’t get the data. Getting the metric itself was actually the easy part. We could measure something like sales conversion at one point in time, run a training program, measure it again a few months later, and see exactly what moved. That part was never the problem.
The problem was correlation versus causation, and it showed up constantly. In some cases, we’d see training followed by a real improvement in the metric. In plenty of other cases, we wouldn’t, even when the training itself was good. It started to feel almost random. Not because the programs were inconsistent, but because we simply didn’t have enough visibility into everything else happening around that metric at the same time to know what was actually responsible for the movement, or the lack of it. Did we contribute? Probably, some of the time. By how much? I genuinely couldn’t tell you, and if I’m honest, most of the people making that claim across the industry couldn’t either.
That’s when I started asking myself a different question. If there are ten factors that influence a business metric, and training is one of them, should we really be the ones owning that metric? I don’t think we should. Not because it doesn’t matter to us. It’s the reason the work exists in the first place. But owning it and being accountable for the other nine factors we have no control over is a different thing entirely, and it’s the thing I think L&D has been quietly setting itself up to fail at for years.
What we should own is something much more in our control, and that thing is a skill.
I don’t think this is a new realization. I think most experienced L&D people have known this for a long time, at least somewhere in the back of their minds. The problem was never that we didn’t understand skill was the more honest thing to measure. The problem was that measuring it well, at scale, in a way that was specific to an individual person rather than a generic pre and post survey, cost more than most organizations were willing to pay. The juice wasn’t worth the squeeze.
But now, almost all of the resource cost (human + financial) attached to this work has gone away. We can verify a person’s skill across 5 different sources by gathering 100’s of data points and analyzing trends against a baseline and determine what an accurate picture of their current skill level looks like against what we want. We can then take it one step further and build out a personalized learning program to close the gap…and when I say personalized, I mean it! It isn’t a pre-built learning journey that we assign to the person. The learning journey is built instantly using a collection of the content you already own. To wrap it all up, we take the same concepts of what we did to know the current skill level of the person to then prove we actually moved the needle.
That shift is what we’ve come to call Know, Grow, Prove. Know where a skill genuinely stands today, not where a job title or a generic assessment assumes it stands. Grow it, deliberately, against that specific gap. Prove it moved, with a real before and after, not a guess dressed up as an outcome.
I want to be careful here, because I know exactly how this can sound if I’m not precise about it. This is not an argument that business metrics don’t matter. They matter more than anything else in the conversation. Revenue, retention, safety, quality, whatever number the business is actually worried about, that’s where every single program should start. If you’re not asking what business problem you’re trying to solve before you build anything, you’re already doing this wrong, full stop. The business metric is the reason the work happens.
What I’m arguing is that the business metric shouldn’t be the thing we’re graded on. It’s the destination, not the scoreboard. The scoreboard should be the skill we specifically set out to move, because that’s the piece of the ten factors that was always actually ours.
Discover how personalized learning experiences can boost employee engagement, support skill development, and drive meaningful growth
Watch Now
For a long time I assumed this was just something I’d landed on through a lot of trial and error, mostly error, with programs that didn’t hold up the way I wanted them to. Turns out the data backs up more of this than I expected, even if the industry hasn’t fully caught up to what it means yet.
The Kirkpatrick Model, which is still the framework most of us were trained on, is used by 71 percent of organizations according to ATD’s most recent research on the topic. It’s the default, by a wide margin. And in that same research, 87 percent of organizations said they find it difficult to isolate training’s actual impact on the business (ATD). That’s not a fringe complaint. That’s nearly nine out of ten people doing this work, telling you the same thing I ran into with sales conversion data years ago.
Speaking of a decade ago, let’s go back to ATD’s 2016 research on the same topic, and only 35 percent of the organizations surveyed said they evaluate business results at all (ATD). Most L&D functions never even get to that step. I don’t think that’s because they don’t care about the outcome. I think it’s because, somewhere along the way, most of them ran into the same wall I did, and quietly stopped trying to climb over it.
In short, this isn’t a new problem.
I’ll add one honest caveat here, because I’d rather tell you the truth than borrow a stat that says more than it should. In conversations I’ve had recently with CLOs and CHROs, I keep hearing a version of this same shift: less interest in a vendor promising a business-impact number, more interest in a vendor who can show a specific skill moved, with real evidence. That’s a real pattern from real conversations. It is not yet a documented industry trend, and some of the broader published research still argues the opposite, that L&D needs to get better at proving business impact, not move away from trying. I think the field is heading toward what I’m describing. I’m not going to pretend it’s already arrived there.
In practice, this changes how a program gets scoped before a single piece of content gets built. You still start with the business metric, because that’s the conversation the business wants to have and it’s the one that earns L&D a seat in the room. But the very next question has to get specific fast: what skill, measured in real terms, is the mechanism we’re actually responsible for moving. Build the measurement before the program launches, not after, so you have a real starting point instead of a memory of where things used to be. Then report on what actually moved, honestly, including when it didn’t move as much as you hoped.
If that skill movement also happens to correlate with the business metric moving in the right direction, that’s a genuine win, and worth pointing to. It just can’t be the finish line, because too much of what drives that number was never ours to promise in the first place.
I think back sometimes to the version of me who was 15 years younger, writing articles insisting that business impact was the only thing that mattered, completely sure I was ahead of everyone else in the room. I wasn’t wrong that the business metric matters. I was wrong about what we should be willing to put our name on. It took getting my hands dirty, watching the same program produce a lift one quarter and nothing the next, to understand the difference.
If you’re still chasing the number the way I used to, I get it. I spent years there. But I’d ask you the same question I eventually had to ask myself: of everything that moves that metric, how much of it is actually yours? And are you willing to build your credibility on the part you can’t control, when there’s a part right next to it that you can?
Sources referenced:
The post We’ve Been Trying to Prove the Wrong Thing first appeared on Infopro Learning.
So that’s what I chased. I wrote a lot of articles about it. I talked to a lot of people about it. The importance of measuring learning through business metrics, not course completions, became something close to a personal crusade. I was sure I’d found the thing everyone else was missing.
Then I actually got my hands dirty, and the whole thing got a lot more complicated.
Here’s what surprised me. It wasn’t that we couldn’t get the data. Getting the metric itself was actually the easy part. We could measure something like sales conversion at one point in time, run a training program, measure it again a few months later, and see exactly what moved. That part was never the problem.
The problem was correlation versus causation, and it showed up constantly. In some cases, we’d see training followed by a real improvement in the metric. In plenty of other cases, we wouldn’t, even when the training itself was good. It started to feel almost random. Not because the programs were inconsistent, but because we simply didn’t have enough visibility into everything else happening around that metric at the same time to know what was actually responsible for the movement, or the lack of it. Did we contribute? Probably, some of the time. By how much? I genuinely couldn’t tell you, and if I’m honest, most of the people making that claim across the industry couldn’t either.
That’s when I started asking myself a different question. If there are ten factors that influence a business metric, and training is one of them, should we really be the ones owning that metric? I don’t think we should. Not because it doesn’t matter to us. It’s the reason the work exists in the first place. But owning it and being accountable for the other nine factors we have no control over is a different thing entirely, and it’s the thing I think L&D has been quietly setting itself up to fail at for years.
What we can actually own
What we should own is something much more in our control, and that thing is a skill.
I don’t think this is a new realization. I think most experienced L&D people have known this for a long time, at least somewhere in the back of their minds. The problem was never that we didn’t understand skill was the more honest thing to measure. The problem was that measuring it well, at scale, in a way that was specific to an individual person rather than a generic pre and post survey, cost more than most organizations were willing to pay. The juice wasn’t worth the squeeze.
But now, almost all of the resource cost (human + financial) attached to this work has gone away. We can verify a person’s skill across 5 different sources by gathering 100’s of data points and analyzing trends against a baseline and determine what an accurate picture of their current skill level looks like against what we want. We can then take it one step further and build out a personalized learning program to close the gap…and when I say personalized, I mean it! It isn’t a pre-built learning journey that we assign to the person. The learning journey is built instantly using a collection of the content you already own. To wrap it all up, we take the same concepts of what we did to know the current skill level of the person to then prove we actually moved the needle.
That shift is what we’ve come to call Know, Grow, Prove. Know where a skill genuinely stands today, not where a job title or a generic assessment assumes it stands. Grow it, deliberately, against that specific gap. Prove it moved, with a real before and after, not a guess dressed up as an outcome.
I want to be careful here, because I know exactly how this can sound if I’m not precise about it. This is not an argument that business metrics don’t matter. They matter more than anything else in the conversation. Revenue, retention, safety, quality, whatever number the business is actually worried about, that’s where every single program should start. If you’re not asking what business problem you’re trying to solve before you build anything, you’re already doing this wrong, full stop. The business metric is the reason the work happens.
What I’m arguing is that the business metric shouldn’t be the thing we’re graded on. It’s the destination, not the scoreboard. The scoreboard should be the skill we specifically set out to move, because that’s the piece of the ten factors that was always actually ours.
Leveraging Learning Experiences to Create a Personalized Growth Journey
Discover how personalized learning experiences can boost employee engagement, support skill development, and drive meaningful growth
Watch Now
I wasn’t as alone in this as I thought
For a long time I assumed this was just something I’d landed on through a lot of trial and error, mostly error, with programs that didn’t hold up the way I wanted them to. Turns out the data backs up more of this than I expected, even if the industry hasn’t fully caught up to what it means yet.
The Kirkpatrick Model, which is still the framework most of us were trained on, is used by 71 percent of organizations according to ATD’s most recent research on the topic. It’s the default, by a wide margin. And in that same research, 87 percent of organizations said they find it difficult to isolate training’s actual impact on the business (ATD). That’s not a fringe complaint. That’s nearly nine out of ten people doing this work, telling you the same thing I ran into with sales conversion data years ago.
Speaking of a decade ago, let’s go back to ATD’s 2016 research on the same topic, and only 35 percent of the organizations surveyed said they evaluate business results at all (ATD). Most L&D functions never even get to that step. I don’t think that’s because they don’t care about the outcome. I think it’s because, somewhere along the way, most of them ran into the same wall I did, and quietly stopped trying to climb over it.
In short, this isn’t a new problem.
I’ll add one honest caveat here, because I’d rather tell you the truth than borrow a stat that says more than it should. In conversations I’ve had recently with CLOs and CHROs, I keep hearing a version of this same shift: less interest in a vendor promising a business-impact number, more interest in a vendor who can show a specific skill moved, with real evidence. That’s a real pattern from real conversations. It is not yet a documented industry trend, and some of the broader published research still argues the opposite, that L&D needs to get better at proving business impact, not move away from trying. I think the field is heading toward what I’m describing. I’m not going to pretend it’s already arrived there.
What this actually looks like
In practice, this changes how a program gets scoped before a single piece of content gets built. You still start with the business metric, because that’s the conversation the business wants to have and it’s the one that earns L&D a seat in the room. But the very next question has to get specific fast: what skill, measured in real terms, is the mechanism we’re actually responsible for moving. Build the measurement before the program launches, not after, so you have a real starting point instead of a memory of where things used to be. Then report on what actually moved, honestly, including when it didn’t move as much as you hoped.
If that skill movement also happens to correlate with the business metric moving in the right direction, that’s a genuine win, and worth pointing to. It just can’t be the finish line, because too much of what drives that number was never ours to promise in the first place.
I think back sometimes to the version of me who was 15 years younger, writing articles insisting that business impact was the only thing that mattered, completely sure I was ahead of everyone else in the room. I wasn’t wrong that the business metric matters. I was wrong about what we should be willing to put our name on. It took getting my hands dirty, watching the same program produce a lift one quarter and nothing the next, to understand the difference.
If you’re still chasing the number the way I used to, I get it. I spent years there. But I’d ask you the same question I eventually had to ask myself: of everything that moves that metric, how much of it is actually yours? And are you willing to build your credibility on the part you can’t control, when there’s a part right next to it that you can?
Sources referenced:
- ATD, “ATD Research: Measuring Learning Program Effectiveness Is Challenging,” 2023: td.org
- ATD, “Evaluating the Impact of Learning Programs,” referencing Evaluating Learning: Getting to Measurements That Matter, 2016: td.org
The post We’ve Been Trying to Prove the Wrong Thing first appeared on Infopro Learning.