Measure Learning Input, Not Output

Aug 4 • Jamie Dixon Article, Management and Leadership, Presentation Skills • 12364 Views • 4 Comments on Measure Learning Input, Not Output


ONLY MEASURE WHAT YOU CAN CHANGE. Return on investment (ROI) measurements only tell you about the past, and as I will explain, there is a much more effective way of measuring learning.

Imagine you attend a training course about conversation skills, and the instructor introduces you to a really useful principle; “Be Interested, Not Interesting”. That’s interesting you think.

Return on investment (ROI) measurements only tell you about the past

A few days later you meet a client on a business trip. You think about how to use this principle on this client and decide that you will ask lots of questions about them. They feel a little bit surprised about how many questions you ask them, but generally they respond quite well. This is good, you think, I’ll try this again.

Over the next few months you continue to do the same thing every time you meet someone new, gradually adjusting the amount of questions you ask to make them feel more comfortable, and continue to have the same positive results. Then one day you meet a guy called Steve. Steve asks you loads of questions about yourself and seems genuinely curious. His facial expression, his fixed gaze and his unwavering attention convince you that he thinks you are a fascinating person. You like Steve, but more importantly you learn from him that showing your curiosity adds to the effect of asking questions. So you add this new skill to your repertoire of effective conversation skills.

Over the next few years you continue to use your questions and genuine curiosity when meeting strangers, always getting really positive results. Then one day, four years after attending that course, you sign the biggest contract in the company’s history: US$2 million. The commission is enough for you to retire. As you invite the client out for a meal to express your gratitude, you ask “what was it that made you sign that contract?”. The client says, “You were the only one that showed a genuine interest in us”.

If your boss was to measure the ROI of you having attended that course, they would probably measure the difference in your sales over a six month or one year period, and then compare the difference with the cost of the course. Most likely, they would find either a very slight increase in ROI, or actually no ROI at all. They might even find an initial negative RIO.

Most likely, they would find either a very slight increase in ROI, or actually no ROI at all.

Your ROI report measures the value of training up until the minute your data stops. This is one of the biggest weaknesses of measuring the value of learning. Who are you to say when the learning stops? In the example above, learning that one principle didn’t show a monetary result for 4 years! Believe it or not, this is actually very common. There are most likely things you learned years ago, maybe even in primary school, things that you don’t even have a conscious memory of learning, that you still have not benefited from yet but will do in the future.

Furthermore, was it the training that you benefited from? Or was it your new friend Steve? Actually, it was a combination of both of those things, plus many, many other factors. Time, sleep, reflection, experience, feedback and so on. All of these things contributed to your learning. So who are you to say that the only source of learning was the training?

Theoretically, of course you can measure the ROI of training. But you need to be aware of just how much value that training created, and the other things that contributed value, and how much time it took to realize that value, and how much of that realized value was due to the learning and how much was due to other factors. To get a precise number, you need to be in possession of some magical powers that allow you to gather that data. If you don’t have those magical powers, then trust me, your time is much better invested elsewhere.

OK, you argue, but I don’t need a 100% accurate number, a rough estimate is more than enough. Well to that I firstly argue, why bother measuring something if you’re going to overlook so many important factors? And secondly I argue that you get far more valuable information about the effectiveness of learning in your organization if you focus on input instead of output.

I argue that you get far more valuable information about the effectiveness of learning in your organization if you focus on input instead of output.

Let me be clear, my second argument here rests on the assumption that the higher the input of learning, the higher the output of learning. But this argument only counts if the learners themselves are proactively engaging in these tasks, not if you are forcing them. As I mentioned here and here  engagement is crucial for learning, if there is no engagement there is no learning.

So provide your staff with multiple options for learning. On top of training, have a library where they can borrow books, have monthly experience sharing discussions, encourage them to attend conferences and seminars, set new challenges for them, let them try something they’ve never done before and so on. Measure these, do not waste your time measuring ROI.

A huge benefit of measuring learning input is that you can find what works and make those things more available immediately. One person really likes going to conferences, so share with them more invitations to these conferences when you receive them. Another maybe really likes reading books, so recommend some books for them, even offer to buy some if there are any they really want to read. This information is invaluable, especially if these are things they are proactively seeking out. Measuring input allows you to identify those inputs that work, whereas measuring output literally tells you nothing, and tells you it too late.

A huge benefit of measuring learning input is that you can find what works and make those things more available immediately.

Don’t assume that all learning comes from training, because it comes from all over the place. Identify the things that are within your control, that your staff can proactively engage in, and that will help them learn new things and encounter new experiences. Keep a record of who does what, and reward them for doing more and more. Having a variety of options increases the chances of your staff finding a way to learn that engages them, and increases the chances of them participating in more and more learning activities. Just like a supermarket that tracks what you buy when you use their membership card; the more you buy a certain product, the more offers and advertisements the supermarket will send you for similar products, increasing the chance of you buying those new products. Aim to get your staff addicted to learning.

As the example at the beginning made clear, learning is ambiguous. You will never be able to fully identify the output of learning, you will never even be able to identify all of the inputs. Yet having a rough idea of something you can change is infinitely more useful than having a rough idea of something you can’t change.

Related Posts

4 Responses to Measure Learning Input, Not Output

  1. Paris says:

    Thanks for sharing your thoughts Jamie. Is your argument that measuring learning input instead of learning output is more valuable to an organization or the the training function? I agree that learning inputs and levels of engagement are good things to track. However I would disagree with getting rid of any post engagement follow-up or measurement activities.

    This topic goes back to the age old Kirkpatrick 4 (now 5) levels of measurement. Learning inputs and activities are captured at level one. You learn what inputs receive the best reception from staff. Once a consistent measurement of activity is established, it is time to move into the science of measurement: looking at performance behaviors and alignment.

    Just because someone reads lots of books or goes off to conferences doesn’t mean that they are performing well on the job. Performance results can only be captured with a well established partnership with HR and the Business line managers. This takes work. If a learning department hasn’t sat down with a senior leader, or even the line manager, to identify what performance results could be seen to recognize that the training has done what it said it would do, then the whole ROI thing doesn’t matter anyway.

    I would say let’s not throw away the ROI equation, but rather focus on the important things first. You can’t establish ROI without measuring inputs at a basic level(your point above). THEN, don’t let this be the only indicator of value you report to the business. Instead, start moving the department closer to establishing itself as a strategic lever, meaning getting better engagement with the performance expectations of the business and the key behaviors that lead to success.

    New business might take 4 years to close a new deal, but you could definitely measure the number of positive phone calls the increase in potential client rapport along the way. These things have value in the sales cycle and can be documented/measured. This will add to the chain of evidence. Then the question can be posed back the business about how training might be able to support in making the full cycle faster.

  2. Morry Morgan says:

    The challenge with a lot of ROI measurements is that they are time delayed. That is, you only find out a month, 6 months, or 12 months later that someone is under-performing. In this way, Jamie’s focus on ‘input’ is identical to our sales approach. We require our sales people to maintain a ‘Quality Score’ over 1000 points. This means that right now, I can look at my CRM system and see who is under-performing. It is instant and measures engagement.

    The same applies to learning. If someone is actively engaged that they will learn. But how do you know that they are ‘active’? Hence, Jamie’s focus on ‘input’ is ideal.

    Of course, that doesn’t mean that you don’t measure performance as well. That’s why you still provide bonuses in sales for hitting targets, since great sales managers know that hitting end of year targets only comes from daily engagement.

  3. Jamie Dixon says:

    Hi Paris,

    Thanks for your reply.

    What I am trying to say is that measuring input is of greater value than measuring output. I don’t believe we should give up on measuring output, although I generally disagree with measuring ROI as it makes many assumptions and overlooks many factors. Behaviour should definitely be measured, as this is what training aims to change, but I believe also the attitude to measuring behaviour should not just be about measuring it, it should be about reinforcing it, with the measurements simply being part of the reinforcement.

    As for Kirkpatrick’s levels of measurement, the vast majority of organisations I work with do very little beyond level 1 measurements (reaction). Whilst I don’t question the logic of Kirkpatrick’s approach, this does makes me question the practicality of Kirkpatrick’s approach, given that it was first published in 1959 and 54 years later so many organisations still don’t fully utilise it.


  4. Jalal Afhim says:

    I think both Paris and Jamie make great points about considering a range of measurements and perspective when looking at how we evaluate learning effort. I also think that both of the approaches they mention are compatible with each-other. They both describe outcomes.

    Yes, a greater uptake in learning initiatives by staff is an outcome, although probably not one you could use in a discussion with the C-suite people to justify training budget! Changes in behaviour after learning interventions are also an outcome, but would, in that same discussion, need to be linked to business results and the bottom line. Do so few organizations measure above Kirkpatrick Level 2 because it is not useful to do so, or because so few C-level sponsors speak the language of learning and behavioural change? How many training departments out there are incentivized to provide convenient reports of numbers, as this is preferable to those in charge?

    What we are talking about, I believe, are both the mechanics of the learning function, and how we articulate this value to others. If we do not articulate this message in a way that makes sense to the senior management, we will not survive.
    The savvy learning professional should, as Paris pointed out, present what they do in terms of enabling strategic goals. In that conversation, draw on the measures that are right for your organization, and are right for the people who control the purse-strings. These might include measures of learning activity, and a whole range of other metrics. If you are very lucky, you may even have the leeway to measure behavioural change and it’s impact on the business.

Leave a Reply

Your email address will not be published. Required fields are marked *

« »

kjøpe bil med forbrukslån | refinansiering | link