The training session is over and trainees have returned to their jobs to begin applying what they’ve learned, so the trainer’s job is done, right? Wrong.
A successful training program is always a work in progress, and the training cycle isn’t complete without an evaluation of training’s effectiveness, which leads to decision-making and planning for future training. Therefore, a useful and informative evaluation program needs to be part of your overall training operation.
To begin with, what should you be looking for in your evaluations?
- Was training delivered as planned, on time, and to the appointed audience?
- Which training methods worked with which topics and which audience groups?
- Which methods did not work with which topics or audiences?
- What specific problems occurred?
- How effective was the trainer at engaging the audience and conveying information?
- How did the training affect employee performance?
- Did the training satisfy regulatory and legal requirements?
- Were all stated goals reached? If not, why not?
This is a lot of information to gather; fortunately, there are many methods and tools available to help you.
Kirkpatrick’s Four-Level Model of Evaluation
Donald Kirkpatrick, author of many books, such as Evaluating Training Programs: The Four Levels and Another Look at Evaluating Training Programs, created a four-level evaluation system more than 40 years ago, which has stood the test of time and continues to be utilized in training programs today. Here are Kirkpatrick’s four levels of evaluation.
Level I: Reaction
Kirkpatrick’s first level evaluates how well participants liked the training session. If you’re interested in running the best training program possible, you want participants to be motivated for and engaged with training. This exercise gives trainees the opportunity to give feedback to the trainer on the pros and cons of the session, which is valuable information that shows trainers specific areas to improve.
Follow these tips when obtaining feedback on trainees’ reactions:
- Observe trainees during the session for your own perception of their reception.
- Get trainee feedback in writing immediately following the session.
- Use measurable and meaningful terms.
- Use uniform feedback forms so results can be quantified and tabulated for the whole group.
Level II: Learning
This level measures how much of the desired principles, techniques, and skills trainees learned in the training session. In order to determine what trainees learn during a session, you need to know what they knew before training.
Follow these suggestions for measuring learning:
- Use pre- and post-knowledge and/or skills testing.
- Use objective measurements to assess what trainees now know or can do that they didn’t know or couldn’t do before training.
- Use a control group of employees who did not attend the training session to compare their performance to employees who received training.
Level III: Behavior
The third level measures employee behavior changes based on training. Your goal is to see how well trainees incorporate learned principles, skills, and knowledge into their jobs on a permanent basis—or at least until they learn a new and better way to perform.
Follow these methods for evaluating behavior:
- Solicit the help of trainers, supervisors, and others who work closely with trainees to observe these employees before and after training, and to give their measurable, objective feedback on performance.
- Continue observations for 3 to 4 months after the training session, so you can get an accurate assessment of whether trainees have made permanent performance improvements based on training.
Level IV: Results
Kirkpatrick’s first three levels focus on trainees and the effect of training on their performance. The last level in Kirkpatrick’s evaluation model focuses on the results of training on the company in terms of:
- Reduction of costs
- Reduction of turnover and absenteeism
- Reduction of grievances
- Increase in quality
- Increase in quantity or production
- Improved morale
Follow these suggestions for measuring the results of training on the company’s overall performance:
- Measure statistics in each of the categories listed above (or whichever categories you included in your goal-setting) before and after training.
- Use a control group, if possible, for comparison.
- Measure more than once over several months to allow time for changes from training to affect the areas you listed.
Evaluation by Return on Investment (ROI) Analysis (or the Fifth Level of Evaluation)
Some training professionals consider ROI analysis to be one method for determining the results of Kirkpatrick’s fourth level of evaluation. Others consider ROI its own level and make it the fifth level of evaluation. In any case, this method is an effective way of measuring the success of your training program. ROI analysis gives the trainer data about the financial impact training programs have on the organization. It differs from Level IV evaluation in the sense that Level IV takes into consideration nonfinancial data such as employee satisfaction. ROI analysis deals strictly with the financial impact of training. It answers the question “For every dollar invested in training, how many dollars does the employer get back?”
Here are three great reasons to use ROI analysis:
- It’s a concrete way to validate your training program as a business tool.
- It can be used to justify the cost of your training program to upper management.
- It can be a useful tool for choosing future training methods.
Many business executives view training as a business expense and, therefore, measure its worth in terms of profits made or savings earned from this expense. You need to make sure training is seen as beneficial to your company. Use the following formula when measuring the ROI for your company to get the percentage of profit earned for every training dollar spent.
ROI (%) = Monetary Benefits – Training Costs x 100
To get the figures for this formula, keep track of training costs, including:
- Design and development
- Delivery (staff or technology)
- Employee wages
After training, keep track of monetary benefits, including:
- Labor savings
- Productivity increases
- Income generation
- New leads
- New products
- Lower turnover costs
ROI Analysis is fast becoming an essential level of evaluation for companies who invest in training.
Tests can be used for collecting information at all five levels of evaluation. They can be given before training, during training, and after training. There are several types of tests from which to choose.
When designing true-false tests, keep these factors in mind:
- Write statements that are clearly true or clearly false.
- Take statements used in training and rephrase them slightly.
- Avoid words that tip trainees to the answer, such as “may” or “generally” for true statements, or “always” or “never” for false statements.
- Avoid double negatives,which make statements unnecessarily confusing.
- Remember that you’re not trying to stump trainees, you’re trying to make sure they learned the material.
These tests may be the most popular with trainees, but they can be the hardest to write for trainers. Follow these suggestions when designing multiple choice tests:
- Cover one topic per question.
- Include only one right answer.Of course, you want all the choices to seem right, but make sure you don’t get so close to right that someone might have a valid argument as to their choice being correct.
- If space allows, use more than three choices. Remember, the more choices you have, the less guessing is involved, e.g., four choices means a guess rate of 25 percent, give choices means 20 percent.
- Avoid always making the right answer the longest choice.
- Avoid making the right answer choice “C” very often. This practice thwarts seasoned multiple choice test takers, who know the ditty, “when in doubt, guess ‘C.’” One way to achieve randomness is to list answers in alphabetical order.
- Avoid giving a grammatical clue in your question, such as using “a” with only one answer beginning with a consonant or “an” with only one answer beginning with a vowel.
- Limit your use of “all of the above” (and “none of the above”). But if you use it for one question, you must use it for at least one more, or trainees will take a clue that it’s probably the right choice for the only question where you included it.
This type of test is used less often but can be fun for the trainee and much easier to design for trainers. Here’s how to write a matching quiz:
- Cover one topic per matching exercise. For example, one test could match a list of chemicals with a list of personal protective equipment the chemical requires. A separate test could match a list of chemical regulations with a list of quotations from those regs. Or one test could match a list of employment laws with a list of the rights they protect and another test could list employment law acronyms with a list of their full names.
- Limit the number of items to around 10. Fewer than 8 can be too easy and more than 12 can get too confusing.
- Lay out the test on one page so trainees don’t have to flip back and forth.
- Make the items in each list brief. Use names, objects, tools, agencies, etc. Avoid making an item longer than one sentence.
Fill in the Blank
These tests can be tricky to write, but they prove that trainees learned the information because they have to produce the right answer without seeing it on the page as in any of the previous tests mentioned. Follow these tips for composing fill in the blank questions:
- Use only one blank per question. Too many blanks don’t give trainees enough information to even grasp the topic.
- Limit the blanks to specific information,such as regulation titles or government agencies, etc. Make the nouns or verbs in a sentence the lank lines, not the adjectives or adverbs.
- Word statements so that there can be only one answer that correctly fills in the blank.
- Place blanks later in the sentence, which helps give trainees the context of the topic.
- Avoid grammatical clues immediately preceding the blank, such as “a” or “an.”
Other Evaluation Methods
Some other evaluation methods include first-hand observation and production data analysis. You can use observation both in the classroom to monitor trainees as they practice new skills and later on in the job to see how they’re incorporating training into their performance. Your goal is to see how well the trainee learned the skill. You can develop a rating scale or simply make comments.
Use production data analysis by keeping track of production quotas and other data before and after training. Compare the results to see measurable ways training improved job performance. Make sure you take into account variables other than training that could have affected performance when you look at these numbers.
Transferring Learning to Work
In order to ensure your training has been effective, you need to do more than evaluate. You also need to take post-training time to help trainees transfer new skills and knowledge to the workplace—and to make these changes stick. You may need to help employees overcome certain obstacles to applying training to the job.
Obstacles to the Transfer
Here are common obstacles employees may encounter:
- Learners can be set in their ways and reluctant to embrace change.
- Upper management may not offer enough support either in terms of morale or materials.
- Learners may lack enough motivation or incentive.
- Learners may not have enough time to practice new skills and achieve a comfort level.
- Co-workers may apply peer pressure against changing methods.
How to Avoid Obstacles
Here’s how you can help trainees avoid obstacles to transferring what they’ve learned to their jobs:
- Give your personal support. A few weeks after the training program, send trainees a note or e-mail, or give them a call. Ask questions like these to their jobs:
- How is what you learned affecting your work?
- Are you having any problems or concerns?
- Do you have any final comments or suggestions about the learning experience?
- Make your support ongoing. Have an open-door policy with employees so they know they can talk with you at any point about any issues that come up as they transfer learning to the job.
- Establish group support. At the end of training sessions, assign trainees to small groups that will meet regularly for a while after training. Participants can use these groups to discuss common problems and concerns, how they are doing in applying the new skills, and to perform practice sessions. Group support gives members a sense of unity and security, which they can fall back on when they run into problems.
- Compose checklists or cheat sheets. Give these out at training sessions for employees to take with them back to work and refer to as needed. These are especially helpful for new procedures or new steps in existing procedures.
- Set up a coaching program. Coaches can be trainers, qualified experienced employees, or members of upper management. The point to this method is to have someone on call to answer questions, give feedback, give support, and to be a role model for proper behavior.