Evaluation

The third “phase” of the HPT process is not really a phase at all. Evaluation is actually an integral part of the entire HPT process and should occur as part of the initial analysis phase, as well as during the design and development of interventions. Evaluation, represented as a third major step, is the part of the process that determines the overall value of the performance analysis and intervention.

Evaluation can be:

Kirkpatrick’s Evaluation Model

This model helps to answer the first question of evaluation. Level 1 and 2 measures are commonly performed through questionnaires or “smile-sheets” and criterion-referenced testing, respectively. Level 3 and 4 measures are generally seen as more difficult or time-consuming and are rarely done in practice (Shrock & Geis, 1999).

Level 1

Determine the immediate reactions of the targeted performers to the intervention

Level 2

Determine the mastery, transfer of knowledge, and competence of the targeted performers

Level 3

Determine the level of application of the intervention as demonstrated in job behavior

Level 4

Determine organizational results or Return on Investment (ROI)

 

Measuring ROI

Measuring the ROI of individual performance improvement projects is a step in the right direction, but it is not sufficient.  Although for many of us, saving $100 thousand may seem heroic, for top executives such figures are “loose change”.  What does capture their attention is valuation. (Gayeski, 2004, p.7)

Return on Investment evaluation can be performed at any time throughout the HPT process…to estimate future returns before a project begins, to formatively identify and adjust plans under development, and to provide figures on the financial impact of implemented interventions. Regardless of when ROI is considered, the basic model is the same:

Performance Value – Cost = Benefit
(Swanson, 1999, p.815)

As stated above, this level of evaluation is rarely performed, though it is beginning to get the attention of many HPT practitioners who want to demonstrate the worth of performance improvement interventions. But how do we measure the value of improving employee motivation or knowledge? Can the impact of successful performance interventions be accurately communicated in terms of a dollar value? What about the intangible outcomes like human capital and customer satisfaction? Some have suggested that evaluation must go beyond this level to be complete.

Evaluation Plus

If you don’t also ask and answer both questions, your evaluation is incomplete. You won’t be able to answer (the) basic question of what happened, and was it successful for all stakeholders – associates, the organization, and external clients. (Kaufman & Unger, 2002, p. 4)

According to Kaufman & Unger (2002), evaluation should answer the questions:

  1. Did we achieve the goals we set out to meet?
  2. Did our we accomplish what we should have accomplished and provide value to all internal and external stakeholders?

This model, developed by Kaufman, Keller, & Watkins, requires practitioners to look at mega-level consequences of performance interventions, adding a fifth level to Kirkpatrick’s model.  Level 5 evaluation provides answers to the question “Did we do good for society and satisfy the needs of the organization?” Factors considered at this level include the long-term profits and human capital development of the organization, health and safety issues for workers and others in society, environmental issues, and other contributions to customers and society in general (Gayeski, 2004; Kaufman, Keller, & Watkins, 1996).

Deciding what, when, and how to evaluate performance interventions is an important part of the work of any HPT practitioner. See the Business and Process of HPT section for more information on how evaluation fits into the process of HPT.

References

Gayeski, D. (2004). IPT 560: Beyond level 4: Tying HPT to valuation of intangible assets, Mod 6 Getting and showing results. Lotus Notes database via Boise State University. Retrieved December 1, 2004.

Kaufman, R., Keller, J., & Watkins, R. (1996). What works and what doesn’t: Evaluation beyond Kirkpatrick. Performance and Instruction, 35(2), 8-12.

Kaufman, R. & Unger, Z. (2002). Evaluation plus: Beyond conventional evaluation. Retrieved December 5, 2004 from www.onap.fsu.edu/download/pub/04.doc

Shrock, S. A. & Geis, G. L. (1999). Evaluation. In H. Stolovich & E. Keeps (Eds.). Handbook of human performance technology (2nd ed., pp. 185-210). San Francisco: Jossey-Bass Pfeiffer.

Swanson, R. A. (1999). Demonstrating return on investment in performance improvement projects. In H. Stolovich & E. Keeps (Eds.). Handbook of human performance technology (2nd ed., pp. 813-839). San Francisco: Jossey-Bass Pfeiffer.

Van Tiem, D. M., Moseley, J. L., & Dessinger, J. C. (2000). Fundamentals of performance technology: A guide to improving people, processes, and performance. Washington, DC: International Society for Performance Improvement.