BY JAMES GRABOWSKI; ROBERT J. GREENE, PH.D.; PETER RONZA; VALERIE JOHNSON; AND LYNN LOHMAN

Employees must view appraisals of their performance as equitable if they are to accept the performance management system. But managers tend to vary in the level of expectations they have for their employees. The perception that standards are uneven across managers can create significant dissatisfaction and the feeling that the playing field is not level. An additional concern is that uneven performance standards can cause a statistically significant adverse impact on protected classes related to pay and promotion.

Principles Underlying Effective Training

Training managers is the most obvious strategy for gaining consistency, but most traditional approaches that focus on identifying various types of rater error have been shown by research to be only moderately effective. The approach that has shown to be the most effective is using “frame of reference” training.

Frame of reference training focuses on developing a consistent set of performance standards. Managers work in groups on a case that consists of several completed appraisals that have commentary about job-related results and behaviors, but no ratings. During the training, individual managers independently arrive at their own ratings, factor by factor and overall. Their individual results are compiled and then presented to the group, which is tasked to come up with a common set of ratings. Those whose ratings vary from the majority initially are invited to provide their rationale and through discussion the group arrives at consensus ratings. The dialogue among participants leads to a common frame of reference as to what constitutes adequate performance, performance that significantly exceeds standards, and performance that does not fully meet standards. This consensus is used to develop rich definitions of what performance at the various rating levels looks like. These standards then are used in the future to guide ratings.

When performance rating distributions across managers are first compared, it is often found that significant differences exist. A metropolitan utility that did a “dry run” of a new performance appraisal system found that the percentage of the 1,300 employees receiving “outstanding” ratings varied from one percent to 60 percent across departments. Even though it is likely that distributions will have some legitimate variation, due to the mix of the workforce in each unit, this large difference is also likely to be caused by a significantly different understanding of what rating levels are intended to mean. If these differences are not analyzed and addressed, the result will be inequitable ratings across departments and a resulting inequity in the administrative consequences. Employees are inherently suspicious that peers working for other managers are getting higher ratings for the same level of performance, due to differences in the level of expectations held by managers. This can erode satisfaction and can lead to legal action if pay or other outcomes are tied to appraisal ratings.

An Application of Frame of Reference Training

A city with a population of 50,000 in the western part of the Chicago metropolitan area implemented a frame of reference training program. The city of Elmhurst and Pontifex Consulting had worked together to develop a new classification and compensation system to ensure pay administration was equitable and competitive. The city then created a new performance appraisal format that would be used for all employees. Management was committed to ensure that equivalent performance standards were being used by all managers, so that appraisals were equitable across departments. Employees not covered by a collective bargaining contract had their pay actions tied to performance appraisal ratings, thereby making equity even more important.

Elmhurst management decided on using a three-level rating system, using the following descriptors:

 

It is common to have five rating levels, which necessitates having an “Outstanding” and an “Unacceptable” rating. However, some believe that unacceptable performance should be dealt with during the year and result in performance improvement plans that should either correct the deficient performance or initiate administrative action if it does not improve. If an Outstanding rating is used, it can be described as: “Individual regularly made exceptional contributions that had a significant and positive impact on the performance of the unit/ organization. Results consistently exceeded performance standards and objectives significantly over a sustained period. Employee has mastered all job-related skills and possesses a broad range of capabilities. Provides a model for excellence and helps others to do their jobs better.”

If a three-level approach is used, the organization should ensure that an employee performing consistently in an outstanding manner is acknowledged in other ways, in order to communicate that extraordinary contributions are valued. In the private sector, cash awards may be used, but in public sector organizations, this is less common. The culture of the organization will have an impact on the approach used to reward those contributing at the highest level. And some jobs are controlled by set procedures, making it difficult for an employee to produce results that necessitate an “Outstanding” rating.

The Training Process

A two-part training program was developed. Part one consisted in a series of group sessions with managers and employees in attendance at the same sessions. The principles of sound performance management were presented, and open discussion was encouraged. The decision to have all types of employees in the sessions together was based on the desire to openly discuss the challenges associated with performance appraisal and to ensure everyone knew that they had all heard the same thing. Organizations often train managers and employees separately, leaving the door open to the suspicion that not everyone heard the same thing.

Once the fundamentals training was completed, the consultants and HR met with department heads and their key staff members to begin work on part two of the training. The Police, Fire, Public Works, and Administrative department directors were each provided with an explanation of the frame of reference approach and were asked to develop a set of dummy appraisals that could be used in the training. The appraisals were written with commentary provided on job-related results and behaviors but no ratings. They consisted of examples of performance at all levels, which would require participants to decide how an overall rating would be arrived at. Once the models were completed for each department, they were distributed to the members of the management/command structure (everyone who would be doing appraisals). The managers rated each of the model appraisals based on the commentary provided for each rating factor and for the overall rating. The results were used in the frame of reference dialogue sessions with all managers of each department participating in the same session. The objective for these sessions was to establish a common understanding of how each of the rating levels should be interpreted and to ensure that the descriptions of what each level of rating meant on the appraisal form were adequate to promote consistency across raters. The appropriate approach to combining individual factor ratings to produce an overall rating was then agreed upon.

In order to ensure the departments were aligned with each other, a review session with the city manager and department heads participating was then held. Definitions of each level of performance were developed through dialogue. It was agreed that forced distribution of ratings should not be imposed, since a different mix of employees performing at the various levels is expected across smaller units. Forced distributions are arbitrary and have not held up in legal proceedings. On the other hand, expected distributions may be communicated as guidelines.

Managers should be informed that if their actual rating distributions vary significantly from the guidelines, supporting evidence will be needed to support them. A further step that can improve alignment is the use of “calibration sessions” at each level. Supervisors meet with their manager as a group and align their ratings; managers meet with their director to align their ratings; and so on. During these alignment sessions, everyone has an opportunity to recommend refinements to the rating level descriptors to make them as clear and consistent as possible.

Conclusion

Each organization must concern itself with internal equity when performance standards are developed and then used to appraise performance. Individuals vary in their interpretation of what constitutes adequate, superior, and poor performance, and in order to minimize the inconsistency across raters, steps must be taken to moderate these differences.

Gaining acceptance by employees that ratings are fair and accurate will always be a challenge. Cognitive bias leads to people believing they are better performers than they are. This form of bias cannot be completely corrected. Recognizing that the bias exists and discussing it with employees can help to alleviate some unwarranted dissatisfaction with ratings that are believed to be too low. People can accept to some degree the difference of opinion between raters and themselves, but suspicions that inequity exists across raters will erode acceptance. When inconsistency across raters is apparent, this reality (or perception) elicits strong feelings of inequity, but organizations can take steps to minimize the amount of distortion it causes. Frame of reference training has been shown by research to be the most effective way to do that.

James Grabowski is city manager, Elmhurst, Illinois (james.grabowski@elmhurst.org).


 

Robert J. Greene, PhD, is consulting principal, Pontifex, (RewardSystems@sbcglobal.net).


 

Peter Ronza is CEO, Pontifex, (pronza@pontifex-hr.com).

 

Valerie Johnson is human resources director, city of Elmhurst, Illinois (valeriejohnson@elmhurst.gov).


 

Lynn Lohman is human resources manager, city of Elmhurst, Illinois (lynn.lohman@elmhurst.gov).