287 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
Criterion Development: The Unknown
Power of Criteria as Communication Tools
WIEBY MM. ALTINK, COERT F. VISSER AND MICHIEL CASTELIJNS
Adviesbureau Psychotechniek Utrecht B.V., Utrecht, The Netherlands
Much has been written about the development of criteria: the standards against which personnel are hired and appraised. We know that criteria should reflect the relevant tasks and responsibilities in the job and organization, that they should meet certain psychometric standards and, moreover, that they should not result in unfair discrimination (against races, gender, etc.). Many tools have been developed to derive criteria that fulfil these purposes. However, less attention has been paid to the process by which criteria are formulated. This process offer many opportunities to communicate with organization members about the goals of the organization and the required and desired performance (see also Anderson and Ostr0ff, Chapter 20 in this volume).
In our opinion, criterion development is not just a manner of applying a psychometric ‘recipe’, as many textbooks still seem to suggest. Criterion development is an activity by which important aspects of behaviors, competencies, and achievements are communicated throughout the organization. Should not all organization members (employees, managers, personnel officers, consultants) work with these important issues that motivate and direct performances within organizations? Why not then pay more attention to their role in the process of criterion development?
This chapter gives some examples that, we hope, trigger ways of thinking about how to develop criteria and the opportunities to involve organization members in this process. You might achieve the same result that a manager of one of our cliënt organizations described in the following way: ‘Now I understand why I should discuss these measures with the members of my team – they simply reflect the goals to which we are heading!”
International Handbook of Selection and Assessment, Edited by N. Anderson and P. Herriot ? 1977 John Wiley & Sons Ltd.
288 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
The question ‘Why should we develop criteria?’ seems redundant in the case of selection and appraisal procedures. Criteria, or concrete measures of performance, define how a candidate fulfils and –in the case of selection- will perform in a certain job. It is a know fact: criteria play a double role. They formulate in a specific manner what individuals have to do, and they also give standards against which we can evaluate whether these goals have been reached. Criterion development follows upon the process of job analysis. First, we state which tasks have to be performed, specify the necessary skills hereto and then we construct specific standards that are necessary to measure, lead, coach and evaluate performance. To give an idea of what criteria are, we show some examples in Figure 14.1. These examples, which we encountered in our practice as external human resource management consultants, show in addition that we sometimes have to deal with behaviors and performances that are not easily described, measured or predicted. Criteria might be formulated on a more abstract or concrete level. They may sometimes concern tasks, competencies, behaviors or capacities and personality requirements. The level of abstraction and the construct to which criteria refer depend on the specific purpose for which criteria are developed and also on several psychometric considerations. However, a central feature of criterion constructs is that they can be transformed into observable skills, behaviors and habits, which can be communicated about with employees to observe, predict and evaluate performance and (future) achievements.
CHANGES IN ORGANIZATIONS AND THEIR ENVIRONMENT AND
THE WAY THEY AFFECT CRITERIA TO BE CHOSEN
Many organizations are experiencing drastic changes in their environments. Technological developments make new products possible. As a consequence of the increasing speed of technological developments product cycles are getting shorter and shorter. New markets are emerging as well as new customer needs and demands. Unfortunately, it is no longer sufficient to be competitive in one’s own national market; world class competitiveness is the new aim in order to achieve sustained success. Workers today are generally more highly skilled and better educated than before. They put different demands on organizations. They have less tolerance for hierarchy, they demand high quality of work and the opportunity to develop themselves, to have a meaningful role and to enjoy their work.
International Handbook of Selection and Assessment, Edited by N. Anderson and P. Herriot ? 1977 John Wiley & Sons Ltd.
289 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
Productivity Products made/delivered within a certain time span
Oral communication Messages communicated to others in a clear and understandable way
Cliënt service Number of complaints by clients; acquired contacts within a certain time span
People management Motivation/productivity of personnel, creation of ‘team spirit’, skills that relate
to enhancing individual and team performance.
Figure 14.1 Examples of criterion constructs and descriptions
Organizations have to be able to adapt to and to cope with the new organizational environments and market demands, which are getting stricter and different. Goals which many organizations set have to do with achieving higher quality, higher speed, lower costs, dependability and adaptiveness (Flood, Gannon and Paauwe, 1995). Finding the right sources of competitive advantage is an important quest. Nowadays, human resources management (HRM) is seen as probably the major source of competitiveness (Pfeffer, 1994; Flood Gannon, and Paauwe, 1995). Human resources are valuable, scarce, difficult to copy and difficult to substitute. Also there are several sources of competitive parity. They imply developments that
organizations have to keep up with in order to keep up with their competitors. Examples are applying new technologies, flattening organization structures and making processes more simple and flexible.
Several drastic changes in work are occurring related to the above-mentioned changes. There are changes related to technological developments. The work place of today is getting informated. New tasks emerge in the application of new technology as well. However, there is also a process going on of sub-tasks, tasks, and jobs disappearing. Knowledge-intensive applications like computers and robots replace human work to some extent. Here there is a parallel with Fordism, the mechanization of work. Also emerging are new ways of
communicating. Of course there is the ‘electronic highway’. People can communicate easily with each other even though they may be far apart. Teleworking is also a new possibility. The need for different ways to management people leads to major changes in organizations too. Human resources are seen as a major factor for achieving company success. Organizations put more emphasis than before on developing a strategic HRM. Making sure that the work force is valuable is a major concern. The development of individuals within organization is too. People demand development opportunities on the one hand; they can be made more valuable for organizations on the other hand. Another way of improving HRM practices is to improve the consistency of different elements, like selecting people, appraising, developing and rewarding them. Line managers are being given the responsibility for these HRM tasks. Topics that get much attention are the flexible work force, broader tasks, more investment in training and development, self-management, and empowerment, management by objectives, ouput-oriented appraisal, flexibility in rewards and raising the internal and external mobility of employees.
International Handbook of Selection and Assessment, Edited by N. Anderson and P. Herriot ? 1977 John Wiley & Sons Ltd.
290 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
There are also some structural changes in the way work is designed. First, there is a process of revision of labor division and simplification of work processes going on (Visser, Altink and Algera, Chapter 21 in this volume). Secondly, there is a tendency to cluster people into teams. Teams can be more flexible than individuals and are more capable of dealing with complex problems. Many organizations throughout the world are experimenting with self-managing teams (Wellins, Byham and Dixon, 1994). Temporary assignments and project work is another trend. To be very flexible and get fast results people from different disciplines work together with a specific assignment until the desired results have been achieved.
As a result of all these changes, there are some new demands on people. High priority is put on aspects like knowledge and the permanent development of knowledge. A bigger emphasis
on intellectual skills and capabilities will be inevitable. Adaptiveness is another criterion, which
will be relevant due to changing work content and contexts. People have to be willing and capable of broadening their skills. Cooperativeness is a requirement because of the emphasis on
teamwork. Workers have to be ready to commit and adapt to team goals. On the other hand, people have to be able to work independently. Employees have to be self-managing. They are
responsible for achieving the results they have committed themselves to. A more intensive co-operation and new ways of communicating places higher demands on communicative skills. To
be more self-managing, mores knowledge about business processes is required. Orientation toward
adding value to organizational goals as well as an orientation toward innovation will be
necessary to meet new market demands.
These developments clearly make the use of criteria important.
PROCEDURES FOR CRITERION DEVELOPMENT
Because of their importance, work and organization psychologists have paid much attention to the development of criteria throughout the years. Algera and Greuter (1989) discuss aspects that have to do with theoretical and conceptual points of view. For instance, they discuss such issues as ‘Should criteria be defined in terms of behaviors, capacities or performances?’; ‘Should we formulate an overall criterion that reflects all performances, or should we apply
several criteria that are more or less independent from each other?’ In addition they discuss the meta-analytic research that indicates the relevance of job criteria against single jobs and jobs overall (see also Landy and Rastetgary, 1989; Guion, Chapter 13 in this volume).
Nevertheless, nowadays a major problem for criterion development is that jobs are no longer clearly ‘visible’, and task activities are rapidly changing. Think, for instance, about work in the IT or health care sectors. Both types of job activities could be recognized clearly in the 1980s. Nowadays these jobs are changing rapidly, and will continue to change in the coming years. From a ‘specialized role’ workers are asked to perform a more ‘broad’ role: they have to service clients and they should have knowledge about many issues beside their own
291 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
specialism. The project work now typical of many jobs is one of the best examples of these developments. Criterion development not only relates to the derivation and description of important job performances, but also to question like: ‘ What is the difference in performance between types of jobs?’; ‘Do these differences relate to the content of work activities, or to the organization in which work is performed or the market and clients that are served?’
Looking at the knowledge available to tackle the above issues, we think that the work of organizational psychologist, and the work of other authors in the field of criterion development, form a sound basis for a discussion about the status of criteria within organizations. These authors have described major principles that relate to the definition and measurement of criteria. A summary of these is given by the performance modeling approach introduced by Martin Greuter (1989). A design procedure is presented by which psychometric pitfalls can be overcome, and which provides direction to the development of adequate criteria for selection and appraisal. This procedure is described in Figure 14.2. It states that you first have to define the problem that is the goal of criterion development (e.g. in the case of selection, the prediction of future work performance of applicants; in the case of appraisal, evaluation of performance of personnel). The second step directs you to defining requirements, such as predictive validity, practicality, and utility. Then you define relevant criterion variables and you formulate the instruments that enable you to predict / evaluate the performance level on these criterion variables. The steps considering structure, form and parameters (steps 4,5, and 6 in Figure 14.2) relate to the combination of the scores obtained or a more ‘clinical’ model (see also Cascio, 1992).
Figure 14.2 Developing performance models. Source: Greuter (1989).
In deriving criteria, a clear vision on the design procedure seems inevitable. The simple reasons for this is that we do not develop criteria in ‘in a vacuum’: we have a special purpose. Sometimes this purpose may be more ‘widespread’: in some cases we would have to develop criteria that serve several purposes (as for instance job redesign, training, coaching,
292 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
development and change). Sometimes criterion development may be directed to specific objectives, as for instance the hiring or minority members, appraisal of certain performances, etc. As Greuter states, the design cycle has the advantage that ‘ it stipulates why required attitudes are relevant for adequate job performance’ (Greuter, 1989, p. 175). This commitment is, in our opinion, extremely important because performance and achievements can only be guided when all organization members understand which direction has to be followed, what they have to do to achieve theory own- of team goals, and the measure against which they will be evaluated.
Unfortunately, the procedures for the development of criteria do not give specific guidance when we have to derive criteria for jobs that are less stable with regard to the content of work activities. In fact, we often see that the process of criterion development doe not go any farther than the identification of general requirements that give little opportunity to coach and direct performance. Criteria, them, may still be useful to predict and evaluate performance and achievement, but they are often remote from the actual job experiences of employees and managers. In such a case there is clearly a risk of criteria ‘existing’ bit not being ‘alive’ and shared by those who are concerned.
We elaborate on this last issue in the next section, because we think that the changing nature of work asks for another view on criterion development.
PSYCHOMETRIC AND ORGANIZATIONAL CONSIDERATIONS
As we have already noted, there seem to be a feeling that we know how to develop criteria, but we are also aware that we might not tackle all the difficult issues in formulating adequate criteria. To illustrate this attitude: the personnel officers and managers that we speak to are seldom enthusiastic about criterion development. They see criterion development as a task that has to be done which leaves little room for innovation. Moreover, because it is not easy to define criteria in an unambiguous way, practitioners (but also researchers) make do with a formulation of those behaviors that can be measured, instead of the behaviors that should be
measure (Akkerman, 1989).
In addition, the number of publication about criterion development is decreasing. Our literature scan through several important journals revealed only articles in which criterion measures are evaluated against certain types of jobs, or general articles that describe how we may ‘technically’ develop criteria. In this regard the conclusions that Landy and Rastegary (1989) draw from their meta-analysis still hold: most studies pay attention to idiosyncratic criteria an less attention is paid to the way in which criteria are actually derived and developed. The recent discussion about behavioral competencies (see Chapter 17) gives new insight but has not yet led to a clear statement of how criteria should be developed, so that they actually work.
293 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
We believe that it is important to realize that the discussion about criteria has up to now-been a purely psychometric one (think about the statement of the manager in our introduction). Requirements are set for criteria: they should be relevant (cover job and tasks), give opportunities for the management of performance (concrete, non-deficient, motivating) and be adequately measurable (reliable, valid, and acceptable for all, free from unfair discrimination). In addition to this, other requirements may be set, like the requirement that criteria should be ‘up to date’: they should be regularly reviewed in response to organizational change (see Figure 14.3).
Reliably measurable Should measure behaviors and outcomes in an objective way
Content valid Rationally related to the job performance activities
Defined specially Important behaviors and outcomes should be included in a comprehensive
Independent Important behaviors and outcomes should be included in a comprehensive
Non-overlapping Criteria should not overlap
Comprehensive No important behaviors or outcomes should be omitted
Accessible Criteria should be phrased and name in a way that is comprehensible
Compatible Criteria should fit in with an organization’s goals and culture
Up to date Criteria should be regularly reviewed in response to organizational change
Figure 14.3 Some requirements for criteria
To think that these psychometric requirements make up the total of the process or criterion development seems mistaken, however. Criteria are the foundation of procedures of selection, appraisal, development, job redesign and, last but not least, organizational change and development. Therefore they constitute concepts that can be used to organize employees, to direct their attention, and to lead their performance. They are also concepts to direct managers to their task, which - in our opinion – can be made more stimulating by giving them the tools
to discuss work with employees in a constructive way.
To achieve this broader perspective it is vital to introduce organizational practices in the process of criterion development. For example, if the organization is using the method of goal setting to motivate and reward performances, then a similar type of procedure could be applied to develop criteria for selection. The advantage is that expectations are clearly communicated in the first encounter between organization and candidates. When the organization consists mainly of professionals, having a high level of expertise in performing their jobs, it might be more fruitful to apply a procedure by which principles of self-management are introduced in the derivation of criteria. In this instance opportunities are
294 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
made to delegate performance management. In sum, the way in which procedures for criterion development are applied should be in line with the goals of the organization and its common managerial practices. This does not imply that we would like to throw away psychometric considerations in the process of criterion development. Relevance can easily be combined with psychometric soundness, as we will illustrate by some examples that we encountered in our consulting practice. These will be discussed next, after the introduction of a general model that summarizes how to combine psychometric procedures for developing criteria with procedures for communication about criteria within organizations.
CRITERIA IN RELATION TO ORGANIZATION PERFORMANCE
Do you recognize the following? You ask your boss, or the personnel officer, for a clear-cut
description of the criteria against which you will be measured. Mostly, at least in our experience, the response is a ‘glazed look’. The term, or name, ‘criterion’ is often exclusively limited to the language of work and organizational psychologist. ‘Subtle’ distinctions are even made between conceptual and operational criteria Conceptual criteria define behaviors and performances at a rather abstract level, while operational criteria describe behaviors and performances that are concretely measurable. What we mean by these terms is often not understood by others, though. In fact, criteria are often take to mean those aspects on which an employee is hired or the instruments by which his or her performance will be predicted (predictors and instruments).
Nonetheless, when you ask your manager, or personnel officer, or employee to state which goals are important in the (or their) work, they come up with very specific statements. So, criteria are recognizable in practice, simply because people have to run a factory, or an organization. The point is to find the right way to communicate about criteria and, subsequently, to define them in an objective and unambiguous way. The latter is extremely important because nowadays jobs are often not easily recognizable (Bridges, 1994). However, this does not mean that criteria will cease to exist. Criteria for group performance may differ from criteria for individual performance, but they still exist. The reason for this is simply that performance has to be managed / guided / predicted within organization, and this cannot be done except by the use of criteria. The people who are performing these management tasks might change though. For instance, self-managing teams may formulate their own criteria. Even more, they might apply their own criteria instead of the criteria that their manager has set. Clearly, there is more need, but also opportunity, to discuss criteria.
It is a reality: criteria change over time, and are not fixed. Criteria have become more and more matters of discussion – and this situation will only be more the case in the future. For instance, the criteria name in Figure 14.1 are nowadays designed in a totally different manner. Cliënt service means that you should have insight into your product / service, should have ‘content’ knowledge and that you should be up-to-date regarding your experience with the
295 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
products of competitors. A simple criterion like ‘oral communication’ is nowadays formulated somewhat as follows: can give insights in a clear manner, knows what he or she is talking about, shows ‘sensitivity’ and knows how to trigger the motivation of others. In sum, criteria are intertwined with organizational goals and change over time, sometimes very rapidly. One of our major cliënt organizations changed the criteria for their total work force (in total about 4000 employees) within a time span of a couple of months, simply because an organizational change was need due to international market competition.
How can we develop criteria in a changing situation and still preserve the psychometric qualities that criteria should meet? In this case we think that it is useful to refer to the fact that criteria can be defines at more than the individual level of analysis: for instance, at the level of organizational change and development and at the level of measurement and prediction. We summarize this point of view in Figure 14.4. In this figure we indicate which roles criteria can have and which requirements should be made for their development in each of the phases. In fact, this model should be seen as an elaboration on the design cycle of Greuter, but this time with an exclusive focus on the process that is followed.
Goals of criteria Definition Measurement Communication
Goals for organization Organizational level Tasks derived from Acceptable for management
(development and change) organizational goals and employees
ble Reflect organizational Goal-directing
Means to enhance Job/group level or project Performance: objective/fair Clear procedure;
performance (motivating team Acceptable (comments are
and rewarding) Possibility for concrete possible)
Relationship to reward feedback
Direction of performance
Behavior development and Career paths, relationships Behavioral clusters that Win-win situation
coaching between tasks indicate relationships Relationship between past.
Present, and future should
Communication about Should direct development be clear
Instruments for Operational Psychometric requirements Relationship to task of
measurement of criterion measurement clear
performance To be connected at Relationship to
selection/appraisal development/appraisal Opportunity for self-
Figure 14.4 Criterion development within the content of organizational activities
1. First, there should be a clear goal (this seems self-evident, but we still encounter situations
in which criteria are formulated from an objective that is different from that which they
actually have to fulfil). At this stage we should not worry about psychometric
requirements too much. More important is the discussion that we start about criteria. An
applicable tool is to think about criteria in terms of metaphors (examples are ‘relationship
directed’; ‘drive’; ‘insight’). We define activities that related to the business of the
296 Wieby M.M. Altink, Coert F. Visser and Michiel Castelijns
organization and that reflect in a clear-cut way the products or services that have to be delivered, as well as the relationship with our clients and competitors. In fact, this stage implies image building as well. We advise that a connection be made at this point with the public relationship activities of the organization. The reason is that the new criteria will become more visible to all. It goes without saying that an integrated vision about management and performance is needed in this phase or will have to be developed.
2. In the next stage we proceed to a rough specification: which methods or procedures do we intend to choose for evaluation or prediction of performance? To what extent is the participation of other organizational members in the criterion development necessary in order to obtain commitment? Are sustaining facilities available to improve job performance? In this stage it becomes important also to think about issues like objectivity, reliability, validity, and fairness. For instance, if it is the case that an organization has appointed few members of minority groups, it is important to discuss how these groups could be integrated into the organization. Perhaps other facilities should be provided first, in order to enhance the job performance of these groups? We often see that employees cannot meet their standards simply because resources are lacking. In this instance we are not referring only to materials, machinery, etc. but also to managerial practices by which feedback is given on performance and successes and failures are discussed. Such omissions are most easily detected by starting up a communication structure between employees and managers.
3. Thereupon we go to the phase in which we pay attention to the development of behaviors and competencies. This implies of course that we have to specify the relationship with education and training, and courses that are available / offered. In this phase it is important to think about issues like trainability, possibilities for coaching on the job, etc. our experience is that this phase is often neglected, simply because another department will take care of this issue. If criteria are being developed that will be used throughout the organization, this issue should be tackled in this phase because otherwise measurements may be constructed that prevent employees from developing their skills. If future developments are difficult to predict for the organization, then it still seems useful to discuss several scenarios. Clarity is needed, even if only some possible future directions are described., to ensure that employees are aware of their future performances and react cooperatively to changes.
4. Finally, we define the criteria that will be applied in selection and appraisal procedures. At this point matters such as psychometric soundness become important, because concrete instruments have to be constructed that measure criterion performance. At this stage communication with workers / candidates and their (future) managers becomes more and more vital. Operational measurements should leave enough room for clarification of mutual expectations and involvement. For instance, the issue of ‘self-selection’ should be
addressed at this stage by choosing adequate tests and questionnaires. In this phase it is important that operational measures are chosen that are flexible. This means that if job
International Handbook of Selection and Assessment, Edited by N. Anderson and P. Herriot
? 1977 John Wiley & Sons Ltd.