9.3 CHARACTERISTICS OF EVALUATION

An evaluation may become useless if certain criteria are not respected. The following five characteristics, if used carefully when conducting a Competency Check, will result in an accurate and effective form of evaluation.

9.3.1 Reliability

Reliability ensures consistent results. As applied to the competency check, this would mean that two identical performances should result in the same score. Human factors can have a significant effect on reliability. Some of these factors are:

  1. Fatigue: insufficient sleep or rest prior to the competency check;

  2. Emotions: work or home personal problems;

  3. Health: cold or flu, etc.

  4. Time of Day: very early in the morning, or late in the day;

  5. Distractions: noise, interruptions, etc.

The ACD should be conscious of these factors and attempt to reduce as many variables as possible. He may recognize some of these factors as a reason for some lack of accuracy in the candidate’s performance. The ACD should also be aware that his/her ability to accurately assess the candidate’s performance could be affected by these same factors.

Another factor that may affect the reliability of an evaluation is to allow learning to take place during the check. It must be emphasized that testing for the purpose of certification must remain clearly removed from teaching. For example, if worded improperly, questions may lead the candidate to the correct answer. If given a second or third attempt, the candidate may finally answer adequately because of the repeated practice. For this reason, a scenario will not be repeated unless one of the following conditions applies:

  1. Discontinuance: Discontinuance of a scenario for valid reasons such as emergency, mechanical, diversion, weather issue or other procedure necessary to modify the original plan;

  2. Misunderstood Request: A legitimate instance when a candidate does not understand an ACD’s request to perform a specific task. A candidate’s failure to know the requirements of a specified request is not grounds for repeating a task.

  3. Other Factors: Any condition where the ACD was distracted to the point that the candidate’s performance of the task (telephone & radio calls, interruption from another department, etc.) could not adequately be observed.

These provisions have been made in the interest of fairness and safety and do not mean that instruction, practice, or the repeating of an item or task unacceptably demonstrated, are permitted during the evaluation process.

9.3.2 Validity

Competency Checks are valid if they measure what they are supposed to measure and nothing else. Assessment of items must remain within the bounds of the appropriate DCC and the guidelines provided in this manual. The scope of the DCC must be such that when candidates pass, they have met the required standards for the issuance of the certificate or its renewal.

9.3.3 Comprehensive

A DCC is comprehensive if it contains a sample of all course material and measures each area of skill and knowledge required to ensure the skill requirements are met.

9.3.4 Discrimination

During the conduct of a DCC, discrimination enables the ACD to detect different levels of achievement among candidates. Discrimination separates standard performance from above and below standard performance. A 1-4 marking scale is designed to reveal how candidates perform and allows for a greater degree of discrimination than one that simply distinguishes between pass and fail.

9.3.5 Objectivity

Objectivity ensures the ACD’s personal opinions will not affect the outcome or assessment of the competency check. Marks awarded must be made in accordance with the applicable DCC criteria. Despite the fact that ACDs must strive to remain objective, competency checks may be marked to some degree on a subjective basis when the ACD is an experienced dispatcher, has sound and adequate background knowledge of the evaluation process and has the expertise to accurately assess candidates without prejudice.

Date modified: