Advanced Qualification Program Evaluator Manual TP 14672

CHAPTER 9 – CONDUCT OF VALIDATIONS AND EVALUATIONS

9.1 General

9.1.1 During a validation or evaluation, except as provided in 9.1.3, evaluators shall refrain from teaching or briefing the candidate on the correct completion of an exercise or from taking any action that will prompt the candidate to take a specific action.

9.1.2 During MPVs, LOEs and OEs repeats are possible. A debriefing of why the manoeuvre(s) was unsatisfactory is permitted. However, the repeats must occur with no training, practice, or coaching.

9.1.3 During MTVs repeats are possible, and training is permitted. Once the training has been completed the candidate must be advised that a validation/evaluation assessment will be made. During the validation/evaluation assessment evaluators shall refrain from teaching or briefing the candidate on the correct completion of an exercise or from taking any action that will prompt the candidate to take a specific action.

9.1.4When acting as ATC for the purposes of a validation or evaluation, evaluators shall:

  1. provide clear and unambiguous clearances and instructions that are appropriate to the area of operation and the aircraft involved;
  2. use standard ATC terminology to the extent possible based on their knowledge and experience;
  3. provide assistance that would normally be available from ATC when necessary to facilitate the objectives of the exercise or when requested by the crew and doing so will not compromise those objectives, such as for instance, providing vectors for an approach, when the script does not require a full procedure, or when requested by the crew to allow time to complete a checklist or evaluate a malfunction; and
  4. not use initiatives intended to prevent the crew from making a mistake, such as for instance, intervening when it appears that a crew will not comply with an acknowledged clearance, or requesting confirmation that the correct facility is tuned and identified.

9.1.5During Online Evaluations (OEs), evaluators are part of the crew (whether in the jump seat or in a pilot seat), and as such, must take appropriate action to ensure a safe flight and that no violations occur. See section8.11 regarding evaluator feedback during OEs.

9.1.6Validations and evaluations may induce tension and feelings of apprehension in even the most experienced pilots. The evaluator shall attempt to reduce apprehension and create an environment in which a true demonstration of ability can occur.

9.1.7In order to minimize sources of stress and distraction during a validation, evaluation or an AQP evaluator monitor, admittance should be restricted to the following individuals, where required:

  1. designated pilot flying (PF);
  2. designated pilot not flying (PNF);
  3. designated second officer or flight engineer, or Cruise Relief Pilot (CRP) if required by the aircraft type/SOPs;
  4. designated TC Inspector or evaluator conducting the event;
  5. designated TC Inspector or QAE monitoring the event, and any other person designated by the air operator who is required to participate in the event;
  6. Evaluator under training, approved at the discretion of the TC Inspector or evaluator; and
  7. where the event is being conducted in a simulator, the simulator operator.

9.2 Evaluation Philosophy

9.2.1 Technologies employed in the design, manufacture and maintenance of aircraft have resulted in improved aviation safety as measured by the steady decline in accidents attributable to these factors. While the introduction of human factors training and crew resource management have had a positive effect on safety as well, it is recognized that this area must continue to evolve if we are to realize a reduction in the number of accidents attributable to flight operations.

9.2.2 Today's strategies continue to focus on the flight crew yet more attention is now being paid to organizational factors (within the aviation company as well as outside organizations such as air traffic control) as indicated by the introduction of safety management system requirements.

9.2.3 Recent developments in assessment techniques focus on threat and error management strategies and performance where it is recognized that from time to time, errors or deviations from standard practices will occur. While not desirable, it is a fact that errors will be committed by flight crews, or by others associated with flight crews (operational or maintenance control, air traffic, etc.), and that these errors, if not recognized and managed effectively, could have disastrous results. Evaluators must focus on how the crew:

  1. recognizes threats (poor weather, aircraft unserviceabilities, unruly passengers, difficult ATC clearances, terrain, distractions, or challenging approaches, etc);
  2. uses effective strategies to deal with these threats (personal flight discipline, knowledge, flying skill, rigorous use of SOPs, awareness, communication of threat, use of all available resources, etc);
  3. avoids errors using SOPs and good CRM teamwork;
  4. recognizes errors when they occur (using good communication, monitoring and feedback, and situational awareness); and
  5. mitigates the effects of errors when they occur (making positive corrections, advising ATC, trusting on-board warning devices such as altitude alerters, traffic collision avoidance systems (such as TCAS) and ground proximity warning systems (such as GPWS), and obtaining the assistance of additional resources to deal with the situation).

9.2.4Threat and error management assessment techniques require the evaluator to go beyond simple error detection. Evaluators must recognize the potential safety threat for any given situation or commission of errors, and then determine the effectiveness of crew actions in managing the situation so as not to jeopardize safety.

9.3 Flight Crew Concept

9.3.1 Validations and evaluations on multi-crew aircraft shall be conducted under the flight crew concept and not on an individual basis. (This does not apply to SKVs, which are individual assessments of knowledge.)

9.3.2 During a validation or evaluation, a manoeuvre or event set may involve duties and/or responsibilities for crewmembers other than the pilot flying (PF). A sequence that is graded as "unsatisfactory" for the PF may, due to inappropriate action on the part of other crewmembers (i.e., the pilot not flying [PNF]), be rated as "unsatisfactory" for the PNF also. In such a case, it is possible that an assessment of "unsatisfactory" may be given to more than one crewmember involved in the same flight sequence.

9.4Assessments

9.4.1 It is impossible to define all instances when a particular manoeuvre or event set should be given a specific rating; however, it is possible to examine each sequence and test its validity against the definition for each rating. By applying this test to all exercises, standardization can be achieved in assessments. Each sequence of the validation or evaluation, including any errors or mistakes, shall be evaluated with respect to the rating definitions.

9.4.2 Common errors and rating assessments are described by a variety of adjectives. Terms such as (un)acceptable, (un)satisfactory, timely, safe, minor, slight, brief, lack, inadequate and excessive are used to describe the candidates' performance. It is difficult to define these adjectives objectively; however, the dictionary definition may be used to provide amplification of meaning and thereby standardization in application. Terms such as (in)complete, (in)correct, exceed and failure are more finite and may be described objectively by referring to the appropriate regulation, AFM or company procedure.

9.4.3 The air operator's approved Qualification Standards provide the basis for assessments. Evaluators must use their knowledge and experience in conjunction with the rating definitions to arrive at their assessments.

9.5Pre-Flight Briefing – Validation or Evaluation Conducted in a Simulator

9.5.1 A pre flight briefing to the candidate is mandatory. It must be sufficiently detailed to avoid failure due to the candidate's misunderstanding of standards or limitations expected by the evaluator.

9.5.2 The briefing for a validation/evaluation conducted in a simulator shall include or state:

  1. the mandatory items to be demonstrated during the validation/evaluation;
  2. the probable duration of the validation/evaluation;
  3. the requirement to operate the simulator in accordance with flight manual requirements and within acceptable tolerances (refer to section10.6 for tolerances);
  4. where known to the evaluator, any differences between the simulator and the aircraft that may affect the performance of the flight crew;

    Note1:Some examples of this would be cockpit configuration and layout, instrumentation, power plant simulations, warning and alert display systems, FMS databases, electronic monitoring systems, etc.

    Note2:Training on differences between the simulator and the aircraft is required to be included in the training program. Evaluators may not be aware of differences and will evaluate flight crew performance with the expectation that any differences will have been covered during training.
  5. simulator safety features;
  6. the identification and role of the Pilot in Command and Second-in-Command, if applicable;
  7. the requirement for the candidate to demonstrate any normal or emergency procedure applicable to the aircraft and that the candidate's technical performance will be assessed in accordance with the air operator's approved qualification standards with reference to the following:
    1. aircraft flight manual, aircraft operating manual or pilot operating handbook;
    2. CAR Part VI and VII;
    3. Operator's operations manual; and
    4. Operator's SOPs;

  8. that if the runway environment is seen at DH or MAP (MDA for stabilized approaches), then the crew should land, otherwise a missed approach should be carried out;
  9. that the crew should treat all malfunctions as real and that should a simulator fault occur, the evaluator will advise the crew immediately;
  10. that normal crew coordination is expected in accordance with the aircraft AOM/AFM or company SOPs as applicable, and that an emergency situation caused by an incorrect or inappropriate action or response on the part of the candidate will not be corrected by the evaluator;
  11. that multiple, unrelated failures will not be required, but the candidate must be prepared to take corrective action on related failures (ex. loss of hydraulics or electrical supply due to a failed engine);
  12. that for the purpose of the validation/evaluation, the weather will vary and may be at or below the weather minima for the approach being carried out, and that the onus is on the flight crew to determine if the departure weather is suitable; (The evaluator will control the visual system to minima appropriate to the exercise being conducted.)
  13. if the crew requires more time to complete checklists or briefings, that they should ask for a hold or delaying vectors and that the evaluator will make every effort to accommodate the request; and
  14. the circumstances and protocols for repeats.

9.6Pre-flight Briefing – Online Evaluation (OE)

9.6.1 A pre-flight briefing to the candidate(s) is mandatory. It must clearly detail what is expected from the candidate(s) and what the candidate(s) can expect from the evaluator.

9.6.2 The briefing for an OE shall include or state at least the following information:

  1. that the OE will continue from check-in to defect reporting at the end of the flight(s);
  2. the number of flight legs and whether they will be flown as PF or PNF;
  3. that normal crew co-ordination and the use of SOPs will be required;
  4. the role of the evaluator in terms of crew duties and oral questioning;
  5. the emphasis on command, decision-making and the use of CRM principles;
  6. that the evaluator may ask technical questions concerning aircraft operations, rules of the air and ATC procedures, SOPs and the operator's Flight Operations Manual;
  7. the circumstances and protocols for repeats; and
  8. that safety is the number one priority during the OE.

9.7Debriefing Procedures

9.7.1 It is mandatory to carry out a debriefing following every validation and evaluation. The debriefing should highlight the strengths and weaknesses of the candidate(s), and be carried out in a positive, non-confrontational manner. The evaluator should always remember that the purpose of any validation or evaluation is to promote the safety of the traveling public, and conduct the debriefing accordingly. The debriefing should promote learning and increase the knowledge and confidence of the candidate(s). Debriefings should be of a reasonable duration corresponding to the performance.

9.7.2 As soon as the evaluator knows the outcome of the validation or evaluation, he or she should advise the candidate(s). Some empathy and discretion may be required for unsatisfactory assessments.

9.7.3 The following items are mandatory to debrief after every validation or evaluation:

  1. any items assessed as "unsatisfactory" or similar;
  2. any written comments made by the evaluator;
  3. anything the evaluator considers to be a safety issue.

9.7.4It is recommended that evaluators use a self-debrief method as much as possible for all successfully completed validations and evaluations. This method focuses on pilot participation, with the evaluator taking on more of the role of a facilitator. NASA has developed the CRM, Analysis and Line Flying (C-A-L) method of debriefing for airline validations/evaluations using these principles. The goal of the facilitator (namely the evaluator) is to assist the crew to bring out CRM issues that may have led to errors or poor performance, analyze why that performance occurred, and then tie it in to line flying. For each sequence going through the C-A-L process, the end result is a discussion about how the sequence can be improved and how to avoid similar errors on the line.

9.7.5Focus your debriefing as much as possible on CRM issues such as leadership, workload management, situational awareness, communication, decision-making, monitoring and feedback, conflict resolution and crew performance. Normally, technical errors have a root cause in one of these CRM issues; hence, identification of, and discussion about the errors will help the crew avoid these errors in the future.

9.7.6Evaluators should make a conscious decision to highlight strengths and reward good performance during their debriefings. While it is sometimes easier to concentrate on the negative (a sign of the "error detector"), the debriefing will have more impact if good performance is recognized and crews complimented. This will often set a positive tone for the debriefing and open the crew's minds to areas where their performance can be further enhanced.

9.7.7Evaluators should ensure that they differentiate between SOPs and techniques during the debriefing. They may suggest techniques, but must insist on SOPs being followed. Recommendations regarding techniques may be made at the evaluator's discretion.

9.7.8Every briefing and debriefing should end by asking for questions so that misunderstandings can be clarified right away, and the candidate(s) have the opportunity to pursue any topic in more detail.

9.7.9In the event of an unsatisfactory performance, the evaluator must advise the pilot(s) of the following:

  1. for LOEs, they have the right to appeal the assessment to the Transportation Appeal Tribunal of Canada (TATC) within 30 days;
  2. how the re-test will be conducted:
    1. for MPVs, as per paragraph 8.6.4, the candidate will only need to repeat the manoeuvres that were unsatisfactory during the initial MPV;
    2. for MTVs, as per paragraph 8.7.4, the candidate will only need to repeat the manoeuvres that were unsatisfactory during the initial MTV;
    3. for LOEs, as per paragraph 8.10.6 and section 8.10.9 (m), remedial training and another complete LOE is required. (The remedial LOE will be conducted in the same manner as a regular LOE);

  3. that the re-test may be conducted by either a Transport Canada Inspector or an AQP evaluator;
  4. the evaluator must offer to provide a copy of the Flight Test Report Pilot Proficiency Check, form 26-0249 (AppendixE) to the candidate(s); and
  5. (e)where applicable and if known, any company-specific procedures to be followed.

9.8General Assessment "Failed"

9.8.1 A LOE will receive a General Assessment of "Failed", if:

  1. the candidate's initial attempt and repeat of any event set are both unsuccessful; or
  2. the candidate is unsuccessful on the initial attempt of three separate event sets.

Regardless of the number of unsatisfactory event sets, unsafe individual or crew performance that would result in significant damage, hull loss or loss of life (e.g., crash) during a LOE constitutes a failure of the LOE.

9.8.2 A LOE is considered a jeopardy event and a failure is reported to TC. In the event of a failure, the entire copy of the candidate(s) LOE report – the Flight Test Report Pilot Proficiency Check, form 26-0249 (AppendixE) as well as the failed event set – will be faxed to Transport Canada within the next two business days for licensing action (i.e. suspension).

9.8.3 A LOE failure will also result in the individual candidates being placed into Special Tracking for at least one training period. While in Special Tracking, candidates are required to undergo another MV/LOE – instead of an MT/LOFT - during their next assessment.

9.8.4During a LOE, an "unsuccessful" assessment of an Instrument Rating related sequence constitutes a failure of the Instrument Rating and the LOE. The Type E Evaluator shall assess the LOE as "failed" at the bottom of the Flight Test Report Pilot Proficiency Check, form 26-0249 (AppendixE). Appropriate administrative action must be carried out in the suspension of any currently existing LOE and Instrument Rating in accordance with section9.9.

Note: Where the PF is assessed an "unsuccessful" grade on an Instrument Rating related sequence, the above failure and associated suspension activity may be relevant to the PNF as well.

9.8.5During a LOE, failure of a LOE related flight sequence that is not related whatsoever to an instrument flight sequence constitutes failure of the LOE only. In this case, administrative action is taken in the suspension of the currently existing LOE only. The currently existing Instrument Rating is not affected, hence remains valid.

Note:In order to be re-instated on the line, at any flight crewmember position and regardless of the type of LOE (including upgrade), remedial training and another LOE must be completed successfully.

9.8.6When a Type E Evaluator decides that a LOE will receive the General Assessment of "Failed", as per section9.8.1, the LOE shall be terminated immediately.

Note: It is possible that the failure could be for an event set flown earlier in the LOE and that the evaluator has only made the unsatisfactory evaluation based on further observation.

9.8.7Where the situation in section 9.8.6 occurs and the evaluator is an instructor pilot, the time remaining in the session may be used as training provided that:

  1. the candidate is advised at the time of failure and agrees with continuing the flight as a training flight;
  2. the evaluator is a designated company training pilot on type; and
  3. no other crewmember is being evaluated;

9.8.8Once a failed LOE has been terminated as per 9.8.6, or upon completion of the training activities described in 9.8.7, the Type E Evaluator must accomplish the following:

  1. the candidate must be debriefed on the reason(s) for failure and where applicable, on the administrative suspension procedures that will follow, including the candidate's rights to a hearing at the Transportation Appeals Tribunal of Canada;
  2. the evaluator must complete the Flight Test Report Pilot Proficiency Check, form 26-0249 (AppendixE) assessed as "failed" and submit the original to Transport Canada within the next two business days; and
  3. if applicable, follow the procedures for LOE and Instrument Rating Suspensions listed in section9.9.

9.8.9In the event of a failed LOE, the air operator shall retain a copy of the Flight Test Report Pilot Proficiency Check, form 26-0249 (AppendixE) as well as the entire LOE grade sheet on the candidate's file for a period of not less than 90 days. This will ensure that evidence is preserved in the case of a request for a hearing by the Transportation Appeal Tribunal of Canada (TATC).

9.9LOE and Instrument Rating Administrative Suspension Procedures

9.9.1 A Type E Evaluator shall carry out the following administrative procedures after failure of a LOE by:

  1. notifying the Chief Pilot and/or Operations Manager of failed items and recommendations as to corrective action;
  2. ensuring that grades of the failed LOE are recorded in the individual's training and validation/evaluation records. A report shall be completed for each LOE, including those that are terminated during pre-flight preparation, or before all event sets are completed, and the candidate is to be offered a copy of the report;
  3. immediately notifying the Transport Canada Principal Operations Inspector (POI), the RMCBA/Superintendent of Aeroplanes, or the Chief, Airline Inspection, that the pilot has not met the standards for a LOE (including the Instrument Rating where applicable). If unable to reach any of these TC officials via telephone, a voice message or a facsimile is an acceptable means of notification;

    Note: A copy of the 26-0249 form and failed event set shall be faxed to Transport Canada for reference purposes.
  4. if the Instrument Rating was failed and is still valid on the pilot's license, drawing a line through the English and French endorsements on the license and inscribing the notation: "Instrument Rating Suspended" or "suspension de la qualification de vol aux instruments" as appropriate, and signing and dating the license.

9.9.2 A TC Inspector will carry out the following administrative procedures after failure of a LOE:

  1. notifying the Chief Pilot and/or Operations Manager of failed items and recommendations as to corrective action;
  2. ensuring that grades and evaluation of the failed LOE are recorded in the individual's training and validation/evaluation records. A report shall be completed for each LOE, including those which are terminated during pre-flight preparation, or before all event sets are completed, and the candidate is to be offered a copy of the report as required by the CARs;
  3. if the LOE failure involves both the LOE and Instrument Rating as described in paragraph 9.8.4 then completing the following procedures:
    1. if the Instrument Rating is still valid on the pilot's license, drawing a line through the English and French endorsements on the license and inscribing the notation "Instrument Rating Suspended" or "suspension de la qualification de vol aux instruments" as appropriate, and signing and dating the license,
    2. issue a Notice of Suspension (form 26-0363) pursuant to subsection 7.1(1) of the Aeronautics Act in consideration of the flight test as such:
      1. name of candidate with address (same as on the license),
      2. candidate's 5802 file number,
      3. check the flight test box,
      4. date of flight test when it occurred,
      5. specify that he/she no longer meets the required standards for a LOE, including an Instrument Rating where applicable (refer to paragraph9.8.4), and the reasons why,
      6. indicate that his/her previous LOE and where applicable, Instrument Rating (including the expiry dates of each as necessary) is hereby suspended,
      7. specify conditions of re-instatement (i.e. conduct a satisfactory LOE),
      8. where the form requests an address to which the suspended document is to be returned, indicate "not applicable",
      9. specify the date (30calendar days from the date of the issuance of the suspension) when the candidate's request for a review by the Tribunal must be received,

        Note:the candidate should be verbally briefed on his/her right for a hearing at the Tribunal, and
      10. sign and date it; and

  4. if the LOE failure involves only the LOE as described in paragraph9.8.5 then the procedures in9.9.2(c)(ii) are to be followed with the exception that no reference is made to the Instrument Rating.
Date modified: