Advanced Qualification Program Evaluator Manual TP 14672

CHAPTER 8 – VALIDATIONS AND EVALUATIONS

8.1 General

8.1.1 AQP validation/evaluation methodologies must meet or exceed the practical test requirements of Part IV and Part VII of the CARs. Air operators must provide a regulatory comparison to demonstrate that their AQP validations and evaluations meet or exceed the regulatory requirements of Pilot Proficiency Checks as well as Instrument and Type Rating Test Standards.

8.1.2 In traditional training programs, the candidate’s performance is not measured until the final check ride. In contrast, AQP features validation points in each phase of training. Assessments are made continuously, from ground school through Flight Training Device (FTD), Fixed Base Simulator (FBS), Full Flight Simulator (FFS) and Initial Operating Experience (IOE). These assessments are used to ensure the candidate’s satisfactory progress in each phase of training.

8.1.3 AQP uses both ‘validations’ and ‘evaluations’ to assess that the Proficiency Objectives of the training module have been met and the candidate is ready to proceed to the next level of training or line operations.

8.1.4Validations and evaluations also serve to validate the effectiveness of the air operator’s training program, policies and established procedures. They provide air operators with valuable data that is used for the continual improvement of the training program and to improve the safety of ground and flight operations.

8.2 Validations

8.2.1 A ‘validation’ is a determination that training has produced the required results as identified in the Qualification Standards. A validation is a confirmation that the individual has met the Performance Objectives of the training segment.

8.2.2 In AQP there are 4 types of validations:

  • System Knowledge Validation (SKV)
  • Procedures Validation (PV)
  • Manoeuvres Validations (MV)
  • First-Look Manœuvres (FLM)

8.3 Systems Knowledge Validation (SKV)

8.3.1 The System Knowledge Validation (SKV) is an assessment of an individual’s technical knowledge with respect to aircraft systems. The intent of the SKV is to ensure an individual’s systems knowledge is at an appropriate level before progressing into the next training phase. SKV may be accomplished via a written, electronic or oral exam and can be conducted in an open- or closed-book fashion, based on the operator’s validation methodology.

8.3.2 The SKV is a determination of the individual candidate's systems knowledge. Each candidate must successfully pass the SKV on his/her own merit - without the assistance of any other individual. During the SKV, candidates are not permitted to work together or assist each other in any manner.

8.3.3 In cases where the SKV is conducted open-book style, candidates may be provided with appropriate reference materials. Suitable references include the appropriate Aircraft Flight Manual (AFM), Aircraft Operating Manual (AOM), Flight Crew Operating Manual (FCOM) and Quick Reference Handbook (QRH). Access to diagram or mock-up of the flight deck is also permissible. As a general guide as to what is appropriate, the candidates should have access to whatever references or materials to which they would normally have access during the performance of their flight deck duties.

8.3.4The air operator will establish a minimum pass mark for the SKV. All incorrect answers must be corrected to 100 percent. An overall score that is less than the minimum pass mark will require retraining and another complete test. Providing the minimum pass mark was attained, a failure of an individual test module or sub-section requires retraining and retesting of that specific module or sub-section only. Consideration should be given to establishing a maximum number of modules or sub-sections that if failed constitute an overall failure of the validation.

8.3.5Any AQP evaluator or instructor may conduct a SKV. TC delegation of authority is not required to conduct this type of validation.

8.4Procedures Validation (PV)

8.4.1 A Procedures Validation (PV) is an assessment of a candidate’s ability to integrate system knowledge and procedural knowledge. This validation addresses the candidate’s ability to assimilate systems and procedural knowledge into the appropriate execution of procedures. A PV can take place in a System Trainer, Flight Training Device (FTD) or a Full Flight Simulator (FFS). The purpose of the PV is to confirm that a candidate’s systems knowledge as well as procedural knowledge and skills are at an appropriate level. This must be ascertained before the candidate progresses into the Full Flight Simulator training phase. Additional training can occur during a PV. Success is accomplished when the individual is trained to proficiency.

8.4.2 Any AQP instructor or evaluator may conduct a PV. TC delegation of authority is not required to conduct this type of validation.

8.5Manoeuvres Validation (MV)

8.5.1 A Manoeuvres Validation (MV) addresses the candidate’s proficiency in the execution of manoeuvres. It must take place in a Level C or higher Full Flight Simulator (FFS).

8.5.2 In order to differentiate between the MV conducted in a Qualification Curriculum (QC) and a Continuing Qualification Curriculum (CQC), the following terms have been established:

  • Manoeuvres Proficiency Validation (MPV) for the Qualification Curriculum
  • Manoeuvres Training and Validation (MTV) for the Continuing Qualification Curriculum.

The essential difference between the MPV for Qualification Curriculum and the MTV for Continuing Qualification Curriculum is the manner in which repeats of unsuccessful exercises are addressed.

8.5.3 The MV must be conducted by a Type "V" or Type "E" Evaluator.

8.5.4The MV forms part of the licensing requirements to renew an Instrument Rating. However, a candidate’s Instrument Rating cannot be suspended as a result of an unsuccessful MV. No licensing action will result from an unsuccessful MV.

8.6Manoeuvres Proficiency Validation (MPV)

8.6.1 A Manoeuvres Proficiency Validation (MPV) is only applicable to the Qualification Curriculum (QC). The MPV in a QC addresses the candidate’s proficiency as Pilot Flying (PF) in the execution of manoeuvres. Candidates must also be assessed while performing Pilot Not Flying (PNF) duties.

8.6.2 A written recommendation from the last Manoeuvres Training FFS instructor is required for admission to the MPV. The instructor who recommends the candidate cannot conduct his/her MPV.

8.6.3 An air operator may elect to have a brief warm-up period prior to the commencement of the MPV. Once this warm-up period is complete, the evaluator will advise the candidates and the MPV will begin. The time used for warm-up will be included in determining the total duration of the session. The significance of this time constraint is discussed in 8.6.4.

8.6.4During a MPV candidates are allowed two (2) repeats of any one manoeuvre or one repeat of any two (2) manoeuvres. A debriefing of why the manoeuvre(s) was unsatisfactory is permitted. However, the repeats must occur with no training, practice, or coaching. If the candidate fails to demonstrate proficiency within the allowed repeat criteria and/or within the time constraints of the simulator session, an Additional Training Opportunity (ATO) is required. After the additional training, the candidate will be re-scheduled for a remedial MPV. During the remedial MPV the candidate will need to repeat only the manoeuvres that were unsatisfactory during the initial MPV.

8.6.5A MPV shall be conducted according to the following protocol:

  1. Prior to the conduct of a MPV, the evaluator shall verify the validity of the candidates’ training records (including a recommendation from the last Manoeuvres Training FFS instructor), pilot license and medical;
  2. The evaluator shall conduct a briefing in accordance with the procedures and protocols established in the air operator’s AQP and PADB documentation, as applicable;
  3. After the pre-MPV warm-up session (if applicable) is completed, the evaluator will announce to the crew that the MPV is now underway;
  4. Following the announcement that the MPV has commenced, the first attempt on any manoeuvre will be considered a validation;
  5. PNF duties will be validated in conjunction with PF duties;
  6. Once a manoeuvre has been successfully completed it will count as a successful validation;
  7. An individual manoeuvre graded as unsatisfactory may be repeated during the MPV. Two(2) repeats of any one manoeuvre or one repeat of any two (2) manoeuvres are allowed for each candidate;

    Note:Crew-initiated repeats of manoeuvres deemed unsatisfactory by the evaluator will also be counted towards the maximum number of allowable repeats.
  8. Each repeat shall be conducted immediately or at the earliest practical opportunity after the unsatisfactory manoeuvre was demonstrated;
  9. For an unsatisfactory manoeuvre, the evaluator may debrief the candidate(s) as to why the manoeuvre was unsatisfactory. However, any subsequent repeats must occur without training, practice, or coaching;
  10. Following the MPV, the evaluator shall conduct a de-briefing in accordance with the procedures and protocols established in the air operator’s AQP and PADB documentation, as applicable;
  11. If a candidate’s MPV is unsuccessful, an Additional Training Opportunity (ATO) will be provided;
  12. Upon completion of the ATO, with the recommendation of a Manoeuvres Training FFS Instructor, the candidate can be re-scheduled for a remedial MPV. During the remedial MPV, the candidate is required to demonstrate only the manoeuvres which were unsatisfactory during the initial MPV;
  13. A remedial MPV is conducted in the same manner as a normal MPV;
  14. If there is simulator time remaining after the completion of a MPV, this time may be used to provide additional training, if required.

8.7Manoeuvres Training and Validation (MTV)

8.7.1 A Manoeuvres Training and Validation (MTV) is only applicable to the Continuing Qualification Curriculum (CQC). A MTV allows the assessment and ensures attainment of technical proficiency prior to the candidate’s evaluation in a Line Operational Evaluation(LOE).

8.7.2 A MTV in a CQC addresses the candidate’s proficiency as Pilot Flying (PF) in the execution of manoeuvres. Candidates must also be assessed while performing Pilot Not Flying (PNF) duties.

8.7.3 An air operator may elect to include a brief warm-up period prior to the commencement of the MTV. Once this warm-up period is complete, the evaluator will advise the candidates and the MTV assessment will begin. The time used for warm-up will be included in determining the total duration of the session.

8.7.4During a MTV candidates are allowed to repeat any manoeuvres. The only limit to the number of repeats is the time available in the simulator. If the candidate fails to demonstrate proficiency within the time constraints of the simulator session, an ATO is required. After the additional training, the candidate will be re-scheduled for a remedial MTV. During the remedial MTV the candidate will need to repeat only the manoeuvres that were previously unsatisfactory during the initial MTV.

8.7.5A MTV shall be conducted according to the following protocol:

  1. Prior to the conduct of a MTV, the evaluator shall verify the validity of the candidate’s pilot license and medical certificate;
  2. The evaluator shall conduct a briefing in accordance with the procedures and protocols established in the air operator’s AQP and PADB documentation, as applicable;
  3. After the pre-MTV warm-up session (if applicable) is completed, the evaluator will announce to the crew that the MTV is now underway;
  4. Following the announcement that the MTV has commenced, the first attempt on any manoeuvre will be considered a validation;
  5. Following the initial validation attempt, training can occur on any manoeuvre. However, when the manoeuvre is re-assessed, for the purpose of the validation, the candidate must perform the manoeuvre without any coaching or prompting;
  6. PNF duties will be validated in conjunction with PF duties;
  7. Once a manoeuvre has been completed successfully it will count as a successful validation;
  8. Individual manoeuvres considered unsatisfactory must be trained and validated to proficiency. The only constraint on the number of repeats is the availability of simulator time;
  9. Training and re-validation of a manoeuvre shall be conducted immediately or at the earliest practical opportunity after the manoeuvre was demonstrated as being unsatisfactory;
  10. Once the training has been completed, the previously unsatisfactory task must be re-validated. For a satisfactory assessment to be made during the repeat, the candidate must perform the task without prompting or coaching. Once proficiency is established on the unsatisfactory manoeuvre, it is considered to be successfully validated;
  11. Following the MTV, the evaluator shall conduct a de-briefing in accordance with the procedures and protocols established in the air operator’s AQP and PADB documentation, as applicable;
  12. If training to proficiency cannot be established on any required manoeuvres within the time constraints of the simulator session, an ATO shall be provided;
  13. After the ATO, the candidate can be re-scheduled for a remedial MTV. During the remedial MTV the candidate need repeat only the manoeuvres that were unsatisfactory during the initial MTV;
  14. A remedial MTV is conducted in the same manner as a regular MTV;
  15. Depending on the number of manoeuvres to be repeated, an ATO and a remedial MTV may be combined in a single simulator session. This is permitted because training is allowed in a MTV;
  16. If there is simulator time remaining after the completion of a MTV, this time may be used to provide additional training, if required.

8.8First-Look Manoeuvres (FLM)

8.8.1 Given adequate data analysis and justification, AQP may allow for extended training and evaluation cycles. In order to modify training and evaluation intervals, the air operator must have previously implemented First-Look Manoeuvres and collected sufficient data through one full Continuing Qualification Cycle in order to establish a base line by which to measure the effect of increased intervals. First-Look Manoeuvres (FLM) are comprised of tasks, procedures or manoeuvres that are identified as likely to be sensitive to loss of proficiency due to infrequent practice.

8.8.2 FLM is conducted by an AQP qualified instructor or a Type E or Type V Evaluator in a Level C or higher FFS. During FLM, instructors and evaluators must employ the same measurement methodology and rating criteria as used in Manoeuvres Validations (MV). FLM grades are analyzed by the air operator to detect trends of degraded proficiency.

Note:Air operators must have a system to ensure that instructors conducting FLM are qualified to perform this function in accordance with their approved AQP.

8.8.3 If a candidate successfully performs specific manoeuvres during FLM, those manoeuvres, if included in the associated MV, do not need to be assessed a second time during the MV.

8.9Evaluations

8.9.1 An Evaluation is an appraisal of an individual to ascertain whether the standards required for a specified level of proficiency have been successfully demonstrated. Interrupting the evaluation session for training is not permitted.

8.9.2 In AQP there are 2 types of evaluations:

  • Line Operational Evaluation (LOE)
  • Online Evaluation (OE)

8.10Line Operational Evaluation (LOE)

8.10.1 The LOE is the primary mode of proficiency evaluation. The LOE is conducted in a level C or higher FFS as approved by Transport Canada. The purpose, administration, and remediation strategy for the Qualification Curriculum (QC) LOE is the same as for a Continuing Qualification Curriculum (CQC) LOE. A LOE cannot be equated to a Pilot Proficiency Check (PPC), but completion of an AQP training syllabus and all validations and evaluations is confirmation that a candidate has met or exceeded the regulatory requirements of a traditional PPC. Successful completion of a MV and LOE is confirmation that the candidate has met all of the requirements for the issuance or renewal of an Instrument Rating and issuance of a Type Rating (if applicable).

8.10.2 A LOE can be conducted only by a Type "E" Evaluator.

8.10.3 A LOE must be completed within 30 days of the MV (MPV or MTV).

8.10.4The LOE addresses the individual’s ability to demonstrate technical and CRM skills appropriate to fulfilling job requirements in a full mission scenario environment. The intent of a LOE is to evaluate and verify that an individual’s job knowledge, technical skills, and CRM skills are commensurate with AQP qualification standards. For the Qualification Curriculum (QC), the LOE is also used to verify that the individual is qualified to begin the Initial Operating Experience (IOE) portion of the Qualification Course.

8.10.5LOEs are graded at the event set level. A LOE consists of a minimum of 8 events sets. During the LOE, an individual event set graded as unsatisfactory may be repeated. Two repeats are allowed for each candidate. No single event set can be repeated more than once. A debriefing of why the event set was unsatisfactory is allowed, but the repeat must occur with no training, practice, or coaching.

Note:Crew-initiated repeats of manoeuvres or procedures which resulted in an unsatisfactory event set assessment by the evaluator will also be counted towards the maximum number of allowable repeats.

8.10.6If any repeated event set is still unsatisfactory, remedial training and another complete LOE is required. Regardless of the number of unsatisfactory event sets, unsafe individual or crew performance that would result in significant damage, hull loss or loss of life (e.g., crash) during a LOE constitutes a failure of the LOE. An unsatisfactory LOE will require an ATO and a remedial LOE.

8.10.7The LOE is considered a jeopardy event and a failure is reported to TC. In the event of a failure, the entire copy of the candidate’s LOE report is faxed to Transport Canada for licensing action (i.e. suspension).

8.10.8A LOE failure will also result in the individual candidates being placed into Special Tracking for at least one training period. While in Special Tracking, candidates are required to undergo another MV/LOE – instead of an MT/LOFT - during their next assessment.

8.10.9A LOE shall be conducted according to the following protocol:

  1. Prior to the conduct of a LOE, the evaluator shall verify the validity of the candidate’s pilot license and medical certificate;
  2. The evaluator shall conduct a briefing in accordance with the procedures and protocols established in the air operator’s AQP and PADB documentation, as applicable;
  3. A LOE is normally comprised of 8 to 11 (with a minimum of 8) event sets and usually starts at the flight planning/dispatch stage and ends at the gate after the parking checklist is completed;
  4. Both technical and CRM topics are evaluated during each event set;
  5. PNF duties will be validated in conjunction with PF duties;
  6. An event set is considered satisfactory when the appropriate minimum standard has been demonstrated;
  7. Individual event sets graded as unsatisfactory may be repeated during the LOE. Two repeats are allowed for each candidate. No single event set can be repeated more than once;
  8. For an unsatisfactory event set, the evaluator can inform the candidate(s) as to which event set was unsatisfactory. However, any subsequent repeats must occur without training, practice, or coaching;
  9. Event Set repeats must be conducted using the same evaluation media during which the initial event set was performed unsatisfactorily in one of two different ways (as appropriate to the situation):
    1. it is repeated naturally within the flow of the script; or
    2. it is repeated at the conclusion of the LOE;

      Note:Repeats, whether initiated by the crew or the evaluator, should not be conducted in a manner that will disrupt the normal flow of the LOE script.
  10. Upon completion of the LOE, including repeats, the evaluator shall conduct a de-briefing in accordance with the procedures and protocols established in the air operator’s AQP and PADB documentation, as applicable;
  11. If a candidate’s LOE is unsuccessful, licensing action as appropriate is taken and Transport Canada is notified within two business days. An Additional Training Opportunity (ATO) will be provided;
  12. After the ATO, the individual is re-scheduled for a complete remedial LOE;
  13. A remedial LOE is conducted in the same manner as a regular LOE.

8.11 Online Evaluation (OE)

8.11.1 An Online Evaluation (OE) replaces and is conducted in the same manner as a traditional Line Check. The primary difference is the additional requirement to collect data and complete grade sheets as required under AQP. Flight crewmembers receiving this evaluation are assessed for their proficiency in their respective duty position. Successful completion of the OE verifies that the individual is adequately trained and is capable of performing his/her duties and responsibilities.

8.11.2 An OE is an Evaluation conducted during normal flight operations (i.e., during a revenue flight).

8.11.3 An OE must be conducted by a Type ‘O’ or Type ‘E’ Evaluator.

8.11.4While OEs provide an opportunity to evaluate flight crew under normal line operations, they also provide an opportunity to evaluate the effectiveness of company policies and procedures that impact line operations (ex. operational control, refueling and de-icing, air traffic control, etc.).

8.11.5OE data is a valuable tool for determining weaknesses or deficiencies in company policies and procedures and can provide a valuable feedback mechanism for evaluating the efficiency and effectiveness of adjustments to company operations.

8.11.6During an OE the candidate must be individually evaluated as to:

  • Proficiency in the particular aircraft, crew position, and type of operation (technical); and
  • Skill and ability to operate effectively as part of a crew (CRM).

8.11.7During an OE, when an evaluator decides that a particular sequence or event was unacceptable (i.e., "unsatisfactory" rating), the OE may be continued at the evaluator’s discretion until all planned legs have been completed. If, in the evaluator’s opinion flight safety could be jeopardized by allowing the OE to continue, or the pilot(s) will definitely require further training to meet the standard, then it shall be terminated as soon as practicable. If the Type O Evaluator is a company qualified IOETC and occupies a flight crewmember seat, the remaining portion of the scheduled flight(s) may be conducted as IOE or an ATO at the discretion of the evaluator.

8.11.8If any task is unsatisfactory, the task must be assessed again. The unsatisfactory item can either be re-assessed by repeating the task during the original OE or during a subsequent OE, if required.

8.11.9Repeats during an OE must occur with no training, practice, or coaching. However, a debriefing as to why a task was unsatisfactory is permitted.

8.11.10The decision to repeat an item during an OE will be at the discretion of the evaluator. Consideration will be given to the nature of the deficiency, the cause of the unsatisfactory performance, the individual’s overall performance and abilities to continue, whether additional opportunities will be presented during the course of the OE and any possible jeopardy to the safety of flight. A repeat is not permitted if it is the evaluator’s opinion that flight safety could be jeopardized by allowing the candidate to repeat the task.

8.11.11 For a satisfactory assessment to be made during the repeat, the candidate must be able to perform the task successfully without prompting or coaching. However, normal crew CRM regarding a sequence is allowed.

8.11.12 If a pilot receives an unsatisfactory overall performance rating on an OE, the pilot must be removed from continued line operations until training or appropriate remediation has been accomplished successfully, and a subsequent OE has been completed satisfactorily. The evaluator will recommend the type of training or remediation to be administered.

8.11.13 If a pilot receives an unsatisfactory overall performance rating on an OE, the pilot cannot progress to line operations until the approved remediation (additional training as required) and a successful OE have been completed successfully.

Date modified: