Advanced Qualification Program Evaluator Manual TP 14672


The following terms are used throughout this document and are defined as follows:

ADVANCED QUALIFICATION PROGRAM (AQP):A voluntary program and alternative method of training, evaluating and qualifying flight crewmembers, instructors and evaluators, that uses a systematic methodology for developing proficiency-based training and evaluation programs in lieu of traditional training programs.

AIR OPERATOR:The holder of an Air Operator Certificate.

ANONYMOUS DATA:Data that cannot be identified with a named individual. Also referred to as DE-IDENTIFIED DATA.

APPLICANT:An air operator that applies to conduct training and evaluation under an AQP.

ATTITUDE:A persisting internal mental state that influences an individual's choice of personal action toward some object, person or event.

AUTHORIZED PERSON:A person who is delegated the authority to issue type ratings and/or instrument ratings by signing the additional privileges section on the back of the candidate's license or by completing the Certification of an Additional Privilege Card (26-0267).

COGNITIVE SKILLS (CS):Those intellectual skills that are prerequisite to the performance of a task, sub-task, element or sub-element. The three primary categories of cognitive skill are discrimination, concept learning and rule using.

CONDITION:One of the three primary components of a proficiency objective (performance, condition and standard). The conditions describe the range of circumstances under which student performance will be measured and evaluated. Conditions may include the natural environment (ceiling, visibility, wind, turbulence, etc.), the operational environment (navigational aid (NAVAID) unserviceabilities, birds, conflicting air traffic, gate change, passengers not seated, etc.) and operational contingencies (abnormal situations and emergencies).

COMPUTER BASED TRAINING:Classroom instruction that is performed individually by trainees at a computer station.

CONDUCT:To take an active role in all phases of a validation or evaluation, including pre flight preparation, briefing, control and pace of the various sequences, assessment of the candidate's performance, debriefing, collection of data and completion of required documents including certification of the candidate's licenses.

CONTINUING QUALIFICATION (TRAINING/PROGRAM):Training that follows initial qualification on a regular basis.

CONTINUING QUALIFICATION CYCLE:The time period during which training and evaluation on all proficiency objectives have been accomplished by all flight crewmembers, instructors or evaluators as applicable.

COURSEWARE:All instructional material that a candidate requires to complete a curriculum, in whatever media required, including manuals, visual aids, lesson plans, flight event descriptions, computer software programs, audio-visual programs, workbooks, handouts, etc.

CREW RESOURCE MANAGEMENT (CRM):The effective use of all available resources - human resources, hardware, and information - to achieve safe and efficient flight.

CRITICALITY:A characteristic of a terminal or supporting proficiency objective for which a substandard task performance would adversely affect safety. The relative need for awareness, care, exactness, accuracy or correctness during task performance. Critical tasks must be accomplished more frequently in training and evaluation than non-critical tasks. All critical tasks must be accomplished during each Evaluation Period.

CURRENCY:A characteristic of a terminal or supporting proficiency objective for which individuals and/or crews can maintain proficiency by repeated performance of the item in normal line operations. For pilots, most currency items may be validated during Online Evaluations (OE), while most non-currency items must be demonstrated during training, validation and evaluation events in a simulator or Flight Training Device (FTD).

CURRICULUM:A portion of an Advanced Qualification Program that covers one of two program areas: Qualification or Continuing Qualification. The Qualification and Continuing Qualification programs address the required training, evaluation and qualification activities for each aircraft (or variant) and for a specific duty position. Qualification and Continuing Qualification program areas may include but are not limited to upgrade, transition, differences and re-qualification curricula.

CURRICULUM DESIGN:The activities involved in organizing, clustering, sequencing and otherwise structuring the elements of instruction (objectives, lessons, evaluations, etc.) into an orderly flow of learning experiences to facilitate student performance.

CURRICULUM OUTLINE:The document that organizes training objectives into curricula, segments, modules, lessons, lesson elements, etc.

CURRICULUM SEGMENT:An integral part of a curriculum, which can be separately evaluated and individually approved, but by itself does not qualify a person for a duty position. (e.g., ground training segment, flight training segment, evaluation segment). The first level of curriculum detail (Segment, Module, Lesson, Lesson Element).

DE-IDENTIFIED DATA:Data that cannot be identified with a named individual.

DUTY:All the actions (tasks, sub-tasks, etc.) required by one's position or occupation.

DUTY POSITION:The operating position of a flight crewmember, or other person. Duty positions include Captain, First Officer, Cruise Relief Pilot (CRP), Second Officer, Flight Engineer, Instructor or Evaluator.

ELEMENT:A component of training analysis or design. In the case of task analysis, he element may be used as a level of analysis: phase of flight, task, sub-task, element, sub-element, etc. In the case of curriculum design, the element may be used as a level of curriculum organization: curriculum, segment, module, lesson, lesson element, etc.

ENABLING OBJECTIVE (EO):An instructional objective created at the level of an element, skill, knowledge, or attitude. Describing the functions of a hydraulic system would be an example. EOs are lower level learning objectives that help students master a higher level objective, such as a Terminal or Supporting Proficiency Objective. The knowledge and skill prerequisites of manoeuvres and procedures are usually trained as Enabling Objectives (EO).

EVALUATION:Careful appraisal of an individual's performance by an evaluator to ascertain whether the standards required for a specified level of proficiency have been demonstrated.

EVALUATION OF PROFICIENCY:Under AQP, either a Line Operational Evaluation (LOE) or Online Evaluation (OE).

EVALUATION PERIOD:A period within the Continuing Qualification Cycle in which each person must receive training and an evaluation on all critical proficiency objectives, during a minimum of one training session and a Manoeuvre Training and Validation/Line Operational Evaluation (MTV/LOE).

EVALUATOR:A person delegated by the Minister, who has satisfactorily met approved AQP evaluator training and evaluation requirements that qualify that person to evaluate the performance of flight crewmembers, instructors, or other evaluators.

EVENT:A training or evaluation situation comprised of a task or sub-task to be performed by the crew under a specified set of conditions.

EVENT SET:A relatively independent segment of a scenario made up of several events, including an event trigger, possible distracters, and supporting events.

FILL-IN or "SEAT FILLER":A qualified crew member who substitutes for a candidate who is unable to attend an evaluation session, thus allowing the rest of that candidate's crew to complete their evaluation with a full crew complement.

FIRST-LOOK MANOEUVRES:The performance and assessment of specific tasks, procedures or flight manoeuvres in accordance with approved program documentation, as a means of assessing performance and proficiency on designated tasks, procedures or flight manoeuvres before any briefing or training has taken place, in order to determine trends of degraded proficiency, if any, within the fleet's flight crewmember group as a whole.

FLIGHT OPERATIONS QUALITY ASSURANCE (FOQA):A program that receives and analyzes information from flight operations, aiming to identify and mitigate potential safety hazards.

FLIGHT TRAINING:Training given in an aircraft, flight simulator, FTD, or other cockpit environment. See ground training.

FLIGHT TRAINING DEVICE (FTD):A full-scale replica of an airplane cockpit that may not have the motion or visual systems associated with flight simulators.

FLIGHT TRAINING EQUIPMENT:Aircraft and those FTDs or flight simulators that are used for any of the following purposes: (1) Required evaluation of individual or crew proficiency; (2) training activities; (3) Activities used to meet recency of experience requirements; and (4) Line Operational Simulations (LOS).

FORMATIVE EVALUATION:Process of reviewing courseware for technical accuracy, instructional soundness, and suitability for use by instructors, evaluators and students. Dry run of the total curriculum with a small group of students to test the effectiveness and efficiency of the training (e.g., small group tryout).

FREQUENCY:Number of occurrences of a task/sub-task in a specific period of duty (1 flight, 1 trip, 1 month, 1 year, etc.) How often a task/sub-task is performed. Frequency may be used to determine currency (see Currency) by comparing the frequency with which activities occur on the line, to the frequency required to maintain proficiency without additional training.

FRONT END ANALYSIS:A generic term for any process used to identify the learning needs of a student population. May include needs analysis, job analysis, task analysis, student entry behavior analysis, performance analysis, competency analysis, etc.

GROUND TRAINING:Aviation/aircraft specific training provided in a classroom, learning centre, lecture hall or other traditional educational setting that occurs outside the cockpit environment.

INSTRUCTIONAL ANALYSIS:A process conducted during the design of instruction to identify the presentational components, or learning events, necessary for the student to master the complete range of skills, knowledge, attitudes, abilities, and CRM factors required for proficient performance.

INSTRUCTIONAL SYSTEMS DEVELOPMENT:A systematic methodology for deriving and maintaining qualification standards and associated curriculum content based on a documented analysis of the job tasks, skills, and knowledge required for job proficiency.

INSTRUCTOR:A person who has satisfactorily met approved AQP instructor training and evaluation requirements that qualify that person to conduct instruction to flight crewmembers, instructors or evaluators.

INTER-RATER RELIABILITY:A program that is conducted periodically to calibrate the assessment standards of instructors and evaluators, so that they will rate performance as closely to the same standard as possible.

ISSUING AUTHORITY:The Regional Manager, Commercial and Business Aviation, the Regional Superintendent for Aeroplanes or the Chief, Airline Inspection, as appropriate.

JOB:The summation of the functions, identified as tasks and sub-tasks, performed by an individual.

KNOWLEDGE:Specific information required to enable a student to develop the skills and attitudes to recall facts effectively, identify concepts, apply rules or principles, solve problems, and think creatively. Because knowledge is covert, students must be assigned overt activities to demonstrate their knowledge base.

LESSON:A meaningful division of learning consistent with the method of study, learning, or testing of performance (proficiency) objectives. The third level of curriculum definition (Segment, Module, Lesson, Lesson Element). Lessons usually contain objectives, training events, student materials, instructor materials, and an evaluation scheme or form.

LESSON ELEMENT OR TOPIC:A subgroup of activities within a lesson. It is the fourth level of curriculum detail (Segment, Module, Lesson and Lesson Element).

LICENSING EVENT:An event required for licensing action during a qualification course. The Qualification Standards for all pilot programs will designate those manoeuvres, procedures and events that must be trained and evaluated as a pre-condition for pilot licensing.

LINE OPERATIONAL EVALUATION (LOE):A proficiency evaluation conducted by a qualified evaluator in an approved simulation device that addresses an individual's ability to demonstrate technical and Crew Resource Management (CRM) skills appropriate to job requirements in a full mission scenario environment.

LINE ORIENTED FLIGHT TRAINING (LOFT):A Line Operational Simulation (LOS) flight scenario designed for training purposes to provide practice in the integration of technical and CRM skills. LOFT is conducted using a complete cockpit flight crew to the maximum extent feasible and is accomplished in a Transport Canada approved simulation device. A LOFT training session is not interrupted by the instructor, unless negative learning begins to occur.

MANOEUVRES VALIDATION (MV):A simulator session in which specific manoeuvres are performed and evaluated to proficiency. See Chapter8 – Validations and Evaluations.

MANOEUVRES PROFICIENCY VALIDATION (MPV):See Chapter8 – Validations and Evaluations.

MANOEUVRES TRAINING AND VALIDATION (MTV):See Chapter8 – Validations and Evaluations.

MEDIA:Physical means for providing the instructional content and experience to the student. Includes the entire set of instructional presentation materials; e.g., workbook, videotape, overheads, Computer Based Training device (CBT), mock-ups, FTDs, simulators, etc.

MODULE:A group of subject matter under a specific curriculum segment. Second of four curriculum levels of detail (Segment, Module, Lesson, Element). Often corresponds to a day of training or a device event, such as FTD #3 or simulator #6.

MOTOR SKILL:Physical actions required to perform a specific task (sub-task or element). Students have acquired a motor skill not when they can simply perform a prescribed procedure, but when their movements are smooth, regular and precisely timed. Those hands-on skills that are prerequisite to the performance of a task, sub-task, element or sub-element.

NOMINEE:A person nominated by an air operator as a candidate for AQP evaluator approval by TC.

OBSERVABLE BEHAVIOR:A behavior whose occurrence during the performance of an event is an indicator that the crew is handling the event properly. Observable behaviors form one part of the performance standards identified for each event. See Performance Standard.

ONLINE EVALUATION (OE):An evaluation conducted by a qualified evaluator during normal flight operations that assesses the candidate's proficiency with respect to the particular aircraft, crew position and type of operations, and his or her skill and ability to operate effectively as part of a crew.

PERFORMANCE/PROFICIENCY DATABASE (PPDB):A database that collects results of performance and proficiency evaluations, and is used to assess the effectiveness of training programs.

PERFORMANCE STATEMENT:One of the three components of an objective. A statement of physical and/or cognitive activities which, when executed or carried out, will complete the work required for a specific portion of a job (in the case of a proficiency objective), or the activities required of a learning goal (in the case of a learning objective). See Proficiency Objective.

PHASE OF FLIGHT:The standard high-level set of activities performed by pilots on all operational flights. For example: Pre-flight, Engine Start, Pushback, Taxi, Take-off, Climb, Cruise, Descent, Holding, Approach, Landing, Taxi and Post Flight Operations.

PROCEDURES VALIDATION (PV):See Chapter8 – Validations and Evaluations.

PROFESSIONAL SUITABLITY:A demonstrated willingness to work cooperatively with Transport Canada to uphold the principles of aviation safety.

PROFICIENCY OBJECTIVE:A statement describing precisely what behavior must be exhibited by the candidate, the conditions under which the behavior will be demonstrated, and the minimum standard of acceptable behavior. A learning objective (usually an enabling objective) can be demonstrated in a classroom or academic type setting, while a performance objective (usually a terminal or supporting proficiency objective), must be demonstrated in an environment equivalent to the operational environment.

PROGRAM AUDIT DATABASE (PADB):A database that is used to analyze the elements of a training program and the supporting task analysis that must be accomplished during any training cycle. It may be used to develop lesson plans and to address deficiencies found in performance and proficiency by the PPDB (performance/proficiency database).

QUALIFIED PERSON:In the case of a simulator, a pilot who holds a valid PPC/LOE (or foreign equivalent) on the same type of aircraft on which the other candidate is being evaluated; a person who has been recommended for a validation or evaluation on that aircraft type; or a qualified training pilot on the same type of aircraft for which the candidate is being evaluated on, where that person is acceptable to both the operator and the valilidation/evaluation candidate.

QUALIFICATION STANDARDS:The terminal and supporting proficiency objectives coupled with test and evaluation strategies (where, how and by whom qualification is measured). Qualification Standards and previous experience provide the baseline for mastery of the duty position. Demonstration that an individual has met certain or all of these standards may lead to certification.

QUALITY ASSURANCE EVALUATOR (QAE):A Quality Assurance Evaluator (QAE) is both a company flight crewmember and an experienced AQP Evaluator, in each case qualified on type, who has been designated by the air operator to perform quality assurance functions for the air operator's AQP evaluation programs. The QAE's duties include monitoring (evaluating) AQP evaluator nominees and AQP evaluators. The air operator may utilize other terms such as "Evaluator Mentor" for individuals acting in this role.

RATER-REFERENT RELIABILITY (RRR):RRR is a correlation reflecting how closely an evaluator's ratings agree with some standard or referent. This method of assessing sensitivity can be used when there is an external, objective basis for defining a referent score. A simple illustration is a situation where we correlate an individual's subjective estimates of the weights of different objects with their actual weights. To the extent that the subjective estimates track or co-vary with the actual weights, the estimates are sensitive and the individual's RRR will be high. RRR can be used to assess evaluators' sensitivity in assessing aircrew performance as long as there is an objective basis for grading performance.

SAFETY PILOT:In the case of a two crew aircraft, a training pilot on the same type of aircraft on which the candidate is being evaluated; or a pilot who holds a valid PPC/LOE on the same type of aircraft on which the candidate is being evaluated.


SIMULATOR:A full sized replica of a specific type of airplane cockpit, including both visual and motion systems.

SKILL:An ability to perform an activity or action. Often divided into motor/hands-on and cognitive categories.

SOPs:Approved Standard Operating Procedures established by an air operator, which enable the crewmembers to operate the aircraft within the limitations specified in the Aircraft Flight Manual.

SPECIAL PURPOSE OPERATIONAL TRAINING (SPOT):A portion of a Line Operational Simulation (LOS) training scenario consisting of flight tasks selected from any phase or phases of flight to provide practice in the integration of technical and CRM skills appropriate to the selected flight tasks. SPOT is conducted using a complete cockpit flight crew to the maximum extent feasible and is accomplished in a simulation device.

SPECIAL TRACKING:A system of monitoring the proficiency of an individual at scheduled intervals. It may be applied to individuals that have failed to demonstrate proficiency during an evaluation (LOE) or as required.

STANDARD OF PERFORMANCE:Observable, measurable parameters of performance with tolerances; e.g., course deviation degrees, + or -. Applies to procedures, manoeuvres, and observable behaviors.

SUB-ELEMENT:A subcomponent of an element. See element.

SUB-TASK:Specific separate step or activity required in the accomplishment of a task. May also refer to categories of a task (e.g., Non-precision approach – VOR, NDB, LOC etc.).

SUMMATIVE EVALUATION:Training program evaluation accomplished in a full operational setting. Usually accomplished during the first full increment of classes with a full student complement.

SUPPORTING PROFICIENCY OBJECTIVE (SPO):A proficiency objective created at the sub-task level. For example: Perform Engine-Out Precision Approach Preparation Procedures.

SYSTEMS KNOWLEDGE VALIDATION (SKV):See Chapter8 – Validations and Evaluations

TASK:A task is a unit of work within a function having an identifiable beginning and ending point, which results in a measurable product, output or behavior. An example of a task applicable to AQP: Perform a normal take-off.

TC INSPECTOR:A Transport Canada Inspector who works in the Commercial and Business Aviation (CBA) Branch and is authorized to conduct validations, evaluations and monitors.

TECHNICAL SKILLS:Within an AQP, technical skills refer to those manoeuvres, procedures and other behaviors that have a high psychomotor component, while CRM skills refer to those communication, decision-making and workload management behaviors that have a high cognitive component.

TERMINAL PROFICIENCY OBJECTIVE (TPO):A proficiency objective created at the task level. For example: Perform Engine-Out Precision Approach.

TPO/SPO HIERARCHY:The hierarchy of all TPOs and SPOs organized by phase of flight.

TRAINING PERIOD:At least one period scheduled at the mid-point of each Evaluation Period where training activities are provided for each person under AQP.

TRAINING PILOT:An instructor pilot who meets the requirements of the applicable CAR Standard or Qualification Standards and for the purpose of Initial Operating Experience (IOE), means a Training Captain.

TRAINING SESSION:A contiguously scheduled period of time devoted to training activities at a facility acceptable to Transport Canada for that purpose.

TRAINING TO PROFICIENCY:Training to a performance level that meets or exceeds a qualification standard. This concept must include enough repetition and practice to ensure that each individual can perform at the qualification standard level over the entire evaluation period or Continuing Qualification cycle.

TRIGGERING CONDITIONS:The conditions whose occurrence defines the beginning of an event.

UPGRADE TRAINING:The training undertaken by a second officer, cruise relief pilot or first officer to become qualified as first officer or aircraft captain, as applicable.

VALIDATION:A determination that required results with regards to performance objectives were produced.

VARIANT:An aeroplane or a group of aeroplanes sharing similar characteristics but having pertinent differences from a base aeroplane. Pertinent differences are those that require different or additional flight crewmember knowledge, skills and/or abilities that affect flight safety.

VITAL ACTION:An action that must be taken by flight crew to alleviate a situation that could jeopardize safety of flight. The action shall be taken in a timely manner consistent with the AOM, FCOM or SOPs as appropriate.

Date modified: