Advanced Qualification Program Evaluator Manual TP 14672

CHAPTER 14 – AQP DATA MANAGEMENT

14.1 Background

14.1.1 This chapter provides general guidance for the management of performance/proficiency data within an AQP. Specific recommendations for collecting, entering, reporting, and analyzing performance data are addressed in the "Data Management Guide". This document was developed by the Data Management Focus Group AQP Subcommittee, which is sponsored by the Air Transport Association (ATA). The "Data Management Guide" is available from Transport Canada.

14.2 Regulatory Requirement

14.2.1 Under an AQP, the air operator is required to collect and analyze performance information on its flight crewmembers, instructors and evaluators. The data collection, analysis and reporting processes employed by the air operator must be acceptable to Transport Canada. This data will enable the air operator and Transport Canada to determine whether the form and content of training and evaluation activities satisfactorily accomplish the overall objectives of the curriculum. Good data management practices are necessary to determine whether an AQP is meeting its objectives.

14.3 Validation

14.3.1 The principal goal of the AQP is true proficiency-based training and qualification. Performance objectives are systematically developed and maintained, then continuously validated through the collection and evaluation of empirical performance data. Data collection and analysis, or data management in short, is therefore an integral part of AQP.

14.4Definition

14.4.1 Data management can be classified into the two broad categories:

  • Individual Qualification Records; and
  • Performance/Proficiency Data.

14.5Individual Qualification Records

14.5.1 These are identifiable records maintained in sufficient detail on each individual flight crewmember, instructor and evaluator who is qualifying or has qualified under an AQP. These records show how and when the individual satisfied the requirements of the curriculum required for their assigned duty position. They may also include demographic and work history information, as well as completion information on the modules and lessons. Air operators may maintain a manual or a computerized record keeping system. The record keeping process in AQP does not differ from traditional record keeping requirements.

14.6Performance/Proficiency Data

14.6.1 In addition to the traditional record keeping requirements described above, AQP also requires the establishment of a separate Performance/Proficiency Database (PPDB). PPDB records are de-identified and maintained separately from the normal qualification records.

14.6.2 This de-identified information represents the results of an individual’s ability to demonstrate the performance objectives of each curriculum successfully. This information is captured during validation and evaluation gates as a crewmember progresses through an AQP curriculum. This data is obtained from each crewmember’s performance and is stored in a collective form in the PPDB.

14.6.3 This data is used to analyze training programs and/or groups of participants, not for tracking individual accomplishment. Successful collection and analysis of this data will allow the air operator to identify and correct problems, validate AQP curriculums, and identify developing trends.

14.7Overview

14.7.1 In AQP, data management is a continual process of data collection, entry, submission and analysis.

14.8Data Collection

14.8.1 AQP data collection is required in all curricula. The specifics are detailed in the air operator’s approved AQP Data Management Plan, which is contained in the Implementation and Operations Plan (I & O Plan).

14.8.2 Data is collected at each validation or evaluation gate. This data consists of graded proficiency objectives using a rating scale with associated reason codes (if applicable). Data collection requirements for the AQP will vary with the curriculum, the type of curriculum activity (training, validation, or evaluation), the type of participant (crewmember, instructor, or evaluator), and the overall management objectives for use of the data. All performance data collected on each proficiency objective must be relative to the applicable AQP Qualification Standards defined for the training and evaluation activities.

14.8.3 For each flight crewmember, instructor or evaluator in a Qualification or Continuing Qualification curriculum, Transport Canada must be able to associate the data records applicable to that person in that curriculum through logical grouping of the records, or linkage by a common de-identified index number, but not by name.

14.9Data Entry

14.9.1 All performance/proficiency data collected throughout the AQP is entered into the air operator’s PPDB. Typically, this is an electronic database for ease in analysis, comparison and reporting purposes.

14.9.2 Considerations for data entry include the method, the hardware/software required for data input, and the hardware/software required for data storage and utilization. Distinct advantages, disadvantages and costs are associated with any method of data entry.

14.9.3 Database design is at the discretion of the air operator, providing that the design can generate the required report table specified, in a manner acceptable to Transport Canada.

14.10Data Submission

14.10.1 Transport Canada has established the minimal requirements for the submission of de-identified data by curriculum. Figure 14-1 summarizes the minimum submission requirements for the Canadian Data Report Table (CDRT). The information in this table is downloaded from the carrier’s PPDB.

14.10.2 The submissions are forwarded electronically, or made available by direct web access to the TC unit identified to receive AQP data files. Data should be compiled in 1 calendar-month blocks and made available within 2 months of collection. Transport Canada will analyze the de-identified data using standard automated queries and reports to identify AQP performance trends.

14.10.3 The CDRT contains a listing of 20 fields that are reported for every measured item, providing a separate record for each. A measured item is a manoeuvre, task, procedure, or event set, and is the main component for data analysis. These fields provide a record of the results of the performance of each measured item along with supporting data for reporting and analysis. Certain supporting data fields (airline designator, curriculum, etc.) repeat for each record and can be automatically generated from a query/software routine. Each field in the CDRT must contain an alpha/numeric, numeric, or text entry.

14.10.4Due to the unique features of each operator’s AQP, TC in consultation with the air operator may require the collection of additional data as deemed appropriate.

14.11 Qualification Curriculum

14.11.1 For Qualification Curricula (including Secondary Curricula as appropriate) the air operator must make available to Transport Canada the following minimum AQP data:

  1. Data for each flight crewmember in training, for each progressive validation module other than LOE, by respective validation module identifier;
  2. Data for each flight crewmember evaluated in a LOE, a single electronic record for the Captain/First Officer/Cruise Relief Pilot/Second Officer/Flight Engineer (or seat filler) identifier codes and for each pilot evaluated;
  3. Data for each flight crewmember receiving an Online Evaluation (OE)

14.12 Continuing Qualification Curriculum

14.12.1 For Continuing Qualification Curricula (including Secondary Curricula as appropriate) the air operator must make available to Transport Canada the following minimum AQP data:

  1. Data for each flight crewmember in training, for First-Look Manoeuvres (FLM) (when applicable);
  2. Data for each flight crewmember in training, for Manoeuvre Validation (MV);
  3. Data for each flight crewmember evaluated in a Line Operational Evaluation (LOE), a single electronic record for the Captain/First Officer/Cruise Relief Pilot/Second Officer/Flight Engineer (or seat substitute) identifier codes and for each pilot evaluated;
  4. Data for each flight crewmember present during an Online Evaluation (OE), the performance data that directly mirrors the content of the Transport Canada-approved OE form.

14.12.2 Table 14-1 lists the minimum AQP Data submission requirements.

Table 14-1: Summary of AQP Data Submission Requirements for Canadian Data Reporting Tool (CDRT)
Data Qualification Curricula Continuing Qualification Curricula
Data Type PV MPV LOE OE FLM MTV LOE OE
File Name/Record Identifier X X X X X X X X
DD/MM/YY X X X X X X X X
Aircraft Fleet ID X X X X X X X X
Curriculum ID X X X X X X X X
Crew/Duty Position   X X X   X X X
PF/PNF   X X X X X X X
PIC/SIC   X X X X X X X
Seat Substitute   X X     X X  
Satisfactorily Completed (Y/N) X X X X   X X X
Item Or Event Set Identifier X X X X X X X X
Event Rating/Grade   X X X X X X X
Reason Code/Skill Category   X X X   X X X
Repeats Required   X X X X X X X
Additional OE Required (Y/N)       X       X
Additional Trng Required (Y/N) X X X X   X X X
TC Simulator ID #   X X   X X X  
Evaluator ID # X X X X X X X X
TC Inspector ID # X X X X X X X X
Geographical Area       X       X
Comments O=optional O O O O O O O O
Table 14-1 (Continued)

PV - Procedures Validation

MPV - Manoeuvres Proficiency Validation

MTV - Manoeuvres Training and Validation

LOE - Line Operational Evaluation

OE - Online Evaluation

FLM - First-Look Manoeuvres

PIC - Pilot In Command

SIC - Second In Command

CRP - Cruise Relief Pilot

FE - Flight Engineer

SO - Second Officer

PF/PNF - Pilot Flying/Pilot Not Flying

14.12.2 For each flight crewmember enrolled in a particular Qualification or Continuing Qualification curriculum, Transport Canada must be able to associate the data records applicable to that flight crewmember in that curriculum through logical grouping of the records or linkage by a common index number.

14.13 Data Analysis

14.13.1 The primary users of data reports are:

  • the air operator personnel; and
  • Transport Canada.

14.14Air Operator Data Analysis

14.14.1 AQP requires that the data collection conducted by the air operator for its own use in monitoring curricula will support more analytical detail and diagnostic functions than the data collected for submission to Transport Canada. Transport Canada expects the air operator to do an in-depth analysis of the effectiveness of the training provided.

14.14.2 Reporting of data is based on the analysis of the PPDB to provide information on the curriculum and participant groups (flight crewmembers, instructors, evaluators). Once the data is collected and entered into the PPDB, an analysis should be performed on the aggregate information. Statistical analysis of the proficiency data enables air operators to make an internal assessment of their performance.

14.14.3 Air operators should tailor these processes and techniques to suit their own requirements. Each air operator ’s data collection and performance assessment processes should be refined over time, based on their own practical experience. That is, the measures and processes should be optimized on an iterative basis to provide the degree of discrimination in crewmember performance needed to establish effective quality control over AQP curricula.

14.15Transport Canada Data Analysis

14.15.1 Data submissions to Transport Canada are primarily ratings and reason codes associated with performance measures taken at validation and evaluation gates and supporting data. The data, presented to Transport Canada in the table previously discussed is analyzed and allows POIs and other Transport Canada personnel to conduct trend analysis to monitor overall program effectiveness.

Date modified: