Government of Canada navigation bar

Symbol of the Government of Canada

Primary site navigation bar

Staff Instruction (SI) No. SUR-001

Surveillance Procedures

Internal Process Bulletin!

PDF PDF format (516 KB)

Internal documents and documents hyperlinked and stored on
Transport Canada's intranet mentioned in this document
are available upon request. See Contact Office below.

Issuing Office: Standards    
PAA Sub Activity Area: Aviation Safety Oversight Document No.: SI SUR-001
File Classification No.: Z 5000-32 Issue No: 04
RDIMS No.: 6022986 V8 Effective Date: 2010-11-17

1.0 INTRODUCTION

1.1 Purpose

The purpose of this Staff Instruction (SI) is to describe the procedures to be used by Transport Canada Civil Aviation (TCCA) personnel when conducting surveillance on Canadian certificate holders.

1.2 Applicability

This document is applicable to TCCA personnel holding appropriate Ministerial Delegation of Authority to perform surveillance activities.

1.3 Description of Changes

  1. The following amendments were introduced in this edition:

    1. Item 2.1(k) was amended to replace reference to CAD 7 with CAD ADM-005;

    2. Section 4.0 was amended to update the status of the Inspection and Audit Manual (IAM) and to clarify when TCCA inspectors can use audit checklist;

    3. Section 5.0 was amended to clarify when TCCA inspectors can use other surveillance activities;

    4. Paragraph 7.4(a) was amended to remove reference to company approved procedures.

    5. Paragraph 7.8.12(2) was amended to add information regarding the sharing of draft findings at the exit meeting;

    6. Section 8.0 is re-titled “Evaluation of Conformance”

    7. Paragraph 8.1(1) was added to identify scoring as an internal TCCA process;

    8. The table at Paragraph 8.2(5) was amended to classify scores of 1 or 2 as not being in conformance, and scores of 3, 4 or 5 as being in conformance;

    9. Throughout the document, several references to scoring were amended and are now expressed as “being in conformance”, “not being in conformance” or “exceeding the basic conformance level”;

    10. Paragraph 9.0(2) was added to establish that the report shall not include scores;

    11. The table in 9.6 was amended to replace the term “score” with “conformance (Y/N)”;

    12. A reference to the AWM 505 PVI worksheet was added in Appendix A, along with revised RDIMS numbers for CAR 703/704 and CAR 406 worksheets;

    13. The scoring criteria in Appendix B were amended to classify scores of 1 or 2 as not being in conformance, and scores of 3, 4 or 5 as being in conformance; and

    14. Removed Table D0 from Appendix B.
Return to Top of Page
Top of Page

2.0 REFERENCES AND REQUIREMENTS

2.1 Reference Documents

  1. It is intended that the following reference materials be used in conjunction with this document:

    1. Part I, Subpart 7 of the Canadian Aviation Regulations (CARs)—Safety Management System Requirements;

    2. Part III, Subpart 2 of the CARs—Airports;

    3. Part IV, Subpart 6 of the CARs—Flight Training Units;

    4. Part V, Subpart 61 of the CARs—Manufacture of Aeronautical Products;

    5. Subpart 573 of the CARs—Approved Maintenance Organizations;

    6. Part VII, Subpart 3 of the CARs—Air Taxi Operations;

    7. Subpart 704 of the CARs—Commuter Operations;

    8. Subpart 705 of the CARs—Airline Operations;

    9. Subpart 706 of the CARs—Aircraft Maintenance Requirements for Air Operators;

    10. Part VIII, Subpart 1 of the CARs—Air Traffic Services;

    11. Civil Aviation Directive (CAD) ADM-005, Issue 01, 2010-01-30—Mandatory Training to be Authorized to Act on Behalf of the Minister as a Civil Aviation Safety Inspector

    12. CAD SUR-008, Issue 01, 2009-01-22—Surveillance Policy

    13. CAD 107-004, Issue 02, 2009-08-19—Aviation Enforcement – Safety Management Systems;

    14. Staff Instruction (SI) SUR-002, Issue 02, 2010-09-22—Enhanced Monitoring;

    15. SI SUR-009, Issue 01, 2009-07-21—National Planning Standards

    16. Advisory Circular (AC) 107-001, Issue 01, 2008-01-01—Guidance on Safety Management Systems Development;

    17. Transport Canada Publication (TP) 8606, Edition 02, 2005-07-30—Inspection and Audit Manual (IAM);

    18. TP 13750, Updated 2004-03-08—Commercial and Business Aviation Inspection and Audit (Checklists) Manual;

    19. TP 13751, Edition 02, 2005-09-01—Aircraft Maintenance & Manufacturing Inspection and Audit (Checklists) Manual;

    20. TP 13798, Edition 01, 2001-11-01—General Aviation Inspection and Audit (Checklists) Manual;

    21. Transport Canada Civil Aviation (TCCA) form number 26-0672, Version 0706-02—Confirmation Request Form;

    22. TCCA form number 26-0673, Version 0710-02—Confirmation Request Tracking Form;

    23. TCCA form number 26-0674, Version 0706-02—Corrective Action Form;

    24. TCCA form number 26-0675, Version 0706-02—Corrective Action Form – Part 1;

    25. TCCA form number 26-0676, Version 0706-02—Finding Form;

    26. TCCA form number 26-0677, Version 0706-02—Parallel Finding Form;

aa.  TCCA form number 26-0678, Version 0510-01—Parallel Observation Form;

bb.  TCCA form number 26-0679, Version 0510-01—Evidence Log; and

cc.  American National Standards Institute (ANSI)/American Society for Quality Control (ASQ) Z1.4—Sampling Procedures and Tables for Inspection by Attributes.

2. Please note that for the purpose of this document, unless otherwise specified all sections referenced apply to this SI.

2.2 Cancelled Documents

Not applicable.

Note: By default, it is understood that the publication of a new issue of a document automatically renders any earlier issues of the same document null and void.
Return to Top of Page
Top of Page

2.3 Definitions and Abbreviations

  1. The following definitions and abbreviations are used in this document:

    1. Acceptance Validation Inspection: a process comprised of a documentation review and an on-site review to confirm that an element is documented, in place and understood, and is being utilized by the certificate holder under review. No score is assigned.

      Note:

      Acceptance Validations Inspections are being used as SMS progresses through the phased implementation and are applied to the requirements of each of the implementation phases. When the acceptance validation inspection is completed and all the outstanding issues addressed, the phase is accepted as meeting the applicable requirements. Acceptance Validation Inspections can also be applied to a new certificate holder as part of the initial certification.

    2. Administrative follow-up: the review of corrective action information that does not require a site visit.

    3. AMO: Approved Maintenance Organization.

    4. AOC:  Air Operator Certificate.

    5. Approved : approved by the Minister.

    6. Assessment : the surveillance activity conducted to evaluate effectiveness and level of compliance with the CARs. 

    7. Assessment Manager (AM) : a person appointed by the Convening Authority responsible for the planning and conduct of an assessment.

    8. Assessment/Program Validation Inspection Plan : the description of all the activities and arrangements for an assessment or a program validation inspection.

    9. ATIP : Access to Information and Privacy.

    10. BP : Best Practice.

    11. CADORS : Civil Aviation Daily Occurrence Reporting System.

    12. CAD : Civil Aviation Directive.

    13. CARs : Canadian Aviation Regulations.

    14. Certificate Holder : the individual or the corporation to which Transport Canada Civil Aviation has issued a certificate to conduct operations in accordance with the Canadian Aviation Regulations. The terms certificate holder, company and organization are interchangeable in this document and have the same meaning.

    15. Component : one of the six components of the Safety Management System Assessment Protocol Framework (Ref. Section 6.0, Table A).

    16. Conformance : the state of meeting regulatory requirements.

    17. Continuous Improvement Finding: means a finding against best practices (BP).

    18. Convening Authority (CA) : the individual responsible for authorizing and overseeing a surveillance activity.

    19. Corrective Action Plan (CAP) : a plan submitted in response to findings. The CAP outlines how the certificate holder proposes to address identified deficiencies and manage the associated risks.

    20. Depth : the time period covered during the surveillance activity.

    21. Determine or making a determination : use of discretion and judgement (i.e. forming conclusions based on facts, knowledge and experiences as related to the conditions of the event, or similar events) when making a decision as to the required course of regulatory action.

    22. Document : certificates, manuals, publications incorporated by reference, procedural instructions, uncompleted checklists and any other papers or equivalent electronic publications that detail company organization, policies or procedures, training curricula, personnel authorizations, etc., and that are required to hold a Canadian Aviation Document. Documents exclude records. 

    23. Element : one of the 17 elements of the Safety Management System Assessment Protocol Framework (Ref. Section 6.0, Table A).

    24. EM : Enhanced Monitoring.

    25. Enterprise : the holder of one or more Civil Aviation Documents. For example, a company holds an Approved Maintenance Organization Certificate, an Air Operator Certificate, an Approved Training Organization Certificate and a Design Organization Authority. The term Enterprise is intended to denote that surveillance is conducted on the whole enterprise rather than on an individual certificate.

    26. Enterprise Manager : the Transport Canada manager accountable for the surveillance and certification of the respective enterprise.

    27. Event : an action, incident, accident, occurrence, hazard or regulatory contravention (singular or repetitive in nature) relating to the operations conducted by a certificate holder.

bb.  Finding : a non-conformance to a regulatory requirement or, in the case of a continuous improvement finding, a non-conformance against a best practice.

cc. Hazard register : a list of the identified hazards in an organization.

dd.  IAM : Inspection & Audit Manual.

ee.  Input : the products, services and materials obtained from suppliers (internal and external) to produce the outputs delivered to customers.

ff.  Local Airport Authority (LAA) : a certificate holder that holds one or more airport certificates issued in accordance with CAR 302.03.

gg.  Non-conformance : the failure to meet regulatory requirements.

hh.  NoS : Notice of Suspension.

ii.  Observation : the outcomes obtained during documentation or on-site reviews that are used to confirm a specific requirement of a system, component or element.

jj.  OPI : Office of Primary Interest.

kk.  On-site follow-up : the review of corrective action information that can only be confirmed through a site visit.

ll.  Organization : the certificate holder subject to an assessment or program validation inspection.

mm.  Output : products, materials, services or information provided to a customer (internal or external), from a process.

nn.  PI : Principal Inspector.

oo.  Policy : a high level overall plan that outlines the goals and objectives of a certificate holder.

pp.  Procedure : a specified way to carry out an activity or process.

qq.  Process : a group of interrelated or interacting activities that convert inputs into outputs.

rr.  Process Inspection : an in depth review of the processes utilised to produce an output. A score is not assigned.

ss.  Process Step : one of several steps identified in a process map.

tt.  Program Validation Inspection (PVI) : a process comprised of a documentation review and an on-site review of one or more components of a Safety Management System (SMS) or other regulated areas of a certificate holder. A score is assigned. PVIs are conducted on a routine schedule and will utilize risk indicators to adjust the frequency as necessary.

uu.  Program Validation Inspection Manager (PVIM) : the person appointed by the CA to conduct a program validation inspection.

vv.  Quality Assurance (QA) : a planned and systematic set of activities implemented in a quality system so that quality requirements are clearly established and the defined process complies with these requirements.

ww.  RDIMS : Records, Documents and Information Management System.

xx.  Record : the specific details of events that have occurred during the performance of activities authorized by a Canadian Aviation Document. Records include, but are not limited to, journey log books and other aircraft technical records, maintenance releases, completed checklists, x-rays, incident reports, equipment-servicing records, training records, quality assurance (QA) findings, and submissions made under internal safety reporting systems, that is, under safety management systems (SMS). 

yy.  Regulatory Requirements : the Canadian Aviation Regulations (CARs) and Standards.

zz.  Review : the basic activity of an assessment or program validation inspection involving the systematic assessment of a component, element or system of a certificate holder to verify conformance to regulatory requirements.

aaa.  Risk Indicator : one of the risk indicators as listed in Appendix A of SI SUR-009.

bbb.  Safety case : a proactive study of the hazards and risks associated with change and the mitigations the organization will put in place to manage them. The safety case represents the justification for changes in the operating environment.

ccc.  Safety Management System (SMS) : a documented process for managing risks that integrates operations and technical systems with the management of financial and human resources to ensure aviation safety or the safety of the public.

ddd.  Safety risk profile : a prioritized list of safety risks specific to an organization.

eee.  Service : all activities, other than surveillance activities, conducted at the request, or for the optional use, of the recipients, and includes the issuance of certificates, licenses, approvals or other authorizations by the Minister to enable activities regulated by the CARs.

fff.  Scope : the functional areas that will be subject to a surveillance activity.

ggg.  SMS Database : a database used to provide or retain information or to assist in an assessment or program validation inspection.

hhh.  Standard : an established criterion used as a basis for measuring a certificate holder’s level of compliance.

iii.  Surveillance : all activities directly related to a certificate holder’s compliance with the applicable regulations and standards including, but not limited to, assessments, validation inspections, process inspections, audits, company visits (formal or informal), ramp, cockpit, cabin or aircraft inspections, facility or shop visits, review and approval of company documentation, correspondence by any means, as well as any other activity that could be used to gather information related to a certificate issued by the Minister.

jjj.  System : a group of inter-dependent processes and people working together to achieve a defined result. A system comprises policies, processes and procedures.

kkk.  Team Leader : the individual appointed to conduct an assessment or program validation inspection.

lll.  Team Member : an individual participating in an assessment or program validation inspection.

mmm.  TCCA : Transport Canada Civil Aviation.

nnn.  Tracing : to follow the progress of an item as it flows through a process. The item being traced may be tangible or intangible.

Return to Top of Page
Top of Page

3.0 BACKGROUND

  1. The introduction of Safety Management Systems (SMS) for the aviation industry is fundamentally changing the way Transport Canada (TC) approaches its surveillance responsibilities. Traditional surveillance methods including audits and inspections will be replaced with assessments, program validation inspections (PVIs) and process inspections as the primary surveillance tools, supplemented by audits when necessary.

  2. Traditional surveillance methods focused solely on determining regulatory compliance using a system of direct inspection of a certificate holder’s aircraft, personnel, records and other systems.  The new approach employing Assessments and PVIs will allow TCCA’s surveillance to evolve beyond compliance auditing and include the review of a certificate holder’s management system effectiveness. These changes are consistent with the principles of safety management systems where the certificate holder is expected to take an ownership role in proactively managing risks and have programs in place to ensure they comply with regulatory requirements. TCCA’s role is to ensure that Certificate Holders have effective policies, processes and procedures in place to accomplish this and that surveillance, in the form of assessments and program validation inspections, confirms that these policies, processes and procedures remain effective.

Return to Top of Page
Top of Page

4.0 INSPECTION & AUDIT MANUAL

  1. The information in this staff instruction supersedes the audit procedures detailed in TP 8606 - Inspection & Audit Manual (IAM).

  2. The audit checklists referenced in the IAM are still available to be used as an inspection tool, as part of the sampling activity during an assessment, a program validation inspection (PVI) or a process inspection, or when a compliance audit is deemed necessary.

  3. The IAM can be referenced at the following website:
    http://www.tc.gc.ca/eng/civilaviation/publications/tp8606-menu-5096.htm
Return to Top of Page
Top of Page

5.0 OTHER SURVEILLANCE ACTIVITIES

5.1 General

  1. Where surveillance resources are still available after the annual surveillance planning is completed in accordance with CAD SUR-008 — Surveillance Policy, other surveillance activities may be planned, as described in SI SUR-009 — National Planning Standards.

  2. These additional surveillance activities are not assigned a frequency in CAD SUR-008 —Surveillance Policy and shall be conducted in addition to regular scheduled surveillance. They are not intended to replace assessments and program validation inspections.

  3. These other surveillance activities are also available to be used as an inspection tool, as part of the sampling activity during an assessment, a program validation inspection (PVI) or a process inspection.
Return to Top of Page
Top of Page

5.2 Examples of Additional Surveillance

The following table contains some examples of additional surveillance:

Activity Description Sample ARASS References
Aerodrome Inspection Determine whether a particular aerodrome can safely support various types of commercial operations. Or to verify the aerodrome operator is meeting that requirements of the CARs. 12D029, A3D018, A3D020,
A3D022, A3D024, A3D028,
A3D028, A3D031, A3D045
Special Purpose Inspection/ Facility Surveillance To ensure that enterprises continue to comply with regulations and standards. 12D041, 13D047, 13D048,
3ND009, 33D038, 14D010,
14D014, 14D041, 14D054,
14D055, 14D056, 14D059,
15D009, 15D010, 15D024, 15D055,16D062
In-flight Inspection To ensure that the minimum operational standards are maintained. Includes flight and cabin crew. May include monitoring for any other deficiencies with respect to the air traffic control system, airports, or the navigation aids. 13D039, 13D040, 13D041,
13D042, 15D047, 15D013,
15D020, 15D034, 15D035,
15D047,
Ramp Inspection Ramp inspections/surveillance are conducted to evaluate the level of compliance with the Canadian Aviation Regulations during day-to-day operations. 13D081, 33D052, 33D057,
14D031, 14D032, 14D033,
14D034, 15D032, 15D049,
15D033, 16D077
Monitor of Instructional Techniques To monitor the performance and standard of instruction provided to the industry by individual instructors. Includes in-flight and ground school monitoring. 22D030, 22D107, 22D108
Regulatory Investigation When and as applicable, respond to complaints, accidents, incidents, or other reports such as CADORS, to determine if an incident involves any regulatory violations, shortcomings in procedures or aeronautical product failures. 22D074, 34D005, 34D006,
2DD017, 3RD023
Aircraft Inspection Inspection, sufficient in nature to determine that the aircraft, engine, propeller, appliances and equipment is approved and it meets regulatory requirements. 33D053, 33D054, 33D055,
15D031, 15D048
Return to Top of Page
Top of Page

5.3 Documentation requirements

  1. Details of the decision to conduct additional surveillance activities shall be recorded in the appropriate file (company, aircraft, NACIS, etc). This record shall contain justification of why the additional surveillance was necessary and what form the activity is to take (see table above for examples).

  2. Details of the surveillance activity, and any findings resulting from it, shall be recorded using existing documentation requirements for the specific activity (Aircraft Inspection form 24-008, Letter of notification 24-0019, Cabin Safety Inspection report 26-0598, etc). Where the activity has no formal documentation requirements, a record containing details of what was inspected or reviewed, the results of the activity and identification of any items requiring follow-up shall be added to the appropriate file.

Return to Top of Page
Top of Page

6.0 SAFETY MANAGEMENT SYSTEM FRAMEWORK

Table A outlines the SMS framework and follows the same structure as the TC SMS model, which is outlined in AC 107-001 - Guidance on Safety Management Systems Development. It has six components and corresponding elements.

 Table A – SMS Assessment Protocol Framework
Component Element
1. Safety Management Plan 1.1. Safety Policy
1.2 Non-Punitive Safety Reporting Policy
1.3 Roles, Responsibilities & Employee Involvement
1.4 Communication
1.5 Safety Planning, Objectives and Goals
1.6 Performance Measurement
1.7 Management Review
2. Documentation 2.1 Identification and Maintenance of Applicable Regulations
2.2 SMS Documentation
2.3 Records Management
3. Safety Oversight 3.1 Reactive Processes
3.2 Proactive Processes
3.3 Investigation and Analysis
3.4 Risk Management
4. Training 4.1 Training, Awareness and Competence
5. Quality Assurance 5.1 Quality Assurance
6. Emergency Preparedness 6.1 Emergency Preparedness and Response
Return to Top of Page
Top of Page

7.0 ASSESSMENT PROCEDURES

7.1 General

An assessment is the evaluation of effectiveness and level of compliance with the CARs.

7.2 Notification

  1. A certificate holder shall be contacted in writing 10 weeks prior to the planned on-site review portion of the assessment to confirm the assessment schedule. The notification period for assessments planned on shorter notice will be at the discretion of the Convening Authority (CA). This will allow sufficient time for the certificate holder to supply requested documentation and for TCCA personnel to conduct a complete review and follow up of the documentation prior to the on-site review.

  2. The method of notification shall advise the certificate holder to provide copies of documentation applicable to the assessment no later than 14 calendar days following the issuance of the notification letter (8 weeks prior to the planned on-site review). This may include approved company manuals, information incorporated by reference and any other documents essential to the assessment that provide information relating to the SMS. Refer to subsection 7.5 for additional information regarding applicable documentation.

  3. Section 7.5 provides additional instructions regarding certificate holders who fail to provide acceptable documentation within the timeline prescribed in (2).

  4. Where TCCA holds current copies of the information referenced in (2), the request for documentation shall specify which documentation is needed.

  5. The letter of notification shall request any Occupational Safety and Health (OSH) information that may be applicable to TCCA personnel during their on-site activities.

  6. The letter of notification shall request confirmation from the certificate holder whether or not they want best practices to be identified during the assessment. Such a letter shall also contain a brief description of what constitutes a best practice.

  7. The letter of notification may also include a copy of the documentation review guide to be completed by the certificate holder prior to the documentation review, along with the due date for its return to TCCA.

Return to Top of Page
Top of Page

7.3 Team Selection

Team selection including team member terms of reference, qualifications and responsibilities are specified in Section 18.0.

7.4 Assessment Plan

  1. The Assessment Manager (AM) shall develop an assessment plan for the CA’s approval. This plan ensures that the assessment will be conducted in an organized manner and in accordance with predetermined criteria. Appropriate sections of the plan will be distributed to each member of the team to provide guidance and direction throughout the assessment. In addition to this, the AM may wish to provide the certificate holder with portions of the plan. The assessment plan shall address the following items, as applicable:  

    1. Objective

      The plan shall state the objective of the assessment, which is to ensure the certificate holder is in compliance with regulatory requirements.

    2. Scope & Depth

      The Scope will specify the functional areas to be reviewed, which can include the whole enterprise (all certificates), specific certificates only or selected bases of operation. The Depth will indicate the time period covered by the surveillance activity.

    3. Certificate Holder Description

      The assessment plan shall provide specific information on the certificate holder. This will provide the reader with an overview of the certificate holder and will include information pertaining to the following:

      (i) a brief description of the organization and its operations including services provided, aircraft types and number, bases of operation and any other pertinent information;

      (ii) the number of management personnel to be interviewed and their location; and

      (iii) the number of employees to be interviewed and their location.

    4. Management Personnel

      The plan shall include a listing of management personnel who are relevant to the assessment including the person’s name, title and contact information.

    5. Team Composition

      The plan shall include team information indicating the following where applicable:


      (i) names of the CA, AM, team leader(s) and roles and responsibilities where necessary;

      (ii) support personnel, principal inspector(s), team members, observers and specialists; and

      (iii) team member contact information.

      Note:

      A table or an organization chart can be used to display this information.

    6. Methodology

      Methodology shall reference documentation reviews and the use of interviews of management and working level personnel from a variety of departments. Sampling of documents and other records will also be used to confirm information obtained during the interview process. Reference can be made to the use of applicable assessment worksheets from this SI.

    7. Communications

      The plan shall identify the communication protocols that the team will follow. This will include internal communications within the team and TC personnel, as well as external communications with the certificate holder, external agencies and the public.

    8. Foreign Travel

      When foreign travel is required or contemplated, the following information shall be researched:

      (i) the requirement for government passports, visas, inoculations, notification of foreign civil aviation authorities, and co-ordination with External Affairs; and

      (ii) the availability of voice and data communications, diplomatic courier service and foreign currency exchange.

      Note:

      The certificate holder may be a source for some of the above information.

    9. Budget

      The following will be indicated in the assessment plan:

      (i) amounts budgeted for overtime, travel, accommodation and daily allowances;

      (ii) budget contingency if requested by the CA; and

      (iii) team member responsibility to report deviations from the budget plan.

    10. Assessment Schedule

      A schedule shall be provided indicating the following team information where applicable:

      (i) team member travel dates to and from the assessment;

      (ii) specialty area assignments including the applicable component and element summary responsibilities; and

      (iii) pre-assessment and on-site assessment assignments including start/complete dates/times.

    11. Parallel Findings and Observations

      The plan will indicate the process team members will follow when parallel findings or observations are identified. Refer to Chapter 5 of the IAM for procedures regarding parallel findings and observations.
Return to Top of Page
Top of Page

7.5 Preliminary Documentation Review

  1. Prior to the start of the documentation review, the AM (or assigned TL) shall perform a preliminary review of the documentation submitted to verify its completeness.

  2. Where the preliminary review identifies no deficiencies with the submitted documentation, the assessment team shall proceed with the documentation review in accordance with section 7.6.

  3. Where the preliminary review identifies minor or moderate deficiencies, the AM shall contact the certificate holder in writing to request that the missing or incomplete documentation be provided to TCCA within 14 calendar days. Failure to meet this deadline shall result in certificate action.

  4. Certificate holders that fail to submit acceptable documentation in response to TCCA’s second request for documentation shall be issued an NOS. The NOS shall come into effect 14 calendar days following its issuance if the certificate holder does not satisfy the Conditions for Reinstatement by submitting the requested documentation. For an assessment, the NOS shall be based on failure to meet the requirements of Section 107.02 of the CARs.

  5. Certificate holders that fail to provide any requested documentation in accordance with the timeline described in the notification letter or certificate holders who have major deficiencies in their document submission shall immediately be issued an NOS that will come into effect 14 calendar days following its issuance if the certificate holder does not satisfy the Conditions for Reinstatement by submitting the requested documentation. For an assessment, the NOS shall be based on failure to meet the requirements of Section 107.02 of the CARs.

Return to Top of Page
Top of Page

7.6 Documentation Review

  1. The documentation review shall take place 4 to 6 weeks in advance of the on-site review in order to ensure that:

    1. inspectors have the necessary time to conduct a complete review of the documentation;

    2. the certificate holder has sufficient time to supply additional requested documentation;

    3. inspectors have the necessary time to conduct a follow-up on any documentation questions prior to the on-site review; and

    4. assessment managers are able to make a timely decision whether to proceed with the on-site portion of the assessment or not.

  2. This activity includes a thorough review of all files and documentation that are relevant to the certificate holder and the scope of the assessment. The following shall be completed during this portion of the assessment where applicable:

    1. ensure that all reference manuals and documents to be used during the assessment are readily available and include the latest approved amendments. A request to the certificate holder shall be made to confirm the revision status of any manual subject to review;

    2. review the certificate holder’s approved manuals for conformance to the appropriate standard;

      Note:

      Documentation requested from the certificate holder will be limited to those publications that provide process, policy and procedures only. Company generated records, such as reports or other data, pertaining to the outcome of individual programs will be reviewed on-site as part of the sampling plan.

    3. review other related information relevant to the certificate holder and the scope of the assessment to include:

      1. previous assessments, validation inspections, process inspections or audits including corrective actions and follow-up where applicable;

      2. accident or incident data, including Civil Aviation Daily Occurrence Reporting System (CADORS) reports;

      3. previous compliance history including any enforcement actions; and

      4. exemptions, waivers, approvals, limitations and authorizations.

    4. identify areas that require further review during the on-site portion of the assessment; and

    5. select the assessment worksheets from Appendix A.

  3. The results of the document review shall be documented. The SMS Document Review Guide, RDIMS 4198634 or a similarly formatted document shall be used to record the results of the documentation review.

  4. Where, as part of the documentation review, assessment team members choose to use process mapping in order to assess the certificate holder’s processes and procedures, they shall follow the procedures detailed in Section 17.0 – Process Inspection.
Return to Top of Page
Top of Page

7.7 Team Meeting prior to the On-site Review

  1. The purpose of the team meeting is to:

    1. understand the administrative and logistical needs for the assessment; and

    2. gain a common understanding, prior to going on-site, of how the different processes function together.

  2. This meeting shall include the following agenda items as applicable for the assessment:

    1. administrative details;

    2. assessment plan review and amendment, ensuring that all team members have received appropriate portions of the plan;

    3. budget information, including tracking of overtime and travel expenses;

    4. conflict of interest and confidentiality;

    5. forms administration;

    6. use of measurement criteria and worksheets where applicable, as well as use of Master Document RDIMS 4651358 for regulatory references to expectations and list of prepared questions;

    7. communications;

    8. discussion on the result of the documentation review, including the interaction between company specific processes;

    9. overview of the on-site review; and

    10. where possible, a briefing by all principal inspectors (including, as applicable, flight operations, maintenance, manufacturing, aircraft certification, aerodromes, air traffic control, cabin safety, occupational health and safety, and dangerous goods) on the certificate holder’s current activities, trends, performance and previous surveillance history including corrective action and follow-up. Where the principal inspectors are not available for the team meeting, alternate means shall be used to ensure the surveillance team receives their input. This can include teleconferences, memos and briefing summaries.
Return to Top of Page
Top of Page

7.8 On-Site Review

7.8.1 On-Site Review Plan

  1. General

    1. The AM shall prepare an on-site review plan. This plan will provide a defined structure for the on-site activities by identifying key personnel to be interviewed as well as employees, records, products, processes and procedures to be sampled in order to gain a better understanding of the organization.

  2. Key Personnel Interviews

    1. The on-site review plan shall contain details of the planned interviews with the certificate holder’s key personnel. These interviews will give certificate holders the opportunity to describe elements of their SMS and, where applicable, demonstrate compliance through the review of records and documents.

    2. The AM shall identify the certificate holder’s key personnel for each surveillance activity.

    3. For the purpose of this SI, key personnel includes the accountable executive, any person assigned certificate responsibilities in accordance with the CARs (DFO, PRM, airport manager, QA manager, etc.), or any other person assigned key responsibilities as part of the certificate holder’s SMS (SMS implementation manager, event investigator, safety officer, etc.)

  3. Sampling of Employees and Records

    1. The on-site review plan shall contain details of planned interviews with the certificate holder’s employees. Employee interviews shall normally apply to technical employees, however TCCA retains the ability to interview any employee who’s duties could impact aviation safety.

      Note: 

      Technical employees include anyone performing functions that are mandated by the CARs. This would include personnel working directly on-aircraft, as well as personnel with off-aircraft responsibilities such as technical records.

    2. The employee interviews shall be used to gather information with regards to the elements of the certificate holder’s SMS.

    3. The AM shall use the chart below to determine the number of interviews to be completed (sample size), based on the number of technical employees working for the certificate holder (lot size).

      Lot Size 1-13 14-150 151-280 281-500 501-1200 1201-3200
      Sample Size All 13 20 29 34 55

    4. Once the number of employee interviews required to be performed has been determined, the AM shall select the employees to be interviewed as follows:

      1. select a random sampling of technical employees from the overall population of technical employees working for the certificate; or

      2. select a stratified random sampling of technical employees where the lot size is divided into different areas and each area is designated the amount of sample interviews in proportion to its size. The AM shall establish these sample areas based on the overall knowledge of the certificate holder.

        For example, an AOC certificate holder could be divided into 3 areas - flight operations, cabin safety and dispatch. If flight operations make up approximately 60% of the overall number of technical employees, it is allocated 60% of the employee interviews.

    5. Based on the documentation review and any risk indicators, the AM shall determine which records to sample on-site.

    6. Once the records to be reviewed have been identified, the AM shall randomly select the records in accordance with the chart included in (c) above.

  4. An On-site Review Plan template is listed in Appendix A. The template provides several examples to illustrate how a sampling plan can be developed. Delete the examples when using the form.
Return to Top of Page
Top of Page
7.8.2 Entry Meeting

  1. The entry meeting is an opportunity for the assessment team to address company management, to outline the assessment process and answer any questions from the company relating to assessment activities.

  2. The meeting shall outline administrative requirements so that the assessment may be conducted both effectively and efficiently and minimize any disruption.

    Note:

    A briefing on the Access to Information and Privacy (ATIP) Act is not normally required. If an ATIP request is received during surveillance activity, the ATIP office will provide the necessary information to comply with the request including information to advise the certificate holder under review.

  3. Members of the assessment team will attend the entry meeting as directed by the AM or team leader, as applicable. Additional TC personnel may attend as directed by the CA.

7.8.3 Assessment Team On-Site Briefing

  1. Once on site, the AM or team leaders shall hold a briefing for the assessment team that shall include, but not be limited to, a discussion regarding the following items:

    1. facility familiarization;

    2. protocol for the use of company resources;

    3. identification of key company contacts; and

    4. briefing on OSH requirements.
Return to Top of Page
Top of Page
7.8.4 On-Site Review

  1. During this phase, the assessment team shall:

    1. conduct interviews and sampling per the sampling plan;

    2. gather evidence to support observations;

    3. analyse observations collected from both the documentation and on-site reviews;

    4. determine if:

      1. the certificate holder is in compliance with regulatory requirements; and

      2. the SMS is effectively managing compliance to the regulations;

    5. prepare findings of non-conformance as applicable.

  2. The results of the on-site review shall be documented. The SMS On-Site Review Guide, RDIMS 4208267 or a similarly formatted document shall be used to record the results of the on-site review.
7.8.5 Interviewing

  1. Interviews are verbal communications with the certificate holder’s personnel and range from an informal discussion to a pre-arranged interview. Interviews are important to assessors in that they permit the assessor to:

    1. gain a thorough understanding of the certificate holder’s processes and procedures;

    2. determine whether the applicable SMS information documented in the approved manual(s) is actually in use;

    3. assess the knowledge of personnel pertaining to their duties and responsibilities; and

    4. determine the effectiveness of an certificate holder’s processes and procedures.

  2. The following guidelines will be useful when preparing for an interview:

    1. prepare an interview schedule and provide a copy to the certificate holder. A schedule shall be provided for all interviews including those considered “informal”;

    2. prior to the interview, carefully prepare by reviewing the question(s) associated with the component or element being reviewed. Where the question(s) in the applicable assessment worksheet do not adequately address the specific circumstance, the question can be modified to suit;

    3. explain why the interview is taking place;

    4. use open ended questions and avoid complex questions or phrases;

    5. the use of “yes” or “no” questions should be avoided. If “yes” or “no” questions are used they should be supplemented with additional questions requesting further explanation;

    6. ensure that questions are understood, paraphrase where necessary or restate the question;

    7. listen carefully to answers and allow the interviewee to do most of the talking;

    8. have two persons attend the interview, so that one can ask questions and the other can record responses. If a second person is not available to assist, explain to the interviewee that you will be pausing to take notes to ensure the accuracy of recorded information;

    9. avoid being side-tracked from your original objectives;

    10. terminate the interview if the atmosphere becomes highly negative; and

    11. thank the interviewee at the conclusion of the interview.

  3. Team members responsible for conducting interviews shall provide a written summary to the AM or TL. This summary along with all other notes taken during interviews must be retained in accordance with Section 19.0 – Information Management. Interview notes shall accurately reflect the information obtained and not refer to the interviewee by name. The assessment team shall establish an interview list that identifies personnel by position title or a specific employee code. Names of those interviewed will not be entered in the assessment report.
Return to Top of Page
Top of Page
7.8.6 Analysis

  1. The AM shall coordinate the analysis of all the information gathered as part of the surveillance activity. This analysis shall consider information resulting from:

    1. the documentation review; and

    2. the on-site review, including all records sampling and interview results.

  2. In addition to assessing the effectiveness of individual SMS elements under review, the analysis shall consider the interactions between each SMS element in order to determine if the management system is, in fact, performing as a system. In most cases, assessing if an organization is meeting expectations for a particular element without considering its impact on other elements is not sufficient to ensure it has been applied system wide.

  3. When assessing effectiveness of an element, team members sampling or interviewing against the same element shall compare their results and objectively analyse them in order to ensure the final score applied to that element is an accurate reflection of the certificate holder’s overall performance in that area. Because interviews can be conducted by different team members in different areas of the organization against the same element, and because the results of these interviews can vary significantly from one area of the organization to another, inspectors must consider all information before making a determination on system compliance and effectiveness.

  4. In addition to assessing the effectiveness of an element, the inspector also makes a determination with regards to compliance to regulatory requirements. In accordance with section 7.8.1, the assessment team shall assess compliance to the CARs by sampling to the extent required in order to establish that the certificate holder’s SMS is robust enough to ensure compliance.

  5. The inspectors shall compare the results of their analysis to the measurement criteria contained in Appendix B and, using the scoring procedures detailed in Section 8.0, the AM shall assign a score to each element under review.

  6. Before assigning a score, if there is doubt as to an expectation being met, the AM must decide if additional documentation reviews, interviews or record sampling is required in order to make a final determination.

  7. If the analysis determines that all level 3 expectations have been met, and the certificate holder has identified that they would like best practices identified, the analysis can continue to determine if any level 5 expectations are present.

  8. The analysis is complete when the assessment team has assessed each element.

  9. The results of the analysis shall be documented in a decision record. This decision record, along with any supporting notes generated during the analysis shall be retained in accordance with Section 19.0 – Information Management.
Return to Top of Page
Top of Page
7.8.7 Use of Assessment Worksheets

  1. Assessment worksheets have been developed for each Element to assist inspectors during the interview process. Appendix A provides a RDIMS reference list for each worksheet. Each worksheet is divided into four sections. Section A contains a list of expectations, Section B contains a list of corresponding questions, Section C provides space to document responses and Section D contains the measurement criteria and scoring levels. A blank page is also included to enter additional questions as necessary.

  2. The form is a Word table and provides space for tombstone information as well as space for additional questions and responses. Team members will use this information during daily briefings and during the analysis to determine if the element is effective and meets the applicable regulatory requirements.

  3. The questions are meant to assist in the assessment and are intended for guidance only. It is not mandatory to ask all the questions. The phrasing of these questions and their categorisation is not meant to be prescriptive. If the assessor feels that enough information has been gathered to effectively assess that element or if a different “open-ended” question will elicit a better response, it can be used. The assessor may also re-phrase the question if it is more appropriate to do so. It is expected that answers to any question would lead to others not shown on this list.

  4. With the exception of questions designated as BP, questions in Section B are designed to address the level 3 criteria. If questions are re-worded, assessors must ensure that the questions used adequately address these expectations. Questions labelled BP are designed to address level 5 criteria.

  5. Certain questions are suggested for direction to the accountable executive, management and non-management and are arranged in that order. Questions addressing BPs are listed at the end of each management/employee group.

  6. Completed worksheets shall be retained in accordance with Section 19.0 – Information Management.

Return to Top of Page
Top of Page
7.8.8 Assessment Findings

  1. Assessment findings must be prepared accurately as they form the basis of the assessment report. Finding Form (form #26-0676) is available in the Transport Canada Forms Catalogue, http://tcapps/Corp-Serv-Gen/5/forms-formulaires/search.aspx.

  2. A finding may be issued for each element during the documentation review and an additional finding issued for each element during the on-site review (maximum of 34 findings).

  3. All non-conformances identified during a surveillance activity shall be documented as a finding against one of the 17 SMS elements. In all cases, findings are generated as a result of the analysis and should be linked back through the sampling plan to the expectation and element being assessed.

  4. When assessing a particular element, there are several possible scenarios:

    1. processes, policies and procedures for a particular element are documented and meet regulatory requirements, and on-site review and subsequent analysis indicates they are being followed in all cases. There are no finding against either the documentation or the on-site portion of the assessment of this element;

    2. processes, policies and procedures for a particular element are documented and meet regulatory requirements but on-site review and subsequent analysis indicates they are not being followed in all cases. Examples would support a finding against the on-site portion of the assessment of this element;

    3. processes, policies and procedures for a particular element are documented and do not meet regulatory requirements but on-site review and subsequent analysis indicates that alternate procedures are being used. Examples would support a finding against the documentation portion of the assessment.

  5. Findings are intended to address regulatory non-conformances at the systems level. Examples used to illustrate this should clearly demonstrate that the system is not functioning as intended. A single example of a minor deviation does not demonstrate a systemic breakdown unless it can be shown that safety has been compromised.

  6. Findings shall be classified using the following criteria;

    1. Minor – A finding is considered minor where a surveillance activity has identified that a component and/or element has been maintained and demonstrated effective, but where the component or element requires administrative enhancement to existing areas (for example, formatting issues).

    2. Moderate – A finding is considered moderate where a surveillance activity has identified that a component and/or element has not been fully maintained and non-conformance findings indicate that the component is not fully effective, but where the certificate holder has clearly demonstrated the ability to carry out the activity required by the component/ element and a simple modification to their process is likely to correct the issue.

    3. Major – A finding is considered major where a surveillance activity has identified examples that the component and/or element has not been established, maintained and adhered to or is not effective, and a systemic failure is evident. A major finding will typically require a more rigorous and lengthy CAP than a minor or moderate finding. An NoS with a 30 day notice period and Enhanced Monitoring should be considered.

      Note: The safety management system is made up of various interconnected elements, all dependent upon each other to meet the system’s objective of managing risk to acceptable levels. A systemic failure can be characterized as any one of these elements failing in a manner that impedes the ability of other elements to function as intended, thus resulting in the management system being unable to demonstrate that it has appropriately reduced risks to acceptable levels and/or is unable to demonstrate the ability to do so. 

      For example: An ineffective Investigative and Analysis element impeding the ability of the Reporting systems to generate effective corrective actions, or, the Safety Management Plan not clearly identifying roles and responsibilities which impedes the ability of the Training Program to ensure that personnel are competent to perform their duties.

    4. Critical– Where a finding resulting from a surveillance activity identifies that a component has not been implemented or is wholly ineffective (i.e. scored as a level (1) a Notice of Suspension (NoS) with immediate effectivity is appropriate in accordance with Section 7.1 (a) of the Aeronautics Act. A lack of a maintained and effective SMS in accordance with the applicable CAR reference would be cited as grounds for suspension. For an assessment, the NoS is issued as a failure to meet Section 107.02 of the CARs and the appropriate paragraphs of Section 107.03 of the CARs and the parts and subparts relevant to the Certificate holder being reviewed. For a PVI, the NoS is issued as a failure to meet the parts and subparts relevant to the Certificate holder being reviewed.

  7. If during the course of the surveillance activity an immediate threat to safety is detected or is determined likely to exist, immediate action shall be taken in accordance with existing procedures. Dependent on the nature of the threat, immediate action could include the issuance of a detention notice for an aircraft, the suspension of an aircraft’s C of A, or the issuance of an immediate threat NoS for the certificate under Section 7.(1) of the Aeronautics Act.

  8. Where possible, findings shall be written against Section 107.02 of the CARs - Establishing a Safety Management System, subsequently they should be written against parts and subparts relevant to the Certificate holder being reviewed.

  9. All evidence and supporting documentation will be included with the completed Finding Form for review by the applicable team leader and the AM. This documentation will not be included in the assessment report, but will be retained in the applicable company file. An evidence control log, which will meet the requirement for an evidence receipt where applicable, is available in the Transport Canada Forms Catalogue, form #26-0679.
Return to Top of Page
Top of Page
7.8.9 Issuance of Findings During the Assessment

  1. Where it is determined that corrective action and subsequent follow-up to a non-conformance is required in a period less than that which occurs through the use of an approved Corrective Action Plan (CAP), an assessment finding may be issued during the assessment. This type of finding is usually made where safety is compromised and corrective action is required immediately, or at the very least, prior to completion of the assessment. The corrective action section of the Finding Form includes a checkbox and a line to specify the date/time that corrective action is required by.

  2. The certificate holder must respond to the finding by the date/time specified in the corrective action section of the Finding Form using a Corrective Action form. Corrective action forms are available in the Transport Canada Forms Catalogue, forms number 26-0674 or 26-0675.

  3. Issuance of findings during the assessment will only be contemplated when the CA, AM and applicable team leader are in agreement with such action.

  4. For the purposes of follow-up to corrective actions taken during the assessment, the applicable team leader or AM will accept or reject submitted corrective actions by signing the applicable Corrective Action form.

7.8.10 Completing the Finding Form

  1. When completing these forms, assessors shall complete the following:

    1. at the top of the Finding Form:

      1. correctly identify the certificate holder; where applicable, use the certificate holder’s name as found on the Canadian Aviation Document;

      2. enter the location of the base or sub-base where the non-conformance applies;

      3. enter the certificate holder’s TCCA file number;

      4. identify the component or element area; and

      5. identify the finding number, e.g. 1.2-1 or 3.3-2.

    2. in the “Non-Conformance with” section:

      1. correctly identify the regulatory requirement to which the non-conformance applies;

      2. check the box “which states in part” when a partial quotation will be used (segmenting), then quote the regulatory requirement or company approved procedure word for word, separating segments as necessary with the notation “…” and ensuring that the quotation is relevant; and

      3. when segmenting, quote a sufficient portion of the text to clearly identify the regulatory requirement while avoiding the use of unnecessary words.

    3. in the “Examples” section:

      1. indicate the most applicable examples from the documentation review or observations taken during on-site interviews that clearly illustrate the systemic related non-conformance with the regulatory requirement;

        Note: 

        The number of examples should be appropriate to the size and complexity of the certificate holder and the specific element under review. One example of an issue regarding the non-punitive reporting program may be appropriate in an organization with fewer than 10 employees but not for an organization employing several hundred.

      2. make reference to any evidence or supporting documentation that confirms the validity of the finding; and

      3. in the “Corrective Action required by” section:

        1. check the appropriate box;

        2. where applicable, specify the date/time that corrective action is required by; and

        3. specify the name of the assessor and the date the finding was made.
Return to Top of Page
Top of Page
7.8.11 Daily Team Briefings

  1. Team briefings shall be held at the end of each day during the assessment to:

    1. ensure adherence to the assessment plan;

    2. validate confirmation requests (if used) and any findings;

    3. review interview results with other team members;

    4. resolve issues or problems arising from the day’s activities; and

    5. provide the team leader with the information necessary to update the AM, where applicable.
7.8.12 Exit Meeting

  1. Upon completion of the assessment, the AM shall convene an exit meeting with the certificate holder’s senior management to brief them on the results. The CA or his representatives from the assessment management team shall attend. When present, the CA may chair the meeting or simply attend with the team.

  2. Where applicable, the assessment team will have briefed the certificate holder’s management on potential findings and areas of concern during the daily meetings so debate between the team and the certificate holder should not occur. Whenever possible, the AM will provide a draft copy of any findings of non-conformance. The meeting chair shall advise the certificate holder that they will have an opportunity to respond formally to the assessment report in their CAP submission.

  3. The AM shall advise the certificate holder that the assessment report will be forwarded to them within the time period referred to in subsection 9.1 and that a CAP must be submitted to TCCA within 30 calendar days after the report has been issued. Details of the corrective action process shall also be discussed.

Return to Top of Page
Top of Page

8.0 EVALUATION OF CONFORMANCE

8.1 Scoring Levels

  1. Scoring is an internal TCCA process used to determine the level of conformance of a Certificate Holder. Scoring provides a standardised methodology to determine appropriate actions resulting from non-conformance with the CARs and may be used in the determination of surveillance frequencies.

  2. Scoring levels are based on a set of defined expectations. The expectations relate to an element being assessed. For example, a safety management plan must contain a safety policy. An expectation of the safety policy is that it must contain a clear declaration of commitment and objectives. As SMS are progressive in their development, we expect to see continuous improvement in the system; we also expect to see changes in the safety policy.

  3. The expectations are not intended to be used as a checklist. They are provided as indicators for understanding what an effective element must contain and for standardizing the assessment or PVI process. To understand whether the element is effective or not, it must be considered with other elements in the context of the processes that make up the component and in some cases, the system as a whole.

  4. Each element to be scored will be assigned a number from 1 to 5 (using whole numbers only) based on a comparison of the observations against a set of specific measurement criteria. The criteria are to be used as guidelines for scoring the various elements.

  5. Scoring awards are assessed in the following manner:

    1. A score of 1; the element is not documented and/or implemented, or it is completely ineffective;

    2. A score of 2; the element is partially implemented but may not be fully effective. In other words, the certificate holder does not have all of the criteria required for an award level of 3;

    3. A score of 3; the element meets the regulatory requirements;

    4. A score of 4; the element exceeds the regulatory requirements. To receive this award level, the element meets all of 3 plus some aspects of 5; and

    5. A score of 5; the element meets all the criteria for an award level of 3 plus all of the additional requirements listed under the criteria for that element. To achieve an award level of 5, a certificate holder would have to meet the regulatory requirements as well as demonstrate industry best practices at a very high level.
Return to Top of Page
Top of Page

8.2 Scoring Using the Measurement Criteria

  1. When awarding a score to an element, the assessor shall consider whether the element meets the established criteria. The assessor shall consider all of the expectations and apply the scoring criteria to determine the award level.

  2. If the assessor cannot confirm that all the level 3 expectations have been addressed following the documentation and on-site review, additional interviews and further sampling may be necessary to gather information and confirm results.

  3. If all of the level 3 expectations are not met for any element, i.e. there are findings of non-conformance generated, then the company can not be awarded a score greater than 2.

  4. When assigning a score, the assessor must use the following scoring rules:

    1. Scores are only assigned at the element and component level. An assessment score will not be assigned.

    2. Scoring an Element

      1. Individual elements shall first be assessed to verify that they meet the level 3 measurement criteria. Once that is established, the assessor can proceed to verify if the elements meet additional expectations that could lead to a score higher than a level 3.

        Note:

        Certificate Holders are expected to achieve a level 3, however TC encourages continuous improvement and, based on the information below, can assign a higher score.

      2. To assign a level 3 score, assessors must ensure that each level 3 expectation for the element in question is documented, implemented and utilized, and that the element is effective and interacts effectively with other SMS elements within the certificated areas of the organization. To interact effectively, the element must have clear linkages with other aspects of the SMS, as appropriate, and be subject to continuous improvement.

        Note:

        An example of a link between elements of an SMS is having an organization’s goals and objectives based on the safety policy.

      3. To assign a level 4 score, assessors must verify that all level 3 expectations and a significant portion of level 5 expectations for the element in question are documented, implemented and utilized, and that the element is effective and interacts effectively with other SMS elements in both certificated and non-certificated areas of the organization.

        Note:

        For the purpose of these rules, a significant portion means equal to or greater than 50% of the level 5 expectations. In the case where there are an odd number of level 5 expectations, the proportion must be above 50 % (e.g. 3 expectations out of a total of 5)

        Note:

        Non-certificated areas may include, but are not limited to, baggage handling, catering, security, fuel service, snow removal, hangar operations, marketing, and any other area not associated with the CARs that supports the activities conducted under the certificate.

        For example, to interact effectively with non-certificated entities (internal and external), it is reasonable to expect the following in relation to a safety reporting program:

        1. it must contain information from key sources other than the certificate holder itself;

        2. the certificate holder must provide basic awareness training;

        3. the certificate holder must provide access to the reporting program; and

        4. it follows-up with the non-regulated organization on submitted reports and on any action taken, as well as the number of reports received.

      4. To assign a level 5 score, assessors must verify that the all level 3 expectations and all the level 5 expectations for the element in question are documented, implemented and utilized, and that the element is effective and interacts effectively with other SMS elements in both certificated and non-certificated areas of the organization. An organization that does not apply its SMS to non-certificated areas cannot be considered a level 5 because it does not have the ability to consider hazards and risks outside of the certificated areas. It does not therefore have a complete safety risk profile. Likewise, an organization that does not subject all certificated and non-certificated areas to continuous improvement cannot be assigned a score of 5 because it is not applying the plan-do-check-act (PDCA) management principles in all areas of its operation.

    3. Scoring a Component

      1. Subject to (ii), assessors shall assign a component score by establishing the average score from the elements that make up the component. Scores shall be rounded down to the nearest whole number from the average score, in accordance with the following table:

        Average Score of Elements Assigned Component Score
        1.0 – 1.99 1
        2.0 – 2.99 2
        3.0 – 3.99 3
        4.0 – 4.99 4
        5.0 5

        Note:

        In single element components, the element score is the same as the component score (for example, components 4, 5, and 6).

      2. To assign a level 3 component score, assessors must verify that all the associated elements meet the regulatory requirement (i.e. score a level 3 or more). If one or more elements of the component received a level 2 score, the component score will be 2 or lower.

      3. To assign a level 4 component score, assessors must verify that all the associated elements meet the regulatory requirements and clearly demonstrate that the elements interact effectively between each other.

      4. A component can receive a level 5 score only if all of its elements have been assigned a level 5 score individually.

  5. The example shown in the table below demonstrates how the assessment tool works using Element 1.2, Non-Punitive Safety Reporting Policy. The table combines all the expectations and separates them into level 3 and level 5 criteria for scoring purposes.

     Safety Management Plan – Non-Punitive Safety Reporting Policy
    Score Level Measurement Criteria
    Not in Conformance
    1 Safety-related reports or inadvertent errors result in punitive action being taken against individuals.
    2 (3) less some aspects
    In Conformance
    3

    1. There is a policy in place that provides immunity from disciplinary action for employees that report hazards, incidents or accidents.

    2. Conditions under which punitive disciplinary action would be considered (e.g. illegal activity, negligence or wilful misconduct) are clearly defined and documented.

    3. The policy is widely understood within the organization.

    4. There is evidence that the organization is applying the policy.

    4 All of (3) plus some aspects of (5)
    5 All of (3), plus all of the following:

    • Personnel express confidence and trust in the policy. The policy is applied throughout certificated and non-certificated areas of the organization.

    • The organization has letters/Memoranda of Understanding (MOU) between employees and/or third party contractors and management. The purpose of these letters is to document the disciplinary policy, and the manner in which it will be implemented.
Return to Top of Page
Top of Page

8.3 Failure to Achieve a Score of 3

  1. Failure to achieve a minimum score of 3 on any of the components will result in the following action:

    1. Where any component is assessed at an award level of 2 and interpretation of the findings indicate minor or moderate issues as indicated in subsections 7.8.8.6(a) and (b), the certificate holder must submit a CAP as described in section 10.0 – Corrective Action Plan.

    2. Where any component is assessed at an award level of 2 and interpretation of the findings indicate minor or moderate issues as indicated in subsections 7.8.8(6)(a) and (b) and, after analysis of the findings as a whole, it is determined that there is no systemic breakdown, the CA may use his/her discretion to provide the Certificate Holder with a letter of acknowledgement stating that the Certificate Holder is now considered to be in conformance with the CARs. The letter will only be issued on the condition that the identified deficiencies are corrected, to the satisfaction of the PI, within 30 calendar days following the CAP approval.

    3. In order to issue a letter of acknowledgement in accordance with (b), all findings must be addressed simultaneously.

    4. A copy of the letter of acknowledgement issued in accordance with (b) shall be appended to the surveillance report. The original component conformance status recorded in the surveillance report shall not be amended.

    5. Where any component is assessed at an award level of 2 and interpretation of the findings indicate major issues as indicated in subsection 7.8.8(6)(c), the convening authority will decide the appropriate course of action, which may include the issuance of an NOS and Enhanced Monitoring. If an NOS is issued the certificate holder is required to address the Conditions for the Termination of the Suspension in the NoS to avoid loss of certificate privileges. Any additional items identified in the assessment or PVI but not specified in the NoS will be addressed through the CAP, as described in section 10.0 – Corrective Action Plan.

      Note: 

      Where findings are considered major, the option offered in paragraph (b) is not available.

    6. Where any component is assessed at an award level of 1 a Notice of Suspension (NoS) shall be issued. At the discretion of the CA, the NoS may have an immediate effective date. If the NoS is issued, the certificate holder is required to address the Conditions for Reinstatement before the certificate is reinstated. Any remaining issues identified in the assessment or PVI that are not specified in the Conditions for Reinstatement that are not safety related will be addressed through the CAP, as described in section 10.0 – Corrective Action Plan.

  2. Where an immediate threat to safety exists, the convening authority shall issue the appropriate documents/restriction.
Return to Top of Page
Top of Page

8.4 Failure to achieve a Score of 4 or 5

  1. Where TCCA has determined that the certificate holder meets level 3 expectations but has minor or moderate findings against best practices, a Continuous Improvement Finding may be issued at the discretion of the CA.

  2. Where a Certificate Holder chooses to address the Continuous Improvement Finding through their CAP, the finding shall be attached to the Assessment Report as Appendix B.

  3. Once the certificate holder successfully closes the CAP to the satisfaction of the PI, a letter of acknowledgement shall be issued stating that the Certificate Holder is now considered to have exceeded the basic conformance level.

  4. A copy of the letter of acknowledgement issued in accordance with (3) shall be appended to the surveillance report. The original component conformance status recorded in the surveillance report shall not be amended.

  5. In order to issue a letter of acknowledgement in accordance with (3), all Continuous Improvement Findings described in (1) must be addressed simultaneously.

  6. Acceptance of continuous improvement findings is not mandatory. A company may designate prior to the surveillance activity or at any time during the activity itself that they do not wish to be notified of these findings. In that case, the assessment team will confine its activities to level 3 expectations and the assessment report will reflect findings against the CARs and not best practices.

8.5 Enhanced Monitoring

  1. Certificate Holders that fail to achieve a score of 3 on an assessment or a PVI may be subject to Enhanced Monitoring (EM) at the discretion of the convening authority. EM is intended to restore a certificate holder’s compliance with regulatory requirements in the shortest possible time.

  2. EM shall be conducted in accordance with the latest issue of SI SUR-002.
Return to Top of Page
Top of Page

9.0 ASSESSMENT REPORT

  1. The assessment report is a document that summarises the results of the assessment, includes findings and continuous improvement findings, and where applicable, corrective actions taken to findings issued during the assessment in accordance with subsection 7.8.8. The report is a factual account of the assessment and shall not include subjective statements, suggestions or recommendations.

  2. No component scores are assigned in the report. The assessment report shall state whether the certificate holder is in conformance or non-conformance with the CARs.

  3. The AM is responsible for the preparation of the report and its approval by the CA. As determined by the CA, the report shall be written in the certificate holder’s preferred language as indicated by the record of communication between TC and the certificate holder. The AM shall determine the language to be used in the preparation of assessment documentation based on the language of work requirements. If necessary, the AM shall have translations prepared.

  4. An assessment report template is included in Appendix A.

9.1 Report Procedures

  1. The assessment report shall be presented to the certificate holder within 30 calendar days calculated from the last day of the on-site portion of the assessment. At the discretion of the CA, reports that require additional time for committee review may be given up to five additional working days to complete. Any delay beyond the above maximums must be documented since the validity of the assessment will be compromised if the report is not presented in a timely manner.

  2. The CA shall sign the report cover letter and ensure that the original copy is received by the certificate holder. Confirmation of receipt, in the form of a signed copy, postal receipt or other acceptable means, is essential as this establishes the date for receipt of the CAP, if necessary. The report shall outline the procedure for responding to findings and specify the required response time of 30 calendar days from the time the certificate holder receives the report.

Return to Top of Page
Top of Page

9.2 Executive Summary

  1. This section of the report is intended to convey an overall view of the assessment results to senior management. It should be as short and concise as possible (1 page is suggested) and avoid the use of subjective statements that would suggest a level of compliance has, or has not, been met. The summary should contain a statement of conformance or non-conformance. Statements suggesting that the certificate holder is showing continuous improvement or the results of the current assessment show improvement over previous assessments are valuable to the certificate holder and would be acceptable. Any suggestion that the certificate holder is “10% better” or a “high degree of improvement was observed” should be avoided. It is best to remain factual and determine whether or not the expectations have been met.

  2. Certificate Holders that actively employ BPs should be recognized for this. The summary should indicate that certain BPs were observed.(where they have achieved a level 4 or 5).

  3. Avoid actually repeating details of findings as the findings speak for themselves.

  4. The executive summary shall include references to any associated decisions based on the surveillance activity, such as:

    1. the issuing of a Notice of Suspension;

    2. placing the certificate holder in Enhanced Monitoring; or

    3. forwarding the report for an enforcement review.

9.3 Objective of the Assessment

This is a short paragraph describing the type and nature of the assessment that was conducted.

  1. Example: “A routine assessment of ABC Airlines’ SMS, to determine conformity (compliance and effectiveness) of CAR 107.02”, or similar statement.
Return to Top of Page
Top of Page

9.4 Scope

The Scope shall include:

  1. A short paragraph indicating which components of the SMS model were reviewed and the applicable time period;

  2. Company Information; and

  3. Assessment Team.

9.5 Assessment Methodology

This portion of the report provides a brief summary of the process and the tools used during the assessment. Reference should be made to this SI regarding the assessment process, use of Expectations and Questions.

9.6 Component Score

The assessment report will indicate whether each component is in conformance or not, using the following or similarly formatted table:

Component Title Conformance
1 Safety Management Plan Y/N
2 Documentation Y/N
3 Safety Oversight Y/N
4 Training Y/N
5 Quality Assurance Y/N
6 Emergency Preparedness Y/N

For instructions on assigning a score, refer to subsection 8.2.

Return to Top of Page
Top of Page

9.7 Component Summaries

  1. This report section will provide a brief summary of each component. The intent is to indicate whether or not conformance was achieved, in accordance with the following guidelines:

    1. If TCCA identifies a non-conformance, the summary statement shall list the expectations that were not met.

    2. If TCCA identifies a conformance, the summary statement will simply indicate that the regulatory requirements were met.

    3. If TCCA identifies that the certificate holder has exceeded the basic conformance level, the summary statement will indicate the best practices that were met.

  2. When a finding is issued against an element (a non-conformance), a brief statement to that effect will be added at the end of the summary (e.g. one finding issued).

  3. The following examples are intended to offer the wording needed to justify component scores. Where an expectation is not met, the summary must clearly indicate this by rephrasing the expectation in a non-conformance manner and providing examples that confirm the non-conformance. Where best practices are observed, the summary must clearly indicate this by documenting the expectations related to best practices and providing examples to support the score.

    Example 1 - Using component 1 - Safety Management Plan, an acceptable summary statement for a level 2 non-conformance would read:

    “The regulatory requirements were not met as the following findings were observed in relation to the defined expectations:

    a.  A qualified person has not been appointed, in accordance with the regulations, and therefore has not demonstrated control of the SMS; and

    b.   Safety authorities, responsibilities and accountabilities have not been transmitted to all personnel, in particular, some employees were not aware of their SMS roles and responsibilities.”

    Example 2 - Where all the regulatory requirement have been met and a component is in conformance, the summary statement will read:

    “The regulatory requirements have been met”.

    Example 3 - Using component 3 – Safety Oversight, a summary statement for a component that exceeds the basic conformance level would read:

    The regulatory requirements were met and the following best practices were observed:

    a.   The organization has a process in place to ensure confidentiality when requested as four reports marked “Confidential” were reviewed and each submitter confirmed that a confidential report was requested; and

    b.  The feed back process provides an opportunity for report submitters to indicate whether they are satisfied with the response as a meeting is held with each submitter and their comments are added to the file and the submitter signs off on the final response.
Return to Top of Page
Top of Page

9.8 Findings

  1. This section of the report shall list the findings in order based on the SMS Element numbers (ref. SMS Model, Table A, Section 6.0). Findings resulting from an assessment or PVI are written against a regulatory requirement. For example, where the assessment determines that the non-punitive reporting program is not fully in compliance with the applicable requirement, the finding would be written against that requirement (e.g. Section 705.152 or Section 573.31 of the CARs). The Assessment Report will contain a maximum of two findings per element (maximum of 34), one for the documentation review and one for the on-site review.

  2. The Example section of the Finding Form must contain sufficient narrative information to indicate the extent of the non-conformance. As an example, proactive processes may be used in some areas of the organization but not in others. It shall be clearly indicated which areas of the organization are not using proactive processes. This information is necessary for the organization to formulate an effective CAP.

9.9 Interpretation of Findings

The report shall include a table interpreting the findings in accordance with the definition detailed in section 7.8.8(6).

Return to Top of Page
Top of Page

10.0 CORRECTIVE ACTION PLAN (CAP)

10.1 General

  1. Findings identify a situation where a company policy, procedure, or process does not conform to the applicable regulatory requirement. The certificate holder shall be required to submit a Corrective Action Plan (CAP) to TCCA within 30 calendar days following the issuance of the surveillance report.

  2. Where the CAP is acceptable, the certificate holder shall be advised and the appropriate information (administrative/on-site follow-up, proposed completion date) shall be entered on the Corrective Action Form, for the purpose of follow-up. Functional area databases can also be used to track the progress of assessment follow up.

  3. Where the certificate holder fails to submit a CAP within the required 30 calendar days, an NOS shall be issued with an effective date 30 calendar days from the date of its issuance.

  4. The principal inspector responsible for the certificate shall have 10 working days to accept or reject the proposed CAP. If rejected, the principal inspector shall provide the certificate holder with a description of the CAP deficiencies.

  5. If a certificate holder submits a CAP that is identified by TCCA as unacceptable, the CA shall, at his or her discretion, take one of the following actions:

    1. where the CAP deficiencies are significant in nature and could not be easily rectified without making major changes to the CAP, issue an NOS with an effective date 30 calendar days from the date of its issuance; or

    2. series of simple changes, return the CAP to the certificate holder for amendment. The certificate holder shall be required to return the amendment to TCCA in 5 working days.

  6. The principal inspector responsible for the certificate shall have 5 working days to accept or reject a CAP amended as described in paragraph (5)(b). If rejected again, the principal inspector shall issue an NOS with an effective date 30 calendar days from the date of its issuance.
Return to Top of Page
Top of Page

10.2 Short-Term and Long-Term Corrective Actions

  1. A short-term corrective action is intended to correct a non-conformance situation ensuring that conformance is established quickly until long-term action is completed to prevent recurrence of the problem.

  2. Short-term corrective action will be completed by the date/time specified in the corrective action section of the Finding Form, or per the approved CAP.

  3. TCCA’s surveillance activity cites examples to a degree necessary to demonstrate that a non-conformance exists. Further examples may exist that will require the company to conduct a more in-depth review to determine the full extent of the non-conformance.

  4. Long-Term Corrective Action has two components:

    1. the first will involve identifying the cause(s) of the non-conformance and indicating the measures or system changes that the company would be required to take to prevent a recurrence; and

    2. the second component will include a timetable for company implementation of the long-term corrective action.

  5. Proposed short and long term corrective action, including any supporting documents, will be forwarded to the appropriate individual(s) identified in the covering letter of the report, for review. Supporting documents may take the form of company documentation or revised procedures, or both, as applicable.

  6. TCCA will monitor implementation of the CAP through the follow-up process. The assessment will be formally closed when each finding has been corrected through the CAP or in a case where implementation of the CAP will exceed 12 months, the proposed CAP has been subjected to a risk assessment per subsection 11.3.

  7. In accordance with section 8.4, where a Certificate Holder has accepted a Continuous Improvement Finding, corrective actions shall be submitted as a separate section of the CAP.
Return to Top of Page
Top of Page

10.3 Corrective Action Plan Submission

  1. The covering letter of the surveillance report shall advise the certificate holder that it must:

    1. where applicable, submit corrective action forms for each finding requiring corrective actions by the date specified in the corrective action section of the Finding Form; and

    2. report. This deadline will not be extended without approval of the CA.

  2. The CA will include the name(s) of the person(s) to whom the CAP shall be sent in the report covering letter. This person will normally be the enterprise manager.

  3. CAPs received from the certificate holder shall include completed corrective action forms and where applicable, supporting documentation that may take the form of technical record entries, purchase orders, memoranda, revised procedure cards, manual amendments, etc.

10.4 CAP Evaluation

  1. Upon receiving a CAP developed by the certificate holder to address any non-conformances identified by TCCA following a surveillance activity, the principal inspector responsible for the acceptance of the CAP shall evaluate it in order to determine that it adequately addresses the non-conformances. The corrective action plan shall, at a minimum, include the following:

    1. A Factual Review of the Non-conformance
      The certificate holder shall have completed a review of the non-conformances and clearly identify what happened, how significant it was, where it occurred in the certificate holder’s system and what type of problem it was (e.g.: policy, process, procedure or culture). This shall correspond to the TCCA finding(s) raised. Factual review of the non-conformance should include:

      1. a detailed synopsis of the non-conformances, including a description of relevant factual information related to the item;

      2. identification of seriousness of the non-conformances; and

      3. identification of the process(es), procedure(s), practice(s) or culture(s) involved.

    2. A Root Cause Analysis of the Non-conformance
      The certificate holder’s analysis shall include a summary of the root cause as well as any causal factors that may have contributed to the non-conformance. Even though the certificate holder may use a causal analysis method that is not familiar to the CAP reviewer (e.g.: MEDA process, “5 Why’s”, etc.), the certificate holder must be able to demonstrate how they arrived at the root cause and it should be clear what caused the non-conformance to occur.

    3. Proposed Corrective Actions
      The proposed corrective actions shall contain detailed descriptions of planned or implemented short-term and long-term actions, including:

      1. a detailed description of actions that address the examples identified in the non-conformance finding(s) and all immediate safety issues;

      2. a detailed description of actions that address the causal factors determined during the analysis of the item(s);

      3. clear identification of the person or persons, within the enterprise, responsible for implementation of the actions; and

      4. any induced hazards or risks associated to the implementation of the corrective actions have been assessed, mitigated or eliminated.

    4. Implementation Timeline
      The CAP shall include timelines for the implementation of each proposed corrective action. The timeline should be aimed at implementing effective corrective actions in the shortest reasonable time period. It should be confirmed that there are due dates, targets and planned follow up to ensure effectiveness of the proposed corrective actions.

    5. Managerial Approval
      The corrective action plan shall be approved by an individual within the certificate holder’s management structure who has the authority to commit the necessary resources required to fulfill the plan.

  2. The role of the principal inspector responsible for the acceptance of the CAP shall be limited to the assessment of the process used by the certificate holder in reaching conclusions regarding the findings. It is not the inspector’s role to second-guess the CAP solutions, but rather to evaluate the process used in developing those solutions.
Return to Top of Page
Top of Page

11.0 ASSESSMENT FOLLOW-UP

11.1 General

  1. Upon completion of the assessment, the CA will transfer follow-up responsibilities to the principal inspector responsible for regulatory oversight of the certificate holder.

  2. The principal inspector referred to above shall ensure that:

    1. the CAP and assessment follow-up has been entered in the functional area database;

    2. the Finding Form have been completed by that date; and

    3. the CAP is submitted in the appropriate time period, and is approved, implemented and effective in rectifying the applicable non-conformances.

  3. Assessment follow-up is completed when:

    1. the principal inspector has accepted all finding corrective actions or for findings that will be completed beyond 12 months, a risk assessment has been completed and the results accepted;

    2. corrective action status has been recorded in the applicable company file;

    3. administrative or on-site follow up has been completed to confirm effectiveness of each corrective action; and

    4. the CA has been advised and a letter forwarded to the certificate holder advising them that the assessment is closed.

Return to Top of Page
Top of Page

11.2 Post Assessment Monitoring

  1. Where the findings are of a minor nature, no threat to aviation safety exists and the certificate holder has a proven effective quality assurance program, an administrative follow-up may be acceptable. In this case, the documents referred to in subsection 10.4 must be reviewed and found acceptable. All other findings require on-site follow-up to ensure that non-conformances have been rectified and that corrective actions are effective.

  2. Progress will be monitored as the certificate holder completes the applicable corrective actions. This will be accomplished by using the follow-up section on the Corrective Action Form (or functional area database). The form identifies the finding number, the type of follow-up (administrative or on-site) and the date upon which the corrective action was completed.

  3. Long-term corrective actions that have been accepted in accordance with subsection 10.4 will be followed-up by the applicable principal inspector who will advise the enterprise manager when the item is complete. This follow-up shall be confirmed through subsequent PVIs.

  4. Personnel assigned assessment follow-up responsibilities shall:

    1. monitor the certificate holder to ensure that the 30 day response time for CAP submission is observed or, where applicable, that corrective actions required by a specific date (indicated on the corrective action section of the Finding Form) have been completed;

    2. ensure that the CAP addresses the most safety critical findings as a priority;

    3. determine to the extent possible that each proposed corrective action will rectify the root cause of the finding to prevent its recurrence;

    4. determine that the certificate holder has developed a reasonable timetable for long-term corrective action and ensure that the proposed completion date is indicated on the appropriate section of the Corrective Action Form or entered in the applicable functional database;

    5. accept the CAP in co-ordination with the enterprise manager and where necessary, the CA;

    6. determine for each CAP item whether the follow-up is to be administrative or on-site and indicate so on the Corrective Action Form or applicable functional database;

    7. monitor the progress of the CAP by maintaining the follow-up section of the Corrective Action Form or applicable functional database and ensuring that the appropriate follow-up (administrative or on-site) has been conducted;

    8. ensure that all completed Corrective Action forms, together with any supporting documentation, are placed in the applicable company file; and

    9. advise the responsible manager when all corrective actions have been completed.
Return to Top of Page
Top of Page

11.3 Assessment Closure

  1. To enable Convening Authorities to close assessments within 12 months (or PVIs within 6 months) following CAP acceptance, the following process shall be applied. The CAP shall aim at having all corrective action in place within 90 calendar days of acceptance by the applicable principal inspector, unless the certificate holder is required to complete it sooner in accordance with section 8.3(1)(b). It is not always possible to meet these deadlines and special consideration may be required to ensure assessment closure in a timely fashion.

  2. Unless otherwise specified as per section 8.3(1)(b), findings will be actioned as follows:

    1. An immediate threat to safety; corrective action must be carried out immediately for the certificate holder to continue their activities. The finding shall be written into the report. Long-term corrective actions will be dealt with as identified in the following paragraphs (b), (c) or (d);

    2. Corrected within 90 days; normally the majority of findings should fall into this category. The accepted CAP must indicate that the long and short-term corrective action will be completed within 90 days. The applicable principal inspector or other assigned person will ensure follow-up;

    3. Corrected between 90 days and 12 months; in cases where it is anticipated that the corrective action will take more than 90 days to complete, a risk assessment shall be completed by the applicable principal inspector or other assigned person before accepting the CAP; and

    4. Longer than 12 months; in cases where it is not possible or reasonable to complete the corrective action within 12 months of acceptance of the CAP, a risk assessment study will be completed by the applicable principal inspector or other assigned person. If the risk assessment confirms that the proposed period of time is justified, an exemption will be issued. The corrective action would therefore be completed.

  3. The assessment shall be closed by the CA no later than 12 months after CAP acceptance (6 months for a PVI), since the corrective action has either been completed or assessed to the point whereby an extension will be issued in accordance with section 11.3.2.

  4. The enterprise manager will confirm that all follow-up actions have been completed, entered in the functional area database and that the CA is advised. The CA will ensure that a letter is forwarded to the certificate holder informing it that the assessment is closed.
Return to Top of Page
Top of Page

11.4 Assessment Report Review Committee

  1. Before the assessment report is issued, the assessment report review committee may be convened at the direction of the CA. The purpose of the assessment report review committee is as follows:

    1. to confirm the technical accuracy of the report with special attention given to the certificate holder’s description, the executive summary, and the assessment findings;

    2. to ensure that the report is an objective account of the assessment and that no subjective statements are included in the report;

    3. to ensure that statements made in the executive summary are supported by the findings; and

    4. to determine if any findings identified in the report should be subject to investigation by the Enforcement Division.

  2. The review committee may consist of the following people, based on current headquarters and regional organization and as applicable to the assessment:

    1. Convening Authority;

    2. Assessment Manager and Team Leader(s);

    3. Regional/HQ Director(s);

    4. Regional/Enterprise Manager(s)/HQ Chief(s);

    5. TCC/Regional/HQ Superintendent(s);

    6. Aviation Enforcement point-of-contact and Manager/Chief; and

    7. Principal Inspector(s).

  3. To facilitate an effective review it will be necessary to provide copies of the report to committee members in advance of the committee meeting, yet it is acknowledged that the meeting must be held shortly thereafter to provide time to make any changes, produce the final report and forward same to the certificate holder within the time periods specified in subsection 9.1.

  4. Recommendations resulting from the committee review will be considered advisory by the CA, as the CA will retain responsibility for the final report.
Return to Top of Page
Top of Page

12.0 ENFORCEMENT

  1. Once the assessment report has been sent to the certificate holder, the CA and the appropriate regional or HQ managers will jointly determine the necessity for any enforcement action. The managers may consult the applicable regional manager of Aviation Enforcement or the Chief of Aviation Enforcement regarding the application of enforcement procedures. A decision to proceed with an Enforcement investigation shall respect the policy stated and referenced in Civil Aviation Directive (CAD) 107-004 — Aviation Enforcement – Safety Management Systems. A decision record will highlight those assessment findings that are to be investigated by Aviation Enforcement. This record will form part of the company file.

  2. The co-ordination outlined above may take place as a discrete activity or, alternatively, as a function of an assessment review committee.

Return to Top of Page
Top of Page

13.0 PROGRAM VALIDATION INSPECTION

13.1 General

  1. A Program Validation Inspection (PVI) is intended to provide a review of sufficient depth to determine the level of compliance and effectiveness of a component. The use of a PVI will provide sufficient assurance that the certificate holder has employed effective policies, processes and procedures to meet regulatory requirements.

  2. A PVI differs from an assessment in that it does not look at the entire SMS. It is used to determine that all the requirements of a particular component of the SMS model or other parts of the regulations are documented, implemented, in use and effective. PVI will be used as the routine surveillance method in place of traditional inspections (the term Mandatory Inspections has been used).

13.2 Quality Assurance Program (QAP)

  1. Regulations requiring all Certificate Holders to have a quality assurance program (QAP) have been published or are in the process of being published and will be implemented in a phased manner. Program validation inspections performed against certificates issued pursuant to Parts III, IV, VII and VIII of the CARs who have not fully implemented a QAP, and against delegates authorized to perform certain duties in accordance with Airworthiness Manual 505, will use a customized PVI worksheet. Appendix A contains an expanded list of all PVI worksheets organized by operating rule. The following list identifies the specific areas that a PVI will review for a certificate holder that has not implemented a QAP:

    1. A PVI performed on a Part III certificate holder will review airport operator’s obligations.

    2. A PVI performed on a Part IV certificate holder will review chief flight instructor responsibilities and operational control.

    3. Where a PVI is performed on a delegate authorized under Chapter 505 of the Airworthiness Manual, that PVI will review the delegate’s obligations.

    4. A PVI performed on a Part VII, Subpart 3 or 4 certificate holder will review company operational control.

    5. A PVI performed on a Part VII, Subpart 5 certificate holder will review the Safety Oversight component.

    6. A PVI performed on a Part VIII certificate holder will review operating certificate requirements.

  2. Certificate holders that have implemented a quality assurance program will use the worksheet developed for QA. This will include organizations issued a certificate pursuant to Subpart 561 of the CARs, STD 566 of the CARs and Subpart 573 of the CARs as well as the maintenance requirements for all Part IV and Part VII Certificate Holders. The information referenced in Assessment/Validation Inspection Worksheet, Appendix A, Table 5.1 will be used as the basis for the PVI.
Return to Top of Page
Top of Page

14.0 PVI PROCEDURES

14.1 Notification

  1. A certificate holder will be contacted in writing six weeks prior to the planned PVI date to confirm the schedule. The notification period for PVIs planned on shorter notice will be at the discretion of the CA. This will allow sufficient time for the certificate holder to supply requested documentation and for TCCA personnel to conduct a complete review and follow up of the documentation prior to the on-site review.

  2. The method of notification shall advise the certificate holder to provide copies of documentation applicable to the PVI no later than 14 calendar days following the issuance of the notification letter (4 weeks prior to the planned on-site review). This may include approved company manuals, information incorporated by reference and any other documents essential to the assessment that provide information relating to the component or element under review. Refer to subsection 14.5 for additional information regarding applicable documentation.

  3. Section 14.4 provides additional instructions regarding certificate holders who fail to provide acceptable documentation within the timeline prescribed in (2).

  4. Where TC holds current copies of the information referenced in (2), the request for documentation shall specify which documentation is needed.

  5. The letter of notification shall request any Occupational Safety and Health (OSH) information that may be applicable to TCCA personnel during their on-site activities.

  6. The letter of notification shall request confirmation from the certificate holder whether or not they want best practices to be identified during the PVI. Such a letter shall also contain a brief description of what constitutes a best practice.

  7. The letter of notification may also include a copy of the documentation review guide to be completed by the certificate holder prior to the documentation review, along with the due date for its return to TCCA.

14.2 Team Selection

Team selection including team member terms of reference, qualifications and responsibilities are specified in Section 18.0.

14.3 PVI Plan

Refer to subsection 7.4 for information to establish a PVI plan. At the discretion of the CA, all the elements of subsection 7.4 may not be required depending on the scope and complexity of the PVI.

Return to Top of Page
Top of Page

14.4 Preliminary Documentation Review

  1. Prior to the start of the documentation review, the PVI Manager (PVIM) shall perform a preliminary review of the documentation submitted to verify its completeness.

  2. Where the preliminary review identifies no deficiencies with the submitted documentation, the PVI team shall proceed with the documentation review in accordance with section 14.5.

  3. Where the preliminary review identifies minor or moderate deficiencies, the PVIM shall contact the certificate holder in writing to request that the missing or incomplete documentation be provided to TCCA within 7 calendar days. Failure to meet this deadline shall result in certificate action.

  4. Certificate holder that fails to submit acceptable documentation in response to TCCA’s second request for documentation shall be issued an NOS. The NOS shall come into effect 7 calendar days following its issuance if the certificate holder does not satisfy the Conditions for Reinstatement by submitting the requested documentation. For a PVI, the NOS shall be based on failure to meet the conditions of issue for the certificate in question.

  5. Certificate holders that fail to provide any requested documentation in accordance with the timeline described in the notification letter or certificate holders who have major deficiencies in their document submission shall immediately be issued an NOS that will come into effect 7 calendar days following its issuance if the certificate holder does not satisfy the Conditions for Reinstatement by submitting the requested documentation. For a PVI, the NOS shall be based on failure to meet the conditions of issue for the certificate in question.

Return to Top of Page
Top of Page

14.5 Documentation Review

  1. The documentation review shall take place 2 weeks in advance of the on-site review in order to ensure that:

    1. inspectors have the necessary time to conduct a complete review of the documentation;

    2. the certificate holder has sufficient time to supply additional requested documentation;

    3. inspectors have the necessary time to conduct a follow-up on any documentation questions prior to the on-site review; and

    4. PVI managers are able to make a timely decision whether to proceed with the on-site portion of the assessment or not.

  2. This activity includes a thorough review of all files and documentation that are relevant to the certificate holder and the scope of the PVI. The following shall be completed during this portion of the PVI where applicable:

    1. ensure that all reference manuals and documents to be used during the PVI are readily available and include the latest approved amendments. A request to the certificate holder shall be made to confirm the revision status of any manual subject to review;

    2. review the certificate holder’s approved manuals for compliance to the appropriate standard;

      Note:
      Documentation requested from the certificate holder will be limited to those publications that provide process, policy and procedures only. Company generated records, such as reports or other data, pertaining to the outcome of individual programs will be reviewed on-site as part of the sampling plan.

    3. review other related information relevant to the certificate holder and the scope of the assessment to include:

      1. previous assessments, validation inspections, process inspections or audits including corrective actions and follow-up where applicable;

      2. accident or incident data, including CADORS reports;

      3. previous compliance history including any enforcement actions; and

      4. exemptions, waivers, approvals, limitations and authorizations.

    4. identify areas that require further review during the on-site portion of the PVI; and

    5. select the applicable PVI Worksheet from Appendix A.

  3. The results of the document review shall be documented using the applicable PVI worksheet listed in Appendix A.

  4. Where, as part of the documentation review, PVI team members choose to use process mapping in order to assess the certificate holder’s processes and procedures, they shall follow the procedures detailed in Section 16.0 – Process Inspection.
Return to Top of Page
Top of Page

14.6 Team Meeting

Refer to subsection 7.7 for information on the team meeting.

14.7 Sampling Procedures

Refer to subsection 7.8 for information on sampling procedures.

14.8 On-Site Procedures

14.8.1 Entry Meeting

Refer to subsection 7.8.2 for information on the entry meeting.

14.8.2 PVI Team On-Site Briefing

Refer to subsection 7.8.3 for information on the on-site team briefing.

14.8.3 On-site Review

  1. During this phase, the PVI team shall:

    1. conduct interviews and sampling per the sampling plan;

    2. gather evidence to support observations;

    3. analyse observations collected from both the documentation and on-site reviews;

    4. determine if:

      1. the certificate holder is in compliance with regulatory requirements; and

      2. the SMS is effective (if it is an SMS organization);

    5. prepare findings of non-conformance as applicable.

  2. The results of the on-site review shall be documented using the applicable PVI worksheet listed in Appendix A.
14.8.4 Interviewing

Refer to subsection 7.8.5 for information on interviewing.

14.8.5 Analysis

Refer to subsection 7.8.6 for information on the analysis.

14.8.6 PVI Worksheets

PVI Worksheets have been developed to assist inspectors in the conduct of program validation inspections. A complete list of PVI worksheets is included in Appendix A. Refer to sub paragraph 7.8.7 for additional information on the use of worksheets.

14.8.7 Findings

Refer to subsection 7.8.8 for information on preparing findings.

Return to Top of Page
Top of Page
14.8.8 Exit Meeting

  1. Upon completion of the PVI, the PVIM shall convene an exit meeting with the certificate holder’s senior management to brief them on the results. The CA or his representative from the PVI management team shall attend. When present, the CA may chair the meeting or simply attend with the team.

  2. Where applicable, the PVI team will have briefed the certificate holder’s management on potential findings and areas of concern during the daily meetings so debate between the team and the certificate holder should not occur. The meeting chair shall advise the certificate holder that they will have an opportunity to respond formally to the PVI report in their CAP submission.

  3. The PVIM shall advise the certificate holder that the report will be forwarded to them within the time period referred to in subsection 14.10 and that a CAP must be submitted to TCCA within 30 calendar days after the report has been issued. Details of the corrective action shall also be discussed.

14.9 PVI Scoring

  1. Refer to section 8.0 for information on scoring a PVI.

14.10 PVI Report

  1. A PVI Report can follow a similar format as the Assessment Report described in section 9.0 or it can be condensed to a letter format depending on the size and complexity of the organization and results of the validation inspection. If a letter format is chosen, it shall clearly state the results of the validation inspection similar to an executive summary and include any other applicable information such as conformance or non-conformance statements, findings, continuous improvement findings, corrective action and Enforcement information.

  2. As indicated in subsection 9.2 , the use of subjective language shall be avoided. Only statements that clearly convey the outcome of the PVI shall be used.

  3. The PVI report shall be presented to the certificate holder within 30 calendar days calculated from the last day of the on-site portion of the PVI.

14.11 PVI CAP Procedures

Refer to section 10.0 for information on corrective action plans.

14.12 PVI Follow-Up and Closure

Refer to Section 11.0 for follow up and closure procedures.

Return to Top of Page
Top of Page

15.0 PROCESS INSPECTION

15.1 General

  1. The purpose of a Process Inspection is to determine whether a certificate holder’s processes meet regulatory requirements and that they are functioning as intended. Process inspections are intended to provide information for decisions about the level of risk associated with a certificate holder and whether additional surveillance is required.

  2. A process inspection is a surveillance method not normally requiring the appointment of a convening authority, official notification, opening or closing meetings or reports to the certificate holder. The certificate holder subject to the process inspection shall be briefed on the results and may be issued documentation in the form of a letter, detection notice or finding, which will also be forwarded to the enterprise manager where it shall be used to determine if further surveillance, in the form of a PVI or assessment is necessary.

  3. A process inspection may generate findings. Where a process inspection reveals safety related issues, a PVI may be conducted on an urgent basis to collect sufficient material to support the issuance of a formal report and an NOS, as necessary.

  4. Process inspections can be scheduled as a routine activity, supplementing scheduled assessments or PVIs or they can be conducted where risk indicators show that certain processes may not be performing as expected.

16.0 PROCESS INSPECTION TEMPLATE

  1. Process inspection templates have been developed and are listed in Appendix A, Safety Profile, Performance Measurement, Emergency Preparedness and Response, Investigation and Analysis and Training. A generic Process Inspection Template is also included in Appendix A to allow inspectors a means to develop a process inspection for other processes.

  2. The templates are designed to illustrate the key process steps. Each step includes the process inputs and outputs and a series of questions. The inputs and outputs for each step are identified during the documentation review and the questions are answered during the on-site review.

  3. The template provides space to include a process map. The map is simply a method to show the key process steps in a pictorial format. This format will aid in the process inspection ensuring that all key steps are identified and reviewed.

  4. The following diagram illustrates the process map for an internal audit process as required by the Quality Assurance component. The remaining processes are structured in a similar manner as indicated in each process template.


    Step 1

    Plan audit
    (specify area)


    → 
    Step 2

    Documentation
    review and prep



    → 
    Step 3

    Conduct audit of specific area


    → 
    Step 4

    Complete audit report


    Note:

    This diagram is an example only. It may not represent the number of steps in every process. Adjust the diagram to reflect the number of steps in the selected process.
Return to Top of Page
Top of Page

17.0 PROCESS INSPECTION PROCEDURES

17.1 Process Selection

Select the process that will be reviewed from Appendix A or the completed Process Inspection Template.

17.2 Documentation Review

This includes a thorough review of all files and documentation that are relevant to the selected process. The following shall be completed during this activity:

  1. Ensure that all reference manuals and documents to be used during the process inspection are readily available and include the latest approved process related information.

  2. Review the process information in the certificate holder’s approved manuals for conformance to the appropriate regulation or standard.

  3. Review the sources of external information relevant to the selected process to include:

    1. Previous assessments, program validation inspections, process inspections or audits including corrective actions and follow-up where applicable;

    2. Accident or incident data, including CADORs;

    3. Previous enforcement action; and

    4. Exemptions, waivers, approvals, limitations and authorizations.

17.3 Process Analysis

  1. Perform an evaluation of company documentation using the applicable process inspection template in Appendix A.

  2. Use the first process step in the flowchart. Go to the approved documentation and identify the location by paragraph number to where the particular process step is located. Make notes of:

    1. all action statements (verb and its noun i.e., collect data);

    2. action statements not addressed (omissions), so they can be researched on-site;

    3. the process and associated procedures if they are vague, if there are gaps in the documentation, or if details to address an action statement cannot be identified then a more detailed on-site examination is required;

    4. specific records, forms or reports that are used along with the identification of the person or place where the records should be available;

  3. In relation to the process step being reviewed, answer the questions (who, what, where, when, why and how) as provided in the process inspection template.

  4. Repeat (2) for each process step.
Return to Top of Page
Top of Page

17.4 On-site Review

  1. Tracing is the method commonly used to follow a process from step to step. It begins with an output (report, meeting minutes, license etc.) and follows it backwards through the process, essentially using a “reverse engineering” approach. The purpose is to connect each output with the applicable input(s) that created it. Using the internal audit process map as an example, any audit report can be traced to the audit that created it, to the doc review and prep and finally to its place in the audit plan. Tracing backwards enables seeing how the process is actually done; conclusions will be representative of reality.

  2. To begin tracing refer to the completed process inspection template: it is the road map for tracing. Select record(s) and follow its progression through the chosen process.

  3. Select a process step. Focus on validating the answers to the questions answered during the process analysis. Be sure to document what is seen, the person from whom the information is gathered, your location, when the action is done, why the action is accomplished and how it is accomplished.

  4. Repeat Tracing steps (2) and (3) until the entire process has been reviewed and the path the selected record(s) traveled through the entire process has been traced.

  5. It is critical to see if all outputs stated are being produced, and are being used as inputs in the following process step. If an output is not being produced or not being used as an input for the next step in the process, then document the specific example.

17.5 Outcomes

The result of the process inspection shall indicate one of the following:

  1. the process meets applicable regulatory requirements and is being followed as published. No further action required.

  2. the process does not meet applicable regulatory requirements for one or more of the following reasons:

    1. not documented;

    2. not implemented; or

    3. not effective.

17.6 Developing A Process Inspection Using The Generic Template

Any process can be inspected using the generic process inspection template referenced in Appendix A. A process flow chart can be developed using the process inspection steps identified in section 17.2 to 17.5.

Return to Top of Page
Top of Page

18.0 TEAM SELECTION AND RESPONSIBILITIES

Note:

Although the information in this section is written to address an assessment, it shall also apply to a PVI.

  1. The assessment team, approved by the CA, will vary according to the size and complexity of the certificate holder under assessment. Assessments of smaller certificate holders may consist of only one inspector where that inspector is responsible for all assessment duties. Assessments of larger certificate holders will normally use a team approach and have an AM, two or more team leaders and team members. Administrative support, specialists and observers can be added at the discretion of the CA.

  2. Non-delegated team members can be used for administrative support during a surveillance activity but shall not be used where a delegation is required to perform the functions (for example, performing duties as per section 8.7(1) of the Aeronautics Act).

  3. It is recognised that an assessment team may not require all the positions listed below and that various duties and responsibilities may be combined or deleted when assumed by a particular team member.

  4. This section outlines the terms of reference, qualifications and responsibilities of the CA, AM, team leader(s) and team member(s).

18.1 Convening Authority

  1. The Convening Authority shall:

    1. determine the objective and scope of the assessment;

    2. appoint an AM for each assessment. A sample letter of appointment can be found in the appropriate functional area control manual (electronic means, including e-mail, is acceptable for this purpose);

      Note: 

      The AM should be appointed from four to six months prior to the planned assessment. This will allow sufficient time for research, familiarization with the terms of reference, selection of the assessment team, budget planning and the development of an assessment plan.

    3. oversee the selection of the assessment team;

    4. approve the assessment plan;

    5. attend the entry meeting, when required;

      Note: 

      The CA will usually not attend the entry meeting, as this would detract from the AM’s authority. The CA may wish to attend the meeting however, where the assessment is convened under extraordinary circumstances.

    6. attend the exit meeting, when practicable;

    7. review and approve the assessment report, sign the report cover letter and ensure that the certificate holder receives the report within the required time-frame;

    8. ensure that action is taken in an appropriate and timely manner for any immediate threat to safety identified by the AM during the on-site assessment;

    9. ensure that appropriate follow-up action is completed after the on-site assessment; and

    10. send a letter to the certificate holder confirming that all assessment findings and corrective actions are complete and that the assessment has been closed.
Return to Top of Page
Top of Page

18.2 Assessment Manager

  1. The AM’s terms of reference will be outlined in the appointment letter, memo or other acceptable method and specify that the AM shall:

    1. report directly to the CA until released from his/her assessment duties;

    2. conduct all assessment-related matters in accordance with the procedures specified in this SI and other associated guidance material;

    3. immediately contact the CA with a recommendation for action in the event of an immediate threat to aviation safety. Delegation of Authority must be exercised in situations where aviation safety is, or may be, compromised;

    4. be authorized to communicate directly with HQ directors and Regional/Enterprise Managers to obtain the required personnel resources. This may be sub-delegated to Team Leaders where applicable; and

    5. where required, be assigned a responsibility centre number with the appropriate funding for travel, overtime and any related expenses to be incurred during the assessment.

  2. Qualifications - the AM shall:

    1. have completed all the Civil Aviation Directive (CAD) 7 training requirements;

    2. have experience related to the type of certificate holder to be assessed;

    3. possess a sound knowledge of aeronautical legislation and regulations;

    4. have demonstrated communication and management skills;

    5. have experience with TCCA administrative procedures; and

    6. possess an acceptable level of surveillance experience.

      Note: 

      Until TC personnel acquire experience in conducting assessments and PVIs, the level of surveillance experience necessary to act as an AM will be at the discretion of the CA. It is recommended that assigned managers possess equivalent experience as an audit manager or have acquired an acceptable level of experience in conducting audits or inspections as a team leader or team member.

  3. Responsibilities - the AM shall:

    1. plan, organize, direct and control the assessment process;

    2. begin planning activities well in advance of the scheduled assessment date;

    3. where applicable, select team leaders in consultation with the CA and confirm their assignment by letter, memo or other acceptable method. Team Leaders may be used on assessments depending on the scope and complexity of the assessment and the size of the team;

    4. maintain assessment documentation including, but not limited to, the AM’s letter of appointment and terms of reference, all working notes, copies of assessment-related documents and a copy of the assessment report;

    5. develop an assessment plan for approval by the CA;

    6. notify the certificate holder by letter of the planned assessment ( refer to subparagraphs 7.2 and 14.1). A sample letter can be found in the appropriate functional area control manual;

    7. co-ordinate personnel requirements for the assessment team with the appropriate Regional directors and HQ managers, as applicable;

    8. ensure that the documentation review is complete;

    9. ensure that team members are knowledgeable in their assigned areas;

    10. co-ordinate with the appropriate TCCA specialty areas to ensure that all non-assessment departmental liaison and activities with the certificate holder are minimized and/or co-ordinated through the AM during the assessment period;

    11. convene a pre on-site team meeting where applicable;

    12. advise the appropriate Regional Manager/Chief, Aviation Enforcement of the planned assessment and request a contact point should an enforcement inspector be required;

    13. establish contact with the CA and regional assessment Offices of Primary Interest (OPIs) to relay fieldwork progress, potential problems, changes in the objectives or scope of the assessment, and other significant matters;

    14. co-ordinate and chair the entry meeting with the certificate holder and maintain communications with the organization’s senior management;

    15. advise the CA immediately of any immediate threat to safety and ensure that the CA is aware of any safety issues identified during the on-site assessment. Delegation of Authority must be exercised in situations where aviation safety is, or may be, compromised;

    16. ensure that any decisions to be made by, or approvals required from, the CA during the assessment are received in a timely manner;

    17. exercise line authority over assessment team members and observers;

    18. ensure that all assessment findings are tied to the applicable regulatory requirement and are supported by specific examples and evidence or other supporting documentation where applicable;

    19. ensure that all draft findings have been discussed with the certificate holder prior to the exit meeting, where it is possible to do so;

    20. co-ordinate and chair the exit meeting with the certificate holder’s senior management;

    21. prepare the covering letter and assessment report for approval by the CA;

    22. provide the CA with recommendations for possible enforcement action arising from the assessment and co-ordinate subsequent action regarding assessment findings with the assigned enforcement inspector;

    23. provide the CA with recommendations for possible enhanced monitoring;

    24. ensure that team members have fulfilled all responsibilities prior to release from assessment duties and confirm their release by letter, memo or other acceptable method if released other than as planned; and

    25. ensure that parallel observations and findings have been completed and distributed in accordance with Chapter 5 of the IAM.

Return to Top of Page
Top of Page

18.3 Team Leader

  1. The team leader’s terms of reference will be outlined in the appointment letter, memo or other acceptable method and specify that the team leader:

    1. will report directly to the AM until released from his/her assessment duties;

    2. will conduct all assessment-related matters in accordance with the procedures specified in this SI and other associated guidance material;

    3. will immediately contact the AM with a recommendation for action in the event of an immediate threat to aviation safety. Delegation of Authority must be exercised in situations where aviation safety is, or may be, compromised; and

    4. where applicable, is authorized to communicate directly with HQ directors and Regional managers to obtain the required personnel resources.

  2. Qualifications - the team leader shall:

    1. have completed all the CAD 7 training requirements;

    2. have experience related to the type of certificate holder to be assessed;

    3. possess a sound knowledge of aeronautical legislation and regulations;

    4. have demonstrated skills in communication and management;

    5. have experience with TCCA administrative procedures; and

    6. possess an acceptable level of surveillance experience.

      Note: 

      Until TC personnel acquire experience in conducting assessments and PVIs, the level of surveillance experience necessary to act as a team leader will be at the discretion of the CA in consultation with the AM. It is recommended that assigned team leaders possess equivalent experience as an audit team leader or have acquired an acceptable level of experience in conducting audits or inspections as a team member.

  3. Responsibilities - the team leader shall:

    1. become familiar with the assessment terms of reference and support and assist the AM;

    2. select the appropriate team members and confirm their assignment by letter, memo or other acceptable method;

    3. direct and control his or her team’s activities;

    4. submit the team budget and activity schedule to the AM for approval;

    5. keep the AM advised of deviations from the above budget;

    6. keep the AM informed of the assessment progress in his or her functional area;

    7. ensure that all assessment findings are tied to the applicable regulatory requirement and are supported by specific examples and evidence or other supporting documentation where applicable;

    8. review and verify specific sections of the assessment report as required by the AM;

    9. ensure that parallel findings/observations are documented on the appropriate forms and forwarded to the AM;

    10. brief the organization’s management on his or her functional area during daily briefings and at the exit meeting; and

    11. prepare an executive summary of the most significant assessment findings. This will form the basis for the team leader’s remarks at the exit meeting and will be included in the Executive Summary of the assessment report.

Return to Top of Page
Top of Page

18.4 Team Member

  1. The team member’s terms of reference will be outlined in the appointment letter, memo or other acceptable method and specify that the team member will:

    1. report directly to the AM, through the team leader where applicable, until released from his/her assessment duties;

    2. conduct all assessment-related matters in accordance with policy and procedures specified in this SI and other associated guidance material; and

    3. immediately contact the AM, or where applicable, the team leader, with a recommendation for action in the event of an immediate threat to aviation safety. Delegation of Authority must be exercised in situations where aviation safety is, or may be, compromised.

  2. Qualification - a team members shall:

    1. have completed all the CAD 7 training requirements, as required for their individual delegation;

    2. have experience related to the type of certificate holder to be assessed; and

    3. possess a sound knowledge of aeronautical legislation and regulations.

  3. Responsibilities - team members shall:

    1. report directly to the assessment manager for the duration of the surveillance activity, until released;

    2. become familiar with the assessment terms of reference;

    3. review conflict of interest guidelines with respect to his or her assessment responsibilities;

    4. become familiar with the certificate holder’s policies and procedures;

    5. conduct assessment fieldwork and document assessment findings;

    6. document parallel findings/observations as they are encountered and forward these to the AM through the team leader, where applicable;

    7. communicate with the team leader to ensure that assessment progress is reported and potential problems are addressed;

    8. review the validity and applicability of assessment findings by ensuring that all findings are tied to the applicable regulatory requirement and are supported by specific examples and evidence or other supporting documentation where applicable; and

    9. provide the team leader with the applicable component or element summaries where requested to do so in the assessment plan.
Return to Top of Page
Top of Page

19.0 INFORMATION MANAGEMENT

Inspectors shall place records of the results of their activities on the applicable company files, except aircraft specific records, which shall be placed on the appropriate aircraft file. Inspectors are reminded that notes and other hand written documents created during the surveillance process are official records and are to be preserved in the official files by scanning or paper profiling according to regional and divisional policies.

20.0 CONTACT OFFICE

For more information, please contact the:
Chief, Program Evaluation & Coordination (AARTT)

Phone:
613-952-7974
Fax:
613-952-3298
E-mail:
CAIRS_NCR@tc.gc.ca

Suggestions for amendment to this document are invited, and should be submitted via the Transport Canada Civil Aviation Issues Reporting System (CAIRS) at the following Internet address:

http://www.tc.gc.ca/CAIRS

[Original signed by Don Sherritt]

Don Sherritt
Director, Standards
Civil Aviation
Transport Canada

Return to Top of Page
Top of Page

APPENDIX A— ASSESSMENT/VALIDATION INSPECTION WORKSHEETS

  1. TABLE 1.1 – Safety Policy (RDIMS# 3713435)

  2. TABLE 1.2 – Non-Punitive Safety Reporting Policy (RDIMS# 3713452)

  3. TABLE 1.3 – Roles, Responsibilities & Employee Involvement (RDIMS# 3713475)

  4. TABLE 1.4 – Communication (RDIMS# 3713501)

  5. TABLE 1.5 – Safety Planning, Objectives & Goals (RDIMS# 3713578)

  6. TABLE 1.6 – Performance Measurement (RDIMS# 3713614)

  7. TABLE 1.7 – Management Review (RDIMS# 3713722)

  8. TABLE 2.1 – Identification & Maintenance of Applicable Regulations (RDIMS# 3713760)

  9. TABLE 2.2 – SMS Documentation (RDIMS# 3713864)

  10. TABLE 2.3 – Records Management (RDIMS# 3713893)

  11. TABLE 3.1 – Reactive Processes (RDIMS# 3714073)

  12. TABLE 3.2 – Proactive Processes (RDIMS# 3714165)

  13. TABLE 3.3 – Investigation & Analysis (RDIMS# 3714261)

  14. TABLE 3.4 – Risk Management (RDIMS# 3714332)

  15. TABLE 4.1 – Training, Awareness & Competence (RDIMS# 3714422)

  16. TABLE 5.1 – Quality Assurance (RDIMS# 3714511)

  17. TABLE 6.1 – Emergency Preparedness & Response (EPR) (RDIMS# 3714543)

  18. Program Validation Inspection Worksheet – CAR 302 (RDIMS# 4422273)

  19. Program Validation Inspection Worksheet – CAR 406 (RDIMS# 5417816)

  20. Program Validation Inspection Worksheet – CAR 703/704 (RDIMS# 4757558)

  21. Program Validation Inspection Worksheet – CAR 801 (RDIMS# 4422373)

  22. On-site Review Plan Template (RDIMS# 4954087)

  23. Assessment Report Template (RDIMS# 4861613)

  24. Letter of acknowledgement (RDIMS#)

  25. Process Inspection Template, Safety Profile (RDIMS# 4676175)

  26. Process Inspection Template, Performance Measurement (RDIMS# 4675951)

aa.  Process Inspection Template, Emergency Preparedness & Response (RDIMS# 4675907)

bb.  Process Inspection Template, Investigation and Analysis (RDIMS# 4675876)

cc.  Process Inspection Template, Training (RDIMS# 4675842)

dd.  Process Inspection Template, Generic (RDIMS# 5003211)

ee.  Program Validation Inspection Worksheet – EASA Supplement (RDIMS# 5290193)

ff.  Program Validation Inspection Worksheet – AWM 505 (RDIMS# 4876771)

Return to Top of Page
Top of Page

APPENDIX B— EXPECTATIONS AND SCORING CRITERIA

Note:

Certificated personnel are employees performing functions required by the Canadian Aviation Regulations. Non-certificated personnel are employees of the company working in non-regulated positions such as human resources. Third parties are external personnel performing duties for the certificate holder under contract for example catering, baggage handling and snow removal.

  Table D1.1 – Safety Management Plan – Safety Policy
Score Criteria
Not in Conformance
1 Senior Management does not demonstrate commitment to an SMS. Safety policies are not well developed and most personnel are not involved in SMS.
2 (3) less some aspects
In Conformance
3

  1. A safety policy is in existence, followed and understood.

  2. The organization has based its safety management system on the safety policy and there is a clear commitment to safety.

  3. The safety policy is agreed to and approved by the accountable executive.

  4. The safety policy is promoted by the accountable executive.

  5. The safety policy is reviewed periodically for continuing applicability.

  6. The safety policy is communicated to all employees with the result that they are made aware of their safety obligations.

  7. The policy is implemented at all levels of the organization.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • The Safety policy clearly describes the organization’s safety objectives, management principles and commitment to continuous improvement in the safety level.

  • The policy is included in key documentation and communication media.

  • All levels of management clearly articulate the importance of safety when addressing company personnel.

  • Management has a clear commitment to safety and demonstrates it through active and visible participation in the safety management system.

  • Management makes the policy clearly visible to all personnel and particularly throughout the safety critical areas of the organization.

  • The safety policy contains a commitment by the organization’s management to the development and ongoing improvement of the safety management system.

  • Safety policy objectives drive the organization’s mission statements.

  • The organization’s goals are linked to the safety policy objectives.



Table D1.2 – Safety Management Plan – Non-Punitive Safety Reporting Policy
Score Criteria
Not in Conformance
1 Safety-related reports or inadvertent errors result in punitive action being taken against individuals.
2 (3) less some aspects
In Conformance
3

  1. There is a policy in place that provides immunity from disciplinary action for employees that report, hazards, incidents or accidents.

  2. Conditions under which punitive disciplinary action would be considered (e.g. illegal activity, negligence or wilful misconduct) are clearly defined and documented.

  3. The policy is widely understood within the organization.

  4. There is evidence that the organization is applying the policy.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • Personnel express confidence and trust in the policy.

  • The Policy is applied throughout certificated and non-certificated areas of the organization.

  • The organization has letters/Memoranda of Understanding (MOU) between employees and/or third party contractors and management. The purpose of these letters is to document the disciplinary policy, and the manner in which it will be implemented.

Return to Top of Page
Top of Page
  Table D1.3 – Safety Management Plan – Roles and Responsibilities
Score Criteria
Not in Conformance
1 No formal designation of authorities, responsibilities and accountabilities for the safety management system exists. A management representative has not been appointed to ensure the SMS is implemented. Safety mandates are not widely disseminated and personnel’s awareness of their role in the SMS is limited.
2 (3) less some aspects
In Conformance
3

  1. There are documented roles and responsibilities and accountabilities for the accountable executive and evidence that the SMS is established, maintained and adhered to.

  2. The accountable executive demonstrates control of the financial and human resources required for the proper execution of his/her SMS responsibilities.

  3. A qualified person has been appointed, in accordance with the regulation, and has demonstrated control of the SMS.

  4. The person managing the operation of the SMS fulfils the required job functions and responsibilities.

  5. Safety authorities, responsibilities and accountabilities are transmitted to all personnel.

  6. All personnel understand their authorities, responsibilities and accountabilities in regards to all safety management processes, decision and actions.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • Safety authorities, responsibilities and accountabilities are reviewed prior to any significant organizational change.

  • Safety authorities, responsibilities and accountabilities of all personnel and third parties are defined and documented in job descriptions.

  • There is clear evidence that the accountable executive not only understands that he or she has ultimate responsibility for safety within the organization, but that he or she demonstrates this commitment on a daily basis.

  • There are documented organizational diagrams, where applicable, and job descriptions for all personnel including non-certificated personnel.

  • Key safety activities are clearly described in senior management duties and responsibilities and incorporated into performance agreements.

  • There is evidence that senior management recognizes the significance of contributions from all levels of the organization and has a mechanism for acknowledging those contributions.

  • Employee involvement and consultation arrangements are documented.



Table D1.4 – Safety Management Plan – Communication
Score Criteria
Not in Conformance
1 The general exchange of information throughout the organization does not permit the system to function effectively. The organizational communication network does not include all personnel, out-stations and outsource functions.
2 (3) less some aspects
In Conformance
3

  1. There are communication processes in place within the organization that permit the safety management system to function effectively.

  2. Communication processes (written, meetings, electronic, etc.) are commensurate with the size and complexity of the organization.

  3. Information is established and maintained in a suitable medium.

  4. There is a process for the dissemination of safety information throughout the organization and a means of monitoring the effectiveness of this process.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • Communications related to safety and quality issues are two way and applied across certificated, non-certificated and third parties.

  • There exists a formal means of communicating with experts in SMS so that personnel can easily and quickly obtain advice. The documentation should indicate where these experts could be located.

  • All personnel are informed as to who is their primary contact for safety related matters.

  • There is a communication strategy that might include electronic or web-based communications, frequent meetings, SMS bulletins, communication frequencies, audience targeting and status updates.

  • There is a documented process to review the effectiveness of communications.

  • There is a process for sharing safety-related information with outside sources that might be impacted by this information.

Return to Top of Page
Top of Page
  Table D1.5 – Safety Management Plan – Safety Planning
Score Criteria
Not in Conformance
1 Safety objectives are poorly defined and/or not communicated. Resources are not allocated for achieving objectives.
2 (3) less some aspects
In Conformance
3

  1. Safety objectives have been established utilizing a safety risk profile that considers hazards and risks.

  2. Objectives and goals are consistent with the safety policy and their attainment is measurable.

  3. Safety objectives and goals are reviewed and updated periodically.

  4. There is a documented process to develop a set of safety goals to achieve overall safety objectives.

  5. Safety objectives and goals are documented and publicized.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • Safety objectives are based on a safety risk profile that includes all areas of the organization (certificated, non-certificated and third parties).

  • The organization has a process for analyzing and allocating resources for achieving their objectives and goals.

  • Safety objectives have been established utilizing a safety risk profile that considers:

    1. hazards and risks;

    2. financial, operational and business requirements;

    3. views of interested parties; and

    4. industry-wide safety risk profile.

  • Objectives and goals are documented and publicized throughout non-certificated parts of the organization and to third parties.


Table D1.6 – Safety Management Plan – Performance Measurement
Score Criteria
Not in Conformance
1 Safety performance measures have not been established.
2 (3) less some aspects
In Conformance
3

  1. There is a documented process to develop and maintain a set of performance parameters that are linked to the organization's goals and objectives.

  2. Procedures have been established and maintained to monitor and measure safety performance on a regular basis.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • Performance measurements have been defined for serious safety risks identified on the safety risk profile.

  • Performance measurements have been defined for each and every critical operational process and procedure such as Flight Data Monitoring Program (FDMP).

  • The analysis and allocation of resources are based on outputs from the performance measurement.

  • Personnel at all levels are aware of the safety performance measures in their areas of responsibility and the results of performance measures are transmitted to them.

Return to Top of Page
Top of Page
  Table D1.7 – Safety Management Plan – Management Review
Score Criteria
Not in Conformance
1 The process for conducting reviews of the safety management system is based on event response rather than on a periodic, scheduled basis.
2 (3) less some aspects
In Conformance
3

  1. There are periodic, planned reviews and reviews for cause of the company’s safety management system to ensure its continuing adequacy and effectiveness as well as a review of company safety performance and achievement.

  2. The safety management system review includes:

    • Internal audit results;

    • Activities to verify that employees understand the SMS and their role and responsibilities in it;

    • Safety objective achievement results;

    • Hazards and occurrence investigation and analysis results;

    • Internal/external feedback analysis and results;

    • Status of corrective and preventive action(s);

    • Follow-up actions from previous management reviews;

    • Changes that could affect the SMS;

    • Recommendations for improvement; and

    • Sharing of best practices across the organization.

  3. There is a documented procedure defining responsibilities and requirements for planning and conducting internal audits of:

    • management policies, controls and procedures concerning all safety critical activities; and

    • the implementation and maintenance of SMS requirements established by the organization.

  4. There is a process to evaluate the effectiveness of corrective actions resulting from management review.
4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • The organization has established a structured committee or board, appropriate for the size and complexity of the organization, consisting of a full range of senior management representatives including certificated, non-certificated and third parties that review the management review report.

  • The organization compares its SMS against other organizations and is an active proponent of SMS within the aviation industry.

  • The management review committee makes recommendations to the accountable executive related to:

    1. the improvement and effectiveness of the SMS;

    2. the initiation and implementation of safety policy and actions across the organization; and

    3. allocation of resources needed to achieve objectives.

  • Management review decisions are explained to employees to demonstrate how the review process leads to new objectives that will benefit the organization.
  • Management review results are used by the accountable executive as input to the improvement processes.


Table D2.1 – Documentation – Identification and Maintenance of Applicable Regulations
Score Criteria
Not in Conformance
1 There is no system for the identification and maintenance of applicable regulations.
2 (3) less some aspects
In Conformance
3

  1. The organization has established, maintained and adhered to documentation requirements applicable to the certificate(s) held, as required by the CARs.

  2. A documented procedure has been established and maintained for identifying applicable regulatory requirements.

  3. Regulations, Standards and exemptions are periodically reviewed to ensure that the most current information is available.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • All pertinent technical and regulatory information is readily accessible by personnel.

  • The organization has defined specific triggers that would lead to a review of the company documentation.

  • The organization actively participates in regulatory development activities and anticipates the introduction of new requirements (Notice of Proposed Amendments, CARAC Technical Committee Meetings, etc.).

Return to Top of Page
Top of Page
  Table D2.2 – Documentation – SMS Documentation
Score Criteria
Not in Conformance
1 SMS documentation is incomplete and maintenance procedures are not well established.
2 (3) less some aspects
In Conformance
3

  1. There is controlled documentation that describes the SMS and the interrelationship between all of its elements.

  2. Documentation is readily accessible to all personnel.

  3. There is a process to periodically review SMS documentation to ensure its continuing suitability, adequacy and effectiveness, and that changes to company documentation have been implemented.

  4. There are acceptable means of documentation, including but not limited to, organizational charts, job descriptions and other descriptive written material that defines and clearly delineates the system of authority and responsibility within the organization for ensuring safe operation.

  5. The organization has a process to identify changes within the organization that could affect company documentation.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • The consolidated documentation is readily accessible by all, including third parties.

  • There is evidence that the company has analyzed the most appropriate medium for the delivery of documentation at both the corporate and operational levels.

  • There is evidence that the company has analyzed all areas of the company including non-certificated areas and provided documentation that is integrated and appropriate for the company as a whole.



Table D2.3 – Documentation – Records Management
Score Criteria
Not in Conformance
1 There are no processes in place for managing SMS output.
2 (3) less some aspects
In Conformance
3

  1. The organization has a records system that ensures the generation and retention of all records necessary to document and support the regulatory requirements.

  2. The system shall provide the control processes necessary to ensure appropriate identification, legibility, storage, protection, archiving, retrieval, retention time, and disposition of records.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

The organization has a policy that defines how long records that are not specifically required by regulations are kept.
Return to Top of Page
Top of Page
  Table D3.1 – Safety Oversight – Reactive Process – Reporting
Score Criteria
Not in Conformance
1 The reporting processes do not ensure the capture of internal information, nor do they promote voluntary reporting of observed occurrences or deficiencies. Reports are not reviewed at the appropriate level of management.
2 (3) less some aspects
In Conformance
3

  1. The organization has a process or system that provides for the capture of internal information including hazards, incidents and accidents and other data relevant to SMS.

  2. The reactive reporting system is simple, accessible and commensurate with the size and complexity of the organization.

  3. Reactive reports are reviewed at the appropriate level of management.

  4. There is a feedback process to notify contributors that their reports have been received and to share the end results of the analysis.

  5. There is a process in place to monitor and analyze trends.

  6. Corrective actions are generated and implemented to respond to event analysis.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • The organization has a process in place to ensure confidentiality when requested.

  • The range and scope of safety-related occurrences or deficiencies that must be reported by employees are explained and defined.

  • There is evidence that personnel are encouraged and supported to suggest corrective actions when submitting a report.

  • There is a process to ensure that information is received from all areas of the organization including certificated, non-certificated and third parties.

  • There is a process to share safety reports and other analyses with report submitters.

  • The feedback process provides an opportunity for report submitters to indicate whether they are satisfied with the response.



Table D3.2 – Safety Oversight – Proactive Process – Hazard ID
Score Criteria
Not in Conformance
1 The existing procedures do not promote the generation of safety data other than on a reactive basis.
2 (3) less some aspects
In Conformance
3

  1. The organization has a proactive process or system that provides for the capture of information identified as hazards and other data relevant to SMS and develops a hazard register.

  2. The proactive reporting process is simple, accessible and commensurate with the size and complexity of the organization.

  3. Proactive reports are reviewed at the appropriate level of management.

  4. There is a feedback process to notify contributors that their proactive reports have been received and to share the end result of the analysis.

  5. There is a process in place to monitor and analyze trends.

  6. Corrective actions are generated and implemented in response to hazard analysis.

  7. The organization has planned self-evaluation processes, such as regularly scheduled reviews, evaluations, surveys, operational audits, assessments, etc.

  8. The organization conducts hazard analyses and builds a safety case for changes that may impact their operations for example:

    • introduction of new aircraft type;

    • change in route structures;

    • change in key personnel;

    • mergers; and

    • management/bargaining agent agreements.

  9. The organization has a clearly defined interval between hazard analyses.
  10. The organization will develop a safety risk profile that prioritizes hazards listed on the hazard register.
4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:
  • The organization has a process in place to ensure confidentiality when required.
  • The range and scope of safety-related hazards that must be reported are explained and defined.
  • All proactive reports are subjected to a risk analysis process to determine their level of priority on the safety risk profile and the extent of further action.
  • The organization has identified multiple sources of information for hazard identification that includes all areas of the organization, such as line management judgment, workplace opinions, minutes of safety meetings, audit reports, flight data monitoring programs (FDMP), Line Operations Safety Audits (LOSA).
  • There is evidence that industry data such as a generic industry wide safety risk profile (from sources such as the Aviation Safety Reporting System, the Securitas TSB reporting system) is considered in the hazard identification process.
  • There is evidence that the organization actively seeks information related to safety from outside sources and utilizes that information in its normal business practices.
  • There is a process to ensure that data is received from all areas of the organization including certificated, non-certificated and third parties.
Return to Top of Page
Top of Page
  Table D3.3 – Safety Oversight – Investigation and Analysis
Score Criteria
Not in Conformance
1 The organization does not routinely conduct investigation and analysis of safety-related occurrences and deficiencies.
2 (3) less some aspects
In Conformance
3

  1. There are procedures in place for the conduct of investigations.

  2. Measures exist to ensure that all reported hazards, incidents and accidents are reviewed and, where required, investigated.

  3. There is a process to ensure that hazards, incidents and accidents are analyzed to identify contributing and root causes.

  4. When identifying contributing and root causes, the organization considers individual human factors, the environment, supervision and organizational elements.

  5. The organization has a staff of competent investigators commensurate with its size and complexity.

  6. Results of the analysis are communicated to the responsible manager for corrective action and to other relevant managers for their information.

  7. There is a process to capture information from an investigation that can be used to monitor and analyze trends.

  8. There is evidence that the organization has made every effort to complete the investigation and analysis process in the established timeframe.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • There is evidence that the organization analyses other types of safety reports received from sources such as the environment, occupational health and safety and third party contractors that may have an impact on aviation safety.

  • There is evidence that third party stakeholders have been consulted during the root cause analysis process for example, manufacturers, suppliers and distributors.

  • The organization provides support to third party stakeholders in the conduct of investigation and analyses of hazards, incidents and accidents outside of the scope of the operating certificate held.



Table D3.4 – Safety Oversight – Risk Management
Score Criteria
Not in Conformance
1 The organization does not have a process for evaluating and managing risks.
2 (3) less some aspects
In Conformance
3

  1. There is a structured process for the management of risk that includes the assessment of risk associated with identified hazards, expressed in terms of severity and probability of occurrence and, where applicable, the level of exposure.

  2. There are criteria for evaluating the level of risk and the tolerable level of risk the organization is willing to accept.

  3. The organization has risk control strategies that include risk control, risk acceptance, risk mitigation, risk elimination and where applicable a corrective action plan.

  4. Corrective actions resulting from the risk assessment, including timelines, are documented.

  5. The organization has a process for evaluating the effectiveness of the corrective actions.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • The organization uses its risk management results to develop best practice guidelines that it shares with the industry.

  • The results of the risk management program are built into the organization’s methods and procedures.

Return to Top of Page
Top of Page
  Table D4.1 – Training – Training, Awareness and Competence
Score Criteria
Not in Conformance
1 Training requirements are not documented, nor does the organization incorporate SMS training into indoctrination training.
2 (3) less some aspects
In Conformance
3

  1. There is a documented process to identify training requirements so that personnel are competent to perform their duties.

  2. There is a validation process that measures the effectiveness of training.

  3. Training includes initial, recurrent and update training, as applicable.

  4. The organization’s safety management training is incorporated into indoctrination training upon employment.

  5. Training includes human and organizational factors.

  6. There is emergency preparedness and response training for affected personnel.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • Training requirements are documented for each area of activity within the organization, including non-certificated areas where training requirements are not defined by regulations. The attendance at symposia should also be considered.

  • Training is provided for all employees.

  • Training is provided for third party contractors working in activities related to the company’s operation.

  • A training file is developed for each employee, including management, to assist in identifying and tracking employee training requirements and verifying that personnel have received the planned training.

  • Employees have a mechanism to request additional SMS training in relation to their role in the SMS.

  • Management recognizes and uses informal opportunities to instruct employees on safety management.

  • Training exercises and methods for all employees is kept current to reflect new techniques, technologies, results of investigations, corrective actions and regulatory changes.



Table D5.1 – Quality Assurance – Quality Assurance
Score Criteria
Not in Conformance
1 The organization does not perform audits of its processes at the operational level.
2 (3) less some aspects
In Conformance
3

  1. A quality assurance program is established and maintained, and the program is under the management of an appropriate person.

  2. There exists an operationally independent audit function with the authority required to carry out an effective internal evaluation program.

  3. The organization conducts reviews and audits of its processes, its procedures, analyses, inspections and training.

  4. * The organization has a system to monitor for completeness the internal reporting process and the corrective action completion.*

  5. The quality assurance system covers all functions defined within the certificate(s).

  6. There are defined audit scope, criteria, frequency and methods.

  7. A selection/training process to ensure the objectivity and competence of auditors as well as the impartiality of the audit process.

  8. There is a procedure to record verification of action(s) taken and the reporting of verification results.

  9. * The organization performs a periodic Management review of safety critical functions and relevant safety or quality issues that arise from the internal evaluation program.*

  10. There is a documented procedure for reporting audit results and maintaining records.

  11. There is a documented procedure outlining requirements for timely corrective and preventive action in response to audit results.

  12. * There is evidence that the quality assurance program has itself been subjected to internal audits.

  13. Competence to perform duties is evaluated.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • Audit scope, frequency and criteria are based on the status and importance of the processes and areas to be audited and results of previous audits.

  • Input from the area to be audited, as well as from other interested parties is considered in the development of internal audit plans.

  • The audit report recognizes excellence to provide opportunities for recognition by management and motivation of people.

  • Evidence exists that the organization encourages the development of a robust audit program and is committed to improving its performance based on the program’s outputs.

  • Where contracted functions exist, the organization performs a quality assurance review on those functions.

* These expectations are only applicable to organizations with an SMS.

Return to Top of Page
Top of Page
  Table D6.1 – Emergency Preparedness – Emergency Preparedness and Response
Score Criteria
Not in Conformance
1 The organization does not have a documented emergency preparedness policy. Roles and responsibilities in the event of an accident are poorly defined.
2 (3) less some aspects
In Conformance
3

  1. The organization has an emergency preparedness procedure appropriate to the size, nature and complexity of the organization.

  2. The emergency preparedness procedures have been documented, implemented and assigned to a responsible manager.

  3. The emergency preparedness procedures have been periodically reviewed as a part of the management review and after key personnel or organizational changes.

  4. The organization has a process to distribute the Emergency Response Plan (ERP) procedures and to communicate the content to affected personnel.

  5. The organization has conducted drills and exercises with all key personnel at intervals defined in the approved control manual.

4 All of (3) plus some aspects of (5)
5 All of (3), plus all of the following:

  • The organization has Memoranda of Understanding (MOU) or agreements with other agencies for mutual aid and the provision of emergency services.

  • The organization has a designated command post where the overall coordination and general direction of the response to an emergency takes place.

  • A procedure exists for recording activities during an emergency response.

Date modified:
2013-01-25