Audit of Aviation Security Regulatory Oversight

Print Version |

April 2011

Table of Contents



Executive Summary

Introduction

As part of its responsibilities for promoting a secure transportation system, Transport Canada (TC) sets the legislative, regulatory and program standards for aviation security. While TC is responsible for setting regulations and security measures, other stakeholders, such as aerodromes, airlines and the Canadian Air Transport Security Authority (CATSA), are responsible for implementing the security requirements. TC monitors and encourages stakeholder compliance of the security regulations through inspection and enforcement activities.

Following the events of September 11, 2001, the federal government invested rapidly and significantly in aviation security. New regulations and security measures were brought into force, including the establishment of CATSA. TC enhanced its regulatory oversight program by hiring a large number of regional inspectorate staff to monitor and enforce stakeholder compliance with the regulations. Today, TC’s Aviation Security Program (AvSec) is responsible for establishing the civil aviation security regulatory framework and carrying out oversight activities. The AvSec Program and functional direction provided to the regional inspectorate have evolved rapidly and in many directions in response to emerging or known threats and the need to align with international requirements. These changes, coupled with the complex nature of aviation security and the potential for regional disparities inherently exposes risks to achieving the objectives of the AvSec Program. In recognition of these risks, TC has set specific priorities and implemented a range of initiatives to modernize, solidify and enhance the AvSec Program.

Audit Objectives & Scope:

The objective of the audit was to provide reasonable assurance that the governance, risk management practices and control processes in place within AvSec are effective and achieving the objectives of the following program activities:

  • Aviation Security Regulatory Framework;
  • Aviation Security Oversight; and,
  • Air Cargo Security.

The audit was national in scope, assessing a broad range of governance, risk management practices and control processes in place within the Department’s headquarters and its five regions.

The audit of Air Cargo Security (ACS) employed a different approach as this program is currently under development. The audit assessed the adequacy of the design of the program’s controls rather than the operating effectiveness of those controls. As such, results related to ACS are reported in a separate section of the report.

Audit Opinion

Based on our examination of the management control framework, it is our opinion that the Department’s efforts to implement adequate and effective governance, risk management practices and control processes are appropriate, but not yet adequate. Strategic direction is appropriately focused and operational risks associated with the AvSec Program will be adequately and effectively addressed, provided that the operational direction is implemented as currently planned.

It is also our opinion that AvSec Program’s most significant outstanding risk is to effectively manage the magnitude of change underway. While changes are being implemented to improve operations, the systematic management of these transformational initiatives will be key going forward. Additionally, an increased emphasis on integrated risk management and program performance management is required to provide assurance that program outcomes will be achieved.

Statement of Assurance/Reliance

It is our professional judgment that the audit has been conducted in accordance with the Internal Auditing Standards of the Government of Canada as prescribed by the Comptroller General. Satisfactory procedures for the audit have been conducted, and sufficient relevant evidence has been gathered to support the accuracy of the opinions provided in this report.

Summary of Key Findings

Air Cargo Security:

The audit examined the degree to which ACS has appropriately designed its management control framework and the controls in place to transition to a new regime. The audit concluded that a strong management culture and due diligence has resulted in robust program management and control and a well-managed approach to change and project management.

AvSec Program - Governance:

The audit examined the suite of practices that collectively set program direction at the strategic and operational levels, as well as the processes that enable the monitoring of results in support of continuous improvement.

The audit concluded that strategic program directions, including the policy, legislative and regulatory frameworks are currently being formalized in a more coherent and consolidated manner than in the past and are in line with stated priorities. Operationally, national program direction in relation to TC’s Regulatory Oversight function is also under development and requires immediate and sustained attention. The prompt finalization of the TC Oversight Framework, oversight philosophy and Modus Operandi is vital in ensuring nationally consistent operations and laying the foundation for other important operational improvements concurrently being pursued by the Program.

AvSec’s management approach has been more reactive than proactive due to the external environment in which it operates. A more proactive management approach is required to integrate performance management at the AvSec program level as the elements of performance management currently in place or under development are not all aligned. Additionally, as new operational protocols are developed, a functioning Quality Assurance (QA) program will be required to establish a nationally consistent AvSec program.

AvSec Program - Risk Management:

The audit examined the degree to which AvSec has implemented appropriate and comprehensive risk assessment practices. As well, because the objective of risk management is to inform decision-making processes, the audit examined the manner in which the outputs of these risk assessment activities (i.e. intelligence, knowledge and insight) are being leveraged to consistently inform decision-making, resource allocation and governance processes.

The audit concluded that considerable progress has been made with respect to the design and implementation of risk assessment methods. Risk-based tools exist for identifying and assessing risks across the AvSec spectrum, including regulatory development, incident management, threat and risk assessment, operational planning and business planning. However, neither the processes nor their outputs were found to be aligned or integrated. As a result, TC is more highly exposed to the risk that operational or strategic exposures will go undetected, be misunderstood or be addressed in an inconsistent or inappropriate manner. Enhanced integration and sharing of risk information is likely to yield opportunities for greater efficiency and more value-added analysis, which will ultimately strengthen the program.

AvSec Program - Controls:

As part of its efforts to modernize the AvSec program, AvSec Operations is undertaking many initiatives that are collectively aimed at updating and improving its oversight activities. For example, efforts are currently being made to update the oversight framework and redevelop the Standard Operating Procedures (SOPs) to promote national consistency. Considering the pivotal role these initiatives will have in successfully realizing the goals of the AvSec Program, the audit has concluded that they must be finalized and implemented as soon as possible.

Although much progress has been made and many key elements of change management have been put in place, a more structured and integrated approach is needed to ensure the successful implementation of the vast array of changes within the AvSec Program.

Management Response

The Aviation Security Directorate (AvSec) of Transport Canada welcomes the Audit and Advisory Services recent audit of its regulatory framework and oversight program activities. As the audit states, “the Department’s efforts to implement adequate and effective governance, risk management practices and control processes are appropriate, but not yet adequate”. AvSec is fully aware of, and committed to, the need to put in place more satisfactory and effective modes of governance, risk management practices and control processes. In addition, we echo the expressed sentiment that "the AvSec Program’s most significant outstanding risk is effectively managing the magnitude of change underway."

Most importantly, AvSec shares the underlying concern expressed throughout the audit; namely, that in order to promote national consistency and overall quality assurance it is critical that Transport Canada take a coordinated and holistic approach to its regulatory framework and oversight program activities.

It must be noted that "effectively managing the magnitude of change underway" is a multi-year initiative. Time is necessary to make sure the sequencing of the various changes is correct and to ensure that all the different components fit into a larger holistic approach to security. To be sure, there are some short-term changes that can be implemented within the next twelve months; however, the bulk of the recommendations will require both a long-term vision and plan. As the now drafted National Civil Aviation Security Plan demonstrates, Transport Canada has a vision and a plan.

Director, Audit and Advisory Services and Chief Audit Executive Signatures

Original signed by

 
Dave Leach (CIA), Director,
Audit and Advisory Services

 

March 21, 2011

 
Date

 

 

Original signed by

 
Laura Ruzzier,
Chief Audit Executive

 

March 21, 2011

 
Date

 



1. Introduction

1.1 Purpose

An audit of Aviation Security Regulatory Oversight was included in the Audit and Advisory Services’ 2010 Risk-Based Audit Plan. The purpose of the audit was to provide assurance that the management controls in place within the Aviation Security (AvSec) Program are effective and adequate to support both the development and maintenance of a Regulatory Framework and Oversight function. In addition, as the Air Cargo Security program remains under development, the audit examined the project and change management functions to ensure the appropriate design of the management control framework.

1.2 Background

Transport Canada (TC) is responsible for promoting a secure transportation system and developing and administering policies, regulations and programs aimed at regulating all modes of transportation including marine, rail and aviation within the federal jurisdiction. Responsibility for the regulation of security in the aviation mode rests with the AvSec Program, the objective of which is to promote a secure and competitive civil aviation system and in so doing, support national security. It does this through the development, administration and oversight of policies, programs, regulations and standards necessary for a secure Canadian civil aviation system, in a manner harmonized with the international aviation community.

Responsibility for aviation security is a shared one. TC sets and administers the regulations to be applied by the stakeholders including aerodrome operators, airlines and the Canadian Air Transport Security Authority (CATSA). Stakeholders are directly responsible for implementing security requirements in compliance with the regulations. TC also performs a regulatory oversight function. Through inspection and enforcement activities, TC monitors compliance of the stakeholders to the regulations and other security measures, in accordance with the principles of shared responsibility for security. TC inspectors are mandated to inspect stakeholder compliance with legislation. If non-compliance occurs, enforcement tools are available to enforce compliance.

The policy and regulatory development function is located at Headquarters (HQ), and includes:

  • Aviation Security Policy, which is responsible for strategic analysis, direction, coordination, and advice on the development, implementation, review, and enhancement of security policies dealing with aviation in Canada.
  • Regulatory Review, which is responsible for the review and updating of existing regulations and security measures, moving them to a performance- and risk-based regime.
  • Regulatory Affairs, which coordinates the ongoing development of regulatory proposals, policy and strategies with safety and security inspection, technical and legal staff to manage the technical aspects of the consultations with stakeholders.

The regulatory oversight function is delivered through a matrix management structure. Functional leadership, policy direction and program oversight is the responsibility of AvSec Operations in HQ. Responsibility for inspection and enforcement activities rests with the regions.

Air Cargo Security (ACS) is responsible for the development and implementation of security measures, regulations, standards and guidelines governing air cargo screening and supply chain security. ACS is also responsible for the development and implementation of a training and accreditation regulatory regime which will involve a variety of regulatory oversight functions, such as inspection, audit, training, and outreach to promote and enforce compliance, security operations systems and support, and general policy development, coordination and guidance.

1.3 Operational History and Risk Environment

Following the events of September 11, 2001, the federal government invested rapidly and significantly in aviation security. New regulations and security measures were brought into force, including the establishment of CATSA. TC enhanced its regulatory oversight program by hiring a large number of regional inspectorate staff to monitor and enforce stakeholder compliance with the regulations. Today, TC’s AvSec Program is responsible for establishing the civil aviation security regulatory framework and carrying out oversight activities. The AvSec Program and functional direction provided to the regional inspectorate have evolved rapidly and in many directions in response to emerging or known threats and the need to align with international requirements.

The two most relevant risk factors characterizing the AvSec program are the ever evolving nature of security threats and the move towards new regulatory approaches. These risk factors are compounded by

  • the high degree of inter-dependency within a shared security regime;
  • the considerable complexity of the security function;
  • a history of ill-defined program expectations in some areas; and,
  • human resources challenges in the operational areas of the program.

To address these challenges and in response to a multitude of reviews and studies, more change is being introduced to the program. As described later in this report, a range of transformational initiatives are being pursued, touching almost all facets of the program. These changes, while aimed at improving operations, also introduce considerable risk into the system, particularly during the transition period.

Appendix A (has been removed and is available upon request) provides a summary of the risks to which the program is inherently exposed. These risks formed an important foundation for this audit, as they were used to focus the scope and the lines of inquiry to ensure that the key controls and practices in place to mitigate the risks were examined.

1.4 Audit Objectives

The objective of this audit was to provide reasonable assurance that the governance, risk management practices and control processes in place within AvSec are effective and achieving the objectives of the following program activities:

  • Aviation Security Regulatory Framework:
    • develops and balances the use of various tools such as policies, guidelines, regulations, standards and measures
    • promotes a secure and harmonized Canadian civil aviation security regime.
  • Aviation Security Oversight:
    • supports compliance with the security framework through inspections, audits, monitoring, surveillance, enforcement and education, when necessary, of the aviation industry.
  • Air Cargo Security:
    • supports the National Security Policy by identifying strategies to enhance the security of air cargo.

These management controls have been categorized into three broad areas namely the Governance, Risk Management, and Control processes. Twelve key management controls were then identified and criteria were developed for each. Appendix B (has been removed and is available upon request) highlights the key controls and criteria against which the controls were audited.

1.5 Audit Scope

The audit was national in scope. It assessed the governance, risk management, and control processes that exist both within HQ and in each of the five regions. At HQ, interviews were conducted with directors and key program personnel in all of the AvSec branches as well as the Security Program Support Directorate, which reports directly to the ADM Safety and Security, and corporate Human Resources (HR).

At the regional level, interviews were conducted with approximately 50% of AvSec employees of all levels of AvSec management and inspectors both at Class I airports and Class II and Other airports. As well the Regional Director Generals (RDGs), the Regional Directors (RDs) of Security, and HR were interviewed.

In addition, a large number of key stakeholder organizations were interviewed. Two of the three CATSA Regional Directors and representatives from approximately half of CATSA operations at Class I airports were interviewed. Aerodrome representatives from close to three quarters of the Class I airports were also interviewed.

1.5.1. Audit Scope Inclusions/Exclusions

Audit scope inclusions and exclusions are described in Appendix C (has been removed and is available upon request).

1.6 Audit Approach

It is our professional judgment that the audit has been conducted in accordance with the Internal Auditing Standards of the Government of Canada as prescribed by the Comptroller General. Satisfactory procedures for the audit have been conducted, and sufficient relevant evidence has been gathered to support the accuracy of the opinions provided in this report. Through the application of analytical techniques, documentation review and interviews, the key management controls in place to mitigate risk and achieve program objectives were identified. The control areas were then assessed for the adequacy of their design and their operational effectiveness.

The ACS component of AvSec was audited differently than the rest of the program. As this program is currently under development, the audit assessed the adequacy of the design of controls, but it did not test their operating effectiveness. As a result of this “program under development” approach, results related to ACS are reported separately.

1.7 Structure of Report

The audit examined twelve (12) key controls against forty-two (42) audit criteria. For clarity, the audit findings are grouped into eight (8) general themes and categorized under the three over-arching categories of Governance, Risk Management and Control. (Appendix D has been removed and is available upon request)

Recommendations are provided both in the Findings and Recommendations section of the report and then repeated in the Management Response and Action Plan section of the report. Each recommendation has been assigned a significance rating based on the significance rating scale found in Appendix E (has been removed and is available upon request).



2. Findings and Recommendations

The efforts being made by the AvSec program to implement effective governance, risk management and control processes are appropriate, but not yet adequate. Strategic direction is appropriately focused and while operational direction still requires formalization, efforts underway, if implemented as planned, will adequately and effectively address the operational risk to which the AvSec Program and TC is exposed.

The most significant outstanding risk facing the AvSec Program is the magnitude of change with which it is faced. While changes are being implemented to improve operations, the systematic management of transformational initiatives will be key going forward. This, together with an increased emphasis on the fundamentals of program management will be necessary to provide assurance on program outcomes.

2.1 Air Cargo Security

Air Cargo Security (ACS) employs a robust and well-managed approach to change and project management.

Since its inception as an initiative in 2008, the program has instituted a robust project management structure. As the initiative moved into program status, the project has evolved into a structured program management regime. The following processes and practices are in existence and thoroughly documented:

  • Formal program governance is articulated as is its objectives, outcomes, risks, assumptions and constraints.
  • A detailed work breakdown structure documents the scope of the program, including interconnections between program activities and other key players in the department and beyond. This is complemented by a detailed program schedule and key activities.
  • Extensive Standard Operating Procedures (SOPs) that cover a wide range of program activities such as program team roles, change request processes, risk management and reporting.
  • Status report to the ACS management committees are provided, including dashboards showing progress against plan and activity summaries.

As well, ACS is in the process of reviewing its Management Control Framework. The objective of this review is to support ACS in achieving its program objectives by reviewing controls, identifying control weaknesses, and making recommendations to strengthen them. Efforts such as these are indicative of a strong and diligent management culture and tone at the top, including a commitment to proactive management.

2.2 Governance

Governance is defined as the combination of processes and structures implemented to inform, direct, manage and monitor the activities of the organization toward the achievement of its objectives[1]. As noted in the introduction to this report, the AvSec program is a highly complex one, characterized by multiple players, each with their respective priorities and practices. Clarity and consistency of program direction, objectives, expected results and accountabilities is critical in such an environment. Good governance allows for information on activities, risks and results to be appropriately shared and considered in support of a resilient, effective, efficient and cost-beneficial program.

The audit findings on governance are grouped under two broad themes namely Leadership and Functional Direction and Managing Program Performance.

2.2.1. Leadership and Functional Direction

Strategic directions are currently being formalized, are more coherent than in the past and respond to known priorities; however, they still need to be finalized.

AvSec is in the process of updating and consolidating the TC Policy Framework for Aviation Security into a comprehensive policy document titled the National Civil Aviation Security Program (NCASP). The NCASP is designed to provide a consolidated set of expectations for stakeholders and TC to guide their respective aviation security responsibilities. It is in the developmental stages but contains a clear overview of the program’s policy goals and principles, its legal and financial frameworks and a description of the AvSec Program.

The NCASP will ultimately contain an updated and consolidated oversight framework that will lay the foundation and directions for TC’s regulatory oversight function. AvSec Operations anticipates finalizing the oversight framework by March 2011. It is imperative that the oversight framework be completed as soon as possible so that other critical operational improvements that depend upon this framework can themselves be finalized.

National program direction in relation to AvSec Operations is under development and needs immediate and sustained attention.

As is noted elsewhere in this report, the extent and sustained nature of AvSec Program change is considerable. In the course of managing change, functional direction must still be clearly defined to support consistent, effective and efficient operations. The audit found there is insufficient clarity and functional direction to guide the AvSec Operations nationally. Many of the issues related to the clarity of direction are also presented in the Regulatory Oversight Protocols section of this report. These issues have led to a situation where regions are provided with insufficient or unclear direction on operational requirements and expectations. Specific concerns include the following:

  • There is not a consistent understanding among the inspectorate of the oversight philosophy. Some inspectors view their objective as one of “policing” the stakeholders while others perceive their roles as working with the stakeholders to ensure a secure system.
  • Some inspectors have the mistaken perception that their delegated authority precludes the need for managerial supervision. While inspectors are fully delegated to inspect and enforce, this does not negate the role of managerial supervision and oversight.
  • As noted later in this report (under Regulatory Oversight and Enforcement Protocols), insufficient attention appears to have been placed on disseminating national direction in relation to operational oversight activities (i.e. inspection and enforcement activities). As a result, regions have taken it upon themselves to create their own inspection tools and approaches, some of which are shared nationally.

The result of the above is a heightened risk of regional inconsistency that is further compounded by the lack of effective monitoring of operations in HQ, (see Managing Program Performance and Quality).

Efforts are being made to inform inspectors of the AvSec philosophy through a series of national inspectors’ awareness workshops and the early results appear to be positive. The current development of SOPs is another positive initiative. These efforts, together with a strong focus on managerial supervision and individual performance management, are therefore strongly encouraged.

2.2.2. Managing Program Performance and Quality

In evaluating the systems and practices that assess and manage performance, the audit focused on the mechanisms that are in place to monitor TC’s performance in relation to aviation security. AvSec Program performance was assessed broadly with the audit examining TC regional operations’ application of TC protocols as well as the practices for the assessment of quality and program results. The availability and integrity of information for use in Program performance monitoring was also examined.

An overarching AvSec Program performance management framework is not in place, increasing the risk that there will be misalignment of many of the program’s key elements. This is a key risk in light of the extent of change taking place in the program.

As noted under Leadership and Functional Direction, AvSec has made significant strides in defining and consolidating its broad objectives and priorities through various mechanisms such as its draft National Civil Aviation Security Program. In addition, management frameworks and initiatives are being independently developed in various areas of the AvSec program. For example:

  • the AvSec program is currently developing regulations that are performance based.
  • the Inspection and Enforcement (I&E) Manual is being updated and re-written into SOPs.
  • an Operational Performance Framework (OPF) is being developed to identify performance expectations of stakeholders.

What does not exist is an overarching AvSec Program performance management framework that clearly articulates the overall AvSec Program objectives and how each of the components and initiatives of the AvSec program contribute are expected to those objectives. Without such an overarching AvSec Program performance management framework, there is a risk that the various components of the AvSec program may not entirely align with the program’s objectives.

The following describes in more detail the above three activities and how they may not be fully aligned with the program’s objectives without an overarching AvSec Program performance management framework.

Regulations: Performance-based regulations are in the early stages of development and regulatory outcomes are being identified using appropriate and robust methods. However, the outcomes that are being defined are those related to the regulatory instrument themselves and are not linked to any broader overarching program performance management framework, although generally a Secure Transportation System (commonly referred to as Strategic Outcome 4) describes the ultimate outcome.

SOPs: Similarly, the Inspection and Enforcement (I&E) Manual is being re-written so that it more clearly articulates performance outcomes and expected standards for inspection activity and data collection. Without broad performance expectations for the program, there is no formal means by which to ensure the outcomes contained in the I&E Manual are aligned to either the regulatory outcomes (whether current or evolving) or the outcomes being set for stakeholder performance in the Operational Performance Framework (OPF).

OPF: The OPF is being developed to measure the overall security of the civil aviation transportation system. This work, which is in the early stages of design, is focused on measuring the effectiveness of the practices of stakeholders. While the audit supports the OPF direction and underlying theory, it identified some concerns with the OPF and its potential ability to measure performance:

  • The OPF is not measuring the performance of TC’s regulatory oversight function (i.e., the effect of inspection and enforcement on the overall outcome of aviation security). Rather, its focus is on the activities of the stakeholders. So, while the OPF may provide important insight into the effectiveness of the stakeholders and thus the security of the civil aviation system, it will not provide any insight into the relative value and effect of TC’s oversight investments.
  • The OPF is expected to use data from stakeholder data or from TC (most likely inspector data held within Security Emergency Preparedness Inspection Reporting System (SEPIRS) or other systems). As outlined later in this report, known SEPIRS issues related to inconsistent inspection data entry and the inability to retrieve data creates a risk for the OPF if those issues are not addressed. The use of stakeholder data may also expose the system to risk. AvSec has acknowledged this risk and early plans are in place to implement independent verification and validation of the data and the systems that generate them. The exact approach to validating the data has not yet been determined.
  • Finally, the successful completion and implementation of the OPF may be at risk given the framework and underlying principles and practice of the OPF are highly complex and knowledge of it resides primarily with a Director "atip removed". There appears to be limited documentation regarding OPF (beyond decks) and currently there is no succession plan in place.

It is important to note that aligning outcomes across performance-based regulations, new and revised SOPs and the OPF is made even more challenging since all of these initiatives are underway concurrently. However, this situation also provides an ideal opportunity to align these important elements along common objectives and overarching expected results.

While the program has employed a Quality Assurance Program in the past, functional reviews of the Regional AvSec regulatory oversight functions have not been conducted since FY 2006/07.

Re-instituted in 2002, the AvSec Quality Assurance (QA) program’s objective involves the evaluation of the inspection and enforcement program. The evaluation included a general review of the quality and completeness of paper and electronic files as well as compliance with the various policies and procedures. Competing priorities led to the abandonment of these reviews in FY 2006/07. The absence of a functioning QA program, particularly during times of considerable change and in the context of a regionally distributed operational model, constitutes a significant risk to the consistency of operations. AvSec management has indicated it is planning to reinstitute the QA.

Managerial supervision in the regions is adequate, although issues of trust, managerial authority and individual performance management remain outstanding in some locations.

Closely related to the issues of the Control Environment, which is presented in the Control section of this report, the quality of managerial supervision directly contributes to overall program performance. The audit examined the supervisory practices in each of the regions and found that, overall, managerial supervision is appropriate. In most locations, roles and responsibilities of inspectors relative to supervisors are clear and well respected and methods exist to ensure that supervision is appropriately carried out. A resistance to managerial oversight and supervision, however, characterizes some locations. This resistance may originate from a misunderstanding of Transportation Security Inspectors (TSI) delegations, or from historical issues and grievances. Regardless of the source of the problem, greater focus is needed to reinforce the appropriateness and necessity of managerial oversight and supervision, without which, effective operational monitoring and continuous improvement cannot fully take place.

2.2.3. Conclusions and Impacts

While investments have been made in laying the foundation for program direction, the timely finalization of functional direction is needed to set, reinforce and communicate operational expectations for the regulatory oversight function. This, coupled with a clear understanding of expected outcomes, will lay the foundation for a standardized national program.

In addition, while some efforts have been made to lay the foundation for AvSec Program performance management, the rapid pace of change and growth within the program has led to insufficient sustained attention to this critical management function. As well, this may be attributable to a managerial culture, which has, in the past, tended to be more reactive than proactive in nature. It is also likely related to the number of ‘moving parts’ associated with the AvSec program and the inherent challenges of keeping them coordinated and aligned.

Program performance and quality management is critical to mitigating the most significant risks to the AvSec program. At present, due to the above-noted conditions, AvSec Operations is still exposed to the risk of insufficient information for decision-making, oversight and continuous improvement. Without an immediate and sustained focus on this area, there is the risk that

  • inconsistent and potentially non-standard TC regional inspections will continue to be pervasive;
  • insufficient or unreliable information will continue as a condition, making it difficult to set and defend priorities, or to support management and governance bodies in their discharge of responsibilities;
  • change management, strategic or operational course corrections and continuous improvement will be severely limited; and
  • accountability, governance and decision-making at all levels may be weakened.

While compensating measures are being taken such as improving lines of communication, ensuring the involvement of inter-connected directorates and regions on working groups tasked with improving the AvSec Program, a significant level of risk remains that warrants additional action.

2.2.4. Recommendations

The Assistant Deputy Minister, Safety and Security should ensure the following:

Significance Recommendation
Very Significant 1. Finalize the drafting of the National Civil Aviation Security Program (NCASP) with particular emphasis on the Aviation Security Oversight Framework component so as to provide staff and stakeholders with a clear and documented oversight philosophy, and lay the foundation for critical operational improvements and changes, including consistent implementation of national standard operating procedures.
Very Significant 2. Develop an overarching logic model to articulate the Program’s expected results chain (activities, outputs, immediate and intermediate outcomes) and define how TC contributes to the Program’s expected outcome, i.e., security of civil aviation. This will help ensure that all program elements, particularly those that are in development or undergoing change, are clearly and explicitly aligned to a common outcome and will facilitate performance tracking and reporting.
Significant 3. Develop a succession plan for the Director responsible for the implementation of the Operational Performance Framework (OPF) so as to ensure the successful completion and implementation of the OPF, a key component of the NCASP and critical to its successful implementation.
Significant 4. Implement a Quality Assurance function that regularly monitors regional delivery of the Aviation Security Program so as to ensure appropriate oversight is carried out, is consistent across regions and supports continuous improvement.
Moderate 5. Finalize the Service Level Agreement between the Aviation Security Program and the Security Program Support Directorate to ensure clear understanding of roles and responsibilities in relation to functions such as training and development of inspectors. Given the pace of change occurring in the Aviation Security Program, this is critical to ensuring training and development needs of Aviation Security personnel are met.

2.3 Risk Management

2.3.1. Risk Management

Increasingly, expectations of good governance, combined with resource constraints, have emphasized that risks should be considered in all planning and delivery activities, at the strategic, operational and tactical levels. Integrated Risk Management (IRM) is a set of business practices that assesses, manages and communicates risk at a level appropriate to the organization’s objectives and risk tolerance. It permits the systematic and proactive identification and management of potential events that could undermine objectives and in this way, enhances decision-making and strengthens governance. Ultimately, IRM allows the organization to capitalize on new opportunities, enhance predictability in achieving results, and protect the organization from negative results.

Within AvSec, threats and risk permeate the operation. Indeed, the very existence of these threats is the raison-d’être for the AvSec Program. As well, the AvSec Program itself is exposed to a range of internal operational risks that may hamper the achievement of its objectives. Managing these risks in an environment of competing priorities, whether financial or otherwise, is imperative. Thus, a fulsome, comprehensive framework and set of practices that allows for threats and risks to be identified, understood, assessed and managed is vital in such an environment.

In light of this, the audit examined the degree to which AvSec has implemented and applied appropriate, comprehensive and documented risk assessment approaches. As well, because the objective of risk management is to inform decision-making processes, the audit examined the manner in which the outputs of these risk assessment activities (i.e. intelligence, knowledge and insight) are being leveraged to consistently inform decision-making, resource allocation and governance processes.

TC has developed a regulatory decision-making framework that outlines the principles and value of risk-based decision-making and performance-based decision-making in a regulatory environment. The framework is both comprehensive and appropriate in its direction, although clear guidance on risk tolerance and acceptable levels of risk taking is needed for its successful deployment.

The draft National Civil Aviation Security Policy Framework (NACSP) contains a draft regulatory decision-making framework that outlines and emphasizes the importance of a risk-based and performance-based foundation, upon which to build proactive approaches for decision-making. Acknowledging that TC’s regulations and other instruments are important means by which systemic security threats are controlled, the regulatory decision-making framework specifies the role of threat and risk analysis in selecting the appropriate regulatory instrument and correctly specifies the importance of risk information and communication between TC and its stakeholders and among key TC parties involved in the regulatory and oversight functions. Lastly, the decision-making framework notes the importance of having a solid understanding of the department’s risk tolerance, as a foundation for risk-based decision-making.

The audit concurs with this direction, particularly the imperative of establishing a common risk tolerance. Risk tolerance is a critical, but often neglected tool for the practice of integrated risk management. It provides all players in a system with guidance on acceptable levels of risk-taking and also provides direction on when and in what circumstances risk information should be shared, either for information or decision-making purposes. Such guidance is necessary to ensure that controls (including regulatory controls) are appropriate to the risk and desired level of risk management. It also helps to ensure a common and consistent response to threats and risk across the aviation security system by various parties who may encounter risk-scenarios. Simply stated, without a strong understanding of tolerance, appropriate, balanced and consistent application of control will not be effectively realized.

Various risk assessment approaches exist within AvSec, but in some cases are not sufficiently robust, nor are they sufficiently integrated.

The audit examined the degree to which risk assessment processes are in use to support various aspects of the operation that ought to be risk-based. It concluded that while a number of appropriate risk assessment approaches are in place, more work needs to be done to formalize and regularize some of them. Alignment and integration of the approaches and their outputs is needed to move the program more closely to IRM and the benefits that can be realized from it.

The following identifies some of the areas within the AvSec Program that are/will be conducting threat and risk assessments.

Strategic Risk Assessment: A well-developed strategic risk assessment process is in place to identify the risk exposure associated with specific threat scenarios. Used for regulatory and other purposes, assessments are done periodically, but are not regularized. The conduct of the strategic risk assessments engage appropriate parties in the assessment process, including AvSec Policy, AvSec Technology, Regulatory Review, Operations and Intelligence. At present, the process is applied on a periodic basis, often (but not always) in response to threats that have already materialized. To ensure such risk assessments are more proactive in nature, the planning and conduct of them should be regularized. AvSec Policy has advised that they are considering this.

Security Plans: A risk assessment method has been developed for use by stakeholders in developing risk assessments for security plans. A formal risk assessment process has been defined to support the Aviation Security Assessments (ASA) and the Aviation Security Plans, to be implemented under the new Aviation Security Program regulations. Stakeholders (beginning first with the Aerodrome operators) will be required to develop a security assessment that will consider risk and which will ultimately inform their security program. Under the new regime, TC inspectors will be required to review and approve the assessment.

The Regulatory Review Branch has developed an approach for risk assessment for use by stakeholders. We understand that the regulations, once drafted, will not oblige the stakeholders to use this method as a common approach. While some guidance will be provided, there remains a risk that if stakeholders use different approaches, inconsistency in security assessments and plans (which are based on risk) may result. Further, if TC inspectors are expected to review and approve the results, unless they have a full appreciation of risk management, their ability to make meaningful judgments on the appropriateness of the methods employed may be compromised. To mitigate this risk, Regulatory Review is considering having all security plans for Class I airports (and their substantiating analysis), approved by a national committee. This direction is strongly endorsed by the Audit.

Lastly, formal training and tools have been developed to support ASA implementation, including the risk component; however, the guidance contains no discussion or definition of TC’s risk tolerance (i.e., risk acceptability), nor are there any guidelines for developing it. Without this understanding of tolerance, the above-noted concerns around consistency of judgment may be exacerbated.

Operational Planning: Risk assessment is used to inform the operational planning of inspection and enforcement activity in the regions. The Security Operations Risk Assessment Method (SORAM) provides a systematic approach to assessing risk associated with the ‘universe’ of inspectable areas within the scope of AvSec responsibilities. It is intended to allow the Minimum Inspection Schedule contained in the Inspection and Enforcement (I&E) Manual to be modified based on identified risk levels thereby promoting inspection activity in the areas of greatest need. The audit found that the SORAM system supports the evaluation of risk at an appropriate level and aggregates those risks at the aerodrome, air carrier and screening operations for each airport. For Class II and Other airports, SORAM has the ability to support and inform the planning of inspections at an appropriate level consistent with the Minimum Inspection Schedule. However, for Class I airports there are a number of sub-areas of inspection in the Minimum Inspection Schedule that are not supported in SORAM for planning purposes. Since SORAM does not support the planning of inspections at a lower level in Class I airports, regions develop their plans using their own methods, outside of SORAM. This situation presents a greater risk of inconsistent analysis and decision-making.

While the audit found evidence that risk assessments were done, the audit did not assess the quality of the risk assessments as this was outside the scope of the audit. Some analysis was conducted on the use of SORAM and it was found generally that the use of SORAM was not consistent across regions. Some regions complete SORAM risk assessment and use the information to develop the annual inspection plan while other regions develop their annual inspection plan then input the information into SORAM. If regions are not using SORAM in a consistent fashion, this limits the validity of any regional or national roll up of risks provided by SORAM, which in turn limits the usefulness of these aggregated results to inform program management.

Incident Management/Response: TC applies a formal threat and risk assessment process for incident management and incident response. A formal, documented threat risk assessment is in place for the purposes of incident management and incident response. More information on this finding can be found in the section of the report on Incident Management.

Threat Risk Assessment: A robust situational threat/risk assessment is in place for use by airports in support of emergency response and management. The Canadian Aviation Security Regulations require the airport operator and air carriers to carry out a threat assessment when they are made aware of a threat against the aerodrome, aircraft or flight. In response, the Canadian Airports Council, Air Transport Association of Canada (ATAC), the Canadian Airports Police and CATSA jointly developed a threat/risk assessment (TRA) methodology. Transport Canada representatives were also involved in the development of the approach and participate (but do not lead) the process when it is initiated.

AvSec Intelligence: While the focus of intelligence sharing with AvSec is improving, more effort is needed to track threats and risks more systematically. TC’s intelligence (Intell) function is responsible for providing threat information to AvSec for their decision-making at the regulatory and operational levels. TC receives intelligence information from many sources. In the past, TC has not often been the recipient of needed intelligence information in a timely fashion, in part because it has not been seen as a security organization. Evidence shows this condition is improving. TC’s more active engagement with other security-related departments and agencies through the Canadian Defence Red Switch Network (CDRSN) system is another indication of this improvement.

Intell’s role in the past was to focus more on simply giving AvSec raw intelligence information. They are increasingly playing an advisory role by examining the information, conducting situational analysis and providing AvSec (and others) with the ‘so what?’ thinking (i.e., what is the impact on TC, the transportation system and Canadians of the information being discussed). The audit views this evolution in focus as an appropriate and effective means of contributing to the overall AvSec risk management regime.

While these developments are positive, Intell’s approach to managing threat and risk information needs to be more structured and formalized. At the present time, a process exists whereby pre-defined responses (including escalation) to threat scenarios are identified based on the risk level associated with the event; however, no common criteria yet exist to determine exposure in the first place. Also, no systematic approach is used to structure analysis of the risk as the threat environment evolves. Judgement and accumulated experience of Intell personnel is relied upon, which is not entirely adequate as a practice. Intell is planning to institute formal risk-based processes to help track the identification of threats and emerging incidents and to analyze the impacts of new threats on the overall threat-risk profile. The goal is to ensure that a continuity of knowledge exists and is documented. The output would be an evergreen threat matrix that could be retained and mined for future analysis. The audit strongly encourages these developments.

Escalation of AvSec Risks: Insufficient formal mechanisms exist to allow AvSec risks to be escalated and considered corporately. While risks to AvSec operations must be considered and analyzed within the AvSec Program, formal examination of the risks may be warranted at a higher level, depending on their significance. There is no formal mechanism by which significant AvSec risks may be escalated to and considered in the corporate risk profile.

A draft Integrated Risk Management Policy is being considered corporately, but has not yet been finalized. In the interim, or perhaps through the timely finalization of the Policy, more formal mechanisms need to be defined to link or integrate risk information at the program level with the corporate framework and profile for risk management.

2.3.2. Conclusions and Impacts

The Final Report of the Air India Commission of Inquiry outlined key characteristics of good risk management including:

  • "A common set of protocols for carrying out risk management;
  • Risk management protocols and methods based on current best practices;
  • A performance standard of continual improvement, delivering levels of risk in all relevant areas that are as low as reasonable achievable; and,
  • Acceptable levels of risk control in all of the domains of risk pertinent to aviation security.[2]"

While TC’s various risk assessment processes are generally well designed, they are not aligned, integrated or anchored on a common, integrated risk management framework or policy. While not all elements need to employ the same methods, common principles, supported by a standard lexicon and underpinned by well-understood guidance on acceptable risk taking (tolerance) should be in place to ensure consistency of practice and comparability and usefulness of results. Without this, the effect may be that significant risks could go undetected, be misunderstood or responded to in an inconsistent fashion or inappropriate manner. Opportunities for greater efficiency and more value added analysis may also be realized when risk information is effectively integrated and appropriately shared.

Although TC has recently drafted a well-defined decision-making framework for risk based decision-making, lack of specific guidance on risk tolerance may impede the department’s ability to successfully implement this framework. With appropriate guidance on risk tolerance, AvSec’s ability to make informed decisions related to regulatory instrument choice, control development and regulatory oversight would be enhanced. Ultimately, this will help the program to strike the right balance between security, efficiency and Canadian values.

2.3.3. Recommendations

The Assistant Deputy Minister, Safety and Security should ensure the following:

Significance Recommendation
Very Significant

6. Develop an overarching Integrated Risk Management Framework (IRM) for the Aviation Security Program that:

  1. Specifies IRM objectives and expected results;
  2. Identifies and communicates roles and responsibilities of all key players in the risk management regime which in turn should form the basis for coordination and alignment on accountabilities, information sharing, and training;
  3. Specifies risk tolerance in the form of broad, yet meaningful indicators of risk impact such that all key players working in the various parts of the risk management regime have a common understanding of what constitutes a high or severe risk, boundaries of risk acceptability and what constitutes reasonable responses to unacceptable levels of risk;
  4. Provides, under the broad umbrella of objectives, approach and tolerance, links to specific risk assessment methods, which themselves should be reviewed to ensure their alignment to the broader framework.
  5. Contains a common taxonomy of AvSec Risks and Threats that would be applicable to and useful for the sub-elements of the risk management regime, including:
    1. strategic risk assessment and regulatory decision-making
    2. ASA approval
    3. inspection planning
    4. corporate and business planning
  6. Aligns to the TC Corporate Risk Profile and its risk management and monitoring strategy by outlining the means by which AvSec risk information will be escalated, as appropriate, to the corporate level, for consideration in the broader departmental risk oversight process.
Significant

7. Should stakeholders have the flexibility to use their own risk assessment methods as the basis for their Aviation Security Assessments (ASAs); in order to support comparability of processes and outcomes, the ADM of Safety and Security should ensure that the Aviation Security Program:

  1. Requires that all inspectors reviewing the risk assessments of airports have a solid understanding of what is an acceptable risk assessment method to be used, in the event that the TC method on which they were trained is not applied. This may include developing common criteria to be in place for assessing the adequacy and appropriateness of stakeholders’ risk assessment methods.
  2. Develops a common understanding of risk tolerance (recommendation 6 c above), against which judgments can be made about acceptable levels of risk.
  3. Ensures that final approval of all risk assessments and ASAs be performed by a national committee to ensure consistency and comparability.
Moderate 8. Strategic Risk Assessments are one of the key processes currently conducted by the Aviation Security Program. To optimize their value, a plan should be developed and implemented to ensure their regular periodic conduct. The plan should be flexible so as to be responsive to emerging or unanticipated situations.
Moderate 9. The Security Programs Support Directorate should implement its plan to improve its intelligence-related threat and risk assessment process by formalizing the ongoing management of threat and risk information. This will help ensure a more systemic and consistent approach to analyzing, sharing and escalating threat and risk information.

2.4 Controls

At its broadest definition, control is defined as any action taken by management to enhance the likelihood that established objectives and goals will be achieved. An organization’s system of internal control (i.e. management control framework) is comprised of its resources (including people), culture, systems and processes that collectively help to mitigate risk towards the achievement of objectives. Controls can be preventive, detective or corrective and they can be both formal and informal in their nature. Often considered to be merely ‘constraints’ on an operation (i.e., prohibitions), controls are also enablers directly supporting the functions needed to achieve an organization’s objectives.

In the context of AvSec, which is a regionally distributed operational model with various parties involved in the operation, the importance of clarity and consistency of the controls cannot be overstated. Similarly, in light of TC’s need to balance security with efficiency and Canadian values, the efficiency and usefulness of controls and the processes that are enabled by them is also an important consideration.

Given this, the audit examined five areas of control:

  • Control environment;
  • People and skills;
  • Regulatory Oversight & Enforcement Protocols;
  • Change Management; and,
  • Incident Management.

2.4.1. Control Environment

Supporting the overall direction and management of the operation is the general control environment, characterized in large measure by the "tone at the top", employee engagement and a commitment to a healthy and respectful work environment. More than merely "informal" practices, they directly contribute to the strength of the overall control framework of an operation. When strong and constructive, they reinforce conscientious behaviour, accountability and compliance. When the environment is hostile, negative or mistrustful, levels of care are diminished with direct and negative impacts on the more formal aspects of the control framework.

In recognition of the importance of this aspect of the control framework, the audit examined the culture, philosophy and tone at the top in relation to AvSec, as well as the means by which values, ethics and standards of behaviour are communicated.

The modernized AvSec oversight framework, which will specify TC’s oversight philosophy, is still under development and needs to be finalized as an important foundation for inspectorate activities.

As noted under Leadership and Functional Direction and under Regulatory Oversight and Enforcement Protocols, AvSec Operations has not yet finalized its oversight framework. We understand that this framework will articulate the inspection and enforcement philosophy, which will help to set the ‘tone at the top’ from an oversight perspective. As such, it will lay the foundation for the operational activities of the inspectorate and will underpin the relationship between TC and its stakeholders. As has also been noted, the audit identified that at present, the cadre of inspectors still do not share a common approach or philosophy in relation to their activities. Variable approaches from “policing” to collaborative working relationships with stakeholders still persist, providing further examples of operational inconsistency. Thus, as has been noted earlier, the prompt finalization and dissemination of the oversight framework is critical to address these issues.

While progress has been made, the informal work environment in some locations continues to be negative. This, coupled with noted concerns around functional direction, may undermine operational effectiveness of the regulatory oversight function.

A negative and mistrustful environment has historically characterized the operational history of AvSec. These issues have been prevalent for some time. The audit sought to determine the degree to which such issues had been remedied. The environment was generally found to be appropriate with considerable effort being made by management to remedy the situation. However, the issues still persist in some areas. The source of the current condition is likely a combination of insufficient local management and a lack of accountability on the part of staff for their own behaviour. It is exacerbated by weaknesses raised elsewhere in this report in relation to functional direction. Thus, while a commitment has been made to remedy these isolated situations, sustained attention is required.

Effort has been made to reinforce the importance of values and ethics in the AvSec environment although the codification of expectations has been delayed pending broader departmental efforts in this regard.

Any regulatory operation is inherently susceptible to the risk of conflict of interest and regulatory capture (i.e., the risk that regulators will lose their objectivity in dealing with stakeholders). Key controls to prevent and detect this risk include the specific codification of acceptable standards of behaviour and the existence of confidential channels for reporting when improprieties are suspected. The audit examined these practices and concluded that while the code of conduct for AvSec inspectors has not been finalized, delays are attributable to factors beyond its control.

AvSec was initially engaged in a multi-modal initiative to develop a code of conduct for all security inspectors. The approach being used was reviewed and found to be comprehensive and appropriate. Strengths that were noted included the extensive consultation with inspectors and the explicit discussion and consideration of various ethical scenarios, specific to a regulatory compliance function. The progress on this initiative has been delayed due to a broader effort to develop a TC-wide code of conduct. While an integrated approach is important, the identification of security-specific expectations will also be important to ensure that the code is meaningful and relevant in the context of AvSec.

The audit also examined the degree to which appropriate mechanisms of disclosure are available for use when suspected improprieties are identified. Interviews held across all regions revealed that although inspectors were generally aware of some disclosure mechanisms, they were uncertain as to the details. More effort is needed to communicate and reinforce these communication channels.

2.4.2. People and Skills

As the aviation security regime evolves, so too must the required competencies and skills of TC personnel. TC’s policy framework re-defines the department’s relationship with stakeholders, moving towards one of partnership while at the same time making stakeholders accountable for systematically and proactively managing risks to their aviation security activities. This change will require a shift in the culture and philosophy of inspectors. Effective management of people and their performance during times of change is critical to ensure sustained improvements at the operational level, without which TC will not be able to effectively and efficiently track and manage the uptake of the changes – whether operational or cultural. Supervisors are key ‘controls’ in this regard.

In light of these imperatives, the audit examined the processes in place to establish, communicate and build competencies for both inspectors and their supervisors. It also examined the mechanisms by which training was planned and delivered to address competency requirements.

An updated, comprehensive and appropriately targeted TSI Developmental Program exists and is anchored on the core competencies that will be required for inspectors under the new regime.

In May 2010, the Ontario region drafted an updated developmental program in consultation with regional HR and all modes. National consultation with Regional Directors of Security and HR in regions and HQ also underpinned the effort. The goal of this program is to recruit and develop a sustainable and appropriately skilled TSI community that is able to readily respond to events and deliver quality regulatory oversight activities to stakeholders and the travelling public. The TSI Developmental program does not provide actual training but rather lays the foundations for making recruitment, training and coaching decisions regarding security inspectors. One of the key components of this program is an updated competency framework for TSIs.

The adoption of the TSI competency framework on a national level will be important to reinforce clear functional direction in relation to the inspectorate.

Core competencies for AvSec managers and supervisors need to be developed and rolled out nationally.

Specific strategies are being pursued to ensure appropriate cultural and technical competencies exist among the inspectorate. The same focus needs to be applied to supervisory and managerial competencies. The importance of the role of the immediate supervisor and manager cannot be over-stated, particularly in times of change and in areas where labour relations issues have historically characterized the program. Supervision is a key control for change management and for ensuring national operational consistency. The absence of a formal focus on managerial skills exposes the program to risk.

Formal training and developmental programs exist for inspectors although some areas of improvement remain outstanding.

A formal TSI training program currently exists for inspectors, the completion of which is tied to the provision of their inspector credentials. In addition to this standard curriculum, which is maintained by Security Programs Support, Recurrent Training for inspectors is being designed for national deployment. Similarly, other specific courses are slated for design and update, as part of a broader initiative to update the TSI learning continuum.

The audit found that coordination on matters of training between AvSec Operations and Security Programs Support requires reinforcement to ensure that all key players are aligned on matters of professional development. At present, no service level agreement between the two organizations exists for the provision of training services. Issues of coordination and consultation need to be improved so that the two organizations are aligned and effectively collaborate to support the learning and development of AvSec inspectors.

In addition, there is no formal process to update inspectors regarding program changes made during an extended leave of absence (i.e., sick leave, parental leave, maternity leave, assignments, etc). No central list is kept of regulatory and program changes that have occurred that would allow inspectors to identify those regulations they need to review for the period they were absent.

2.4.3. Change Management

Since September 11, 2001, when the program was greatly expanded, major changes have occurred. Changes have been required and initiated to respond to internal requirements as well as in response to external threats and various external reviews. The magnitude of these changes coupled with the inherent reactivity of such an operational program makes the management of change even more challenging. However, these same factors also make the need to manage these changes that much more critical, particularly in light of the important role that the AvSec program has in the overall security of the transportation system.

Overall, the audit found that although some areas, such as Air Cargo Security employ formal and structured change management methods, a more structured, systematic and integrated approach to change management is needed in AvSec Operations.

Although progress has been made, AvSec Operations needs to put in place a more structured and integrated approach to change management to ensure the successful implementation of the important and vast array of changes.

As has been outlined in the introduction of this report, in response to many previous reviews and studies, AvSec Operations is on a course of major change. These operational and cultural changes are taking place against the backdrop of other fundamental changes in AvSec, including the establishment of new directions, regulations and the ever-evolving nature of security threats and security programs. The regionally distributed model for operations and the inherent reactivity of the operation increases the complexity of this situation.

A Change Management Strategy for AvSec Operations was developed in 2009 within which the mandate for change management is formally articulated: “To facilitate [the move] from an inspection and enforcement only Regulatory Oversight [regime] to a more rigorous, clearly articulated, and well understood Continuously Improved Results Oriented and Risk-based Regulatory Oversight Framework”. The strategy is anchored on the 2007 Policy Framework, Moving Forward, which outlines the parameters of systems-based regulation and the imperative of culture change that is needed to be successful in this regard.

The audit examined governance, project management, communications and monitoring practices in relation to the change management strategy.

Governance: The audit first examined the governance processes in relation to the AvSec changes. Responsibility and accountability is clearly described, although governance can be strengthened. The Director of AvSec Operations, working in concert with the Regional Directors, retains ultimate responsibility for the change. Responsibility for the long term planning of the strategy, the development and execution of the implementation plan and the communication of changes to the national inspectorate has been delegated to a Project Manager. An AvSec Transition Steering Committee is to be struck in order to provide leadership and feedback with respect to the operationalization of the change strategy.

Project Management: The audit also examined the project management regime under which the changes are being operationalized. While a documented action plan exists, there is no overarching work breakdown structure for the Change Management Strategy, nor an integrated project management framework or plan. While many important pieces of change management have been instituted, more focus is needed to integrate the elements and their dependencies. The importance of this is underlined by the fact that the action plan is comprised of 39 sub-projects with numerous working groups in place to assist, advise or direct the changes. Many of the working groups and taskforces are interconnected and inter-dependent, yet no formal mapping of all players, their outputs and their issues has been completed.

Communications: The communication of operational changes was also examined. One of the primary objectives identified in the AvSec Operations project scope is the development of a communications strategy to ensure a consistent understanding by developing various communications products. Although no formal communications plan is currently in place, a wide range of training and awareness events has taken place, and based on inspector feedback, have been effective in communicating operational changes. Another form of communication is through the single point of contact on day-to day operational issues, which rests with the Compliance and Inspection unit within AvSec. They are responsible for communicating protocols and expectations in relation to compliance and enforcement. The audit identified however, that in some instances, the head of this unit is not sufficiently integrated into the change management process, limiting their ability to be a conduit of change information to the inspectorate.

Monitoring: Finally, we examined the processes for monitoring the change management initiatives. Although much activity is underway, and there is some informal monitoring of cultural changes, given the complexity and the magnitude of the changes being pursued, we remain concerned that there is insufficient monitoring of progress and results. For instance, the change management strategy envisions the conduct of a risk assessment; however, no integrated risk analysis has been undertaken. Also, a logic model for the change management strategy exists, but no performance measures have been defined for use in measuring the effectiveness of change. While monitoring may at first appear to be a luxury in an environment as fast paced and ever-changing as AvSec, it is precisely for these reasons that monitoring becomes so critical. Without it, the need for course corrections cannot easily be identified nor implemented which potentially undermines the very objectives of the continuous improvement model that is at the core of the Change Management strategy.

2.4.4. Regulatory Oversight and Enforcement Protocols

The rapid pace of change and growth, coupled with operational realities, has led to insufficient attention being placed on matters of regulatory oversight and enforcement protocols.

There are a number of inspectorate tools to support regulatory operations available to the AvSec inspectors and many of them are found in the Security & Emergency Preparedness Incident Reporting System (SEPIRS). It provides the Inspection and Enforcement (I&E) Manual, related legislation and some tools required to perform their duties, as well as other general information (i.e. Q&As, Operational Newsletters, and Minutes of Regional Teleconferences). The audit found however that although this functionality could provide a good resource for inspectors, its value is greatly diminished, as the content is neither complete nor kept current.

The I&E Manual is the main source of guidance to inspectors on their regulatory oversight and enforcement function. It provides the AvSec inspection and enforcement framework that includes governing principles and the responsibilities of the inspectors. The I&E Manual outlines the process for inspection preparation, conduct, compliance assessment, documentation and reporting. It also provides the Ministerial Delegation of Authority Schedule, Minimum Inspection Schedule (MIS) and Code of Conduct.

The audit found that the I&E Manual is neither current, nor does it provide sufficient guidance on how inspections should be carried out. Specific concerns include the following:

  • The I&E Manual does not include the complete set of required inspections such as the more recently developed "Restricted Area Identity Card (RAIC) audit" and the "Passenger Protect Program Verification". In the past, the HQ Compliance and Inspection unit within AvSec Ops updated the I&E Manual on a semi-annual basis but this activity has not been completed for some time, due to competing priorities.
  • While the I&E Manual prescribes what needs to be performed by inspectors, it does not specify how inspections should be carried out. For this, SEPIRS contains ‘checklists’ for use by inspectors. A review of the checklists found that they are simply lists of related legislation requirements and are not precise enough in their description of what should be examined and how the examination should take place. Through regional interviews, the audit found that in the absence of useful and user-friendly checklists, some regions have developed their own checklists and inspection programs. The use of different inspection checklists across regions creates inconsistency in the oversight function.

To address this, the AvSec Change Management unit is currently working to refine the oversight framework and a new working group has been tasked with developing mandatory prescriptive Standard Operating Procedures (SOPs) to replace the I&E Manual. Once written, the SOPs will include prescriptive checklists written as procedures or audit programs. For each aspect of inspection, the templates will identify the links to the key regulations, the expectation for performance and the specific steps to be taken by inspectors. This approach to documenting the inspection methodology is strongly supported by the audit as an important means of clarifying expectations and ensuring national consistency. The Change Management unit is also working on a new risk-based Inspection Program to replace the MIS.

The anticipated completion of the SOPs is March 2011, but given the early stage of the revision exercise and the competing priorities of working group members, management has expressed concern that the deadline may be ambitious. SOPs are critical to support consistent operations and any delays in finalizing these guidelines will continue to place the quality, consistency and integrity of the inspection process at risk.

In the event that regulatory clarification for inspectors is required, a formal request for Regulatory Advice and Interpretations exists. All regional inspector requests are directed to the HQ AvSec Compliance and Inspection unit. This group provides operational guidance and legislative or regulatory clarifications are addressed by Regulatory Affairs. Although this process is appropriate, the audit found that a formal process does not exist for logging the requests received by the Compliance and Inspection unit. This results in the risk that requests may not always be responded to or made available to all inspectors. Regional interviews identified that HQ does not always provide a response to requests, and when they do, it is not always timely.

Inconsistent inspection data entry, the inability to effectively retrieve the data, and a dormant AvSec QA program results in unreliable management information for use in program oversight.

The audit found that the combination of inadequate SEPIRS functionality and the lack of guidance regarding the data entry of inspection activities in SEPIRS have resulted in inconsistent inspection data entry, and thus, unreliable reporting of inspection information. While regional training was delivered on SEPIRS in 2006, audit interviews indicated that the training emphasized the operation of the system, not the data entry protocols.

The Security Expertise Branch within Security Programs Support Directorate had begun developing the Transportation Security Information System (TSIS) that would ultimately replace SEPIRS. The first phase of the TSIS project was a query and reporting module on existing SEPIRS data. This module exists on a test basis only and is available only to the Compliance and Inspection Unit. The TSIS project was recently cancelled, leaving the data collection and analysis issues in AvSec unaddressed. As a result, AvSec has committed to rolling out the query and reporting module, which will partially address these issues.

Faced with an inspection database that is not useful, regions maintain hard copy files and primarily rely on them for their inspection planning and analysis. As a result, HQ does not have ready access to the regional inspection information, restricting their ability to exercise national oversight of regional inspection activities.

2.4.5. Incident Management

The Emergency Preparedness (EP) Branch within the Security Program Support Directorate has overall functional authority for national EP activities:

  • emergency planning, exercises, response, training and awareness;
  • EP quality assurance;
  • managing and facilitating the use of the National TC Situation Centre (TCSC); and
  • participating in NATO EP activities and the National Critical Infrastructure Assurance Program (NCIAP).

As such, the audit evaluated AvSec processes and practices to ensure that AvSec incidents are appropriately reported up to the EP Branch within Security Program Support. In addition, the audit looked at EP processes and practices to ensure AvSec related security intelligence. In evaluating the processes and practices in place to respond and recover from incidents, the audit focussed on the existence of appropriate practices and protocols and observed that appropriate attention has been placed on the function relating to aviation incident management.

TC has a framework and procedures for the management of incidents both at HQ and in the Regions.

The audit examined the overall framework and tools available to TC personnel in identifying, responding to and escalating information in relation to aviation security incidents. It found that work done by the EP Branch within Security Programs Support was adequate and effective to set the broad direction and requirements in this area. Directives, roles, responsibilities and accountabilities, are clearly articulated, including the roles of the following decision-making bodies:

  • The Incident Management Team (IMT), which is responsible for analyzing and responding to incidents and in managing the emergency response of TC; and,
  • The executive Crisis Management Team (CMT), which is responsible for making high-level decisions and taking high-level actions to alleviate the emergency.

Numerous documents relating to incident management exist which describe TC and stakeholder obligations as well as the procedures and protocols to be applied in the event of an incident.

Complementing this, interviews indicated that relationships between EP and AvSec Operations are working well and continually improving. EP has also been focussing on collaboration with other relevant groups over the past few years, which appears to have resulted in improvements. As evidenced in the December 2009 incident, and noted under Risk Management, historically, TC has not been seen as a security player and thus has often not been fully engaged or informed by other government departments and agencies on such matters. Efforts to enhance the position of TC on security matters are being taken. One example is TC’s involvement in the Canadian Defence Red Switch Network (CDRSN), which allows the department to connect with key players via secure communications during incidents. Interviews suggested that during the recent Air Cargo incident, TC was more actively engaged and informed throughout the incident. The program also performs formal, post incident event “hot-washes” or post-mortems and develops recommendations to address any gaps or weaknesses experienced.

To support and complement the HQ role, functional direction is provided to regions via Policy Directives that set out regional Duty Inspectors’ responsibilities. These directives clearly instruct regional Duty Inspectors to report all incidents to the HQ Duty Inspector.

2.4.6. Conclusions and Impacts

As noted above, a clear, robust and consistent management control framework, reinforced by strong leadership and common direction is necessary to ensure a robust and nationally consistent regulatory oversight program. Without these core elements, there is a greater risk of inconsistent inspection activity and ineffective operational and individual performance management. The audit concluded that at present, insufficient clarity on operational protocols exists. This, coupled with the concerns raised on functional direction in the Governance section of this report, suggests that the program is exposed to high levels of risk of inconsistent oversight activity.

This is particularly germane given the extent of change facing the AvSec Program. While some considerable advances have been made in the management of change, the rigor of project and change management does not appear to be fully commensurate with the risk and complexity of the initiatives being pursued. The lack of an integrated project management framework limits the ability to manage and monitor the overall performance and risk associated with the strategy’s deployment. Constant change, even with strong communication and effective change management, increases the risk of inconsistent understanding and non-standardized operations.

2.4.7. Recommendations

The Assistant Deputy Minister, Safety and Security should ensure the following:

Significance Recommendation
Very Significant

10. Given the magnitude and complexity of change taking place within the Aviation Security Program, develop and implement an overarching change-management plan and integrated project management framework to:

  1. support clear communication
  2. ensure clear accountabilities
  3. align projects and priorities to effectively manage interdependences
  4. track, measure and report progress in this area
Very Significant 11. Complete and implement as soon as practical, National Standard Operating Procedures for all aspects of TC’s inspection and enforcement activities, including entry and management of inspection data. Standardized operating procedures are essential to ensuring a nationally-consistent application of the inspection and enforcement activities. In the interest of timely dissemination of the standard operating procedures, the Aviation Security Program should roll out new procedures as they are developed, with a deployment plan and communication strategy.
Significant 12. The query and reporting functionality for the SEPIRS database should be finalized and rolled out to provide management with the ability to analyze inspection data that supports timely decision-making and functional oversight
Significant

13. An employee performance management process should be fully implemented on a priority basis.

In addition:

  1. Core competencies should be developed for supervisors
  2. A national performance management mechanism should be established to ensure consistent and effective performance management.


3. Management Response and Action Plan (MRAP)

Each recommendation has been assigned a significance rating based on the rating scale found in Appendix E (has been removed and is available upon request).

3.1 Governance

The Assistant Deputy Minister, Safety and Security should ensure the following:

Significance Recommendation Management Action Plan with Expected Completion Date
Very Significant 1. Finalize the drafting of the National Civil Aviation Security Program (NCASP) with particular emphasis on the Aviation Security Oversight Framework component so as to provide staff and stakeholders with a clear and documented oversight philosophy, and lay the foundation for critical operational improvements and changes, including consistent implementation of national standard operating procedures.

Finalize the draft NCASP, in accordance with the following milestones* and timelines:

  • Complete the oversight philosophy component.
    Target Date: May 2011
  • Develop implementation and communication plan
    Target Date: Jul 2011
  • Seek cabinet support as required.
    Target Date: Fall 2011
  • Release document
    Target Date: Dec 2011

*milestones will change should document not be made public

Very Significant 2. Develop an overarching logic model to articulate the Program’s expected results chain (activities, outputs, immediate and intermediate outcomes) and define how TC contributes to the Program’s expected outcome, i.e., security of civil aviation. This will help ensure that all program elements, particularly those that are in development or undergoing change, are clearly and explicitly aligned to a common outcome and will facilitate performance tracking and reporting.

Complete overarching logic model in accordance with the following milestones and timelines:

  • Update current logic model to clearly articulate the results chain.
    Target Date: May 2011
  • Map how all program elements and change initiatives contribute to the program’s ultimate outcomes.
    Target Date: Sep 2011
  • Finalize and adopt logic model for all of the AvSec Directorate. Utilize the Program Management Office to ensure integration into project level planning and priorities.
    Target Date: Apr 2012
Significant 3. Develop a succession plan for the Director responsible for the implementation of the Operational Performance Framework (OPF) so as to ensure the successful completion and implementation of the OPF, a key component of the NCASP and critical to its successful implementation.

One year extension to incumbent Director sought.
Target Date: Apr 2011

Implement a migration plan to ensure effective transition from Director which will include appropriate documentation and knowledge transfer.
Target Date: Jan 2012

Significant 4. Implement a Quality Assurance function that regularly monitors regional delivery of the Aviation Security Program so as to ensure appropriate oversight is carried out, is consistent across regions and supports continuous improvement.

Develop a new quality management program with implementation options based on availability of resources.
Target Date: Sep 2011

Roll out the quality control program incrementally as new SOPs come online.
Target Date (for QC against all SOPs): Oct 2011

Design a program wide quality management function.
Target Date: Dec 2011

Review and revise the quality control program as required.

Moderate 5. Finalize the Service Level Agreement between the Aviation Security Program and the Security Program Support Directorate to ensure clear understanding of roles and responsibilities in relation to functions such as training and development of inspectors. Given the pace of change occurring in the Aviation Security Program, this is critical to ensuring training and development needs of Aviation Security personnel are met.

Signed March 2011.

3.2 Risk Management

Significance Recommendation Management Action Plan with Expected Completion Date
Very Significant

6. Develop an overarching Integrated Risk Management Framework (IRM) for the Aviation Security Program that:

  1. Specifies IRM objectives and expected results;
  2. Identifies and communicates roles and responsibilities of all key players in the risk management regime which in turn should form the basis for coordination and alignment on accountabilities, information sharing, and training;
  3. Specifies risk tolerance in the form of broad, yet meaningful indicators of risk impact such that all key players working in the various parts of the risk management regime have a common understanding of what constitutes a high or severe risk, boundaries of risk acceptability and what constitutes reasonable responses to unacceptable levels of risk;
  4. Provides, under the broad umbrella of objectives, approach and tolerance, links to specific risk assessment methods, which themselves should be reviewed to ensure their alignment to the broader framework.
  5. Contains a common taxonomy of AvSec Risks and Threats that would be applicable to and useful for the sub-elements of the risk management regime, including:
    1. strategic risk assessment and regulatory decision-making
    2. ASA approval
    3. inspection planning
    4. corporate and business planning
  6. Aligns to the TC Corporate Risk Profile and its risk management and monitoring strategy by outlining the means by which AvSec risk information will be escalated, as appropriate, to the corporate level, for consideration in the broader departmental risk oversight process.

Create an overarching IRM Framework in accordance with the following milestones and timelines:

a-b. Develop IRM policy and governance framework document that outlines the objectives and expected results, and program level direction with respect to risk management. Roles and responsibilities of all key players will be included as well as linkages to various risk practices currently in place. Roll out will be aligned to available resources and internal reallocation.
Target Date: Mar 2012

c-d. Provide program-level risk management guidance documentation which outlines the program’s overarching approach to risk and tools for risk management.
Target Date: Mar 2012

e. Ensure definitions and language (risk taxonomy) are aligned with Department wide work on horizontal and vertical risk alignment when available.
Target Date: Sep 2011 (will revise as required)

f. Review and revise framework, risk tolerance and risk assessment methods as required to align to Corporate and Strategies and Integration documentation when available.
Target Date: Sep 2011

Provide training and/or awareness, where required, for integrated risk management for all staff and management within the AvSec Directorate.
Target Date: Apr 2012

Significant

7. Should stakeholders have the flexibility to use their own risk assessment methods as the basis for their Aviation Security Assessments (ASAs); in order to support comparability of processes and outcomes, the ADM of Safety and Security should ensure that the Aviation Security Program:

  1. Requires that all inspectors reviewing the risk assessments of airports have a solid understanding of what is an acceptable risk assessment method to be used, in the event that the TC method on which they were trained is not applied. This may include developing common criteria to be in place for assessing the adequacy and appropriateness of stakeholders’ risk assessment methods.
  2. Develops a common understanding of risk tolerance (recommendation 6 c above), against which judgments can be made about acceptable levels of risk.
  3. Ensures that final approval of all risk assessments and ASAs be performed by a national committee to ensure consistency and comparability.
  1. Provide training and guidance on acceptable risk assessment methodology for inspectors after Gazette publication of regulations.
    Target Date: Jun 2012 – based on anticipated Gazette publication date OR target date will occur 6 months after actual publication date
  2. Communicate to the directorate the risk tolerance, updating as required to ensure alignment to Corporate and Strategies and Integration documentation when available.
    Target Date: Mar 2012
  3. Create a National Committee, chaired by the Director of AvSec Operations and supported by the Regional Directors, with a mandate and terms of reference which will aid in the consistency and the comparability of process and outcomes of security assessments.
    Target Date: Jun 2012 – based on anticipated Gazette publication date OR target date will occur 6 months after actual publication date
Moderate 8. Strategic Risk Assessments are one of the key processes currently conducted by the Aviation Security Program. To optimize their value, a plan should be developed and implemented to ensure their regular periodic conduct. The plan should be flexible so as to be responsive to emerging or unanticipated situations. Create and implement a cyclical strategic risk assessment plan.
Target Date: Aug 2011.
Moderate 9. The Security Programs Support Directorate should implement its plan to improve its intelligence-related threat and risk assessment process by formalizing the ongoing management of threat and risk information. This will help ensure a more systemic and consistent approach to analyzing, sharing and escalating threat and risk information.

Security Program Support (SPS) will formalize the management of threat and risk information through the use of the “Risk Audit Matrix” (RAM), in accordance with the following milestones and timelines:

  • Develop flexible implementation plan, present to Security DGs and seek approval for process.
    Target Date: May 2011
  • Organize TRA workshop(s) to address information gaps.
    Target Date: Sep 2011
  • Train employees who will be responsible to contribute
    Target Date: Jan 2012
  • Communicate matrix in awareness sessions within TC and within the intelligence community
    Target Date: Feb 2012

3.3 Control

Significance Recommendation Management Action Plan with Expected Completion Date
Very Significant

10. Given the magnitude and complexity of change taking place within the Aviation Security Program, develop and implement an overarching change-management plan and integrated project management framework to:

  1. support clear communication
  2. ensure clear accountabilities
  3. align projects and priorities to effectively manage interdependences
  4. track, measure and report progress in this area

Establish an AvSec Program Management Office to provide program management, in accordance with the following milestones and timelines:

a-d) Complete program management plan through the design to scale based on various costing options. Communicate design and options and seek feedback throughout process.
Target Date: Aug 2011

  • Implement plan based on available funding.
    Target Date: Oct 2011
  • Transition to full integration and self-sustainable.
    Target Date: Aug 2012

The AvSec Program Management Office will provide control and risk management along with governance as indicated in the Audit.

Very Significant 11. Complete and implement as soon as practical, National Standard Operating Procedures for all aspects of TC’s inspection and enforcement activities, including entry and management of inspection data. Standardized operating procedures are essential to ensuring a nationally-consistent application of the inspection and enforcement activities. In the interest of timely dissemination of the standard operating procedures, the Aviation Security Program should roll out new procedures as they are developed, with a deployment plan and communication strategy.

Develop the Standard Operating Procedures in accordance with the following milestones and timelines:

  • Develop
    Target Date: Jun 2011
  • Test
    Target Date: Sep 2011
  • Implement
    Target Date: Oct 2011
Significant 12. The query and reporting functionality for the SEPIRS database should be finalized and rolled out to provide management with the ability to analyze inspection data that supports timely decision-making and functional oversight.

Create the query and reporting function of SEPIRS in accordance with the following milestones and timelines:

  • Analysis/Design and Development
    Target Date: completed
  • Deploy and Train
    Target Date: June 2011
Significant

13. An employee performance management process should be fully implemented on a priority basis.

In addition:

  1. Core competencies should be developed for supervisors
  2. A national performance management mechanism should be established to ensure consistent and effective performance management.

Develop core competencies for supervisors and enhance the national performance management mechanism in accordance to the following milestones and timelines:

  1. Identify core competencies for the Security Management Team (TI-07, TI-08 and PM-06)
    Target Date: Sep 2012
  2. Deliver the new HR course(s) on Managing Employee Performance, for security supervisors. A post-course focus group(s) will be conducted to identify future specific needs.
    Target Date: Sep 2011
    Further identify enhancements to the existing HR performance management system.
    Target Date: Dec 2011

Appendix have been removed and are available upon request

  • Appendix A – Risks to the Aviation Security Program
  • Appendix B – Key Management Controls and Audit Criteria
  • Appendix C – Highlight Of Areas Included And Excluded From The Audit
  • Appendix D – Mapping of Audit Key Controls to Audit Report Themes
  • Appendix E – Description of Ratings


1 Institute of Internal Auditors, International Professional Practices Framework.

2 Air India Commission of Inquiry. Final Report. 2010. p.181.



The following document is available for downloading or viewing:

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: