1. Home
2. Air Transportation
3. Aviation Safety
4. Policy and Regulatory Services
5. Aviation Safety Intelligence
6. Safety Study on Risk Profiling the Air Taxi Sector in Canada

Safety Study on Risk Profiling the Air Taxi Sector in Canada

Report to Management

September 2007
Civil Aviation

• Executive Summary
• Introduction
  • Risk Management and Aviation Safety
  • Air Taxi Sector
  • Safety of Air Taxi Operations (SATOPS) Task Force
  • Hazards and Risks in Complex Systems
    • Domino Model (Sequential)
    • Reason’s Model of Organizational Accidents
    • Rasmussen’s Risk Management Framework (System)
  • Method
    • Study Design
• Results
  • Trend Analysis of Accident Data
  • Effect of the SATOPS Project
  • AGEMS Analysis of Accident Data
• Accident Mapping - ACCIMAP
  • Inspector Interviews and Questionnaires
• Discussion
• Conclusion
• Recommendations
• Appendix A
  • Levels of Performance
  • The Analytical Process

Executive Summary

Safety Study on Risk Profiling the Air Taxi Sector in Canada

Due to a number of serious accidents over a short period, involving small aircraft in British Columbia (B.C.), Transport Canada Civil Aviation managers directed staff to conduct a study into the safety of the air taxi sector in the Pacific Region. The purposes of the study were to:

• Examine the historical air taxi accident pattern and to the extent possible, determine whether recent accidents constituted a change in the risk level for the flying public and workers in the air taxi sector;
• Evaluate the effectiveness of the Safety of Air Taxi Operations (SATOPS) Task Force, the last significant means of review and regulatory intervention in the air taxi sector;
• Develop a risk profile for the Pacific Region air taxi sector;
• Identify potential risk-reduction strategies where appropriate.

The study team gathered considerable information on the air taxi sector, examined historical accident rate patterns, reviewed the SATOPS final report, and analyzed approximately 125 air taxi accidents occurring between 1996 and 2006 in the Pacific Region. The team tested analytic tools, including an adaptation of James Reason’s Generic Error Modeling System (AGEMS), Rasmussen’s Risk Management Framework and accident mapping (ACCIMAP) techniques; the team also administered a series of questionnaires and interviews.

The team found that the 2006 spike in fatal accidents is probably not indicative of a general deterioration of air taxi safety. The nature of accident rate statistics is such that a trend can only be confirmed after the trend has become established. Currently, accident rates appear to be returning to historical patterns, indicating that the 2006 spike was anomalous. The analysis of accidents and survey/interview information was used to develop an understanding of the hazards and conditions which constitute a risk for air taxi passengers and employees. The hazards clustered around four themes. Three are geographic and the fourth is related to equipment failure: mountainous terrain, coastal terrain, weather and equipment.

The geographic hazards interact with other factors at different levels within the aviation system, such as low experience for pilots and self-dispatch practices. These risk factors are most appropriately dealt with through non-regulatory intervention, concentrating on giving pilots adequate guidance to assess available information and then to select appropriate options.

The tests of the AGEMS method and Rasmussen’s risk management framework showed that these methods can be used in studies of this type and yield a rich understanding of individual hazards, but more importantly, how hazards and conditions interact to create elevated risk levels. These methods are, however, time-consuming and require specialized knowledge of both operational and human/organizational factors to be used effectively. They should be adopted as standard approaches for combining operational and behavioural science expertise to understand risks and advance safety.

Introduction

A number of serious accidents involving small aircraft in British Columbia led to questions about the safety of the air taxi sector. The relatively short time between events caused concern and drew media attention. Questions were raised over whether the safety of air taxi operations in B.C. was deteriorating.

In mathematical terms, accidents are relatively rare events, so we are dealing with small numbers. This means that changes in the rate of occurrence in the short to medium term (up to 2 years) must be interpreted cautiously. By early summer 2006, the accident trend data was ambiguous. The increase in the number of accidents, particularly fatal accidents, was a cause for concern. This variation in the accident rate, however, was not sufficient to support a conclusion that a new trend was beginning. Trends are detectable only retrospectively. If the accident rate returned to historical levels, no trend would be established. If, on the other hand, accidents rates remained at the elevated rate of late 2005 and 2006, a trend would be established.

Senior management within Transport Canada (TC) decided that waiting for a trend to be confirmed or refuted was not an acceptable course of action. TC staff were directed to conduct a study into the safety of the air taxi sector.

The purposes of the study were to:

• Examine the historical air taxi accident pattern and to the extent possible, determine whether recent accidents constituted a change in the risk level for the flying public and workers in the air taxi sector;
• Evaluate the effectiveness of the SATOPS Task Force, the last significant means of review and regulatory intervention in the air taxi sector;
• Develop a risk profile for the Pacific Region air taxi sector;
• Identify potential risk-reduction strategies where appropriate.

Risk Management and Aviation Safety

Transport Canada, the aviation industry, and the Canadian public all have a stake in advancing aviation safety. The Transport Canada Civil Aviation (TCCA) mission—to develop and administer policies and regulations for the safest civil aviation system for Canada and Canadians, using a systems approach to managing risks¹ —clearly defines the regulator’s contribution to aviation safety. Other stakeholders share the goals, but contribute to achievement in different ways, depending on their roles as providers or consumers of air transportation.

“For the purposes of the Civil Aviation Program, and to bring our strategic goals into focus, Transport Canada Civil Aviation (TCCA) defines safety as the condition where risks are managed to acceptable levels² ”. Accomplishment of the TCCA mission therefore requires that risks be understood, so that they may be eliminated or reduced to acceptable levels. Understanding risks demands an overall understanding of the concept at a systems level, and of how particular hazards pose risks during particular operations. There is a need for a macro and a micro view of risk.

Civil Aviation Directive (CAD) 20 provides guidance for the exercise of due diligence regarding frequency of inspection. Risk indicators are meant to act as triggers for increased oversight activities. Currently, there is no systematic approach for the application of the following risk indicators:

• Financial change
• Labour difficulties
• Management practices
• Internal audit or quality assurance program
• Change in operational scope or additional authorities
• Changes in contracting for services
• High turnover in personnel
• Loss of key personnel
• Additions or changes to product line
• Poor accident or safety record
• Merger or takeover
• Regulatory record

Air Taxi Sector

Activity is diverse. The holder of a 703 air operator certificate may be engaged in one or more of a variety of operations within this sector. Scheduled air services, air courier operations, heli-skiing, some aspects of heli-logging, fire suppression activities, game hunting operations and sightseeing fall within the air taxi operations envelope.

One challenge in the focus of the study was frequent overlaps with activity outside of the sector. It is not uncommon for an operation to include aerial work (702) in conjunction with operations that fall within the 703 sector. For instance, heli-logging operations primarily involve aerial work, but may include the transport of individuals at times; that particular aspect of the activity falls within 703. Another example would be fire suppression activities alternating with the transport of fire crews.

Safety of Air Taxi Operations (SATOPS) Task Force

In 1996, following a TC review of accident data from 1990 through 1995, Transport Canada and the aviation industry collaborated on the Safety of Air Taxi Operations Task Force. The objective was to identify ways to improve the safety of the air taxi segment of the aviation industry and to address the outstanding Transportation Safety Board (TSB) recommendations affecting air taxi operations. The SATOPS Task Force used several information-gathering techniques:

1. Industry consultation meetings were held at industry-accessible locations across Canada;
2. Information on the SATOPS project, including a feedback function, was established on the Transport Canada Civil Aviation Web site;
3. A questionnaire was distributed at industry meetings;
4. A toll-free telephone “hotline” was established; and
5. A feedback document was distributed to industry.

Pacific Region industry consultation meetings were held in Vancouver, Campbell River, Terrace, Prince George and Kelowna. The final SATOPS report, containing 71 recommendations, served as a basis for evaluating SATOPS.

Hazards and Risks in Complex Systems

For over five decades, scientists have been devoting considerable effort to understanding and explaining how accidents happen (Rasmussen, 1997). Understanding systems is a pre-requisite for understanding accidents. Systems may be tightly coupled or loosely coupled. In a loosely-coupled system, events and conditions in one area have little or no impact on other areas. In a tightly-coupled system, a small change or problem in one area can have a significant impact elsewhere. A bad weather day at Pearson International Airport will inconvenience the entire country since the national air transport system is tightly coupled. Interactions between system components are becoming more complex. Technology is more advanced, and the knowledge required for the design, operation, maintenance and decommissioning of an advanced system is more demanding. To deal safely with advanced systems, our understanding of the nature of failures, particularly human error, has had to evolve.

Over the past few decades, three principal accident causation models have been used in accident investigation and analysis. The Domino model is a linear, event-based model. Reason’s organizational model takes a broader, epidemiological view, and Rasmussen’s Risk Management Framework is a system model. There are advantages and disadvantages to each model. However, as our understanding of the relationship between humans and technology in socio-technical systems has progressed, it is becoming evident that system models provide a truer representation of the world and help us begin to understand the nature of human-technology breakdowns in normal work contexts.

Domino Model (Sequential)

The Domino Model was published in 1931 by Heinrich and was the first generally used accident model that allowed a greater emphasis on human error (in Hollnagel, 2004). The Domino Model pictured accident causation as a series of dominos standing on end (Figure 1). Each falling domino would lead directly to the next. This model begins with the social or historical background of an individual. Factors at this level lead that individual to make an error. The error leads to an unsafe act or condition, which in turn leads to an accident and an injury.

This was a groundbreaking model at the time of its publication because it led investigators beyond the unsafe act and the error, to underlying conditions. However, the relationship between the various events are depicted as direct and linear, leaving the impression that there is only one error path to an accident, and one root cause for an accident.

Figure 1 : Heinrich’s Domino Model of Accidents

The simplicity of this model helped to make it popular, but it is this simplicity that limits its usefulness in investigations of accidents in complex socio-technical systems. This model cannot reflect the reality of complex systems.

Reason’s Model of Organizational Accidents

In the early 90s, Reason developed a theoretical framework to describe accidents in complex systems, and coined the term ‘‘organizational accident’’ (Reason, 1997). This was a significant progression in accident causation thinking and has guided accident investigation and analysis methods ever since. The framework describes accidents as the result of several causal factors, which combine to create a “trajectory of opportunity” (Reason, 1997) through multiple defences in a system (Figure 2). According to Reason, organizational accidents result from active and latent failures that occur within five layers of an organization - decision makers, line management, psychological preconditions, productive activities and defences.

Figure 2: Reason’s Model of Organizational Accidents

Despite significant efforts to identify and reduce active and latent failures through various approaches, safety programs and the like, Reason tells us that human error is inevitable. To mitigate the problem of errors, he suggested providing a variety of defences to intervene between the human actions and the adverse consequences that can result from these actions. (Reason, 1997). However, in a system designed with defences-in-depth, operators can violate individual defences without experiencing negative effects. If violating certain defences leads to better performance without immediate negative consequences, operators will likely continue to violate defences. Over time, the defences will slowly erode, increasing the risk of an accident. The creation of defences-in-depth, as advocated by Reason’s model, may delay the occurrence of an adverse event, but accidents will continue to happen in environments with apparent conflict between safety and productivity goals.

Managing risk by identifying potential sources of human error (deviation from prescriptive norms) and by putting in place multiple lines of defence against both active and latent failures, have not prevented several recent complex system accidents (Bhopal, Walkerton, NASA Columbia). One reason for this is that humans rarely perform work with strict adherence to prescribed rules or instructions. Procedures are often developed for a particular task in isolation from the work context. Performance requirements impose constraints on operators beyond what was considered (and available) when the task instructions were written. To get the job done, people work outside the defined rules. This is why studies of humans, even those working in high-risk, complex systems, have found that operators modify instructions and violate rules in ways that are quite rational given the actual workload and timing constraints (Dekker, 2006).

Additionally, the defence-in-depth strategy is problematic because the actual system state with respect to safety depends on the defences-in-depth being intact and operational. When individuals violate safety defences in the context of normal work, the margin of safety in a system decreases. However, defences are controlled by different parts of an organization. As individuals in one area breach defences to locally optimize their performance, latent failures are introduced that affect the margin of safety available to other areas.

Rasmussen’s Risk Management Framework (System)

Rasmussen’s framework for modeling risk management is a system framework with two components. The first is a structural hierarchy describing the actors—individuals and organizations—in a system. The second considers the dynamics of the system as it migrates towards the boundary of safety. An understanding of both of these components is required to model how and why accidents occur.

Structure

Socio-technical systems are designed to produce a product or service while managing risk (Figure 3). Work-level factors are conditions associated with the particular process being controlled (e.g., nuclear power plant, healthcare system, commercial aviation). Activities of the individual staff members who interact directly with the process being controlled (e.g., control room operators, front line hospital staff, pilots) are at the Staff level. Factors at the Management level are related to supervision of operational staff. Company-level factors include the activities of the company as a whole. The next level represents the activities of the regulators or professional associations that are responsible for constraining the activities of companies in that particular sector. Professional associations in this context are those that exert regulatory authority over the profession, such as the College of Physicians and Surgeons or the law society of a province. Factors at the Government level are related to the activities of government, both public servants and elected officials, who are responsible for setting public policy.

When a system functions, decisions at higher levels of the system propagate down the hierarchy. Simultaneously, information about the current state of affairs should percolate up the hierarchy. Two-way flows of information, direction and feedback, are critical to the successful functioning of the system. If instructions from above are not formulated or carried out at the lower levels, system states they intend to safeguard will not be protected. If information from below is not collected or conveyed to decision makers at higher levels, decisions cannot reflect the available capacity and limitations of the system or the constraints facing the system. The result is that the system can become unstable and start to lose control of the hazardous process that it is intended to control. From this perspective, safety can be viewed as an emergent product of a complex socio-technical system.

Threats to safety usually result from a loss of control caused by a lack of vertical integration, or mismatches between levels of a complex socio-technical system, not just from deficiencies at any one level alone. All layers play critical, albeit different, roles in maintaining safety. A lack of vertical integration is frequently caused, in part, by a lack of feedback between levels of a complex system. Actors at each level cannot see how their decisions interact with those made by actors at other levels, so the threats to safety are not obvious before an accident occurs because no one has a global view of the entire system.

Figure 3: The socio-technical system involved in risk management

Dynamics

The second component of the framework, shown in Figure 4, considers the dynamic forces that can cause a complex socio-technical system to modify its structure and behaviour over time. Financial pressures that result in a cost gradient push the people in the system to reduce costs. Psychological pressures result in an effort gradient pushing people in the system to work in a more mentally or physically efficient manner. Although this effort gradient is sometimes interpreted as laziness, especially after an accident, this is usually an over-simplification rooted in the normal human tendency to attribute actions to personality factors and underestimate the influence of local conditions on behaviour. When all appears well and there are no accidents, the effort gradient will be viewed as positive, encouraging people to seek out new, better ways of getting the job done. We are all familiar with the maxim ‘‘do more with less’’ as an illustration of the benefits of an effort gradient. This process of trial and innovation can be particularly important when people are being required to take on more responsibilities with fewer resources.

Figure 4: Rasmussen’s Dynamic Socio-Technical System

As a result of cost and effort gradients, work practices will be subject to an exploratory but systematic change over time. Financial and psychological forces inevitably lead to people finding the most economic ways of performing their job. Moreover, the modification of work practices can occur at several levels of a complex socio-technical system simultaneously. Over time, this migration causes people to cross the official boundary of work practices, shown on the near left in Figure 4. People are forced to deviate from procedures and cut corners because they are responding to requests or demands to be more cost-effective. As a result, the system’s defences-in-depth degrade and erode gradually over time, not all at once.

One might think that a lack of procedural compliance and the resulting degradation in safety would raise an immediate warning flag, but this does not happen for two reasons. First, the migration in work practices is required to get the job done, given the stresses that the system is undergoing. That is why “work to rule” campaigns can cause complex socio-technical systems to come to a grinding halt. Second, the migration in work practices does not usually have any visible, immediate negative impact. The threats to safety are not obvious before an accident because the violation of procedures does not immediately lead to catastrophe. At each level in the hierarchy, people are working hard, striving to respond to cost-effectiveness measures, but they do not see how their decisions interact with those made by other actors at different levels of the system. Yet, the sum total of these uncoordinated attempts at adapting to environmental stressors is slowly but surely building the conditions for an accident (Rasmussen, 1997).

As a result, the migration of work practices continues. People try harder and harder to work in more efficient ways, and with each new innovation, they are coming closer and closer to the real boundary of safety. The boundary, however, is usually invisible; people do not know whether the system as a whole is close to or far away from disaster. Migrations from official work practices can persist and evolve for years without mishaps until the real safety boundary is reached. After an accident, workers may wonder what happened because they did not do anything differently than they had been doing in the recent past. In other words, accidents in complex socio-technical systems do not usually occur because of an unusual action or an entirely new, one-time threat to safety. Instead, they result from a combination of a systematically induced migration in work practices and an odd event that winds up revealing the degradation in safety that had been occurring all the while.

Rasmussen’s framework is a systems approach in that it focuses on the vertical interaction between the levels through directions/instructions/decisions (flowing down) and feedback (flowing up). This is important in the air taxi sector, since accidents often result from a series of decisions made and communicated by actors at different vertical levels.

Rasmussen’s framework helps us to visualize and understand the correlation between factors at different vertical levels, enabling the study of issues that are only visible by looking at the system as a whole. This can then be further generalized into a pro-active risk management framework, where issues affecting the system at large are addressed, as opposed to focusing on specific issues associated with individual errors taken out of the context in which they occurred.

The Rasmussen framework also addresses the dynamic nature of the air taxi sector. In Figure 4, Rasmussen models the “drift” effect in organizations under economic and workload pressures. Rasmussen presents the idea of the two pressure gradients, production and effort, that push actors into higher risk levels through gradual shifts in practice. This migration, associated with adapting to circumstances, does not necessarily present actors with negative feedback immediately after each “drift”. In fact, the immediate feedback may be very positive, resulting in higher production and reward.

Method

Study Design

Statistical Evaluation: Accident data for Canada and British Columbia from 1996 to 2006 were examined and analyzed to identify patterns of accidents over the ten-year period and to evaluate the possibility that a trend towards more fatal accidents was developing.

SATOPS Evaluation: A thorough review of the SATOPS final report and responses to the recommendations was conducted. At the time of the SATOPS, TCCA’s risk management program had not been formalized, so the terms ‘‘risk’’ and ‘‘hazard’’ were not commonly used. To allow an analysis of SATOPS in terms of hazards and risk, concepts that are a part of the current language of aviation safety, the study team had to infer the hazards associated with the various safety issues identified. During the analysis of accidents (described below), lists of hazards were generated, and the hazards identified during SATOPS were compared to this list to determine whether they still applied.

Accident Analysis: By analyzing accident investigation data, it is possible to identify the unsafe acts which put people in harm’s way and factors which led to those acts. When the nature and origins of the unsafe acts are understood, it is often possible to develop countermeasures to prevent or reduce the likelihood of recurrence. An adaptation of Reason’s AGEMS model (see Appendix A for details) was used to analyze errors or unsafe acts which led to accidents. This approach requires identifying the sequence of events and acts leading up to an accident and classifying each unsafe act as a type of error. Contributing factors, or antecedents, are then identified. The aim of this analysis is not to determine where people went wrong, but to establish how people’s assessments and actions made sense at the time, given the circumstances that surrounded them.

One hundred and twenty-five accident reports were analyzed. The TSB Aviation Safety Information System (ASIS), the National Aviation Company Information System (NACIS) and the Flight 2005 database were used to identify all air taxi accidents in the Pacific Region during the ten year period extending from 1996 to 2006.

Accident Mapping: Following the AGEMS analysis, accident mapping (ACCIMAP) techniques developed by Rasmussen were used to graphically display how hazards and decisions at different levels of the Rasmussen risk hierarchy contributed to accidents. The ACCIMAP technique was applied to the TSB investigation reports to portray accident processes. Generic ACCIMAPs and aggregations of individual ACCIMAPs were generated to reveal all relevant, alternative flow paths that may have potentially led to the critical event, as well as related prevention and mitigation strategies in place (Svedung and Rasmussen, 2002).

Inspector Interviews and Questionnaires: Aviation safety has progressed to a very high standard because civil aviation regulatory authorities have used accidents and incidents to learn more about how mishaps occur. Modern accident causation models and theories consider accidents to be an outcome of a process rather than a discrete event. If the process is well understood, then changes can be introduced to affect the untoward outcome. This approach is essentially reactive, but has elevated the level of aviation safety to the very high standard we now enjoy. TCCA and other authorities are now trying to become more proactive. To do this, we must learn how to use available information to predict untoward events, so that appropriate preventive measures can be applied. The next step in the study was to explore the possibility that information available to TCCA staff could be used to predict potential accidents by identifying elevated risk levels associated with certificate holders.

Fifty-three air taxi operators who had had at least one accident and 33 air taxi operators who had not had an accident were selected for further analysis. Principal inspectors (operations and maintenance) completed three questionnaires on the operators. The purpose of this was to determine whether inspectors could discriminate between operators with respect to a number of factors believed to be related to risk, and whether these factors, as perceived by inspectors, are related to the probability that an operator will have an accident.

Three questionnaires were used and three open-ended interview questions were asked of inspectors. One hundred and twenty-six interviews were conducted. Ninety-two accidents were addressed, representing 47 air taxi operators.

Seriousness Scale Questionnaire

The Seriousness Scale questionnaire consisted of five Likert³ scale questions and one question in which inspectors were asked to select the most appropriate category for the occurrence. The questions were:

1. How serious was this accident in terms of the overall safety of operations for the company involved (not serious at all to extremely serious)?
2. This accident was an aberration; it was completely unexpected (strongly disagree to strongly agree).
3. The company will learn from this accident and reform their organization (extremely likely to not likely at all).
4. Please rate the probability of recurrence (not likely at all to extremely likely).
5. Please rate the severity of the accident (negligible to catastrophic).
6. What was the main event category for this accident (Maintenance, Flight Operations, Air Navigation Services and Airspace, Aerodrome Safety, Other)?

CAD 20 Accident and Current Questionnaire

Civil Aviation Directive (CAD) 20 provides guidance to managers and inspectors in assessing the risk associated with particular operators. CAD 20 lists nine risk-associated factors, or risk indicators. The list was produced based on logical considerations and has never been empirically evaluated. A questionnaire was administered to principal operations and maintenance inspectors, consisting of nine ‘‘yes/no/unknown’’ questions reflecting risks to the operator, and eight ‘‘yes/no/unknown’’ questions reflecting risks to the regulator.

Risks to the operator:

1. Contracted Services (Yes/No/Unknown).
2. Merger/Sales (Yes/No/Unknown).
3. Labour Problems (Yes/No/Unknown).
4. Management Problems (Yes/No/Unknown).
5. Key Person Changes (Yes/No/Unknown).
6. Staff Turnover (Yes/No/Unknown).
7. Quality Assurance Program Effective (Yes/No/Unknown).
8. Scope – Change in Fleet/Facilities (Yes/No/Unknown).
9. Product Line – Change in Routes/Services (Yes/No/Unknown).

Risks to the regulator:

10. Contracted Services (Yes/No/Unknown).
11. Fiscal Pressure (Yes/No/Unknown).
12. Labour Problems (Yes/No/Unknown).
13. Management Problems (Yes/No/Unknown).
14. Key Person Changes (Yes/No/Unknown).
15. Staff Turnover (Yes/No/Unknown).
16. Quality Assurance Program Effective (Yes/No/Unknown).
17. Scope – Change in Fleet/Facilities (Yes/No/Unknown).
18. Product Line – Change in Routes/Services (Yes/No/Unknown).

Interview

The interview consisted of three open-ended questions:

1. Is there anything else you would like to add about your certificate holder? Strengths? Weaknesses? Opportunities? Threats (hazards and risks)?
2. Is there anything else you would like to add about the industry? Strengths? Weaknesses? Opportunities? Threats (hazards and risks)?
3. Is there anything else you would like to add about the regulator? Strengths? Weaknesses? Opportunities? Threats (hazards and risks)?
   

Results

Trend Analysis of Accident Data

Figure 5 was taken from the SATOPS final report and illustrates the proportion of air taxi accidents relative to the total number of accidents in commercial operations from 1990 to 1995. The first column in each year represents the total number of accidents in all commercial operations. The second column represents the number of helicopter and fixed-wing air taxi accidents. The third column represents the number of helicopter accidents (included in the second column) for comparison purposes. In 1990, 82% of all commercial aircraft accidents involved air taxi aircraft. In subsequent years, air taxi aircraft represented 83% in 1991, 69% in 1992, 72% in 1993, and 79% in 1994 and 1995.

Figure 5: SATOPS—Commercial Accidents 1990-1995

NOTE: Before 1995, the TSB was not categorizing accidents in accordance with the Canadian Aviation Regulations (CARs). This explains the discrepancy in the number of accidents in 1995 between the pre-SATOPS graph and the post-SATOPS graph (Figure 6).

Following 1995, the number of accidents decreased considerably for about five years and has remained relatively constant since 2000. In 2005, the total number of air taxi accidents was 56, which is less than half of the 1995 total of 118. Since 1998, there have been fewer than eight fatal air taxi accidents per year. The decrease in the number of fatal accidents per year since 1996 is quite propitious. It is unlikely that the decline is random fluctuation, since it has continued for several years. There appears to be a systematic effect, which indicates that something has changed in the air taxi sector.

The number of helicopter accidents that occurred during air taxi operations has also declined since 1995, although not as much as the fixed-wing or overall numbers. Figure 6, below, illustrates the declines graphically.

Figure 6: Commercial and Air Taxi Accidents 1995-2006

The worst year in B.C. was 1996, when 32 air taxi accidents happened (Figure 7). The lowest number of air taxi accidents—eight—was recorded in 2003. In 1999, and in the period from 2001 to 2004, there was only one fatal air taxi accident per year. However, in 2005, there were three fatal accidents in B.C., accounting for 5 fatalities and 4 missing (presumed dead). This three-fold increase in fatal accidents draws our attention, but it is too early to determine whether this is the beginning of a new trend. Two or three more years of comparative data will be required in order to tell whether 2005 is an anomalous year, or if the annual rate of fatal accidents in B.C. is on the rise.

Figure 7: Air Taxi Operations in British Columbia - Accidents and Fatal Accidents

On a national basis, the fatal accident record over the period from 2003 to 2006 is steady. Figure 8, below, shows the annual fatal accident rate for Canada and B.C. for comparison purposes. A rise can be seen in the 2000 B.C. fatal accident rate, comparable to the 2005 rate, but the rate then returned to a low level for the next three years (2001-2004).

Figure 8: Air Taxi Operations in Canada - Fatal Accidents and Fatalities

Note: The data from 1990 to 1994 were taken from the TSB database and were not re-categorized according to the CARs; therefore, these data may contain errors.

Figure 9 allows us to compare the number of air taxi accidents in Canada with the number of aircraft registered and operating as an air taxi in Canada by year. The number of aircraft registered increased steadily from 1990 to 2002, to reach 3 694, but it has been decreasing since 2003. As of June 7, 2006, there were 2 137 aircraft registered. Using the number of registered aircraft as an indicator of activity, we can observe that despite an increase in activity between 1997 and 2002, the number of accidents decreased during this time period.

Figure 9: Air Taxi Operations in Canada - Accidents by registered aircraft operating as an air taxi

Figure 10 shows the number of aircraft registered in Canada and operating as an air taxi, broken down by single-engined versus multi-engined. The number of single-engined aircraft registered increased steadily from 1990 until 2002, but has declined since then. The number of multi-engined aircraft also grew from 1990 until 2002, but at a slower rate than single-engined aircraft.

Figure 10: Air Taxi Operations in Canada - Aircraft registered and operating as an air taxi (single-engined and multi-engined)

Figure 11 demonstrates the number of aircraft involved in accidents in Canada during air taxi operations, broken down by multi-engined versus single-engined. From 2000 on, we can see that the number of accidents has been quite constant, and considerably lower than the previous 10 years. The decrease coincides with the implementation of the SATOPS recommendations.

Figure 11: Air Taxi Operations in Canada - Accidents (single-engined and multi- engined)

Figure 12 demonstrates the number of single-engined aircraft accidents in relation to the number of registered single-engined aircraft involved in air taxi operations. Despite an increase in the number of registered single-engined aircraft (an indicator of activity), there has been a decrease in the number of single-engined accidents in air taxi operations.

Figure 12: Air Taxi Operations in Canada – Accidents involving single-engined aircraft registered and operating as an air taxi

Despite an increase in the number of registered multi-engined aircraft (an indicator of activity), there has been a decrease in the number of multi-engined accidents in air taxi operations. This is shown graphically in Figure 13.

Figure 13: Air Taxi Operations in Canada – Accidents involving multi-engined aircraft registered and operating as an air taxi

While historical review does not support any cause and effect conclusions, a number of important facts do come to our attention. Air taxi activity, measured by the number of aircraft registered, increased steadily from 1990 until about 2003. From 1996 to 2003, the number of accidents and fatal accidents declined. Since 2003, the rates have been steady. There is no doubt, based on these numbers, that there is a decreased risk to the flying public and to those involved in the aviation industry.

Although it is the most reliable indicator we have of activity in the air taxi sector, the number of aircraft registered and operated under air taxi operating certificates has limitations that must be noted. The air taxi fleet grew over a period of approximately 12 years, and it also evolved. More multi-engined aircraft were acquired. There was also an increase in the number of turbine-powered aircraft registered by air taxi operators. These aircraft are more capable, more reliable, and less demanding in terms of maintenance, than aircraft with reciprocating engines. It is possible, therefore, that while the size of the fleet has contracted, the activity level, in terms of aircraft movements, passenger miles flown, cargo transported, or even hours flown, has remained constant or grown in the past two years. It is a known fact, for instance, that Northern Alberta, particularly the Fort McMurray area, has seen exponential growth in aircraft movements during this period.

During the period from 1996 to 2006, there were a number of changes that likely contributed. In 1996, Transport Canada changed the type and style of regulation. The Canadian Aviation Regulations replaced the Air Navigation Orders. Regulations became less prescriptive and more performance-based. In 1997, TC began implementation of the more than 70 recommendations of the SATOPS Task Force. During this period, as well, more pilots were graduating from provincially-regulated colleges or, in the case of Quebec, from CEGEP programs. New arrivals in the commercial pilot community may be different, in areas related to aviation safety, than the population of pilots that preceded them. It is impossible to say whether such factors contributed to risk reduction, and if so, to what degree.

Effect of the SATOPS Project

The above trend analysis provides unequivocal evidence that accident rates declined in the wake of SATOPS. Although informative, a simple comparison of accident frequencies is not appropriate for determining the effect of the SATOPS recommendations. Given that, statistically speaking, accidents are rare events, accident counts are a poor criterion for evaluating complex interventions. Additionally, we do not have sufficient control over potentially confounding variables to conclude cause and effect relationships.

After the SATOPS Task Force ended in 1997, TCCA tracked the recommendations to verify that they were followed up. Sixty-one of the 71 recommendations have been implemented; a few were overtaken by events and became unnecessary or redundant. TCCA tracking methods used in the late 1990s were very good for ensuring that recommendations and decisions were followed up, but measurement of the effectiveness of initiatives was weaker. This is a very common condition in government and industry. Therefore, a retrospective qualitative analysis was conducted.

In Flight 2010, TCCA defined safety as the condition where risks are managed to acceptable levels. An argument was presented earlier that counting accidents is not a good measure of safety, since accident counts and rates tell us only about outcomes and very little about the level of risk involved. The study team decided to try to track risk-related variables over time to determine whether safety benefits could be attributed to SATOPS.

The study team carefully reviewed the SATOPS final report. Each SATOPS recommendation was analyzed to identify the underlying hazard or hazards addressed. The team members were able to reach consensus fairly easily. The hazards identified in the SATOPS review were then compared to the hazards identified in the current study.

Six hazard and risk factors isolated in the SATOPS study were not found in the current study:

1. Lack of relevant statistics;
2. Lengthy certification process for parts and supplementary type certificates (STCs);
3. Inconsistency of in-house safety programs;
4. Lack of helmet use among rotorcraft pilots;
5. Lack of navigation charts; and
6. Unannounced changes in fuel storage.

It is possible that these hazard and risk factors were mitigated through the implementation of the SATOPS recommendations.

On the basis of hazard and risk factors substantiated in the current analysis, seven isolated in the SATOPS study were found in the current study:

1. Inadequate management of pilot resources;
2. Competitive pressure;
3. VHF frequency congestion;
4. Risks of itinerant pilots;
5. Lack of aerodrome support;
6. Inconsistent communications equipment; and
7. Difficulty in obtaining weather information.

It is possible that these hazard and risk factors were reduced in terms of probability, severity and exposure through the SATOPS recommendations. However, it is also possible that the original issues persist, or that new issues surrounding these hazards have appeared. Subject matter experts should reassess the risks associated with these factors.

Six hazard and risk factors isolated in the SATOPS study were seen in the potential factors in the current study:

1. Deficiency in TC-industry communication;
2. Inadequacies in inspection activities;
3. Inadequacies in training regulations;
4. Training deficiencies;
5. Operational pressure; and
6. Worker fatigue.

Subject matter experts should reassess the risks associated with these factors once they have been substantiated through other data sources.

As well as hazards, a number of strengths in the air taxi sector were discussed above. It is possible that some of these strengths resulted from the SATOPS efforts. Given the evidence presented in this study, if we compare the results of this study to the SATOPS study, qualitatively it can be concluded that the SATOPS recommendations did have a positive effect on the safety of air taxi operations in British Columbia. However, as with all ongoing improvement activities, additional efforts may be required in the British Columbia air taxi sector, depending on the level of risk associated with the factors found to currently exist.

AGEMS Analysis of Accident Data

It was very difficult for the study team to gather accurate data on companies for the purpose of identifying air taxi (CAR 703) operators of interest. The team used NACIS, ASIS and the Flight 2005 database to identify companies that had experienced accidents while operating under CAR 703, and to determine which companies were still in business and which had gone out of business. Many records had to be amended when the team met in British Columbia because the NACIS records were out date. The principal inspectors had to tell the team who was still operating. The inspectors were not keeping NACIS up to date.

One hundred and twenty-five accidents that occurred during air taxi operations, involving air taxi operators, were selected for inclusion in the study. Time constraints and unavailability of inspectors prevented all accidents from being included. Of the 125 accidents, 35 were investigated by the TSB (Class 3 Investigations). Table 1 shows the distribution of the 125 accidents, including the number of fatal accidents. Eighteen fatal accidents, involving 40 fatalities, were identified. Most of the accidents occurred during the day.

Of the 125 accidents, 58 involved fixed-wing aircraft and 67 involved rotary wing aircraft. Ten of the fatal accidents involved fixed-wing aircraft while 8 involved rotary wing aircraft. Most accidents occurred during passenger operations.

Aircraft Type	Accident Type	Total
Fixed-Wing	Accident	48
	Fatal Accident	48
Fixed-Wing Total		58
Rotary Wing	Accident	59
	Fatal Accident	8
Rotary Wing Total		67
Grand Total		125

Table 1: Distribution of sample data analyzed by aircraft type and accident type

Forty-four of the fixed-wing aircraft were single-engined aircraft, while 14 were multi-engined aircraft. The majority of fixed-wing aircraft were on floats (33) and 21 were on wheels.

Using AGEMS to guide the classification of unsafe acts, 62% of unsafe acts were intentional, consisting of 60% mistakes and 2% violations (see Appendix A). Ten percent of unsafe acts were unintentional, consisting of 7% slips and 3% lapses. Twenty-eight percent were undetermined due to limited data availability (Class 5 accident reports).

Knowing the type of error involved helps to identify the factors that led to the unsafe act. It was determined that the condition of the landing surface was unsafe in 35% of the accidents involving fixed-wing aircraft. Equipment and weather conditions were also identified as important unsafe conditions (Figure 14). Understanding the levels of performance involved in operational settings can lead to an understanding of mishaps and help to devise preventive measures.

Figure 14: Fixed-Wing Aircraft - Unsafe Conditions

Accident Mapping - ACCIMAP

ACCIMAPs are a compilation of accident-related data presented in the context of Rasmussen’s multi-layered structure (1997). Rather than a “cause and effect” process, an ACCIMAP follows the flow of events in the context of influences on decisions across the layers within the whole organization. The data used to populate the ACCIMAP was drawn from Transportation Safety Board aviation accident reports, Pacific Region, between 1996 and 2005. The level of detail in a TSB report that is required for constructing an ACCIMAP precludes the use of Class 5 reports; therefore, only Class 3 TSB reports were used in this component of the study. Thirty-two accidents were reviewed, covering the target period.

Generic ACCIMAPs are compilations derived from individual ACCIMAPS that have been grouped into categories based on similar factors. The synthesis of the ACCIMAPS generated four generic ACCIMAPS outlining the following hazard sources: equipment, weather, mountainous terrain and coastal terrain. To the extent possible with the data source, the levels within each generic ACCIMAP are populated with the information relevant to the decisions, acts, conditions and events.

Hazard Source: Mountainous Terrain

The mountainous terrain predominant in the Pacific Region produces a number of factors that are common within this category of accident. Fifteen ACCIMAPs were synthesized to generate the Mountainous Terrain generic ACCIMAP.

• Equipment & Surroundings: Wind conditions include strong vertical flows and mechanical turbulence, particularly in the vicinity of dramatic topography. Visibility can vary and whiteout conditions can develop rapidly, largely due to the topography. Terrain is erratic, with narrowing valleys and dramatic vertical changes. VHF communications are restricted due to their ‘line-of-sight’ characteristic. Aircraft loaded to, or near, maximum certified weight and/or configured with floats or external loads may not have the performance capability necessary to maintain acceptable obstacle clearance.
• Physical Processes & Actor Activities: A low experience level in mountain flying plays a large role in decisions to commence or continue flight in conditions illustrated in the ‘‘Equipment & Surroundings’’ level. Selecting a route or landing site, and coping with instrument meteorological conditions (IMC) with limited or no suitable instrument flight rules (IFR) options appear to be common factors in the generic map. The terrain component of the previous level limits manoeuvring capability, and combined with experience level, can lead to over control, lack of adequate control, or loss of control of the aircraft. Frequently, the damage, loss or injury associated with accidents is severe.
• Technical & Operational Management: Flight dispatch decisions were shown to have limited consideration for adequate crew rest and equipment performance capabilities, e.g. hover out of ground effect (HOGE). The placement of ELT equipment, to minimize the effect of impact forces, was contributory to survivability and damage results.
• Local Government: A high priority is placed on client satisfaction; however, there is evidence of a low level of client awareness of aviation safety. Management personnel, who are often the owners, may have little or no aviation background. Reliance on pilot self-dispatch, coupled with the above factors, influences the decisions at the Actor level to initiate flight. When considered in the light of low experience level as an Actor level factor, hazard/risk elements become prominent.
• Regulatory Bodies and Associations: Regulations have addressed passenger safety regarding heli-skiing operations. These have been shown to be beneficial in a number of accidents. Consideration of HOGE in mountainous terrain is not contained within the current regulations.
• Government: No common factors identified.

Hazard Source: Equipment

The wide range of aircraft employed in the air taxi sector introduces a large set of data within this hazard source. Ten ACCIMAPs were synthesized to generate the Equipment generic ACCIMAP.

• Equipment & Surroundings: By their nature, the legislative parameters of air taxi operations involve the use of light helicopters, light twin-engined aircraft and single- engined fixed-wing aircraft. In the Pacific Region, these aircraft operate in an area that is largely remote, northern, heavily-forested and mountainous, or combinations thereof. A characteristic of such an environment is that facilities and technical support are limited.
• Physical Processes & Actor Activities: Accidents illustrating this hazard source show events such as: failure to follow airworthiness directives (AD), non-standard procedures (rigging), component failures, engine failures, and structural failure.
• Technical & Operational Management: Decisions at this level are made in the context of operational needs, interpretation of legislation, and ability to effect those changes. This has resulted in the use of non-standard parts or components, a failure to meet technical specifications (e.g. AD not followed) and documentation deficiencies. Often a pilot or engineer will be held responsible for all aspects of company operations. This is largely a result of the influence of the remoteness of the operations and/or the limited support facilities.
• Local Government: A common factor found at this level for the hazard source (equipment) is that there are incidents of unapproved, non-standard, uncertified parts or components entering the supply chain at the contractor level. This can influence the lower levels (supervisory and actor) in that awareness and acceptance, or lack of awareness, results in their continued use. The common factors at the lower levels illustrate several occurrences.
• Regulatory Bodies and Associations: At the Regulatory level, there have been efforts made to address design or modification issues through supplementary type certificates, airworthiness directives, design approvals and legislation. Also, the campaign to eliminate unapproved parts was directed at the entry of these components into the supply chain. External load regulation has been impaired by its complexity and resultant inaccurate documentation at the supervisory level.
• Government: No common factors identified.

Hazard Source: Weather

The coastal and mountainous environment of the Pacific Region carries with it weather conditions that collectively may produce unique and hazardous situations. Four ACCIMAPs were synthesized to generate the Weather generic ACCIMAP.

• Equipment & Surroundings: Weather phenomena that introduce a threat to operations can take several forms that produce potentially hazardous situations. Strong gusty winds are a frequent characteristic in mountainous areas, and can produce severe turbulence and vertical winds that may be beyond the capability of aircraft performance. Precipitation can be heavy and often continuous, restricting visibility, affecting engine operation, or resulting in standing water on runways. Poor lighting is evident in precipitation and in shadowed terrain. Reflective glare is given off by water and snow-covered areas. Conditions change during flight, often rapidly, and with the variable weather, icing conditions can be difficult to predict.
• Physical Processes & Actor Activities: The extremes and intensity of some of the weather factors impact the Process & Actor level by challenging the decision-making process. The decision to continue a flight into adverse conditions appears to be a prominent factor at this level. The resultant activity may be having to deal with a hydroplaning aircraft, potential runway overrun, rotor decay, flameout or reduced visibility—all in mountainous or coastal terrain.
• Technical & Operational Management: The weather-related accidents reports which contained the data for the generic ACCIMAPs did not identify factors above the bottom two levels. Therefore, no data was available to populate the upper levels.
• Local Government: No common factors identified.
• Regulatory Bodies and Associations: No common factors identified.
• Government: No common factors identified.

Hazard Source: Coastal Terrain

The entire western boundary of the Pacific Region is seacoast. Three ACCIMAPs were synthesized to generate the Coastal Terrain generic ACCIMAP.

• Equipment & Surroundings: Common factors identified from ACCIMAPs related to Coastal Terrain were wind and water conditions, limited manoeuvring space, lighting effects produced by time of day, a mix of traffic that included aircraft and watercraft, and the existence of multiple standard radio frequencies.
• Physical Processes & Actor Activities: The experience level of some air taxi crew members contributed to decisions to continue into deteriorating conditions of poor visibility. Experience level also appeared in critical decisions related to visual flight rules (VFR) navigation in a coastal environment; e.g. route selection involving similar looking inlets. Harbour traffic can be a mix of aircraft and small watercraft, where there is little or no means for intercommunication. Multiple standard radio frequencies observed at the Equipment & Surroundings level create a hazard in communications between aircraft in the same vicinity. The prominent risk experienced with this hazard source was collision—either with another aircraft or watercraft.
• Technical & Operational Management: Assigning crew with limited experience appears to be a factor at this level in the map. This influences the decision making process at the next lowest level.
• Local Government: Weather reporting along the coastal area of the region is regarded as insufficient. Landing areas, with a few exceptions, are unsupervised or unmonitored with respect to watercraft, and when coupled with the identified communications disparities, the risk level becomes elevated. Also, at the local government level (company management), there is an indication that sufficient underwater egress training is required.
• Regulatory Bodies and Associations: Surface vessels are governed under a set of regulations that, like aviation, do not integrate with other modes.
• Government: No common factors identified.

Inspector Interviews and Questionnaires

The severity scale and risk indicator scale were subjected to psychometric analysis to evaluate specific aspects of the scales. The purpose of administering the scales was to determine whether:

1. Inspectors are able to rate operators consistently; and
2. Inspectors performing the rating could be used to differentiate between those operators who had accidents from those who did not.

It is well accepted that accidents are not randomly distributed amongst operators. Some operations are safer than others. The expression “accident waiting to happen” is a common reference to this generally held belief. If TCCA was able, in a systematic and scientifically sound way, to discriminate between operators on the relative probability of an accident occurring, the oversight program could be made more efficient and effective by devoting resources to operators exhibiting the highest levels of risk. No predictive system will ever be perfect. James Reason tells us unequivocally that safe companies do have accidents and unsafe companies may not incur any mishaps for long periods (Reason, 1997).

For a scale to be a useful predictor, it must exhibit two statistical properties: reliability and validity. Reliability is a measure of the degree of consistency associated with the scale. A ruler, for instance, is a reliable measuring device to the extent that, if the same object is measured repeatedly, the same length is observed. If the ruler were made of elastic rubber, however, a different length might be observed on successive measurements. Validity is the degree to which the scale measures or predicts that which it is designed to measure or predict.

Severity Scale: The Severity scale was assessed to determine the degree to which principal operations inspectors (POI) and principal maintenance inspectors (PMI) agreed when rating aspects of the severity of accidents. Table 2 summarizes the ratings assigned. The degree of agreement was not as high as could be desired, but given the lack of familiarity with the task and guidance on how to assign the ratings, the results are encouraging. Table 3 summarizes the extent to which pairs (1 POI and 1 PMI) agreed on the various ratings.

Table 2: Severity Scale Summary

Severity Item	Response Option Range	No. of Ratings a	Mean	Std. Dev.
1. How serious was this accident in terms of the overall safety of operations for the company involved?	1. Not serious at all to 5. Extremely serious	108	3.7	1.0
2. This accident was an aberration; it was completely unexpected.	1. Strongly agree to 5. Strongly disagree	107	2.4	1.1
3. The company will learn from this accident and reform their organization.	1. Extremely likely to 5. Not likely at all	103	2.2	1.0
4. Please rate the probability of recurrence.	1. Not likely at all to 5. Extremely likely	102	2.4	0.8
5. Please rate the severity of the accident.	1. Negligible to 5. Catastrophic	107	2.2	1.1
6. What was the main event category for this accident?	Maintenance Flight Operations Air Navigation Services & Airspace Aerodrome Safety Other. Total	0 5 27 52 24 108	-	-

^a A total of 126 ratings were done, based on 90 different occurrences.

Table 3: Severity Scale- Agreement Amongst Raters

Severity Item	No. of Valid Pairs of Raters	No. of Pairs of Raters that Agree	Percent Of Valid Pairs That Agree
1. How serious was this accident in terms of the overall safety of operations for the company involved?	24	15	63%
2. This accident was an aberration; it was completely unexpected.	24	13	54%
3. The company will learn from this accident and reform their organization.	24	13	54%
4. Please rate the probability of recurrence.	21	10	48%
5. Please rate the severity of the accident.	24	19	79%
6. What was the main event category for this accident?	24	15	63%

In general, the level of agreement between raters is not as high as desirable. Lower levels of agreement may suggest that perhaps the raters are not trained to the same standards, or find it difficult to evaluate certain risk items, or have access to different information regarding the operations and the accident. For example, there was only about 50% agreement on items 2, 3 and 4: whether the accident was an aberration, whether the company will learn from the accident, and the probability of recurrence. Two of these items ask the rater to predict future events rather than report on aspects of the accident itself. It is likely that, in addition to having access to different sources of information, the raters were being tasked to form a summative judgement based on a variety of information. Without explicit guidelines on what type of information should be used for these items, and how it should be used, it is likely that the POIs and PMIs would come to different conclusions regarding the likelihood of future occurrences. Therefore, it is not surprising that there was disagreement on these questions.

While access to different sources of information may partly explain why the POIs and PMIs rated the probability of future accidents differently, it does not solve the problem of what to do with this conflicting information. If the severity items are to be used in indicating which operators might need more or less frequent inspection, then the discrepancy between operators becomes an important issue. Educating raters regarding the signs of future problems may be one way to approach this issue. In addition, it may be useful to have the POI and PMI share information before they make their independent ratings, and then discuss and resolve discrepancies where possible.

Risk Scale: POIs and/or PMIs were asked to rate nine items intended to assess the risk associated with each operator. The nine items consisted of two items relating to financial risk factors (contracted services and fiscal pressures), four items relating to personnel risk factors (labour problems, management problems, key person changes and staff turnover) and three items assessing the effectiveness of quality assurance programs, changes in fleet/facilities, and changes in routes or services. Each of these items was assessed with respect to the operator’s status, and with respect to Transport Canada’s status at the same time. Each item was rated on a three-point scale (Yes, Unsure or No).

Unlike the severity items, which rated the severity of a specific accident, the risk items dealt with organizational characteristics. Therefore, it was possible to evaluate operators for risk at the time of an accident or at any other time. In addition, operators who had not had an accident could also be assessed on these items. This resulted in ratings being produced under one of two circumstances. One group was rated in relation to an accident. The second group was operators whose risk was assessed at the “current” time (July 2006), not in response to any specific accident.

On average, when dealing with operators at the time of an accident, raters agreed on 68.8% of their ratings (Table 4). However, the rate of agreement for pairs of raters ranged from 22.2% for one pair of raters (i.e., they agreed on only 2 of their 9 ratings) to 100% (the POI and PMI agreed on all nine of their ratings). The average rate of agreement is clearly better than chance (which would be 33.3% since there were three valid response options). The item on which POIs and PMIs exhibited the least agreement was whether the company had an effective quality assurance program. This is likely because POIs are not familiar with the administration of quality assurance programs.

Table 4: Agreement per Item for the Risk Items—Operator Assessment at Time of Accident

Risk Scale: Operator at Time of Accident	No. of Valid Pairs of Raters	No. of Pairs of Raters that Agree	Percent Of Valid Pairs That Agree
1. Contracted services	32	24	75%
2. Fiscal pressure	32	25	78%
3. Labour problems	32	22	69%
4. Management problems	31	24	77%
5. Key person changes	32	24	75%
6. Staff turnover	31	21	68%
7. QA program effective	22	5	23%
8. Scope – change in fleet/facilities	31	20	65%
9. Product line – change in routes/services	32	19	59%

Table 5. Agreement per item for the Risk Items -- Operator Assesment at Current Time

Risk Scale: Operator at Time of Accident	No. of Valid Pairs of Raters	No. of Pairs of Raters that Agree	Percent Of Valid Pairs That Agree
1. Contracted services	62	46	74%
2. Fiscal pressure	63	54	86%
3. Labour problems	63	52	83%
4. Management problems	63	52	83%
5. Key person changes	63	47	75%
6. Staff turnover	63	44	70%
7. QA program effective	37	20	54%
8. Scope – change in fleet/facilities	63	46	73%
9. Product line – change in routes/services	61	38	62%

Furthermore, a multivariate procedure, called discriminant analysis, showed that the ratings could be used to differentiate those companies who had accidents from those who had not.

Inspector Interviews: During the interviews, inspectors identified a large number of potential hazards relating to the air taxi sector in British Columbia. Combining inspector insight with hazards and ACCIMAPs led to a practical understanding of how accidents happen, and steps that may reduce the risks. The greatest gains are to be made in the area of providing better support to decision making in operational settings. For instance, self-dispatch, a common practice in the air taxi sector, leads to contact with the geographical hazards (mountainous terrain, coastal terrain, and weather), especially when the self-dispatching pilot lacks extensive experience.

Within TCCA, there is room for better coordination between operations and maintenance inspection activity. This is consistent with the fact that the level of agreement between POIs and PMIs on the severity and risks scales was lower than desirable.

Discussion

Historical Review: The historical accident rate does not indicate that the level of risk for passengers and employees in the air taxi sector has increased. To date, the 2007 accident count in the Pacific Region is lower than 2006, and more like previous years. It may well be that 2006 was anomalous, similar to the pattern experienced in 2000.

The historical review was made more difficult by the fact that NACIS was out of date and many records had to be manually updated. If TCCA is to become a data-driven organization, then data requirements should be closely examined. They must be determined on the basis of the jobs people do and the way they do them. This is the principle of human-centred design and should be adopted and as the TCCA standard.

Methodology: The use of the AGEMS model and the ACCIMAP procedure led the study team to an excellent understanding of the dynamics of accidents and the interplay of conditions and factors that lead to mishaps. The procedures are time consuming and require an advanced level of knowledge and understanding of human and organizational factors and the operational and technical settings pertaining to the occurrence(s). This combination is unlikely to be found in any individual, so application of the techniques requires a team approach. Overall, the methods are judged to be valuable tools in understanding the structure and dynamics of aviation risk. These methods should be adopted as Safety Intelligence standards and used in future complex analyses.

Risk Patterns: Analysis of accidents occurring between 1996 and 2006, using the AGEMS and ACCIMAP techniques, confirms that the Pacific Region presents significant challenges in the form of geographic hazards—mountains, coastal terrain and weather. The terrain can make radio communications difficult. Rugged terrain reduces the options available should any kind of problem arise during a flight, and the remote locations served by Pacific air taxi operators make it difficult or impossible for assistance to be provided to a flight crew experiencing problems. This is not news, but the technique showed how these hazards interact with normal human characteristics to produce untoward outcomes. The geographical hazards are particularly dangerous when combined with low pilot experience levels, self-dispatch protocols, and less-than-stringent supervisory practices. Client and business pressures can also play roles in elevating risks associated with particular flights.

The most dangerous part of a pilot’s career is the period between 100 and 500 flight hours. This has often been attributed to an overconfidence effect, but there is another explanation that is, scientifically, more plausible and offers better options for the development of risk reduction measures. For the first part of a pilot’s career, he or she is typically closely supervised and guided by a flight training unit. After the pilot earns a licence, the supervision is relaxed. At this time, the pilot assumes more responsibility for operational decisions. Although an accident can happen anytime, a less-experienced pilot is particularly vulnerable to finding him or herself in a high-risk situation.

Cockpit decisions can be perceived as having two components: situation assessment and selection of a course of action. The difficulty involved in decision making depends primarily on two factors: “the degree of clarity of the cues signifying the problem and the nature of the response options available in the situation”.⁴

Cues, or information about the situation, can vary between clear and ambiguous. The clearer the cues, the less mental effort is required to interpret them. In an unfamilar situation, it is very difficult to interpret available cues. The inexperienced pilot is often faced with unfamilar situations. If you have not seen something before, how do you know what you are looking at? The difference between a novice and an expert is that the expert has seen it all before and is able to interpret the situation very quickly, even automatically.

Response options vary from highly defined to unspecified. The more choice and the less firm guidance about what to do, the more difficult the selection of an option becomes. Novice pilots are, for the most part, better off when the choice of what to do in a situation is clear. For instance, student pilots were guided that, in the event of an engine failure after takeoff, they should land straight ahead unless they had enough altitude to turn back. Novices are poor judges of what constitutes “enough” altitude. In fact, the likelihood of fatality was eight times higher when turning back.

Countermeasures: The more complex and difficult a decision is, regardless of whether the difficulties are in the situation assessment component or in the response selection component, the more likely the outcome of the decision is to be less than ideal. TC and the industry can work together to provide more support to all pilots in the air taxi sector in Canada. This support is most likely to be effective if it provides the pilot with more guidance on what specific cues are important, what they mean, and what action is appropriate. Generalized guidance such as ‘‘be careful’’ or ‘‘watch out for winds at …’’ are not sufficient. Information can be provided on local hazards, the signs that indicate that a hazard is active, and how it should be handled.

In the late 1990s, Canada 3000 took such an approach to controlled flight into terrain (CFIT) avoidance at southern destinations. The company developed written and video guidance on approaches at their southern destinations, with particular emphasis on hazards and countermeasures. Before flying to an unfamiliar destination, pilots could refer to the destination package to learn, or refresh themselves on the approach. This is decision support. Generalized pilot decision-making courses, especially those based on attitude change, are not effective.

Risk Indicators: It was apparent that POI and PMI ratings of some factors showed a lower degree of agreement than is desirable. This is a way to estimate reliability. When assessing a predictor, reliablity is very important since the reliablity establishes a limit on validity. If assessors cannot agree on a factor, they cannot all be right, so predictions based on their perceptions are not likely to be precise. This was, however, a very challenging test of the approach. Inspectors were not trained and were working with minimal guidance. They did agree on some variables, and the risk indicator scale was able to discriminate those operators who had accidents from those who had not.

Given the mathematical characteristics of the criterion used (accident/no accident) and the inspectors’ lack of familiarity with the task, this is very encouraging. TCCA can be assured that a valid risk indicator protocol, based on inspector perceptions, can be developed. Such development must be approached cautiously to provide for adequate definition of the task, guidance in interpretation of the scales and available information, training, and a standard procedure for gathering and analyzing the information, using appropriate multivariate statistical procedures.

Conclusion

There is no evidence at this time that the level of safety afforded Pacific Region air passengers and employees has deteriorated. The variance in the fatal accident rate in 2006 is within a range that can be expected in this type of data. So far in 2007, the accident rate appears to be regressing to the historical mean.

The last significant intervention in the air taxi sector was the SATOPS project in 1996-97. The accident rate decreased in the period following SATOPS, until 2000, when it reached a plateau; it has remained fairly constant since then. The accident analysis and interviews with TCCA inspectors revealed a number of hazards that combine in particular ways leading to accidents. It is likely that TCCA, in cooperation with the industry, can develop better decision support for air taxi pilots. This could benefit all pilots, but would be especially helpful to low-time pilots who have most of the accidents.

The SATOPS Task Force developed 71 recommendations, most of which have been implemented. Following that, the accident rate in the air taxi sector improved. This study found that several of the hazards identified during SATOPS were not present now. It cannot be stated that SATOPS lowered the accident rate, but it is likely that SATOPS, the transition from the Air Navigation Orders to the Canadian Aviation Regulations, and other progress in the industry, combined to better manage risks and lower the accident rate.

It has been verified that inspectors can rate operators on a number of variables and that their ratings can be manipulated mathematically to predict elevated levels of risk. Further work on the development of a predictive risk indicator is warranted.

Recommendations

The understanding gained from applying the ACCIMAP technique indicates that, for the most part, individual hazards do not lead to accidents in and of themselves. The risk elevates when hazards such as mountainous terrain, weather and pilot inexperience interact. Eliminating hazards is preferred, but mountains cannot be changed and only time can remedy inexperience. Work can be done to reduce the risk of these unavoidable interactions.

Rasmussen’s Risk Management Framework and the ACCIMAP technique proved to be practical techniques which will guide the development of effective countermeasures, increasing our understanding of how hazards interact. Rasmussen’s Risk Management Framework is a relatively easy to understand model of how risks and technical and human factors combine to present opportunities for mishaps. The ACCIMAP technique is a complex application of the Risk Management Framework, which is not likely to be widely employed, at least in the near term, by most inspectors. The technique, however, can be used on a regular basis by Safety Intelligence personnel to conduct analyses and to support the inspectors and managers.

Despite the less than desirable reliability of the safety indicator scales tested in the study, the technique showed promise by demonstrating that inspectors, under unfavourable conditions (lacking guidance, standards or training) were able to assign ratings which could discriminate between air operators likely to have accidents and those less likely to have mishaps.

Appendix A: Adapted Generic Error Modelling System (AGEMS)

Individual actions and decisions, viewed out of context, can appear to be virtually random events, defying explanation. Human behaviour, however, is not random. It usually conforms to some pattern and can be understood. The reason for using a model of human error is to guide the analyst in:

1. identifying the information needed to complete an analysis; and
2. analyzing the information to arrive at an understanding of the factors that lead to specific errors.

The model is predicated on research which indicates that people in operational settings do not usually use an analytical approach to decision making. Rather, they use much more efficient methods which capitalize on their training, experience and knowledge of the systems they are working within.

Levels of Performance

The model provides for three levels of performance, ⁵ distinguished from each other by the degree to which the performance requires conscious information processing.

Skill-Based Performance: When people are performing familiar work under normal conditions, they know by heart what to do. They react almost automatically to the situation and do not really have to think about what to do next. For instance, when a skilled automobile driver is proceeding along a road, little conscious effort is required to stay in the lane and control the car. The driver is able to perform other tasks, such as adjusting the radio or engaging in conversation, without sacrificing control. Errors committed at this level of performance are called slips or lapses.

Rule-Based Performance: The rule-based level of performance is used when tackling problems that can be diagnosed, and for which there are readily-available solutions. People have all kinds of “rules” stored in their memories. These are not necessarily regulations; they are more like: “If this condition exists, then do that.” Rule-based performance requires more conscious thought than skill-based performance. The amount of mental effort will depend on the clarity of cues available to diagnose the situation, and whether the response is prescribed, or a choice is required from the options presented. Errors committed at the rule-based level of performance are called mistakes.

Knowledge-Based Performance: The final level of performance is used when the situation cannot be readily diagnosed, or there is no procedure to follow. The person in this situation has to rely on his/her knowledge of the system and creativity to devise a new way out of the problem. This level of performance involves the highest level of mental effort. One of the most famous examples of knowledge-based performance in aviation is the Sioux City DC-10 crash, where the flight crew devised a way to control the aircraft after a complete hydraulic failure. There were no procedures, so the crew developed a creative solution and in doing so, saved lives. Errors at this level are also called mistakes.

The three levels of performance form a sort of hierarchy of responses. People tend to be most comfortable at the skill-based level. Only when it is necessary do we progress to the rule-based level. If an appropriate rule comes to mind, it will become the plan. Only after the possibility of identifying a rule is exhausted will we progress to the knowledge-based level of performance.

Types of Error

Errors can be classified a number of different ways. In the most basic terms, however, there are two kinds of errors—execution errors and planning errors. Planning errors involve thinking. Execution errors do not⁶. Execution errors occur at the skill-based level of performance and planning errors occur at the rule and knowledge-based levels of performance. Violations are a particular type of planning error. Most planning errors can probably be described as “honest mistakes”. Violations, however, exhibit a willful disregard of standards or regulations. For the purposes of this study, unsafe acts were classified as execution errors, planning errors, or violations.

The Analytical Process

Arriving at this classification was accomplished by a systematic step-by-step process, using accident investigation reports and Class 5 occurrence summaries published by the Transportation Safety Board.

Step one was to read a report to understand the chronological sequence of events, actions, and conditions that produced the accident or incident.

Step two consisted of identifying the unsafe act, acts, or conditions that were apparent from the sequence.

Step three was to determine the type of error. This involved two sub-steps:

1. Determine whether the unsafe act was intentional or unintentional. Did the person intend the action? Unintentional actions are those that did not go as planned. They are errors of execution. Intentional actions are those that are carried out as planned, but the actions are inappropriate; these are errors in planning.
2. Identify the error type that best describes the error. The choices are: slip, lapse, mistake, or violation.

Step four was, to the extent possible from the investigation report or occurrence summary, to identify what events, conditions, knowledge or skill levels were contributory to the unsafe acts. Lack of experience, regulatory inadequacy, fatigue, design features of the aircraft, and operating pressures are examples of such antecedents identified by Reason⁷ and others.

The sample of accident reports examined included the seventy accidents referred to by the Transportation Safety Board in the preamble to Recommendation A96-10, as well as other accidents in the TSB database identified as potentially relevant by TC safety programs.

Each working group member was assigned responsibility for analyzing a number of reports and was encouraged to collaborate and discuss their determinations to achieve a high degree of consistency in the analysis. Each accident and incident constituted a record which was transcribed onto a computerized spreadsheet. Before an unsafe act was classified as a violation, at least two working group members had to agree that the investigation report provided convincing evidence of willful contravention of a regulation or standard.

Once the individual cases were analyzed and records transcribed, the working group met to consider the data and determine what systemic safety problems were identifiable, particularly with regard to the adequacy of the margin of safety afforded by current VFR visibility minima.

On the basis of the events and conditions described in the investigation reports or occurrence summaries, the working group was able to identify several accident patterns.

_____________________

¹Flight 2010: a Strategic Plan for Civil Aviation. TP 14469
² IBID
³ A Likert scale is a 5-point scale devised by Rensis Likert. It is probably the most popular scale used in survey research.
⁴ Orasanu, Dismukes, Key, and Fischer, Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting, Seattle, Washington, 1993.
⁵ Human Factors for Aviation: Basic Handbook. 1997. Transport Canada TP 12863.
⁶ See James Reason Human Error. 1990. Cambridge. Cambridge University Press.
⁷ James Reason. Human Error. 1990. Cambridge. Cambridge University Press.

Date modified:

2010-03-31

Primary navigation