CANUTEC SURVEY

Alternate format

EVALUATION OF CANUTEC SERVICES
- A SURVEY OF FIRST RESPONDERS

M. Provencher

Transport Dangerous Goods Directorate

June 2009 TP 14901

ABSTRACT

Everyone involved in emergency response knows the importance of having access to critical and timely information during an incident. The Canadian Transport Emergency Centre (CANUTEC) assists emergency response personnel in handling dangerous goods emergencies. To evaluate the level of impact that CANUTEC has amongst first responders, a survey was conducted early in 2009. A 2-page questionnaire was sent to fire departments and police services across Canada. 46% responded. More than 85% of respondents were aware of CANUTEC, similarly for the Emergency Response Guidebook. 43% of respondents have used CANUTEC’s emergency telephone service while 68% of respondents used the Guidebook for training, simulation or actual emergency response. The great majority of respondents who used the services were satisfied with the assistance obtained.

INTRODUCTION

CANUTEC is the Canadian Transport Emergency Centre operated by Transport Canada to assist emergency response personnel in handling dangerous goods emergencies. This national bilingual advisory centre was established in 1979 and is part of the Transport Dangerous Goods (TDG) Directorate. CANUTEC’s scientists, using the information accessible to them and their professional experience and knowledge, provide immediate advice and recommend actions to be taken and actions to avoid in dangerous goods emergencies.

As part of Transport Canada’s commitment to providing relevant, timely and quality services, and more simply because it is important to know the difference one makes, the TDG Directorate with the support of the Canadian Association of Fire Chiefs and the Canadian Association of Chiefs of Police undertook a survey evaluating CANUTEC services.

This paper presents the methodology adopted, the instrument or questionnaire used, and the results obtained.

OBJECTIVE

The objective of the survey is to evaluate the knowledge first responders have, and the use that they make, of CANUTEC’s services.

APPROACH

As early as the fall of 2006, a survey was being considered and a questionnaire was built. Input was sought from CANUTEC and representatives of the fire and police departments¹ on how to improve the questionnaire’s content and delivery. The questionnaire is intended for first responders (firefighters and police officers). The National Directory of Emergency Services 2006 provided the addresses and telephone numbers of fire services and police services in Canada. A hard copy of the directory was ordered, received and the scan to convert its content into an Excel file was completed in April 2007². A total of 4927 such services were listed: 3738 fire departments and 1189 police services. This is the best frame for the targeted population i.e. all fire departments and police services in Canada. It appears that no more than one location is listed per type of service, per city. That is for a larger municipality only the fire department and not all fire stations are listed, similarly for police services. Figures 1 and 2 present the distribution of these services by geographical region.

The entire population was solicited rather than a sample as the typical response rate in public opinion telephone surveys is around 10% to 20% [1]. A representative sample for such a participation rate would require 70% of the population to obtain a reasonable margin of error (5%) on the estimated proportions. Therefore the cost difference in opting for the entire population was minimal.

From prior discussions, mailing was the preferred solution with returns either by mail (self addressed, stamped envelopes were enclosed in the original packet) or fax. All that was readily available for the entire population were the addresses and telephone numbers listed in the directory. A telephone survey would not have worked well because volunteer firefighters are not always at the fire department. The other issue with a telephone survey is reaching and interviewing the right individual: a person familiar with dangerous goods. An electronic survey publicised on the TDG, the fire or police associations’ web site might have been convenient. However it could have resulted in duplication from some of the services and a strong bias in the answers (it is like preaching to the converted).

The project was set aside and resurrected in the fall of 2008. The questionnaire and accompanying letter were refined (see the Annex for an example of the final product). The printing and distribution were completed early in January 2009. A February 2^nd cut-off date for the returned questionnaires was provided.

RESULTS

Participation Rate

Responses started coming in on January 13 and were received in large numbers up to mid-February³. Still more questionnaires kept coming in. A second mailing of questionnaires was made on April 3^rd to a random sample of 116 services that had not yet responded, to determine if non-responding services were much different from responding services. Table 1 illustrates the flow of questionnaires over the January – May period. 3 services sent their questionnaires by email as an address was provided for information purposes in the letter, a few more faxed their responses (179) and most services (2065) mailed them⁴. Through the survey we were informed of some restructures and closures: 35 services were closed or amalgamated with others. 177 questionnaires were returned because of a wrong address. Considering that a 2006 directory was used, this number is not surprising and is low at 4% of all services.

A total of 2247 questionnaires were received out of an adjusted total of 4893 services. Consequently the participation rate is 2247/4893 = 46%, that is, much higher than expected.

Follow-up calls were not made to improve the response rate but to clarify some answers or comments made by the services. These few contacts confirmed the difficulty in reaching volunteer firefighters by telephone. Several calls were made to the same service sometimes without success.

A quick look at the distribution of respondents by geographical region provides some comfort as to a reasonable representation by region. For fire departments the highest participation rates are found in Quebec (57%), Ontario (55%), New Brunswick (52%) and the lowest in the Northwest Territories (15%) and Yukon (29%). With police services the participation rates are a little higher, all above 40%, with higher participation rates found in Alberta (58%), Prince Edward Island (55%), and Manitoba (53%). Overall the fire departments had a 45% participation rate while the police services had a slightly higher participation rate of 49%. Could it be because a lot of fire services are volunteer/part-time services?

Shortly after the questionnaires were mailed on January 12, it was realised that the web site address provided in the questionnaire for the guide was no longer correct. Fortunately this problem was quickly solved on January 14: the current and previous paths were both operational.

No imputation per se was made to the responses provided. Imputation is often used to assign values to missing items: a replacement value from another respondent with similar characteristics is imputed for the missing value [2]. Only corrections, also called deductive imputations, were introduced. For example if one service did not respond to the question "Have you ever contacted CANUTEC?" and then responded to the question on why they contacted CANUTEC, the answer to the first question was set to "Yes".

Some of the comments were direct requests for guidebooks or documentation. These comments do not show in this report. There were a little over 50 such requests: some requests were handled through the provincial/territorial point of contact, others were handled directly by the TDG Directorate.

Assessment by Question

Unless otherwise specified, all percentages presented in the responses to the questionnaire are relative to the overall number of questionnaires received/analysed i.e. 2247.

Question 1: Which category best describes your organization?

The first question sought to identify the respondents. In particular, we were interested in knowing if there were a lot of volunteer/part time fire services. As shown below, an overwhelming 61% of respondents (1367 services) were volunteer fire departments, another 25% of respondents were police services. The rest was split amongst composite (11%) and full-time (3%) fire services, or other (1%).

Responses	Category
1367	Volunteer fire department5
67	Full-time fire department
236	Composite fire department (composed of volunteer and full-time firefighters)
570	Police service
13	Other, please specify

The “Other” category consisted of 13 services. Amongst them, a few specified “municipality” or “inter-city agreement”. Some indicated they dispatch for both police and fire services or assume both policing and firefighting tasks. Two services indicated that they respond to medical emergencies.

Assuming the respondents are representative of the entire population, the great majority of fire departments are volunteer fire departments.

Question 2: Which size of population best describes the area your organization serves?

Responses	Category
42	Large urban (250,000 or more)
85	Urban (80,000 to 249,999)
208	Small city (20,000 to 79,999)
1309	Smaller community (1,000 to 19,999)
601	Rural (under 1,000)

It is not surprising to find that 85% of services are in smaller communities or even rural areas since a majority of respondent services are volunteer fire departments.

Here the aim of the question was to obtain the size of the population served. It seems that because of some of the qualifiers, a few respondents were confused when the size of the population was relatively large but composed mostly of farmers. They would choose “smaller community” and add that it was rural or check mark both. The adopted answer was that of the bigger population size.

Question 3: Are you aware of CANUTEC, the Canadian Transport Emergency Centre, a 24-hour bilingual service for accidents involving dangerous goods? (call collect 613-996-6666, cellular *666)

Responses		Category
1942	86%	yes
299	13%	no

86% of respondents were aware of CANUTEC, but more interestingly, who was not? It appears that from the 299 respondents that were unaware of CANUTEC, 174 are fire departments, 102 are RCMP services, and the rest are provincial or municipal police services. For fire departments, it should be noted that the province of Saskatchewan had 52 fire departments declare that they were unaware of CANUTEC (11% of all fire departments in that province) while Newfoundland and Labrador had 29 such fire departments (10% of all fire departments in the province). For the RCMP services, the provinces where the 24-hour service seemed to be less known were Alberta, Saskatchewan and British Columbia at 20, 19 and 17 services respectively.

Question 4: Have you ever contacted CANUTEC?

Responses	Category
973 43%	yes
1112 49%	no

These percentages added together exceed the 86% of services that are aware of CANUTEC. That is because 86 respondents checked both answers. Depending on the situation, their service has called CANUTEC and at other times not.

Question 5: You have not used CANUTEC services because

Responses	Category
844 38%	you never hdad a dangerus goods situation?
146 7%	your regular personnel handled the dangerous goods situation(s)?
19 1%	you have your own expert team for these situations?
78 3%	you called upon an external service? Please elaborate

49% of respondents did not contact CANUTEC: 38% because they never had a dangerous goods situation, 7% because they handled the situation with their own personnel, 1% because they have their own expert team and 3% because they called another (external) service.

Of those who called an external service, 40 called a fire department (most of the 40 were police services), 11 called the Environment (a provincial or federal organisation), another 11 called a company for clean-up or information, and 16 called other organisations mostly HAZMAT teams. It is unknown however whether these were private or public services or if they themselves called CANUTEC.

Question 6: Why did you contact CANUTEC?

Responses	Category
385 17%	general information on products
670 30%	emergency response measures to take during an incident
215 10%	reporting an accident
300 13%	simulation/exercise
14 1%	other, explain

43% of respondents contacted CANUTEC. The reasons for contacting CANUTEC are about: emergency response measures to take during an incident for 30% of respondents, general information on products for 17% of respondents, simulation/exercise for 13%, reporting an accident for 10% and other reasons for 1% of respondents. It is clear that the same service may have called one or many times for different reasons. The other reasons raised for calling CANUTEC (there were 14 such reasons mentioned), are essentially for educational material, training, and clarifications on reporting or planning.

Question 7: Customer satisfaction – CANUTEC

Percentages are relative to the number of respondents to this question.

98% or more of the services that contacted and evaluated CANUTEC were satisfied with the timeliness provided by CANUTEC, the politeness and professionalism with which the specialist handled the call, the clarity and relevancy of the instructions/information provided, and the helpfulness of the information. 99.7% of these respondents (893 services) would recommend CANUTEC to others.

To the question on how CANUTEC could improve on the service provided, 123 respondents commented. Amongst them, 14 indicated that CANUTEC does excellent work, 42 indicated that CANUTEC does a good job, 9 had issues with language, 36 mentioned promotional material. The comments on promotional material were mostly suggestions for additional material such as wallet cards, documentation (electronic or otherwise), training, or a toll free number.

Question 8: Are you aware of the Emergency Response Guidebook (ERG2008)?

(A Guidebook for First Responders during the initial phase of a dangerous goods/hazardous materials incident)

Responses	Category
1948 87%	yes
265 12%	no

Just as CANUTEC is very well known by respondents, so is its Emergency Response Guidebook as 87% of respondents indicated that they were aware of it. Amongst the services that were not aware of the Guidebook, 180 were fire departments, notably 47 from Saskatchewan and 33 from Newfoundland and Labrador.

Question 9: What percentage of your vehicles (police cruisers/fire trucks) has a copy of the Guidebook?

1966 services responded to this question. This is the one question where the value, a percentage, is not relative to the total number of questionnaires received (2247). Figure 5 illustrates the distribution of percentages provided by the respondents through a table of frequencies with corresponding histogram.

124 services indicated that they had no copy of the Guidebook in their vehicles while 1584 services had at least one copy in all of their emergency vehicles. Therefore 81% of respondent services to this question have a copy of the Guidebook per emergency vehicle. Ideally the TDG Directorate would like a copy of the Emergency Response Guidebook in all emergency vehicles in Canada.

Question 10: Has your organisation used the Guidebook for training, simulation or actualemergency response?

Responses	Category
1524 68%	yes
504 22%	no

Figure 5: Frequency and Histogram Table of the Percentage of Vehicles with a Copy of the Guidebook

87% of respondents were aware of the Guidebook, 68% have used it in training, simulation or actual emergency response.

Question 11: Customer satisfaction – Emergency Response Guidebook

(hard copy or electronic versions)

Percentages are relative to the number of respondents to this question.

98% or more of the services that contacted and evaluated CANUTEC were satisfied with the timeliness provided by CANUTEC, the politeness and professionalism with which the specialist handled the call, the clarity and relevancy of the instructions/information provided, and the helpfulness of the information. 99.7% of these respondents (893 services) would recommend CANUTEC to others.

98% or more of the services that used and evaluated the Guidebook found it well organised and easy to use and found the information that they needed in a reasonable time. To the question on how to improve the Guidebook, 185 respondents commented. 79 services indicated that the Guidebook works well the way it is, several indicated that they would like a larger print (11), others indicated that tabs (9) and spiral rings (7) would be useful. 19 services mentioned electronic versions (some are already available, others were proposed). The rest of the comments were quite varied in nature: greater availability of the guide, more frequent updates, etc.

Question 12: Any additional comments?

An opportunity was given at the very end of the questionnaire for any additional comments. 266 respondents took advantage of this opportunity. Most of these comments were general praise for the service (143 respondents with comments such as “Keep up the good work” or “Great service”). 52 respondents commented on the guide, often to indicate how they valued the document, some out of concern for availability or price. 21 respondents commented on the web site and the electronic availability/ease of use of the guide and other information.

Non-respondents

All in all the results of this survey are very positive for CANUTEC. First a 46% participation rate was obtained, that is much better than the 20% anticipated, and awareness of the telephone service and the Emergency Response Guidebook both of which at the heart of CANUTEC’s activities is high, more than 85% of respondents indicated so.

As mentioned earlier 4927 questionnaires were mailed early in January. Despite the high participation rate, 116 additional questionnaires were sent early in April to a random sample taken from the services that had not yet responded. 34 of these services responded and their responses are included in the results. This added sample provides some understanding as to whether the services that did not respond are very different from those who did. Using the sample we find that:

- the estimated awareness rate for CANUTEC’s 24-hour service is 76% with a 95% confidence interval around the actual awareness rate of non-respondents of [62%, 91%] – 86% is the value obtained from all respondents

- the estimated rate for contacting CANUTEC is 24% with a 95% confidence interval of [9%, 38%] – 43% is the value obtained from all respondents and is not in that confidence interval

- the estimated awareness rate for the Guidebook is 79% with a 95% confidence interval of [66%, 93%] – 87% is the value obtained from all respondents

- the estimated rate of use for the Guidebook is 50% with a 95% confidence interval of [33%, 67%] – 68% is the value obtained from all respondents and is not in the confidence interval.

This small sample provides indication that the non-respondents are not all that different from the respondents in terms of their knowledge of CANUTEC and its Emergency Response Guidebook, however these services might be less inclined to either use the Guidebook or contact CANUTEC. It is to be noted however that all sampled services that used CANUTEC services were satisfied.

CONCLUSION

Early in January 2009, questionnaires were sent to 4927 fire departments and police services in Canada. A participation rate of 46% was obtained. Analysis of the responses shows that 86% of the respondent services are aware of CANUTEC’s 24-hour service while 87% are aware of the Emergency Response Guidebook, to a lesser extent in the provinces of Saskatchewan, Newfoundland and Labrador. The fire departments or police services that either called CANUTEC or used the Guidebook are satisfied with the service. Comments were generally very positive and some will help improve on the service provided.

REFERENCES

[1] Best Practices in Public Opinion Research – Improving Respondent Cooperation for Telephone Surveys. Published by Public Works and Government Services Canada. March 2007.

[2] Sampling: Design and Analysis. S.L. Lohr. Duxbury Press. 1999.

ANNEX A

SAMPLE LETTER AND QUESTIONNAIRE

---

The following document is available for downloading or viewing:

• Evaluation of Canutec Services (TP14901E)(PDF Version 1,314 kB)

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet:

• Adobe Reader
• Foxit Reader
• Xpdf
• eXPert PDF Reader