Follow-up Progress Assessment of Audit of IM/IT Project Life Cycle Controls

This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

 

Table of Contents

EXECUTIVE SUMMARY

A follow-up assessment of management's progress addressing the recommendations from the 2011 Audit of IM/IT Project Life Cycle Controls was included in the 2013/14 audit plan due to the significance of the audit findings. Internal Audit follow-up on management action plans are part of key requirements of the Treasury Board Policy on Internal Audit and the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing because it helps ensure timely implementation of audit recommendations.

In 2011 the IM/IT Project Life Cycle Controls Audit was completed. The objective was to assess the adequacy of the Department's management control framework for IM/IT projects, identify control gaps, if any, and test the operating effectiveness of existing controls.

The audit identified significant weaknesses in the Department's management control framework for IM/IT investments. As a result, six recommendations were made to Transport Canada senior management. All six recommendations were accepted by management and a Management Action Plan was recommended for approval by the departmental Audit Committee in July of 2012.Management attested that all recommendations had been addressed, with the exception of the implementation of an updated governance framework.

The objective of this exercise was to provide an assessment of the status of the implementation of each of the six recommendations from the original audit. In addition, given the pace of change across government since the audit, the scope of this engagement also included an assessment of steps being taken to streamline the 379 systems in the Department and the extent to which IM/IT plans/planning processes are identifying readiness related risks and challenges as the Department and government move to greater centralization of IT services and applications.

Overall, significant improvements to the Department's IM/IT project life cycle controls were observed during the follow-up engagement. Streamlined processes, enhanced reporting mechanisms and improvements to accounting procedures were noted.Also, prior to approval, proposed new projects now require more detailed options analysis and measurable outcomes. At the time of this assessment there were 11 IM/IT projects underway in the Department with a total estimated cost (TEC) of approximately $22M, compared to the 43 IM/IT projects with a TEC of $97M at the time of the original 2011 audit. In addition, plans are starting to be developed to consolidate, rationalize and integrate the Department's systems in order to achieve greater efficiencies.

1. INTRODUCTION

1.1 PURPOSE

A follow-up assessment of management's progress addressing the recommendations from the 2011 Audit of IM/IT Project Life Cycle Controls was included in the 2013/14 audit plan due to the significance of the audit findings. Internal Audit follow-up on management action plans are part of key requirements of the Treasury Board Policy on Internal Audit and the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing because it helps ensure timely implementation of audit recommendations.

1.2 BACKGROUND

In 2011 the IM/IT Project Life Cycle Controls Audit was completed. The objective was to assess the adequacy of the Department's management control framework for IM/IT projects, identify control gaps, if any, and test the operating effectiveness of existing controls.

In assessing the Department's management control framework, the audit looked at the following:

  • the governance structure that senior management has put in place to direct, manage and monitor IM/IT projects;
  • the Department's risk management practices for IM/IT projects; and
  • the Department's controls to ensure that IM/IT projects deliver the expected benefits and are completed based on the approved budget, schedule and scope.

The audit identified significant weaknesses in the Department's management control framework for IM/IT investments. As a result, six recommendations were made to Transport Canada senior management. All six recommendations were accepted by management and a Management Action Plan was recommended for approval by the departmental Audit Committee in July of 2012.Management attested that all recommendations had been addressed, with the exception of the implementation of an updated governance frameworkFootnote 1.

1.3 OBJECTIVES, SCOPE AND APPROACH

The objective of this exercise was to provide an assessment of the implementation status for each of the six recommendations from the original audit. The scope of the assessment included assessing control design; control operating effectiveness was not assessed due to the fact that many of the controls have only recently been implemented and the population from which to draw a sample would be limited. Where applicable, examples (i.e. completed templates) were reviewed as part of the design assessment.

Table Description: The table below provides a list of four possible ratings with the descriptions for each rating.

Rating Description
Complete All criteria were met and required design elements of the recommendation have been fully implemented.
Substantially Complete Although not fully complete, the vast majority of criteria were met and the resources required to complete the work are minimal.
Partially Complete Some of the criteria were met, however, many of the design elements of the recommendation have not been implemented and the resources required to complete the work are considerable.
Incomplete Few to none of the criteria were met and significant resources are required to fully implement all design elements of recommendations.

Given the pace of change across government since the audit, the scope of this engagement also included an assessment of:

  • the steps being taken to streamline the 379 systems in the Department through integration; and
  • the extent to which IM/IT plans/planning processes are identifying readiness related risks and challenges as the Department and government move to greater centralization of IT services and applications.

During the planning phase, a risk assessment was performed which included the review of documentation and preliminary interviews. An assessment program was then established to assess the implementation status of each audit recommendation. The assessment phase of the engagement consisted of interviews and documentation review, as well as a walkthrough of controls.

2. OBSERVATIONS AND ASSESSMENT

Overall, significant improvements to the Department's IM/IT project life cycle controls were observed during the follow-up engagement. Streamlined processes, enhanced reporting mechanisms and improvements to accounting procedures were noted. Also, prior to approval, proposed new projects now require more detailed options analysis and measurable outcomes. At the time of this assessment there were 11 IM/IT projects underway in the Department with a total estimated cost (TEC) of approximately $22M, compared to the 43 IM/IT projects with a TEC of $97M at the time of the original 2011 audit. In addition, plans are starting to be developed to consolidate, rationalize and integrate the Department's systems in order to achieve greater efficiencies.

Recommendation # 1

The Deputy Minister should ensure there is an approved IM/IT plan for the Department, that this plan is updated regularly, and that there are regular reviews of performance against the plan.

Assessment of implementation status: Complete
Significant work was completed in order to create detailed strategic plans for both 12/13-15/16 and 13/14-16/17 as well as a mid-year update for 12/13 (the mid-year update for 13/14 is scheduled for November 2013).

Observations

Evidence was provided to support that TIMSD submitted the 12/13-15/16 plan and the 12/13 IM/IT Strategic/Investment Plan mid-year report (mid-year report) to TMX; however, there was no evidence within TMX decision records of a presentation, discussion or decisions related to the documents.TMX decision records clearly provide evidence of presentation and discussion of the 13/14 -16/17 plan.

The 12/13 mid-year report provided a dashboard of the status of six performance indicators: scope, schedule, budget, operational requirements, risk management and business fit. The mid-year report also included an appendix with a summary of each project. However, the status of individual projects was not included in the mid-year report.

Opportunities for Management Consideration

  • Determine if the plan and mid-year report should be presented and discussed at TMX, in keeping with best practices, and if so, ensure that decision records clearly identify resulting discussion, review and decisions.
  • Review the content and format of the mid-year report and consider adding information to more clearly show the performance for each project (for example, a one page standard report for each project).

Recommendation # 2

The ADM, Corporate Services, should provide TMX annually a list of all recommended IM/IT project proposals with sufficient information (e.g. whether or not the proposed project can be obtained at a lower cost through adaptation of an existing application) to allow for informed decisions and oversight.

Assessment of implementation status: Substantially Complete
A list of proposed future investments is submitted as part of the IM/IT strategic investment plan, an improvement over the information provided in the past. An appendix to the report now includes additional details for each project such as project description, expected outcomes, and total estimated cost; however, there is no information with respect to options analysis that would determine whether or not the proposed project could be obtained by adapting an existing application.

Observations

A proposed future IM/IT investments listing (list of investments) is part of the IM/IT Strategic/Investment Plan. The scope of the information provided to TMX to support decision making has been enhanced since the audit. Project descriptions, expected outcomes and objectives are now included. However, the list of investments does not include information regarding whether or not the project could be adapted from an existing application. The estimated timeframe is also not included. We observed that a high level options analysis is a mandatory component of the Investment Project Justification Template (Justification Template), and a more detailed analysis is part of the Project Approval Document (PAD). As the PAD is a very detailed document, it is not completed until a project is notionally approved by TMX. However, the high level options analysis within the Justification Template is completed prior to preparation of the list of investments that is sent to TMX, but this analysis is not included.

In a review of three PADs it was observed that the quality of the information within the options analysis was enhanced since the time of the original audit. Examples of options considered included purchasing Commercial-Off-the-Shelf (COTS) systems as well as enhancements to current systems already in place.

Opportunities for Management Consideration

  • Include the options analysis in the proposed future IM/IT investments listing that is provided to TMX as well as an estimated project timeframe/schedule. Ensure the information is sufficient to assess alignment to the IM/IT strategy, including recent direction on multi-modal requirements.

Recommendation #3

The ADM, Corporate Services supported by the CIO and the DG, Finance and Administration, should review the control framework with the view to making it workable by streamlining it and addressing fundamental gaps. In particular, the project management framework should be scalable to take account of project size, nature, complexity and risk.

Assessment of implementation status: Complete

Extensive work was completed to improve the IM/IT project life cycle controls. The improvements resulted from a collaborative effort that included a wide range of internal stakeholders, revision of documentation, communication and training for both NCR and regions. This work has resulted in a clear and concise IM/IT project life cycle process and procedures.

Observations

Both Finance and TIMSD performed and documented a review of the various policies and procedures that formed their respective project process documents, TP117 Chapter 210 Project Approval Process and IM/IT Project Management Framework, in order to simplify and streamline the processes. Both Groups used a strong project management process in performing this work by actively engaging stakeholder review and feedback, documenting meetings and working papers, formally communicating the new/revised processes, and providing training sessions.

The TC TP 117 Chapter 210 Project Approval Process now provides all the required information including links to templates and related processes.

The IM/IT Project Management Framework is a single clear and concise reference document that describes the project management processes in appropriate detail from planning to closure and lessons learned. It also provides links to all templates and related processes.

Requirements related to the Department's System Development Life Cycle are clear and mandatory documents have been established.

Opportunities for Management Consideration

N/A

Recommendation #4

The ADM, Corporate Services, supported by the CIO, should significantly strengthen monitoring of and reporting on IM/IT projects and, effective immediately, report bi-annually to TMX and, when significant risks are identified report on these more frequently. Progress reports should identify all scope, schedule and budget changes and significant issues/risks.

Assessment of implementation status: Partially Complete
Project Overview Reports, as well as quarterly financial update reports together identify scope, schedule and budget changes and significant issues/risks to IM/IT projects underway in the Department. This has enhanced the visibility of the status of IM/IT projects within the Department.

Since revisions to the IM/IT governance model are still ongoing the implementation status of this recommendation is assessed as partially complete.

Observations

Many improvements have been made to project monitoring information. A quarterly Project Overview Report is prepared for each of the three SO committees as well as the Internal Services Management Board (ISMB). This report provides IT project performance information on the status of six performance indicators, a financial overview, and variance explanations. A Project Overview Report summary is also provided to the ISMB. In addition to the overview report, the CIO presents a quarterly deck to ISMB, providing the status of IM/IT projects, as well as a summary listing of recommended actions (i.e. continue project, discontinue project) for each IM/IT project.

As directed by TMX, the Project Overview Reports are sent to the ISMB, SO planners and SO committees but not to TMX. Although this process varies somewhat from the original audit recommendation, the information is reaching most of the intended audience as all TMX members are members on the SO committees and ISMB, with the exception of the Deputy Minister and Associate Deputy Minster.

In addition, TIMSD provides the Project Overview Reports to Corporate Finance that in turn summarizes the information into a quarterly financial update report that presents the status of all departmental projects, including IM/IT, to TMX. However, Corporate Finance's report includes only a high level summary of the performance indicators (e.g., green, yellow or red status for budget, scope, risk, etc.) and does not indicate the status of individual projects. As this is a Finance presentation, the format and content is determined by the Corporate Finance group.

While reports prepared for ISMB and TMX show the status of scope, schedule and cost; if a project has been revised (via a PAD revision) the status indicators are based on the revised scope, schedule and cost, not on the original. As a result, the Department does not have a complete picture of a project's performance when it has a revised PAD.

Finally, TIMSD updated the IM/IT governance model for the Department based on consultations with TMX members and stakeholders. However after it was presented to TMX in April 2013, TMX requested that the model be refined to clarify and detail TMX's overall role. Now that the Department is taking steps to establish a comprehensive business and system architecture strategy, the governance model is being reviewed to take this strategy into consideration. The target is to present a revised governance model to TMX in December 2013.

Opportunities for Management Consideration

  • Review the scope and format of the information TMX receives on the performance of IM/IT projects and determine if it is adequate.
  • Track project performance against both original and revised (if applicable) scope, schedule and costs to provide a more complete picture of the outcomes of IM/IT projects.

Recommendation #5

ADM, Corporate Services, supported by the CIO, should ensure that each IM/IT project proposal to TMX has clear measurable outcomes, that there is a post-project review of every IM/IT project, and that findings of post-project reviews are provided to TMX.

Assessment of implementation status: Complete
Significant improvements have been made to the measurement of performance outcomes for IM/IT projects. An executive summary report of business performance measurement outcomes has been developed and is provided to TMX for their review.

Observations

As per the MAP, an update to the Project Approval Document's Performance Outcome Measurement instructions to support Project Sponsors in developing measurable outcomes was made. While the information is somewhat limited (for example it does not provide examples of good measurements) the document does direct individuals to consult the Enterprise Project Management Office for additional support to define performance outcomes. Updates were also made to the IM/IT Management Framework to provide performance measurement guidance. Five recent PADs were reviewed and we observed that all five had measureable outcomes with information on how each outcome would be measured, demonstrating an improvement since the original audit.

A formal project close process, including a mandatory close-out report, was established and an annual report has been developed that provides a high level summary of the achievements of each project, highlighting the qualitative value of the project to the Department and/or program. It was observed, however, that two projects that closed during the time period of the report were not included. The last report was sent to TMX secretarially, with the option for the CIO to present the findings at a future TMX meeting.

Opportunities for Management Consideration

  • Report on all projects that have been completed during the period covered by the annual report, even if the close-out report is outstanding.

Recommendation # 6

The ADM, Corporate Services, should ensure controls are sufficient to provide reasonable assurance of the accuracy of the IM/IT project costs within its financial statements.

Assessment of implementation status: Substantially Complete
The monitoring of Work in Progress (WIP) accounts, not only for IM/IT projects but for all departmental projects, has been significantly improved compared to the process in place at the time of the original audit. A weakness was identified in the process related to the timing of clearing completed projects from the WIP account to the asset account.

Observations

A quarterly listing of capital projects in WIP is maintained and regularly reviewed against the information in the Department's financial system (i.e., Oracle). This is a significant improvement compared to the process in place at the time of the original audit.

As per the MAP, Finance was to "Request quarterly confirmation of Work-in-Progress (WIP) project status by FMAs/RCFAs to ensure timely monitoring, reconciling and clearing of WIP projects". While quarterly requests are sent out, Finance does not ensure that completed projects are transferred from WIP to the Asset module in a timely manner. According to the Department's Daily and Period-end closing procedures: "Capital assets related to completed WIP projects will be recorded in the Asset module in the period in which the assets are placed in service (productive use)." Upon review of the WIP status listing it was observed that 12 IM/IT projects with a value of approximately $19M, completed as at March 31, 2013, remained in WIP as of August 13, 2013. By the end of August, 4 of these projects with an approximate value $7M were moved from WIP and put in the Asset module.

Internal Audit was able to verify that a clean-up project of the WIP clearing accounts was undertaken. This was evidenced by working papers provided by Finance. As a result of the clean up, approximately $40M was removed from the WIP.

Given the significant amount of the adjustment that resulted from the cleanup exercise we asked whether or not an analysis had been prepared to assess the impact of the adjustment on the financial statements or if the adjustment was documented within the schedule of misstatements, as required by the Departments' Chapter 610 Materiality and Departmental Financial Statements policy. According to the Policy, the Director, Accounting Operations is required to prepare a consolidated schedule of misstatements greater than $100k and this schedule is to be used by the Director to assess whether or not the statements are materially misstated. We were informed that the schedule had not been prepared for the last few years nor had an analysis been prepared on the adjustment.

Opportunities for Management Consideration

  • Enhance the current process by establishing procedures to provide greater assurance that at year-end all completed projects are accurately recorded in the asset module. In addition, review and update, as necessary, procedures documentation (i.e. current requirement to move completed projects from WIP to asset module in same month they are completed) to ensure alignment with procedures in place and that roles and responsiblities are clear and understood.
  • Re-establish practice of completing schedule of misstatements as part of year-end procedures and establish a process to ensure errors/control weaknesses are communicated appropriately within the Department, including the Internal Controls over Financial Reporting group.

3. INTEGRATION AND CENTRALIZATION OF IM/IT RESOURCES

The scope of this progress assessment included an assessment of the steps being taken to streamline the systems in the Department through integration and the extent to which IM/IT plans/planning processes are identifying readiness related risks and challenges as the Department and government move to greater centralization of IM/IT services and applications.

To complete this assessment, Internal Audit conducted interviews with members of TIMSD and TMX, reviewed documentation including the 2013/14-16/17 IM/IT Strategic/Investment Plan and attended meetings with TIMSD and the contractor engaged to develop the architecture strategy for the Department.

Observations

Strategic Plan

The purpose of the IM/IT Strategic/Investment Plan is to provide departmental direction in the area of IM/IT in support of efficient and effective program delivery, managing risks and providing continued service levels for business operations.

The 2013/14-2016/17 IM/IT Strategic/Investment Plan (Plan) is a substantial document representing significant effort and thought. As there was no Treasury Board Secretariat (TBS) process or format for TIMSD to follow for an IM/IT strategic planning process and plan, TIMSD created their own. TBS reviewed TC's plan as part of the annual Management Accountability Framework assessment and determined that the Department had met TBS requirements.

The Plan includes elements of strategic initiatives, a report on accomplishments and IM/IT project investment priorities. The amount of detail, as well as the portion of the plan reporting on TIMSD accomplishments, can be distracting to the reader.

In addition, the IM/IT vision described in the Plan does not clearly address all the strategic initiatives underway in the Department. For example, although revised governance and investments are addressed in the Plan, initiatives underway to consolidate and rationalize the Department's systems are not. Based on interviews with TIMSD, at the time the Plan was being finalized, these aspects of the Department's vision for IM/IT had not been determined. The concepts of consolidation, rationalization and integration are now well established and work is underway to support these initiatives. For example, TIMSD has prepared high level plans including cost estimates for the Department to transition from Oracle to SAP as well as from its HR application, TIPS to PeopleSoft. In April of 2013, TIMSD made a presentation to TMX on the plan to move to SAP, including identification of the 20 applications currently integrated with Oracle.

The consolidation of application development and support in the Department, (as per TC's Deficit Reduction Action Plan) is progressing according to the schedule provided by TIMSD. The majority of applications and support staff are now members of TIMSD and as a result of the consolidation TIMSD has decommissioned 27 applications to date.

A cross-section of stakeholders, including TMX, were asked to provide feedback on the draft Plan. Based on a review of the disposition paper used to track comments, TIMSD received feedback from only one TMX member. One-on-one interviews were not held with TMX members as part of the strategic planning process due to time constraints associated with a new requirement to prepare the Plan and proposed investments at the same time. In the past, the Plan was prepared first and then proposed investments in support of the plan were put forward at a later date. However, it was noted that one-on-one interviews were held with some TMX members as part of the work to revise TC's IM/IT governance framework.

A TC IM/IT risk profile was not developed as part of the Plan. Instead, IM/IT-related risks from the Internal Services risk profile were used. As a result, some risks we would have expected to be addressed by the Plan were not. For example the ability of the Department's information systems to support the Department's operational decisions and performance reporting was not addressed in the plan.

Change Management was identified as a risk in the Plan and other risks, such as transitioning to Shared Services Canada and workforce capacity, also noted aspects of change management. Mitigation actions related to change management cited implementation of specific IM/IT processes such as portfolio management; however, the Plan did not indicate that a change management process at the departmental level was in place (or was needed).

Work underway to Develop an Architecture Strategy

Consistent with government-wide direction to reduce costs and generate efficiencies, Transport Canada initiated a process to examine the Department's current business and systems architecture in order to make improvements to ensure it supports the fulfillment of the Department's mandate.

In fall of 2013 TIMSD engaged a contractor to develop a strategy to develop departmental business and systems architectures. A business architecture shows what state a business wants to achieve, as well as its current state. A systems architecture depicts the IM/IT processes and systems that support the business. In terms of "best practice", the process starts with the business first, then information, security, technology and application/ systems are identified to support the business. The strategy will serve as the foundation for further rationalization of the Department's applications as well as the implementation of other government initiatives such as centralization and integration. While the current effort is to develop a strategy for establishing architecture, implementation will take multiple years and likely have a significant cost.

In addition to leading the systems architecture initiative, TIMSD has been asked to recommend to TMX a strategy to establish a business architecture. According to the project plan, consultations are to be scheduled with TMX members, TC program area representatives, and IM/IT practitioners in order to document the current architecture and identify a strategy for future architecture and high level plans to implement the strategy.

As of late September, the process to establish the architecture strategy had just started and therefore the opportunity to assess progress was limited. TIMSD did have a project plan with established timelines and detailed consultation schedules with IM/IT practitioners, TC Program Area representatives, and TMX members. Sessions with IM/IT practitioners were underway and sessions with program area representatives and TMX members were scheduled for early October. TIMSD has stated that they are committed to complete the Strategy Report and present it to TMX in November.

Opportunities for Management Consideration

Strategic Plan

  • In addition to engaging the Strategic Outcome Management Boards, consult directly with TMX members one-on-one to develop the IM/IT Strategic/Investment Plan.
  • Conduct a risk assessment of TC IM/IT risks. Ensure the risk assessment adequately addresses risks associated with current initiatives underway within the Department such as the Department's capacity to rationalize the Department's applications.
  • Articulate an end-state vision for IM/IT at TC that encompasses all strategic initiatives, including many that are currently underway (i.e. consolidation/ rationalization/integration of applications). In addition, ensure the Plan incorporates and aligns to the approved architecture strategy.
  • Review the Plan's content and format to ensure that the key IM/IT strategic information is the primary focus.

Architecture Strategy

  • Ensure this significant departmental initiative is driven and fully supported by TMX. Clearly define, as part of the Strategy Report and resulting implementation plans, the role of TMX, how support and direction will be communicated by TMX within the Department, and how TMX will monitor and measure progress.
  • Appoint a business champion (TMX level) to work with TIMSD as soon as possible to ensure that the business architecture is business driven and input from the business lines is captured both during the development and implementation of a business architecture strategy.
  • Ensure that the systems architecture addresses all key strategic initiatives underway and planned (consolidation, rationalization, integration, multi-modal requirements, etc.).
  • Ensure that the architecture strategy report is written in clear, non-technical language. In addition, consider:
    • including clear, brief definitions of terms such as consolidation, rationalization, integration, etc, based on how terms are defined at Transport Canada and ensure consistent use of the terms in related documentation; and
    • obtaining feedback on the Strategy Report from a select group of employees outside TIMSD to ensure it is clear and easy to understand.
  • Ensure that change management needs are clearly identified and form part of the Strategy Report and resulting implementation plans.