Audit of Spectrum Application Modernization—Commercial Software Implementation

Recommended for approval to the Deputy Minister by the Departmental Audit Committee on July 12, 2012

Approved by the Deputy Minister on July 17, 2012

Table of Contents

• 1.0 Executive Summary
  • 1.1 Introduction
  • 1.2 Main Findings and Recommendations
  • 1.3 Audit Opinion
  • 1.4 Statement of Assurance
• 2.0 About the Audit
  • 2.1 Background
  • 2.2 Objective and Scope
  • 2.3 Audit Approach
• 3.0 Findings and Recommendations
  • 3.1 Introduction
  • 3.2 Change Management
  • 3.3 User Requirements Management and Solutions Definition
  • 3.4 Project Management
  • 3.5 Vendor and Contract Management
• 4.0 Overall Conclusion
• Appendix A: Audit Criteria
• Appendix B: Management Action Plan

1.0 Executive Summary

1.1 Introduction

The Spectrum Application Modernization – Commercial Software Implementation (SAM-CSI) project represents a major effort to stabilize and replace the existing legacy Spectrum Management System (SMS). It is one of the largest systems initiatives undertaken by Industry Canada in the past decade, and is a key departmental priority. The project is administered by the Spectrum Management Operations Branch in the Spectrum, Information Technologies and Telecommunications Sector (SITT).

The SAM-CSI project was conceived to address the impact of aging IT applications on the Spectrum Management program. The diverse and complex SMS has become difficult to maintain and is a limiting factor in the continuing evolution of the program.

SAM-CSI is pursuing a "Commercial-Off-The-Shelf" (COTS) software solution, as opposed to in-house custom development, to replace the outdated SMS system. The future system will provide new ways of issuing and managing radiocommunications licences in real time with a more streamlined automated process and a more stable core system. The vendor selected to provide the solution has an international client base, which will allow the program to leverage global expertise in Spectrum Management.

The SAM-CSI team includes SITT employees from headquarters and across the regions. The team regularly consults with program experts for input and advice. In addition the project has a strong internal governance structure and has a dedicated project management office that oversees the day to day operations of the project.

The SAM-CSI project plan sees the effort divided into planning, project definition and implementation phases.

• During the planning phase, the program defined its operational requirements and compared them with potential vendor solutions based on documentation received through a Request for Information.
• The project definition stage, referred to as Release 0, includes an examination of all aspects of the vendor's suite of software products. The main activities of this stage include a Gap-Fit Analysis to confirm software functionality and how it fits with the program's business requirements, and Proof-of-Concept activities to explore and test a few key technology-oriented requirements. Where the selected off-the-shelf product does not incorporate all the features required by Industry Canada, the project's strategy is to address the balance firstly through business process change, and if that is not feasible, through software customization. The Release 0 activities provided the required information for implementation and were used to support the Effective Project Approval (EPA) submission process, which was successfully granted ahead of project schedule. Furthermore, the project has been granted stage gate 3 approval by the department's Project Oversight Committee. Both the EPA and stage gate 3 approval represent significant milestones for the project as it signals that the project can proceed to the implementation/execution phase.
• During the implementation phase, the project aims to test and roll out the software in a phased approach, with the final release scheduled for November 2016.

In accordance with the approved Industry Canada 2011-14 Multi-Year Risk-Based Audit Plan, the Audit and Evaluation Branch undertook an audit of the Spectrum Application Modernization – Commercial Software Implementation project.

The objective of this audit was to assess the adequacy and effectiveness of controls supporting the achievement of pre-established outputs and outcomes of the SAM-CSI project in the areas of:

• Change management
• User requirements management and solutions definition
• Project management (scope; schedule; cost; quality; risk)
• Vendor and contract management.

This audit revealed that the SAM-CSI project has established, with some exceptions, adequate and effective controls to support the pre-established outputs and outcomes in the areas of: change management, user requirements and solutions definition, project management, and vendor and contract management.

1.2 Main Findings and Recommendations

Change Management

A change management network including Regional Champions and Program Ambassadors has been established and supports the change management initiative.

Ongoing assessment activities of the organization's capacity and readiness for change have taken place and are planned to occur over the course of the project.

While a change management strategy exists, the project needs to develop a formal, comprehensive plan, with identified resource requirements, that addresses the impact on the entire organization, over the period of change.

Recommendation 1:

The Project Director should ensure that a change management plan that addresses the impact of change on the entire organization be finalized and identify resource requirements.

Certain activities that support communication initiatives have been delayed.

Recommendation 2:

The Project Director should ensure that resources are in place to support communication initiatives for change management.

User requirements management and solutions definition

Business requirements were defined, reviewed and analyzed by appropriate stakeholders during the planning and definition phase of the project.

Business requirements have been defined to allow the project to assess fundamental differences between the current system and the proposed COTS product. Detailed requirements will be defined and tested in the next phase of the project.

Project Management (scope; schedule; cost; quality; risk)

Appropriate tools and processes are in place to help the project monitor the overall project status and ensure that customer and stakeholder requirements and outcomes are met.

There are sound processes in place to help define and control the scope of the project.

A Quality Management Plan was not completed and no Quality Assurance audits had been performed at the time of the audit.

Recommendation 3:

The Project Director should complete the Quality Management Plan. In addition, Quality Assurance audits should be scheduled at project milestones and integrated within the project schedule.

The SAM-CSI project team has key elements in place to manage the budget and ensure that project expenditures are in line with the spending authority.

There are cost discrepancies between key project reports presented to Senior Management and Oversight bodies due to different sources of cost information being used to generate reports.

Recommendation 4:

The Project Director should ensure that project cost information reported to stakeholders is consistent with departmental financial system data.

The SAM-CSI project team has sound processes and tools in place to support effective risk management.

Vendor and Contract Management

Appropriate Processes are in place to manage external vendors and monitor compliance with contractual requirements.

The Contract Deliverables Requirements List includes detailed descriptions of both the Conference Room Pilot (CRP) and Proof-of-Concept (PoC) activities. These are key activities defined by the project team that facilitates an evaluation and testing of the COTS software.

There is no formal sign-off on decisions regarding business process changes or software customization; some post Special Spectrum Operations Committee (sSOC) consensus sheets had outstanding decisions to be made for some business requirements requiring customization.

Recommendation 5:

The Project Director should ensure that all post special Spectrum Operations Committee consensus sheets are finalized and formally approved prior to the development of detailed requirements associated with each release phase.

1.3 Audit Opinion

In my opinion, the SAM-CSI project has established adequate and effective controls and processes with no material weaknesses. There are medium to low risk exposures related to certain control activities where there are opportunities for improvement.

1.4 Statement of Assurance

In my professional judgment as Chief Audit Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided and contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed on with management. The opinion is applicable only to the entities examined and within the scope described herein. This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada.

Susan Hart
Chief Audit Executive, Industry Canada

2.0 About the Audit

2.1 Background

In accordance with the approved Industry Canada 2011-14 Multi-Year Risk-Based Audit Plan, the Audit and Evaluation Branch (AEB) undertook an audit of the Spectrum Application Modernization – Commercial Software Implementation (SAM-CSI) project.

The telecommunications infrastructure and radio spectrum are resources that require attentive and efficient management within both a domestic and a global context. Their effective use is crucial to the social and economic well-being of Canadians. Management of these resources is a federal obligation that is performed by Industry Canada through its Spectrum/Telecom program.

The Spectrum/Telecom program generates approximately $260 million a year in revenues from radio licences and fees. More than 70,000 Canadians and businesses have radio licences.

The program's existing Spectrum Management System (SMS) is the fundamental tool used by program staff to manage the radio spectrum and to deliver licensing services to its clients across Canada.

The SAM-CSI project seeks to address the shortcomings of the SMS, including the system's limitation in meeting new program requirements, by purchasing and deploying a "Commercial-Off-The-Shelf" (COTS) software solution that satisfies a large portion of SMS requirements. The project is administered by the Spectrum Management Operations Branch in the Spectrum, Information Technologies and Telecommunications Sector (SITT).

The project plan states that, where possible and cost-effective, the program will identify, plan and implement business process changes to reflect the functionality available "out-of-the-box" in order to minimize the need for software customizations. In the first instance SAM-CSI will adopt internal measures to minimize the need for any such custom development. The Director, Spectrum Operations is accountable for working closely with SAM-CSI to proactively identify potential business changes, and for working with the national Spectrum Operations Committee to confirm, plan, and implement such changes.

The project has previously received Preliminary Project Approval (PPA). Fully loaded project costs are estimated at $55 million. The project is intended to be funded from within existing departmental reference levels. Recently, the project received Effective Project Approval (EPA) for the implementation phase.

Project Governance Structure

Project governance for SAM-CSI is composed of various oversight streams as shown below:

• A dedicated Project Steering Committee includes the Chief Financial Officer (CFO), the Chief Information Officer (CIO), the Chief Executive Officer of the Canadian Intellectual Property Office and representatives from Treasury Board Secretariat (TBS).
• The departmental Project Oversight Committee oversees the project through the department's stage gating process.^{Footnote 1}
• TBS Executive Project Oversight Committee and includes TBS's gating process for IT-related projects.

The Project Steering Committee provides advice and oversight to the ADM-SITT, who also chairs the committee. The committee has delegated authority to provide Program Services Board approval to the project in matters pertaining to project management and procurement.

At the departmental level, the project presents regularly to the Information Technology Strategic Management Committee to ensure its activities are aligned with departmental IT investments.

The Core Team working under the direction of the project office consists of specialists and subject-matter experts drawn from units within SITT that have a direct interest in the operations of the SMS. Regional and operational experts have joined the project full-time as Core Team members.

Project Definition Phase—Release 0

The SAM-CSI project is currently in the Release 0 phase of a proposed multi-phase implementation. The first vendor contract was signed in August 2011 for Release 0 activity. This phase – scheduled to end in June 2012 – focuses on the evaluation of the proposed software solution that meets the specified needs of the business owners. The introduction of a COTS solution often necessitates a trade-off between functionality and business process change.

As part of the Release 0 phase, Industry Canada along with the vendor, is conducting a Gap-Fit Analysis via a series of Conference Room Pilots to clarify the difference between what the vendor's COTS solution offers and what the organization needs. The project is also conducting Proof-of-Concept activities that evaluate the implementation of essential aspects of the technical architecture of the COTS solution.

Current Developments

Public Works and Government Services Canada (PWGSC) and the project team are negotiating the Pilot Release contract with the vendor. The contract will include the Pilot Release phase and subsequent releases. The Pilot Release will establish the groundwork for the development of subsequent releases as increased functionality is built into the product.

The SAM-CSI project has been granted stage gate 3 approval by the department's Project Oversight Committee. The project has also received Effective Project Approval, enabling the implementation phase to begin.

2.2 Objective and Scope

The objective of this audit was to assess the adequacy and effectiveness of controls supporting the achievement of pre-established outputs and outcomes of the SAM-CSI project in the areas of:

• Change management
• User requirements management and solutions definition
• Project management (scope; schedule; cost; quality; risk)
• Vendor and contract management.

The audit scope included document review, interviews of Industry Canada staff in the National Capital Region and in the regions, and interviews with the vendor, PWGSC, the CFO, Shared Services Canada and the CIO. The audit covered project activities up to March 31, 2012. Where possible, AEB reviewed documentation and activities that went beyond this date to substantiate findings.

Data migration activities have been excluded from the audit scope because the activities are scheduled to occur during the implementation phase of the project, thereby falling outside the time frame of this audit.

The SAM-CSI project team has assessed data migration as one of its top risks that could cause delays and cost overruns, given the complexities of the current SMS data model. Recognizing this risk, the project office has conducted preliminary data migration work through a Proof-of-Concept during the Release 0 phase, to gain a better understanding of the work required to perform a full scale data migration.

2.3 Audit Approach

This internal audit was conducted in accordance with the TB Policy on Internal Audit and the Internal Auditing Standards for the Government of Canada.

The planning phase for this audit took place from January to March 2012. AEB performed a detailed risk assessment to confirm the audit objective and key audit areas. Based on these key areas, AEB developed audit criteria reflecting both Treasury Board Secretariat Standards and the COBIT (Control Objectives for Information and Related Technology) Framework for IT Governance and Control. Appendix A of this report lists the audit criteria.

The conduct phase for this audit took place from March to May 2012. Audit testing included reviews of SAM-CSI project documentation, such as plans; documentation for EPA and PPA submissions; meeting minutes (e.g., Project Steering Committee meetings) and records of decision (e.g., conference room pilot consensus documents); reports (e.g., risk register; schedule and budget); and selected communications (e.g., e-mails and presentations).

The sample sizes for the audit tests were determined in relation to the frequency of the control (e.g., annual, quarterly, monthly, weekly, daily, or many times per day). Judgmental and random selection methods were used in selecting samples. The timeframe for the selection of samples varied based on the nature of the testing.

A debrief meeting was held with management on May 31, 2012 to validate the accuracy of findings contained in this report.

3.0 Findings and Recommendations

3.1 Introduction

This section presents detailed findings from the audit of the Spectrum Application Modernization – Commercial Software Implementation project. The findings are based on evidence and analysis from both the initial risk assessment and the detailed audit work.

In addition to the findings below, AEB has communicated to management findings of conditions that were of minor impact, verbally and/or in a management letter, for consideration.

3.2 Change Management

To accommodate and maximize the benefits of a COTS solution, the Spectrum Management program will have to implement business process changes. As a result, change management is necessary to address the organization's readiness for and acceptance of new IT application tools as well as business process changes.

In support of the change management initiative, the SAM-CSI project has the backing of key decision-makers as well as an established change management network as described below:

• The executive sponsor of the project is the Senior ADM of SITT and the project sponsor is the Director General of the Spectrum Management Operations Branch (DGSO). Both have signed the project charter. This document describes responsibilities and accountabilities for the project and ensures agreement from the key stakeholders on the principles and terms of the project.
• The Senior Director of Spectrum Management Operations (DOS), who reports to the DG of DGSO (the top level business owner for Spectrum Operations), is responsible and accountable for decisions regarding business processes related to the SAM-CSI project. The Senior Director of DOS chairs the Spectrum Operations Committee (SOC), whose membership includes regional directors and managers. Therefore, the Senior Director of DOS in alliance with SOC maintains the link between the sponsors and managers responsible for national operations.
• A Change Management advisor/lead within the Project Organization reports directly to the SAM-CSI Project Director. His role includes defining communications and training strategies as well as drafting and implementing the change management plan. The change management network also includes Program Ambassadors and Regional Champions.
  • Program Ambassadors are the primary project communicators within their region. They were involved in the planning and definition phase of the project as subject matter experts. This provided them with a good knowledge of the system. They also participate in the development and delivery of communications and presentations. The intent is to have these experts and ambassadors build consensus and buy-in with their peers for the COTS solution.
  • Regional Champions are district directors who are known and respected in the program areas affected by the project. They provide advice and guidance on specific information requirements to meet regional needs.

The change management network informs all program staff of specific change initiatives, either formally through presentations and meetings or informally through peer discussions. Presentations were held in the regions in December 2011. Additional project updates and software demonstrations have begun for all regions and are scheduled to be completed in May-June 2012. The purpose of these communications is to foster buy-in to the project and reduce the risk of organizational resistance to business changes.

To address organizational readiness, the project engages the special Spectrum Operations Committee (comprising regional managers) in its business decisions; change management is now a standing agenda item at the DGSO meeting, which includes regional directors. Furthermore, the project office regularly provides detailed project status reports to program management.

The project has undertaken initial steps to assess the organization's capacity and readiness for change. Examples include:

• During the planning phase, the program defined and compared its operational requirements with potential vendor solutions based on documentation received through a Request for Information. This process involved subject matter experts from the regions as well as headquarters. According to SITT management, this work provided the program an opportunity to garner support or buy-in for the COTS solution.
• Presentations have been and will continue to be rolled out to the regions. According to the Change Management lead, they provide a good way of fostering a common understanding of the project and receiving feedback.
• The Change Management lead also obtains feedback from the regions regarding acceptance and readiness for the COTS solution through the intermediary of the change management network and management meetings.
• A training strategy for the Pilot Release was developed and presented at the DGSO meeting. According to the strategy, the vendor will train Industry Canada trainers who in turn will train the staff.

As stated in the EPA business case, an organizational readiness plan will assess organizational capabilities and readiness to change. For the plan, SAM-CSI will assess the willingness and ability of the business units, SMS users, and clients to accept the new COTS solution.

The project has developed a change management strategy that speaks to change management requirements such as training, communications and organizational readiness. The strategy calls for a change management plan to be finalized. The program has indicated it is developing a formal plan.

Key lessons learned from the previous initiative to replace SMS pointed to the lack of a formal change management plan during that initiative. EPA documentation acknowledges that (project) success will depend on effective management of the transition through substantial and sustained effort over many years.

The plan should include appropriate resources for work related to business process changes that fall under the responsibility of the program. These changes to the Spectrum Management program's business model are a prerequisite to successful SAM-CSI project implementation. Currently, change management resources accounted for by the project are limited to training costs, as well as several key change management facilitator roles in the Project Office, but do not include resources for implementing business process changes.

With the absence of a formal and comprehensive change management plan, there is a risk that the organization may not be able to adapt to the required change over an extended period.

Recommendation 1

The Project Director should ensure that a change management plan that addresses the impact of change on the entire organization be finalized and identify resource requirements.

Communication is an important element in the overall change management initiative to keep SITT employees up to date and foster support for the project. Some of the communications tools include a project web site, regional visits and presentations.

Audit interviews indicated that key communication deliverables such as updates to the web-site have been delayed. The Communications and Change Analyst position was vacant at the time this report was written. The program has set aside a budget for this position and is engaged in the staffing process. The responsibilities of the position include involvement in the development and preparation of project communication and training tools, including presentations, newsletters, regular communiqués, manual, etc., and creation and maintenance of the project intranet site.

There is a risk that information critical to achieving staff buy-in may not be communicated as early as possible, thereby impacting the effectiveness of stakeholder communications.

Recommendation 2

The Project Director should ensure that resources are in place to support communication initiatives for change management.

3.3 User requirements management and solutions definition

Defining business requirements and expectations and communicating them amongst the relevant parties is a key step in achieving the expected benefits of a COTS solution. Business requirements, if not defined and managed properly, may adversely affect the project budget or delay the achievement of project outcomes.

The audit examined the SAM-CSI approach to defining, reviewing and analyzing business requirements over two phases.

During the project planning phase (Pre Release 0):

• Business requirements were defined, reviewed and analyzed through the AS-IS, FIT-DIF and Operational Review (OR) processes. These activities included consultation across SITT (including regions) and involved business owners and subject matter experts.
• The AS-IS process developed a comprehensive inventory of what the Spectrum Management System users need (i.e. a baseline of how the business operates today). It also highlighted opportunities for improvement.
• The FIT-DIF process identified where potential COTS solutions met or diverged from Industry Canada requirements by referencing documentation obtained through a Request for Information.
• The OR process examined the results of the FIT-DIF analysis. Where differences were identified, the OR examined options, including changes in operations/business processes or customizations to the vendor product. The OR team presented its recommendations for consideration by the Spectrum Operations Committee, chaired by the Senior Director of Spectrum Management Operations. SOC's membership includes regional representation, engineers and regulatory experts.

During the project definition phase (Release 0):

• The project team is conducting a Gap-Fit Analysis via a series of conference room pilots (CRPs). The project team and the vendor compare business requirements (approximately 2,300) with features available in the COTS software solution. CRPs are held to identify areas where the software capabilities meet or diverge from the stated business requirements for the program's eleven functional areas. The project team includes Industry Canada business and subject matter experts and regional representation. The outcomes of each CRP are captured in a consensus work sheet document that describes how requirements will be addressed.
• Gaps identified through the CRPs will be resolved either through business process changes or by customizing the software. The project aims to minimize changes to the COTS base system by accepting customization only for mandatory requirements (i.e. legislation, regulation and policy) or when it provides a cost benefit to the business.
• CRP consensus documents, augmented by presentations that describe proposed business process changes and customizations, are presented to the special Spectrum Operation Committee for its approval. Special SOC was created to ensure the right people are in attendance for consideration of a particular business line. Membership includes regular SOC members plus part time subject matter experts who attend meetings relevant to their area of expertise.

Based on the above activities, the audit team found that business requirements were defined, reviewed and analyzed in the planning and definition phase of the project.

The business requirements being reviewed in the CRPs were written at a high level and do not include detailed business rules. For example, a high level business requirement could be that documents need to be scanned and converted to an electronic format. For the purposes of Release 0, details regarding size of document (e.g., legal, letter), file format, etc. were not defined and therefore could not be assessed against the functionality available from the COTS product.

The goal for the CRPs is to understand at a high level any fundamental differences between the AS-IS Spectrum Management system and the proposed COTS product, and to determine whether these differences are acceptable to the Spectrum Program.

The project has stated that business requirements will be analyzed at a more detailed level during the implementation phase (post Release 0), when testing will be performed. The project expects to encounter issues during implementation when looking at detailed requirements where new gaps may be identified between the software and the business requirements.

As a result, there is a risk that cost estimates for long-term budgeting purposes may not be as accurate as they could have been had the project team defined requirements at a detailed level to fully understand the degree of customization versus business process changes. Furthermore, by analyzing detailed business requirements during the implementation phase, there is a risk that this could have a direct impact on the level of work required for change management activities (e.g., additional training, additional business process changes, etc.).

To mitigate these risks, the general approach taken by the project to adapt to the COTS software will be similar to the one being used in Release 0, i.e. business process changes will be implemented where possible, with the view to limit customization and associated costs. Furthermore, the project has formally identified the risk associated with the amount of business process changes that could be required and have identified, as a mitigation activity, the development and implementation of a change management plan.

3.4 Project Management

Schedule

The overall effectiveness of key project management processes is critical in achieving effective implementation and future maintenance and operation of the COTS solution. It is therefore important that the project have the appropriate controls in place to ensure that the project scope (including requirements management), schedule, costs, risks and quality are properly managed.

The project charter is a key document that sets up the framework for project management. The audit noted that the project charter provides an overview of the project including project goals, business outcomes and objectives, project scope and milestone schedule. The charter was signed by the project's executive sponsor, the project sponsor and the project manager.

In addition to the project charter, the SAM-CSI Release 0 activities were presented as a baseline schedule at the Project Steering Committee meeting on January 20, 2012 and approved by the committee.

The SAM-CSI project team has developed tools, including a project schedule and work view, to track and monitor progress against the baseline schedule. These tools are regularly updated by the project office and the project schedule is reviewed on a bi-weekly basis at project team meetings. Any variances from the original plan are reported to the Project Steering Committee.

A formal Change Request process to control proposed changes to baselines defining project scope, timelines, budget, or quality has also been documented by the project team to help support project management activities. The purpose of the Change Request process is to ensure that:

• All changes to the approved Project Management Plan baseline are documented
• The impacts of proposed changes are assessed
• Approved changes are verified and implemented in a controlled manner, and
• Changes are traceable through the project activities back to the initial project baselines.

At the time of the audit, no changes had been made to the approved baselines defining project scope, timelines, budget or quality.

Scope

The project office has put together a number of processes that consist of documentation and tools that help define and control the scope of the project. These include the following:

• A project charter provides an overview of the project, including the project goals, business outcomes and objective, and project scope. In terms of scope, the SAM-CSI project aims to modernize the SMS across all of its business functional areas.
• The Contract Deliverables Requirements List (CDRL) defines Release 0 activities in detail and reflects the project scope.
• The project schedule lists all activities related to the CDRL and allows the project to track tasks defined under the scope of the project. The schedule is officially approved by the Project Steering Committee, which monitors progress on a regular basis.
• As noted above, a Change Request process has been defined and documented for addressing change requests that affect the scope of the project. As part of the process, a Change Control Board is responsible for approving any major changes to the scope of the project.

Quality

A Quality Management Plan should include the following components:

1. quality standards,
2. quality control and assurance activities,
3. quality roles and responsibilities, and
4. a plan for reporting quality control and assurance problems.

The audit team noted at the time of the audit that a Quality Management Plan was not in place. However, the project team has developed a Quality Management Guide that describes project management best practices. The guide states that:

• Quality assurance is the ultimate responsibility of the project manager but will require the active involvement of the Quality Assurance (QA) advisor and all project team members.
• QA audits shall be conducted at specific milestones during the project.
• The QA advisor is responsible for defining applicable QA policies, standards and procedures and for measuring project team results against the QA program. The QA advisor is also responsible for conducting QA audits (structured reviews of quality management activities).

At the time of this audit, no formal QA audits had been performed on project deliverables or processes. Furthermore, QA audits are not mentioned in the project schedule. The audit also noted that there was a delay in staffing the QA advisor position, which contributed to the fact that no audits had been performed.

The QA advisor is now in place and is charged with developing a Quality Management Plan that will define the quality management activities to be applied to project processes and deliverables.

Without a completed Quality Management Plan, quality management activities may not be formally defined. The absence of QA audits creates a risk that the project team may fail to identify potential non-compliances with respect to project processes and deliverables.

Recommendation 3

The Project Director should complete the Quality Management Plan. In addition, Quality Assurance audits should be scheduled at project milestones and integrated within the project schedule.

Cost

With a project of this magnitude, it is important to have processes in place to actively manage and control the budget to ensure that project expenditures are in line with approved spending authority.

The audit specifically examined whether project costs are estimated, captured and analyzed as frequently as required throughout the execution of the project to determine if corrective action is needed. Through interviews and document review, AEB observed the following in relation to budget management:

• An operational budget is prepared annually and approved by the DG of Spectrum Operations. The budget is prepared by the Project Director in consultation with project team members and the Policy and Planning group at SITT.
• Estimates for the operational budget are in line with what was presented as part of the PPA submission. These estimates were revised as the project completed Release 0 activities and engaged in negotiations with the vendor for the Pilot Release.
• The financial administrative team at SITT is responsible for recording and tracking actual project expenditures in the departmental financial system (the Integrated Financial and Materiel System or IFMS) against the budget. This is done via a monthly budget reconciliation tool prepared by Finance.
• The Project Director in collaboration with SITT Finance regularly reviews the reconciliation tool along with actual costs tracked in IFMS and re-forecasts where necessary. Funds in expense categories that are in surplus may be re-allocated to another expense category if needed. For example, excess contingencies that were part of the PPA spending authority were refunded. Funds may also be re-profiled to the next fiscal year if the project experiences delays. For example, some vendor payments were shifted to the 2012-13 fiscal year because of delays in signing the Release 0 contract.
• The Project Director reviews and approves all project expenditures before they are recorded in IFMS. The Director reviews invoices to ensure goods have been received or services rendered prior to approving payment. The Director's sign-off represents FAA Section 34 expense approval and is an important cost control mechanism.

Financial cost information pertaining to the project is reported regularly to senior management and oversight bodies including the Project Steering Committee, TBS and the Deputy Minister. Oversight bodies receive actuals reports (comparing budget vs. actual expenditures) or Executive/Project dashboard reports (providing an overview of the overall health and status of the project). Some of the reports prepared for oversight bodies are:

• Deputy Minister: Quarterly Reporting (IFMS) / Monthly Project Dashboard Reporting (Clarity System)
• Treasury Board Secretariat: Executive Project Dashboard (Clarity System)
• Project Steering Committee: Steering Committee Actuals report (IFMS).

Based on the above, there are two systems used by the project team to report cost information:

• IFMS: A Government of Canada Integrated Financial and Material System. It is used by Industry Canada to generate the departmental financial statements.
• Clarity: The Departmental Project Portfolio Management system now required for all project reporting for various departmental governance committees.

The audit examined a sample of reports to oversight bodies to compare cost information presented as at March 31, 2012. The audit noted discrepancies in the reporting of actual costs between reports using IFMS and reports using Clarity for the same reporting period.

By not ensuring that reported project costs are reconciled to IFMS, the project risks giving stakeholders differing views of the financial status of the project. This could affect outstanding activities or the use of remaining funds.

Recommendation 4

The Project Director should ensure that project cost information reported to stakeholders is consistent with departmental financial system data.

Risk

Effective risk management practices are crucial to the success of the project in implementing the COTS solution. These practices include planning, identifying, analyzing, responding to and monitoring risks and mitigation activities as they pertain to the project.

During the audit, AEB made the following observations with respect to the project's risk management practices:

• A Risk Management Plan describes the formal risk management approach used by the SAM-CSI project. It summarizes the methods used to identify, track and mitigate risks.
• The project office maintains a project risk register, which is the office's working tool to track identified risks, impacts, probabilities and mitigation actions.
• For each documented risk, an executive champion and project office member are identified and assigned responsibility for mitigation. The project office members meet with the executive champion on an as-needed basis to discuss and review risk and mitigation strategies.
• Significant project risks are presented at Project Steering Committee meetings for discussion and review.
• The risk register is discussed at project team meetings on a bi-weekly basis and then updated by the project office. Changes can include adding/removing risks, updating mitigating strategies, revising the risk rating, etc.
• The project team holds workshops on as-needed basis to conduct a comprehensive review of each risk in the risk registry. The purpose of the workshops is to get a good understanding of the risks that have the most affect on the project in the current operating environment and then assess the impact of those risks. The risk register is updated based on the results of the workshops.

3.5 Vendor and Contract Management

The SAM-CSI initiative signals a change from in-house development to the procurement of a COTS software. Hence project management processes need to adequately manage risks associated with a contracted solution, including those related to vendor management. Effective vendor management control mechanisms are key to successful COTS implementations.

The audit noted the following with respect to vendor and contract management:

• A Vendor Management Plan identifies vendor management processes and participant roles and responsibilities. It includes a process to manage and review vendor contract deliverables.
• PWGSC is the contracting authority for the Release 0 and Pilot Release contracts with the vendor and have specialized personnel in software commodity procurement involved in managing and negotiating the contract with the vendor. PWGSC has obtained internal support and advice throughout the contracting process for both the Release 0 and Pilot Release contracts.
• The contracting authority has reviewed and approved the Release 0 contract with the vendor as evidenced by sign-offs on the contract.
• Contract deliverable requirements are monitored, reviewed and formally accepted by management before milestone payments are approved. The review process involves experts from the project office as well as the Project Director. The Director's sign-off on the milestone invoice constitutes formal acceptance of the contract deliverable requirements. Sample audit testing indicated that vendor deliverable reporting associated with each milestone aligns to the contractual requirements for each payment milestone.

The CDRL is a key document that forms part of the Release 0 contract with the vendor. The document outlines the responsibilities of both the vendor and Industry Canada (IC). It includes detailed descriptions of both CRP and Proof-of-Concept activities and requirements, which are the principle Release 0 activities. Evaluation/test plans for these activities have been defined and documented.

The Gap-Fit Analysis allows the vendor and the project team (i.e. subject matter experts from headquarters and across the regions) to understand Industry Canada business requirements and how they may or may not be met by the COTS product. This analysis is being facilitated by a series of CRPs, which involve demonstrations and discussions of the COTS functionality. The evaluation plans for CRPs are organized to identify how the COTS product might meet IC needs "out-of-box."

The Proof-of-Concept activities focus on evaluating the implementation of several essential aspects of the COTS solution technical architecture. The idea is to test technical aspects of the software and develop IC comfort in installing and managing the COTS product in the IC technical environment. Proof-of-Concept test plans identified resources including subject matter experts to execute tests and evaluate test results.

A controlled and dedicated test environment was set up to mirror the future production environment. As per the Release 0 contract, the lab environment supplied is to simulate up to 20% of production level processing requirements (e.g., servers, operating systems, and interconnectivity and network devices). Shared Services Canada is responsible for hosting the COTS application and therefore was involved in setting up the lab environment. The vendor installed and configured the COTS software and the IC architecture team reviewed the results.

Based on the above, the audit team found that the CRP and Proof-of-Concept processes are key activities that facilitate the evaluation and testing of the COTS software prior to the implementation phase.

During each CRP session, decisions regarding business requirements are summarized in a CRP consensus sheet for each functional area. IC's Gap-Fit Analysis Teams (aided by the project office) are responsible for documenting decisions made concerning business changes, selection of options, configurations, requests for customization quotes, etc.

The CRP consensus sheets indicate next to each business requirement, among other things, which ones require customization of the COTS system, any action items for IC and/or the vendor as a result, and the estimated magnitude of the costs involved (i.e. low, medium or high).

Once the CRP participants complete a CRP consensus sheet, the Functional leads then table the sheet and provide presentations to the special Spectrum Operations Committee (sSOC). The presentations give an overview of how business processes are carried out today, how the processes would operate under the vendor solution, and the approach to be taken (i.e. business process change, customization or base system). Members of sSOC review and decide on the final approach to be taken.

The audit found that there is no formal sign-off on decisions regarding the approach to be taken by the chair of the sSOC. The SAM-CSI project developed a formal sign-off process for business requirements that require customization or business process change. However, the process was deemed impractical for the purpose of sSOC and was therefore not used by the project team.

Meeting summaries for sSOC indicate that members reviewed CRP consensus sheets to discuss customization vs. proposed business and process changes. The audit noted, however, that there is no clear evidence that action items raised at the meetings are consistently followed up in subsequent meetings.

The audit found that some post sSOC consensus sheets had outstanding decisions to be made on the final approach for certain business requirements requiring customization. At the time of the audit, approximately 38 user requirements (out of 168) had outstanding items regarding customization.

The project office has stated that action items on consensus sheets will be resolved over time and that some of these items may not be completely determined before the implementation phase, due to begin in September 2012. For the Pilot Release, the consensus sheets are expected to be finalized by December 2012.

The absence of a formal sign-off on decisions regarding business process changes or customization creates a risk that stakeholder acceptance and commitment may not be secured prior to development of the detailed requirements in the implementation phase. In addition, not finalizing outstanding requirements could have a direct impact on the scope of future releases and vendor designs.

Recommendation 5

The Project Director should ensure that all post special Spectrum Operations Committee consensus sheets are finalized and formally approved prior to the development of detailed requirements associated with each release phase.

4.0 Overall Conclusion

This audit revealed that the SAM-CSI project has established, with some exceptions, adequate and effective controls to support the pre-established outputs and outcomes in the areas of: change management, user requirements and solutions definition, project management, and vendor and contract management.

Appendix A: Audit Criteria

Appendix B: Management action plan