Audit of Data Centre Management

January 2011

Recommended for Approval to the Deputy Minister
by Departmental Audit Committee on January 25, 2011

Approved by the Deputy Minister on January 31, 2011

Table of Contents

• 1.0 Executive Summary
  • 1.1 Introduction
  • 1.2 Overall Conclusion
  • 1.3 Main Findings.
  • 1.4 Recommendations
  • 1.5 Statement of Assurance
  • 1.6 Audit Opinion
• 2.0 About the Audit
  • 2.1 Background
  • 2.2 Objective
  • 2.3 Scope
  • 2.4 Methodology
• 3.0 Findings and Recommendations
  • 3.1 Introduction
  • 3.2 Governance
    • Finding 1.0: Functional Authority at an Operational Level
    • Recommendation 1.0
  • 3.3 Internal Control
    • Finding 2.0: Control over the Physical Access to the ICDC
    • Recommendation 2.0
    • Recommendation 2.1
    • Finding 3.0: Server Evergreening
    • Recommendation 3.0
    • Finding 4.0: Facilities Maintenance Agreements Documentation
    • Recommendation 4.0
  • 3.4 Risk Management
    • Finding 5.0: IT Continuity and Business Continuity Plans
    • Recommendation 5.0
• 4.0 Appendix A—Audit Criteria Used
• Management Action Plan

1.0 Executive Summary

1.1 Introduction

The Data Centre Management (DCM) Directorate is a unit within the Infrastructure Services Division (ISD). ISD reports to the Chief Informatics Officer (CIO). DCM's mandate is to maximize management and operations of Industry Canada's Data Centre (ICDC), while providing leadership and expertise to the Department's business units in the domains of servers, enterprise data storage and e-mail.

The objective of the audit engagement, as approved in the risk-based audit plan and confirmed in the planning phase of the audit, was to provide assurance that ICDC has adequate controls in place to protect the confidentiality, integrity and availability of Industry Canada's (IC) data and systems.

The scope of the audit engagement covered all aspects of the data centre facility housed at the C.D. Howe complex. The audit also examined DCM's exercise of governance, risk management and control. Furthermore, the audit reviewed the degree to which functional authority is established and exercised over computing facilities where sectors have retained responsibility for the development and support of program-specific Information Technology (IT) systems.

The audit did not include an examination of the computer and server rooms in operation outside of the ICDC (i.e. in Place du Portage, Jean Edmond Towers and regional offices). In addition the audit did not review logical access controls related to any particular applications, databases, or network devices (such as hubs, routers, switches and firewalls).

1.2 Overall Conclusion

The DCM function is mature in terms of operational processes and procedures. The Department has adopted and implemented the project management methodology outlined in the Treasury Board Secretariat (TBS)/CIO Enhanced Management Framework. Review bodies provide oversight to incidents, problems and changes. Furthermore, data centre performance is monitored and reported against service level agreements. The physical environment of the data centre is appropriately monitored and controlled.

However, opportunities exist to address and improve management practices related to the exercise of functional authority over the operation of the Department's computer and server rooms, control over physical access to the data centre, and activities to ensure the continued availability of data centre operations.

1.3 Main Findings

Governance

The DCM function is mature in terms of operational processes and procedures. The Department has adopted and implemented the project management methodology outlined in the TBS/CIO Enhanced Management Framework and the Information Technology Infrastructure Library service delivery and service support processes. Review bodies provide oversight to data centre incidents, problems and changes. However, improvement could be made in the following area:

Internal Controls

The data centre's performance is monitored and reported against service level agreements. The physical environment of the data centre is appropriately monitored and controlled. However, improvements could be made in the following areas:

Risk Management

Threat and Risk Assessments are appropriately prepared and addressed and review committees discuss and assess risk on an ongoing basis. Problems and incidents are studied to determine root causes and to identify changes required to prevent recurrences. However, improvement could be made in the following area:

1.4 Recommendations

1.5 Statement of Assurance

In my professional judgment as Chief Audit Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided and contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed on with management. The opinion is applicable only to the entities examined and within the scope described herein. This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada.

1.6 Audit Opinion

In my opinion, the management of Industry Canada's Data Centre has strengths and weaknesses, with moderate risk exposures related to risk management, control, and governance processes relative to the assurance of IT continuity, asset life-cycle management, and the exercise of functional authority that require management attention.

Susan Hart
Chief Audit Executive,
Industry Canada

2.0 About the Audit

2.1 Background

Data Centre Management Directorate:

The Data Centre Management Directorate is a unit within the Infrastructure Services Division. ISD reports to the Chief Informatics Office. DCM's mandate is to maximize management and operations of Industry Canada's Data Centre, while providing leadership and expertise to the Department's business units in the domains of servers, enterprise data storage and e-mail. Its mission centers on strengthened relationships and continuous collaboration with partners, stakeholders, and clients.

The mission and vision of the organization are as follows:

DCM manages the ICDC, located in the C.D. Howe building. The 6,200 square-foot facility houses approximately 400 servers (mostly managed by DCM), and more than 200 Tera-bytes of data storage.

Core Services:

In order to deliver on the mission and vision, DCM Directorate provides the following services:

Computing Facilities

The ICDC is housed in a secure and managed facility at 235 Queen Street. It is a state-of the art facility with independent climate control, fire suppression systems, uninterruptible power supply, continuous monitoring, and generators.

Server Management

The Server Management Team manages and supports the enterprise application server infrastructure. As part of an enhanced service delivery model, the Server Management Team has restructured into two distinct groups: Windows Server Management and UNIX Server Management.

Storage Management

The Storage Management Group is responsible for providing enterprise data storage through multi-tiered storage area network architecture. It engineers data backups, develops archiving and retention strategies, manages data storage capacity and monitors usage trends.

Messaging

The Messaging Services group is responsible for engineering and system management of the corporate e-mail system.

Public Key Infrastructure (PKI)

The PKI service comprises a comprehensive portfolio of capabilities, functions and procedures that maintain secure systems and permit the communication of sensitive information.

2.2 Objective

The objective of the audit engagement, as approved in the risk-based audit plan and confirmed in the planning phase of the audit, is to provide assurance that ICDC has adequate controls in place to protect the confidentiality, integrity and availability of Industry Canada's data and systems.

2.3 Scope

The scope of the audit engagement covered all aspects of the data centre facility housed at the C.D. Howe complex. The audit also examined DCM's exercise of governance, risk management and control. Furthermore, the audit reviewed the degree to which functional authority is established and exercised over computing facilities where sectors have retained responsibility for the development and support of program-specific IT systems.

The audit did not include an examination of the computer and server rooms in operation outside of the ICDC (i.e. in Place du Portage, Jean Edmond Towers and regional offices). In addition the audit did not review logical access controls related to any particular applications, databases, or network devices such as hubs, routers, switches and firewalls.

2.4 Methodology

The audit criteria focus on providing assurance of the adequacy of the DCM's controls in protecting the confidentiality, integrity and availability of the Department's IT assets. Linkages were made between the Control Objectives for Information and related Technology (COBIT — a set of best practices for information technology management) and Treasury Board Secretariat's Core Management Controls (CMCs); therefore the assessment of the CMCs is also an assessment of the corresponding COBIT criteria. Refer to Appendix A for a list of audit criteria used.

This internal audit was conducted in accordance with the Treasury Board Policy on Internal Audit and Internal Auditing Standards for the Government of Canada. The audit approach consisted of the following:

• Documentation review: 29 key documents were reviewed.
• Interviews: A total of 14 interviews with CIO, DCM, and Building Management staff were conducted for inquiry and corroboration.

The information gathered through these procedures was analyzed and assessed against the audit criteria developed during the planning stage of the audit. These criteria have been shared with the client.

Audit fieldwork was conducted during June and July 2010.

3.0 Findings and Recommendations

3.1 Introduction

This section presents detailed findings from the audit of DCM. Findings are based on the evidence and analysis from both the initial risk analysis and the conduct of the audit.

3.2 Governance

However, improvement could be made in the following area:

Finding 1.0: Functional Authority at an Operational Level

The 2009 Treasury Board Directive on Management of Information Technology states, in part, that "The departmental CIO or equivalent is responsible for…

• Developing and maintaining efficient and effective departmental IT management practices and processes, as informed by ITIL (Information Technology Infrastructure Library) and COBIT (Control Objectives for Information and related Technology), with priority on IT asset management, the IT service catalogue and IT service costing and pricing, as appropriate…
• Aligning departmental IT management practices, processes and technology architecture with federal government strategy, directions, standards and guidelines as they become available and as they evolve under the guidance of the CIOC (Chief Information Officer of Canada) …
• Reviewing and assessing IT services periodically to identify opportunities for enhancing efficiency, effectiveness and innovation as determined by governance and in collaboration with service providers, service users and other stakeholders…"

Although the CIO exercises full responsibility and authority over the operation and maintenance of the ICDC, the CIO does not exercise functional authority over all of the Department's server and computer rooms (e.g. those of Communications Research Centre Canada and Spectrum, Information Technologies and Telecommunications). Functional authority in this context refers to the authority to direct how the Department's server and computer rooms operate in terms of incident management, problem management, change management, release management, and access and environmental controls through the development, communication and monitoring of policies, directives, procedures and standards.

The CIO is responsible for the Department's core IT infrastructure and must also ensure that those connecting to the infrastructure will not create any risks or security issues.

There has been and continues to be an evolution towards centralized IT management within Industry Canada. The Department has a stated policy following from the BearingPoint study (August 2004) that all infrastructure procurement and IT contracting must be approved by the CIO; however, the CIO has confirmed that this policy is not consistently applied by business units.

The Department's IT Governance framework effectively addresses IT planning and investment decision-making and project management through the IT Senior Management Committee and the Project Oversight Committee.

The IT Standards and Architecture Committee, chaired by the CIO, has the mandate to define, approve, implement, evolve, promote and enforce Departmental IT standards and architecture at Industry Canada. The CIO has developed practices and processes for DCM. However, the CIO has not fully ensured Departmental IT management practices and processes are in place in all of the Department's server and computer rooms nor has the CIO reviewed and assessed the Department's computer and server rooms periodically to identify opportunities for enhancing efficiency, effectiveness and innovation.

In addition to being non-compliant with the Treasury Board Directive on Management of Information Technology, the absence of the exercise of functional authority and direction with respect to the operation and maintenance of the Department's computing facilities increases the risk of inconsistent and inefficient business processes and practices, and misalignment with Departmental and government-wide directions.

Recommendation 1.0:

It is recommended that the CIO exercise functional authority with respect to the management of the Department's IT infrastructure, computer rooms and server rooms in operation outside of the ICDC.

3.3 Internal Control

However, improvements could be made in the following areas:

Finding 2.0: Control over Physical Access to the ICDC

The Treasury Board Operational Security Standard on Physical Security states that "Departments must control access to restricted-access areas using safeguards that will grant access only to authorized personnel."   In accordance with the Government Security Policy, Section 10.11, this standard provides baseline physical security requirements to counter threats to government employees, assets and service delivery and to provide consistent safeguarding for the Government of Canada.

The C.D. Howe complex is currently protected by several physical and operational security measures such as a 24/7 guard force, closed-circuit television, and the implementation of proximity card readers. A procedure has been documented for physically accessing the ICDC that includes signing the access log book and swiping the card reader.

Access log book procedures require CIO personnel to reference a Product Change Record (PCR) each time they enter the Data Centre. The procedures require that other building personnel (i.e. IC Facilities Management, building management) stipulate a reason for their visit.

A review of the access log book by the audit team indicated that procedures for CIO personnel were not always followed—PCRs are not always referenced and PCRs are sometimes referenced after their stop dates.

Card reader access rights to the data centre are managed by both CIO and Facilities Management. The audit team reviewed a non-statistical sample of individuals assigned access rights to the C.D. Howe (East Tower) Mezzanine Level Main Entrance of the ICDC, and found that rights were assigned to the access cards of three individuals who had left the Department, two of whose cards were still active at the time of the audit.

Industry best practices stipulate that access rights to IT resources should be on a need-to-know basis where there is a legitimate business requirement. As well, access rights should be maintained (and updated periodically) to reflect the current business requirements of personnel whose roles and responsibilities have changed.

Although the data centre is protected by numerous access controls, the effectiveness of these controls depends on their faithful execution. Effective access control is necessary to maintain the data centre's Protected B certification, to protect IT assets and to ensure continuity of availability of IT services.

Recommendation 2.0:

It is recommended that the CIO ensure that the sign-in log book is reviewed periodically and that the importance of proper completion of the log book be underscored by:

• Contacting individuals who did not properly complete log entries; and
• Reminding all ICDC clients of the procedure to be followed when accessing the data centre.

Recommendation 2.1:

It is recommended that the CIO, in conjunction with Facilities Management, ensure access rights are reviewed immediately, and periodically confirm that all individuals with card access rights to the various ICDC card readers have a current requirement for such access.

Finding 3.0: Server Evergreening

Managing the IT infrastructure asset inventory through a life-cycle approach responds to the need to replace and upgrade an asset ("evergreening"), make changes to an asset and meet the requirements of new initiatives. 

According to the 2010–2011 CIO Business Plan, "Evergreening Industry Canada's IT infrastructure remains a challenge. Each year the CIO reviews equipment that has reached end of life and plans its replacement based on criticality of the system, risks, business priorities, and funds available. The assets include telecommunications equipment, servers, data storage systems, and related equipment. The CIO received major investment funds from the Department in 2005–06 and 2006–07 through the Department's long-term capital plan....IT infrastructure investments made at that time are now reaching end of life and are in need of replacement."

A December 2009 inventory of servers provides evidence of evergreening—a high percentage of DCM servers (87%) are 4 years old or less, a result of significant investments in 2005–2007. These assets typically have a life span of 5 years¹ and best practices dictate that investments be made every year. However, the server inventory also indicates that approximately 32% of these servers are three years old and 25% are four years old; they will require replacement in the coming year (2011–2012).

In 2008, the CIO formulated a replacement strategy for its infrastructure as a means of maintaining IT assets year over year. This strategy was never formally presented to Senior Management.

The CIO is currently focusing on software and infrastructure rust out, to support TBS's follow-up work from the spring 2010 report of the Auditor General (AG) on Aging IT Systems. As the AG report stated, "Without sufficient and timely investments to modernize or replace aging systems, the ability of departments and agencies to serve Canadians is at risk."

If the servers, storage and other IT assets in the ICDC are not life-cycle managed, there is an increased risk of service interruption to Departmental clients and the public, with a resulting loss of productivity and/or reputation. Furthermore, the delay of evergreening investments could result in the need for much more significant investment in server/storage replacement at a future date.

Recommendation 3.0:

It is recommended that the CIO finalize and implement a replacement strategy for servers acquired in 2005–2007 in support of the Department's evergreening initiative.

Finding 4.0: Facilities Maintenance Agreements Documentation

The ICDC is housed on the mezzanine level of the east tower of the C.D. Howe Building. The C.D. Howe Building has been managed since April 1, 2005 by SNC-Lavalin O&M on behalf of Public Works and Government Services Canada (PWGSC).

Some Data Centre maintenance services are provided through Facilities Management (within the Comptrollership and Administration Sector), which interacts with PWGSC and SNC-Lavalin to engage contractors to perform necessary maintenance on important environmental and support equipment (e.g. air conditioners and Uninterrupted Power Supply (UPS)). This equipment is required to ensure the ongoing integrity and availability of the ICDC.

Requirements are provided by DCM to Facilities Management, which asks PWGSC to put in place a maintenance contract. PWGSC, in turn, sends the request to SNC-Lavalin, which in turn requests and evaluates proposals from suppliers and enters into a contract for the required maintenance service.

DCM, however, had not received any confirmation that a maintenance contract had been let for the maintenance of ICDC's UPS at the time of the audit. Without such confirmation, the Director of the DCM Directorate has no formal assurance that the required maintenance contract has been put in place.

Recommendation 4.0:

It is recommended that the Director, DCM Directorate, in conjunction with Facilities Management, ensure that contract confirmation is received, providing assurance that ICDC facilities-related maintenance contracts are in place, prior to the cessation of the existing contracts.

Footnotes

¹Asset Life Cycle Management Tools and Processes, Gartner Research Article: ID Number G00153023, 12 December 2007 (Return to reference 1)

3.4 Risk Management

However, improvement could be made in the following area:

Finding 5.0: IT Continuity and Business Continuity Plans

The Treasury Board Operational Security Standard—Business Continuity Planning (BCP) Program states that departments must establish a Business Continuity Planning (BCP) Program to provide for the continued availability of:

1. Services and associated assets that are critical to the health, safety, security or economic well-being of Canadians, or the effective functioning of government. Unavailability would result in a high degree of injury to Canadians and government.
2. Other services and assets when warranted by a threat and risk assessment.

As noted in the Control Objectives for IT (COBIT), continuity plans are designed to reduce the impact of a major disruption on key business functions and processes. The plans should be based on an understanding of potential business impacts and should address requirements for resilience, alternative processing and recovery capability of all critical IT services. They should also cover usage guidelines, roles and responsibilities, procedures, communication processes, and the testing approach.

The audit team reviewed two business continuity planning documents (BCP—Adapted for H1N1 and CIO Services BCP). Neither plan contained all of the required elements of a comprehensive BCP. As well, neither included a testing approach. The team noted that the CIO Services BCP is still in draft form.

Without a comprehensive IT Continuity plan in place and tested, there is an increased risk that unplanned data centre outages could negatively affect the delivery of Industry Canada's programs for a longer period of time than would otherwise be the case.

Recommendation 5.0:

It is recommended that the CIO ensure its IT Continuity Plan and Business Continuity Plan are finalized and tested periodically.

4.0 Appendix A—Audit Criteria Used

Management Action Plan