Evaluation of the Communications Research Centre

Final Report

January 2012

Recommended for approval to the Deputy Minister by the Departmental Evaluation Committee on January 25, 2012

Approved by the Deputy Minister on February 7, 2012

---

Appendices (Separate document)
Appendices are available via an Access to Information

Appendix	Title
Appendix 1	Description of Research Programs
Appendix 2	Crosswalk Between the Research Programs and Core Competency Groupings
Appendix 3	Document Review
Appendix 4	Literature Review
Appendix 5	Interview Guides
Appendix 6	Client Survey Questionnaire
Appendix 7	Client Survey—Results
Appendix 8	Managers' Survey Questionnaire
Appendix 9	Review of Other Labs—Questionnaire and List of Participating Organizations
Appendix 10	Excellence Indicators

List of Tables

Table	Title
Table 1	2011–2012 Source of Funds
Table 2	CRC Employees by Activity
Table 3	Shirleys Bay Buildings
Table 4	Recent Awards for Scientific Excellence
Table 5	CRC Research Outputs
Table 6	Summary of 2007–2010 Outcomes
Table 7	Why clients choose to work with CRC
Table 8	Campus Operation Costs
Table 9	Profile of Organizations Participating in Review of Other Labs
Table 10	Comparison of Corporate Services Provided Internally

List of Figures

Figure	Title
Figure 1	Overview of CRC Budget Allocation, 2011-12
Figure 2	Percentage Split between Research Expenditures and Corporate Expenditures between Comparable Organizations

Acronyms, Abbreviations and Definitions used in this Report

Acronym	Meaning
AEB	Audit and Evaluation Branch
CEB	Certification and Engineering Bureau
CRC	Communications Research Centre
CSA	Canadian Space Agency
DND	Department of National Defence
DPR	Departmental Performance Report
DRB	Canadian Defence Research Board
DRDC	Defence Research and Development Canada
DRTE	Research Telecommunications Establishment
FTE	Full-time equivalent
GDP	Gross domestic product
IC	Industry Canada
ICT	Information and Communication Technologies
IEEE	Institute of Electrical and Electronics Engineers
IT	Information Management and Information Technology
IP	Intellectual Property
ITU	International Telecommunication Union
LAC	Library and Archives Canada
MOU	Memorandum of Understanding
NCC	National Capital Commission
NRC	National Research Council
O&M	Operating and Maintenance
OGD	Other Government Departments
R&D	Research and Development
SITT	Spectrum, Information Technology and Telecommunications
TTO	Technology Transfer Office

Executive Summary

Program Overview

The Communications Research Centre (CRC) is the federal government's primary laboratory for research and development in advanced telecommunications and is a centre of excellence in Information and Communications Technologies (ICTs). The organization oversees twenty individual research programs and a full suite of in-house corporate services ranging from a technology transfer office to graphic design services. Additionally, the CRC acts as landlord to the Shirleys Bay Campus in Ottawa, where the organization is co-located with a number of other government departments as well as some private industry.

Created in 1969 and part of Industry Canada since 1994, the organization is currently housed in the department's Spectrum, Information Technology and Telecommunications (SITT) Sector. Its annual funding is approximately $52.7 million, of which approximately $18.0 million is funded by other government departments and the private sector.

Evaluation Purpose and Methodology

This evaluation is intended to inform management decision-making with respect to future priorities and resource allocation. This evaluation was conducted in accordance with the Treasury Board Policy on Evaluation and the Directive on the Evaluation Function.

The evaluation findings and conclusions are based on the analysis of multiple lines of evidence. The methodology included a review of documents and literature, an administrative data review, key interviews, a client survey, expert panels and a review of other federal labs.

Findings

Relevance

The CRC is responding to the roles and responsibilities of the federal government as set out by legislation via both its function in ICT research as well as its capacities to provide unbiased technical advice in the policy and regulatory forums.

The CRC's activities align with the federal government's priorities as stated in the June 2011 Speech from the Throne, as well as the Science and Technology Strategy, Mobilizing Science and Technology to Canada's Advantage. The CRC's activities also align with Industry Canada's strategic outcomes; however, the CRC research lines reflect a very broad interpretation of them. This interpretation should be narrowed, and research should be reviewed for relevance more frequently in order to ensure work carried out by the organization is appropriately aligned.

The CRC sits at the technological intersection of spectrum management, innovation and regulation in Canada. Evidence indicates that there is a continued need for the organization due to the increasing importance of the ICT industry as well as reliance on the CRC by its client base which includes Industry Canada's SITT, other government departments, academia, industry and other players in the international forum. The CRC is the Government of Canada's source of sound and unbiased advice to inform policies and regulations, enabled by unique in-house expertise and equipment.

Key interviews with clients from the private sector and OGDs revealed that they do not have the capacity that the CRC offers in-house. Paying clients consider any alternatives as biased in their research priorities and competitive interests. This is especially true for large government departments who require input on matters of regulation, national security and public safety. In the client survey, the majority of clients indicated that at least part of the work they commissioned the CRC to do may be sourced elsewhere, but comments indicated that issues of quality and shortfall of professional expertise would likely arise without CRC involvement in their work.

Expert panels put forward the importance of Canadian research remaining at the forefront of emerging trends, citing that such work is important for economic development, as well as matters of scientific priority to government. Expert panels also identified niche areas within the organization that may be doing research that is currently of a low degree of importance to the Government of Canada: while the majority of research conducted at the CRC was deemed highly relevant, expert panels noted that some research has outlived its relevance and a refocus of the corresponding research groups was recommended.

Performance

The research being carried out by the CRC is world-class. Of the twenty research programs the organization houses, sixteen boast of winning awards in the past five years. Researchers collectively lay claim to several hundred publications, thousands of citations and spoke at over 200 engagements over the same period of time. The organization attracts clients and collaborators from all over the globe and clients unanimously rated the CRC's work as being of high quality in key interviews. Some of the CRC's research was flagged for refocus by expert panels as the excellence of the work had lapsed over time.

At the time of this assessment, the CRC was engaged in a joint exercise with SITT to articulate objectives and performance measures. As such, it was difficult to conclude on whether or not CRC is meeting its expected outcomes. Nonetheless, the evaluation used a combination of primary data sources gathered as part of the assessment as well as different data sets and studies available from the CRC to assess performance against general outcomes stated in the CRC's 2007–2010 Strategic Plan, along with the Departmental Performance Report (DPR). There is evidence to suggest that the CRC has made progress toward the achievement of their objectives. The CRC is expected to have a more robust performance measurement strategy following the completion of the joint exercise.

Multiple lines of evidence suggest that clients benefit from the CRC's unbiased advice in terms of scientific findings, technical solutions, and IP licensing. The client survey indicated that the majority of clients believe the work the CRC conducts for their organization offers genuine benefits, with 91.2% indicating that the benefits to their organization were either very good or excellent. However, clients do not believe they are making full use of the organization's capacities and indicated that a client development function would enhance their ability to exploit the full suite of offerings the CRC houses.

This assessment was unable to report on remuneration for work conducted for clients. While incoming funds are documented in financial systems, the CRC cannot extract information relating to corresponding time spent on individual research contracts. Without appropriate administrative tracking systems, as available in other federal labs, it is not possible to discern whether or not the CRC is charging appropriately.

The CRC's governance of the Shirleys Bay campus is in line with Treasury Board policies and the organization is maximizing the recovery of costs. However, the current model detracts from the organization's capacity to focus on its research mandate. Further, significant time and energy is directed toward managing the campus and addressing infrastructure issues.

The CRC's corporate costs are relatively high compared with other research centres. In part, this is because some of the organization's corporate services are allocated towards supporting campus operations. Efficiencies may be gained if some functions were centralized, consolidated, or outsourced.

Finally, research activities at the CRC are not currently functioning at optimal levels with respect to efficiency. This is not a by-product of the capacity of researchers, but is rather an artifact of historical management practices and a silo-based organizational structure. Some reconfiguration of research resources is required to optimize the excellence, talent and capacity to achieve inherent in the organization.

Recommendations

The conclusions of the evaluation led to the following recommendations:

1. The CRC should seek internal efficiencies and respond to the modern realities of ICT research within government.
   1. Consider the recommendations of the expert panels, particularly those which address areas of overlap, inefficiencies and topics of lower strategic value to the organization
   2. Seek more efficient solutions for corporate services
   3. Set regular intervals for the review of research lines in terms of their excellence, relevance and impact
   4. Continue to work toward a governance model that will focus the attention of the organization on its research mandate
2. The CRC should improve its tracking of resource use on research and work performed for clients.
   1. Researchers should track the time they spend on all projects to determine the true cost of services to clients and to ensure appropriate billing
   2. Systems should be established to ensure that reports on client-based activities by research program can be produced
3. To better communicate its capabilities, the CRC should strengthen its client/business development interface. Consideration should be given to:
   1. Articulating and promoting of the CRC's capabilities to clients and stakeholders
   2. Collecting market intelligence on new and emerging trends
   3. Actively updating a client feedback mechanism at the conclusion of research projects
   4. Formally acting as a catalyst for a stronger working relationship with SITT
4. To better evaluate the performance of research, the CRC should continue to work on performance measurement for outputs as well as identifying and measuring immediate, intermediate and ultimate outcomes.