Evaluation of Industry Canada's Contribution to the Canada Foundation for Innovation

To obtain a copy of this publication or an alternate format (Braille, large print, etc.), please fill out the Publication Request Form at www.ic.gc.ca/Publication-Request or contact the:

Web Services Centre
Industry Canada
C.D. Howe Building
235 Queen Street
Ottawa, Ontario K1A 0H5
Canada

Telephone (toll‑free in Canada): 1‑800‑328‑6189
Telephone (Ottawa): 613‑954‑5031
TTY (for hearing-impaired): 1‑866‑694‑8389
Business hours: 8:30 a.m. to 5:00 p.m. (Eastern Time)
Email: ic.info-info.ic@canada.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Industry Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Industry Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Industry Canada.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the Web Services Centre (see contact information above).

© Her Majesty the Queen in Right of Canada,
as represented by the Minister of Industry, 2015

Catalogue Number Iu4‑186/2015E‑PDF
ISBN 978‑0‑660‑03329‑7

Aussi offert en français sous le titre Évaluation de la contribution d'Industrie Canada à la Fondation canadienne pour l'innovation.

Final Report

May 2015

Presented to the Departmental Evaluation Committee on

Approved by the Deputy Minister on

Table of Contents

List of Acronyms Used in this Report
Acronym Meaning
AEB Audit and Evaluation Branch
APCF       Automotive Partnerships Canada Fund
CFI Canada Foundation for Innovation
CFREF Canada First Research Excellence Fund
CIHR Canadian Institute of Health Research
CIIF College Industry‑Innovation Fund
CMS Corporate Management Sector
FAA Financial Administration Act
HQP Highly Qualified Personnel
IC Industry Canada
IOF Infrastructure Operating Fund
IP Intellectual Property
JELF John R. Evans Leader Fund
LEF/NIF Leading Edge/New Initiative Fund
MSI Major Science Initiatives
NSERC Natural Sciences and Engineering Research Council
OECD Organization for Economic Co‑operation and Development
O&M Operations and Maintenance
OMS Outcome Measurement Study
OPEA Overall Performance Evaluation and Value‑for‑Money Audit
PAA Program Alignment Architecture
PERAF Performance, Evaluation, Risk and Audit Framework
PDF Postdoctoral Fellow
PL Project Leader
PPR Project Progress Report
PM Performance Measurement
PU Principal User
R&D Research and Development
SIS Science and Innovation Sector
S&T Science and Technology
SSHRC Social Sciences and Humanities Research Council
UK United Kingdom
US United States

Executive Summary

Program Overview

The Canada Foundation for Innovation (CFI) is a not‑for‑profit corporation that funds research infrastructure in order to strengthen the capacity of Canadian universities, colleges, research hospitals and non‑profit research institutions to carry out world‑class research and technology development for the benefit of Canadians. The CFI is the primary source of federal funding for research infrastructure in Canada. The CFI funds up to 40 percent of a project's research infrastructure cost. In partnership with provincial governments and other public, private and non‑profit organizations, institutions secure the remaining 60 percent of the required funding.

The CFI funds priority research infrastructure projects proposed by eligible institutions across all research disciplines. To do so, the CFI has designed a funding architecture that covers a broad spectrum of infrastructure, including projects to attract a leading researcher, team‑led innovative projects that have a structuring effect for an institution or a region, and large‑scale national projects.Footnote 1

Evaluation Purpose and Methodology

The objectives of this evaluation are to address the core issues of relevance and performance in accordance with the Treasury Board's Directive on the Evaluation Function

Under the CFI's 2010 Funding Agreement with the Government of Canada, the CFI was responsible for carrying out an independent third‑party overall performance evaluation of its funding to eligible projects by . To meet federal evaluation requirements under the Financial Administration Act and the Treasury Board Policy and Directive on Evaluation, Industry Canada is also required to conduct an evaluation of its contribution to the CFI.

Industry Canada's evaluation used the CFI's third‑party evaluation as the primary source of information to evaluate the CFI's progress toward achieving its expected results outlined in the 2010 Funding Agreement, as well as the extent to which the CFI demonstrated efficiency and economy. The third‑party evaluation also provided some information that was used to assess one aspect of relevance (i.e., continued need) of the program. In addition to this, Industry Canada's evaluation examined all three core issues of relevance as per the Treasury Board Directive on the Evaluation Function as well as the efficiency and economy of Industry Canada's monitoring of its contributions to CFI via a document and literature review and interviews.

Findings

Relevance

Research infrastructure is recognized as a key element in science and innovation systems that drives economic prosperity and well‑being for Canadians. By providing research infrastructure support, the CFI fosters research and technology development of benefit to Canadians. The ongoing demand that exists for the program in Canada is also a reflection of its continued need. 

Support for the CFI is consistent with federal government priorities and with Industry Canada's strategic outcome related to advancing S&T, knowledge and innovation in order to strengthen the Canadian economy. Federal support for the CFI is also consistent with the roles and responsibilities of Industry Canada as set out in the Department of Industry Act and is consistent with what is being done in other countries. The CFI also makes a specific and unique contribution to the federal government's support for the research ecosystem via its funding for large‑scale research infrastructure.

Performance

The CFI has contributed to enhancing the capacity of recipient institutions to: attract and retain the world's top research talent; support private sector innovation and commercialization; train the next generation of researchers; and enable researchers to undertake world‑class research and technology development. There is less of an evidence base on whether research and technology development supported by CFI is contributing to social, environmental and economic benefits for Canadians; however the analysis that has been done suggests a high level of impact in this area.

Overall, the CFI has been delivered efficiently and economically during the evaluation period. Results suggest that further efficiencies could be gained for the CFI and its support to recipient institutions through the move to a longer funding cycle. Industry Canada's monitoring could be further improved by developing its own Performance Measurement Strategy.

Recommendation

The third‑party evaluation report led to one recommendation specific to the CFI, which can be found in Appendix A along with AEB's corresponding analysis. The findings of this evaluation lead to the following recommendation for Industry Canada:

  1. Following the CFI's refreshment of its Performance, Evaluation, Risk and Audit Framework (PERAF), Industry Canada's Science and Innovation Sector (SIS) should consider developing its own PM Strategy to further ensure all objectives and expected results are being appropriately tracked and to guide future decision making regarding its contributions to the CFI.

1.0 Introduction

1.1 Program Description

The Canada Foundation for Innovation (CFI) is a not‑for‑profit corporation that funds research infrastructure in order to strengthen the capacity of Canadian universities, colleges, research hospitals and non‑profit research institutions to carry out world‑class research and technology development for the benefit of Canadians. Funded infrastructure includes equipment, specimens, scientific collections, computer software, information databases, communications linkages and other intangible property used for carrying out research, including housing and installations essential for the use and servicing of those things. 

The CFI funds up to 40 percent of a project's research infrastructure cost. In partnership with provincial governments and other public, private and non‑profit organizations, institutions secure the remaining 60 percent of the required funding.

The CFI funds priority research infrastructure needs identified by eligible institutions across all research disciplines. To do so, the CFI has designed a funding architecture that covers the full spectrum of infrastructure, including projects to attract a leading researcher, team‑led innovative projects that have a structuring effect for an institution or a region, and large‑scale national projects.Footnote 2 CFI funding is awarded to institutions; and all funding proposals must support the institutions' strategic research plans. Eligible Canadian institutions apply for support through a CFI fund, and all proposals are assessed on three main criteria: the quality of the research and need for infrastructure; the project's contribution to strengthening the capacity for innovation; and the potential benefits of the research to Canada.

During the evaluation period (2009–10 and 2013–14) the CFI had six funds from which awards were made, which are listed and described below. The first three funds were introduced shortly after the inception of the CFI in 1997. The last three are newer funds that were introduced relatively recently. More specifically, Industry Canada provided its first payment to the CFI for the College Industry–Innovation Fund and Major Science Initiatives Fund in 2012–2013, and for the Automotive Partnership Canada Fund in 2010–2011.  

  • Innovation Fund [formerly the Leading Edge Fund/New Initiatives Fund (LEF/NIF)] – invests in innovative infrastructure projects that sustain and enhance areas of activity in which the CFI has already invested and provide support to explore promising new research directions.
  • John Evans R. Leadership Fund (JELF; formerly the Leaders Opportunity Fund) – helps universities attract world‑class researchers by funding infrastructure to enable cutting edge research and contributing to a portion of operating and maintenance costs. 
  • Infrastructure Operating Fund (IOF) – allocates 30% of each capital investment provided to an institution to support operations and maintenance (O&M) costs.
  • Major Science Initiatives Fund (MSI) – contributes to the O&M costs of CFI‑funded large scale research facilities.
  • College–Industry Innovation Fund (CIIF) – funds state‑of‑the‑art, industry‑relevant research infrastructure to foster partnerships between colleges and the private sector.
  • Automotive Partnership Canada Fund (APCF) – funds research infrastructure in support of collaborative R&D intended to benefit the Canadian automotive industry under the umbrella of the NSERC‑led Automotive Partnership Canada.

According to the 2010 Funding Agreement, the expected results for the CFI are to enhance the capacity of ultimate recipients to:

  • Attract and retain the world's top research talent;
  • Enable researchers to undertake world‑class research and technology development that lead to social, economic and environmental benefits for Canada; 
  • Support private sector innovation and commercialization; and
  • Train the next generation of researchers.

Industry Canada's Science and Innovation Sector (SIS) is responsible for ongoing management and monitoring of the contributions to the CFI. The CFI is governed by a Board of Directors that, subject to the by‑laws of the foundation, exercise all of the foundation's powers. There are 13 directors on the CFI Board , six of whom (including the Chair) are appointed by the Governor‑in‑Council on the recommendation of the Minister of Industry on a rotational basis. The other 7 directors are appointed by Members (described further below).
 
The Directors are appointed for a three‑year renewable term and bring with them expertise in the private, institutional, academic, research, and government sectors. The Board of Directors provides overall policy direction for the CFI's programs, reviews the results of merit review assessments of project applications and makes final decisions on projects to be funded. An Industry Canada official attends board meetings as an observer. 

The Board of Directors reports to Members—a higher governing body similar to a company's shareholders, but representing the Canadian public. Similar to the Board of Directors, Members bring with them expertise from private, institutional, academic, research, and government sectors. Members are responsible for the appointment of 7 of the 13 Board of Directors. They also receive audited financial statements, appoint auditors, and approve the Annual Report at their annual meeting. 

1.2 Program Resources

The federal government has committed a total of $5.5 billion to the CFI since its inception in 1997, $1.25 billion of which was committed between 2009 and 2013. To date, the CFI has funded its operating expenses using the investment income from the monies it has been granted by the Government of Canada. In addition, the CFI has reinvested more than $1.4 billion of investment generated income in support of CFI programs.


2.0 Methodology

This section provides information on the evaluation strategy, the objectives and scope, the specific evaluation issues and questions that were addressed, the data collection methods, and data limitations for this evaluation.

2.1 Evaluation Strategy

Under 42(1) of the Financial Administration Act each department is required to conduct a review every five years of the relevance and effectiveness of each ongoing program for which it is responsible. Under the CFI's 2010 Funding Agreement with the Government of Canada, the CFI was responsible for carrying out an independent third‑party overall performance evaluation of its funding to eligible projects by . To meet federal evaluation requirements under the FAA and the Treasury Board Policy and Directive on Evaluation, Industry Canada is also required to conduct an evaluation of its contribution to the CFI.

Industry Canada's evaluation used the CFI's third‑party evaluation as the primary source of information to evaluate the CFI's progress toward achieving its expected results outlined in the 2010 Funding Agreement, as well as the extent to which the CFI demonstrated efficiency and economy. The third‑party evaluation also provided some information that was used to assess one aspect of relevance (i.e., continued need) of the program. An Industry Canada Evaluation Directorate representative sat on the CFI's Evaluation Steering Committee to help ensure alignment between the two evaluations. The third‑party evaluation report led to one recommendation specific to the CFI, which can be found in Appendix A along with AEB's corresponding analysis.

In addition to this, Industry Canada's evaluation examined all three core issues of relevance as per the Treasury Board Directive on the Evaluation Function as well as the efficiency and economy of Industry Canada's monitoring of its contributions to CFI.

The evaluation of Industry Canada's contribution to the CFI that was conducted by the Audit and Evaluation Branch will be referred to as "the evaluation".

2.2 Evaluation Objectives and Scope

The objectives of this evaluation are to address the core issues of relevance and performance in accordance with the Treasury Board Directive on the Evaluation Function. Industry Canada's evaluation of the CFI adopted a goal‑based approach in that it focuses on the expected results of the program as stated in the 2010 Funding Agreement. The evaluation was also calibrated to account for the comprehensive third‑party evaluation undertaken in 2014. Similar to the third‑party evaluation, the evaluation focused primarily on the period of 2009 to 2013.

2.3 Evaluation Questions

The evaluation addresses the following questions on relevance and performance:

Relevance

  1. Is there a continued need for research infrastructure as supported by the CFI?   
  2. To what extent do the objectives of the CFI align with priorities of the federal government and the strategic outcomes of Industry Canada?
  3. Does support for the CFI align with federal roles and responsibilities?

Performance

  1. To what extent has Industry Canada's contribution to the CFI achieved its expected results? 
  2. How efficiently and economically is Industry Canada's contribution to the CFI being delivered?

2.4 Data Collection Methods

Data collection and analysis was primarily undertaken in the third‑party evaluation conducted by Bell Browne Molnar and Delicate Consulting for core issues of performance. The third‑party evaluation employed five data collection methods, including a document review, a survey of users of the CFI‑funded infrastructure [i.e., Project Leaders (PLs) and Principal Users (PUs)], key informant stakeholder interviewsFootnote 3, meta‑analysis of the CFI's Outcome Measurement Studies (OMSs)Footnote 4, and analysis of the CFI's Project Progress Reports (PPRs)Footnote 5 and administrative data. Some key lines of evidence used in the third‑party evaluation are described below.

Outcome Measurement Studies

The CFI's OMS are designed to assess the degree to which the CFI's investment in research infrastructure is a contributing factor in the realization of expected outcomes.Footnote 6   Each OMS examined a particular research theme at an institution, covering multiple projects (typically between 10–20 projects). Each OMS involves an in‑depth questionnaire submitted to institutions with a follow‑up validation by an expert panel whose report is the key output of the exercise. In total 28 OMS were considered in the third‑party evaluation, with particular emphasis on the 12 most recent studies completed between 2009 and 2011.

Project Progress Reports

The PPRs were also particularly useful for the third‑party evaluation for assessing the achievement of expected outcomes. PPRs are annual progress reports that collect data on the implementation, operation and maintenance of the funded infrastructure, as well as the results and funded projects and their linkages to the attraction and retention of researchers, the development of highly‑qualified personnel, the formation of research partnerships and collaborations, and knowledge and technology transfer. The third‑party evaluation analysed PPRs for projects under the LEF/NIF and JELF projects submitted to CFI between 2008–09 and 2012–13.

Survey of Project Leaders and Principal Users

The third‑party evaluation used an online survey to gather feedback from the users of the CFI‑funded infrastructure, including both Project Leaders (PLs) and Principal Users (PUs). The survey consisted of key questions pertaining to the impacts of CFI‑funded infrastructure.

The survey included all PLs and PUs associated with all approved projects for the following funds and years: LEF/NIF: 2006, 2009 and 2012; and JELF: 2006–2012. Just over 6,000 invitations were sent to participate in the online survey. The overall response rate was 25% (1,470 responses) which exceeded the target response rate of 20%.

The AEB led evaluation included a further document review, literature review, and interviews, the details of which are explained below.

2.4.1 Document Review

The review included the third‑party evaluation report, federal budgets and Speeches from the Throne, Treasury Board Submissions and other relevant policy documents, Departmental Reports on Plans and Priorities, Departmental Performance Reports, as well as CFI Annual Reports, Corporate Plans and other documents. 

2.4.2 Literature Review

This review was conducted primarily to address the core evaluation issues of continued need and federal roles and responsibilities. Specifically, the literature review sought to examine the continued need to increase research and innovation capacity and the role of federal funding in supporting R&D generally and large‑scale research infrastructure more specifically, in Canada and other jurisdictions.

2.4.3 Interviews

The objective of the interviews was to collect views, explanations and factual information from Industry Canada and CFI management that address the evaluation questions, as well as to obtain additional information/clarification in relation to the third‑party evaluation report where required. A total of seven interviews were conducted, three with CFI management and four with Industry Canada representatives [3 SIS staff and 1 Corporate Management Sector (CMS) staff member].

2.5 Limitations

This evaluation relied heavily on the CFI's third‑party evaluation for findings related to performance. As such, this evaluation faces the same data limitations as the third‑party evaluation. These limitations and mitigation strategies are outlined in Appendix A.

An additional limitation impacting this evaluation relates to the time to achieve results. A significant amount of time can lapse between the announcement of CFI funding, implementation of infrastructure and realizing outcomes. These factors limit the extent to which the effectiveness of the program can be assessed. In order to mitigate this challenge, the third‑party evaluation included earlier funding programs for which continued need and/or outcomes were realized during the time frame of the evaluation. For example, some projects funded in the early part of the previous decade have only recently been commissioned (e.g. the Research Hospital Fund, which provided funding in 2007–08 and 2008–09) just before the last evaluation was completed. Thus while the scope of both evaluations generally covers 2009 to 2013, some findings relating to the achievement of ultimate outcomes relate to projects funded prior to 2009.


3.0 Findings

3.1 Relevance

3.1.1 Is there a continued need for research infrastructure as supported by the CFI?

Key Finding: Research infrastructure is recognized as a key element in science and innovation systems that drives economic prosperity and well‑being for Canadians. By providing research infrastructure support, the CFI fosters research and technology development of benefit to Canadians. Other countries similarly support research infrastructure in order to remain competitive. The continued need for the program is also reflected in the ongoing demand that exists for the program.

The need for ongoing research infrastructure support was clearly identified in the literature. The 2012 State of the Nation report produced by the Science, Technology and Innovation Council notes that Canada's success in the 21st century will be significantly determined by its ability to harness science, technology and innovation to drive economic prosperity and societal well‑being. Footnote 7 This is further echoed in the recently released federal Science, Technology and Innovation strategy, which notes that, while gains have been made since 2007, Canada needs to continue to invest in S&T in order to remain competitive as the pace of scientific discovery and technological innovation continues to accelerate.Footnote 8

Research infrastructure is recognised as a key element in science and innovation systems, helping boost scientific knowledge generation, accelerate technology development, and provide advanced training for new generations of scientists and science managers.Footnote 9 The Council of Canadian Academies' 2012 Expert Panel on the state of S&T notes that the nation's S&T performance is critically dependent on access to research infrastructure and facilitiesFootnote 10 alongside the more direct forms of support provided to researchers. Further, the recently released 2014 OECD Science, Technology and Innovation Outlook report indicates that countries are focusing on strengthening research infrastructure in order to remain competitive, and in particular to attract and retain talent and knowledge‑based assets which are particularly valuable and mobile.Footnote 11 In Canada, the CFI is the primary source of federal funding for research infrastructure. The CFI supports research and technology development in Canada (which in turn leads to innovation and benefits for Canadians) by providing support for research infrastructure to universities, colleges, research hospitals and non‑profit research institutions across the country. 

The extent of ongoing demand for large‑scale research infrastructure support in Canada can also be used as an indication of a program's continued need. Third‑party evaluation interviewees pointed out that across all sectors, demand for infrastructure and hence infrastructure funding exceeds available funding. They also noted that funding pressures have increased in a number of provinces, due to fiscal restraint and/or provincial funding priorities. Third‑party evaluation survey respondents also confirmed that they anticipate continuing to look to the CFI to support infrastructure funding and further stated that "findings suggest that a substantial level of requests from institutions could be anticipated over the next five years". For example, 59% of respondents indicated that, during the next five years, their institution would likely ("definitely" or "probably") submit an application to the CFI for renewal of infrastructure. 71% indicated that their institutions would likely submit an application for new infrastructure.Footnote 12

3.1.2 To what extent do the objectives of the CFI align with the priorities of the federal government and the strategic outcomes of Industry Canada?

Key Finding: Support for the CFI is consistent with federal government priorities as outlined in the Government's Science and Technology (S&T) Strategies, its Digital Canada 150 plan and recent budget announcements. The objectives are also aligned with Industry Canada's strategic outcome related to advancing S&T, knowledge and innovation in order to strengthen the Canadian economy.

The CFI's objectives are aligned with the Government's priorities related to its 2007 Science and Technology (S&T) Strategy. Specifically, the CFI is aligned with the three Canadian S&T advantages to be fostered (i.e. Knowledge Advantage, People Advantage and Entrepreneurial Advantage). 

The S&T Knowledge Advantage states that "Canadians must be positioned at the leading edge of the important developments that generate health, environmental, societal, and economic benefits." Supporting world‑class research is a core element of the CFI's mandate and is one of the three pillars of the CFI's Strategic Roadmap 2012–2017. More specifically, the CFI strives to ensure that "Canada's best researchers have access to the state‑of‑the‑art infrastructure necessary to conduct world‑class research across all disciplines".Footnote 13

The S&T People Advantage states that "Canada must be a magnet for the highly skilled people we need to thrive in the modern global economy with the best‑educated, most‑skilled, and most flexible workforce in the world." Supporting world‑class talent is another core aspect of the CFI's mission and is the second core pillar in the CFI's Strategic Roadmap 2012–2017, which states that the CFI will help "attract and retain the best talent from around the world and provide a vibrant environment in which to train the next generation of researchers and innovators".Footnote 14

Finally, the CFI also aligns with the S&T Entrepreneurial Advantage that "Canada must translate knowledge into commercial applications that generate wealth for Canadians and support the quality of life we all want." Supporting world‑class innovation is the third and final pillar in the CFI's Strategic Roadmap 2012–2017, which states that the CFI will "ensure that universities, colleges and research hospitals use their research infrastructure to support innovation in partnership with innovators in the public, private and voluntary sectors."Footnote 15

Moving forward, the CFI will continue to be aligned with the Government's priorities as outlined in its recently released Science, Technology and Innovation strategy, Seizing Canada's Moment: Moving Forward in Science, Technology and Innovation, which explicitly notes that the availability of state‑of‑the‑art research infrastructure is critical for success and will allow Canada to remain internationally competitive.

The new strategy also continues with the People and Knowledge pillars but enhances and broadens the Entrepreneurial pillar to encompass Innovation. The CFI will continue to align with this pillar in that one of the program's expected results is to enhance the capacity of recipient institutions to support private sector innovation.

Digital Canada 150

On April 4, 2014, the Minister of Industry announced the Digital Canada 150 plan. This plan aims to support Canada in the digital age through five pillars, one of which is "Economic Opportunities". The CFI is aligned with this pillar, particularly through its contribution to Compute Canada, which delivers high‑performance computing capabilities to support researchers across the country. Digital Canada 150 highlights Compute Canada as a notable investment in this area and reiterates the Government's commitment – announced in Budget 2013 – to invest $50 million through existing accrued interest of the CFI "for future investments by the CFI to address acute needs in cyber infrastructure at post‑secondary institutions, particularly high‑performance computing."Footnote 16

In the 2014 Budget, the Government of Canada announced the creation of the Canada First Research Excellence Fund (CFREF). The CFREF is a $1.5 billion dollar investment that addresses the need for Canada's research‑intensive universities to compete on the world's stage and develop and attract the research talent that will ensure Canada's future prosperity. As noted earlier in the report, the CFI is aligned with this direction given the foundational research infrastructure it provides in support of leading‑edge research.

The CFI is also aligned with Industry Canada's 2014–15 Program Alignment Architecture (PAA) and in particular Industry Canada's Strategic Outcome 2: Advancements in science and technology, knowledge and innovation strengthen the Canadian economy. More specifically, the contributions to the CFI are consistent with the priorities listed under the Science, Technology and Innovation Capacity Program Activity which falls under this Strategic Outcome. This Program Activity sets the strategic direction for policies and programs that support and stimulate research, development and innovation in Canada. Under this Program Activity, Industry Canada, in collaboration with its portfolio partners, other government departments and external stakeholders from the private and public sectors, fosters an environment that is conducive to innovation and promotes scientific excellence. Contributions to the CFI are consistent with the Strategic Outcome and associated Program Activity outlined in that the CFI, through its large‑scale research infrastructure support, facilitates and supports knowledge and innovation to strengthen the Canadian economy.

3.1.3 Does support for the CFI align with federal roles and responsibilities?

Key Finding: Federal support for the CFI is consistent with the roles and responsibilities of Industry Canada as set out in the Department of Industry Act and is consistent with what is being done in other countries. The CFI also makes a specific and unique contribution to the federal government's support for research via its contributions to large‑scale research infrastructure.

Industry Canada's mandate for supporting Canadian S&T activities and policy goals stems from the Department of Industry Act, 1995, which provides the Minister with the powers, duties and functions related to industry, technology, and science in Canada. More specifically, it gives the Minister the responsibility to strengthen the national economy, foster and promote science and technology in Canada and to encourage the fullest and most efficient and effective development and use of science and technology. This closely aligns with the CFI's expected result to enhance the capacity of its ultimate recipients to "enable researchers to undertake world‑class research that lead to social, economic and environmental benefits for Canada."

Federal support for large‑scale research infrastructure is also consistent with what is done in other countries, including Australia, Germany, the UK and the US. For example, the US President's 2012 Budget proposes to invest $2.4 billion in research infrastructure, defined as support for research and development (R&D) facilities construction, renovation, and the purchase of major capital equipment for R&DFootnote 17. In Germany, the federal government provides the majority of funding for large‑scale equipment in basic research with an annual budget of around 1.1 billion eurosFootnote 18.

Evaluation evidence also suggests that the CFI makes a specific and unique contribution in Canada by supporting large‑scale research infrastructure. While the granting councils also fund research infrastructure, SIS staff indicate that this support tends to be on a much smaller scale and generally does not exceed 15% of the councils' supported research projects. Genome Canada also funds research infrastructure on a much smaller scale. Provincial governments play a complementary role to the CFI in that they are the primary co‑funders of CFI‑supported projects.
The key role played by the CFI was further supported by the results of the third‑party evaluation, wherein key informant interviewees in recipient organizations noted that CFI funding is a critical component of the funding and investment available from both federal and provincial governments. They further reported that the CFI has a specific focus on funding research and development infrastructure that is not replicated elsewhere.

3.2 Performance

3.2.1 To what extent has Industry Canada's contribution to the CFI achieved its expected results?

Key finding: The CFI has contributed to enhancing the capacity of recipient institutions to: attract and retain the world's top research talent; support private sector innovation and commercialization; train the next generation of researchers; and enable researchers to undertake world‑class research and technology development. There is less of an evidence base on whether research and technology development supported by CFI is contributing to social, environmental and economic benefits for Canadians; however the analysis that has been done suggests a high level of impact in this area.

Key findings based on the third‑party evaluation report are presented below for each expected result as per the 2010 Funding Agreement. Note that each expected result is framed as an evaluation question below.

3.2.1.1 To what extent has the CFI contributed to enhancing the capacity of ultimate recipients to attract and retain the world's top research talent?

Evaluation evidence suggests that the CFI has been successful in helping institutions to attract and retain high calibre researchers. According to the third‑party evaluation report, a high to very high impact on recruitment and retention was reported for most of the OMS cases, including the 12 most recent ones. A medium to very high impact of attracting faculty from abroad, including repatriations, was also reported.

Survey results also revealed that CFI‑funded infrastructure was one of the most important factors affecting the decision of infrastructure users (PLs and PUs) to either remain in their current position at the funded institution, or to move to a new position at another institution. The majority (78%) of respondents rated CFI‑funded infrastructure as important in this decision, the second most important factor following the general research environment at the institution (83% rated it as important).Footnote 19 Furthermore, high proportions of LEF/NIF and JELF project leaders reported that the availability of their CFI‑funded infrastructure was an important factor in attracting post‑doctoral fellows, PhDs and Masters students.

The third‑party evaluation's PPR analysis of two of the CFI's core funds, the LEF/NIF and the JELF, also suggests the importance of CFI‑funded infrastructure in project leaders' decisions to stay at their institutions. The rate was high for LEF/NIF (between 94% and 100% rated it as "very important" or "important / somewhat important" over the five reporting years that were reviewed). Although it still indicates that JELF projects factored in project leaders' decisions to stay at their institutions, the rate was lower for these projects (ranges of 51–58% "very important" and 35–41% "important / somewhat important" in the 2009–10 and 2010–11 reports, and 60–71% and 21–32% respectively for the 2011–12 and 2012–13 reports).

3.2.1.2 To what extent has the CFI contributed to enhancing the capacity of institutions to enable researchers to undertake world‑class research and technology development that lead to social, economic and environmental benefits for Canada?

The achievement of this expected result was looked at from two perspectives. First, the CFI's contribution to enhancing the capacity of institutions to enable researchers to undertake world‑class research and technology development; and, second, the associated social, economic and environmental benefits.

World‑class research and technology development

Third‑party evaluation evidence suggests that the CFI has helped institutions' capacity to achieve world‑class research and technology development. The 12 OMS cases pointed to the CFI's impact on the quantity and quality of research as ranging from medium to very high. In 8 of the 12 cases, the research theme was ranked to be of international calibre, 3 were ranked to be of national calibre, and 1 mixed national/international calibre. The 12 OMS cases also identified, in most cases, a high to a very high impact on each institution's overall competitiveness in terms of attracting and retaining researchers, and funding. Footnote 20

The production of high‑impact research linked to CFI‑supported projects also demonstrates the CFI's impact in this regard. The PPR analysis of LEF/NIF projects showed that the percentage of projects with research publications (e.g., peer reviewed publications, conference presentations, books, technical reports) remained high across all four reporting years and increased over time from 70% of projects to 98% of projects. The PPR analysis of JELF projects showed similar findings with the percentage of projects with research publications increasing from 75% in year one to 96%, across five reporting years.  

Social, economic and environmental benefits for Canadians

It is important to frame expectations accordingly for this area based on the associated expected result. The CFI is responsible for providing high‑quality research infrastructure support to enhance the capacity of recipient institutions to enable world‑class research and technology development that "leads to" social, economic and environmental benefits for Canadians.

The third‑party evaluation report notes that "there is not a broad and deep evidence base existing about the contribution that the CFI has made to recipient institutions being able to contribute to social and economic benefits for Canadians". It further notes that quantification of benefits is a challenge not just for the CFI but other funders in Canada and in other countries. It states that the "various players in the results chains leading to ultimate outcomes, the lengthy time period to move from investment in research, to discovery and then to innovation and commercialization, as well as the difficulties in gathering data, all add to the complexities, and associated costs and time associated with quantifying benefits."

Despite these challenges, some analysis has been done to suggest that the CFI has helped recipient institutions contribute to social, economic and environmental benefits for Canadians. The most compelling evidence stems from a recent independent pilot Socio‑Economic Impact Analysis (SEIA) study of medical imaging R&D that the CFI commissioned in partnership with the Canadian Institutes of Health Research (CIHR). The case study for the analysis was computed tomography perfusion (CTP), an advanced imaging procedure that can be performed in just a few minutes using scanners readily available in hospitals' emergency departments for diagnosis in acute stroke. Sophisticated but easy to use tools were developed using CFI infrastructure and CIHR support. This study was unique in that the distinct contributions of the CFI and CIHR were considered, along with those of provincial partners and universities. The study showed net economic benefits attributable to the CFI/CIHR and partners as being $42 million to $86 million, which represented a benefit‑to‑cost ratio of between 1.5 to 1 and 2.3 to 1.Footnote 21

3.2.1.3 To what extent has the CFI contributed to enhancing the capacity of institutions to support private sector innovation and commercialization?

This expected result was added as part of the 2010 Funding Agreement and relates primarily to two of the CFI's funds: the College‑Industry Innovation Fund (CIIF) – which focuses on helping colleges enhance their capacity to support private sector innovation and commercialization, and the Automotive Partnerships Canada Fund (APCF) – which supports collaborative research and development intended to benefit the Canadian automotive industry. With both of these funds being introduced relatively recentlyFootnote 22, it is too early to adequately assess their associated outcomes.

More generally, however, the third‑party evaluation reports that CFI‑funded infrastructure has helped create an environment conducive to networks and collaborations. In the survey, 59% of respondents rated CFI‑funded infrastructure as important for fostering collaborations with end users outside of academia. Industry interviewees noted that the attraction for collaboration was not only in access to specialized equipment that they would not have been able to access on a commercially viable basis, but also to the academic expertise (that is, faculty, post doctoral fellows, graduate students and technical staff) affiliated with the CFI‑funded infrastructure to resolve their particular issue or area of interest.

The CFI's Research Facilities Navigator was built in order to facilitate connecting private sector and other end users to research infrastructure available in institutions. It provides a searchable inventory of research infrastructure including capabilities. The Research Facilities Navigator was launched in November 2013, and was a focus of a special information feature in The Globe and Mail newspaper on . It is prominently displayed on the home page of the CFI website. As of August 2014, over 360 laboratories and facilities at 63 institutions across Canada were included. The impact of this service was not assessed in the third‑party evaluation report because it was still early days in its implementation.

Third‑party evaluation evidence also indicates that CFI‑funded infrastructure helped to increase the number of productive networks and collaborations between institutions and the private sector. For example, PPR analysis of LEF/NIF and JELF projects showed that CFI funding was considered to have a major impact on the ability to attract additional research funding from other sources, one of which included corporations/firms. In the OMS cases, the CFI's impact on end user partnerships and contributions was rated very high in 11 of the 12 cases.Footnote 23

The CFI also aims to strengthen Canada's innovation capacity by encouraging knowledge transfer/translation.Footnote 24 Third‑party evaluation evidence suggests that the CFI has been successful in this regard as well. The survey showed that 70% reported that they had actively explored practical applications of the research conducted using the CFI‑funded infrastructure. Of these, 61% had explored direct technology transfer for new or improved industrial products, processes or services, and 58% had explored indirect technology and knowledge transfer. Knowledge and technology transfer took place through various modes, including IP actions / rights, licensing and company creation. The OMS cases showed that knowledge and technology transfer took place, but the extent to which each mode was employed depended upon the research theme.

3.2.1.4 To what extent has the CFI contributed to enhancing the capacity of ultimate recipients to train the next generation of researchers?

Evidence provided in the third‑party evaluation report suggests that the CFI has made a contribution in this area as well. Of the 12 OMS cases reviewed, the impact on the total number of research trainees varied from low to very high, but at least a medium impact on at least one type of research trainee – that is, Masters students, Doctoral (PhD) students or Postdoctoral Fellows (PDFs).Footnote 25 PPR analyses also showed that a large majority of PLs also reported technical training on their CFI‑funded infrastructure, especially during the early years of its operations.Footnote 26

PPR analysis also indicated that for LEF/NIF and JELF projects, CFI‑funded infrastructure was reported to have played a significant role in the research work undertaken by PDFs, PhDs and Masters students. The percentage of PLs indicating that CFI‑funded infrastructure played a significant role in the research work undertaken by these researchers ranged from 78% to 100% across both types of projects. The importance of this outcome and the contribution made by the CFI in increasing the numbers of PDFs, PhDs and Masters students was also confirmed by the third‑party evaluation team through key informant interviews. 

3.2.2 How efficiently and economically is Industry Canada's contribution to the CFI being delivered?

Key Finding: Overall, the CFI has been delivered efficiently and economically during the evaluation period. Further efficiencies could be gained for the CFI and its support to recipient institutions by a longer funding cycle. Industry Canada's monitoring could be further improved by developing its own Performance Measurement Strategy to reflect its contribution to the CFI and inform future investment decisions.

This section examines efficiency and economy in terms of the CFI's operations and then in terms of Industry Canada's monitoring of contribution agreements.

Efficiency and economy of the CFI

The third‑party evaluation examined the CFI's efficiency and economy by assessing the CFI's resource utilization, as well as program design and delivery. Key findings for each of these areas are discussed below.

Resource utilization

The ratio of operating to total costs is a common way to assess the efficiency of third‑party delivery organizations.Footnote 27 The third‑party evaluation reports that the CFI's operating expenses as a percentage of disbursements ranged from 2.4% to 3.4% from 2009–10 to 2013–14. Although the report cautions there are differentiating factors between the CFI and the granting councils (e.g., the CFI includes direct program costs in their operating costs whereas the granting councils do not, there are also differences in the average size and number of grants and contribution provided by each agency), it notes that the CFI's operating expenses as a percentage of disbursements compare favourably with that of the granting councils, which ranged from 2.2% to 6.5% between 2010–11 to 2013–14 according to the documents cited in the report.

Program design and delivery

The third‑party evaluation examined the CFI's program design and delivery by assessing: the extent to which there is stakeholder support for the current CFI model (as opposed to other cost‑effective alternatives); the design and delivery of the CFI's core funds, the LEF/NIF and JELF; and the challenges, barriers and opportunities for improvement related to the CFI's current design and delivery.

Stakeholder support for the CFI's current program design and delivery

The third‑party evaluation report indicates that key informant interviewees did not identify any alternatives to the current program design and delivery of the CFI and expressed support for the current CFI model, although the need for ongoing coordination with the other federal funding agencies was emphasized. They noted that having one independent organization accountable for funding support for research infrastructure means that it can develop a strong depth of knowledge and allows it to be flexible and responsive to changing stakeholder needs.   

Design and Delivery of LEF/NIF and JELF

The third‑party evaluation survey showed that the design and delivery of LEF/NIF and JELF were highly rated by project leaders. In the survey, design referred to "the basic elements of the CFI's funds such as objectives or eligibility criteria", while delivery referred to "the basic administration of the funds by the CFI, the information provided for the preparation of applications, the granting, payment, and control of contributions through the transparency of the decision‑making process". Based on a 1 to 5 scale rating (where 1 is "poor" and 5 is "excellent"), 70% of project leaders assigned a 4 or 5 rating for the design of the LEF/NIF, and 63% assigned a 4 or 5 rating for its delivery. Regarding the JELF, 78% of project leaders assigned a 4 or 5 rating for its design and 65% assigned a 4 or 5 rating for its delivery.

Challenges, barriers and opportunities for improvement

While supporting the CFI's current program design and delivery, some challenges and barriers, as well as opportunities for improvement were identified by third‑party evaluation key informants as well as in the PL/PU survey, some of which were also identified via document review and interviews conducted as part of this evaluation. Key findings are as follows:

Lack of long‑term stable funding for the CFI's support of Ultimate Recipients

The lack of long‑term funding for the CFI was identified as a key challenge/issue by all categories of third‑party evaluation key informants. The third‑party evaluation notes that this presents difficulties for the CFI in terms of its own sustainability and has implications for human resource planning. It also has impacts for provinces being able to plan their funding support and institutions being able to develop strategic research plans. This was further echoed in interviews conducted with CFI representatives as part of this evaluation. For example, it was reported that there have been cases where institutions, concerned that there may not be another opportunity in the near future, submit applications earlier than necessary and end up purchasing new infrastructure that could have been purchased later.

Funding from Other Sources

The design of the CFI's funding programs requires that 60 percent of funding come from other sources, such as institutions and their funding partners – e.g., provinces, private sector, other funders. The third‑party evaluation reports that other funders may have difficulties in committing to matching funding or meeting their funding obligations as economic conditions change or their priorities change.

Document review and interviews conducted as part of this evaluation indicate that the recently introduced MSI fund has been experiencing some of these challenges. More specifically, institutions are finding it difficult to secure partner funding from either provincial governments and/or from federal departments originally engaged in the funding of research activities. While the responsibility for obtaining matching funding rests with the recipient institutions and CFI funding is only provided when partner funding is secured, the CFI has its own measures in place to mitigate the risk that partner funding does not materialize (e.g., assessing sustainability as part of the application review process, increased monitoring and governance).Footnote 28

Industry Canada Oversight

Industry Canada oversight includes the development and implementation of the funding agreements, attending CFI Board of Directors meetings as an observer, reviewing CFI reporting documents and providing ongoing analysis and advice to the Minister of Industry and Minister of State (Science and Technology) in relation to the CFI. Evidence collected as part of this evaluation suggests that Industry Canada is managing these tasks efficiently. SIS estimates that approximately one FTE is dedicated to core CFI oversight activities for this contribution, made up of both staff and management time. Corporate Management Services (CMS), who was also consulted as part of this evaluation, had no suggestions for improvements to efficiency/economy within the department. CMS noted that although some additional steps need to be taken given the size of the CFI contribution (i.e., over $25M), the CFI file requires minimal attention from CMS as SIS undertakes the bulk of the monitoring and oversight function.

SIS also has service standards in place that outline time frames for handling key aspects of the administrative process and have performance targets tied to each one. For example, regarding payments, their goal is to release payment within 45 calendar days of satisfying the requirements of the funding agreement 95% of the time.Footnote 29 SIS staff report that these standards have generally been adhered to for the CFI during the evaluation period.

While SIS collects information regarding the CFI's performance on a regular basis, having its own Performance Measurement (PM) Strategy to reflect the Government's contributions to the CFI would further ensure that information is being collected in a systematic and meaningful way for the department. It could also be used as an important tool for supporting future investments and strategic decision‑making for the department related to the CFI, as well as for routinely assessing the extent to which SIS is managing its contributions to the CFI efficiently and economically.

The 2014 Contribution Agreement requires that the CFI refresh its Performance, Evaluation, Risk and Audit Framework (PERAF) by March 31, 2015. This would be an opportune time for SIS to develop its own PM Strategy based on the information that the CFI is already collecting and reporting on. Note that a PM Strategy has been done for other S&T partner organizations (e.g., Genome Canada) within the SIS portfolio.


4.0 Conclusions and Recommendations

4.1 Relevance

Regarding relevance of the program, the evaluation determined that:

  • There is a continued need for research infrastructure as supported by the CFI. Research infrastructure is recognized as a key element in science and innovation systems that drives economic prosperity and well‑being for Canadians. By providing research infrastructure support, the CFI fosters research and technology development of benefit to Canadians. The ongoing demand that exists for the program in Canada is also a reflection of its continued need. 
  • Support for the CFI is consistent with federal government priorities and with Industry Canada's strategic outcome related to advancing S&T, knowledge and innovation in order to strengthen the Canadian economy.
  • Federal support for the CFI is consistent with the roles and responsibilities of Industry Canada as set out in the Department of Industry Act and is consistent with what is being done in other countries. The CFI also makes a specific and unique contribution to the federal government's support for research ecosystem via its contributions to large‑scale research infrastructure support.

4.2 Performance

Regarding the effectiveness of the program, the evaluation determined that:

  • The CFI has contributed to enhancing the capacity of recipient institutions to: attract and retain the world's top research talent; support private sector innovation and commercialization; train the next generation of researchers; and enable researchers to undertake world‑class research and technology development. There is less of an evidence base on whether research and technology development supported by CFI is contributing to social, environmental and economic benefits for Canadians; however the analysis that has been done suggests a high level of impact in this area.
  • Overall, the CFI has been delivered efficiently and economically during the evaluation period. Further efficiencies could be gained for the CFI and its support to recipient institutions through a move to a longer funding cycle. Industry Canada's oversight could be further improved by developing its own Performance Measurement Strategy.

4.3 Recommendation

This evaluation led to the following recommendation:

  1. Following the CFI's refreshment of its Performance, Evaluation, Risk and Audit Framework (PERAF), Industry Canada's Science and Innovation Sector (SIS) should consider developing its own PM Strategy to further ensure all objectives and expected results are being appropriately tracked and to guide future decision making regarding its contributions to the CFI. 

Appendix A

Third‑party evaluation recommendation and corresponding AEB analysis
Third‑party Evaluation Recommendation Corresponding AEB analysis
In developing its next Performance, Evaluation, Risk and Audit Framework (PERAF), the CFI should consider including all aspects of the Expected Results in its funding agreements in the logic model, performance measurement framework and evaluation framework. Further, the CFI should consider adding to its logic model some immediate outcomes which go beyond "enhancing capacity" of recipient institutions, and add what the CFI achieves for its other stakeholders as well. Such outcomes could be derived from aspects now in the CFI's Strategic Roadmap.

IC's evaluation focused on assessing the CFI's effectiveness based on the expected results outlined in the 2010 Funding Agreement. However the 2014 Contribution Agreement includes national objectives and expected results. It is also important to note that the CFI still operates under a 1997 Funding Agreement that includes four "national objectives". Taking this into consideration, the approach being taken by the CFI in developing its PERAF appears reasonable as it provides measures for both national objectives and expected results, and includes a cross‑walk showing the link between the two.

The latter portion of this recommendation (i.e., that the CFI consider adding immediate outcomes which go beyond "enhancing capacity" of recipient institutions, and add what the CFI achieves for its other stakeholders) relates to specific adjustments that the CFI could make to its PERAF to include elements outside of what is specified in the currently active funding and contribution agreements. As such, this recommendation is not applicable to SIS.

Appendix B

Note that this information is taken directly from the third‑party evaluation report.

Third‑party evaluation limitations and mitigation strategies
Limitation Impact/Potential Impact Mitigation Strategy
Analysis of Project Progress Report data

Changes to the project reporting requirements were introduced for the 2011–12 reporting year. As a result, projects that completed award finalization prior to the end of 2010–11 continued to be required to submit reports for each of the five years following finalization. However, projects that completed award finalization after 2010–11 were required to submit reports starting when the infrastructure became operational, for four years when the CFI contribution was less than $1 million and five years when over $1 million.

In 2010–11, modifications were made to some of the questions in the project reporting questionnaire.

Changes to the project reporting requirements and/or questions in the project reporting questionnaire meant that some of the data was not comparable over time. Data impacted by changes in project reporting requirements and/or questions asked were analyzed and reported separately.
Key Informant Interviews
Stakeholders invited to be key informants were selected across a range of stakeholder categories to reflect the stakeholder community, rather than to be a representative sample. In this context, key informant interviews provided insight into a process or problem, and, as such, are not conducive to counting up responses.
There may be an expectation by readers that information gathered in key informant interviews can be extrapolated across all stakeholders. The range of responses was presented in addition to their frequency.
Survey of PLs/PUs

The census (entire population) of PLs and PUs was selected for the online survey (i.e., all PLs/PUs associated with the two funds and award years listed above). The response rate was 25%. This exceeded the target response rate of 20%, which is typical of similar online surveys, and was the rate achieved in the survey conducted for the 2010 Overall Performance Evaluation and Audit (OPEA).

Because the census of PLs and PUs was selected, it was not appropriate to apply a margin of error / confidence interval to the survey data. This is in accordance with the Government of Canada standards for the conduct of online surveys. Section 4.6 states that for census surveys, "do not state a margin of sampling error, as this does not apply to census surveys because no sample is drawn": http://www.tpsgc-pwgsc.gc.ca/rop-por/enligne-online-eng.html#s4.6

There still may be questions by readers as to the extent that the survey results can be considered to be representative of the entire population of PLs and PUs. They may also suggest that a margin of error / confidence interval should be applied to the survey data. A comparison was made between the respondents who completed the survey with the total population. The two groups of data were compared on the following parameters for which data were available for the population: language, number of projects, fund type, year of award decision, amount of award and province of the respondent's institution. Based on these comparisons, we found that the two sets of data were very similar. There were a few minor variations; e.g., respondents tended to represent a slightly higher proportion of more recent projects; the LOF/JELF fund was slightly over‑represented; and, larger awards were slightly under‑represented. But given the differences were small for each parameter, we determined that there was no need to statistically weight the data.
Survey of PLs/PUs
Key survey questions from the survey for the 2010 OPEA were replicated in this survey. However, in some cases there were changes made in the questions, for example, in an attempt to better understand the employment history and choices of PLs and PUs.
Differences in the questions between the survey questionnaires used for the 2010 OPEA and the current survey have an impact upon the extent to which comparisons can be made between the findings for the two surveys. Care was taken in this report as well as in the survey working paper to highlight cases where differences in the questions limit the ability to infer trends.
Financial data
Financial data mapped to outputs or outcomes in the logic model was not available for the program.
There may be an expectation by readers that such data would be available and be used to assess the CFI's allocation and utilization of resources, regarding operational and allocative efficiency. The report explains that the CFI's business cycles are not conducive to analyzing resources by output or outcome. Other lines of evidence – meta‑analysis of the OMS studies, key informant interviews and the PL/PU survey were used to gather anecdotal evidence.

Management Response and Action Plan
Recommendation Management Response and Planned Action Management Accountability Action Completion Date

Recommendation 1:

Following the CFI's refreshment of its Performance, Evaluation, Risk and Audit Framework (PERAF), Industry Canada's Science and Innovation Sector (SIS) should consider developing its own PM Strategy to further ensure all objectives and expected results are being appropriately tracked and to guide future decision making regarding its contributions to the CFI. 

Agreed. We will develop a detailed performance measurement strategy to guide the next five‑year evaluation of the CFI, building on the CFI's existing Performance, Evaluation, Risk and Audit Framework. Director General, Program Coordination Branch


Date modified: