Evaluation of the Canadian Institute for Advanced Research – 2017

This publication is available online at www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03844.html.

To obtain a copy of this publication or an alternate format (Braille, large print, etc.), please fill out the Publication Request Form at www.ic.gc.ca/Publication-Request or contact:

Web Services Centre

Innovation, Science and Economic Development

Canada

C.D. Howe Building

235 Queen Street

Ottawa, ON K1A 0H5

Canada

Telephone (toll-free in Canada): 1-800-328-6189

Telephone (Ottawa): 613-954-5031

TTY (for hearing-impaired): 1-866-694-8389

Business hours: 8:30 a.m. to 5:00 p.m. (Eastern Time)

Email: info@ic.gc.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Innovation, Science and Economic Development Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Innovation, Science and Economic Development Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Innovation, Science and Economic Development Canada.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the Web Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Innovation, Science and Economic Development Canada, (insert year of publication).

Aussi offert en français sous le titre Évaluation CIFAR

PDF version

Evaluation of the Canadian Institute for Advanced Research – 2017

805 KB, 33 pages

March 2017

Presented to the Performance Measurement and Evaluation Committee on April 20, 2017

Approved by the Deputy Minister on May 29, 2017


Table of contents

List of acronyms

AEB
Audit and Evaluation Branch
CIFAR
Canadian Institute for Advanced Research
FTE
Full-time Equivalent
GDP
Gross Domestic Product
IIA
Inclusive Innovation Agenda
ISED
Innovation, Science and Economic Development Canada
OECD
Organization for Economic Cooperation and Development
RPP
Report on Plans and Priorities
S&T
Science and Technology
SIS
Science and Innovation Sector

List of tables

Figure

Figure 1: CIFAR logic model

top of page

Executive summary

Program overview

CIFAR is an independent not-for-profit corporation that supports networks of distinguished Canadian and international researchers. Its mission is to increase Canadian research capacity in areas of importance to Canada, and strengthen the Canadian research environment through the promotion of excellence and engagement with the international research community at universities and research institutes worldwide.

Since 1987, Innovation, Science and Economic Development Canada has invested a total of $119 million in funding to CIFAR. Budget 2017 committed an additional $35 million in funding over five years, starting in 2017–18. It also announced an investment of $125 million in CIFAR to support a Pan-Canadian Artificial Intelligence Strategy for Canada.

Evaluation purpose and methodology

The purpose of this evaluation was to assess the relevance and performance of CIFAR. The evaluation covered the period from 2012–13 to 2015–16 and built upon the results of CIFAR's third-party evaluation completed in December 2015. The evaluation employed four primary data collection methods including a document and literature review, interviews and operational benchmarking.  

Findings

Relevance

There is a continued need for publicly-funded fundamental research as a means to foster innovation and provide social, environmental and economic benefits to Canadians. CIFAR responds to this need by using global, interdisciplinary and collaborative research networks to deliver large-scale research programs designed to achieve knowledge breakthroughs.

CIFAR's objectives align with federal government priorities to invest in fundamental research and innovation. ISED's support for CIFAR is also consistent with the federal government's responsibility to foster science and technology. Further, the federal government has a role to play in funding fundamental research as a general contribution to societal need.

Performance

CIFAR has developed and broadened the research capabilities of Canadian and international researchers by providing them with the opportunity to explore novel research approaches and collaborate with the best researchers in the world in a face-to-face and interdisciplinary fashion. Further, early-career researchers have enhanced their capabilities through involvement in CIFAR.

The institute has been successful in facilitating interdisciplinary research and national and international collaboration. This approach has contributed to significant research achievements.

CIFAR has produced high-calibre research that has contributed to social and economic benefits for Canadians. Further, researchers have produced and disseminated a significant amount of knowledge products. The program's outreach activities have increased in recent years and are highly regarded by participants.

The institute's virtual research model of serving as a facilitator of research collaborations suggests efficiencies compared to more traditional models used by other advanced research organizations. In addition, the program has been successful in leveraging external funding, exceeding federal leveraging requirements. Further conclusions could not be reached due to data limitations related to the organization's efficiency and economy.

Recommendations

The third-party evaluation led to six recommendations specific to CIFAR, which can be found in Appendix A. These recommendations were made in the context of CIFAR's Strategic Plan 2012–2017 (CIFAR 2.0) and are broader than the expected results stipulated in ISED's funding agreement.

The findings of this evaluation led to the following recommendation:

top of page

1.0 Introduction

This report presents the results of an evaluation of Innovation, Science and Economic Development's (ISED) contribution to the Canadian Institute for Advanced Research (CIFAR). The purpose of the evaluation was to assess the relevance and performance of CIFAR. The report is organized into four sections:

1.1 Program context

ISED funds a number of arms-length organizations to support science and technology objectives of the Government of Canada. One of these organizations is the Canadian Institute for Advanced Research (CIFAR). CIFAR was established in 1982 as an independent Canadian institute of advanced study, forming unique partnerships between international researchers, research institutions, public funders, private donors and the broader community. ISED has funded CIFAR continuously since 1987; to date, the department has provided a total of $119 million in funding to the organization.

Positioning Canada as a leader in artificial intelligence

With funding from ISED as well as other sources, CIFAR has been supporting international collaboration in the area of artificial intelligence since 2004 through its Learning in Machines and Brains program.

This includes pioneering a technique called "deep learning", a computer technique inspired by the human brain and neural networks, which is now used by the likes of Google and Facebook.

The institute's early exploration in this area has helped position Canada as a leader in artificial intelligence.

Budget 2017 announced an investment of $125 million in CIFAR to support a Pan-Canadian Artificial Intelligence Strategy for Canada.

1.2 Program description

CIFAR is an independent not-for-profit corporation that supports networks of distinguished Canadian and international researchers. Its mission is to increase Canadian research capacity in areas of importance to Canada, and strengthen the Canadian research environment through the promotion of excellence and engagement with the international research community at universities and research institutes worldwide. CIFAR currently supports research programs in 14 areas, drawing upon approximately 400 eminent researchers from around the world.

The primary intended beneficiary of CIFAR is the Canadian research community. This includes individual researchers and students, universities, research institutions and partners. While researchers are the primary target population, it is also expected that the research undertaken through CIFAR research programs will have benefits for the public and private sector, and Canadians more generally in terms of long-term social and economic benefits.

With ISED funding, CIFAR is expected to achieve the following results:

While some of ISED's expected results are more operational, they generally align with the broader strategic goals (herein referred to as objectives) outlined in CIFAR's 2012–2017 Strategy (CIFAR 2.0): addressing complex, important challenges for the future of humanity; broadening global reach; building the next generation of research leaders; creating impact by putting research goals into action; and building a culture of partnerships.

Research Model

The CIFAR research model is "virtual" in nature. The essence of CIFAR's approach is to develop a multidisciplinary network of eminent researchers from different national and international institutions who remain at their home institutions. CIFAR facilitates and supports interaction and collaboration amongst the researchers through repeated in-person meetings and extensive networking.

Key Activities

CIFAR's key activities fall under three pillars:

CIFAR provides program members with funds designed to facilitate their research and involvement in the research network. It arranges and supports meetings of program members with national and international guests, typically two or three meetings a year for each program. CIFAR also provides its research programs with funding to increase collaborations amongst program members who wish to partake in high-risk research opportunities related to the program's overarching research goals.

1.3 Governance

CIFAR is led by the President and CEO and governed by a Board of Directors, which is advised by a Research Council and Council of Advisors. Members of the Board of Directors are drawn from Canadian business, research and professional communities. Its key functions are to contribute to and approve the strategic and financial direction and plans of the Institute and to approve the Institute's research programs.

The Research Council is responsible for monitoring the overall direction and quality of CIFAR's research, advising the President on the approval and renewal of research programs and providing insight into the Canadian and international research environment. The Council is appointed and chaired by the President. Its members are drawn from the senior ranks of Canadian and international universities and research-related organizations.

The Council of Advisors assists the Board of Directors and the President by providing advice as requested. The Council comprises former members of the Board of Directors, the Research Council and past participants of CIFAR's research programs.

The President and Chief Executive Officer of CIFAR reports to the Board and is responsible for developing and leading the Institute in an overall strategic direction. Reporting to the President is the executive group.

The Program Coordination Branch within ISED's Science and Innovation Sector (SIS) is responsible for administering ISED's contributions to CIFAR, as well as ongoing management and oversight of the funding.

1.4 Resources

Since 1987, ISED has provided a total of $119 million in funding to CIFAR. Budget 2017 committed an additional $35 million in funding over five years, starting in 2017–18. It also announced an investment of $125 million in CIFAR to support a Pan-Canadian Artificial Intelligence Strategy for Canada.

Other key stakeholders providing funding to CIFAR include the provincial government, private sector, and research organizations.

1.5 Logic model

The logic model below depicts CIFAR program theory. It shows how the program's activities are expected to lead to certain outputs and various levels of outcomes. CIFAR has developed and refined its logic model since the original one was created in 2002. The current version, represented in Figure 1, was developed in 2014 in response to a recommendation in the 2010–2011 third-party evaluation.

Figure 1: CIFAR Logic Model

Logic model of CIFAR (the long description is located below the image)
Description of logic model

Figure 1 depicts the logic model of the Canadian Institute for Advanced Research (CIFAR). A logic model shows how program activities use inputs to produce outputs, which in turn, are expected to lead to different levels of results or outcomes.

The inputs presented in the logic model are:

  1. Researchers & Collaborating Organizations,
  2. CIFAR Partnerships,
  3. Public & Private Support, and
  4. CIFAR Organization & Management. The program's activities use these inputs to produce outputs.

The CIFAR logic model has three categories of activities:

  1. Research,
  2. Knowledge Outreach, and
  3. Global Academy.

The "Research" category of activities includes:

  1. Portfolio development: Identifying important questions and creating global networks,
  2. Research Activities: Supporting collaborations, interactions and global engagement, and
  3. Program Assessment: Annual assessment and peer review.

The activities under the "Knowledge Outreach" category are:

  1. Stakeholder Engagement: Convene stakeholders and identify research priorities & partners,
  2. Outreach Activities: Identify outreach opportunities and engage with leading knowledge users, and
  3. Outreach Activities: Track performance and conduct 5-year reviews.

Lastly, the activities under "Global Academy" are:

  1. Academy Design: Global academy development, creating training opportunities and selecting Global Scholar candidates, and
  2. Academy Programs: Global Scholars meetings, schools for early-career researchers and international mobility programs.

There are three outputs that are direct products of the activities above.

  1. Interdisciplinary teams: novel, interdisciplinary teams, and new research collaborations (from activities under the "Research" category);
  2. Knowledge Outreach: Bringing together researchers and decision-makers, and outreach activities (from activities under the "Knowledge Outreach" category); and
  3. Trained Researchers: Global Scholar Fellowships, Global Scholar meetings, and advanced training opportunities (from activities under the "Global Academy" category).

There are three short-term outcomes (expected to be achieved on 0–5 years) that are directly attributed to the three outputs above. These short-term outcomes are, respectively:

  1. Knowledge Creation: Research publications, inventions and other IP, and summaries of collective results,
  2. Engagement: Researchers informing policy development, and application of knowledge, and 3) Early-Career Researchers: Increased research capacity in priority areas, and attraction of HQP to Canada.

The three short-term outcomes, respectively, result in three medium-term outcomes (expected to be achieved in 6–15 years):

  1. Knowledge Impact: Discoveries and new perspectives, significant breakthroughs, and dissemination of critical advances,
  2. Solving Challenges: Answering complex questions, new frameworks of knowledge, and social and technical innovation, and
  3. Research Capacity: Retention of research leaders and international recognition of Canada's research leadership.

Finally, there are three ultimate outcomes (expected to be achieved in more than 15 years) that directly result from the three intermediate outcomes:

  1. Innovative Economy: Transformative social and technical innovation, and advances in policy,
  2. Enriched Human Life: Knowledge and innovation that sustains long-term development of Canadian society, and
  3. Advanced Research: Increased capacity and value for interdisciplinary research.

top of page

2.0 Methodology

2.1 Evaluation scope and objectives

CIFAR undertook an independent third-party evaluation that was completed in December 2015Footnote 1 (hereafter referred to as the third-party evaluation) in accordance with its federal funding agreement with ISED. Its primary objective was to provide a mid-term assessment of CIFAR's progress in implementing its Strategic Plan 2012–2017 (CIFAR 2.0)Footnote 2.

As part of ISED's commitments under the Financial Administration Act (FAA), it was required to conduct its own evaluation of CIFAR in 2016–17. The scope of this evaluation built upon the results of CIFAR's third-party evaluationFootnote 3, covering all core evaluation issues of relevance and performance, the latter of which focused on the expected results of the program as stated in the 2012 Funding Agreement.

The evaluation covers the period from April 1, 2012 to March 31, 2016.

2.2 Evaluation approach

This evaluation study was managed by the Audit and Evaluation Branch (AEB) and was primarily conducted by Ference and Company Consulting Ltd.

The evaluation approach relied on evidence derived from various sources and methods. An analysis was undertaken to identify gaps in the information and results obtained from the third-party evaluation of CIFAR in comparison to the current ISED evaluation requirement. The results focused this evaluation to target primary data collection efforts towards issues of relevance and efficiency and economy.

2.3 Evaluation questions

Based on CIFAR's Performance Measurement Strategy and consultations with ISED's Audit and Evaluation Branch (AEB) and the Program Coordination Branch (SIS), the evaluation addressed the following questions:

Relevance

Performance

2.4 Data collection methods

Multiple lines of evidence were used to address all evaluation questions. Data collection methods included a document review, literature review, interviews, and operational benchmarking.

Document Review

The document review provided an understanding of CIFAR, its relevance and achievement of expected outcomes. Key documents included program foundational documents; previous evaluation reports (CIFAR-led and ISED-led); Government priority-setting documents (e.g. Budgets, Speeches from the Throne); CIFAR documentation including Annual Performance Reports and Annual Corporate Plans; and other documents (e.g. research papers, publications, and survey reports).

The document review was also used to summarize the performance results from the third-party evaluation, which employed a methodology consisting of a data, document and literature review, interviewsFootnote 5, and online surveys of CIFAR Fellows and Global Scholars.

Literature Review

The literature review built upon work conducted for previous evaluations and addressed the core evaluation issues of continued need, and federal roles and responsibilities. The literature reviewed included scholarly (peer-reviewed) literature as well as relevant ‘grey literature', such as working papers, conference proceedings, government documents, and commissioned research reports, such as those from the Science Technology and Innovation Council, the Council of Canadian Academies and the Conference Board of Canada.

Interviews

Interviews were conducted with a total of 21 participants, including:

Operational benchmarking

Four international research organizations, as well as other ISED-funded arms-length organizations, were examined to identify potential financial benchmarking information.Footnote 7

2.5 Limitations

Attribution of Outcomes to CIFAR

The majority of projects supported by CIFAR are jointly funded, making it difficult to attribute the success of outcomes achieved specifically to CIFAR. Determining attribution and incrementality of outcomes for CIFAR is also challenging as CIFAR researchers often contribute to these outcomes in their primary work, which is sponsored by sources other than CIFAR. Finally, measurement of societal impacts of research is difficult because of four common problems:

Data on Efficiency and Economy  

CIFAR's Performance Measurement Strategy identifies one indicator for assessing efficiency and economy related to comparing its proportion of resources to similar organizations. SIS identified four international research organizations for benchmarking in this evaluation. However, direct comparisons were ultimately not feasible due to key differences in size, model and limited availability of financial information.Footnote 9 Further, no targets were established by SIS in its funding agreement on the ratio of non-program to total cost. These factors limited the extent to which evaluators could draw conclusions on CIFAR's efficiency and economy. The evaluation provides a recommendation to strengthen this area.

Timing of Data Collection

The main challenges encountered during the third-party evaluation resulted from the timing of data collection. These activities were conducted over the summer, when many program members were on leave. In addition, at CIFAR's request the timeframe for conducting the evaluation was reduced to accommodate changes in leadership of the team responsible for overseeing this evaluation. Both surveys recorded high participation rates (over 50%); however the small sample obtained for the Global Scholars survey limited the extent to which corresponding findings could be generalized. The third-party evaluation mitigated this limitation through the triangulation of findings from different lines of evidence.

top of page

3.0 Findings

3.1 Relevance

3.1.1 Is there a continued need to support CIFAR?

Key finding: There is a continued need for publicly-funded fundamental research as a means to foster innovation and provide social, environmental and economic benefits to Canadians. CIFAR responds to this need by using global, interdisciplinary and collaborative research networks to deliver large-scale research programs designed to achieve knowledge breakthroughs.

This section examines need from two perspectives: the need to support fundamental research and its benefits to society, and the need to support interdisciplinary and collaborative research.

Need to support fundamental research and its benefits to society

Publicly-funded fundamental research, also known as basic research, responds to a need to foster innovation and provide economic and social benefits to Canadian society. Innovation performance is recognized by the Organization for Economic Cooperation and Development (OECD) as a crucial determinant of competitiveness and national progress. By pursuing their best ideas, researchers help to build a broad base of knowledge, understanding and expertise that can seed longer-term social and economic opportunities. Many technological and social innovations, with deep and positive social and economic impacts, have had their roots in public research and came from findings that were impossible to foresee. The most recognized way in which fundamental research generates benefits and fosters innovation is through the creation of new knowledge, information and ideas that are available to firms and other 'users' such as public sector researchers and policy makers.Footnote 10  

Policy and scholarly literature, in Canada and internationally, supports the continued need for publicly-funded basic research. This is largely due to the range of potential benefits derived from research activities, along with their recognized contribution to a strong knowledge economy. For instance, Forster and SeegerFootnote 11 provide an extensive list of literature supporting the notion that fundamental research contributes to the wealth and economic growth of a country which is supported by conclusions drawn by Nelson & RomerFootnote 12 and AghionaFootnote 13. In addition, a recent report by Frontier EconomicsFootnote 14 provides an overview of the evidence related to the rate of return in investment in science and innovation. It found that government-funded investments in science and innovation provided returns to society of between 20% and 40%.

The Council of Canadian Academies stated that fundamental research in natural sciences and engineering is a key driver in the creation of many public goods and that scientific advances help catalyze innovation, create new knowledge, foster economic prosperity, improve public health, enable better protection of the environment, strengthen national security and defence, and contribute to other national and sub-national policy objectives.Footnote 15

CIFAR responds to the continued need for fundamental research by focusing on research that has the potential to generate transformative knowledgeFootnote 16. It does this by developing and managing research programs that focus on key issues which have the potential to contribute to improved health, and environmental, societal and economic benefits to Canada. Some examples are as follows:

Need to support interdisciplinary and collaborative research

There is evidence that the types of complex questions examined by fundamental research benefit from an interdisciplinary and collaborative approach. In particular, the literature suggests that complex questions require interdisciplinary approachesFootnote 17,Footnote 18,Footnote 19,Footnote 20; collaborative teams increase productivity and the likelihood of translational outcomesFootnote 21; and, international collaborations lead to greater contributions to science as measured by impact factors and citations.Footnote 22

An essential element of the CIFAR model is the creation and support of domestic and international networks that approach problems through a collaborative and interdisciplinary approach. Stakeholders interviewed in the third-party evaluation noted that CIFAR has been able to attract the best foreign and Canadian researchers to come together to work in a highly demanding international research network. This was considered a significant achievement as "almost no (Canadian) university has enough critical mass to be globally significant in major (research) areas." Interviewees noted that CIFAR allows Canadian researchers to become part of the "global supply chain of ideas."

3.1.2 To what extent do the objectives of CIFAR align with federal government priorities?

Key finding: CIFAR's objectives align with federal government priorities to invest in fundamental research and innovation.

CIFAR's objectives are consistent with the federal government's priorities to invest in fundamental research and innovation, as set out in the 2015 Ministerial mandate letters, recent Budgets and the 2016 Inclusive Innovation Agenda.

Ministerial mandate letters identify overarching goals and priorities of the government to invest in research. Specifically, the Minister of ISED is tasked with improving the quality and impact of programs that support innovation, scientific research and entrepreneurship. The Minister of Science is to support scientific research to create sustainable economic growth and support and grow the middle class.

Budget 2016 recognizes the need to invest in innovation and notes that "the advancement of basic science and the development of intellectual capacity is the foundation for innovation". Budget 2017 provided funding of $35 million, starting in 2017–18, to CIFAR. In addition, CIFAR was selected as the delivery vehicle for a $125-million federal investment in Artificial Intelligence.

Further, CIFAR's current strategy, CIFAR 2.0 aligns with the Government's Inclusive Innovation Agenda. Specifically, the Institute has set its views on building the next generation of research leaders who will put research goals into action. The institute seeks to attract top talent from around the world and to work collaboratively in a diverse ecosystem of highly qualified workers in order to address complex, important challenges. The fundamental research this work supports promotes world-class research excellence in Canada and fosters collaboration across academic disciplines.

3.1.3 Is the funding of CIFAR consistent with the roles and responsibilities of the federal government?

Key finding: ISED's support for CIFAR is consistent with the federal government's responsibility to foster science and technology. Further, the federal government has a role to play in funding fundamental research as a general contribution to societal need.

ISED's support to CIFAR is consistent with the mandate and responsibilities set out in the Department of Industry Act, 1995 with respect to S&T activities. Specifically, these responsibilities include encouraging "the fullest and most efficient and effective development and use of science and technology" as well as fostering and promoting science and technology in Canada. In addition, the Act states that the Minister shall "promote, assist and provide support services for, and investment in, Canadian industry, goods, services, science and technology". CIFAR's work in fundamental research contributes to the fulfillment of these duties.

A review of academic literature pertaining to fundamental research confirms a role for the federal government. The OECD suggests that, "The fundamental justification for government support of research is the classical market failure argument: the market does not provide sufficient incentives for private investment in research owing to the non-appropriable, public good, intangible character of knowledge and the risky nature of research. In addition to basic research, public research is also needed to meet specific needs of national interest, such as defense, and of the population at large, e.g. regarding health programs".Footnote 23

The importance of government support for basic research was also clearly highlighted in an expert panel report, Innovation Canada: A Call to Action, which reviewed federal support to R&D (referred to as the Jenkins report). It notes that federal and provincial governments play an important role in fostering an economic climate that encourages business innovation – for example, by supporting basic and applied research and related training of highly-qualified, skilled people.Footnote 24

The 2013 Horizontal Review report by Science Metrix found that increasingly, academic research [as conducted by CIFAR] is also being called upon to "contribute to the solution of societal problems and to support innovations and economic growth." The legitimacy of research institutions is largely based on the perceived benefits they provide, i.e., their ability to contribute to economic competitiveness, cultural enrichment and social progress, which ultimately justify the substantial investments made to them, including via public funds.

3.2 Performance

3.2.1 To what extent has CIFAR been able to develop and broaden the research capabilities of high-quality personnel?

Key finding: CIFAR has developed and broadened the research capabilities of Canadian and international researchers by providing them with the opportunity to explore novel research approaches and collaborate with the best researchers in the world in a face-to-face and interdisciplinary fashion. Further, early-career researchers have enhanced their capabilities through involvement in CIFAR projects and unique learning experiences provided to them through CIFAR.

CIFAR builds the research capacity both of Canada's top researchers and its more junior researchers. Senior fellows enhance their capabilities through exposure and interaction with top researchers internationally and through unique interactions with peers in other disciplines. Literature suggests that face-to-face meetings, a key element of the CIFAR approach, contribute to boosting research productivity.Footnote 25

Table 1 demonstrates both the reach of CIFAR's activities and the quality of its participants. In terms of reach, in 2015–16, there were 381 CIFAR researchers at 30 Canadian universities and 95 international universities. Significantly, CIFAR's target of maintaining a share of 40% of senior fellows who are in the top 1% of their field (by citation count) was achieved in three out of four of the most recent years for which figures were available. Further, 81 CIFAR researchers received a national or international award in 2015–16.

Table 1: Reach of CIFAR Programs and Quality of Participants
2011–12 2012–13 2013–14 2014–15 2015–16
Number of Canadian universities with CIFAR researchers 29 27 29 27 30
Number of international universities with CIFAR researchers 75 83 75 88 95
Number of Canadian and international researchers engaged in CIFAR research 389 377 340 349 381
Number of researchers who received national or international awards 76 75 66 78 81
Percentage of researchers identified in the "Outstanding" or "Excellent" category in five- year peer review assessmentsFootnote 26 78% 79% 82% 89% 88%Footnote 27
Percentage of Senior Fellows that are in the top 1% of their fields by citation count 43% 38% 40% 42% N/A
Source: CIFAR Annual Performance Reports

CIFAR also provides opportunities for the development of junior researchers through its Global Academy program which offers a range of opportunities for graduate students, postdoctoral fellows and assistant professors in Canada and across the globe to learn from and collaborate with some of the world's top minds, positioning them to become the leading thought innovators of their generation. In addition, CIFAR fellows supervise graduate students and postdoctoral fellows and CIFAR funding provides these trainees with opportunities to participate in some program interaction meetings, attend advanced training sessions, and/or make research visits to the laboratories of collaborating CIFAR fellows.

The Global Scholars program – one element of the Global Academy program – was redesigned in 2015–16. This program switched its target to exceptional early career researchers in the first years of their assistant professorship from the previous focus on post-doctoral fellows. The rationale was three-fold. First, there are far fewer supports for early assistant professors and it is a particularly vulnerable time in their careers. Second, assistant professors, with their stronger institutional connections, create more opportunities for CIFAR to build relationships. Third, assistant professors deal with more competing demands (e.g. gaining tenure) for which the leadership component of the Global Scholars program could better help with.

Table 2 highlights the reach of the Global Academy Program and its impacts. In terms of reach, there have been approximately 18–20 Global Scholars per year in recent years, scaling down as the old program was phased out and a new one phased in. (The appointment of 18 new Global Scholars was announced in September 2016.) The figures also indicate that Global Scholars strongly believe that the program offered them training and experience that they would not have been able to acquire through other programs. Further, over 1,000 doctoral students have been supervised by CIFAR researchers each year.

Table 2: CIFAR Global Academy Activities: Reach and Impacts
2011–12 2012–13 2013–14 2014–15 2015–16
Number of CIFAR Global ScholarsFootnote 28 20 21 18 8 18
Percentage of Global Scholars who believed their fellowship provided them with "value-added" training and experience for their future career that they would not have been able to acquire through other postdoctoral programs 56% 82% 87% N/A N/A
The number of doctoral students supervised to completion by Canadian CIFAR researchers. This includes graduate students + postdocs 1,614 1,141 1,390 1,466 N/A
Source: CIFAR Annual Performance Reports

According to the document review and findings from the survey of Global Scholars conducted as part of the third-party evaluation, junior scholars' participation in CIFAR's Global Academy program provided numerous benefits. Examples include training that would not be available in Canada or abroad, networking with senior and junior researchers, funding for scholarly events that would not be possible otherwise, exposure to cutting‐edge research ideas and mentoring by leading researchers in the field.

3.2.2 To what extent have CIFAR activities led to increased interdisciplinary research and increased national and international research interactions and collaboration?

Key finding: CIFAR has been successful in facilitating interdisciplinary research and national and international collaboration. This approach has contributed to significant research achievements.

The essence of CIFAR's model is to support international research networks that facilitate collaboration and networking between outstanding Canadian researchers and their international peers. This is demonstrated in CIFAR's membership which as of March 2017 included 411 researchers (18 Global Scholars, 324 Fellows, 69 advisors) from 144 institutions in 17 countries.

CIFAR continues to have a commitment to interdisciplinary collaborationFootnote 29. Peer reviews of CIFAR programs over the last five years have concluded that every program has created unique, interdisciplinary collaborations that otherwise would not have arisen. Some examples of the interdisciplinarity of projects include the:

Stakeholders interviewed in this evaluation believe that Canadian universities, granting councils and other research organizations to a large extent continue functioning as silos with CIFAR facilitating inter-organization collaboration. They suggested that this is because most Canadian research funding is directed to a researcher or an organization which does not foster collaboration whereas the CIFAR model focuses on bringing the best researchers from across Canada and around the world to address specific research questions. They also suggested that CIFAR's involvement of junior researchers early in their careers encourages them to think more of collaboration and interdisciplinarity later in their careers.

The survey of CIFAR fellows conducted as part of the third-party evaluation found that CIFAR research programs have resulted in major research achievementsFootnote 31 since 2012–13, and that 66.5% of program members surveyed rated the frequency of interactions with other program members as a factor that greatly aided their capacity to achieve these achievements. The evaluation also found that, as a result of their membership in a CIFAR research program, program members have experienced major changes in the directions of their research and that the aspects of CIFAR that most influenced these changes were the quality of interactions at meetings (94%) and the calibre of CIFAR collaborators (88.5%).

The third-party evaluation suggests that low representation of countries outside of North America could lead to missed opportunities for collaborations with international researchers outside of North America.Footnote 32 Several stakeholders interviewed as part of this evaluation explained that it is difficult to get non-North American researchers to attend events in Canada and a good proportion of the top researchers in the world are located in the US. Further, the window of opportunity is small as most participants are selected when CIFAR's research programs are launched or reviewed every five years. Nevertheless, CIFAR has made progress in this regard. Non-U.S. participation of international research members increased gradually from 23% in 2011–12 to 27% in 2013–14 and more significantly to 33% in 2014–15 when four new programs were launched.

CIFAR has also started work to forge partnershipsFootnote 33 with other research organizations to increase collaboration as well as heighten its global profile and reach, and to strengthen its knowledge outreach efforts. According to the third-party evaluation, over the past five years the number of partnerships with other research organizations has increased from 0 in 2011–12 to 19 in 2015–16, with institutions in Germany, United Kingdom, Sweden, China, Korea, US and Canada.

3.2.3 To what extent has CIFAR generated research knowledge been promoted and disseminated?

Key finding: CIFAR researchers have produced and disseminated a significant amount of knowledge products. The program's outreach activities have increased in recent years and are highly regarded by participants.

CIFAR researchers disseminate their findings through publications both print and web-based, presentations at symposiums and lectures. The third-party evaluation notes that "the number of knowledge products generated by CIFAR program members has been significant, and many of these products have been published in some of the most highly regarded publishing outlets in their respective knowledge fields." Further evidence of this stems from program data. For example, the annual number of publications and other peer-reviewed contributions by program members averaged 1,554 from 2011–12 to 2014–15.

The general view of stakeholders interviewed was that, in the past, CIFAR depended on researchers to do their own promotion and dissemination and researchers tended to focus primarily on solving their research problems without sufficient emphasis on promotion and dissemination. However, starting in 2012–13, under CIFAR 2.0, CIFAR placed a heightened emphasis on creating impacts through knowledge outreach and exchange.Footnote 34 Further, in 2013–14, it launched IdeasExchange, which are interactive dialogues that bring together CIFAR fellows and knowledge users to share ideas and perspectives on how to act on the insights emerging from CIFAR programs. Continuing in 2014–15, it began to require that newly established and renewing programs develop knowledge outreach plans and a series of roundtables with government representatives was started. Most recently, in 2015–16, CIFAR increased opportunities to engage scientific leaders and early-career researchers with venture capitalists and industry to identify opportunities where scientific discoveries can be commercially developed.

Table 3 below provides evidence that CIFAR's outreach events are of perceived high-quality and are useful to participants. Specifically, 79-81% of participants surveyed rated outreach events as very good or excellent; 68%-88% of knowledge users surveyed reported increased understanding of topics presented; and 65%-81% of knowledge users surveyed planned to integrate the new ideas in their work.

Table 3: Effectiveness of CIFAR Outreach Activities
  2013–14 2014–15 2015–16
Excellence of eventsFootnote 35 (% who rated event as very good or excellent) N/A 79% 81%
Percentage of knowledge users whose understanding of a topic area has been improved through the engagement 88% 68% 75%
Percentage of knowledge users who planned to integrate the new ideas into their workFootnote 36 81% 65% 75%
Source: CIFAR Communications and Knowledge Outreach Analytics

3.2.4 To what extent have CIFAR activities resulted in high-calibre research yielding economic and social benefits to Canadians?

Key finding: CIFAR is producing high-calibre research that has contributed to social and economic benefits for Canadians.

Despite the challenges associated with measuring societal impacts of research (as discussed in section 2.5), CIFAR does track some of these impacts. These are outlined below in Table 4. A key measure of the calibre of research is citation count which reflects how often an article is cited in other sources. As demonstrated below, an average of 41% of CIFAR researchers were in the top 1%Footnote 37 of their fields from 2011–12 to 2014–15. Results discussed in section 3.2.1 regarding awards received and ratings of outstanding researchers via peer review assessments also reflect the high-calibre research supported by CIFAR.

Table 4: CIFAR Impacts
2011–12 2012–13 2013–14 2014–15
% of Senior Fellows that are in the top 1% of their fields by citation count 43 38 40 42
Number of contributions by CIFAR programs that have resulted in transformative social or technical innovation 5 8 11 12
Number of contributions from CIFAR research programs that have resulted in significant economic policy changes 3 0 2 2
Number of contributions by CIFAR programs that have resulted in public policy changes, improved delivery of social services and health care, and increased life satisfaction 8 7 12 8
Sources: CIFAR Annual Performance Reviews and CIFAR Program Peer Reviews

The above table shows that over the last four years, CIFAR programs have resulted in 36 contributions that have resulted in transformative social or technical innovations; 35 contributions that have resulted in public policy changes, incidences of improved delivery of social services and health care, and increased life satisfaction; and 7 contributions that have resulted in economic policy changes.

Specific examples of social and economic benefits gleaned from CIFAR's research programs are as follows (additional examples are included in Appendix B):

These findings align with the third-party evaluation, which concluded that CIFAR's research programs have contributed to social and economic impacts. While the report acknowledges that not all fundamental research will lead to these results, it notes that 15% of program members surveyed identified some form of social and economic impacts, including patents and spinoffs, changes to policies and practices or teaching and learning outcomes. Also, in light of the increasing focus on innovation, the third-party evaluation provided evidence of CIFAR's contributions to new technologies, products or services with potential for commercial or economic value. Specifically, evidence was found of applications in speech recognition technologies, and research that has informed patent applications for technologies related to genetic networks. Some technologies are already finding their way into the market via start-ups and technology transfer to companies, such as Google.

It is important to note that CIFAR's focus is on fundamental research where social and economic impacts are often hard to predict and stakeholders agree that this should remain its focus. This is reflected in the third-party evaluation findings that translation of research findings into social and economic impacts is not a primary goal of program members. However, stakeholders interviewed as part of this evaluation believe that CIFAR is putting a greater emphasis on exploring the potential applications of CIFAR research under CIFAR 2.0 and is encouraging researchers to engage with end-users to look for applications.

3.2.5 To what extent does CIFAR demonstrate efficiency and economy?

Key finding: CIFAR's virtual research model of serving as a facilitator of research collaborations suggests efficiencies compared to more traditional models used by other advanced research organizations. In addition, the program has been successful in leveraging external funding, exceeding federal leveraging requirements. Further conclusions could not be reached due to data limitations related to the organization's efficiency and economy.

CIFAR's efficiency and economy was assessed at two levels: the efficiency and economy of the organization itself and ISED's management and oversight of the contribution.

Efficiency and economy of CIFAR

CIFAR employs a "virtual" research model, meaning that it does not have a physical location to house its researchers as is the case for most other Canadian research organizations. CIFAR's approach is to allow researchers to stay with their home research institution in Canada or elsewhere, with CIFAR facilitating and supporting interaction and collaboration amongst researchers through in-person meetings. As a result, CIFAR does not incur costs for "bricks and mortar" research facilities or have to pay the salaries and benefits of researchers and their staff. Instead, CIFAR pays a relatively small amount to its researchers and leverages other sources of funding available to the researcher.

CIFAR has been successful in leveraging external funding. As indicated in Table 5, for every dollar invested by the federal government, CIFAR has leveraged an average of $1.62 of other funding, exceeding the federal minimum requirement of $1 in each year per federal dollar invested.

Table 5: Funding Leveraged from Federal Contribution
2012–13 2013–14 2014–15 2015–16 AVERAGE
$ leveraged from other sources per $1 federal funding $1.67 $1.31 $1.52 $1.98 $1.62
Source: CIFAR Summarized Financial Statements

The proportion of total funding devoted to non-program costs ranged from 14% to 16% during the evaluation period. The lack of comparable organizations with which CIFAR could be benchmarked coupled with limited efficiency and economy indicators and data (as discussed in section 2.5), presented challenges for the evaluators to draw further conclusions on CIFAR's efficiency and economy.

Efficiency and economy of ISED's oversight

SIS employs approximately 1/2 of an FTE for oversight activities. This includes the development and implementation of funding agreements, reviewing CIFAR reports and providing ongoing analysis and advice to the Minister of ISED and Minister of Science in relation to CIFAR. Given the size of the contribution, this suggests that ISED is managing these tasks efficiently.

top of page

4.0 Conclusions and recommendations

4.1 Relevance

4.2 Performance

4.3 Recommendations

The third-party evaluation led to six recommendations specific to CIFAR, which can be found in Appendix A. These recommendations were made in the context of CIFAR's Strategic Plan 2012–2017 (CIFAR 2.0) and are broader than the expected results stipulated in ISED's funding agreement.

The findings of this evaluation led to the following recommendation:

top of page

Appendix A: Third-party evaluation recommendations

Recommendations

Relevance

  1. In the context of the CIFAR 2.0 strategy, the institute should deepen its efforts to optimize the potential to engage researchers from around the world to address some of the emerging trends in the global research landscape. While it can continue to capitalize on program members' efforts towards diversification of research and funding partnerships, it could do more also in terms of purposeful scouting for program members, Global Scholars and potential long‐term institutional partnerships. Mechanisms such as the Global Call or those underpinning the attraction and retention of research leaders in Canada can be used even more strategically to suit such purpose.

Transformative Knowledge Outcomes

  1. The evaluation encourages CIFAR to continue to address the recommendation from the 2013 performance evaluation in regards to developing additional performance measures to better capture the effectiveness of CIFAR's activities and outputs with a view to assessing immediate and intermediate outcomes. A set of indicators in the PM Strategy centred on interactivity, the dynamics of research networks, social network analysis, and how such dynamics influence the research directions of program members would be better aligned with CIFAR's theory of change and business model.

Knowledge Outreach Outcomes

  1. CIFAR 2.0 should continue to closely monitor the level of influence that it exerts on the willingness and capacity of program members to identify, engage and respond to the needs of potential knowledge users. At the same time, it is recommended that CIFAR constantly communicate to its different stakeholders the value proposition that an enhanced knowledge outreach strategy potentially brings to the work of program members, and research programs more generally.

Building the Next Generation of Research Leaders

  1. In line with recent efforts to refocus the Global Academy program, CIFAR should revisit its operational strategy related to the broader objectives of attraction and retention of HQP to Canada with CIFAR's assistance. CIFAR should either identify the means by which it can influence the achievement of the outcome related to the attraction and retention of highly qualified personnel, or no longer hold itself accountable for this outcome.

Partnership Building

  1. CIFAR should make efforts to further enhance its visibility, reputation and capacity to establish long-term institutional partnerships with the research and funding communities beyond North America. There is an opportunity for CIFAR to further leverage the personal networks and contacts of program members, as well as challenging itself to become a leading international model, as a unique organization that is able to mobilize, with few resources, the collective efforts and capacities of groups of well-established and emerging researchers towards addressing questions of relevance to the world. The Institute should explore more actively partnership opportunities emerging from the growing presence of researchers and research funding organizations from the developing world.

Demonstration of Efficiency and Economy

  1. In order to draw more meaningful conclusions about the efficiency and economy of CIFAR's activities, it is recommended the Institute carry out additional studies that examine program delivery mechanisms and processes. The Global Call offers a good opportunity. While this is the first time that the Institute has carried out a competitive research program development, the mechanism of competitive calls is already well established among STI funders in Canada and abroad. The merits of the CIFAR approach should be contrasted with similar processes carried out elsewhere. CIFAR can also contrast its different approaches to research support, research capacity building and knowledge outreach in light of similar/alternative processes carried out elsewhere.

top of page

Appendix B: CIFAR's Research Programs

PROGRAM DESCRIPTION
Azrieli Brain, Mind & Consciousness Seeks to examine the neural underpinnings of consciousness, leading to better treatments for mental health disorders and insights into the most profound questions about human nature.
Bio-inspired Solar Energy Aims to improve our ability to use solar energy by learning lessons from the remarkable effectiveness of photosynthesis in plants, algae and photosynthetic bacteria.
Child & Brain Development Explores the core question of how social experiences and settings affect developmental biology and help set early trajectories of lifelong development and health.
Cosmology & Gravity Seeks to tell a comprehensive story of the structure and evolution of the universe, from its first moment of existence to its ultimate fate.
Genetic Networks Devoted to mapping the biological networks that translate genomes into complex traits and improving our fundamental understanding of biological systems to enable new treatments and preventive measures.
Humans & the Microbiome Examines the human microbiome — the microbes that live in and on us — and the role it plays in human development and behaviour, as well as how it is affecting our evolution and society.
Institutions, Organizations & Growth Takes an integrated approach to basic questions, such as what makes some countries rich and others poor; what makes some societies violent and others peaceful; what makes some states strong and others weak; always examining the crucial roles of different types of institutions.
Integrated Microbial Biodiversity Explores the diverse microbial world that surrounds and permeates human life. Program members are transforming our understanding of biodiversity and changing approaches to medicine and health, environmental sustainability and evolutionary biology itself.
Learning in Machines & Brains Aims to contribute to the understanding of the computational and mathematical principles that enable intelligence through learning, be it in brains or in machines.
Molecular Architecture of Life Seeks to untangle the details of the complex molecular processes that underlie all living systems, with implications for everything from our understanding of evolution to our ability to treat disease.
Quantum Information Science Unites computer scientists and physicists in an effort to harness the strange and fascinating properties of the quantum world, where the mere act of observing an object changes its nature, with the aim of building quantum computers.
Quantum Materials Invents and explores materials whose novel and unusual electronic properties, like superconductivity, could revolutionize technology.
Social Interactions, Identity & Well-Being Develops a new approach to understanding economic and social issues by exploring the complex ways in which health, happiness and well-being are shaped by social identities and interactions.
Successful Societies Explores the roots and effects of social inequalities and asks the question: What makes a society successful?
Source: CIFAR 2015–16 Annual Performance Report

top of page

Appendix C: CIFAR Research impact examplesFootnote 38

Significant economic policy changes

Public policy changes, improved delivery of social services and health care and increased life satisfaction

Transformative social or technical innovation

top of page

Management Response and Action Plan

A - For inclusion in the report

The findings and recommendation of the Canadian Institute for Advance Research Report were presented to program management. Management has agreed with the findings included in this report and will take action to address all applicable recommendations by October 2017.

B - For follow-up purposes - Detailed actions to address the recommendations in the report

Detailed actions to address the recommendations in the report
Recommendation Planned Action on the Recommendation Responsible Official (position) Target completion date

Recommendation 1

In order to draw more meaningful conclusions about the economy and efficiency of CIFAR's operations, SIS should work with CIFAR to update its Performance Measurement Strategy to include additional efficiency indicators. Consideration could be given to identifying efficiency-related comparables (i.e. benchmarking) and targets to further assess CIFAR's operations.

Agreed.

SIS will work with CIFAR, ISED's Audit and Evaluation Branch (AEB) and Corporate Management Sector (CMS) to establish economy and efficiency measures as part of the work to update CIFAR's Performance Measurement Strategy to take into account renewed and enhanced funding as announced in Budget 2017. This will include:

  • qualitative indicators that provide evidence of continuing efforts to strive for optimal efficiency at CIFAR; and
  • searching for appropriate benchmarks for expenditures on administration and overhead.

Data sources and methods for collecting the information will also be established.

 

Director General, Program Coordination Branch

 

October 2017

top of page

Date modified: