Evaluation of Industry Canada's Contribution to CANARIE
This publication is also available online in HTML at www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03749.html.
To obtain a copy of this publication or an alternate format (Braille, large print, etc.), please contact: The Audit and Evaluation Branch — Office of the Chief Audit Executive 343‑291‑2356.
Permission to Reproduce
Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Industry Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Industry Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Industry Canada.
For permission to reproduce the information in this publication for commercial purposes, please contact the:
Web Services Centre
C.D. Howe Building
235 Queen Street
Ottawa, Ontario, Canada
Telephone (toll‑free in Canada): 1‑800‑328‑6189
Telephone (Ottawa): 613‑954‑5031
TTY (for hearing‑impaired): 1‑866‑694‑8389
Business hours: 8:30 a.m. to 5:00 p.m. (Eastern Time)
© Her Majesty the Queen in Right of Canada,
represented by the Minister of Industry, 2014
Catalogue Number Iu4-163/2015E-PDF
Aussi offert en français sous le titre Évaluation de la contribution d'Industrie Canada à CANARIE.
Presented to the Departmental Evaluation Committee on
Approved by the Deputy Minister on
Table of contents
- Executive Summary
- 1.0 Introduction
- 2.0 Methodology
- 3.0 Findings
- 4.0 Conclusions and Recommendations
- Appendix A – Executive Summary, Evaluation of CANARIE, Final Report
- Appendix B – CANARIE Logic Model
- Appendix C – Overview of Third‑Party Evaluation Data Collection Methodology
- Evaluation of Industry Canada's Contribution to CANARIE Management Response and Action Plan
|AURP||Association of University Research Parks|
|BL‑NCEs||Business Led Centres of Excellence|
|CAF||Canadian Access Federation|
|CANARIE||Canada's Advanced Research and Innovation Network|
|CDS||Content Delivery Service|
|CECR||Centres of Excellence for Commercialization and Research|
|CIO||Chief Information Officer|
|CFREF||Canada First Research Excellence Fund|
|DAIR||Digital Accelerator for Innovation and Research|
|DC 150||Digital Canada 150|
|ICT||Information and communications technology|
|ICTAM||ICT Association of Manitoba|
|MOU||Memorandum of Understanding|
|NCE||Networks of Centres of Excellence|
|NREN||National Research and Education Network|
|OECD||Organisation for Economic Co‑operation and Development|
|ORAN||Optical Regional Advanced Network|
|PAA||Program Alignment Architecture|
|PAES||Performance, Audit and Evaluation Strategy|
|REANNZ||Research and Education Advanced Network New Zealand|
|RDC||Research Data Canada|
|SITT||Spectrum, Information Technologies and Telecommunications Sector|
List of Figures
|Figure #||Figure title|
|1||CANARIE Network Map|
Canada's Advanced Research and Innovation Network (CANARIE) is a not‑for‑profit corporation that provides research and education communities in Canada with a high‑speed network to transmit data which supports Canadian‑based research, discovery and innovation. It is supported by membership fees, with the major investment provided by the Government of Canada. It was created to ensure Canadians could engage in leading‑edge research that requires more capability and more capacity than the commercial Internet. In addition to the network, CANARIE also manages a Research Middleware funding program and provides funding to the research community. This funding helps develop research platforms and components that accelerate discovery, reduce duplication in software development and enables broad use of digital infrastructure, including the advanced computing infrastructure hosted by Compute Canada.
Evaluation Purpose and Methodology
The objectives of this evaluation are to address the core issues of relevance and performance in accordance with the Directive on the Evaluation Function.
Under CANARIE's 2012 contribution agreement with the Government of Canada, CANARIE was responsible for commissioning an independent third‑party evaluation of its funding to eligible projects by . To meet federal evaluation requirements under the FAA and the Policy and Directive on Evaluation, Industry Canada was also required to conduct an evaluation of its contribution to CANARIE.
Industry Canada's evaluation used the independent third‑party evaluation as the primary source of information to evaluate CANARIE's progress toward achieving its expected results against the objectives outlined in its Contribution Agreement. Industry Canada's data collection methods (document and literature review and interviews) were mainly used to validate and supplement the third‑party evaluation.
There is a continued need for the funding of CANARIE. CANARIE helps to address Canada's innovation gap and more specifically its large scale research collaboration gap. Its continued need is reflected in the growing demand for its services and the potential negative impacts of its discontinuation on research and education.
The funding of CANARIE is aligned with federal government priorities and with Industry Canada's Strategic Outcome relating to advancements in science and technology, knowledge and innovation that strengthen the Canadian economy. Federal support of CANARIE is also consistent with the roles and responsibilities of Industry Canada and is similar to support provided in other countries around the world. Private sector and provincially sponsored options are not feasible to provide the same level of effectiveness.
CANARIE has generally been successful in achieving its objectives and related short‑term outcomes. Specifically, over the evaluation period, it has expanded and upgraded its network capabilities, increased and improved access to the network and kept up with network demand. Further, CANARIE has facilitated research collaboration and innovation and garnered international respect.
During its current mandate, CANARIE has developed 48 reusable software platforms and 12 network enabled platforms. Through this and other activities, CANARIE has helped researchers to seamlessly access distributed data, tools and research instruments across the world resulting in increased collaboration and innovation.
In terms of spurring private sector innovation, CANARIE has served over 250 users through its Digital Accelerator for Innovation and Research (DAIR) program, providing a test bed for product development and testing. The impact on participants has been to reduce product development costs and risks and help them develop proof of concept. As a result, participants reported that DAIR helped them in accelerating innovation and reducing time to market.
CANARIE's success in achieving longer‑term outcomes can be better assessed as new program components have time to realize longer‑term effects and as the performance measurement system begins to capture the longer‑term outcomes.
CANARIE has demonstrated continuing efforts to achieve efficiencies, has kept overhead costs within the maximum amount allowed under the Contribution Agreement, and is generally on track to meet its cost‑recovery targets. Industry Canada's relatively small investment in CANARIE is efficient in that it provides the necessary infrastructure to support the significantly larger total government investments in R&D.
The evaluation findings lead to the following recommendation:
- SITT should continue to monitor the implementation of CANARIE's performance measurement system to ensure that long‑term outcomes are appropriately tracked to support public reporting, departmental decision‑making and future evaluations.
Canada's Advanced Research and Innovation Network (CANARIE) was established in 1993 as a not‑for‑profit corporation that is supported by membership fees, with the major investment provided by the Government of Canada. CANARIE's membership includes more than 70 organizations from academia, industry, government and the not‑for‑profit sector. It was created to ensure Canadians could engage in leading‑edge research that requires more capability and more capacity than the commercial Internet.
CANARIE supports Canadian‑based research, discovery and innovation by providing research and education communities in Canada with a high‑speed network to transmit data. Over 19,000 kilometres of fibre‑optic cable connects over one million usersFootnote 1 at over 1,100 Canadian institutions, including universities, colleges, research institutes, hospitals, and government laboratories (See Figure 1). In addition to the network, CANARIE also manages the Research Middleware funding program and provides funding to the research community. This funding helps develop research platforms and components that accelerate discovery, reduce duplication in software development and enables broad use of digital infrastructure, including the advanced computing infrastructure hosted by Compute Canada.
The CANARIE network gives users the ability to manipulate massive amounts of data, and access collaborative software platforms, high performance computing, remote sensors and complex scientific instruments both within Canada and internationally. The commercial Internet cannot technically or cost‑effectively handle this high volume research traffic. Through CANARIE, researchers and educators also have access to over 100 international peer networks in 80 countries to facilitate global research collaboration. All nation members of the Organisation for Economic Co‑operation and Development (OECD) have a publicly funded network similar to CANARIE.
Recipients of federal research funding use CANARIE to access data and facilitate research collaboration, including the Networks of Centres of Excellence (NCEs), the majority of the Centres of Excellence for Commercialization and Research (CECR), Canada Research Chairs, and the Business Led Centres of Excellence (BL‑NCEs). CANARIE connects over one million users at universities, colleges, hospitals and health institutions, research consortia, federal and provincial government labs, cultural institutions, and hundreds of thousands of K–12 students at over 2,000 K–12 schools.
CANARIE and the Government of Canada (represented by the Minister of Industry) outline the objectives for CANARIE for the period of to in the signed 2012 funding agreement:
- Network Operations – continue to operate the CANARIE network as essential research infrastructure;
- Technology Innovation – develop, demonstrate, and implement next generation technologies to advance the CANARIE network as a leading edge research network; and,
- Private Sector Innovation – leverage the CANARIE network to assist firms operating in Canada and Canadian universities to advance innovation and commercialization of products and services to bolster Canada's technology innovation capabilities.
The Information and Communications Technologies Branch within the Spectrum, Information Technologies and Telecommunications Sector of Industry Canada is responsible for ongoing management and monitoring of the contributions to CANARIE. CANARIE is governed by a Board of Directors comprised of leaders in education, research and innovation. They are elected for one‑year renewable terms. CANARIE relies on the commitment of its Board members to review and approve new initiatives that set direction for the organization, make final decisions on programs to be funded, approve corporate policies, and oversee compliance.
The federal government has supported five generations of CANARIE's advanced research network with a total investment of $529.5 million since 1993. Most recently, the Government of Canada provided CANARIE with a $62 million contribution over three years from 2012–13 to 2014–15. CANARIE's funding agreement includes a provision that CANARIE may reserve up to 17.4 percent of its contribution ($10,788,000 over three years) to support costs and expenses related to the administration and management of the CANARIE network, including leased premises, salaries for 21 employees, costs pertaining to federal government reporting requirements, communication and demonstration activities, and for the retention of financial, legal, audit, performance measurement and evaluation resources directly related to the CANARIE network.
Industry Canada's operating costs related to the monitoring of CANARIE are approximately $60 thousand per year. This includes salary costs for a 0.5 FTE and travel costs for a Director General to attend CANARIE's Board of Director meetings. Industry Canada's 2012–2015 funding agreement with CANARIE also required it to explore and implement a cost‑recovery strategy. CANARIE's performance in this regard is further discussed in Section 3.2.2.
It should be noted that CANARIE is the national backbone of the network and that the Optical Regional Advanced Networks (ORANs) provide the regional connections to end users. In 2012, CANARIE and its network partners undertook a rigorous exercise based on standard financial accounting practices to more precisely identify their respective contribution to the overall network. Through this exercise, it was estimated that 52% of the total networks costs are contributed by the provinces and territories through the ORANs and 48% by federal government through CANARIE.
A logic model is a visual representation that links a program's activities, outputs and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program, policy or initiative is expected to achieve its objectives. It also provides the basis for developing evaluation strategies, including the evaluation matrix.
The logic model for the program (see Appendix B) was developed by CANARIE as part of its 2012–2015 Performance, Audit and Evaluation Strategy (PAES). Note that this evaluation assesses CANARIE's performance based primarily on the objectives outlined in Industry Canada's contribution agreement with CANARIE as opposed to reporting explicitly on each specific outcome in the logic model. This is in line with the approach taken by the third‑party evaluation for its evaluation report, which was a key source of information for this evaluation.
This section provides information on the evaluation strategy, approach, objectives and scope, the specific evaluation issues and questions that were addressed, the data collection methods, and data limitations for this evaluation.
2.1 Evaluation Strategy
Under 42(1) of the Financial Administration Act each department is required to conduct a review every five years of the relevance and effectiveness of each ongoing program for which it is responsible. Under CANARIE's 2012 contribution agreement with the Government of Canada, CANARIE was responsible for commissioning an independent third‑party evaluation of its funding to eligible projects by . To meet federal evaluation requirements under the FAA and the Policy and Directive on Evaluation, Industry Canada is also required to conduct an evaluation of its contribution to CANARIE.
Industry Canada's evaluation used CANARIE's third‑party evaluation as the primary source of information to evaluate CANARIE's progress toward achieving its expected results against the objectives outlined in its Contribution Agreement. Industry Canada`s data collection methods (document and literature review and interviews) were mainly used to validate and supplement the more comprehensive third‑party evaluation already conducted.
The focus of the third‑party evaluation was CANARIE's progress toward achieving its three objectives, the continued need for and relevance of CANARIE, the alignment of CANARIE with Government priorities and with federal roles and responsibilities, CANARIE's effectiveness in achieving program objectives and expected outcomes, and the efficiency and economy of CANARIE in addressing its mandate (Appendix A provides the executive summary of the third‑party evaluation). An Industry Canada Evaluation Directorate representative was a member of CANARIE's Evaluation Steering Committee to help ensure alignment between the two evaluations.
The evaluation of Industry Canada's contribution to CANARIE was conducted by the Audit and Evaluation Branch at Industry Canada and will be referred to as "the evaluation" throughout the rest of the document.
2.2 Evaluation Scope and Objectives
The objectives of this evaluation are to address the core issues of relevance and performance in accordance with the Directive on the Evaluation Function. Industry Canada's evaluation of CANARIE adopted a risk‑based approach and the evaluation was calibrated to account for the comprehensive third‑party evaluation undertaken in 2014. The evaluation covers the period of to present.
2.3 Evaluation Questions
The evaluation will address the following questions on relevance and performance:
- Is there a continued need for Industry Canada's contribution to CANARIE?
- To what extent are the objectives and support of Industry Canada's contribution to CANARIE aligned with: i) departmental strategic outcomes; ii) federal priorities and strategies?
- Does providing support to CANARIE align with the roles and responsibilities of the federal government?
- To what extent has Industry Canada's contribution to CANARIE achieved its objectives and related expected outcomes?
- How efficiently and economically is Industry Canada's contribution to CANARIE being delivered? Is there a more cost effective way of achieving expected results?
2.4 Data Collection Methods
Data collection and analysis was primarily undertaken in the independent third‑party evaluation conducted by Nordicity in conjunction with Bytown Consulting. The Industry Canada evaluation sought to verify the findings of the third‑party evaluation while providing greater focus on the Industry Canada monitoring of the program. The third‑party evaluation employed five data collection methods: document and file reviews; a survey of Chief Information Officers (CIOs), Vice‑Presidents Research (VPRs), researchers and developers, Digital Accelerator for Innovation and Research (DAIR) program users; stakeholder interviews; case studies; and a comparative analysis of international peer organizations/networks. Additional information regarding the third‑party evaluation's methodology can be in found in Appendix C.
This evaluation included a further document review, literature review, and interviews, the details of which are explained below.
2.4.1 Document Review
The review included the third‑party evaluation report, Federal Budgets and Speeches from the Throne, Treasury Board Submissions and other relevant policy documents, Departmental Reports on Plans and Priorities, Departmental Performance Reports, as well as CANARIE Business Plans, Annual Reports and other organizational documents.
2.4.2 Literature Review
This review was conducted primarily to address the core evaluation issues of continued need and federal roles and responsibilities. Specifically, the literature review examined the continued need to increase research and innovation capacity and the role of federal funding in supporting R&D generally and high‑speed networks more specifically, in Canada and other jurisdictions.
The objective of the interviews was to collect information, including views, explanations and factual information from Industry Canada and CANARIE management that address the evaluation questions, as well as to obtain additional information/clarification in relation to the third‑party evaluation report where required. A total of eight interviews were conducted, five with CANARIE management and board members, two with Industry Canada representatives, and one with the former IC program Director General.
This evaluation relied on the independent third‑party evaluation, particularly to address the evaluation questions related to performance. The firm responsible for completing this evaluation was selected by CANARIE's Board of Directors; as such, there is a risk that the third‑party evaluation may not be perceived as neutral. A representative from Industry Canada's Evaluation Directorate severed as a member of Steering Committee for the third‑party evaluation, providing insight into the quality and impartiality of the work done. In addition, in an effort to validate the independent third‑party evaluation, IC conducted this supplemental evaluation of its contribution to CANARIE, involving additional interviews, a document and literature review.
CANARIE receives funding from IC to undertake three primary activities (objectives) as outlined in Section 1.1. In this section, evidence for the continued need related to each of these activities is presented. It should be noted that the latter two activities serve to enhance the functionality of the network and to expand its usage to private sector innovative applications.
The Jenkins Report, Innovation Canada: A Call to Action, suggests that Canada suffers an innovation gap and more specifically a large scale research collaboration gap. It further describes innovation inputs that can address these gaps, one of which is "networks, collaborations and linkages that enable innovative partners to pool staff and resources, share information, risks and costs".Footnote 2
CANARIE supports Canadian‑based research, discovery and innovation by providing research and education communities in Canada with a high‑speed network to transmit data. Over 19,000 kilometres of fibre‑optic cable connect over one million users at over 1,100 Canadian institutions, including universities, colleges, research institutes, hospitals, and government laboratories. CANARIE is the national backbone that connects researchers across Canada to each other and to research data, colleagues and instruments in over 100 countries.
As reported in the third‑party evaluationFootnote 3, academia and industry suggest that CANARIE is fundamental to research and education (R&E), and to achieving a successful innovation eco‑system as it facilitates collaboration and data transfers between researchers and educators, in Canada and across the globe, and makes digital content readily accessible. Further, stakeholders reported that data‑intensive research activities are highly dependent on the CANARIE Network to transmit large volumes of data globally in a reliable, efficient and secure manner across a broad spectrum of academic disciplines including the natural, health, and social sciences; engineering; and the humanities.
The literature suggests that the availability of cost‑effective and cutting‑edge National Research and Education Network (NREN) services (such as CANARIE) enables and encourages technological spillover into the commercial sector, which ultimately benefits society as a whole. Conversely, the absence of such facilities hampers such development and can exclude countries from achieving advances that could help their economic development.Footnote 4
CANARIE's increase in network traffic over the evaluation period is also a reflection of the need for the program. During the last five years, CANARIE traffic has grown by 385 percent. According to CANARIE traffic reports, this averages to approximately 77 percent growth a year from 2009–10 to 2013–14. Third‑party evaluation interviewees and survey respondents mostly agreed that the demand for high‑speed transmission of data is growing exponentially for scientific and engineering research projects and for the development of new innovative technology applications. Industry Canada and CANARIE staff and board members interviewed as part of this evaluation also agreed that growth in demand will continue to be exponential.
The third‑party evaluationFootnote 5 also looked at the potential negative impacts of discontinuation of the services provided by CANARIE. It concluded that, "In all, the absence of CANARIE would have a profoundly negative effect on research, education and innovation in Canada". Third‑party evaluation interview and survey respondents outlined some of the following potential impacts:
- "Without CANARIE, interprovincial and international linkages for Canadian researchers and educators would be seriously jeopardized and the Canadian position on the international arena, fragmented;"
- "Without CANARIE and its community of resources, research would become more isolated and happen in silos. It would become much less creative and less productive."
- "It would put Canada at a significant disadvantage, nationally and internationally."
Furthermore, multiple third‑party evaluation interviewees indicated that the results of discontinuing CANARIE would be "catastrophic" and "disastrous," and that some research and educational activities would "come to a halt." The evaluation suggests that the impact would be far greater on the smaller provinces, as they are dependent to a higher degree on the connectivity afforded by the CANARIE Network, and many institutions would be hard‑pressed to find alternatives to current levels of connectivity through CANARIE, as equivalent services are not available over commercial channels.
In addition to the network, CANARIE also manages a range of funding programs that advance digital infrastructure in Canada. The third‑party evaluationFootnote 6 concludes that CANARIE enables research collaboration and technology innovation by supporting the development of applications and interface tools and software platforms that allow researchers to access globally distributed research data, tools and colleagues. It further concludes that the discontinuation of CANARIE programs and services would have a highly negative impact on users, as well as Canada's ability to maintain leadership in research and development.
Private Sector Innovation
The primary CANARIE activity to support private sector innovation is the DAIR program. DAIR leverages the CANARIE network by offering a Canadian cloud‑computing platform to Canadian small and medium size businesses to accelerate product development and decrease time to market. The third‑party evaluationFootnote 7 provides evidence of the need for DAIR as reflected in the importance that surveyed users reported with respect to indicators of how the program helped them in developing innovative products and services. Specifically, eighty‑eight percent of DAIR users reported that the program was "very" or "critically" important to them for reducing their development costs; 67 percent for reducing the risk associated with conducting research; 66 percent for helping them to create proof of concept; and 62 percent for accelerating innovation.
3.1.2 To what extent are the objectives and support of Industry Canada's contribution to CANARIE aligned with: i) departmental strategic outcomes; ii) federal priorities and strategies?
CANARIE's objectives during its most recent mandate were directly aligned with the Government's priorities related to its 2007 Science and Technology (S&T) Strategy. Specifically, CANARIE contributed to the three Canadian S&T advantages to be fostered (i.e. People Advantage, Entrepreneurial Advantage, and Knowledge Advantage).
The S&T People Advantage states that "Canada must be a magnet for the highly skilled people we need to thrive in the modern global economy with the best‑educated, most‑skilled, and most flexible workforce in the world."Footnote 8 CANARIE contributes to this by connecting researchers, professors and students at every university in Canada and enabling new ways to train highly qualified people and facilitating science and research that spans many scientific disciplines, technologies and industrial sectors.
The S&T Entrepreneurial Advantage states that "Canada must translate knowledge into commercial applications that generate wealth for Canadians and support the quality of life we all want."Footnote 9 CANARIE contributed to this by bolstering Canada's international reputation in advanced networking, with benefits flowing to both the research and business sectors; delivering advanced networking capability to researchers who partner with industry, and extending the research capacity of companies that participate in collaborative projects; and awarding funds to companies and their research partners to accelerate research and development in areas that rely on advanced networking, such as Green IT.
Finally, CANARIE supported the S&T Knowledge Advantage that "Canadians must be positioned at the leading edge of the important developments that generate health, environmental, societal, and economic benefits."Footnote 10 CANARIE contributed to this by enabling Canadian participation in data‑driven research, big science projects and global R&D collaborations, enabling researchers to develop new knowledge and expertise.
Moving forward, CANARIE will continue to be aligned with the Government's priorities as outlined in its recently released S&T strategy, Seizing Canada's Moment: Moving Forward in Science, Technology and Innovation. This strategy highlights the role that CANARIE has played in ensuring that Canada's research infrastructure is world‑class and points to its continued support to CANARIE for the provision of high‑speed networking and software tools to enhance Canada's research capacity.
The new strategy also continues with the People and Knowledge pillars but enhances and broadens the Entrepreneurial pillar to encompass Innovation. CANARIE will continue to be aligned with this pillar in that one of CANARIE's explicit objectives is to support private‑sector innovation and one of its internal operational objectives is to advance its services through technology innovation.
Digital Canada 150
On , the Minister of Industry announced the Digital Canada 150 plan. This plan aims to support Canada in the digital age through five pillars, one of which is "Economic Opportunities". CANARIE is aligned with this pillar, particularly through DAIR, which has facilitated the acceleration of innovation and commercialization, and allowed "Canadian companies large and small [to] use digital tools to boost productivity, develop their businesses, and capture growing markets at home and abroad." Digital Canada 150 highlights the DAIR program as a notable investment in this area, further illustrating CANARIE's direct alignment with the national digital strategy.
CANARIE is also aligned with the Digital Canada 150's pillar, "Connecting Canadians," which aims to increase and improve connectivity across Canada to enable "e‑commerce, high resolution video, employment opportunities and distance education—providing rural and remote communities with faster, more reliable online services." Through partnership with the ORANs, CANARIE has significantly contributed to parallel advancements in the Canadian research and education landscape, and its mandate is complementary to this digital strategyFootnote 11. CANARIE activities are also aligned with the "Digital Government" pillar, as the network is among the resources used by 50 federal government departments and laboratories.Footnote 12
CANARIE is also contributing to the federal government's "Open Science" initiative by providing the network, enabling connections to Compute Canada (for data storage and computation) and funding research software that makes it easy to archive, tag, and retrieve data.Footnote 13
In the 2014 Budget, the Government of Canada announced the creation of the Canada First Research Excellence Fund (CFREF). The CFREF is a $1.5 billion dollar investment that addresses the need for Canada's research‑intensive universities to compete on the world's stage and develop and attract the research talent that will ensure Canada's future prosperity. CANARIE is aligned with this direction as it provides the backbone digital infrastructure, services and software tools that support and accelerate Canadian research excellence and innovation, and promote accessibility and mobility of knowledge.Footnote 14
CANARIE is also aligned with Industry Canada's 2014–15 Program Alignment Architecture (PAA) and in particular Industry Canada's Strategic Outcome 2: Advancements in science and technology, knowledge and innovation strengthen the Canadian economy. More specifically, the contributions to CANARIE are consistent with the priorities listed under the Science, Technology and Innovation Capacity Program Activity which falls under this Strategic Outcome. This Program Activity sets the strategic direction for policies and programs that support and stimulate research, development and innovation in Canada. Under this Program Activity, Industry Canada, in collaboration with its portfolio partners, other government departments and external stakeholders from the private and public sectors, fosters an environment that is conducive to innovation and promotes scientific excellence. Contributions to CANARIE are consistent with the Strategic Outcome and associated Program Activity outlined in that CANARIE through its network infrastructure, technology innovation support and private sector support, facilitates and supports knowledge and innovation to strengthen the Canadian economy.
3.1.3 Does providing support to CANARIE align with the roles and responsibilities of the federal government?
In terms of the roles and responsibilities of Industry Canada as set out in the Department of Industry Act of 1995, there is a clear alignment with the objectives underlying the department's support to CANARIE. The Act provides the Minister with the powers, duties and functions related to industry and technology, science and telecommunications, all which relate to funding of CANARIE. Further, it gives the Minister the responsibility to strengthen the national economy, foster and promote science and technology in Canada and to encourage the fullest and most efficient and effective development and use of science and technology. Again the objectives of support to CANARIE are consistent in that CANARIE is a technology that supports S&T in Canada to provide the economic benefits associated with innovation. Finally, the Act provides the Minister with the authority to implement national programs (such as CANARIE) consistent with objectives set out for the Department in the Act and to provide grants and contributions in conjunction with such programs (as it does for CANARIE). Industry Canada staff also suggested that CANARIE is an essential infrastructure element that underpins all other federally funded research and development.
CANARIE is the national backbone that connects regional and territorial networks and provides international connections to over 100 countries. Popov suggests that, "…
the government plays and should play a vital role in the development an institution/organization responsible for the internal and external connectivity of the national research and education institutions". This is also consistent with the argument that both science and education have a crucial place in creating economic growth, fostering national and international understanding and cooperation, and thus creating the right climate for political and social stability that improve the well‑being and the life of every citizen.Footnote 15 In terms of overlap or duplication with the provinces, the Leadership Council, a voluntary collective of leaders who represent key stakeholders that have a part to play in the creation of a world‑leading, advanced digital infrastructure ecosystem for Canada, concludes that CANARIE and the regional networks have a culture of collaboration and have worked to avoid duplication.Footnote 16
In considering alternate service providers, the potential options that might be considered are the private sector and the provinces. The third‑party evaluationFootnote 17 reports that stakeholders suggest that if the private sector were given the mandate to connect university and government research, it would be more expensive, less seamless, and services would not be as geographically complete. In terms of a provincial alternative, the third‑party evaluation stakeholders suggested that the most likely arrangement would be a process whereby the provincial ORANs would form a common buying group and broker commercial telecommunications providers. The stakeholders suggested that such a model would be inefficient, and require more staff to be hired by the provinces to develop the relationships and knowledge to orchestrate national high‑speed networks. Moreover, with different mandates and funding capacity, a buying group would not provide the specialized services nor conduct the R&D that further leverages the investment in research.
Federal support for CANARIE is also consistent with what is done in many other countries. Similar networks exist in all OECD countries. For example Australia's Academic and Research Network (AARNET), was a non‑profit organization established in 1989 by a group of Australian universities and research institutions, with the aim of providing high‑speed internet connectivity to serve the academic and research community. Research and Education Advanced Network New Zealand (REANNZ) is the Crown‑owned company that owns and operates a high‑speed, unrestricted broadband network for the New Zealand education, research and innovation communities.
The US has invested heavily in computational facilities and high‑speed networks – both regional and national. Much of the public discussion has focused on the benefits of such investments for industrial competitiveness, the requirements of big science for extreme computing and more recently data intensive life sciences.Footnote 18
3.2.1 To what extent has Industry Canada's contribution to CANARIE achieved its objectives and related expected outcomes?
This section provides an assessment of the extent to which the objectives and expected outcomes for Industry Canada's contribution to CANARIE outlined in Section 1.1 have been achieved.
It should be noted that the Technology Innovation activities serve to enhance access to and functionality of the network and would be expected to contribute to the expected long‑term outcomes for the network. As such, the analysis contained in this section focuses on the expected short‑term outcomes for the Technology Innovation activities and the long‑term achieved outcomes for the network will be assumed to be due to both Network Operations and Technology Innovation activities.
It is also important to note that, under its Contribution Agreement with Industry Canada, CANARIE was required to develop a Performance, Audit and Evaluation Strategy (PAES), which included performance indicators associated with expected outcomes related to each of the three CANARIE objectives. Further, it was obliged to submit to the Minister, an Annual Performance Report based on the indicators outlined in the Strategy.
CANARIE did develop such a Strategy and has been submitting the required Annual Performance Reports based on CANARIE's performance measurement system from its 2012–2015 PAES. However, a review of the most recent CANARIE Annual Report found that it did not report on many of the activity and output indicators or any of the outcome indicators outlined in the PAES. CANARIE staff said that they had previously indicated to Industry Canada that, given CANARIE's three‑year mandate and the cost‑recovery requirement during the first year, it was not feasible to report these indicators with less than two years of operational period data for CANARIE's activities. CANARIE staff indicated that their intention is to continue to collect information related to all of the performance indicators contained in the PAES. With respect to long‑term outcomes, it should also be noted that these were recently developed as part of CANARIE's 2012–2015 PAES and were expected to come to fruition within 3–5 years.
The consequences for the third‑party evaluation and this evaluation were that some of the metrics were not available that could provide a complete picture of the success in achieving the objectives for CANARIE. Nevertheless, the third‑party evaluation was able to collect through surveys and interviews and other methodologies, sufficient information for the evaluation.
As indicated at the onset of this report, this section relies primarily on the information collected in the third‑party evaluation. The third‑party evaluation also relied on a summary of stakeholder consultation findings from the consultations conducted by Monitor Deloitte in the development of CANARIE's 5‑year strategic plan which was provided by CANARIE.
The remainder of this section is organized into each of the three CANARIE objectives.
The degree of success in addressing the objectives related to network activities was assessed in relation to the following evaluation indicators:
- Evidence CANARIE has expanded and upgraded advanced network capabilities and infrastructure for use by Canada's research and education community;
- Degree to which the capacity of the CANARIE network has kept up with demand;
- Evidence CANARIE has increased and improved access to and use of the network by real and virtual organizations;
- Degree to which CANARIE has enhanced Canada's involvement in networking and networking‑focused collaborations for scientific and other research; and,
- Evidence CANARIE has helped reinforce Canada's position as a recognized leader in the development and use of advanced research networks.
Expanded and upgraded advanced network capabilities
The third‑party evaluation documented the activities that CANARIE has undertaken to expand and upgrade the network capabilities. In this regard, the Monitor Deloitte report concluded that, CANARIE is committed to continual infrastructure upgrades to adapt to emerging trends. In the current mandate, CANARIE is in the process of implementing a dedicated ultra‑high‑speed (100G, or 100 billion bits per second) network across Canada, which the third‑party evaluation reported is on schedule.
The CANARIE Network serves all the provinces and territories of Canada and the third‑party evaluation concluded that it plays an important role in Canada's overall digital infrastructure. This conclusion was based on responses from interviewees and open‑ended comments from survey respondents in the CANARIE 2014 Survey, which showed that there was agreement that CANARIE is an important component of Canada's research infrastructure and will continue to be needed for a future of new and innovative data intensive applications emerging in fields such as genomics, astronomy, and bioinformatics.
CANARIE traffic has increased by 385 percent from 2009–10 to 2013–14 and the third‑party evaluation reported that users indicated that CANARIE provided reliable and sufficient capacity to meet their evolving needs, especially but not exclusively in applications involving very large datasets. It further concluded that, during its evaluation period (April 2010–May 2014), CANARIE had become more accessible, particularly to private industry connections and consortia, the education sector, research centres and parks, libraries, and cultural institutions.
Degree to which the capacity of the CANARIE network has kept up with demand
Interview and survey respondents to the third‑party evaluation indicated that R&E users were satisfied that CANARIE had been able to meet the rise in demand for its services, and mostly agreed that CANARIE was well‑positioned to meet further increases in future traffic demand.
CANARIE staff indicated that it looks ahead to anticipate future requirements while monitoring current requirements to ensure that it doesn't have to pay for excess capacity. One way that CANARIE is able to keep up with demand without developing excess capacity is by operating a hybrid network. The third‑party evaluation explains that as a hybrid network, it is equipped with leading‑edge optical and routing equipment, enabling CANARIE to offer traditional IP network services and Lightpath services (dedicated end‑to‑end connections) while continuing to develop new network service offerings based on the most advanced technology available. With this hybrid infrastructure, CANARIE offers greater flexibility in service offerings in order to meet the changing needs of users, scaling up from users with smaller data exchange needs up to very intensive data applications, such as high energy physics applications.
The third‑party evaluation concludes that, over the past 15 years CANARIE has introduced major improvements through at least three significant upgrades, and during every upgrade they have been deemed excellent, by those consulted for its evaluation, in working with their partners to facilitate the changes.
Increased and improved access to and use of the network by real and virtual organizations
CANARIE and its NREN partners connect more than:
- 89 universities, 101 colleges, and 47 CEGEPs
- 127 provincial and federal government labs and research parks
- 62 hospitals and health networks
- 24 cultural institutions
- more than 2,000 K–12 schools
- 100+ international peer networks in over 80 countries and other partners in Canada's innovation systemFootnote 19
CIOs who responded to the third‑party evaluation survey estimated that on average 56 percent of their institutions' total research and education traffic (as a percentage of monthly gross traffic) was carried by the CANARIE Network or by regional ORANs. The evaluation concluded that, the success of CANARIE working with its ORAN partners in reaching all potential users varies across provinces. Larger regions and provinces are typically very well served, while smaller regions and provinces tend to lag behind their larger counterparts.
Enhanced Canada's involvement in networking and networking‑focused collaborations
The third‑party evaluation survey responses of researchers and technology developers showed the importance of CANARIE's role in facilitating their capabilities for technology innovation through collaboration. Eighty percent of respondents reported that CANARIE was "very" or "critically" important for collaboration with researchers outside Canada; 75 percent reported that it was "very" or "critically" important for collaboration with other researchers in Canada; and 50 percent indicated that it was "very" or "critically" important for collaboration with other researchers in Networks of Centres of Excellence.
These responses about collaboration were further confirmed in that researchers also reported in the survey that since 2010 they individually undertook via CANARIE, an average of 5 joint innovation projects involving remote collaboration with Canadian partners within their respective provinces; an average of 5 joint innovation projects with Canadian researchers outside their provinces; and an average of 6 joint innovation projects with researchers outside Canada. Furthermore, since 2010, researchers individually published an average of 9 scientific articles co‑authored with their collaborators outside Canada. Researchers also indicated that their use of CANARIE funds enabled them to author, on average, the following:
- 4 scientific articles in peer‑reviewed academic journals;
- 7 technical reports; and,
- 7 invited presentations given at conferences, workshops and meetings.
The third‑party evaluation concluded that, CANARIE effectively promotes collaboration between academia and industry, nationally and internationally.
Canada's position as a recognized leader in the development and use of advanced research networks
The third‑party evaluation provided several examples of the leadership CANARIE had demonstrated in setting world records for data transmission speeds at different points in time over the evaluation period. It indicated that, during the period of evaluation, CANARIE had joined and become a member of a blue ribbon international initiative comprising the CEOs of the top twenty NRENs in the world. It reported that its interviews with stakeholders confirmed that CANARIE was an active, leading member of this initiative, and had garnered respect and recognition as a leading‑edge contributor by its peers.
Overall, the third‑party evaluation concluded that, CANARIE was well‑recognized by its peers in other countries as being one of the leading advanced networks in the world.
The degree of success in addressing the objectives related to technology innovation activities was assessed in relation to the following evaluation indicators:
- Degree to which CANARIE programs and activities led to the development and increased availability of software interfaces and tools that facilitate easier, flexible use of distributed research equipment and resources; and,
- Degree to which CANARIE programs and activities led to increased collaborative partnerships for the development of research platforms.
CANARIE programs and activities led to the development and increased availability of software interfaces and tools that facilitate easier, flexible use of distributed research equipment and resources
CANARIE staff reported that there was a gap in middleware development which it has attempted to fill. The third‑party evaluation reported that during the current mandate, 48 reusable software components and 12 network enabled platforms have been funded by CANARIE. To better support and guide this software initiative, CANARIE increased its internal expertise in the software development area by staffing a small team to build the infrastructure to manage the services being developed and offered by the community, and to work with the community to oversee the architectural evolution of the research software program.
Researcher respondents to the third‑party evaluation survey were unanimous about the importance of CANARIE funding for the development of tools and platforms for conducting innovative research. All respondents indicated they considered CANARIE funding "very" or "critically" important for developing application tools and software interfaces, and generally for conducting innovative research and technology development.
Overall, the third‑party evaluation concludes that CANARIE has facilitated technology innovation by researchers by helping them to use the Network to seamlessly access distributed data, tools and research instruments across the world. However, while CANARIE stakeholders were pleased with the role played by CANARIE in middleware development, the Leadership Council, a voluntary collective of leaders who represent key stakeholders that have a part to play in the creation of a world‑leading, advanced digital infrastructure ecosystem for Canada suggest that there is still a long way to go. Specifically, it suggests that:
- There has been inadequate attention to the development of the software and middleware needed to make infrastructure work according to users' expectations and usable across multiple public and private resources and platforms.
- Even with the new CANARIE programs, a better system‑wide approach for the development, adoption, and support of middleware is required, including a mechanism to evolve integrated development platforms.Footnote 20
It should be noted that this criticism suggests that the issue is larger than the scope of activities undertaken by CANARIE.
CANARIE programs and activities led to increased collaborative partnerships for the development of research platforms
Researchers surveyed in the third‑party evaluation were asked if they considered CANARIE funding support to develop the software, tools, and research platforms that empower them to access globally distributed data, tools and instruments, to be important for collaboration. The results showed that researchers considered the research software program and other funding support as a relevant contribution by CANARIE. The response by researchers was almost unanimous in considering CANARIE funding an important factor for successful collaboration and innovation with other researchers in Canada, with international researchers, and with counterparts in private industry.
The Monitor Deloitte consultations found that from the perspective of international peers, CANARIE was proactive at establishing international connections to support global collaborations.
Private Sector Innovation
The degree of success in addressing the objectives related to private sector innovation activities was assessed in relation to the following evaluation indicators:
- Evidence CANARIE initiatives helped to increase information and communications technology (ICT) by small and medium‑sized enterprises; and,
- Extent to which CANARIE has been effective in helping Canadian ICT researchers and SMEs to validate and commercialize their products faster.
Increase ICT by small and medium‑sized enterprises
As mentioned earlier, CANARIE offers the DAIR program to leverage CANARIE resources by stimulating private‑sector innovation. The third‑party evaluation reported DAIR helped users to understand how cloud computing can make a difference for them, reducing barriers to entry in the competitive market, and providing a test‑bed for confirmation of the workability of their products and service offerings before they go to market.
DAIR has served over 250 users during CANARIE's current mandate. The third‑party evaluation noted that through DAIR, CANARIE has a growing list of relationships with local Canadian technology incubators and accelerators, including the City of Fredericton's goFrednetwork, Startup Calgary, TEC Edmonton, Accelerator YYC, and the ICT Association of Manitoba (ICTAM). Further, it noted that DAIR has also being deployed in support of colleges' (Algonquin College, New Brunswick Community College, Centennial College, George Brown College, Humber College, and Red River College) applied research initiatives. Finally, CANARIE recently signed a Memorandum of Understanding (MOU) to partner with the Association of University Research Parks (AURP) Canada, a consortium of 26 Canadian research parks which connects to nearly 1,400 knowledge‑based businesses.
Help Canadian ICT researchers and SMEs to validate and commercialize their products faster
The third‑party evaluation reported the following findings in relation to this indicator:
- Sixty‑six percent of CIOs surveyed considered CANARIE of some importance for the creation of intellectual property and 83 percent considered CANARIE of some importance for innovations in products and services
- Seventy‑nine percent of DAIR users said that DAIR was "very" or "critically" important to them for testing their new products or services; 68 percent for effectively using cloud‑based computing services; and 72 percent for testing or evaluating a new model for rolling out their businesses.
- Eighty‑eight percent of DAIR users reported that the program was "very" or "critically" important to them for reducing their development costs; 67 percent for reducing the risk associated with conducting research; 66 percent for helping them to create proof of concept; and 62 percent for accelerating innovation.
- Seventy percent of DAIR users reported that the program was "very" or "critically" important in getting their products and services to market faster. Users estimated that, on average, DAIR helped them get to market 20 weeks sooner. Further, 60 percent reported that DAIR helped them to evaluate the market potential of their products or services were "very" or "critically" important.
The evaluation also provided the example of Galdos Systems Inc. It reported that the DAIR program enabled Galdos to test their product to the extent that they were able to enter into an agreement with Expedia, the world's largest travel company, an agreement to use Galdos INdicio, a product which enables a whole range of location‑enabled applications. The evaluation indicated that the impact of DAIR in this case was that it allowed Galdos to demonstrate the commercial viability and performance of its product and without DAIR this would not have been possible because Galdos did not have the resources at the early stages of development.
3.2.2 How efficiently and economically is Industry Canada's contribution to CANARIE being delivered? Is there a more cost effective way of achieving expected results?
This section examines efficiency and economy in terms of CANARIE's operations and then in terms of Industry Canada's monitoring. In examining CANARIE's efficiency and economy, there are several aspects to consider:
- Appropriate and balanced spending to achieve the objectives set out by Industry Canada;
- Ratio of overhead costs to total budget;
- Cost recovery efforts;
- Measures employed to garner operational efficiency; and
Appropriate and balanced spending
The third‑party evaluation examined the appropriateness and balance of CANARIE spending through stakeholder interviews and surveys. It concluded that CANARIE has been living within its means, is well focused on its mandate and related initiatives and that its funds are distributed proportionately and sufficiently commensurate with expected outcomes. It further concluded that CANARIE is delivering its programs efficiently. Industry Canada staff are satisfied that the funds are spent in accordance with the objectives and requirements of the contribution agreement.
Ratio of overhead costs to total budget
The ratio of overhead costs to total budget is a common way of assessing the efficiency of government sponsored organizations. For CANARIE, the actual and projected overhead and operating expenses for 2010–2011 through 2015–2016 are 15.5% which is below the 17.4% allowed under the most recent Industry Canada contribution agreement.
Industry Canada's 2012–2015 funding agreement with CANARIE required it to explore and implement a cost‑recovery strategy. Responding to this requirement, CANARIE, after consultations with its stakeholders developed a business plan outlining a range of cost‑recovery initiatives, including cost‑sharing with provincial and territorial partners (ORANs) and the collection of user fees.
In its business plan, CANARIE set an $11.7 million cost‑recovery target for the period of to . Cost‑recovery measures include revenues from membership and user fees, matching funds, in‑kind contributions, and cost avoidance measures. For the duration of its current mandate, from to , CANARIE is forecast to achieve cost‑recovery of $11.4 million; $0.3 million shy of its $11.7 target.
Over the evaluation period, CANARIE has made efforts to maximize its efficiency and economy. The third‑party evaluation concludes that CANARIE has demonstrated best practices in managing and operating the Network. These include its co‑delivery model with the ORANs, regularly analyzing key performance metrics, delivering a best‑effort service, scalable and flexible network operations (garnering low legacy costs when upgrades are needed), and operating a hybrid system to serve the needs of large and small users.
CANARIE staff explained how they keep up with demand by making sure the network has sufficient capacity and looking several years ahead. On the other hand, they monitor the network regularly to avoid excess capacity. One example was that they delayed moving to a 100 GB network so that CANARIE was not paying for the capacity before it was needed.
CANARIE's Network is a hybrid system, which offers greater flexibility in service offerings in order to meet the changing needs of users, scaling up from users with smaller data exchange needs to very‑intensive data applications, such as high energy physics applications. This scalability ensures costs are minimized.
Finally, CANARIE has created efficiency by developing reusable software. CANARIE realized that many of the research platforms share common tasks that each implemented independently (e.g. methods to access advanced computing resources). As well, the platforms have developed unique internal services that other platforms would find useful instead of developing their own, for example special data visualization tools and workflows. CANARIE convinced the existing platforms to modify appropriate services, and new platforms to design their new services such that these software services could be accessed externally, separate from their platform. This has resulted in the availability and re‑use of very specialized research software services across multiple platforms, reducing the time and costs of creating similar software and services.
CANARIE is one of 12 delivery partners in the system that delivers advanced high‑speed networking across Canada. CANARIE operates as the national backbone and provides the international connections. The ORANs provide the regional networks which connect with local users. Further, the Industry Canada investment of $62 million for the current mandate is leveraged by CANARIE to include ORAN funding from the provinces, membership and service fees. The third‑party evaluations concluded that, with its ORAN partners, "CANARIE has leveraged the Network and services efficiently, assisting R&E institutions and the private sector to innovate and commercialize products and services, while realizing cost‑savings for users and participants in its programs."
The introduction of DAIR is also an example of leveraging the network. Because the private sector usage is very small (less than 5%) compared to that of researchers, it has had minimal impact on network capacity while leveraging it to support provide sector innovation.
Industry Canada Monitoring
Industry Canada monitoring comprises the development and implementation of the funding agreements, sitting as ex‑officio members on various CANARIE committees, and reviewing CANARIE reporting documents. Spectrum, Information Technologies and Telecommunications Sector (SITT) estimates that less than one FTE is dedicated to monitoring, which would seem to be relatively efficient in relation to the most recent $62 million in funding provided to CANARIE.
In interviews conducted for this evaluation, both Industry Canada and CANARIE staff were satisfied with the role that Industry Canada provides through its ex‑officio roles on committees and the monitoring provided by SITT. However, most CANARIE staff and board members interviewed believed that reporting requirements could be streamlined to eliminate redundancies and focus on key elements. Those interviewed as part of this evaluation and the third‑party evaluation also suggested that a three‑year funding agreement causes considerable inefficiency in that there are set up and wind down costs continually incurred. Further, front end development time and mandate end renewal preparations can reduce actual program delivery time to as low as18 months out of the 3 years.Footnote 21 It was also suggested that a longer funding cycle would allow CANARIE to be more efficient in making funding decisions in relation to longer infrastructure life cycle considerations. CANARIE staff also suggested that advance notice that their mandate would be renewed could help reduce or eliminate some wind down costs and encourage further investments from other network partners.
The third‑party evaluation also suggested that the Industry Canada investment in CANARIE can be considered to be highly efficient in that the relatively small investment provides the necessary infrastructure to support the total government R&D investments. Specifically, it notes that in 2013, the federal government targeted $6 billion for R&D funding and $5.3 billion for higher education. Compared to this combined annual budget of $11.3 billion, CANARIE's annual funding as a public investment is a very small fraction (0.18 percent of total R&D and education spending), ensuring that the larger R&E community, and private sector participants in DAIR, are well served with an essential, advanced backbone communications, data transmission, cloud, and content provider network.
Evidence of socio‑economic benefits from CANARIE is limited. However, a cost benefit analysis conducted by Nordicity and Bytown Consulting in 2011 estimated that between 1993 and 2010, each dollar invested in the operations of the CANARIE R&E network generated economic benefits of $2.61 in the form of GDP throughout the Canadian economy.Footnote 22
4.0 Conclusions and Recommendations
Regarding relevance of the program, the evaluation determined that:
- CANARIE helps to address Canada's innovation gap and more specifically its large scale research collaboration gap.
- CANARIE's continued need is reflected the growing demand for its services and the potential negative impacts of its discontinuation on research and education.
- The funding of CANARIE is aligned with federal government priorities and with Industry Canada's Strategic Outcome relating to advancements in science and technology, knowledge and innovation that strengthen the Canadian economy.
- Federal support of CANARIE is consistent with the roles and responsibilities of Industry Canada and is similar to support provided in other countries around the world. Private sector and provincially sponsored options are not feasible to provide the same level of effectiveness.
Regarding the effectiveness of the program, the evaluation determined that:
- CANARIE has generally been successful in achieving its objectives and related short term outcomes.
- CANARIE's success in achieving longer‑term outcomes can be better assessed as new program components have time to realize longer‑term effects and as the performance measurement system begins to capture the longer‑term outcomes.
- CANARIE has demonstrated continuing efforts to achieve efficiencies.
- Industry Canada's relatively small investment in CANARIE is efficient in that it provides the necessary infrastructure to support the significantly larger total government investments in R&D.
The evaluation findings led to the following recommendation:
- SITT should continue to monitor the implementation of CANARIE's performance measurement system to ensure that long‑term outcomes are appropriately tracked to support public reporting, departmental decision‑making and future evaluations.
Appendix A – Executive Summary, Evaluation of CANARIE, Final Report
Nordicity and Bytown Consulting
Introduction and Background
In order for Canada to continue to produce world‑class research, to innovate ground‑breaking technologies, and to remain connected to some of the planet's leading edge scientific developments, the nation requires access to the necessary digital infrastructure. To that end, Nordicity and Bytown Consulting have been engaged to evaluate the extent to which CANARIE, Inc. (CANARIE) is helping Canada attain those goals.
Based on CANARIE's records and primary research data obtained between March and May 2014 (via a series of interviews and an online survey), the project team evaluated CANARIE in light of five key issue areas typical to reviews of federally funded programs and organizations:
- Achievement of Expected Outcomes: the extent to which CANARIE is meeting the goals outlined in its contribution agreement (e.g., regarding network operations, technology innovation and private sector innovation);
- Relevance and Continued Need: the extent to which CANARIE remains helpful for current users and will continue to be a critical resource as users' needs evolve in the future;
- Alignment with Government Priorities: the extent to which CANARIE remains aligned with specific directions outlined by Canada's federal government (e.g., Digital Canada 150, the Federal Budget 2014, and the 2007 Science and Technology [S&T] Strategy);
- Alignment with Federal Roles and Responsibilities: the degree to which CANARIE is a necessary and critical function for the federal government to provide;
- Demonstration of Efficiency and Economy: the extent to which CANARIE has been efficiently operating and managing the CANARIE Network, contributing to technology innovation and leveraging the Network to assist firms and Canadian universities to advance innovation and commercialization of products and services.
Summary of Findings
Based on the five above key issues areas, the findings of the project team can be summarized as follows:
Achievement of Expected Outcomes
CANARIE has met – or is in the process of meeting – its stated objectives:
- During the evaluation period ( to present), CANARIE has made significant progress against its objectives for evolving and extending the network and its services, leading the development of research software tools, and stimulating ICT innovation and commercialization.
- CANARIE continued to build on previous mandate successes by further developing high‑speed backbone connectivity across the country, reaching out with its Optical Regional Advanced Network (ORAN) partners to some 1,965 connected institutions.
- The CANARIE Network serves all the provinces and territories of Canada, and during the current mandate, CANARIE's completion of a dedicated ultra‑high‑speed (100G, or 100 billion bits per second) network from coast to coast, from Victoria to Halifax, is on schedule.
- CANARIE traffic has increased by 385 percent from 2009–10 to 2013–14, and users have indicated that CANARIE has provided reliable and sufficient capacity to meet their evolving needs, especially but not exclusively in applications involving very large datasets.
- CANARIE continues to be an essential component of Canada's digital infrastructure and is deemed by those consulted in academia and industry to be fundamental to research and education, and to achieving a successful innovation eco‑system by facilitating collaboration and data transfers between researchers and educators, in Canada and across the globe, and by making digital content readily accessible.
- CANARIE is well‑recognized by its peers in other countries as being one of the leading advanced networks in the world. CANARIE works closely with its international National Research and Education Network (NREN) peers in advancing the state of the art in high‑speed networking across the globe.
Relevance and Continued Need
CANARIE is highly relevant to users, and will continue to remain so in the coming years, including in the following ways:
- Data‑intensive research activities are highly dependent on the CANARIE Network to transmit large volumes of data globally in a reliable, efficient and secure manner across a broad spectrum of academic disciplines including the natural, health, and social sciences; engineering; and the humanities.
- Services such as those of the Canadian Access Federation (CAF) and the Content Delivery Service (CDS) allow educators to provide access to a wide range of resources and content with minimal administrative and financial burden.
- The need for CANARIE programs and services will grow over the next three years, as researchers, educators and entrepreneurs engage in activities that require the collection and transmission of increasingly large volumes of data.
- The discontinuation of CANARIE programs and services would have a highly negative impact on users, as well as Canada's ability to maintain leadership in research and development.
- There will be an increased need for CANARIE programs and services to facilitate research and education both with international partners and within Canada.
- Digital Accelerator for Innovation and Research (DAIR) program users predict an increase in the use of that program, with particular interest paid to the use of cloud computing tools.
Alignment with Government Priorities
CANARIE and its programs and services both directly and indirectly support the objectives of several relevant policy documents. For example:
- CANARIE supports the objectives of Digital Canada 150 (DC 150), particularly the creation of "economic opportunities" and "connecting Canadians" – two key pillars of the plan – in the following ways:
- By providing cloud infrastructure to Canadian entrepreneurs and emerging businesses, CANARIE's DAIR program directly aligns with the creation of "economic opportunities," and
- By connecting research institutions (and researchers) across Canada CANARIE has contributed to "connecting Canadians," and also has helped to address the "Open Science" element of DC 150.
- CANARIE contributes to the goals of the 2014 Federal Budget, by promoting "research excellence" and acting in parallel to planned investments to be made through the Canada First Research Excellence Fund (CFREF), in particular through its support of post‑secondary institutions and facilitation of Canada's participation in world‑leading research projects (e.g., the ATLAS Experiment).
- CANARIE has led to the development of new knowledge, attracted (and retained) highly qualified persons, and led to the commercialization of new products and services – thereby directly contributing to three pillars of Canada's S&T strategy.
Alignment with Federal Roles and Responsibilities
CANARIE provides, and likely will continue to provide, a service that Canadian researchers and entrepreneurs need – one that cannot readily be provided by the private sector. To that end:
- Stakeholders and users of CANARIE mostly agree that CANARIE provides programs and network services that are appropriate for federal government support and aligned with its roles and responsibilities – ensuring that Canadian researchers and educators from coast to coast have an accessible and reliable leading edge, advanced high‑speed network available to them.
- Without CANARIE, research institutions would have to form some other organized effort to enable collaboration, communication and access to Canadian and international research – likely at greater expense than the cost of CANARIE.
- If institutions were forced to pursue private sector solutions, the costs incurred would likely render many research activities cost‑prohibitive.
- Helping SMEs grow via programs like DAIR is generally supported by the research community as a legitimate mission for CANARIE.
- Vice Presidents of Research at academic institutions, researchers, partner ORANs, and incubators/accelerators all have a clear understanding of the purpose and role of CANARIE, though that understanding seems somewhat less clear among Chief Information Officers (CIOs) of academic institutions.
Demonstration of Efficiency and Economy
As in its previous mandate, CANARIE is living within its means and delivering its programs efficiently:
- CANARIE is well‑focused on its mandate and related initiatives and funds are distributed proportionately and sufficiently commensurate with expected outcomes.
- For the duration of its current mandate, from to , CANARIE is forecast to have achieved cost‑recovery of $11.4 million, just $0.3 million shy of its $11.7 million cost‑recovery target.
- With its ORAN partners, CANARIE has leveraged its network and services efficiently, assisting Research and Education (R&E) institutions and the private sector to innovate and commercialize products and services, while realizing cost‑savings for users and participants in its programs.
- CANARIE has demonstrated best practices in managing and operating the network. These include its co‑delivery model with the ORANs, regularly analyzing key performance metrics, delivering a best‑effort service, scalable and flexible network operations (garnering low legacy costs when upgrades are needed), and operating a hybrid system to serve the needs of large and small users.
Note that CANARIE's 2012–2015 Performance, Audit and Evaluation Strategy (PAES) states the following regarding the logic model: "
The logic model starts off by identifying the activities that CANARIE will undertake under each of its Objectives. And by completing those activities, CANARIE expects to achieve the results that are listed in the following row, called Outputs. These Outputs are expected to be fully achieved as CANARIE activities are nearing completion. These Outputs will have a significant impact on Short‑Term Outcomes, presented in the following row of the Logic Model. The Short‑term Outcomes will be achieved in 2‑to‑3 year timeframe. As the Short‑Term Outcomes are coming to fruition, the results of the Outputs and Short‑Term Outcomes combined will lead to Long‑Term Outcomes in the 3–5 year timeframe. The full impact of Government of Canada's investments in CANARIE will be realized in the 5–10 year timeframe in the form of Ultimate Impacts, as highlighted in the last row of the Logic Model."
The third‑party evaluation employed five data collection methods: document and file reviews; an online survey; interviews with key stakeholders and users; case studies; and a comparative analysis of international peer organizations/networks. Additionally, CANARIE provided the research team with access to a summary of stakeholder consultation findings from the research surrounding the development of CANARIE's 5‑year strategic plan (as conducted by Monitor Deloitte).
The following table outlines the over 150 stakeholders whose input was integrated into the report based on the various methodologies used in the evaluation.
|Survey Respondents||Evaluation Interviews||Monitor Deloitte ConsultationsFootnote *||Total Represented|
|University and College Leaders (Total)||14||7||12||33|
Funded software developers
|DAIR users and partners||19||7||26|
|Researchers and developers||21||2||7||30|
|CANARIE Network Partner Representatives and International Partners||8||15||23|
|Industry and Private Sector Partners/Associations||2||13||15|
|Government departments and agencies (users, researchers, and others)||6||6||12|
|CANARIE Staff & Management||5||5|
|Others (research funders and innovation ecosystem partners)||7||7|
- Document and File review: Documents provided by CANARIE included previous evaluations and performance audits, annual reports, and customized reports on the CANARIE organization and its role in Canada's science and technology community. CANARIE also supplied copies of reports submitted by funding recipients. In addition, where relevant, government documents (strategies, budgets, policies) were reviewed to determine CANARIE's alignment with the broader objectives of the Canadian government. The research team was also provided with a summary of the findings of stakeholder consultations conducted as part of the development of CANARIE's 5‑year strategic plan (2015–2020), which accounted for a total of 60 various stakeholders. Documents relating to other National Research and Education Networks (NRENs) were also reviewed in order to ascertain CANARIE's position among its peers on the international research and education stage.
Online Survey: The research team worked with CANARIE to administer an online survey. The survey invitation was distributed by CANARIE to a total of 335 contacts, including chief information officers (CIOs) at Canadian universities, vice‑presidents of research (VPRs) at Canadian universities, researchers and developers, and small‑ and medium‑sized businesses using DAIR. It should be noted that the survey was only distributed to DAIR users that had one or more years of experience with the DAIR program (at the time of the survey). The online survey was launched on and closed on .
A total of 54 individuals responded to the online survey. The highest response rate was in the researcher category: 33% of invited researchers completed the survey (see Table 2 below). The DAIR user category also received a high level of responses, with 29% of invited users completing the survey. While only 11% of CIOs responded, those CIOs represent 196,254 full‑time and part‑time enrolled students (graduates and undergraduates), or 15.4 percent of all enrolled students in the Association of Universities and Colleges of Canada (AUCC) member institutions; and represent 8,177 faculty members, or 14 percent of all faculty in AUCC member institutions.Footnote 23 As such, while figures containing survey results from CIOs should be treated with some caution, they remain broadly indicative of the overall environment.
However, only one VPR from the 92 invitees responded to the survey. Due to the extremely low response rate in that category, the results were omitted from the analysis included in this report.
While some of the response rates listed in the above table may seem low, the third‑party evaluation report notes that it is important to remember that participation in this exercise was completely voluntary—and no incentive was offered to either CIOs or VPRs to participate. Furthermore, not all stakeholders engage directly with CANARIE, so some potential respondents may not have felt adequately informed to participate in the survey. At the same time, two other consultation exercises have recently been undertaken regarding CANARIE: a cost recovery consultation in late 2012‑early 2013 and the consultations regarding CANARIE's 5‑year strategic plan (consulted by Monitor Deloitte).
The survey questionnaire inquired about respondents' experiences with and perceptions of CANARIE programs and services, including:
- Current and anticipated future use of the CANARIE Network, and its programs and services;
- CANARIE's sufficiency in meeting current and anticipated future demand;
- Outcomes of CANARIE‑supported research, education and innovation activities;
- Projected impact if CANARIE ceased to exist;
- Cost‑savings resulting from use of the CANARIE Network, and participation in programs and services; and,
- Role of CANARIE in establishing and maintaining Canada's leadership position in research networking.
- Interviews: In total, 37 stakeholders were interviewed by phone and in person, including CANARIE staff and management, partner networks, current users, and representatives from Industry, research centres, government departments and international peer networks. Interviewees for this study were selected by CANARIE, with input from the research team. Some key stakeholders were not included as interviewees for this study as they had already been consulted regarding the development of CANARIE's 5‑year strategic plan. As mentioned above, some of the results of that consultation process have been incorporated into this document. The breakdown of interviewees according to stakeholder groups was presented in Table 1. The interviews were designed to corroborate and complement the findings of the online survey and document review. Each stakeholder group was asked a different set of questions, addressing evaluation issues most relevant to their relationship with CANARIE.
- Case Studies: Four case studies were completed as part of the evaluation, in an effort to take a closer look at the evaluation issues through specific examples. The following is a list of case studies completed: TRIUMF; Project Whitecard; Metafor Software; and the Canadian Brain Imaging Research Network (CBRAIN)/Global Brain Imaging Research Network (GBRAIN). The case studies were developed primarily through interviews, supplemented with a review of relevant websites and documentation (e.g., funding reports obtained from CANARIE).
International Peer Organizations/Networks: In order to provide some comparative perspectives on CANARIE operations, several international partners and peer networks were examined as a part of this evaluation. In addition to two interviews, which were conducted with representatives from Internet2 and the International Center for Advanced Internet Research (ICAIR), document reviews provided additional details on international research based networks, including those from Australia, the Netherlands, the UK, New Zealand, Scandinavia and the US.
Examination of NRENs included a review of publicly available information on their respective official websites, other online information about the initiatives and activities of these organizations, and a review of the TERENA Compendium of National Research and Education Networks, 2013 Edition (www.terena.org/compendium). The review of the TERENA Compendium in particular was an important source of information to help the evaluation team to identify CANARIE best practices. This Compendium has grown over the more than ten years since its inception and has become a recognized authoritative reference source for researchers and organizations that are interested in the development of research and education networking.
Evaluation of Industry Canada's Contribution to CANARIE
Management Response and Action Plan
|Recommendation||Management Response and Planned Action||Management accountability||Action Completion Date|
- Date modified: