Evaluation of Industry Canada's Contribution to the Ivey International Centre for Health Innovation

December 2013

Recommended for approval to the Deputy Minister by the Departmental Evaluation Committee on November 29, 2013

Approved by the Deputy Minister on December 12, 2013

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Program Context
  • 1.2 Program Overview
  • 1.3 Program Resources
  • 1.4 Program Design and Delivery
  • 1.5 Expected Results
• 2.0 Methodology
  • 2.1 Evaluation Approach
  • 2.2 Objective and Scope
  • 2.3 Evaluation Issues and Questions
  • 2.4 Data Collection Methods
  • 2.5 Limitations to the Data Collection Methods
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance
• 4.0 Conclusions and Recommendations
  • 4.1 Conclusions
  • 4.2 Recommendations

Acronyms, Abbreviations and Definitions used in this Report

List of Tables

List of Figures

Executive Summary

In 2009, Industry Canada committed $5 million in contributions to Western University to provide seed funding for the creation of the Ivey International Centre for Health Innovation (Ivey Centre). The Ivey Centre was created to identify, assess and commercialize innovative technologies, systems and processes that are expected to benefit Canada's health care providers, patients and the health care system. This is accomplished through the Ivey Centre's three activity areas, education and leadership development, thought leadership, and health innovation projects (HIPs). At the time of funding, the intention was for the Ivey Centre to be self-sustaining by year five through the generation of revenue streams.

In terms of education and leadership development, the Ivey Centre offers a variety of programs and courses in health innovation, management and leadership. These programs are provided to students of Western University's MBA and HBA programs as well as executives and managers within the health sector. Activities related to "thought leadership" include the annual Ivey Global Health Conference, symposia, applied research and white papers. Health Innovation Projects are experimental or applied research projects that bring students, faculty, subject matter experts, and health care organizations together to study an issue, problem or opportunity within the health sector. A sub-category of HIPs are demonstration projects. These projects are the primary mechanism through which the Ivey Centre supports the innovation, commercialization and adoption of health innovations in the health care sector.

Evaluation Purpose and Methodology

In accordance with the Treasury Board Policy on Evaluation and the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of Industry Canada's contribution to the Ivey Centre. The evaluation covers the period of 2009-10 to 2013-14 and findings are based on the analysis of multiple lines of evidence. The lines of evidence include a document review, a literature review, an environmental scan, interviews and data analysis.

Findings

Relevance

The innovation, commercialization, and adoption of health care technologies, systems and processes are essential for the development of a more efficient and effective health care system. Advancements in this area contribute to the economic prosperity and societal well-being of Canadians. The three primary activity areas of the Ivey Centre respond to the key challenges associated with the commercialization and adoption of health innovations. The mandate and objectives of the Ivey Centre align with federal priorities and are consistent with the responsibilities of Industry Canada.

Performance

The Ivey Centre was expected to result in personnel trained in the commercialization of healthcare technologies, systems and processes, as well as highly qualified personnel with the potential to be future health industry leaders. While the Ivey Centre offers a range of formal and informal training opportunities for students and health sector executives, a lack of data prevented a comprehensive assessment of the Ivey Centre's success in training personnel and developing future health industry leaders.

The Ivey Centre was also expected to promote and disseminate knowledge in health innovation and commercialization suited to Canada's health care system. The Ivey Centre has demonstrated capacity to assemble and promote knowledge through multiple platforms. The Ivey Global Health Conference, along with the development of white papers, are recognized as effectively contributing to the advancement and dissemination of knowledge on topics of health innovation relevant to the Canadian context.

The Ivey Centre was expected to identify and assess innovative health care technologies, systems and processes with a view towards commercialization. Through its demonstration projects, the Ivey Centre has made some progress in these areas. However, more time is needed to demonstrate commercialization for some projects, and others lack a specific commercialization focus.

Additional contributions will be needed for the Ivey Centre to meet Industry Canada's July 2014 leveraging targets. Furthermore, as innovations are yet to be commercialized, more time will be needed for the Ivey Centre to achieve sustainability. The use of a third-party delivery model is an efficient means to achieve targeted S&T policy goals. However, relatively high administrative requirements for IC's oversight and management of the funding agreement with the Ivey Centre were noted. Finally, the Ivey Centre's governance mechanisms are consistent with recognized practice and support the organization in administering funds and delivering programming in an efficient and effective manner.

Recommendations

In the event of funding renewal, the following recommendations should be considered:

1. Science Partnerships Directorate should require the Ivey Centre to develop a business strategy with clearly defined approaches to achieving self-sustainability.
2. Science Partnerships Directorate should assess the need to maintain the existing level of compliance controls and reporting requirements against the risk level and materiality of the funding agreement.
3. Science Partnerships Directorate should collaborate with the Ivey Centre to refine performance measures to better capture the effectiveness of the Ivey Centre's activities. In particular, the need for performance data to measure the achievement of outcomes related to the development of highly qualified personnel and the commercialization of technologies, systems, and processes.
4. Science Partnership Directorate should ensure that future activities supported by Industry Canada have a strong focus on commercialization.

1.0 Introduction

This report presents the results of an evaluation of Industry Canada's Contribution to the Ivey International Centre for Health Innovation (the Ivey Centre).

The purpose of the evaluation was to assess the relevance and performance of the Ivey Centre. The report is organized into four sections:

• Section 1 provides the program context and profile of the Ivey Centre;
• Section 2 presents the evaluation methodology along with a discussion of data limitations;
• Section 3 presents the findings pertaining to the evaluation issues of performance and relevance; and
• Section 4 summarizes the study's conclusions and provides recommendations.

1.1 Program Context

Arms-length organizations have been used as tools to deliver federal public policy since the 1990s, particularly in areas such as research and development and education. Through its Science and Technology Partnerships Program, Industry Canada manages federal contributions to a number of third-party organizations, including the Ivey Centre. These investments are guided by the policy commitments set out in the federal Science and Technology (S&T) Strategy, Mobilizing Science & Technology to Canada's Advantage.

The S&T Strategy outlines four priority areas, one of them being, "health and related life sciences and technologies".^{Footnote 1} Within the S&T policy framework, Budget 2009 committed
$5 million in contributions to Western University to provide the seed funding for the creation of the Ivey Centre.

The Ivey Centre was created to identify, assess and commercialize innovative technologies, systems and processes that are expected to benefit Canada's health care providers, patients and the health care system. The Ivey Centre also strives to produce highly skilled personnel with the combination of medical knowledge and business capabilities to help drive innovation in the Canadian health care system.

The Ivey Centre was developed in collaboration between Western University's faculties of business and medicine and the London Health Sciences Centre, a separate legal entity and one of the largest teaching hospitals in Canada. A part of its operating model is to attract other public and private sector partnerships, either through collaborations or investments. The intention of the Ivey Centre was to be self-sustaining by year five with revenue streams from technology transfer and licensing (from successful commercialization), the market intelligence library, and the Executive Education Program.

1.2 Description of the Program

The Ivey Centre operates under three primary activity areas: education and leadership development, thought leadership, and health innovation projects.

In terms of education and leadership development the Ivey Centre offers a variety of programs and courses in health innovation, management and leadership. These courses are offered as electives to Ivey's Master of Business Administration (MBA) and Honors Business Administration (HBA)^{Footnote 2} students in the health sector stream, as well as students from other faculties including medicine, health sciences, computer science, engineering and law. Other educational opportunities are provided to clinicians, scientists and senior executives through various formats including executive development programs, webinar series, and guest lectures.

Activities related to "thought leadership" include the annual Ivey Global Health Conference, symposia, applied research and white papers. The purpose of the Global Health Conference is to exchange ideas related to innovation and to prioritize health systems challenges in need of innovation. Each conference produces a white paper focusing on the emergent trends and business-related issues of health care innovation adoption. In addition, symposia are held on an ad-hoc basis and focus on specific topics. Faculty and/or graduate students also produce research papers focused on health innovation that are published and disseminated in academic and trade or industry journals. Finally, faculty and postdoctoral fellows associated with the Ivey Centre are commissioned to produce point of view papers aimed at encouraging discussion around health innovation and adoption.

Collaborative research projects conducted at the Ivey Centre are based on the Health Innovation Project (HIP) approach. HIPs bring students, faculty, subject matter experts, and health care organizations together to study a cross-enterprise issue, problem or opportunity of significance to a private industry or other health sector partner. Demonstration Projects are a unique category of HIP designed to support the adoption of health innovations such as current or emerging technologies and process improvements in the health care sector. The research team examines the impact and effectiveness of a new product or process in a real-world setting. Depending on the research results of the project, the private industry partner makes the decision to pursue broader implementation strategies, including commercialization. Intellectual property and commercialization opportunities related to specific innovations are determined on a case-by-case basis.

1.3 Program Resources

Industry Canada's $5 million contribution to the Ivey Centre was intended to support its start-up costs for the first five years of operation. The majority of funds were allocated to support the management and administration of the Ivey Centre ($3 million), including the salary and benefits of the Chair of Health Innovation, the Executive Director, visiting fellows and administrative staff. The remaining funds were to be used for the Demonstration Projects ($1.6 million), and outreach activities ($0.4 million), such as the Global Health Innovation Conference.

The funding profile is as follows:

1.4 Program Design and Delivery

The Science Partnerships Directorate (part of the Program Coordination Branch within the Science and Innovation Sector of Industry Canada) is responsible for the delivery of the federal contribution to the Ivey Centre, as well as the ongoing management and oversight of the funding.

The Ivey Centre is situated within the Ivey School of Business. The Dean of the School, along with the Chair of Health Innovation, set the direction for the Ivey Centre and ensure that its objectives are met. The Chair's responsibilities include: recruiting faculty (including visiting professors) and executives-in-residence; designing and implementing the program for students enrolled in the Health Sector MBA; recruiting companies for involvement in the Ivey Centre's projects; and implementing the annual Global Health Innovation Conference. The Chair is assisted by a small staff including an Executive Director who reports to the Chair and undertakes administrative duties for the Ivey Centre.

The work of the Ivey Centre is supported by a number of councils and committees whose members provide strategic advice in a range of areas including governance and innovation. These bodies include the:

• Advisory Council: The Ivey Centre is informed by an Advisory Council composed of representatives from the private, public, and not-for-profit sectors. This Council advises on the operations of the Ivey Centre and provides access to other key members of the health sector. Activities are coordinated across partners through formal meetings of the Advisory Council as well as specific sub-committees.
• Innovation Council: The Innovation Council consists of members of the Advisory Council and other additional experts. The Ivey Centre and its students identify potential projects and bring proposals to the Council for advice on their viability. The Council reviews and recommends proposals, offers guidance and expertise to Ivey Centre staff and students for the successful completion of demonstration projects and other health innovation projects, and connects students and staff with additional resources that may assist in the successful completion of promising projects. 

The key stakeholders for the Ivey Centre are:

• Partners and Collaborators: One of the Ivey Centre's activities is to build collaborations and networks in the health sector for the purpose of generating innovative ideas, supporting innovation adoption, facilitating collaboration, developing and testing innovations and disseminating information. Partners include organizations from the private, public, and not-for-profit sectors.
• Ministries of Health: The administration and delivery of health care services is the responsibility of each province or territory. The provinces and territories fund these services with assistance from the federal government in the form of fiscal transfers.
• Canadian Life Science Industries: The life sciences sector spans research, development and manufacturing. Industry players include small and medium-sized companies developing diagnostics, biopharmaceuticals, pharmaceuticals and medical devices, as well as global companies with operations in Canada, serving both domestic and international markets.^{Footnote 3} These industries are an important driver of Canada's innovation economy, and potential benefactors of the Ivey Centre's mandate.

1.5 Logic Model

The Ivey Centre's overall objective is to become an internationally recognized centre dedicated to identifying, assessing and commercializing innovative health care technologies, systems and processes that will bring about positive change for health care providers, patients and the health care system.

The expected results of Industry Canada's contributions are to:

• Increase knowledge in Canada in the innovation and commercialization of health care technologies and processes suited to Canada's health care system;
• Develop specialized talent with the requisite scientific and commercial knowledge needed to commercialize health care technologies, systems, and processes;
• Create a commercial model focused on driving innovation and commercializing ideas to achieve a return on investment and ensure that the most promising ideas are pursued and brought to the health care system as quickly as possible, and lead to the development of businesses, products lines and jobs; and
• Position Canada to take full and privileged advantage of the economic and social benefits of research in the field of health and related life sciences and technologies, including efficiency, cost savings and health care system improvements.

A logic model for the program (represented in Figure 1) was first developed in 2009 as a part of the Ivey Centre's Performance Measurement Strategy and is the basis for the evaluation. The logic model outlines the program's inputs, activities, and outputs, as well as the intended short-term, intermediate, and longer-term outcomes.

Figure 1: Logic Model for the International Centre for Health Innovation

Download high resolution version (PDF, 66 KB, 1 page)

2.0 Methodology

This section provides information on the evaluation approach, objective and scope, the specific evaluation issues and questions that were addressed, the data collection methods, and data limitations for the evaluation.

2.1 Evaluation Approach

Like many evaluations of government programs, this evaluation was based on expected outcomes of the program as stated in the program's foundational documents and logic model.

2.2 Objective and Scope

The objectives of this evaluation was to address the core issues of relevance and performance in accordance with the Directive on the Evaluation Function and to fulfill the Ivey Centre's requirement for an evaluation in 2013-2014 as per the Financial Administration Act.

The Ivey Centre is a relatively new organization that has only been in operation since 2009. As such, the evaluation focused on the immediate and intermediate outcomes.

The evaluation covers April 1, 2009 to August 31, 2013.

2.3 Evaluation Issues and Questions

Based on the program Performance Measurement Strategy, and subsequent consultations with the program, the evaluation addressed the following questions:

Relevance

• Is there a continued need for innovation and commercialization of health care technologies, systems and processes? Does the Ivey Centre respond to this need?
• To what extent do the objectives of the Ivey Centre align with the priorities of the federal government and the strategic outcomes of Industry Canada?
• To what extent does support for the Ivey Centre align with the roles and responsibilities of the federal government?

Performance

• To what extent have personnel been trained in the commercialization of health care technologies, systems and processes? To what extent has the Ivey Centre developed personnel who have the potential to be future health industry leaders?
• To what extent has the Ivey Centre assembled and promoted knowledge in health innovation and commercialization suited to Canada's health care system? To what extent has the Ivey Centre improved and disseminated this knowledge?
• To what extent has the Ivey Centre identified and assessed innovative health care technologies, systems, and processes? To what extent has the Ivey Centre commercialized or catalyzed the commercialization of health innovations that have led to new businesses, processes, and product lines?
• To what extent does the current program model demonstrate efficiency and economy?

2.4 Data Collection Methods

Multiple lines of evidence were used to address all evaluation questions. The data collection methods included a document review, a literature review, an environmental scan, interviews and data analysis.

Document Review

The document review was conducted to gain an understanding of the Ivey Centre, its alignment with government priorities and its achievement of expected outcomes. Key documents included:

• Program foundational documents (e.g. Treasury Board Submission, Funding Agreement and Performance Measurement Strategy)
• Program reporting documents (e.g. Annual Reports, Quarterly Reports, Corporate Plans)
• Other key program documents (e.g. Research Papers and Publications, Global Health Innovations Conference Reports, other Ad-Hoc Reports on Progress Achieved); and
• Government priority-setting documents (e.g. Budgets, Speeches from the Throne)

Literature Review

The literature review was undertaken to primarily address the core evaluation issues of continued need, and federal roles and responsibilities. Specifically, the literature review examined the continued need to support health innovation and commercialization, and the role of federal funding in supporting R&D in Canada and other jurisdictions.

Environmental Scan

An environmental scan was conducted to address the core evaluation issue of roles and responsibilities. Specifically, the environmental scan examined the existence of other organizations and government programs (both federal and provincial) with similar activities and assessed the degree of duplication and/or complementary activity.

Interviews

The objective of the interviews was to gather in-depth information, including views, perceptions and factual information that address the evaluation questions. The interviews were designed to obtain qualitative feedback from a range of respondents. The majority of interviews were conducted by telephone due to the disperse locations of the interviewees.

In all, the evaluation team engaged 30 interviewees, either individually or in a group interview setting.^{Footnote 4} The interviews included:

• Industry Canada (5)
• Academic Chair of Health Innovation (1)
• Executive Director (1)
• Dean of the Ivey School of Business (1)
• Executives-in-Residence, Visiting Fellows and/or Visiting Fellows (2)
• Industry and non-profit partners (6)
• Health Sector Advisory Board and the Innovation Council (5)
• Students (4) and recent graduates (3)
• External stakeholders (2)

Data Analysis

An analysis of leveraging and sustainability data was conducted to address the evaluation issues of efficiency and economy. The analysis examined the Ivey Centre's ability to leverage federal contributions over time and the Ivey Centre's ability to achieve sustainability through revenue generation.

Two qualitative methods were used to compliment the data analysis. An analysis of the efficiency of using third party organizations as delivery mechanisms to achieve public policy objectives, and a review of administrative requirements for both the Ivey Centre and Industry Canada (Science and Innovation Sector) to manage and administer the contribution agreement were considered.

2.5 Limitations to the Data Collection Methods

The following were the limitations to the methodology:

• Availability of Commercially Sensitive Information: For many of the demonstration projects, the specific details of the projects are confidential in order to protect commercial development. This limited the evaluation team's ability to conduct a thorough assessment of the demonstration projects that were undertaken, such as a review of project files or access to interview participants. The evaluation team mitigated this risk by focusing on the demonstration project approach and how it has contributed to the success of developing an innovation, and not on the specific technical or commercial details of the project. In order to reduce selection bias, the evaluation team worked closely with the Ivey Centre to select and recruit willing participants based on predetermined criteria.
• Access to Students and Graduates: At the time of the evaluation, due to privacy restrictions, it was not possible to access a representative sample of current students or graduates of the Ivey Centre. Further, there was no available performance data on the effectiveness of the Ivey Centre's training activities. This limited the assessment of the extent to which personnel have been trained, as well as the extent to which the Ivey Centre has developed personnel who have the potential to be future health industry leaders. As such, the assessment was largely limited to a descriptive analysis of activities conducted, an accounting of the aggregate number of personnel engaged in the Ivey Centre's activities, as well as a qualitative analysis of interview data from a limited number of current students (4) and recent graduates (3).

3.0 Findings

3.1 Relevance

3.1.1 Is there a continued need for innovation and commercialization of health care technologies, systems and processes? Does the Ivey Centre respond to this need?

Need for innovation and commercialization of health care technologies, systems and processes

The ability to innovate, or find "new and better ways of doing valued things", is a key factor in creating economic prosperity and societal well-being. Innovation drives productivity gains, stimulates wealth creation and improves the standard of living of Canadians. This is particularly evident in the health sector. The life sciences industry sector is an important contributor to Canada's innovation economy, spanning the research, development and manufacturing continuum. The sector fosters economic diversification, employs highly educated professionals, and develops intellectual property. Further, academic and public policy literature link the sustainability of Canada's health care system, among other things, to its ability to develop and deploy innovations that have the potential to lower costs and improve the performance of health systems. In the context of rising health care expenditures, the adoption of health innovations can play a key role in developing a more efficient and effective health care system.

In the life sciences sector, industry players include small and medium-sized companies as well as global companies with research, development and manufacturing operations in Canada. As of 2013, the sector has been slowly recovering from the effects of the global economic recession, which has negatively affected investments in R&D. Nevertheless, the sector has had historically low levels of R&D expenditures and the productivity gap within the sector has been widening over the last decade when compared to the United States. For instance, one measure of productivity is the pharmaceutical R&D to sales ratio.^{Footnote 5} In recent years there has been a significant decline in the ratio of pharmaceutical R&D to sales in Canada compared the United States.^{Footnote 6}

Findings from the literature review and interviews also point to various barriers to the commercialization and adoption of health care innovations across both the private and public sector. Businesses face obstacles navigating federal regulatory policies and provincial procurement systems, securing venture capital, developing strategic partnerships, and accessing the specialized expertise needed to evaluate and assess innovations. On the receptor side, there is a lack of incentives, mechanisms and resources for local hospitals and health care organizations to adopt innovations.^{Footnote 7}, ^{Footnote 8} As a result, interviewees suggest that Canadian companies are often forced to look abroad (typically the United States or the European Union) for more conducive markets within which to commercialize and deploy their new products. Collaboration amongst the different partners in the healthcare system (including academic institutions, the private sector, health care organizations and policy makers) is seen as an increasingly important means to move towards system level changes.

Interviewees across all stakeholder groups not only confirmed the growing need and urgency for innovation adoption across the health care system, but also perceived an accompanying increase in awareness of the need to find solutions and the potential economic and societal benefits that could result from innovation adoption. Literature suggests that such societal benefits include both improved health outcomes for Canadians and better service delivery within the health care system.^{Footnote 9} Other societal needs addressed by health innovation include the ability to address future health challenges. For instance, the country's aging population, the growing impact of diseases such as diabetes and dementia, and the emergence of new infectious diseases have been cited.^{Footnote 10} These challenges are coupled with the rising costs of Canada's health care system and introducing health innovations can potentially contain rising expenditures.^{Footnote 11} This is not to imply that the Ivey Centre in-and-of itself will address these challenges; rather, it is one of the handful of players within the health innovation system working towards this collective goal.

Responsiveness of the Ivey International Centre for Health Innovation

At a broad level there is alignment between the mandate of the Ivey Centre, and the need for health innovation:

"The Centre's goal is to be an internationally recognized centre dedicated to identifying, assessing, commercializing and driving adoption of innovative health technologies, systems and processes that will bring about positive change for health providers, patients and the health care system."

Further, the objectives and activities of the Ivey Centre largely respond to key challenges associated with the commercialization and adoption of innovations. These include developing the capacity for cross-sectoral (industry-academia-government) collaboration and knowledge transfer, targeting academic research to business and public sector needs, along with the provision of specialized training and learning opportunities.

The importance of the higher education sector in the innovation system, generally and in Canada in particular, has been highlighted in policy and academic literature. Given Canada's research strengths and existing capacity in the higher education sector alongside the country's relatively poor performance in business innovation, several scholars, policy-makers and advisory boards have identified the need for increased and diverse types of support for academic-industry collaboration, knowledge transfer and other forms of direct support.^{Footnote 12},^{Footnote 13},^{Footnote 14},^{Footnote 15} Within the health sector, the need to increase the capacity for knowledge transfer, exchange and collaboration around the adoption of innovations is widely acknowledged.

The Ivey Centre aspires to be a hub for health innovation by building a network of key players, across academia, business and government, in order to accelerate the commercialization and adoption of health innovations. Interviewees, particularly private sector partners, perceived the Ivey Centre to be well positioned to form effective networks and negotiate new cross-sectoral partnerships. Specifically, they found the Ivey Centre was demonstrating the capacity to bridge the gap between academia, government and industry by: offering R&D services targeted towards innovation adoption through demonstration projects and contract research; producing relevant policy research responsive to the needs of both industry and policy sectors; and facilitating knowledge outreach events that bring together diverse stakeholders.

Linked to the Ivey Centre's efforts in collaborative R&D are its activities related to developing and promoting the mobility of highly qualified personnel. Industry-academia R&D collaboration is often seen as a vital form of knowledge transfer "on two feet" through the movement and interplay of people.^{Footnote 16} In general, academic and policy literature has identified the need to better integrate the science, technology and innovation sector with business knowledge; to enhance work-integrated learning; and to focus on the education of future corporate leaders.^{Footnote 17} Literature specific to the health sector further highlights the shortage of skilled workers in this area,^{Footnote 18} the need to invest in the development of the types of skills needed to manage a more innovative process,^{Footnote 19} along with the lack of management talent that has experience bringing companies through the commercialization process.^{Footnote 20}

The Ivey Centre provides a variety of specialized training and learning opportunities to students from a wide range of disciplines and backgrounds. Activities include the development of specialized curriculum on health innovation, opportunities for students to perform work-integrated learning as part of consulting and demonstration projects with industry and government partners, and access to mentoring support from health-innovators-in-residence, faculty, and partners. Leadership development, focused on developing the skills needed to manage commercialization and innovation processes within both the private and public sector, was most often cited by interviewees as the area through which the Ivey Centre was best positioned to address health innovation challenges.

3.1.2 To what extent do the objectives of the Ivey Centre align with the priorities of the federal government and the strategic outcomes of Industry Canada?

The Government of Canada set its current agenda for supporting science and technology in 2007 with the introduction of the federal S&T Strategy. Under this Strategy, the Government outlined its intention to foster three distinct Canadian S&T advantages: a Knowledge Advantage, a People Advantage, and an Entrepreneurial Advantage. This focus is consistent with departmental priorities which indicate the need to "further develop the federal Science and Technology strategy and priorities to strengthen support for business innovation and continue to build Canada's advantages in knowledge, people and entrepreneurship".^{Footnote 21}

The Ivey Centre focuses its mandate and strategic objectives around three integrated activity areas: thought leadership; education and leadership development; and health innovation projects. These areas align with the three S&T Advantages as follows:

• Knowledge Advantage: is based on the premise that Canadians must be positioned at the leading edge of the important developments that generate health, environmental, societal, and economic benefits. The S&T Strategy identifies health and related life sciences and technologies as one of four priority knowledge areas. The Ivey Centre's thought leadership activities (e.g. development of white papers, symposiums, and conferences) strive to generate new knowledge specific to commercialization and innovation adoption in health systems.
• People Advantage: is based on the premise that Canada must be a magnet for the highly skilled people we need to thrive in the modern global economy with the best-educated, most-skilled, and most flexible workforce in the world. The Ivey Centre's education and leadership development activities strive to equip the next generation of health leaders with the required skills to facilitate the adoption of innovative technologies, processes and management systems.
• Entrepreneurial Advantage: is based on the premise that Canada must do more to translate knowledge into commercial applications. Through its health innovation project activities, particularly demonstration projects, the Ivey Centre strives to bring together academic researchers with private and public sector partners to support the development, commercialization and adoption of innovations.

Subsequent Budgets (Table 2 below) have supported the Government's commitment to these S&T Strategy priorities.

Support for the Ivey Centre also aligns with Industry Canada's priorities under the Science, Technology and Innovation Capacity Program Activity of Industry Canada's Program Alignment Architecture. This Program Activity contributes to Industry Canada's Strategic Outcome 2: "Advancements in science and technology, knowledge, and innovation strengthen the Canadian economy." This focus is consistent with the objectives of the Ivey Centre's three integrated activity areas, the organization's overall mission "to be a global leader in health innovation, adoption and leadership education", along with their vision "to be a catalyst for a health system that is sustainable, productive and embraces innovation".

3.1.3 Does support for the Ivey Centre align with federal roles and responsibilities?

The assessment of federal roles and responsibilities for support of the Ivey Centre considered three factors:

• legitimacy of the federal government's role in supporting S&T in the higher education sector;
• alignment of support for the Ivey Centre with the mandate of Industry Canada; and
• degree of complementary and/or duplication between the Ivey Centre's activities and those of other organizations or programs supporting the commercialization and adoption of health innovations.

Role and rationale for federal government support of R&D in the higher education sector

A review of S&T policies from several developed countries all identify a key role for direct and indirect public sector support for research and innovation to ensure the country/region achieves a strong and competitive economy. This encompasses basic and applied research conducted in the higher education sector, including activities in this sector that contribute to downstream business innovation. Specific to the Canadian context, the importance of government support (both federal and provincial) for research in the higher education sector was clearly highlighted in the Expert Panel Report on Federal Support for R&D.^{Footnote 22}

Using research institutions to deliver S&T policy goals also appears to be an appropriate practice and has been observed in other countries. These organizations can facilitate interdisciplinary and international collaborative research and negotiate cross-sectoral partnerships. Further, research centres can provide academics with the legitimacy to act in selected focus areas (e.g., in "health innovation"), as well as provide them with opportunities and resources to optimally fulfill stated objectives in these areas. Partners in other sectors, and society more generally, are therefore more likely to view these types of organizations as credible actors that can help address societal issues related to their specific area of focus.^{Footnote 23}

In addition, public policy literature identifies and examines several types of rationales used to justify federal support for R&D. These rationales range from the traditional "market failure" argument (that federal support for R&D is necessary because firms do not provide adequate support for these activities), to more recent efforts to apply evolutionary theories. The latter approach generally views government funding as a means to enhance competitive performance and promote structural change (e.g. diversity and connectivity) within the S&T system. The interaction between academic institutions and firms can be viewed in this framework as an interactive and dynamic process,^{Footnote 24} which contributes to the capacity of firms to take up and exploit new knowledge.^{Footnote 25} Funding should therefore be delivered as an adaptive "policy mix" that considers the interactions across a multi-level and multi-actor system (rather than a generic approach such as that which might be prescribed to correct market failures). Accordingly, Industry Canada uses a variety of mechanisms and works with several partners to achieve S&T and innovation policy goals, including support for a number of arm's length organizations such as the Ivey Centre.

Alignment of support for Ivey Centre with mandate of Industry Canada

Industry Canada's mandate and responsibilities for Canadian S&T activities and policy goals stem from the Department of Industry Act, 1995, subsections 4(1) (a) and (b), "industry and technology in Canada" and "science in Canada" respectively. Section 5 states that these responsibilities include encouraging "the fullest and most efficient and effective development and use of science and technology" as well as fostering and promoting science and technology in Canada. Section 14 of the Act provides the Minister the authority to make contributions to facilitate the implementation of any program or project to fulfill its mandate.

As a higher education institution delivering activities and services that align with federal S&T priorities, support for the Ivey Centre falls under Industry Canada's broad mandate to foster and promote S&T in Canada. In the area of health innovation, given the inter-connectivity of actors within the health care system, public support crosses jurisdictional boundaries. For instance, responsibility for regulatory and other policy issues falls under the jurisdiction of Health Canada while procurement of health technologies and the delivery of health services fall under provincial jurisdiction.

Within the broader health innovation sector the Ivey Centre's area of focus, "identifying, assessing, commercializing and driving adoption of innovative health technologies, systems and processes" remains consistent with the S&T responsibilities of Industry Canada. Specifically, R&D activities supporting the commercialization and innovation efforts of businesses in the life sciences industry sector and the development of highly qualified personnel trained in the management of commercialization and innovation processes, are areas that align with the mandate of Industry Canada. However, there are indications that the focus of the Ivey Centre's activities are shifting towards broader system-level issues. While these activities contribute to strengthening innovation within the health care system, additional analysis would be needed to ensure that the future focus of Ivey's activities continues to best align with support from Industry Canada.

Complementarity of Ivey Centre activities and other organizations and government programs

Given the magnitude and breadth of the health sector, there are a number of programs and organizations that support the commercialization and adoption of health innovations. These can be found at both the federal and provincial level, and are delivered by public, private and not-for-profit organizations. An analysis of the complementarity or duplication of the Ivey Centre's main activity areas was conducted through an environmental scan of programs that: provide R&D services to business; deliver education and leadership development activities; and develop and disseminate knowledge.

R&D services to business

At the federal and provincial level there are numerous programs that provide support to private sector firms to encourage R&D activities. This support can be provided through direct support for a firm's internal R&D activities, or indirectly through tax credits on R&D expenditures. These programs are largely complementary to the types of activities and services provided through the higher education sector.

The federal government's most significant means to supporting R&D relating to health innovation through the higher education sector is through project funding provided to researchers and trainees by the Canadian Institute of Health Research (CIHR), and program support to research clusters or non-profit consortiums of academic, not-for-profit and industry sector partners through the Networks of Centres of Excellence of Canada (NCE). R&D activities relating to health innovation supported through the CIHR largely focus on early stage research, such as the development of emerging health technologies. This research focus is complementary to the later stage applied and experimental research conducted at the Ivey Centre. Research networks funded through the NCE, such as MaRS Innovation, follow a collaborative research model and conduct later stage applied and experimental research. However, they are largely seen as following a "supply-push" model of R&D focusing on transferring knowledge developed in the higher education sector to industry. This is in contrast to the Ivey Centre's "demand-pull" model of R&D where demonstration projects are industry driven.

The Ontario government recently partnered with MaRS to launch MaRS Excellence in Clinical Innovation and Technology Evaluation (MaRS EXCITE) program. In interviews, MaRS EXCITE was often identified as the program offering services most similar to the Ivey Centre. MaRS EXCITE is collaboration between a range of stakeholders in the health technology sector. The organization helps companies get innovative health technologies to market faster by offering pre-market testing to demonstrate that their technologies meet the needs of the health system and to generate data needed to support regulatory approval and provincial assessment processes. While the Ivey Centre provides similar R&D activities, it also provides support to companies to address adoption challenges after they have received regulatory approval. Further, it is important to note that MaRS EXCITE is a provincial program in contrast to the Ivey Centre's national reach. Additionally, some interviewees suggested that there is the potential for the Ivey Centre to build stronger linkages with both MaRS Innovation and MaRS Excite.

Education and Leadership Development

Many universities across Canada offer MBA programs that focus on health care management or policy, as well as providing executive MBAs tailored to professionals working in the sector. Among these programs, the Ivey Centre's specific focus on commercialization and adoption of health innovations is unique. Interviewees further distinguished the Ivey Centre's educational activities from other institutions by noting its cross-disciplinary approach, the variety of experiential learning opportunities provided to students, and the fact the Ivey Centre is a health innovation institute embedded within a business school.

Knowledge Development and Dissemination

A wide range of organizations develop and disseminate knowledge around public policy and other issues relating to the health care system. In the higher education sector, Dalhousie University, Royal Roads University, University of Toronto, McMaster University and McGill University all have research centres that focus on aspects of health care. These centres have received funding from federal partners such as CIHR, NCE, Health Canada, and Public Health Agency of Canada. Other non-government organizations who work in the space of policy research include the Canadian Foundation for Health Care Improvement and the Health Council of Canada which have been historically supported by Health Canada. Last, there are independent think tanks who work in the field of innovation and health care improvement, including the CD Howe Institute, the Fraser Institute and the Conference Board of Canada.

Each of these organizations conducts and disseminates research around similar topics (such as health care transformation) through their unique mandates, with the Ivey Centre's being the commercialization and adoption of technologies. While some degree of overlap in research is inevitable given the inter-connectedness of various aspects of the health system, one interviewee noted that research in similar areas is not necessarily counter-productive as it plays its part in building consensus towards change. Further, some interviewees perceived the Ivey Centre to be in a position to provide a distinct perspective when tabling their white papers given the Ivey Centre's breadth of experience conducting applied and experimental research with a wide variety of stakeholders.

Overall, findings from the environmental scan reveal that while the Ivey Centre's activities largely complement other programs there is some degree of overlap in specific activity areas. When viewed as an integrated program, the Ivey Centre occupies a unique and important niche due to its specialization in the commercialization and adoption of health innovations across all three activity areas. Most interviewees, particularly private sector partners and advisory board members, emphasized the strategic importance to Canada of having a higher education sector organization specializing in health innovation, as well as noting the importance of the synergies that the Ivey Centre is achieving through integrating its three activity areas.

3.2 Performance

3.2.1 To what extent have personnel been trained in the commercialization of health care technologies, systems and processes? To what extent has the Ivey Centre developed personnel who have the potential to be future health industry leaders?

The Ivey Centre offers a range of courses and other types of training opportunities for students and health sector executives. The focus of these activities includes targeted training in the commercialization of health care technologies, systems and processes, as well as topics that address broader innovation issues and management practices within the health care sector.

As part of the health sector stream MBA program, the Ivey Centre offers three health sector elective courses.^{Footnote 26} Of the three courses, the "Health Innovation and Leadership" course directly addresses the Ivey Centre's mandate to develop specialized talent in the commercialization of health care technologies, systems and processes. The Academic Chair of the Ivey Centre delivers the course and enrolment is capped at 60 students per semester. The course was consistently highly rated by students in post-course teaching assessments, and the Academic Chair was recognised by the University's Faculty of Heath Sciences Teaching Award Committee for excellence in teaching. Additionally, HBA and medical students are offered a health sector elective course focusing on leadership and management issues in the Canadian health care context. This course addresses broader issues of innovation in the health care system and does not focus specifically on the commercialization of health innovations.

Complementing the health sector courses, the Ivey Centre offers a number of experiential learning opportunities to students through participation in Health Innovation Projects (HIPs). HIPs include short-term Health Consulting Projects, Health Venture Projects and longer-term demonstration projects. Examples of short-term projects include developing marketing strategies, drafting business plans, and performing market research. These projects provide students with opportunities to tackle real-world problems of health sector organizations or start-up companies, and provide partners with student-driven consulting advice. While most demonstration projects focus more specifically on the commercialization of health innovations, the short-term consulting projects generally have a complementary focus allowing students to apply related business knowledge and skillsets. In interviews, students were very supportive of the HIP approach as projects were perceived to provide unique and valuable learning experiences that enable students to address pertinent health care issues while engaging in meaningful interactions with health sector stakeholders. Since 2009, a total of 46 students participated in 11 demonstration projects. Data confirming the total number of students participating in other types of HIPs was not available at the time of the evaluation, but it was noted that HIPs are a core requirement for the students enrolled in the Health Sector HBA and MBA streams.

The Ivey Centre also offers executive development programs, webinar series, and guest lectures to clinicians, scientists and senior executives. These programs mix lectures, discussions, case studies and workshops and are typically delivered over a short-term period (i.e. one to two days). For example, in 2012 the Ivey Centre delivered a two-day program, "Innovation and Leadership in Biopharmaceuticals", to the genomics scientific community. The executive program was attended by 86 delegates composed of executives from the finance, biopharmaceutical and life sciences sectors, as well as senior researchers. In total, six such programs have been conducted by the Ivey Centre, as well as four guest lectures and two webinars. At the time of the evaluation there was no available data (e.g. course evaluations or follow-up information) on the effectiveness of these courses in developing the commercialization and innovation skills of these executives.

In terms of the personnel trained by the Ivey Centre, available data is limited to the total aggregate number of personnel (students and executives) that have participated in the range of learning opportunities offered. Depending on the type of training, the breadth and depth of these learning opportunities varies significantly. For instance, the aggregate reported figures include personnel who have participated in one-time short-term activities (e.g. a special lecture, a conference presentation, a webinar) as well as more in-depth training (e.g. HBA and MBA courses, HIPs, executive development programs, or internships). Further, there was no information available as to the extent that the different activities focused on training in the commercialization of health innovations and contributed to the development of new knowledge, skills, or capacities. Moving forward, availability of data disaggregated by type of learning activity, along with post-activity assessment to provide data on the effectiveness of each learning activity, would assist in measuring the success of these efforts in developing new knowledge, skills, or capacities related to the commercialization of health innovations.

In terms of the extent to which the Ivey Centre has developed personnel with the potential to become health industry leaders, there was no data available on the career progression of graduates. At the time of the evaluation the Ivey Centre was developing an alumni strategy that will enable the collection of data from graduates in the future. The evaluation study was therefore limited to assessing the perceptions of a small number of students (4) and recent graduates (3) as well as anecdotal evidence provided by Ivey Centre management and partners. Due to both the limited number of interviews, as well as their purposeful selection,^{Footnote 27} these results cannot be generalized or taken as representative of the population of students trained by the Ivey Centre.

Overall, student and graduate interviewees perceived the training they received to have a positive impact on their potential to become future health industry leaders. Specifically, the development of management and innovation knowledge and skills specific to the health sector, along with experience gained conducting applied R&D oriented to the needs of health sector partners, was perceived to provide the students with a competitive advantage entering the job market. Further, the ability of the students to access the Ivey Centre's network of health sector partners for mentoring advice, along with individualized support provided by Ivey Centre faculty and management, contributed to their capacity to plan and pursue leadership careers. In terms of career direction, some students and graduates reported that the cross-sectoral approach of the Ivey Centre broadened their perspectives and increased their awareness of and interest in different health sector career options within both the private and public sector. Additionally, two students reported an increase in their awareness of the need for personnel within the health sector to advocate for innovation and two industry partners noted the ability of Ivey Centre students to demonstrate the type of analytic and communication skills needed to champion innovation.

3.2.2 To what extent has the Ivey Centre assembled and promoted knowledge in health innovation and commercialization suited to Canada's health care system? To what extent has the Ivey Centre improved and disseminated this knowledge?

The Ivey Global Health Conferences, as well as related symposia, are the main mechanisms for assembling and promoting knowledge in health innovation and commercialization. The conferences are used as a forum for different health sector stakeholders to exchange ideas related to innovation and to prioritize health systems challenges. Since 2009 four conferences and three symposia have been hosted by the Ivey Centre. These events include keynote addresses, panel discussions and networking opportunities. The keynote speakers represent the major health care stakeholders, including industry chief executive officers (CEOs), academic experts, as well as provincial and federal government officials. A file review of the conference programs demonstrate that the content and focus of the conferences is rooted squarely in improving the conditions for innovation for Canada's healthcare system. Some of the covered topics in these sessions included:

• The role of health policy in creating an environment for innovation;
• The financial challenges and opportunities facing health care, and the financial drivers to innovation; and
• The means to adapting, implementing and adopting innovative ideas into Canada's health care system.

The number of participants attending the conferences have steadily increased year over year, from 130 in 2009 to 230 in 2012. The results of post conference surveys conducted by the Ivey Centre were consistently positive, rating highly in overall satisfaction and providing positive feedback on the networking opportunities that were facilitated. In interviews, the Advisory Council members perceived the conferences to be effective in bringing together the right mix of speakers and addressing pertinent challenges in the health sector. Moving forward, Advisor Council members suggested that it would be beneficial for the Ivey Centre to further its reach and attract a broader group of participants, as the right stakeholders in attendance can be just as critical as the right speakers.

In terms of other forms of knowledge promotion, the Chair of the Centre has been actively involved in presentations both nationally and internationally. Since 2010, the Chair has delivered over sixty speaking engagements, with five being delivered in the United States and another four in Europe. These speaking engagements included delivering white paper findings, speaking to Ivey Centre's achievements and programs, and other ad hoc initiatives.

The Ivey Centre had also recently begun developing online tools that are designed to promote and foster health innovation and are accessible to the general public. One such tool is the Innovatexchange (www.innovatexchange.ca) website. Although still in development, the tool is intended to be an online platform that shares innovative projects from around Canada in hopes of connecting innovators with each other. To date, there are over 100 projects listed on the site. In addition, the Ivey Centre is active on social media, leveraging platforms such as Twitter, Facebook and Linkedin. For Twitter specifically, the Ivey Centre has a relatively sizable following with over 1,700 followers, which at the time of the evaluation was similar to other organizations involved in health innovation.^{Footnote 28} Although it is too early to assess the impact of these activities, their development demonstrates the Ivey Centre's ability to assemble and promote knowledge through multiple platforms.

The Ivey Centre's main contribution to improving knowledge is the development of their white papers which are tabled at the annual conferences and are intended to focus on emerging trends and business-related issues in health innovation.^{Footnote 29} To date, six such white papers have been produced, as shown in Table 3.

These papers are point-of-view papers that consolidate existing literature, conduct some specific research and provide general recommendations based on its findings. The papers do not typically have a specific target audience, although the majority are policy oriented, implying government as the primary audience. Specifically, the second white paper focuses on British Columbia's need to encourage information technology in its health care sector, providing recommendations for the provincial government. The remaining white papers centre on improving Canada's health care systems as whole, and are presented on a national level.

In interviews Advisory Council members, representing leading stakeholders in the health sector, perceived the topics of the white papers as being relevant to current health innovation issues. Further, the content of the papers was perceived to advance available knowledge in this area. Some Advisory Council members suggested that the papers are "required reading" for healthcare stakeholders. Additionally, one council member suggested that the white papers differentiate themselves from similar research and policy papers by their usefulness and pragmatism as opposed to more theoretical academic research. External stakeholders noted the contribution of white papers to consolidating available knowledge, verified that the topics are relevant, and suggested that along with other similar research, the Ivey Centre's white papers are helping to build the required consensus for policy change.

In interviews, the Ivey Centre management and Advisory Council members suggested that early impact of the papers' influence in effecting health care policy and practice changes can be measured by interest in the papers by policy makers. In particular, the white paper for adopting innovative medical devices received significant attention from provincial governments for addressing procurement issues, while the cost vs. value paper received international attention, and catalyzed a collaboration with the UK's National Association of Primary Care to provide policy advice on dementia care.

Building partnerships and fostering networks is a key means through which the Ivey Centre disseminates information and extends influence. The original target for the Ivey Centre was to establish 50 private sector partners by the end of 2012-13. The Ivey Centre defines a partnership as any agreement with another party to collaborate on the advancement of innovations in health care. These partnerships can either be formal and informal and can take the form of discussions about partnering, formal agreements, as well as participation in committees. By this definition, the Ivey Centre had achieved its partnership target at the time of the evaluation, and overall has over 200 partners listed in their annual reports. These partners include academic institutions, businesses, public sector organizations, and non-profit organizations, both national and international. Although the partnership figures demonstrate the Ivey Centre's ability to engage stakeholders, a clearer definition of partnerships and a delineation of targets into informal and formal partnerships would be helpful in measuring the impact of the collaborations.

3.2.3 To what extent has the Ivey Centre identified and assessed innovative health care technologies, systems, and processes? To what extent has the Ivey Centre commercialized or catalyzed the commercialization of health innovations that have led to new businesses, processes, and product lines?

Demonstration projects are the primary mechanism through which the Ivey Centre identifies and assesses innovative health care technologies, systems, and processes. The demonstration project approach examines and evaluates innovative technologies and models of care by using a framework to assess proof of concept (evaluating the utility or effectiveness of the innovation), proof of relevance (measuring the impact and relevance of the innovation for the intended population) and proof of value (measuring the impact, effectiveness, and sustainability of an innovation scaled up to the level of the health system). Projects are conducted by academic teams that may include students, faculty and/or health innovators in residence. Partners may include private sector businesses or not-for-profit organizations. The results of the project enable partners to make future decisions for broader implementation.

By March 31, 2013 the Ivey Centre was expected to have identified and assessed 17 innovative healthcare technologies, systems or processes. As of August 2013, the Ivey Centre's demonstration project portfolio included a total of 14 projects – 2 of which were completed and 12 others in progress. While the total number of projects, particularly completed projects, is lower than expected, a number of factors need to be considered. First, as a start-up organization the Ivey Centre required time to recruit personnel, establish governance structures, develop networks, and design activity areas. The first two years of the Ivey Centre were largely dedicated to these activities, with the Ivey Centre being in a position to make a concerted effort to engage with industry partners and accelerate the development of demonstration projects in its third year. Of the 14 projects that have been undertaken, four were initiated in 2011-12 and the remainder in the following years. Second, depending on the type and stage of the innovation being assessed the time needed to complete a project varies. This is particularly evident with the Ivey Centre's recent shift towards larger scale projects that place emphasis on health care system-level impacts. These projects can involve multiple partners and products and involve testing and assessment processes that span several years.

Of the two completed demonstration projects, one resulted in a product that is expected to enter the market in 2015. The other project was in the process of securing patents, as well as completing a commercialization agreement. As the remaining twelve demonstration projects are still in progress, it is too early to assess the extent to which these projects are leading to the commercialization of new businesses, processes and product lines. Further, the nature of non-disclosure agreements with demonstration project partners limits access to confidential information that could provide insight into the potential commercial success of these projects. Thus, in order to capture early indications of project success, a limited review of data (namely brief project descriptions) was conducted along with interviews with Ivey Centre representatives, Advisory Board/Innovation Council members, and demonstration project partners. Interviews focussed on the strengths and limitations of the demonstration project approach, along with perceptions of the potential commercial success of ongoing projects.

The data review revealed that the majority of the demonstration projects at the Ivey Centre are essentially information technology related, with technologies that provide information management, diagnostics services and communication systems. Examples include an electronic tracking board for hospitals, an electronic cognitive screening test for dementia, and a mobile digital health record application for physicians and patients. These types of innovations fit well with the strengths of Ivey Centre's demonstration project approach, and can be reasonably tested by groups of students in an applied setting.

Alongside these predominant project types, the Ivey Centre appears to conduct some demonstration projects that do not have a specific and direct connection to commercialization. One project is an assessment of the costs associated with home care for Alzheimer's patients. Although such research findings are valuable and may inform future business decisions, no commercialized innovation is expected to emerge from the research findings. Another example is a project that involves the Ivey Centre conducting an impact assessment and evaluation of the British Columbia Ministry of Health's health innovation policy. A subset of this study may provide recommendations on innovation adoption at the health system level, but there is no specific focus on commercialization. Finally, the innovatexchange website is classified as a demonstration project by the Ivey Centre. Described as a, "qualitative study of innovation across the global health care continuum", the website itself is not a commercial venture, but hopes to foster the creation of an health innovation network, which may in turn lead to further innovations. Although these projects support innovation within the health care system, they do not directly contribute to Industry's Canada's expected result of commercialized innovations.

Overall, the perceptions of the effectiveness of the demonstration project approach were largely positive. Specifically, the demonstration project approach enables industry partners to test and improve their products through proof-of-concept testing in a health care environment. This was identified as an effective strategy to address procurement barriers as it provided health care organizations with greater exposure to the benefits of adopting innovative products. Further, partners reported that this type of approach helps to demonstrate the value, scalability and impact of their products. For example, one industry partner pursued a demonstration project for a product that was already commercialized owing to the perceived benefits of measuring the impact of the product in a health care environment as a means to accelerate adoption.

The Ivey Centre, as an independent academically-based institution, was perceived to bring an important level of credibility to the assessments, particularly when interacting with health care organizations and purchasers. Demonstration project partners and Advisory Board members further identified the strength of the Ivey Centre's leadership as a contributing factor to the success of projects. The current leadership was seen as instrumental in developing the types of networks needed to support demonstration projects as well as providing the oversight and expertise needed to ensure the quality of the demonstration project activities. The capacity of the current leadership (one Academic Chair and one Executive Director) to sustain and grow the Ivey Centre's operations was raised as a concern going forward, particularly as Ivey Centre management indicated that current demand is above what can be undertaken.

The involvement of students in the conduct of demonstration projects was regarded positively by Advisory Board/Innovation Council member and industry partners. Of the 14 demonstration projects, 11 involved the participation of students. Students were perceived to bring a distinct perspective and enthusiasm to projects and the involvement of students from different disciplines was found to be particularly effective. For instance, one demonstration project, a software program developed to replace a paper-based assessment process, involved the participation of medical students to administer the assessment, computer science students to assist with programming and business students to measure the economic benefits. Having multiple disciplines involved ensures a system wide view, as no one solution from a single stakeholder would be adequate. While positive benefits were attributed to student involvement, some partners also pointed to the potential for challenges as students balance academic demands with the business needs of industry partners.

The strengths of the demonstration project approach were also noted to vary depending on the type of partner. Interviewees across stakeholder groups noted distinct advantages accruing to small and medium size enterprise (SME) partners. Specifically, the ability for SMEs to tap into the Ivey Centre's extensive network was identified as a major benefit. Start-up SMEs typically do not have established networks in the health sector, as the majority come from bioscience, engineering or computer science. In this regard, the Ivey Centre acts as a gateway to industry and academic networks, providing partners the ability to tap into the expertise of industry players and academic researchers. The Ivey Centre coaches these start-up organizations on how to enter the marketplace and navigate the health care system. Last, SMEs typically operate with limited resources at their disposal to conduct the type of research equivalent to a demonstration project. Depending on the agreement negotiated with the Ivey Centre, SMEs are able conduct projects without having to provide up-front fees, in return for downstream royalty payments. Five of the 14 demonstration projects involve SME partners.

Regarding the potential for successful commercialization results, demonstration project stakeholders noted that more time is required. Nonetheless, interviewees perceived that there are demonstration projects underway that show promise. Further, some interviewees were confident that as a result of demonstration projects a number of innovative technologies and processes will enter the marketplace, and new businesses may be formed, within the next five years. Monitoring the progress of these new innovations and businesses will help to ensure that results and impacts in these areas are captured.

3.2.4 To what extent does the current program model demonstrate efficiency and economy?

The extent to which the current program model demonstrates economy and efficiency was reviewed by assessing:

• the extent to which the Ivey Centre has leveraged federal contributions and achieved sustainability;
• the relative efficiency of using a third-party delivery model to achieve S&T goals; and
• the degree to which appropriate governance structures are in place to ensure the efficient and effective use of resources.

Leveraging and Sustainability

Leveraging is the funding secured from other sources in relation to the funding provided by IC. Interrelated with leveraging is sustainability, which is defined as the Ivey Centre's ability to become financially sustainable beyond the term of the IC funding through its own revenue streams and contributions from other sources. These are important indicators of efficiency for an arms-length organization, since leveraging and sustainability contribute to policy goals at no additional cost to the government (and the taxpayer).^{Footnote 30}

For the Ivey Centre, the main types of funding secured include: general investments towards the Ivey Centre to support its mandate; targeted investments to support demonstration projects; and sponsorship contributions towards the Ivey Centre's annual conference. When the Ivey Centre was first established, the target was to attract an additional $5 to $10 million from private and government sources (for a leveraging ratio of 1 to 2), alongside $4 million from charities or endowments, by July 2014.

Considering contributions that have been received and committed, the Ivey Centre is not on pace to achieve these targets. For the period 2009-2014, The Ivey Centre anticipates receiving approximately $3.7 million from private and public sources. Of this total, $2.6 million in cash has been received as of October 2013, with the remaining $1.1 million committed by the end of the 2013-14 fiscal year.^{Footnote 31} An additional $1.45 million in committed contributions are expected to be received after the July 2014 target.^{Footnote 32} Additional contributions ($1.3 million from private and public sources, and $4 million from charities or endowments) would have to be committed and received by July 2014 in order for targets to be met.

From 2009-10 to 2013-14, contributions (both received and committed) have resulted in a cumulative leveraging ratio of 0.74. From an annual perspective, in 2009-10 the leverage ratio was relatively high, reflecting the low proportion of IC contributions allocated to the first year and the original seed funding from Western University and the Ivey Business School. Afterwards, the annual leverage ratio consistently grew from 0.29 in 2010-11, to 1.23 in 2013-14. These increases are reflective of the Ivey Centre's growth as an organization. As the Ivey Centre has matured and expanded it has progressively attracted greater levels of funding. Figure 2 demonstrates the annual leverage ratio over the last five years.

Figure 2: Leverage ratio of the Ivey Centre from 2009-10 to 2012-13

In addition to cash contributions, the Ivey Centre has also leveraged in-kind contributions. In-kind contributions are the non-monetary transactions, such as any goods and services donated to carry out the activities of the Ivey Centre. Over the last four years the Ivey Centre has estimated its in-kind contributions at almost $5 million. The largest contributor to in-kind contributions was Western University as the Ivey Centre leveraged existing services and infrastructure. Other tangible services that have been provided to the Ivey Centre include in-kind consulting services, and the volunteer time of Advisory Council members, Health-Innovators in Residence and guest lecturers.

While the Ivey Centre has made advances in securing private and public sector contributions, the attraction of funding from charitable donations and endowments has been lagging. The Ivey Centre has made recent efforts to address the gap, and is in the process of developing a philanthropic campaign strategy. Further, the Ivey Centre has recently been granted official status as a research centre within Western University, and as such is now eligible for matching contributions to fund the Academic Chair position.

With regard to sustainability, the Industry Canada contribution to Western University was provided as start-up funding for the Ivey Centre, with the intention that the organization would be self-sustaining by year five. Sustainability was anticipated through revenue streams generated from technology transfer and licensing (from successful commercialisations); the market intelligence library; and the Executive Education Program. Overall, self-sustainability from the Ivey Centre's anticipated revenue streams was not yet realised at the time of the evaluation. While leveraged contributions have increased in recent years, they are typically targeted towards specific project activities. Revenues streams remain an important source of funding as they can be allocated at the discretion of the organization to support the activities needed to sustain and grow an organization's operations. Specifically, administrative and overhead costs, such as salaries, are typically difficult to secure from other sources. Over the last four years, Industry Canada was the predominant contributor to the Ivey Centre's administrative expenses.

Interviewees from Ivey Centre management, Advisory Council members, and private sector partners perceived that the Ivey Centre has high potential to generate sufficient revenue streams, particularly through licensing and other commercialization agreements, within the next five years. The mix of demonstration projects, in terms of those securing up-front fees and those with the potential for downstream revenue generation through licensing or other commercialization agreements, was perceived to be an important consideration for the organization's financial sustainability. Of the fourteen demonstration projects, one project has a commercialization agreement in place and four others are anticipated to secure agreements.

It was further noted that the expectation of sustainability within five years may have been overly optimistic given the time needed for the organization to become operational, launch demonstration projects, and get successful products to the health sector market. This is consistent with the findings of recent assessments of the NCE Centres of Excellence for Commercialization and Research (CECR) program. The assessments recognize that sustainability may not be possible within a five-year funding cycle, and note that the time a centre needs to become operational impacts the time needed to achieve sustainability. The findings further emphasize the need for better performance measures to track progress towards sustainability, and the need for "highly developed business plans" with defined commercialization goals.^{Footnote 33}, ^{Footnote 34} As the Ivey Centre anticipates commercialization revenues over the next five years, these findings are especially pertinent.

Relative Efficiency of Using a Third-Party Delivery Model

The Government of Canada has a well-established history of supporting third-party organizations to deliver public policy goals and such practices have been increasingly observed in other international jurisdictions.^{Footnote 35} Industry Canada representatives reported a number of advantages to using this delivery model compared to building internal capacity to deliver similar programming. Specifically, arms-length organizations are better positioned to more efficiently and effectively: attract specialized expertise; develop cross-sectoral partnerships; attract additional funding; develop the structures and procedures that best meet the needs of targeted R&D and training activities; and provide a capacity for independent non-partisan decision making. Additionally, support for arms-length organizations provides IC with access to specialized expertise and knowledge that can inform the development of S&T policy.

The assessment of the efficiency of using arms-length organizations included a review of the administrative requirements associated with this model. Administrative requirements include both IC's requirement to oversee and manage the funding agreement and the Ivey Centre's requirement to report on how funds were administered and programming delivered.

In terms of IC's administrative requirements, a relatively high number of labour-intensive compliance and reporting requirements were noted by IC management. The time needed for IC personnel to fulfill these requirements, particularly the detailed claims procedures, was found to be significantly higher than the time needed to oversee and manage other funding agreements with third-party organizations in IC's portfolio. In terms of the Ivey Centre's administrative requirements, financial and performance/corporate reporting requirements were considered. Ivey Centre management noted that financial reporting requirements for IC ran parallel to their own internal reporting requirements and further efficiencies were gained by leveraging in-kind financial services of Western University. However, the timelines (specifically the difference in fiscal year between the university and IC) and the frequency of corporate and performance reporting requirements were found to add additional administrative burden to the management of the organization. Going forward, the need to maintain the existing level of controls and reporting requirements could be re-assessed against the risk level and materiality of the funding agreement.

Overall, literature and interviews support the use of third-party delivery models. This model is considered a relatively efficient means to achieve targeted S&T goals while still ensuring an adequate level of accountability, and is consistent with the provision of IC's contribution in support of the activities and objectives of the Ivey Centre.

Analysis of Governance Structures

Governance practices and processes are used to guide decision-making within an organization on how resources are allocated and used to achieve objectives in an efficient, effective and economical manner. As such, a qualitative review of the degree to which the Ivey Centre has appropriate governance structures in place was conducted.

The Ivey Centre's Advisory Council is the organization's main governance body. The composition of the Advisory Council members reflects best practices by including participation of industry (both large companies and SMEs), with a mix of suppliers, distributors, users and funders.^{Footnote 36} The Advisory Council provides guidance on strategic direction, performance, and assists in the achievement of financial goals. A review of Advisory Council meetings minutes demonstrates that the Council has been providing advice in these areas, guiding both the efficient and effective use of funds.

Sub-committees of the Advisory Council are the Innovation Council and Executive Committee.^{Footnote 37} The Innovation Council reviews and recommends proposals for demonstration projects, thus functioning similar to a peer review panel. Appropriate assessment frameworks with clearly defined criteria are in place to guide the assessment of proposals. This process helps to ensure that resources are not allocated to projects that may not be feasible or sufficiently developed. By recommending only those proposals that are deemed most likely to succeed and achieve their intended outcomes, this process helps to guide the effective allocation of resources. In turn, the Executive Council advises the Academic Chair on a quarterly basis on the use of the Ivey Centre's financial resources and ensures that all funding is being appropriately and economically utilized. Additionally, by operating within an academic institution, the Ivey Centre is subject to the same expenditure management controls and policies of both the Ivey Business School and Western University, ensuring proper spending and oversight. Further, Advisory/ Innovation Council members and Ivey Centre management interviewees perceived that the existing governance and expenditure management controls are helping to ensure that resources allocation decisions demonstrate value and are made in an efficient and economical manner.

4.0 Conclusions and Recommendations

4.1 Conclusions

Relevance:

• The innovation, commercialization, and adoption of health care technologies, systems and processes are essential for the development of a more efficient and effective health care system. Advancements in this area contribute to the economic prosperity and societal well-being of Canadians. The three primary activity areas of the Ivey Centre respond to the key challenges associated with the commercialization and adoption of health innovations.
• The mandate and objectives of the Ivey Centre align with federal priorities and departmental strategic outcomes. Supporting research and development in the higher education sector is an appropriate role for the federal government and is consistent with the responsibilities and mandate of Industry Canada.

Performance:

• The Ivey Centre was expected to result in personnel trained in the commercialization of healthcare technologies, systems and processes, as well as highly qualified personnel with the potential to be future health industry leaders. While the Ivey Centre has developed and delivered a range of formal and informal training activities to students and health sector executives, data limitations prevented a comprehensive assessment of the effectiveness of the Ivey Centre's activities in support of training and development outcomes.
• The Ivey Centre was also expected to promote and disseminate knowledge in health innovation and commercialization suited to Canada's health care system. The Ivey Centre has demonstrated capacity to assemble and promote knowledge through multiple platforms. The Ivey Global Health Conference, along with the development of white papers, are recognized as effectively contributing to the advancement and dissemination of knowledge on topics of health innovation relevant to the Canadian context.
• The Ivey Centre was expected to identify, and assess innovative health care technologies, systems and processes with a view towards commercialization. The Ivey Centre has made some progress in these areas. However, more time is needed for innovations to achieve commercialization and some projects lack a specific commercialization focus.
• Additional contributions will be needed for the Ivey Centre to meet Industry Canada's July 2014 leveraging targets. Furthermore, as innovations are yet to be commercialized, more time will be needed for the Ivey Centre to achieve sustainability. The use of a third-party delivery model is an efficient means to achieve targeted S&T policy goals. However, relatively high administrative requirements for IC's oversight and management of the funding agreement with the Ivey Centre were noted. Finally, the Ivey Centre's governance mechanisms are consistent with recognized practice and support the organization in administering funds and delivering programming in an efficient and effective manner.

4.2 Recommendations

In the event of funding renewal, the following recommendations should be considered:

1. Science Partnerships Directorate should require the Ivey Centre to develop a business strategy with clearly defined approaches to achieving self-sustainability.
2. Science Partnerships Directorate should assess the need to maintain the existing level of compliance controls and reporting requirements against the risk level and materiality of the funding agreement.
3. Science Partnerships Directorate should collaborate with the Ivey Centre to refine performance measures to better capture the effectiveness of the Ivey Centre's activities. In particular, the need for performance data to measure the achievement of outcomes related to the development of highly qualified personnel and the commercialization of technologies, systems, and processes.
4. Science Partnership Directorate should ensure that future activities supported by Industry Canada have a strong focus on commercialization.