Evaluation of the Strategic Aerospace and Defence Initiative

This publication is available online at www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03862.html.

To obtain a copy of this publication or an alternate format (Braille, large print, etc.), please fill out the Publication Request Form at www.ic.gc.ca/Publication-Request or contact:

Web Services Centre
Innovation, Science and Economic Development Canada
C.D. Howe Building
235 Queen Street
Ottawa, ON K1A 0H5
Canada

Telephone (toll-free in Canada): 1-800-328-6189
Telephone (Ottawa): 613-954-5031
TTY (for hearing-impaired): 1-866-694-8389
Business hours: 8:30 a.m. to 5:00 p.m. (Eastern Time)
Email: info@ic.gc.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Innovation, Science and Economic Development Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Innovation, Science and Economic Development Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Innovation, Science and Economic Development Canada.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the Web Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Innovation, Science and Economic Development Canada, 2017.

Aussi offert en français sous le titre Évaluation de l'initiative stratégique pour l'aérospatiale et la défense

PDF version

March 2017

Presented to the Performance Measurement and Evaluation Committee on March 30, 2017

Approved by the Deputy Minister on May 19, 2017

Table of contents

List of acronyms

AEB
Audit and Evaluation Branch
IRAP
Industrial Research Assistance Program
ITO
Industrial Technologies Office
OECD
Organization for Economic Co-operation and Development
R&D
Research and Development
SADI
Strategic Aerospace and Defence Initiative
SMEs
Small and Medium-sized Enterprises

List of tables

List of figures

Executive summary

Program overview

SADI was launched in 2007, and is one of the largest contribution programs at ISED. The program provides repayable contributions for industrial research and pre-competitive development projects in the Canadian aerospace, defence, space and security (A&D) industries.

The program's objectives are to:

In 2007, SADI was provided an initial budget of $900 million over five years. Budget 2013 announced that the Government would continue to provide stable funding for SADI and allocated close to $1 billion over five years to the program. As of March 2016, SADI had authorized $1.32 billion in assistance to 39 projects.

The Industrial Technologies Office (ITO), a Special Operating Agency of ISED, manages and delivers the program.

Evaluation purpose and methodology

The objectives of this evaluation were to address the core issues of relevance and performance in accordance with the Directive on the Evaluation Function, with particular focus on the design of the SADI program and how it responds to the needs of the aerospace and defence sector.

The assessment covered the period from April 1, 2012 to March 31, 2016 as the program was last evaluated in 2011–12. The evaluation employed eight data collection methods, including a document review, literature review, environmental scan, interviews, social network analysis, sector survey, case studies, and program data analysis.

Findings

Relevance

Aerospace and defence is a key sector of the Canadian economy. There is a need for a program to support these industries in Canada in order to ensure they are competitive internationally. SADI, however, may not be fully responding to industry needs as currently designed.

The program is aligned with the federal government's responsibility to foster competitiveness and to support R&D and is complementary to other programs that support similar work at the federal and provincial levels.

SADI has kept pace with priorities of the government in support of Canadian R&D in strategic sectors. It further responds to the Inclusive Innovation Agenda's aim to support the growth of globally-leading Canadian companies, invest in new technologies, and develop people through collaboration between academia and the private sector.

Performance

While SADI's reach has been restricted by a number of factors since inception, including challenges in identifying and reaching out to potential recipients, program design issues and working to measure results against a small recipient base, the evaluation found evidence that the program is largely meeting its objectives.

There is evidence that SADI recipients increased their investments in R&D as a result of the program. There are also indications that SADI accelerated innovation and kept R&D in Canada that might have been conducted elsewhere and that SADI-supported R&D activities resulted in the development and commercialization of new products, processes, services and technologies. 

Further, SADI recipients are collaborating with academia and the private sector, sometimes making joint decisions. The requirement to allocate one per cent of funding to collaboration is having a positive effect on recipients' interactions with academia. More significant efforts to encourage collaboration could deepen the program's impact in this respect.

There is also evidence that SADI contributed to the competitiveness of recipient firms. Additional performance information would shed light on whether SADI projects are leading to increased R&D, as well as the longer-term impacts of funding. It should be noted that SADI has faced some restrictions on performance measurement due to its small recipient base and the need for confidentiality, as well as long timeframe of R&D projects to fully realize economic outcomes.

There are several elements of the program that make it less attractive for potential applicants. The amount risk-sharing taken on through the program, the application process, and the program's reporting burden may all be contributing to low uptake of the program. There are a number of key lessons learned that can be taken from this program.

Lessons learned

Issues such as the low level of uptake and program elements that diminish SADI's attractiveness to potential applicants suggest that the program design of SADI requires careful consideration going forward. Further, improved performance measurement would result in a better understanding of the long-term impact of large funding programs like SADI and enhance reporting on results.

The following lessons learned were developed to inform the design and delivery of the newly announced $1.26 billion five-year Strategic Innovation Fund as well as future G&C programs providing support to key sectors of the Canadian economy. These lessons learned also take into account the Budget 2017 commitment to consolidate and simplify existing business innovation programming, including the Strategic Aerospace and Defence Initiative, and to expand support to other dynamic and emerging sectors.

top of page

1.0 Introduction

This report presents the results of an evaluation of the Strategic Aerospace and Defence Initiative (SADI).

The purpose of the evaluation was to assess the relevance and performance of SADI. The report is organized in four sections:

1.1 Program context

Canada is among the leading aerospace nations in the world. Its aerospace industry is the fifth largest, and the second largest relative to the size of the economy.Footnote 1 Over the years, the Government of Canada has played an active role in providing support to the aerospace, defence, space and security (A&D) industries for research and development. Programming has evolved from the Defence Industrial Productivity Program to Technology Partnerships Canada to the current Strategic Aerospace and Defence Initiative, launched in 2007.

In 2011, the Government of Canada announced that it would initiate "a comprehensive review of all policies and programs related to the aerospace/space industry to develop a federal policy framework to maximize the competitiveness of this export-oriented sector and the resulting benefits to Canadians." The resulting report entitled the Aerospace Review Report (2012), led by the Honourable David Emerson, included seventeen recommendations to optimize federal support for aerospace and space focusing on innovation, market access and supplier development, skills and procurement.

The report recommended that SADI funding levels be maintained, and that SADI modify its terms and conditions to make it a more effective program for stimulating the development of the aerospace and space technologies of the future. Instead, changes were made to SADI that included expediting the application process for smaller applicants and increasing the share of the total eligible project costs covered by SADI from thirty to forty percent.

1.2 Program description

One of the largest contribution programs at ISED, SADI provides repayable contributions for industrial research and pre-competitive development projects in the Canadian aerospace, defence, space and security (A&D) industries.

The program's objectives are to:

SADI is accessible to Canadian incorporated companies of all sizesFootnote 2, located in Canada, that perform R&D in the country. Currently, large companies account for the majority of total authorized assistance provided, with the aerospace sector receiving the largest share. That said, the majority of projects have been undertaken by SMEs.

1.3 Program delivery

SADI is managed and delivered by the Industrial Technologies Office (ITO), a Special Operating Agency of ISED. Footnote 3 ITO is led by an Executive Director and is made up of five directorates, employing 68 full-time equivalents that manage various aspects of ITO's three main programs including SADI, the Technology Demonstration Program and the Post-Secondary Institutions Strategic Investment Fund. ITO also manages legacy programs including the Technology Partnerships Canada Program and the Defence Industrial Productivity Program.

SADI Highlights as of March 31, 2016:

The delivery of SADI comprises the following three phases:

  1. Application phase: ITO analysts screen each application received to ensure that the application contains adequate information upon which to start a due diligence review and that the applicant and project meet the eligibility requirements. ITO analysts then conduct a detailed review of each project proposal to ensure it meets ITO's assessment criteria, which include an evaluation of the applicant's capability to achieve the economic and social benefits that it expects will result from the R&D activities. During this step, site visits by ITO officials take place and third-party technical and market experts are consulted. If the applicant passes each step of the application phase, decision documents are prepared and a contribution agreement is signed by the government and the applicant.
  2. R&D (Work) phase: ITO monitors recipient progress towards meeting the contractual requirements outlined in the contribution agreement, including the outcomes and benefits associated with the project. As part of the monitoring process, ITO will conduct site visits, review recipient progress reports and verify claims for reimbursement. Projects in the R&D phase typically last five years.
  3. Repayments (Benefits) phase: Once projects have completed the R&D phase, they move to the repayments phase with repayment starting two years following the termination of the R&D phase. In addition to ISED collecting repayments, ITO prepares repayment forecasts and monitors the recipients' overall progress in achieving expected outcomes. There are two different types of repayment terms for SADI projects. If repayment is unconditional, there is a fixed annual amount. If repayment is conditional, repayment is based on the recipient's gross business revenues. In both cases, repayments are typically repaid over a 15 year period.

1.4 Program resources

SADI was provided an initial budget of $900 million over five years (2007–08 to 2011–12). In 2013, the Government announced it would continue to provide stable funding for SADI and allocated close to $1 billion to the program over five years. As of March 31, 2016, SADI has authorized $1.32 billion in assistance to 39 projects. Within the current portfolio, approved project funding ranged from $300,000 to $300 million.

1.5 Logic model

The logic model in Figure 1 depicts SADI's program theory. It shows how the program's activities are expected to lead to certain outputs and various levels of outcomes and ultimately to one of ISED's strategic outcomes.

Figure 1: SADI Logic Model

SADI Logic Model (the long description is located below the image)
Description of Figure 1

Figure 1 depicts a logic model for SADI. The logic model is a diagram showing how SADI is expected to work in theory by conceptually separating the initiative into SADI objectives, activities, outputs, immediate outcomes (0-5 years), intermediate outcomes (greater than five years), and ultimate outcomes. It shows the linkages between these program elements.

There are three SADI objectives:

  1. Encourage strategic R&D that will result in innovation and excellence in new products and services.
  2. Enhance the competitiveness of Canadian aerospace, defence, space and security (A&D) companies.
  3. Foster collaboration between research institutes, universities and colleges, non-profit organizations and the private sector.

There are six activities:

  1. Communicate program availability, eligibility, processes, expected outcomes and benefits.
  2. Receive application, assess eligibility of applicants and their R&D projects.
  3. Conduct due diligence on the company and its project.
  4. Formulate decision documents and seek appropriate approval.
  5. Preparation of Contribution Agreements.
  6. Monitoring of Agreements, R&D activities, repayments and program outcomes.

One activity (communicate program availability, eligibility, processes, expected outcomes and benefits), leads to an output of website development, proposal guidelines, program information, applicant consultants. This leads to an immediate outcome of awareness of SADI by Canadian aerospace, defence, space and security companies, which in turn leads to an intermediate outcome of Canadian public better informed of program benefits.

The other activity boxes lead to two outputs; the execution of contribution agreements and amendments (if warranted) of Contribution Agreements.

These outputs lead to three immediate outcomes which are:

  1.  Increased strategic R&D of recipients.
  2. Leveraging of private sector investments in R&D and
  3. Collaboration between research institutes, universities, colleges, and the private sector.

The immediate outcomes lead to two intermediate outcomes which are:

  1. Increased development and commercialization of innovative products, services and technologies processes
  2. Longer collaboration benefits

These lead to two ultimate outcomes to increase competitiveness of Canadian aerospace and defence firms and to contribute to the achievement of broader technological, economic, environmental and social benefits for Canadians.

top of page

2.0 Methodology

2.1 Evaluation scope and objectives

The objectives of this evaluation were to address the core issues of relevance and performance in accordance with the Directive on the Evaluation Function along with issues identified by SADI's management. Specifically, SADI management requested that the evaluation consider the design of the program and how it responds to the needs of the aerospace and defence sector. The assessment covered the period from April 1, 2012 to March 31, 2016 as the program was last evaluated in 2011–12.

2.2 Evaluation questions

Based on the program's Performance Measurement Strategy, consultations with ITO and research commissioned by AEB to assess SADI's intermediate outcomes, the evaluation addressed the following questions:

Relevance

  1. To what extent does SADI address a demonstrable need?
  2. To what extent does SADI align with the federal government's role and responsibilities to support R&D? Are there overlapping programs and services?
  3. To what extent do the objectives of SADI align with the priorities of the federal government?

Performance

  1. To what extent is SADI reaching its target groups?
  2. To what extent have recipients increased their investments in R&D as a result of SADI repayable contributions?
  3. To what extent has SADI increased the competitiveness of its recipients?
  4. To what extent has SADI fostered collaboration between research institutes, universities, colleges and the private sector?
  5. To what extent has SADI-supported R&D activities increased the development and commercialization of innovative products, processes, services and technologies?
  6. To what extent is SADI using the most appropriate and efficient means to achieve its objectives?

2.3 Evaluation approach

This evaluation adopted a goal-based approach to address the program's stated outcomes, meaning the evaluation questions were based on certain outcomes of the program logic model and tested to see whether or not the program was achieving them. In order to ensure the program's impact was considered in the appropriate context, theory-based analysis was employed in evaluating the program's contribution to innovation. AEB staff undertook most of the data collection; a contractor was engaged to conduct the non-recipient survey, the economy-efficiency analysis and a portion of the interviews.

2.4 Data Collection Methods

The multiple lines of evidence used to address the evaluation questions are described below.

Document review

The document review addressed both relevance and performance. It was conducted to gain an understanding of SADI, its alignment with government priorities and the achievement of expected outcomes. Further, it provided insight into the achievement of program objectives and identified changes made to the program since the last evaluation.

Literature review

This review contributed to informing the evaluation with respect to both relevance and performance questions, providing insight on the appropriateness of government support for R&D, alternative approaches, and the extent to which SADI's design is sound.

Environmental scan

The environmental scan described the ecosystem surrounding the aerospace and defence industry in Canada including the economic impacts of the sector in comparison to the manufacturing sector, its global context, and an overview of international trends in the United States, European Union, and other Organization for Economic Co-operation and Development (OECD) countries.

Interviews

Interviews were employed to address both relevance and performance. The evaluation team conducted a total of 54 interviews with SADI recipients and other stakeholders including:

Social network analysis

A network analysis was undertaken to determine the depth and significance of collaboration activities taking place via SADI projects. The analysis was based on data collected by ITO from recipients at the request of AEB at the time of the evaluation. Respondents were asked to classify the depth of their collaborations using a seven point scale, slightly modified from the original published in the American Journal of Evaluation (2006).Footnote 4

Sector survey

The survey assessed issues of relevance and performance with respect to companies in the aerospace and defence sector, as identified by the program. A total of 300 firms were identified and valid contact information was obtained for 258. Questions focused on the financing needs of each firm as well as their awareness of the SADI program and the capacity of respondents to make use of the program. A total of 92 responses were received for a response rate of 35.7%.

Case studies

The program has conducted 13 case studies on SADI projects. These studies were analyzed for the assessment and provided specific project information that contributed to addressing evaluation questions on outcomes, efficiency and economy. Two additional case studies conducted by AEB helped assess the extent to which the design of SADI responded to the financial needs of the aerospace and defence sector.

Program data analysis

The administrative and financial data review addressed questions on relevance and outcomes as well as economy and efficiency.

2.5 Limitations of the study

Data availability

The names of SADI project collaborators are self-reported by recipients through qualitative means (e.g. through written updates to the program). Consequently, lists derived from program documentation were not necessarily comprehensive and presented inconsistencies. To address this, the program was asked to collect lists of collaborators by type from its recipients at the time of evaluation.

Further, the program has been restructured since the last evaluation and has taken on the delivery of additional programs, as well as new corporate responsibilities. For this reason, the O&M and salary costs of delivering SADI only were not available for use in analysis.

Confidentiality

Some recipient information of interest to evaluators, such as the identity of collaborators and depth of relationships, is commercially sensitive, which precluded reporting on the relationships of individual firms or industries. Aggregate data is equally sensitive because of the small pool of SADI recipients. This restriction is addressed in the report by providing overall information on collaborative relationships associated with the SADI program rather than looking at them on an industrial or firm level.

Surveying Canadian A&D sector firms

Canadian businesses in general receive many requests for information from government but often have limited resources to respond. The burden issue may have impacted the sector survey response rates. In order to maximize response rates, a professional firm was commissioned to collect survey data and conduct appropriate follow-up activities.

The program had amassed a contact list of just under 300 known firms within the sector (up from just under 100 at time of the previous evaluation). While the program works with policy experts within the department to maintain this list, it cannot be known how representative it is relative to the full sector population. For that reason, the sector survey is used to explore the sentiments of those firms on the list, but is triangulated with other lines of evidence.

top of page

3.0 Findings

3.1 Relevance

3.1.1 To what extent does SADI address a demonstrable need?

Key finding: Aerospace and defence is a key sector of the Canadian economy. There is a need for a program to support these industries in Canada in order to ensure they are competitive internationally. However, SADI may not be fully responding to industry needs as currently designed.

The evaluation examined whether there is an ongoing need for government support to R&D in the aerospace and defence sectorFootnote 5 and whether SADI is meeting that need.

Aerospace and defence is a key sector of the Canadian economy and a major contributor to the R&D work conducted within the nation's borders. Canada's aerospace industry alone contributed more than $28 billion to the nation's GDP and 211,000 jobs in 2015.Footnote 6 Among Canadian manufacturing industries, aerospace is the largest investor in R&D, accounting for nearly 30% of total R&D in 2015.Footnote 7

A 2014 report by the Canadian Association of Defence and Security Industries on the defence industry stated that industry accounted for some 63,000 jobs spread throughout Canada, and contributed $6.7 billion to GDP in 2014.Footnote 8 Further, a 2015 report by the Canadian Space Agency estimates that Canada's space sector contributes approximately $5.4 billion annually to the Canadian economy and employs just under 10,000 highly-qualified personnel.Footnote 9 The report also underscores that the space sector fosters the creation of knowledge and innovation within Canada's research and scientific communities.Footnote 10

In 2012, the federal government released the Aerospace Review, highlighting the strategic importance of the industry to Canada. The report speaks to changes in technology and the world stage as well as the need for government to respond to these changes and ensure the health of the industry in Canada:

Failure to respond and adapt to changing global circumstances will not mean maintenance of the status quo but rather, steady decline, significant lost opportunities, diminished industrial capacity, fewer rewarding jobs in advanced manufacturing, and the gradual eclipse of an industry that has been a major contributor of the country's well-being.

Further, there are populous and geopolitically ambitious nations aiming to build up a competitive aerospace and defence sector within their own borders.Footnote 11 The world is in the midst of a digital revolution that has already had a profound impact on this sector.Footnote 12

The review of literature underscores the importance of government intervention in R&D in general, in particular with respect to sectors of economic importance to a nation.Footnote 13 Strong public investment signals to business that a government is committed to fostering a particular industry.Footnote 14 This in turn can attract new companies to a market as well as sustain or grow existing ones. Making significant investments in R&D is a critical component of encouraging the pursuit of innovation.Footnote 15

There was consensus among recipients interviewed that there is a fundamental need for government support for R&D in the aerospace and defence sector in Canada. According to several recipients such support is crucial for establishing a level playing field for the country to be competitive in an international context. Aerospace and defence are known as "pay to play" Footnote 16 sectors given the substantial levels of support provided by governments in other countries. The international market is also increasingly competitive, with rapidly evolving technology.

In the sector survey conducted for the evaluation, firms cited the most common barrier to increasing R&D as a lack of funding and resources (38%). The review of literature affirmed that innovation of any kind is high risk, and that banks and other traditional sources of financing tend to be risk adverse.Footnote 17 Footnote 18 Literature also suggested that businesses and markets underperform in innovation without government's involvement and direction. Footnote 19 Footnote 20 Footnote 21

The environmental scan conducted for this assessment echoed the need for government intervention in the sector, pointing to shifts in the economic landscape both in Canada and abroad since the inception of SADI in 2007. Further, climate change and environmental concerns are increasingly driving regulatory agendas as well as consumer behaviour. Regulatory response to climate change such as strengthening emission standards can also affect the features of aircraft and require innovation to meet new requirements. The SADI Program Highlights report notes environmental benefits such as improvements to energy efficiency (12 projects), production efficiency (17 projects) and the reduction of the use of natural resources (7 projects).

All this said, SADI does not appear to be fully responding to the needs of the aerospace and defence sector. Interviewees as well as the review of literature, applicant case studies and a recent audit of the program revealed that the lack of government risk-sharing, application process and reporting burden associated with the program all present on-going challenges for potential recipients.

For example, in interviews, recipients said that the program design does not provide adequate risk sharing by government. They were critical of the fact that SADI's repayable contributions must be reimbursed based on total revenues regardless of whether a product has been developed or commercialized. Where SADI supports projects at all levels of innovation – from fundamental research through to more advanced proof of concept – interviewees asserted that repayment does not always make sense.

This was re-enforced by the review of literature, which suggests R&D requires government investment to thrive within any nation.Footnote 22 It is also of note that the government introduced the Technology Demonstration Program since the last SADI evaluation in order to provide non-repayable funding to earlier stages of R&D, but SADI has not been modified in turn.

This aspect of the program, along with the application process and the reporting burden required by the program, is discussed in greater detail in section 3.2.6, which addresses the impact of program design on effectiveness.

3.1.2 To what extent does SADI align with the federal government's roles and responsibilities to support R&D? Are there overlapping programs and services?

Key finding: SADI is aligned with the federal government's responsibility to foster competitiveness and to support R&D. While there are other programs that support similar work at the federal and provincial levels, SADI complements these programs.

SADI is aligned with the federal government's roles and responsibilities to support R&D. Its objectives align with the 1995 Department of Industry Act, which tasks the Department with "…increase[ing] international competitiveness of Canadian industry, goods and services…"and "encourage[ing] the fullest and most efficient and effective development and use of science and technology."

Interviewees agreed that there is a clear role for the federal government in providing support for R&D in the aerospace and defence sectors. Given the international competition for manufacturing in aerospace and defence industry, they said such support is essential for ensuring the sector is present in Canada. In fact, it is common for other countries to support this sector. For example, the United States, France, Germany and the United Kingdom all make significant efforts to grow these industries domestically.

Additional funding options are available to companies conducting R&D in Canada from other federal programs and provincial governments, but SADI's size and terms are unique in Canada, making SADI complementary to other programs intended to support the aerospace and defence sectors. For example:

OECD data highlights that Canada, compared to almost all other countries, relies more heavily on tax credits, or indirect support, as a means of supporting business R&D.

Without federal support, some firms, large enterprises or subsidiaries may be able to obtain funds from sources such as their shareholders, or by redirecting funds from internal operations. That said, the literature review conducted for this assessment indicates that long-term innovation involves failures, setbacks and uncertainties, making firms risk-adverse to investing in it. Government support takes on risk that the private sector and traditional lending sources, such as banks, may not.Footnote 24 Footnote 25 Footnote 26

3.1.3 To what extent do the objectives of SADI align with priorities of the federal government?

Key finding: SADI has kept pace with priorities of the government in support of Canadian R&D in strategic sectors. It further responds to the Inclusive Innovation Agenda's aim to support the growth of globally-leading Canadian companies, invest in new technologies, and develop people through collaboration between academia and the private sector.

There is strong evidence that SADI's objectives have kept pace with federal government priorities supporting Canadian R&D in strategic sectors. The program is reflective of the overarching goal identified in the 2015 mandate letter for the Minister of Innovation, Science and Economic Development, which is "to help Canadian businesses grow, innovate and export so that they can create good quality jobs and wealth for Canadians." The letter highlights the importance of investments in key growth sectors where Canada has the ability to attract investment or grow export-oriented companies.

In Budget 2016, the Government defined a new vision for Canada's economy as a global leader in innovation, anchored by the Inclusive Innovation Agenda, which aims to build partnerships to drive skills development, invest in new technologies, and support the growth of globally-leading Canadian companies. The Agenda notes that over the past decade, there has been insufficient investment in people, technology and companies and it seeks to improve these investments. SADI's objectives are, in fact, directly aligned with these priorities.

For example, the program's objective to encourage strategic R&D that will result in innovation and excellence in new and improved products, services and processes highlights an emphasis on technology. By focusing on the aerospace industry, SADI is supporting the number one R&D investor across manufacturing industries in 2015, accounting for nearly 30% of total manufacturing R&D investments.

The program also aims to enhance the competitiveness of Canadian A&D companies by creating a supportive environment in which they can develop advanced products, services, processes and technologies. This reinforces federal leadership in a critical and strategic sector as well as helps stimulate new investments and grow world-class companies.

Finally, following its objective to foster collaboration between research institutes, universities, colleges and the private sector,Footnote 27 the program seeks to ensure there is a continuing pool from which to recruit talent. Interviewees stated that the program makes a strong contribution to creating highly-qualified personnel as well as helps to keep them in Canada. Where the program contributes to retaining talent within the nation's borders, it in turn makes the country a more attractive place to invest.

All of this follows with SADI's previous evaluation in 2012, which found equally strong links between government priorities and program objectives, demonstrating that the program has corresponded with the Government's priorities since inception in 2007.

3.2 Performance

3.2.1 To what extent is SADI reaching its target groups?

Key finding: SADI faces challenges in identifying and attracting potential recipients. Lack of in-depth knowledge of the program's target groups may be hindering outreach efforts.

SADI defines its target groups as firms conducting R&D in the aerospace, defence, space and securities industries. There are varying opinions with respect to how many firms are in this target population, with estimates ranging from slightly fewer than 100 to over 2000.Footnote 28 According to the Aerospace Review, in 2012 Canada had close to 700 aerospace and defence companiesFootnote 29 and just over 200 space firms.Footnote 30 The vast majority were SMEs, while about 30 were large firms. SADI itself had a list of about 300 companies that it considered to be eligible for funding at the outset of the evaluation.Footnote 31

Between its inception in 2007 and this assessment, SADI had approved a total of 39 projects with 30 recipient firms (an increase of 15 projects with ten firms since the last evaluation). The bulk of SADI support has gone to large firms (87%) in the aerospace sector (83%). See Tables 1 and 2.

Interviews with most groups revealed a perception that SADI has only been successful in reaching large industry players working at the higher technology readiness levels (TRLs 7-9). Several experts and ISED staff said there was a gap in funding for medium-size firms who require funding above the $1 million limit of the National Research Council's IRAP, but for whom SADI's administrative burden is too heavy.

Table 1: SADI Projects and Assistance by Firm Size
Size of Firm # of SADI projects % of SADI assistance
Source: Program data
Large >500 staff 10 87%
Medium 100 – 499 staff 11 9%
Small <100 staff 18 4%
Total 39 100%
Table 2: SADI Projects and Assistance by Sector
Sector # of SADI projects % of SADI assistance
Source: Program data
Aerospace 18 83%
Defence 14 14%
Space 3 2%
Security 4 <1%
Total 39 100%

The sector survey found that more than a third of firms on the program's list of known companies that might be eligible for SADI had never heard of the program. Many of the survey respondents who were aware of the program learned about it from a friend or colleague (29%) and very few learned about it through SADI outreach activities such as industry trade shows (4%) or the SADI website (4%). This suggests that the program's outreach strategy may require renewal and that follow-up work to determine the impact of outreach activities conducted may be required.

ISED staff and provincial government representatives agreed that SADI needs to strengthen its outreach. To date, the program's outreach efforts have focused on industry association and interest group events. Challenges identified in conducting increased outreach include a lack of funding, the lack of a client relationship management system to track client interactions, and the loss of regional staff who could conduct outreach.

A couple of representatives of industry associations said that SADI, as currently designed, has reached the full extent of its target audience. The low uptake of the program supports that premise. In 2015-16, only two new projects were approved with a total value of approximately $3.2 million, a small fraction of the value of assistance approved in each of the three previous years.Footnote 32

This echoes the findings of the last evaluation of SADI in 2012, when it was found that the actual reach was considerably smaller than its potential reach. Multiple factors impacting the reach of the program were identified as contributing to low uptake, including program design issues discussed elsewhere in this report, such as the application process and the administrative burden. The economic downturn was also identified as a possible explanation for the lower numbers of applicants.

3.2.2 To what extent have recipients increased their investments in R&D as a result of SADI repayable contributions?

Key finding: There is some evidence that SADI recipients increased their investments in R&D as a result of the program. There are also indications that SADI accelerated innovation and kept R&D in Canada that might have been conducted elsewhere. Additional performance information would shed light on whether SADI projects are leading to increased R&D or sustaining previous levels.

Program data points to total R&D investments of $1.32 billion provided to 39 SADI projects since the program's inception. Further, SADI has, on average, leveraged $1.82 in private contributions for every dollar of claims paid over the past five years, increasing the total R&D investment in Canada.

Most SADI recipients, experts and ISED staff agree that SADI has increased the amount of R&D taking place in Canada. Some said that without the program, some of that R&D would have taken place in other countries. A small number of firms said they undertook projects that would not have taken place at all without SADI funding. Interviews and case studies suggest that SADI encourages recipients to undertake projects that may not have otherwise moved forward in the immediate future.

In-depth analysis of the program's case studies revealed that SADI funding had made a significant difference in eight of the 13 projects included and some measure of difference in four. About a third of case studies suggested that the projects might have gone forward but would have had a significantly reduced scope without SADI funding.

SADI contributions can also accelerate the R&D process. About half of recipients interviewed for the assessment indicated that this was the case for their projects. One in particular stated that as a result of funding a new product was developed in three years that was expected to be developed in ten. Echoing this, a third of case studies indicated that the time frame for the research project would have been longer without SADI financing.

Despite this, it is difficult to determine if there has been an increase in R&D expenditures attributable to SADI in Canada, or if previous levels of R&D throughout recipient organizations were simply sustained by SADI. Additional indicators, including baseline information collected before funding and additional data collected following funding, would help to better determine how SADI contributed to overall corporate R&D activity.

3.2.3 To what extent has SADI increased the competitiveness of its recipients?

Key finding: There is evidence SADI contributed to the competitiveness of recipient firms. Additional performance information would shed light on the long-term impacts of the program.

Interviews and program reporting showed that more than two-thirds of SADI recipients have increased their competitiveness. According to program reporting, 25 of SADI's active projects resulted in successful commercialization of new and improved products, services, processes or technologies. In interviews, 16 of 23 firms said that SADI funding had contributed to their competitiveness. About a third of firms said that SADI funding helped them move new products or services more quickly to market.

In interviews, several company representatives said that SADI support had helped them grow their research team, as well as to use co-op hires to access new talent. Several recipients interviewed also said that it allowed them to grow their companies and increase the number of jobs in Canada. 

According to the program case studies, SADI funding helped seven of the 13 companies expand their market base, identify new sectors to pursue or achieve increased sales. Nine of the 13 SADI projects examined in the case studies resulted in the creation or maintenance of full-time jobs. SADI helped two other companies expand their market share and contributed to positioning six companies as industry leaders.

Case studies showed that SADI enabled two companies, one large and one medium-sized, to enhance their existing research facilities. Further, two SADI projects resulted in the recipient companies receiving supplier awards from foreign Original Equipment Manufacturers. Program documentation offers further insight:

While the qualitative evidence collected for this assessment provides insight into the impact of the program on the competitiveness of recipients, better defined outcomes and more comprehensive performance measures would have allowed for a more thorough assessment of results. The performance data required to conduct a full analysis of the program's long term economic impact was not being collected at the time of this assessment, and the program's small recipient population presented significant barriers for analysis for confidentiality reasons. Data to assess the program's impact on the innovation ecosystem that influences firms' choices to innovate would also be of use in understanding program performance.

3.2.4 To what extent has SADI fostered collaboration between research institutes, universities, colleges and the private sector?

Key finding: SADI recipients are collaborating with academia and the private sector, sometimes making joint decisions. The requirement to allocate one per cent of funding to collaboration is having a positive effect. More significant efforts to encourage collaboration may deepen the program's impact.

In 2009, the program introduced an eligibility requirement to enhance collaboration between research institutions, universities, colleges and the private sector. The program now requires that 1% of funds be allocated toward collaboration with post-secondary institutions. 

Since the addition of this requirement, all projects have reported collaborative relationships. While collaboration with one organization is sufficient to meet the program's requirements, most projects reported multiple collaborators. It is also of note that while recipients are only required to collaborate with academia, there is a plethora of private sector collaborative relationships around projects as well.

Both the literature and document review underscore the value of collaboration in innovative environments. The document review revealed an emphasis on collaboration with post-secondary institutions, which is aligned with common practices identified in the review of literature. Footnote 33 The literature review indicated that the genesis of innovation lies in the vast array of consumers, subcontractors, consortia and academic institutions within any given sector. Footnote 34 Footnote 35

The social network analysis identified just over 450 collaborative relationships associated with SADI projects. Most collaborative relationships reported to the program are among private sector organizations, representing nearly two-thirds of SADI project-related collaborations. It is of note that very few businesses collaborated with more than one recipient.

There were 113 unique relationships with post-secondary institutions, representing about 30% of all collaborative relationships. There were 37 institutions in this group, with some schools having as many as ten different SADI projects making use of the knowledge, expertise and/or experience of faculty and students.

At the time of the last evaluation of SADI, the program's emphasis on collaboration was relatively new. About half of recipients then reported that collaboration had increased as a result of the program, but most of these collaborations involved hiring more students. A handful also indicated engaging in more research-oriented collaborations.Footnote 36

Through a brief, one-question collaboration survey, this assessment was able to measure the depth of 303 collaborative relationships and found that just over half involved some level of joint decision making between partners, representing a much deeper than expected impact. Details are shown in Figure 2, below.

Figure 2: Type of collaborative relationships held with SADI recipients (N=303)Footnote 37

Bar chart illustrating the type of collaborative relationships held with SADI recipients (N=303) (the long description is located below the image)
Description of Figure 2

This is bar graph representation of the types of collaborative relationships held with SADI recipients. The legend identifies 3 categories: Other, Post-Secondary and Private.

The Y-axis has seven types of relationships. From top to bottom it says: Full collaboration (decisions reached by consensus); Coalition/Idea Sharing (All members have a vote in decision making); Coordination/Defined roles (some shared decision making); Defined shared resources/funding (some shared decision making); Information sharing; Hired co-op students; and, Networking only.

The X-axis represents the total number of collaborative relationships. The scale starts at 0 and increases by 20 up to 140.

From top to bottom, the bar for 'full collaboration'Footnote *** indicates 4 other, 23 post-secondary and 113 private.

The bar for 'coalition/idea sharing'Footnote ** indicates 8 post-secondary and 7 private.

The bar for 'coordination/defined roles'Footnote * indicates 2 other, 5 post-secondary and 1 private.

The bar for 'defined shared resources/funding'Footnote * indicates 10 post-secondary and 5 private.

The bar for 'information sharing' indicates 3 other, 15 post-secondary, 89 private.

The bar for 'hired co-op students' indicates 10 post-secondary

The bar for 'networking only indicates' 4 for post-secondary and 4 private.

Many interviewees said that the requirement to have 1% of funding allocated to collaboration was having a positive impact, but said more needed to be done. The 2012 Aerospace Review indicated that SADI does not do enough to encourage collaboration among different companies and researchers.Footnote 38

Considering the apparent evolution of partnerships between recipients and academia since the last evaluation of SADI, more significant efforts to encourage collaboration as well as the collection of detailed performance data on collaborators and the nature of these relationships to facilitate analysis may enhance the impact of the program.

3.2.5 To what extent has SADI-supported activities increased the development and commercialization of innovative products, processes, services and technologies?

Key finding: There was evidence that SADI-supported R&D activities resulted in the development and commercialization of new products, processes, services and technologies.

According to program data the majority of SADI projects (33 projects) had resulted in the development of new or next generation products, services, processes or technologies as of November 2016. Most recipients interviewed said that they had developed or commercialized innovations as a result of their SADI-supported projects. Program documentation and case studies of 13 SADI projects showed that all had resulted in new or improved products or technologies, although not all of those were market-ready by the end of the SADI project. For example:

While SADI is to be commended for developing case studies of its successful initiatives, in-depth research of its less successful initiatives could also provide valuable information for program management and improvement.

3.2.6 To what extent is SADI using the most appropriate and efficient means to achieve its objectives?

Key finding: There are several elements of the program that make it less attractive for applicants. The amount risk-sharing taken on through the program, the application process, and program and reporting burden may all be contributing to low uptake of the program.

While the majority of recipients interviewed praised the quality of service provided by ITO staff, SADI did not meet its targets for the number of projects approved in each year of the evaluation period. Further, an average of 18% of available contribution funding has not been disbursed in each of these years, meaning that more contributions could be allocated should appropriate projects present themselves.

The multiple lines of evidence employed for this assessment have found that government intervention is critical to encouraging the risk-taking required to foster innovation. Government support is also key to ensuring the success of the aerospace and defence sectors in Canada. It follows that government programming must be appropriately designed to meet the needs of potential recipients and encourage an appropriate level of risk taking.

Three problematic elements of SADI's program design were identified by the evaluation: the application process, the lack of risk-sharing by the program and the reporting burden placed on small and medium-sized recipients.

Lack of Risk-Sharing

Evidence suggests that the lack of risk-sharing on SADI-funded projects may be diminishing the program's attractiveness for SADI's target groups. Many staff members and the majority of recipients suggested that repayment of SADI contributions should be linked to project success so that the program assumes greater risk for R&D. Recipients suggested that loans should be conditionally repayable based on project royalties, not gross revenues, as was the case with the Department's former Technology Partnerships Canada program.

This is line with the review of literature, which found that companies can under-invest in R&D without government intervention because of the high-risk nature of the work, the expense and the long time horizons involved.Footnote 39 The Aerospace Review in particular identifies SADI's repayment terms as a "fundamental shortcoming" of the program and identifies a perception within the sector that SADI's funding terms are similar to a public version of conventional loans. The report states that this raises questions regarding SADI's capacity to fund high-risk innovation and charges that SADI funding should be provided on a greater risk-sharing basis.

Other contribution programs have repayment requirements that are more reflective of the value of funding and the risk profile of recipients. For example, under the new terms and conditions of the Automotive Innovation Fund, which offers repayable and non-repayable contribution funding, projects must demonstrate their strategic importance with respect to pre-determined elements in order to be eligible for non-repayable contributions. 

Application Process

ITO's service standard is to process applications of under $10 million within six months and under $2 million within four months. Applications requesting more than $10 million require Treasury Board and/or Cabinet consideration, which may result in additional time for approvals. Generally speaking, the program has met its service standards, with rare exceptions.

The application process, however, received critical comments across several lines of evidence. The majority of sector survey respondents (58%) considered the length of the application process to be unreasonable and a quarter of recipients interviewed also referred to the process as being too long and complex. Further, the long timeframe for project approval was considered by some interviewees to be an impediment for SMEs. It should be noted that an application process perceived to be too long by its target group may also indicate inefficiencies within the program's processes.

The application process may be improved by implementing a staged approach similar to what is found in programs such as ISED's Technology Demonstration Program, the Automotive Innovation Fund, and Western Economic Diversification's Western Innovation Initiative. For example, the Western Innovation Initiative application process involves a determination of eligibility through a formal pre-screening process.Footnote 40 Only those applicants deemed eligible are asked to provide more information, and then advance to a more detailed assessment. Other programs, such as the National Research Council's IRAP, have a project analyst work with the applicant to develop the application.

Further, case studies involving applicants who had not yet obtained funding indicated that ITO staff may in some cases not communicate issues early in the process with the result that a firm may spend resources preparing an application for a project that will not move forward. The evaluation could not fully assess how well the process is working as the program does not track its initial interactions with potential applicants.

This leaves a lack of administrative data on the counts of potential versus actual applicants and the reasons companies did not move forward with a SADI application. This information would be valuable in analyzing program uptake. Tracking inquiries would have additional value in measuring the impact of outreach activities since information on how potential applicants learned about the program could be collected and leveraged to enhance new outreach work or assess existing work.

Reporting Burden for SMEs

SADI monitoring and reporting requirements are currently the same regardless of the amount of project funding or the overall risk assessment of a project. Interviews with SADI recipients suggest that the program's reporting burden is felt most by SMEs that have received less funding than larger corporations and have fewer resources to assist with the administrative burden imposed by SADI reporting requirements.

In interviews, about a third of recipients asserted that the program's reporting requirements were excessive for a repayable contribution program. This was echoed by many SADI staff members, who were further critical of the requirement for firms to provide three years of financial statements and audits, a requirement at the time of application. The practice is viewed as too stringent for SMEs that may have innovative products and ideas, but less capacity to report and possibly fewer than three years of corporate history.

Instead of having the same reporting requirements for all recipients, SADI could align its progress reporting with the amount of funding and the risk profile of the recipient. This approach may encourage more businesses to apply for funding. That said, the right balance between risk and reporting requirements must be struck.

The lack of uptake and identification of key components that make the program both less effective in fostering innovation and less attractive for potential applicants in this assessment suggests that the program design of SADI requires careful consideration going forward. It should also be noted here that previous studies, including the last evaluation in 2012 and the Aerospace Review, identified similar issues. It is likely that the program's impact on the Canadian economy will be limited by its design without consideration of substantive change.

top of page

4.0 Conclusions and lessons learned

4.1 Relevance

4.2 Performance

4.3 Lessons learned

Issues such as the low level of uptake and program elements that diminish SADI's attractiveness to potential applicants suggest that the program design of SADI requires careful consideration going forward. Further, improved performance measurement would result in a better understanding of the long-term impact of large funding programs like SADI and enhance reporting on results.

The following lessons learned were developed to inform the design and delivery of the newly announced $1.26 billion five-year Strategic Innovation Fund as well as future G&C programs providing support to key sectors of the Canadian economy. These lessons learned also take into account the Budget 2017 commitment to consolidate and simplify existing business innovation programming, including the Strategic Aerospace and Defence Initiative, and to expand support to other dynamic and emerging sectors.

Date modified: