Evaluation of the Strategic Aerospace and Defence Initiative

3.0 Constatations (continued)

3.2 Performance

3.2.1 To what extent have SADI recipients increased their investments in strategic R&D as a result of SADI repayable contributions?

SADI has led to an increase in the total amount of R&D investments by A&D firms and SADI repayable contributions investments have leveraged private sector investments. Since the creation of SADI in 2007, the program has contracted 24 projects totalling $824 million of repayable contributions, leveraging an additional $1.8 billion in R&D investments for a total investment of $2.6 billion on innovation. This represents a leverage ratio of more than $2 for every dollar of SADI repayable contributions.

The question of whether there was a net increase in R&D investments that can be attributed to SADI is more complex. In the interviews, some recipients were unequivocal about the positive impact that SADI had in increasing their total R&D spending. Of the 19 firms that were interviewed, eight SADI recipients attributed an increase in their R&D activities directly to SADI.

Most firms reported a beneficial impact from SADI, but not necessarily an increase in their R&D spending as a result of the program. For these firms (11 of 19), the main impact of SADI repayable contributions was to accelerate their planned R&D spending. Given the necessity to invest in technology and to position themselves to take advantage of potential opportunities, these recipients reported that they would have eventually generated sufficient revenues from internal operations or found other sources to fund the necessary strategic R&D investments. That said, SADI played a critical role. The acceleration of planned R&D spending was beneficial because it helped reduce their time to market. For some firms where there was a competitive advantage in being first-to-market, this impact was significant.

SADI repayable contributions provided other related benefits to recipients. The program reduced the perceived level of risk of the R&D investment. It allowed recipients to be in a more cash positive position during commercial development and this made it easier for the firm to make the initial investments. The existence of SADI repayable contributions also allowed recipients to maintain their R&D capacity despite the fall in sales during the recent economic downturn. One firm stressed that it would have cancelled its R&D project because of the recession if it had not been for SADI.

Overall, case study participants firms were generally positive about the impact of SADI. They indicated that SADI repayable contributions helped them take on more R&D projects; broaden the scope of their projects; and accelerate the timelines of their projects. SADI repayable contributions also allowed firms to develop new areas of focus, and in some cases take on riskier projects. The case study participants were the earliest of the SADI projects so it is possible that the full impacts of SADI repayable contributions on later recipients will become more evident over time.

3.2.2 To what extent has collaboration between recipients and post-secondary institutions (universities, colleges and research institutes) increased as a result of SADI?

In the summer of 2011, AEB conducted an independent stocktaking of the status of collaboration activities between recipients and post-secondary institutions. As part of the review, the program identified nine SADI projects that included a new or enhanced collaboration between a recipient and a post-secondary institution (See Appendix G). The remaining 15 SADI projects were characterized as continuations of existing collaboration activities between recipients and post-secondary institutions. These existing collaborations had been established prior to SADI.

In interviews with researchers and other university stakeholders, the review found that the overall perception of post-secondary institutions was that they had not benefitted from new and enhanced collaborations under SADI. In most cases, the impetus for current collaborations was attributed to the activities and funding available through the Natural Sciences and Engineering Research Council and the Consortium for Research and Innovation in Aerospace in Quebec, and to a lesser extent smaller programs such as the Ontario Centres of Excellence or the Ontario Research Fund. This perception was compounded by the fact that many university collaboration projects have several partners and their timelines often preceded SADI, making it difficult to attribute increased collaboration to a specific program.

For the evaluation, AEB conducted interviews with recipients. In these interviews, almost half of the recipients (9 of 19 interviews) reported an increase in collaboration with post-secondary institutions that could be directly attributed to SADI. Specifically, five recipients reported that SADI repayable contributions had helped them hire more students and another two firms reported that SADI repayable contributions had led to increased research collaboration. For two recipients, SADI had led to both enhanced research collaboration and the hiring of more students.

An innovative example of collaboration came from the CAE case study. In addition to helping foster a number of research collaborations and the hiring of students, SADI played a direct role in enhancing CAE's collaboration with École Polytechnique. In 2011, CAE began collaborating with École Polytechnique to create a laboratory environment at the university for CAE's Augmented Engineering Environment (AEE). The AEE is a simulation product that enables Original Equipment Manufacturers to evaluate, test, and validate a range of aircraft models and systems during the development phase. In early 2012, the contribution of early versions of AEE licenses to the lab will help CAE because the contribution will serve as an end-user test platform for the product and help identify refinements.

Overall, the renewed focus on collaboration within SADI also raised awareness among recipients about the benefits and the potential for collaborations with post-secondary institutions. According to a senior manager at one recipient, the firm had struggled in the past to find effective ways of collaborating with universities and did not have a history of looking for ways to collaborate with universities. As a result, the recipient's SADI project originally did not include a collaboration element. Given the greater emphasis on post-secondary collaboration placed by the program, the recipient decided to try using students to test the quality of a recent software release. This project gave the students valuable work experience and provided the recipient with an opportunity to identify potential hires. According to the senior manager, the firm may never have come up with this idea for collaboration if SADI had not communicated the value that it placed on collaboration.

3.2.3 Recognizing that results may take years to occur given the nature of the program, to what extent have SADI-supported R&D activities increased (or will be likely to increase) the development and commercialization of innovative products, processes and services?

One of the challenges facing firms in the A&D industries is that they need to invest in research and the development of new technologies, but they will only see the benefits of these investments in the far future. Given that the program is only in its fourth year, all of the SADI projects are still in the R&D Phase. Among the 24 projects, the earliest year that any of the recipients will enter the Repayment Phase is 2013-14. The Repayment Phase is when the commercialization of products, processes and services is expected to take place.

When a firm submits its proposal to SADI, program officers look at the degree of strategic innovation and excellence as well as the feasibility of the research and development in terms of its likelihood to lead to product applications or service applications. The result is that the program likely screens out weak proposals that are unlikely to result in development and commercialization.

In this context, recipients reported that many projects have already resulted in the development of new products, services or processes. According to the SADI Annual Program Benefits Report of July 2011, five of the 24 SADI projects (21%) have already resulted in the development of new products, services, or processes. Interviews with recipients revealed a slightly more positive picture than the program's data. While almost half of the recipients (8 of 19) believed that it was too early to tell whether their SADI projects would result in full commercialization, the majority of recipients (11 of 19) reported progress towards the development of products, services or processes.

The case studies provided the richest source of detailed information on the success of recipients in developing and commercializing new products and services. There are three prominent examples of how SADI has already contributed to successful commercialization.

1. EMS: This recipient supplies high-speed data communications equipment for commercial and military aircraft. Supported by SADI funding, EMS has completed development on several products including the AMT-700 high gain antenna, an ARINC 781 transceiver set, and the Aspire 200 LG airborne communications system. In July 2011, EMS announced sales of the Aspire 200 LG to Anderson Air. This will deliver feature-rich voice and data connectivity for small and medium-sized business aircraft and allow passengers to make telephone calls and have Internet access while in-flight.
2. Norsat: This recipient produces communication solutions for remote and austere locations including satellite ground equipment which enables the high-speed transmission of data, audio and video signals. With the help of its SADI project, Norsat has developed a number of new microwave and satellite products. The firm also added X-band and improved frequency capabilities to its product families in order to enable their use by military organizations.

   Norsat has successfully sold products associated with its SADI-funded R&D project to several military customers. In August 2011, the firm announced that it had won a $1.3 million contract with the NATO Consultation, Command and Control Agency. The firm's equipment and services will play a critical role in NATO's communications in remote regions in Afghanistan.

3. SkyWave Mobile Communications: This recipient designs and manufactures satellite terminals and operates satellite network systems to assist customers in tracking, monitoring and controlling their fixed and mobile assets. In August 2011, the firm launched a new satellite service for machine-to-machine communications that allowed tracking and messaging for commercial transport and government fleets, the transmission of telemetry information from oil and gas distribution equipment; and the remote management and control of fixed assets.

3.2.4 Has SADI resulted in any unintended impacts or outcomes?

There were no significant unintended negative impacts arising from SADI. However, individual recipients experienced benefits and outcomes from SADI that had not been foreseen at the time that their applications were approved. These emerging benefits included cultural changes, improved reputation, and the potential for growth in entirely new product areas.

1. New Areas for Growth: Some case study firms reported that the products that had been developed through their SADI projects were being used for markets that had not been anticipated at the time of their application to the program. These recipients are entering new geographic markets, new adjacent product markets and establishing partnerships in new countries.
2. Cultural Changes: For one recipient, SADI's reporting requirements and the underlying need to develop products and commercialize its research led to a cultural change within the firm's engineering group. In the past, the engineering group would conduct research purely for the sake of research. It investigated particular technologies and features without having an end customer in mind. Because of the firm's participation in SADI, there is now a much stronger emphasis on defining concrete deliverables and developing project schedules with clear milestones and associated project risks. All of the firm's R&D activities now identify its intended customers.

   According to another recipient, the firm has become more philanthropic as a result of its SADI project. The technology that it developed has been used to support disaster relief efforts in Japan, Chile, and Haiti. In these circumstances, communicating is often as important as providing the basics for survival (i.e., food, water, and shelter). The recipient also helped an isolated First Nations community use this technology to set up distance learning and establish communication links with other First Nations communities. As a result of these sales and visits to support equipment installations, the firm reported that it has a greater appreciation of communities in need and is making more charitable contributions.

3. Reputational Benefits: Two recipients reported that participation in SADI has been used as a tool to market their firms. In particular, one recipient stressed that "the Government of Canada investment" in the firm through SADI has helped it present itself as a reputable, high quality supplier in the United States and this has contributed to greater sales.

3.2.5 How efficient is ITO in the delivery of the SADI program?

The key measures for assessing the efficiency of the program were the time required to process applications and the efficiency of the organization in terms of operating expenses as a percentage of total program expenses.

SADI Application Processing Times

Since the inception of the program, the average time required from the start of the due diligence phase to obtaining departmental sign-off on Contribution Agreements was 8.9 months for all projects. To better manage its performance, ITO introduced a service standard in July 2009 to complete these activities within six months for SADI projects that were less than $10 million.

Since July 2009, the average time required to complete these activities across all SADI projects under $10 million was 6.3 months, slightly exceeding the service standard. The range was between 4.6 and 8.2 months. These average processing times are a noticeable improvement from the average time that it took for these activities prior to the introduction of the service standard. It took an average time of 9.6 months for equivalent projects in 2008-09 and an average of 11.3 months in 2007-08.

According to the program, the main challenges in meeting the service standard included delays by the applicants in providing necessary information, challenges in scheduling site visits (especially during holiday and vacation periods), and the availability of external experts to conduct due diligence.

The full process that applicants actually experience remains longer than what the service standard measures. In particular, there are several steps that occur in the application process prior the start of the due diligence phase. Specifically, SADI officers will provide guidance and feedback on draft applications and conduct a suitability assessment on a submitted proposal. According to SADI staff, the time from initial discussions of eligibility with an applicant to the due diligence phase can take from 6 to 12 months.

The challenges that ITO has faced in processing SADI applications since the inception of the program were reflected in the interviews with recipients. Recipients recognized the need for due diligence and accountability when disbursing taxpayer funds. They also found ITO officers to be diligent and client-focused. However, recipients found the application process to be onerous. Half of the recipients (10 of 19) believed that a weakness of the program was the application process. It was seen as too complicated, too slow, and involving too much paperwork. Some recipients experienced many requests for additional details or the same information in a different format. The length of the application process was cited by two A&D firms as a contributing factor in their decision not to apply to SADI.

The delays in the application process also had negative impacts on recipients. Because they did not have clarity on whether their planned R&D expenses would be approved, some firms reported that they slowed down their R&D activities and held off on hiring decisions, and this affected their time-to-market. Some recipients also reduced the scope of their planned SADI project.

Actual versus Planned Efficiency

When the program was established, ITO was allocated funding to develop an internal operating capacity to meet the eventual expected demand for SADI repayable contributions. During the initial years, some additional overhead would be required to design, promote and manage SADI as the program gradually ramped up. Over the longer term, the operating budget was expected to represent 5.4% of total program funding.

The evaluation used this 5.4% figure to assess the efficiency of the program. It should be noted that this measure for operational efficiency is an indicative, rather than conclusive, measure of the desired efficiency. Further, the figure is based on operating costs allocated to both SADI and TPC.

In order for SADI to demonstrate efficiency, the use of resources would be broadly in line with the anticipated design. If actual operating expenses were greater than 5.4% of total program funding, then SADI would be operating less efficiently than anticipated. If actual operating expenses were lower than 5.4% of total program funding, then SADI would be more efficient than anticipated.

Table 4 presents operating costs as a percentage of total program funding after the initial year of SADI operations. Operating costs included salary expenses and operating and maintenance expenses. The total costs of the program included these expenses plus SADI repayable contributions.

Table 4: SADI Operating Expenses as a Percentage of Total Program Expenses

program funding	2008-09	2009-10	2010-11
Operating Efficiency	13.8%	9.0%	4.2%

From 2008-09 to 2009-10, the operating efficiency percentages were higher than 5.4%, which indicates that SADI operated at a lower level of efficiency than anticipated. This can be attributed to the fact that the expected demand for SADI repayable contributions did not materialize. As a result, the program's capacity was greater than required for actual demand. Although the lower-than-expected operating efficiency percentage was not desirable, the program leveraged its extra capacity and improved other aspects of SADI service delivery. In particular, SADI undertook stakeholder engagement activities, developed program guidelines and tools, and improved its internal processes. The program also expedited amendments to Contribution Agreements and claims processing.

Over time, ITO decreased operating expenditures. Specifically, resources dedicated to managing TPC and SADI were reduced and the total number of full-time-equivalents fell from 99 at the inception of the program to 84. In total, SADI operating costs were reduced by 25% from 2007-08 to 2011-12.

These reductions also coincided with an additional flow of disbursements of repayable contributions. Disbursements of SADI repayable contributions increased from $62.0 million in 2009-10 to $114.6 million in 2010-11. The increased efficiencies led SADI operating expenses to fall below 5.4% of total program funding. The net result was that the program operated in 2010-11 and beyond at a greater level of efficiency than what had been anticipated at the time of SADI's inception.