Evaluation of the Technology Partnerships Canada Program

This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Archived

This page has been archived on the Web

Archived

Summary Report

Submitted to:
Industry Canada

Submitted by:
Performance Management Network Inc.

October 14, 2003

Foreword

This summary report is intended for the use of Industry Canada management. It contains highlights of the formative evaluation of Technology Partnerships Canada conducted in 2002-03. As such, this report contains summary findings, conclusions and recommendations, but it does not comprehensively address all aspects of the study. For more information, readers are referred to the Evaluation of the Technology Partnerships Program Detailed Findings Report (September 29, 2003) produced under separate cover.

Table of Contents

Executive Summary

Study Context (Section 1)

This Technology Partnerships Canada (TPC) summary report is based on the issues andmethodologies laid out in the Technology Partnerships Canada Evaluation and AccountabilityFramework of July 2001. The study covered specific research questions related to:

  • program relevance and rationale;
  • program governance and management;
  • program design and delivery;
  • progress in delivering the program and achieving success; and,
  • lessons learned.

The evaluation methodologies include a review of program files and documents, an analysis ofTPC and IRAP-TPC management information system electronic files, interviews with TPC andIRAP managers and staff, other co-delivery partners and clients, telephone surveys of a sample of90 of 133 TPC clients and 120 of 323 IRAP-TPC clients with funded projects, and six casestudies (four TPC and two IRAP-TPC).

These multiple lines of evidence were used to develop findings and conclusions on the variousresearch questions.

This study was designed as a formative evaluation - focussed on assessing the governance,management, design, delivery and early results of TPC. For this reason the evaluation offers onlylimited information on overall economic impacts or the industrial policy relevance of theprogram.

Summary Findings

Relevance (Section 2)

In an era of Federal Government deficit reduction, TPC has been asked to cover a myriad of industrial policy objectives. The program was given a wide reaching mandate to create jobs and foster innovation in three very different sectors. At the same time, TPC's role in the innovation assistance process was limited by the requirements of investment payback, as well as being subjected to intense international scrutiny. These constraints have led to several logical inconsistencies in its set-up, as compared to previous and other existing contribution programs:

  • The program focuses on repayment of investments, yet it must fund (increasingly since the WTO decision of 1998) high risk innovation - often in emerging areas of technology.
  • TPC must operate in a consistent, transparent and 'fiscally responsible' manner, leading to a lengthy multi-step assistance process, yet it is by definition mandated to serve areas of high technology risk and market uncertainty - areas which require speed and flexibility.
  • The program was essentially developed in many respects as a 'son of DIPP'. The Defence Industries Productivity Program (DIPP) was designed to serve the needs of the mature, cold-war aerospace and defence industry of the 1970s and 1980s. This approach is likely not consistent with a program being asked to assist sectors (even that of the modern aerospace industry) facing a completely different market situation.

In conclusion, TPC has, in theory and rhetoric, been established to serve several publicly stated industrial innovation goals and sectors, while subjecting itself to the constraints of economic development assistance in the modern era. In reality, it would appear that the 'one-size fits all' approach that has been taken up to now may not be up to the task.

Governance (Section 3)

While TPC is identified as a Separate Operating Agency, it has, at present few of the flexibilities, such as increased spending authority, often associated with SOA status. Evidence suggests that TPC's partnerships with NRC's IRAP and NRCan's CCAF and TEAM have improved program relevance and success. Also, the TPC Advisory Board, originally identified as an important source of private sector advice for program management, has not met in several years.

Design and Delivery (Section 4)

The TPC assistance process is complex in terms of both its number of steps and the number of groups involved. Some of this complexity is likely unavoidable due to the numerous constraints placed on modern industrial assistance of this magnitude. However, when combined with a low investment in program administration, the effect has been to unfairly handicap the speed, flexibility and 'learning ability' of this program.

Success (Section 5)

Within the limits of the methods used for this formative evaluation, it appears that TPC and IRAP-TPC achieved reasonable project incrementality, significant technical and early job creation impacts. Commercial, therefore repayment, results have been limited to date. This can partially be attributed to the long time horizons required for many projects to realize commercial results and partially due to economic circumstances (i.e., a downturn in the markets for the sectors involved).

Case studies and survey responses suggested that there were 'public good' types of outcomes which were significant for a number of TPC investments. Unfortunately, a narrow public communications focus on repayment levels, combined with a lack of program investment in intermediate outcomes measurement, have worked against TPC in terms of telling its full performance story.

While TPC funding is important to every firm supported, it appears to play a critical role in support to the aerospace sector, due to the lack of other support programs. On average, since 1996, TPC has supported a large fraction of the major R&D projects and TPC contributions have been about 15 - 20% of the annual R&D expenditures in the aerospace and defence sector. In the other sectors, there are calls for an increased share of TPC funding, as the demand for TPC support is much greater than can be met by available funds. However, there are other government programs and private sector sources to provide funding support, and TPC plays a much less dominant role to the enabling and environmental sectors.

In terms of operational objectives, TPC has recently exceeded the targets of 3% 'overhead costs' and 50 Full Time Equivalent (FTE) program staff levels, that were found to be unrealistic for a program of this magnitude and complexity.

Lessons Learned (Section 6)

The use of a single program, essentially based on an assistance model from a previous era, to assist three critically different sectors in high risk innovation appears to have hampered the effectiveness of TPC.

Recommendations

As a formative or mid-course evaluation, the mandate of the study is to address program design and delivery issues. The study is not aimed at making judgments on the fundamental public policy instrument that underpins the TPC program, or to make summative judgements on the success or contribution of this program to the country's innovation agenda. Nevertheless, there is no evidence to suggest that the government should not remain committed to the "public good" ideals intended with this partnership investment.

It is clear that the objectives at inception of this program were significantly changed to adhere to the "pre-commercialized" dictates of the WTO decision of 1998 and to address the emerging economy of enabling and environmental technologies outside of the aerospace and defence sector. It is also clear that the program design and delivery parameters were never re-aligned to the new realities and objectives of an altered TPC program. These include program criteria, risk assessment, performance and outcome expectations, repayment plans and approval processes.

In order to fully exploit the relevance, effectiveness, transparency and ultimately the success of TPC, a mid-stream course correction is recommended. Accordingly, Industry Canada should undertake a review of TPC to ensure its suitability to the needs of the modern aerospace, enabling and environmental sectors. Such a review should encompass the domains of policy, governance and delivery processes. It is not the recommendation of this study that a "moratorium" be placed on this program but rather that Industry Canada map out a process to realign this program in accordance with the following recommendations:

Policy

1. In order to improve the relevance and effectiveness of Canada's innovation assistance, Industry Canada should undertake a policy review of TPC with regard to its relevance and suitability to the needs of the modern aerospace, enabling technologies and environmental technologies sectors. Options for consideration may include:

  • the establishment of clearly separate program elements under a TPC umbrella;
  • the development of completely separate programs; or,
  • other structures which will preserve this important element of innovation

assistance.

Irrespective of the conclusions of the policy review proposed above, the following recommendations are made with regard to the governance and design and delivery of the program.

Governance

2. TPC should determine what structural flexibilities would assist in delivering the program in a more effective and efficient manner, and seek approval for those changes. Options for consideration may include:

  • increasing TPC' s independence from Industry Canada; and / or,
  • seeking new opportunities for further delivery and co-funding partnerships.

3. TPC should reinstitute the TPC Advisory Board as soon as possible and involve the Advisory Board in the strategic oversight and management of the program.

Delivery Processes

4. TPC should review and revise the project selection and approval process, with a view to significantly reducing the length of time for decision-making and making funds available. Options for consideration may include:

  • negotiating higher levels of spending authority; and /or,
  • developing different levels of due diligence and approval processes for differing levels of funding and risk.

5. In order to reduce or eliminate the high level of funds carried forward, TPC should continue to develop more aggressive cash management practices. Options for consideration may include:

  • a strategy of over commitment on the number of projects and resources being considered for funding based on experience; and /or,
  • reconsidering the present practice of committing funds at the project selection stage, which is a major factor in under-utilization of available funding.

6. TPC should develop a more appropriate performance measurement and reporting approach, that captures operational performance information, as well as a broader range of early, intermediate and longer term technological, economic and societal benefits of projects.

7. Following from the above, TPC should communicate a more complete view of this innovation focused program and its technical and commercial outcomes to stakeholders and the public. This would include the high level of technical success being achieved by projects, as well as the lower level of commercial success achieved to date. TPC should communicate the high risk nature of projects, the fact that not all will be successful and that overall repayments will not likely meet the initial forecasts identified in proposals.

8. In order to achieve the process improvements noted above, TPC should increase the current levels of funding and staffing for policy, planning and program management to a more appropriate level, in line with requirements. The new levels should be determined by a needs analysis and comparison to other related programs.

1.0 Introduction

1.1 TPC Program Overview

In 1996, the Technology Partnerships Canada (TPC) program was announced as part of the 1996 Budget Speech, as a key element of the Government's Jobs and Growth Agenda, focussing on technology development in the aerospace industry, environmental technologies and information technologies. TPC provides funding for large scale, private sector pre-commercialization projects that support government policy objectives. The National Research Centre's (NRC) Industrial Research Assistance Program (IRAP) delivers the TPC program to small and medium sized establishments (SMEs) for relatively small projects, up to $3,000,000. The annual budget of TPC and IRAP-TPC in 2002-03 was $315,000,000. In the seven years of operations to March 31, 2003, TPC and IRAP-TPC have made over $2 billion in investments.

In response to a 1999 World Trade Organization (WTO) Panel Report, TPC was restructured to move the program away from funding near to market projects, towards earlier stage product development and pre-commercialization projects. At that time, Treasury Board Secretariat identified the need for a formative evaluation. To meet this requirement, Industry Canada commissioned an interim evaluation study that was begun in December 2002 and completed in August 2003. This report provides a summary of the evidence, conclusions and recommendations of the study. A more complete discussion of the findings and evidence collected on each of the evaluation issues can be found under separate cover in the report entitled Evaluation of the Technology Partnerships Program Detailed Findings Report, September 5, 2003.

For a more detailed description of TPC, see section 1.2 of the Detailed Findings Report.

1.2 Study Issues and Methodologies

The issues covered and methodologies utilized are based on those recommended in the Technologies Partnerships Canada Evaluation and Accountability Framework of July 2001. The study covered specific research questions related to:

  • program relevance and rationale;
  • progress in delivering the program and achieving success;
  • program design and delivery; and,
  • lessons learned.

The specific methodologies included in the evaluation of TPC were:

  • review of program files and documents received from TPC;
  • analysis of TPC and IRAP-TPC management information system electronic files;
  • interviews with 26 TPC and IRAP managers and staff, and other co-delivery partners;
  • telephone surveys of a sample of 90 of 133 TPC clients and 120 of 323 IRAP TPC clients with funded projects;
  • in-depth interviews with seven TPC and four IRAP-TPC clients; and,
  • six case studies (four TPC and two IRAP-TPC) selected to highlight the impacts of funded projects on the firm and wider community.

These multiple lines of evidence were used to develop credible findings and conclusions on the various research questions. The telephone surveys in particular included responses from a relatively large percentage of funded firms, ensuring that the survey data would accurately reflect the views of the client group, with little non-response bias.

For more details on study issues and methodology, see section 2.0 of the Detailed Findings Report.

1.3 Summary Report

The remainder of this report summarizes the evidence and conclusions reached on each of the research questions addressed under the evaluation issues. Recommendations to improve program design and delivery are provided where appropriate.

2.0 Rationale / Relevance of TPC

The methods and level of analysis required to fully address this issue are beyond the scope of this project. Past efforts to address this question from similar programs have typically exceeded the budget for this entire evaluation study. Given the scope limitations, the methods used to contribute to the understanding of this issue included a document and file review, interviews, case studies and the client survey.

2.1 Relevance to Meeting Government and Industry Needs for an Industrial Development Fund to Maintain and Increase Economic Growth and Support Sustainable Development

TPC was formed in 1996 in order to meet a need to create a "fiscally responsible" program that supported Government of Canada objectives of job creation, growth and sustainable development by supporting development of technologies in existing high technology sectors that could take their technologies to the marketplace, supporting private sector development of enabling technologies and attracting and holding "footloose" investments. The February 27, 1996 Speech from the Throne noted that the Government of Canada would invest in knowledge and technology to create enduring jobs for Canadians, and "support technology development in the aerospace industry, in environmental technologies and in critical enabling technologies such as biotechnology...". Since its inception, TPC has been viewed as a government instrument to create and maintain jobs in selected, strategically important, knowledge intensive sectors of the economy, and was, in fact, designed to achieve this objective. TPC has also been used as a government instrument to support public policy objectives related to connectedness, e-commerce, sustainable development and reduction in greenhouse gases.

TPC is the latest embodiment of government support to the aerospace and defence sector during the latter part of the 20th century. In this regard, TPC can be seen as replacing the Defence Industries Productivity Program (DIPP). Evidence suggests that this type of support has contributed to a very successful Canadian aerospace industry, internationally competitive in a number of selected areas, such as aircraft engines, landing gear and regional aircraft production. The Government has also made use of TPC as the vehicle to support the involvement of the Canadian aerospace and defence sector in development and demonstration work for the U.S. Joint Strike Fighter project. These projects are intended to position Canadian firms to compete effectively to produce components for this multi-use, low cost, next generation fighter aircraft. This initiative is expected to generate major levels of future revenues and to support the continued success of the Canadian aerospace and defence industry. TPC is also used to support government objectives related to the environment, sustainable development, and connectedness.

TPC was also designed to meet industry needs in the three supported sectors for additional financing to complement inadequate internal financing, to move quickly to develop a technology or product to match an identified window of opportunity, and to build technological capability and competitiveness. Having TPC investments conditionally repayable is also important to industry, as it becomes a form of pseudo-equity investment, repayable only upon commercial success, and does not appear as debt. Over 75% of TPC clients surveyed gave TPC a high rating for meeting their identified needs.

For more information on this issue, see section 3.2.1 of the Detailed Findings Report.

2.2 Evolution of TPC to Meet Changes in the National and International Environment

TPC has evolved to meet a number of changes. The most important change in the international environment was the WTO decision in 1998 that TPC was operating as a de facto export subsidy to the Canadian regional aircraft industry. This led to TPC modifying its program design and delivery to move away from 'close-to-market' projects towards funding late stage R&D and pre-commercialization initiatives. While the WTO finding applied only to the regional aircraft sector, TPC applied the changes to all program elements, and no longer considered export sales information as part of project selection. TPC did not, however, adjust its objectives or performance measures as a result of WTO.

More recently, TPC has responded to changes in the competitive position of the Canadian aerospace supplier group (Tier 3 and Tier 4) by instituting a pilot project to assist SMEs in this sector to develop and incorporate world-class business and manufacturing practices and technologies required to meet increased requirements from component purchasers.

Another change has been an increase in the number of requests for funding support from companies involved in development of enabling and environmental technologies. Unfortunately, to a large extent, TPC has been unable to respond to these requests and is limited to funding only a fraction of proposals from these firms.

For more information on this issue, see section 3.2.2 of the Detailed Findings Report.

2.3 The Role of TPC in the National Innovation Strategy

TPC and IRAP-TPC are the main government programs to support innovation projects carried out by private sector firms, and are positioned at the product development and pre-commercialization end of the innovation continuum. They are considered closest to market of all federal innovation programs. While this is the primary focus, occasionally, TPC funds earlier stage R&D projects. TPC is designed to be complementary to existing related programs. Most government agencies such as NSERC, NRC and other government programs and laboratories support earlier stage R&D and some smaller scale technology development. TPC supports relatively large, multi-year technology development projects in strategically important areas related to aerospace, environmental and enabling technologies. IRAP-TPC supports smaller scale (up to $3,000,000) pre-commercialization late stage technology development projects conducted by SMEs.

For more information on this issue, see section 3.2.3 of the Detailed Findings Report.

2.4 Conclusions

As a formative or mid-course evaluation, the mandate of the study is to address program design and delivery issues. The study is not aimed at making judgments on the fundamental public policy instrument that underpins the TPC program, or to make summative judgements on the success or contribution of this program to the country's innovation agenda. Nevertheless, there is no evidence to suggest that the government should not remain committed to the "public good" ideals intended with this partnership investment.

In an era of Federal Government deficit reduction, TPC appears to have been asked to cover a myriad of industrial policy objectives. The program was given a wide reaching mandate to create jobs and foster innovation in three very different sectors. At the same time, TPC's role in the innovation assistance process was limited by the focus on requirements for investment payback, as well as being subjected to intense international scrutiny. These constraints have led to several logical inconsistencies in its design, as compared to previous and other existing contribution programs:

  • The program focuses on repayment of investments, yet it must fund (increasingly since the WTO decision of 1998) high risk technology development and innovation - often in emerging areas of technology.
  • TPC must operate in a consistent, transparent and 'fiscally responsible' manner, leading to a lengthy multi-step process, yet it is by definition mandated to fund projects of high technology risk and market uncertainty - areas which require speed and flexibility.
  • The program was essentially developed in many respects as a 'son of DIPP'. The DIPP program was designed to serve the needs of the mature, cold-war aerospace and defence industry of the 1970s and 1980s. The evidence suggests that this approach is likely not consistent with a program being asked to assist sectors (even that of the modern aerospace industry) facing a completely different market situation.

In conclusion, TPC has, in theory and rhetoric, been established to serve several publicly stated industrial innovation goals and sectors, while subjecting itself to the constraints of economic development assistance in the modern era. In reality it would appear that the 'one-size fits all' approach that has been taken up to now may not be up to the task.

It is clear that the objectives at inception of this program were significantly changed to adhere to the "pre-commercialized" dictates of the WTO decision of 1998 and to address the emerging economy of enabling and environmental technologies outside of the aerospace and defence sector. It is also clear that the program design and delivery parameters were never re-aligned to the new realities and objectives of an altered TPC program. These include program criteria, risk assessment, performance and outcome expectations, repayment plans and approval processes.

In order to fully exploit the relevance, effectiveness, transparency and ultimately the success of TPC, a mid-stream course correction is recommended. Accordingly, Industry Canada should undertake a review of TPC to ensure its suitability to the needs of the modern aerospace, enabling and environmental sectors. Such a review should encompass the domains of policy, governance and delivery processes. It is not the recommendation of this study that a "moratorium" be placed on this program but rather that Industry Canada map out a process to re-align this program in accordance with the recommendations made throughout this report.

2.5 Rationale / Relevance Recommendation

Recommendation 1: In order to improve the relevance and effectiveness of Canada'sinnovation assistance, Industry Canada should undertake a policy review of TPC withregard to its relevance and suitability to the needs of the modern aerospace, enablingtechnologies and environmental technologies sectors. Options for consideration mayinclude:

  • the establishment of clearly separate program elements under a TPC umbrella;
  • the development of completely separate programs; or,
  • other structures which will preserve this important element of innovation assistance.

3.0 Program Governance

The file review, interviews and survey components of this formative evaluation study produced strong evidence on this issue category.

3.1 Impact of SOA Status and Possible Changes

The terms and conditions of the formation of TPC granted the program limited SOA status. TPC has a separate budget, a separate management structure, and authority for a 20% carry forward / back year to year transfer of funds, as well as the ability to recycle repayments into the program on an annual basis. However, the 3% limitation on resources available for program management and administration is also part of the SOA agreement. In addition, TPC still has to take almost all projects delivered directly by the program to the Industry Canada PSB for approval. TPC also has to seek approval for higher levels of funding from the Industry Canada ADM or Minister or Cabinet, depending on the level requested.

For more information on this issue, see sections 5.2.7, 5.2.8 and 5.2.9 of the Detailed Findings Report.

3.2 Partnerships with Other Program Delivery Organizations

The most significant partnership is with NRC's IRAP. TPC recognized early on that it was designed to deliver a relatively small number of large projects, and sought out a partner to deliver relatively small projects to SMEs. In 1998, TPC entered into an agreement with IRAP to deliver projects valued at up to $1,500,000 to SMEs. In April, 2003, this limit was raised to $3,000,000. Since the agreement began, IRAP has provided funding to over 300 projects, about twice as many as TPC, with an average contribution of about $385,000. Results from the survey of IRAP-TPC clients are very positive and show that IRAP is an effective partner in delivering the TPC program. TPC has also entered into formal agreements with the NRCan Climate Change Secretariat Climate Change Action Fund (CCAF) and the Technologies Early Action Measures (TEAM) to co-fund environmental technologies projects associated with sustainable development and reduction of greenhouse gases. The CCAF and TEAM funding is in addition to TPC's contribution to make projects in the environmental sector more commercially viable.

For more information on this issue, see section 5.2.4 of the Detailed Findings Report.

3.3 Responsibility for Management and Delivery of TPC

The responsibilities for management and delivery of TPC have evolved somewhat from the original design. While there are detailed policies and procedures to govern program delivery and the project selection and approval processes, the original design for high level program management and overview has not been followed. Specifically, the original plan was for TPC to be market driven and results oriented, with advice as to overall direction to be provided by a private sector based Advisory Board. This Board was to meet at least twice yearly. However, the Board has not met at all in almost two years, and it is unclear whether a Board still exists. The lack of private sector advice and influence provided through an Advisory Board exposes TPC to criticism about lack of stakeholder involvement of the nature provided to other innovation funding programs like IRAP, Canarie and PRECARN.

For more information on this issue, see section 5.2.10 of the Detailed Findings Report.

3.4 Conclusions

While TPC is identified as a Separate Operating Agency, it has, at present few of the flexibilities, such as increased spending authority, normally associated with SOA status. Evidence suggests that TPC's partnerships with NRC's IRAP and NRCan's CCAF and TEAM have improved program relevance and success. Also, the TPC Advisory Board, originally identified as an important source of private sector advice and involvement, has not met in several years.

3.5 Program Governance Recommendations

These recommendations are irrespective of the conclusion of the policy review proposed in recommendation 1.

Recommendation 2: TPC should determine what structural flexibilities would assist in delivering the program in a more effective and efficient manner, and seek approval for those changes. Options for consideration may include:

  • increasing TPC' s independence from Industry Canada; and /or,
  • seeking new opportunities for further delivery and co-funding partnerships.

Recommendation 3: TPC should reinstitute the TPC Advisory Board as soon as possible and involve the Advisory Board in the strategic oversight and management of the program.

4.0 Program Design and Delivery

4.1 Operational Design and Delivery

Satisfaction of Firms with Program Delivery

TPC funded clients surveyed were relatively satisfied with the advice and assistance they received from staff, but much less satisfied with the project selection process, speed of decision making, and reporting requirements. There were also some problems with the transfer of responsibility from TPC to the Industry Canada Program Policy and Management Branch (PPM) for the repayment phase, following completion of the work phase. Where comparisons were possible, levels of satisfaction of TPC clients were lower than for those of other federal funding programs. IRAP-TPC clients surveyed were generally more satisfied than TPC clients with all phases of design and delivery.

Consistency of Funding Decisions with Program Terms and Conditions

As stated previously, evidence suggests that funding decisions are made primarily on the basis of alignment with government objectives, innovativeness of the project and technical and financial feasibility of completing the work phase. Less emphasis has been placed on the likelihood of commercial success.

Beneficiaries of TPC

The industrial sector benefiting most from TPC is the aerospace and defence sector, which was the target of DIPP funding and continues to receive the largest fraction of TPC funding.

The characteristics of the individual firms benefiting from TPC are largely determined by the characteristics of the three sectors supported by the program. TPC supports relatively large, multi-million dollar projects in the aerospace and defence sector, and enabling (IT and biotechnology, advanced manufacturing and materials) and environmental technologies.

In the seven years from program inception to March 31, 2003, TPC's aerospace and defence sector funded 78 projects with 50 firms, an average of 11 per year. Participants were mostly firms involved in communications and electronic systems (21), aircraft component manufacturing (13) and aircraft engine manufacture and maintenance (4). Firms from the latter two groups were most likely to have multiple projects.

Similarly in the enabling technologies area, there were a total of 39 firms with 42 funded projects. Firms from the telecommunications and information technologies sector had 22 projects, an average of 3 per year. Firms in the biotechnology sector, primarily pharmaceutical firms, had 14 projects. There were very few firms with more than one project.

Unlike the other two funding components, the environmental technologies area funded projects with firms from a wide range of industrial sectors. Over the seven years, a total of 25 firms received funding for 31 projects. The largest number of firms is involved in alternative energy production and systems (8), with information technology and control systems, water purification and recycling, and aircraft and other engine manufacturing and maintenance being also represented with three projects each. Other funded firms are from the aquaculture, automotive, oil and gas production, and chemical products sectors.

Efficient and Effective Delivery of TPC and Suggested Changes

The evaluation identified a number of items that affect the efficient and effective delivery of TPC. The main points are presented.

  • Project Approval Processes

    TPC processes associated with project selection and approval processes and the development of the contract are of major concern. These processes are considered, by funding applicants and staff, to be very long and bureaucratic, with numerous steps in the approval process. In the past, first time applicants have been often confused about what was expected, and have made ineligible or incomplete applications. This results in inefficiency and use of scarce resources. For those proposals that are eventually accepted, the time from first application to receipt of funding is typically more than a year. In fact, TPC is not in control of the review and approval process, which involves Industry Canada Sector Branches and other government departments (OGD) for technical review, Justice Canada and Public Works and Government Services Canada (PWGSC) for review of contracts. All applications for TPC funding over $500,000, which is the vast majority, must go to the Industry Canada Program Services Board for approval. By contrast, for the relatively small IRAP-TPC projects, the review and approval process is entirely under IRAP program management.

  • Cash Flow and Resource Management

    In each of the first five years of operation, TPC did not fully utilize its allocated level of resources. To some extent, this was the product of the large size of TPC projects, the need to fund projects over their full workphase and the uncertainty in negotiating contracts. TPC has not lapsed most of these funds due to the 20% carry forward / back negotiated with Treasury Board as part of the Separate Operating Agency agreement. The situation has improved somewhat, as TPC has learned from experience. However, as of the 2002-03 fiscal year, TPC still has over $100,000,000 in funds carried forward.

  • TPC's Performance Measurement, Management and Reporting System

    Overall, it appears that TPC management's collection and use of performance information for program management is at an early stage of development. The SOA Framework that was part of the formation of TPC identified a number of program design and delivery principles and performance indicators. These indicators focused primarily on the financial and economic characteristics of the program, such as leverage, repayments, rate of return, and creation of jobs. There was no measurement of the impacts of the program related to improved technical capability and increased competitiveness of the firm. In 1998, TPC gave notice of plans to broaden the types of performance information collected, but has not yet done so. The 3% limitation on program management and administration is identified as a reason for lack of progress.

    In the fall of 2002, TPC provided the first report to management about the performance indicators and what they showed about the program. This report was identified as a prototype, and it is not clear what use TPC management has made of this information to date. The Annual Report, which is identified as the main mechanism by which TPC demonstrates its accountability, provides information on about half of the original performance indicators. TPC does not produce a separate performance report.

  • Communications and Performance Reporting

    Evidence from the study shows that TPC communications have focused on project forecasts related to commercial success, long term jobs and repayments. This focus is unrealistic and leads to unfortunate expectations among stakeholders and the public about TPC. As discussed previously, little attention has been paid to the high level of technical success that is being achieved by TPC and IRAP-TPC funded projects. These projects have been shown to lead to improved technical capability and firm competitiveness. Unfortunately, the high risk, breakthrough nature of many TPC projects, means that a considerable number of them, like all R&D, are unlikely to achieve commercial success.

  • Resources for Program Management and Administration

    On the surface, keeping the level of resources and staffing available for program management and administration to 3% and 50 FTEs respectively seems to be a good method to ensure efficient operation. In fact, the study found that this level of resources is far below what is needed to operate the program in an effective and efficient manner. For example, limiting FTEs to 50 has resulted in the hiring of a number of consultants to accomplish the needed work, at a higher cost than hiring program staff. The study found numerous deficiencies in program delivery that can be linked to too few resources and staff. For comparison purposes, other innovation related programs such as PRECARN and Canarie allow 10% of program funding for management and administration.

For more information on this issue, see sections 5.2.5, 5.2.6 and 5.2.10 of the Detailed Findings Report.

4.2 Conclusions

The TPC assistance process is complex in terms of both its number of steps and the number of groups involved. Some of this complexity is likely unavoidable due to the numerous constraints placed on modern industrial assistance of this magnitude. However, when combined with a low investment in program administration, the effect has been to unfairly handicap the speed, flexibility and 'learning ability' of this program.

4.3 Program Design and Delivery Recommendations

In addition to supporting recommendation 1, the evidence and conclusions presented in this section lead to a series of new recommendations. However, these recommendations are irrespective of the conclusion of the policy review proposed in recommendation 1.

Recommendation 4: TPC should review and revise the project selection and approval process, with a view to significantly reducing the length of time for decision-making and making funds available. Options for consideration may include:

  • negotiating higher levels of spending authority; and /or,
  • developing different levels of due diligence and approval processes for differing levels of funding and risk.

Recommendation 5: In order to reduce or eliminate the high level of funds carried forward, TPC should continue to develop more aggressive cash management practices. Options for consideration may include:

  • a strategy of over commitment on the number of projects and resources being considered for funding based on experience; and / or,
  • reconsidering the present practice of committing funds at the project selection stage, which is a major factor in under-utilization of available funding.

Recommendation 6: TPC should develop a more appropriate performance measurement and reporting approach, that captures operational performance information, as well as a broader range of early, intermediate and longer term technological, economic and societal benefits of projects.

Recommendation 7: Following from the above, TPC should communicate a more complete view of this innovation focused program and its technical and commercial outcomes to stakeholders and the public. This would include the high level of technical success being achieved by projects, as well as the lower level of commercial success achieved to date. TPC should communicate the high risk nature of projects, the fact that not all will be successful and that overall repayments will not likely meet the initial forecasts identified in proposals.

Recommendation 8: In order to achieve the process improvements noted above, TPC should increase the current levels of funding and staffing for policy, planning and program management to a more appropriate level, in line with requirements. The new levels should be determined by a needs analysis and comparison to other related programs.

5.0 Progress in Delivering the Program and Achieving Success

The formative nature of this evaluation means that study evidence is comparatively strong for early outcomes such as incrementality, participant reactions and perceptions. The evidence is more limited in its contribution to understanding intermediate and longer term economic impacts.

5.1 Level of Understanding of TPC

The level of awareness of TPC and understanding of its objectives is highly variable among the target client groups. Most firms with funded projects are generally aware of program objectives (i.e., help firms perform R&D, creation of employment and help Canada compete internationally) as a result of the detailed application and selection process. Firms in the aerospace sector are generally very familiar with TPC, probably due to their long association with the program and its predecessor DIPP and the efforts of the Aerospace Industries Association of Canada to keep them up to date on the program. Firms in other sectors served by the enabling technologies and environmental technologies elements are less aware, as TPC is only one of a number of government programs that support R&D initiatives in these sectors. Firms in these highly innovative, emerging sectors also make use of a number of other government programs and private sector options for funding R&D, such as stock offerings, and venture capital. As mentioned previously, TPC funding is a much smaller factor in these sectors than in aerospace, where it is the dominant source of R&D support. SMEs in particular are often not aware of the program.

TPC managers and staff identified a major objective of TPC not mentioned by clients, that relates to the overall program rationale. They noted that the program objective was to fund projects in private sector firms that help achieve government public policy objectives such as sustainable development, reduction in use of nonrenewable resources, and connectedness.

For more information on this issue, see section 4.2.1 of the Detailed Findings Report

5.2 Extent to Which TPC is Achieving Program Objectives

TPC program objectives include both technological and economic benefits. Funded projects are expected to achieve technological benefits first, during the work phase of projects, with some economic benefits occurring during the work phase, and the majority occurring in the years following, as the technology and improved competitiveness of the firm resulting from the project manifest themselves in firm growth.

In order for either technological or economical benefits to occur, TPC funding must be incremental to the project. That is, government funding must be required for the project to proceed with the appropriate scope and timing. The following table provides a summary of survey results regarding incrementality. Full project incrementality means that respondents reported that the project would not have proceeded at all, and no incrementality means that the project would not have been affected adversely if funding had not been provided.

Project Incrementality
Program Full High Low No
TPC 24% 61% 15% 0%
IRAP-TPC 31% 58% 11% 0%

As can be seen, over 85% of projects had full or high incrementality, and all projects were incremental to some extent. Comparison to similar data for other innovation related programs shows that TPC and IRAP-TPC have lower levels of full incrementality than some earlier programs. On the other hand, TPC and IRAP-TPC cases always showed at least partial incrementality. This is an improvement over the findings for earlier innovation programs (e.g., IRDP1-Innovation showed roughly 10% 'no' incrementality).

There is strong evidence that firms with TPC and IRAP-TPC funded projects are receiving technological benefits. At least 90% of firms surveyed (both work phase completed and in progress projects) reported that new or improved technologies had been

1 IRDP - Industrial Regional Development Program.

developed and that the firms technological capability had improved because of the project. Over two thirds of firms surveyed reported that their competitive position had improved.

Most projects have not yet completed the work phase, and the longer term economic benefits resulting from improved technology and competitiveness are not yet expected to have occurred for these firms. Most projects that have completed the work phase, have done so only recently, whereas most economic benefits are expected to occur over the longer term. It is encouraging that over 40% of firms surveyed with work phase completed projects reported increased revenues and over a quarter reported increased profitability. This is in spite of recent problems in the equity markets for enabling technologies, the aerospace sector and the general economic downturn of the past few years.

Survey respondents also noted that funded projects have contributed to the maintenance and creation of highly skilled jobs and job growth in firms. Over 90% of firms surveyed reported that the funded projects had led to the maintenance of highly skilled jobs and over 80% reported that new highly skilled jobs had been created. In addition, over 40% of firms also reported long term job growth. Firms with TPC funded projects reported an average increase in the number of employees in highly skilled jobs of 29%, and of total employment of 17%, from an average of 439 to 515 employees. For firms with IRAPTPC funding, the average increase in number of highly skilled jobs was 13%, and of total employment was 48%, from an average of 16 to 24 employees. While the full amount of these increases cannot be directly attributed to TPC and IRAP-TPC project funding, the evidence suggests that the funding has clearly contributed to the increase.

The study also showed that TPC is supporting the achievement of government public policy objectives, such as sustainable development and connectedness.

For more information on this issue see section 4.2.2 of the Detailed Findings Report.

5.3 Extent to Which TPC Management is Achieving Operational Objectives

TPC has several specific operational objectives related to costs of program management and allocation of funds between the three sectors (aerospace, enabling and environmental technologies).

In 1996, when the TPC program began, a target was established for program management and delivery costs to be kept to 3% of program budget. At the present time, the annual TPC A base budget is about $300,000,000, which results in a target of $9,000,000 for administrative costs. Until recently, TPC has kept costs below 3%. However, in the past two years, administrative costs have climbed to 3.4% of annual budget in 2001-02, and about 3.8% in 2002-03.

Similarly, there was an initial agreement to keep TPC FTEs at 50. This target was also met until two years ago, when the staff levels began climbing, to a forecast level of 70 FTEs for 2002-03.

Evidence collected for the study suggests that these targets are unusually low for federal programs. Other government innovation programs, such as PRECARN and Canarie, allocate 10% of funding to program management and administration.

TPC also has a target of 2/3 of funding assigned to the aerospace and defence sector and the other 1/3 divided between the enabling and environmental technologies sectors. The allocation of funding has varied widely from year to year, depending on external factors and the economic situation. Since 2001, the amount of funding going to the aerospace sector has declined, and the 2002-2003 TPC business plan identified the plans of program management to seek a reduction in the fraction of funding going to aerospace, with a corresponding increase to the other elements. The cumulative percentage of total TPC project funding to the aerospace sector is now 56%, compared to the 66% target.

For more information on this issue see section 4.2.3 of the Detailed Findings Report.

5.4 Extent to Which Projects are Achieving Objectives Identified in Proposals

Broadly speaking, a large proportion of projects have achieved and are achieving project level technical objectives. The previous discussion of the achievement of program objectives included considerable information about the achievement of project objectives, and support this conclusion. Additional information is provided from the client survey, in which over 80% reported that their project was very or completely technically successful (8 or more on a 10 point scale). In a number of cases, the work phase of the project takes somewhat longer to complete than planned, due to unexpected problems associated with risky, uncertain R&D projects.

As previously reported, there has been less success in achieving commercial objectives. Just under 60% of clients surveyed reported that their project was very or completely commercially successful. To date, few, if any, completed projects have achieved the post project commercial success forecast in terms of sales, increased revenues and jobs. Reasons given are linked to the technological and economic downturn of the past few years, which occurred just as many projects were being completed. Another factor is that these forecasts could be considered optimistic, in that they presume that events will unfold in a very positive manner. These presumptions have not been met in reality. A large number feels that many of the forecast economic impacts will still come.

For more information on this issue see section 4.2.4 of the Detailed Findings Report.

5.5 Extent to Which Project Impacts Have Extended Beyond the Funded Firm

While a detailed economic analysis of regional and community impacts was beyond the scope to this project, it appears that the growth of firms in terms of number of employees and revenues has spill over effects into the community. Some limited evidence to address this question came from several sources. Survey respondents reported that suppliers, subcontractors and consultants benefited from funded projects. Also, the most recent TPC Annual Information Update (AIU), estimated that subcontractors provided about 10% of the 10,872 actual and projected jobs for all TPC funded projects. IRAP-TPC data was not included.

For more information on this issue see section 4.2.5 of the Detailed Findings Report.

5.6 Long Term Job Creation Achieved

This study was conducted relatively early in the life of this program. Since the program funds predominantly multi-year projects, most were not completed at the time of this study. However, there is some evidence to address the question of job creation for TPC funded projects available from the AIU that funded firms provide. For the 54 projects with the work phase completed as of December, 2001, the actual number of jobs created and maintained during the work phase was 4,460, compared to the original forecast level of 3,805, an increase of just under 20%. During the corresponding benefits phase, the actual number of jobs was 1,122, about 30% less than the original forecast level of 1,705. These numbers are consistent with earlier discussions about the level of technical success of projects and difficulties with commercialization.

For more information on this issue see section 4.2.6 of the Detailed Findings Report.

5.7 Conclusions

Within the limits of the methods used for this formative evaluation, it appears that TPC and IRAP-TPC achieved reasonable project incrementality, significant technical and early job creation impacts. Commercial, and therefore repayment, results have been limited to date. This can partially be attributed to the long time horizons required for many projects to realize commercial results and partially due to economic circumstances (i.e., a downturn in the markets for the sectors involved).

Case studies and survey responses suggested that there were significant 'public good' types of outcomes for a number of TPC investments. Unfortunately, a narrow public communications focus on repayment levels, combined with a lack of program investment in intermediate outcomes measurement, have worked against TPC in terms of telling its full performance story.

While TPC funding is important to every firm supported, it appears to play a critical role in support to the aerospace sector, due to the lack of other support programs. Since 1996, TPC has supported a large fraction of the major R&D projects and, on average, TPC contributions have been about 15 - 20% of the annual R&D expenditures in the aerospace and defence sector. In the other sectors, there are calls for an increased share of TPC funding, as the demand for TPC support is much greater than can be met by available funds. However, there are other government programs and private sector sources to provide funding support, and TPC plays a much less dominant role to the enabling and environmental sectors.

In terms of operational objectives, TPC has recently exceeded the targets of 3% 'overhead costs' and 50 Full Time Equivalent (FTE) program staff levels, that were found to be unrealistic for a program of this magnitude and complexity.

5.8 Success Recommendations

Recommendations to improve program success are included in the previous recommendations in Sections 2, 3 and 4.

6.0 Key Lessons Learned

6.1 What Lessons Have Been Learned

The study found that a TPC program with a single set of selection criteria and policies and procedures was not able to meet the differing needs of the three sectors supported (aerospace and defence, enabling and environmental technologies). These sectors are at much different levels of maturity, market acceptance and types of projects required.

TPC staff recognize that the program has in place a detailed due diligence procedure and approval process that is extremely time consuming and causes dissatisfaction among clients. As well, staff and management recognize that the restrictions on resources available for program management and delivery have affected timeliness of project approval, management of funded projects and overall program delivery.

Another major lesson learned is that the narrow focus on job creation, commercial success and repayability have misrepresented the broader, overall objectives of the program that, especially since the change of program direction in 1999 following the WTO ruling, include improving technological capability and competitiveness of the firm as well as commercial success. In addition, the high risk nature of many TPC projects and the possibility of commercial failure, even though the project may be technically successful has not been properly considered in developing performance indicators and communicating the performance of the program.

For more information on this issue see section 6.2.1 of the Detailed Findings Report.

6.2 Factors That Have Facilitated / Impeded the Effective Delivery of TPC

Many of the impeding factors have already been discussed in the previous sections. To a considerable extent, the original design of TPC and its implementation in the policies, procedures and delivery of the program have had some unintended as well as intended consequences.

On the positive side, the familiarity of the aerospace and defence sectors with DIPP, the predecessor of TPC, facilitated early access to TPC by firms from this sector. Support from the Industry Canada Automobile and Aerospace Sector also helped firms from these sectors to make well written proposals that achieved a high approval rate. Also, the decision to have IRAP act as a co-delivery partner to deliver TPC to SMEs for smaller projects, up to $3,000,000 was a major factor in making the program widely accessible to this target group. Similarly, the linkages with the NRCan CCAF and TEAM initiatives helped facilitate access to funding for firms involved in technology to reduce greenhouse gas emissions and related sustainable development initiatives.

On the negative side, as noted in sections 3 and 4, for TPC funded projects, the extensive due diligence and multiple approval process resulted in most approved projects taking over a year for the selection, approval and contract process. The length of time and effort to complete the process is seen as a major problem by firms, and consumes valuable program resources as well. These observations do not apply to TPC-IRAP projects, which are smaller and more quickly dealt with. Another factor affecting TPC program delivery is the high level of demand for funding of environmental and enabling technologies projects. In the past, much of this demand has been to fund projects outside the objectives of the program. This indicates that many first time applicants are unfamiliar with the specific selection criteria, and waste time and efforts in applying. These inquiries also result in inefficient use of program resources in reviewing a large number of nonqualifying proposals.

For more information on this issue see section 6.2.2 of the Detailed Findings Report.

6.3 Factors That Have Facilitated / Impeded the Achievement of TPC Objectives

Much of the discussion in previous sections apply to internal factors that have affected the achievement of program objectives. The extensive due diligence and project approval process previously mentioned have delayed project start up and, in so doing, have reduced the ability of funded firms to move quickly to build technical capability and develop technologies to meet an identified window of opportunity.

However, the single largest factor to date impeding the achievement of TPC objectives related to long term jobs and wealth creation is external to the program and outside the ability of the program or the government to control. This is the technological and economic environment of the past few years. The dot.com crash, 9/11 and the economic downturn have all affected the ability of firms with completed projects to make sales and achieve commercial success. In addition, the severe downturn in the aerospace sector has negatively affected the ability of firms in this sector to fund ongoing projects, resulting in project delays.

For more information on this issue see section 6.2.3 of the Detailed Findings Report.

6.4 Conclusions

The use of a single program, essentially based on an assistance model from a previous era, to assist three critically different sectors in high risk innovation appears to have hampered the effectiveness of TPC.

6.5 Lessons Learned Recommendations

Recommendations related to the key lessons learned presented in this section are imbedded in the other recommendations made previously.

 Report a problem
Date modified: