Language selection

Search

Industrial and Technological Benefits Policy Evaluation Report

From: Innovation, Science and Economic Development Canada

May 2020

This publication is available online at http://www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03917.html.

To obtain a copy of this publication, or to receive it in an alternate format (Braille, large print, etc.), please fill out the Publication Request Form at www.ic.gc.ca/publication-request or contact:

ISED Citizen Services Centre
Innovation, Science and Economic Development Canada
C.D. Howe Building
235 Queen Street
Ottawa, ON K1A 0H5
Canada

Telephone (toll-free in Canada): 1-800-328-6189
Telephone (international): 613-954-5031
TTY (for hearing impaired): 1-866-694-8389
Business hours: 8: 30 a.m. to 5: 00 p.m. (Eastern Time)
Email: ISED@Canada.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from the Department of Industry, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that the Department of Industry is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, or as having been made in affiliation with, or with the endorsement of, the Department of Industry.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the ISED Citizen Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Industry, 2020.

Cat. No. xxx
ISBN xxx

Aussi offert en français sous le titre Évaluation de La politique des Retombées industrielles et technologique.

List of abbreviations and acronyms

DND
Department of National Defence
GDP
Gross Domestic Product
IRB
Industrial and Regional Benefits (Policy)
ISED
Innovation, Science and Economic Development Canada
ITB
Industrial and Technological Benefits (Policy)
PSPC
Public Services and Procurement Canada
R&D
Research and Development
RFP 
Request for Proposal
SME
Small- and Medium-Sized Enterprise
VP
Value Proposition

List of figures

List of tables


Executive summary

This report presents the results of an evaluation of the Industrial and Technological Benefits (ITB) Policy. The program is administered by Innovation, Science and Economic Development Canada (ISED) and was announced as part of the Defence and Procurement Strategy in February 2014 to replace the Industrial and Regional Benefits (IRB) Policy, which had been in place since 1986. The ITB Policy helps ensure that defence and security procurements going forward are better leveraged to create jobs and economic growth in Canada.

Program overview

The ITB Policy requires that companies awarded defence procurement contracts undertake business activities in Canada, equal to the value of the contract (as under the IRB Policy). The ITB Policy also includes a requirement for companies to provide specific, targeted proposals (known as the Value Proposition or VP) for how they will contribute to economic growth in Canada through targeted investments. In addition, some changes to the banking policy, the goal of which is to encourage bidders to engage Canadian companies early and invest in Canada well ahead of securing a defence contract, were made with the introduction of the ITB Policy.

The ITB Policy has the following five core objectives:

Between February 1, 2014 and January 31, 2019, 32 contracts valued at almost $6.2 billion were initiated under the ITB Policy.

Evaluation purpose and methodology

The objectives of this evaluation were to examine the relevance, performance and efficiency of the ITB Policy in accordance with the Treasury Board Policy on Results and address issues identified by ITB Branch management (e.g., engagement with industry prior to contracting). The evaluation covered the period from February 1, 2014 to January 31, 2019. The evaluation employed multiple data collection methods: a document review, interviews, administrative data review, survey of industry, input/output (I/O) analysis, and case studies.

Findings

Relevance

The ITB Policy, including expenditure requirements and VPs, is aligned with ISED priorities related to the competitiveness of Canadian firms, economic growth, job creation, innovation and R&D. Further, the ITB Policy is aligned with the needs of the Canadian defence industry by supporting the growth of prime contractors and suppliers in Canada.

Performance

Roughly one-third of ITB Policy recipients have developed new business relationships with prime contractors as a result of ITB Policy-related work – about half have become qualified suppliers, while a majority said that ITB Policy contracts allow them to sell to new clients.

The ITB Policy is contributing to the development of a Canadian supplier base, as it enforces sourcing from Canada-based suppliers. Most recipients report that the ITB Policy contracts improved their work processes and contributed to their R&D and innovation capacity. According to data analyses, the ITB Policy was estimated to have contributed to the maintenance/creation of as many as 6,450 jobs per year in the private sector between 2014-15 and 2018-19. The impact on Canada's Gross Domestic Product (GDP) was estimated to have been as high as $3.3 billion for this period.

The ITB Policy has led to improvements in the engagement process with industry. Early engagement has allowed industry to provide views on the development of the VP framework before the Request for Proposals are issued, and engagement is generally viewed as being timely.

Further, Value Propositions increase benefits for Canada in many areas, including R&D, innovation, skills development, and economic development. Two-thirds of prime contractors interviewed were satisfied with the Value Proposition approach. However, one-third expressed concerns about the lack of clarity of the VP statements, their complexity and limited flexibility of the approach.

In terms of the banking mechanism, it is generally being utilized as planned and is encouraging prime contractors to make investments in Canada.

Finally, the ITB Branch research and analytics function is used for decision-making and reporting purposes and is deemed useful within ISED.

Efficiency

ITB Branch operations were efficient over the 2014-15 to 2018-19 period, with no major issues or gaps identified with respect to the efficiency of operations. However, there are opportunities to improve the consistency of the crediting process (i.e., the process whereby contractors submit reports to get their expenses credited as per their VPs and obligations).

Recommendations

The evaluation findings led to the recommendations noted below.

Recommendation 1: Value Proposition Approach

ISED should continue to work with contractors to ensure that the Value Proposition approach on a procurement-by-procurement basis is as flexible as possible for prime contractors to fulfill requirements, while continuing to meet the policy objectives.

Recommendation 2: Crediting Process

ISED should enhance internal guidance related to the application of crediting guidelines for contractor expenses to improve the consistency of the crediting process.

Recommendation 3: Program Understanding

ISED should continue industry engagement and outreach activities to enhance stakeholder understanding of the ITB Policy, particularly the Value Propositions.

1.0 Introduction

1.1 Report overview

This report presents the results of an evaluation of the Industrial and Technological Benefits (ITB) Policy. The program is administered by Innovation, Science and Economic Development Canada (ISED) and was announced in February 2014 to replace the Industrial and Regional Benefits (IRB) Policy, which had been in place since 1986. The ITB Policy helps ensure that defence and major Canadian Coast Guard procurements going forward are better leveraged to create jobs and economic growth in Canada.

The purpose of this evaluation is to assess the relevance, performance, and efficiency of the ITB Policy. The report is organized into four sections:

At a glance:

1.2 Context

Federal procurement of goods and services has the potential to generate significant industrial benefits for Canadian firms. Since the mid-1970s, the Government of Canada has been pursuing the use of industrial benefits as part of federal procurement contracts that fall outside of international trade agreements or for which a National Security Exception is invoked. Under this approach, firms bidding on government defence and major Canadian Coast Guard contracts are evaluated on the basis of the economic benefit of their proposal to Canada, as well as price and technical requirements. Since 1986, 144 projects have leveraged a total of $43.8 billion in economic obligationsFootnote 1.

Canada's Defence Procurement Strategy, announced in February 2014, transformed Canada's IRB Policy into the ITB Policy. As with the IRB Policy, the ITB Policy requires that companies awarded defence procurement contracts undertake business activities in Canada, equal to the value of the contract. However, the ITB Policy also includes a requirement for companies to provide specific proposals (known as the Value Proposition or VP) for how they will contribute to economic growth in Canada through targeted investmentsFootnote 2.

1.3 Objectives and description

The ITB Policy is part of the procurement process used to purchase new equipment and services for the Canadian Armed Forces and Canadian Coast Guard. Companies that bid on a defence contract submit a proposal to the Government of Canada, which includes a technical submission which is evaluated by the Department of National Defence (DND), a financial submission evaluated by Public Services and Procurement Canada (PSPC), and a VP evaluated by ISED. The proposal with the highest combined score wins the contract.

The ITB Policy requires the winning bidder to invest an amount equal to the value of their contracts back into the Canadian economy. This is accomplished by forming partnerships and awarding work to firms in Canada. The winning bidder could also invest in new technologies and research with post-secondary institutions, as well as with Canadian companies both inside and outside the defence industry, helping to grow the Canadian economy and make Canada more innovative.

The ITB Policy has the following five core objectives:

A core element of the ITB Policy approach is a rated and weighted VP that requires bidders to put forward activities that support the five core objectives of the ITB Policy and provide economic benefits to Canada (weightings for each objective can vary on a procurement-by-procurement basis). Under the IRB Policy, bids were evaluated on the basis of price and technical merit only. Under the ITB Policy, VPs are also factored into the evaluation of bids. Bidders receive points for VPs that support the five core objectives of the ITB Policy. After a contract is awarded, the contractor is required to start fulfilling its commitments (i.e., business activities directly related to the equipment or service being procured by Canada) and to identify further business activities in Canada, as may be required to meet the overall ITB Policy obligation (i.e., 100% of the contract value).

The change from the IRB Policy to the ITB Policy included a new emphasis on federal government engagement with prime contractors and Canadian industry as a whole, before contracts are awarded, which gives industry an opportunity to provide input into the development of a Request for Proposal (RFP), including the VP framework. The change to the ITB Policy also included enhanced monitoring and reporting on the economic impacts of the ITB Policy after contract award (e.g., updates on the level of Canadian SME involvement).

Banking Policy

The banking policy aspect of the ITB Policy (created in 2009 under the IRB Policy) allows companies to undertake business activities in Canada prior to a procurement and "bank" these investments for credit against their IRB/ITB obligations later, when they secure a defence contract. The goal of the banking policy is to encourage bidders to engage Canadian companies early and invest in Canada well ahead of securing a defence contract.

Five changesFootnote 4 to the banking policy were made to the ITB Policy in 2015 to significantly enhance the attractiveness of investing in Canada in advance of a contractual requirement and motivate prime contractors to extend successful business relationships with Canadian suppliers above and beyond their contractual requirements:

As before, banked transactions may be used to fulfill up to 50% of a contractor's overall IRB/ITB obligation, thereby ensuring that at least one-half of a company's obligation comprises new investment commitments. For example, if a company has invested $20 million into the Canadian economy prior to winning a $100 million ITB Policy contract, the company can withdraw these investments from their ITB bank account and apply them against the $100 million obligation.

The banking policy also allows companies that achieve more than their contractual obligations to bank the overachievement for future credit. This encourages prime contractors to fulfil their obligations ahead of schedule and continue investing in Canadian suppliers with which they have successful relationships.

1.4 Delivery and governance

The ITB Branch is responsible and accountable for the administration and management of the ITB Policy. The ITB Branch established a set of processes, including controls and approvals, which are recorded in a Process Management Information System. Employees are required to use this system, which in turn has a series of tasks officers must complete for each procurement to which the ITB Policy applies. During the pre-contract stage and the verification activity stage, team leaders are responsible for ensuring officers are in compliance with the ITB processes while providing value-added input into ITB procurement documents (e.g., VP evaluation framework). Senior management has the responsibility of reviewing procurement documentation, providing comments, and approving procurement documents. In the Contract and Verification Activity stage, team leaders are required to review verification material and reports developed by officers and make any necessary changes before being approved by Directors.

ITB Branch management conducts regular reviews to ensure all processes are followed, correct information is being entered, and proper amounts are being credited in the ITB Process Management Information System. These processes include required consultations with the following organizations:

1.5 Resources

Between February 1, 2014 and January 31, 2019, 32 contracts valued at almost $6.2 billion were initiated under the ITB PolicyFootnote 6. These contracts were distributed among four sectors:

Projects under the ITB Policy's predecessor (IRB Policy) awarded since the launch of the Defence Procurement Strategy in 2014, valued at $6.1 billion, were excluded from this evaluation since they exclude some of the key elements of the ITB Policy being evaluated, notably the VPs.

Between 2014-15 and 2018-19, total ITB Branch expenditures more than doubled, from about $3.9 million to $8.1 million, with operations and maintenance expenditures almost triplingFootnote 7. The majority of the financial resources (60.6%) over this five-year period were dedicated towards staff salaries.

Table 1: ITB Branch Expenditures
Fiscal Year 2014-15 2015-16 2016-17 2017-18 2018-19 Total

Operations and Maintenance

$1,057,196

$2,723,138

$3,646,804

$3,703,863

$3,147,245

$14,278,246

FTEs

31

48

54

56

56

245

Salaries

$2,853,131

$4,257,189

$4,577,735

$5,331,207

$4,967,399

$21,986,661

Total

$3,910,327

$6,980,327

$8,224,539

$9,035,070

$8,114,644

$36,264,907

Source: ISED financial data

1.6 Logic model

The 2018 logic model for the ITB Policy is presented in Figure 1. A logic model is a visual representation that links a program's activities, outputs and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program, policy or initiative is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies.

Figure 1: ITB Policy Logic Model

Text version

The 2018 logic model for the ITB Policy is presented in Figure 1. A logic model is a visual representation that links a program's activities, outputs and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program, policy or initiative is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies.

The four activities described in Figure 1 are:

  1. Stakeholder Engagement: Includes outreach; engage PMO and RDAs; and website, phone, emails.
  2. Pre-RFP Activities: Includes support announcements; VP strategy development; industry engagement; and support governance decisions.
  3. RFP Activities: Includes VP documentation; VP evaluation; and contract award and announcement.
  4. Post-Contract and Verification Activities: Includes assess 2nd and 3rd tranche; and verify annual reports.

The four outputs described in Figure 1 are:

  1. Program communications and decisions on VP applicability.
  2. Development of VP strategy for RFP.
  3. VP scoring influences winner selection.
  4. Compliance monitoring – verification: planning, activities and reports.

The four immediate outcomes (expected to occur within 0-5 years) described in Figure 1 are:

  1. Increased awareness of ITB Policy; efficient ITB procurement process.
  2. Improved high quality of industrial proposals that focus on priority areas.
  3. Increased work and investments for Canadian companies related to procurement.
  4. Increased work and investments for Canadian companies on supplemental work.

The four intermediate outcomes (expected to occur within 5-15 years) described in Figure 1 are:

  1. Growth for defence sector.
  2. Growth of Canadian suppliers and SMEs.
  3. Enhanced R&D activities in Canada.
  4. Increased export potential for Canadian firms.

The long-term outcomes (expected to occur after 15 years) described in Figure 1 are the growth of Canadian industrial base in terms of technological capabilities, competitiveness and contribution to economy; and development of next generation technologies and services.

The strategic outcome described in Figure 1 is "Industrial Competitiveness and Capacity – Economic Outcomes from Procurement".

2.0 Methodology

This section provides information on the evaluation context, objectives, scope and approach, issues and questions, data collection methods, and limitations.

2.1 Evaluation context

The previous evaluation of the ITB Policy (formerly known as the IRB Policy) was completed in January 2015. The evaluation found a continued need for federal support for industrial development in Canada's defence and security sector. In terms of performance, the evaluation noted that prime contractors were fulfilling their IRB obligations, that IRB recipients had benefitted from IRB transactions to a varied extent, and that the IRB Policy had led to the growth of IRB recipients and the development of new business relationships. The evaluation recommended that the ITB Branch should:

2.2 Objectives

An evaluation of the ITB Policy is a Treasury Board submission requirement. The objectives of this evaluation were to examine the ITB Policy in accordance with the Treasury Board Policy on Results and address issues identified by ITB Branch management (e.g., engagement with industry prior to contracting) not covered under the logic model.

2.3 Scope and approach

The evaluation was conducted by Goss Gilroy Inc. and covered the period from February 1, 2014 to January 31, 2019. It examined the relevance, performance and efficiency of the ITB Policy, including the economic and social benefits of the Policy. In the assessment of performance, the evaluation focused only on progress made towards the achievement of the immediate outcomes identified in the logic model, as the ITB Policy has been in existence for only five years. The logic model suggests that intermediate outcomes require 5 to 15 years to be measureable.

Contracts for the IRB Policy that were active during the evaluation period are not covered by this evaluation, since they exclude some of the key elements of the ITB Policy being evaluated, notably the VPs.

2.4 Evaluation issues and questions

The evaluation addressed the following questions:

Relevance

  1. Is the ITB Policy, including the VP, aligned with ISED priorities?
  2. Is the ITB Policy in line with the needs of the Canadian defence industry, in particular the SMEs?

Performance

  1. To what extent is engagement with industry prior to contracting effective and timely?
  2. To what extent has the VP approach been implemented as planned?
    1. Are some VP components more beneficial than others?
  3. To what extent has the ITB Policy led to the development of new business relationships between prime contractors and recipients?
    1. How many recipients have then become qualified suppliers of prime contractors?
    2. How many recipients have then become primary (tier 1) or secondary (tier 2) contractors?
  4. What are the economic and social impacts of the ITB Policy?
    1. What are the direct economic benefits of the ITB Policy in terms of employment and economic growth?
    2. To what extent have recipients benefited from the ITB Policy, including VP? From supplemental business opportunities with other firms after the initial ITB Policy transactions?
    3. To what extent have prime contractors benefitted from the ITB Policy, including VP?
    4. To what extent have other organizations benefitted from the ITB Policy, including VP?
  5. To what extent does use of the banking mechanism lead to an impact on recipients?
  6. To what extent is the ITB Branch research and analytics function providing value to ISED and other departments?
    1. To what extent does ISED or the ITB Branch data support decision-making?

Efficiency

  1. To what extent has the implementation of the ITB Policy demonstrated efficiency? Are there alternatives to improve efficiency?

2.5 Data collection methods

Multiple lines of evidence were used to address the evaluation questions. The data collection methods included a document review, interviews, administrative data review, survey of industry, input/output (I/O) analysis, and case studies.

Document Review

A document review was conducted to gain a thorough understanding of the program and to provide insights into relevance and performance. The review included ISED program and foundational documents, ITB annual reports, evaluation reports, corporate plans and administrative data.

Interviews

The objective of the interviews was to gather in-depth information related to the relevance, performance, and efficiency questions. The interviews were semi-structured in nature to help collect qualitative information from a range of key stakeholders. Interviews were conducted either in-person or by telephone.

The identification of key stakeholders was confirmed with program management. In total, 32 interviews were conducted as part of the evaluation with the following three groups:

Administrative Data Review

The administrative data reviewed included ITB Branch financial tables and data on results against service standards. The data was summarized using tables and analyzed to assess program efficiency. Contract dollars data was also analyzed as part of the I/O analysis.

Survey of Industry

An online survey (based on a sample of 194 private sector firms and organizations that were part of an ITB Policy transaction) was conducted of all companies (including prime contractors) that were recipients on one or more ITB Policy transactions. The survey resulted in 45 completed surveys (and 14 partially completed surveys) for a response rate of 23% (30% response rate if including partially completed surveys). The response rate is in line with other comparable surveys.

The distribution of survey respondents was almost identical to the distribution of the original sample of 194 (e.g., 50% of the original sample was companies in the air sector versus 49% for the completed or partially completed surveys).

Input/Output (I/O) Analysis

The economic impact analysis undertaken for this evaluation utilized data from the results of the survey of industry undertaken as part of this evaluation. The data obtained was then analyzed using Statistics Canada's National Accounts and I/O tables. The analysis also includes an I/O analysis of the ITB Policy contracts (as a separate analysis). The I/O analysis was conducted by ISED economists within the ITB Branch. More detailed information on the I/O analysis and the economic impact of the ITB Policy is provided in Appendix B and Appendix C.

Case Studies

Five focused case studies were completed of specific ITB Policy contracts to get a more in-depth understanding of the ITB Policy. The cases were selected in consultation with the ITB Branch and covered contracts in three domains (air, marine, and land). The respondents from the five cases and the 14 prime contractors interviewed as part of the interviews covered more than half of the prime contractors working under ITB Policy contracts.

After the case selection, the ITB Branch provided file information related to each of the cases, including contract amounts, project descriptions and lists of sub-contractors. The main representatives of the private sector firms were contacted for an interview by phone. Other interviews were conducted based on their recommendations, including some with the Canadian sub-contractors and recipients of the VPs. Up to three interviews were conducted per case.

2.6 Limitations

Limited Quantitative Information

The interviews were conducted and analyzed using qualitative methods. While this allows for a more in-depth data collection and analysis, it presents a risk of bias on the part of the evaluation analysts. This was mitigated by focusing the analysis on factual information and results that were supported by explanations (versus simple opinions without any rationale provided). When possible, the evaluation team also reviewed documentation to ensure that the information was in line with formal program documentation.

Case Studies

The case studies included only five ITB Policy contracts awarded. Therefore, they are not representative of all ITB Policy contracts; nonetheless, they helped inform and corroborate other lines of evidence.

3.0 Findings

3.1 Relevance

3.1.1 Is the ITB Policy, including the Value Proposition (VP), aligned with ISED priorities?

Key Finding: The ITB Policy, including expenditure requirements and Value Propositions, is aligned with ISED priorities related to the competitiveness of Canadian firms, economic growth, job creation, innovation and R&D.

According to documents reviewed, the ITB Policy aligns with ISED's departmental result that Canadian companies are globally competitive and achieve high growthFootnote 9. The application of the ITB Policy has enabled the development of new technologies and skilled labour, acquisition of new technologies, and increased innovationFootnote 10. In addition, the Policy generates intangible benefits to Canadian companies such as knowledge transfer, improves internal processes, and enhances access to other markets and supply chains. These tangible and intangible benefits allow Canadian companies to become competitive and grow in the aerospace and defence sector.

In ISED's 2019-20 Departmental Plan, there is a strong focus on redesigning the innovation policy landscape in Canada. It states that the Innovation and Skills Plan is "a new microeconomic framework to drive growth – one that aims to succeed by building a culture of innovation and a globally recognized brand as one of the most innovative and competitive countries in the world."Footnote 11 Moreover, this multi-year plan builds on Canada's innovation strengths, including exporting and scaling-up globally competitive companies. Since the ITB Policy is used as an investment attraction tool for the Canadian economy, it is well aligned with ISED's priorities. More specifically, the Policy has been developed to ensure that purchases of defence equipment and services results in economic growth, innovation and success in export markets.

In addition, the incorporation of Key Industrial CapabilitiesFootnote 12 into the ITB Policy framework enables the Government of Canada to bring a more strategic approach to leveraging procurement that reflects key upcoming defence procurement requirements for which significant economic and commercial benefit may be derived while still allowing for broad investments. This is documented in the VP GuideFootnote 13.

The ITB Policy also aligns with the Innovation and Skills Plan, one of ISED's key priorities, which emphasizes the development of partnerships through bringing stakeholders together from across the innovation system. The ITB Policy indeed supports this element through encouraging stronger partnerships between contractors and Canadian companies, as well as between industry and research institutions. In fact, the VP Guide also notes that since the introduction of the Policy in 2014, potential bidders are "engaging earlier with Canadian companies to form partnerships and identify high quality business investments up-front on a project." It is also stated that through VPs, R&D investments will be encouraged that involve strategic and long-term partnerships with Canadian SMEs, for example, to provide engineering support that results in knowledge transfer that will help an SME to scale-up and advance the commercialization of innovative ideas and products. Industry partnerships with Canadian post-secondary academic and public research institutions are encouraged under the ITB Policy, including the formation of research consortia, to create an environment that supports world-class innovation and allows for the fusion of hands-on, industrial experience with academic knowledge and inquiry. Furthermore, to motivate bidders to partner with Canadian universities and colleges, additional points towards the evaluation of the VPs can be awarded based on the R&D that bidders and their major suppliers propose to undertake with accredited Canadian post-secondary institutions.

ISED's Departmental Plan also highlights the need to embrace inclusivity and fosters the participation of traditionally underrepresented groups in the innovation economy. The introduction of the Gender and Diversity Plan in the second edition of the VP Guide supports this objective as bidders are now required to provide these plans, at the prime contractor level, describing their approach to achieving gender balance and increasing diversity within their Canadian corporate structures and broader supply chains in Canada. Some interviewees emphasized the addition of the Gender and Diversity Plan requirement, which aligns with GBA+ initiatives being undertaken by the broader government agenda and priorities. However, none of the firms covered in the case studies had a Gender and Diversity Plan, as this was not a requirement when they signed their contracts – which was during the first edition of the VP Guide before the Plan was introduced. Other firms had a Plan, but these existed before their ITB Policy contract came into effect.

The majority of ISED interview respondents indicated that the transition from the IRB Policy to the ITB Policy brought about more opportunities to align with ISED priorities due to the introduction of the VPs. Some government respondents discussed how the ITB Policy allows for flexibility for companies to respond to ISED priorities, which is in contrast to the previous pass/fail structure of the IRB Policy. Respondents from DND generally agreed that the ITB Policy is aligned with DND priorities as well.

3.1.2 Is the ITB Policy in line with the needs of the Canadian defence industry, in particular the small- and medium-sized enterprises (SMEs)?

Key Finding: The ITB Policy is aligned with the needs of the Canadian defence industry.

Canadian defence SMEsFootnote 14 have multiple needs that must be met in order for them to enter markets, grow, and be successful. According to interviews, two of these key needs relate to accessing and integrating into global supply chains by making themselves known to the main contractors and becoming qualified suppliers of the main contractors. Becoming a qualified supplier entails a number of requirements, including having the appropriate certifications, and management and quality control systems, among other requirements.

The document review indicated that, through leveraging defence and major Canadian Coast Guard procurement to create jobs and economic growth, the ITB Policy supports the long-term growth and sustainability of Canada's defence sector, supporting the growth of prime contractors and suppliers in CanadaFootnote 15. Moreover, the ITB Policy requires companies awarded defence contracts to do business in Canada equal to the value of their contracts.

According to interview findings, the ITB Policy requirements for direct expenditures in Canada meet the needs of Canadian businesses, as it allows them to contact prime contractors and offer their services, knowing that the prime contractors will need services from Canadian suppliers in order to meet their ITB obligations. According to survey findings, 46% of recipients have become qualified suppliers as a result of ITB Policy contracts. Case study evidence shows that many suppliers provide support and training for Canadian SMEs in order for them to become qualified:

One contract in the area of aircraft maintenance shows how prime contractors can help SMEs become qualified suppliers. This long-term maintenance contract for the Hercules and Aurora aircrafts, has provided the opportunity for Canadian SMEs to receive training from the prime contractor to allow them to work on these aircraft that were designed to DND's special requirements. They are also required to meet the standards of the International Organization for Standardization. The contracts received allow SMEs to develop new capabilities and expand their markets.

The survey findings also indicate that the ITB Policy contracts have contributed to improving recipients' profiles and credibility (76% agreed) and 26% said that the ITB Policy contracts contributed to their participation in the prime contractor's global supply chain. Almost all interview respondents agreed that this requirement (for direct expenditures in Canada) led to more business for Canadian suppliers.

3.2 Performance

3.2.1 To what extent is engagement with industry prior to contracting effective and timely?

Key Finding: The ITB Policy has led to improvements in the engagement process with industry. Early engagement has allowed industry to provide views on the development of the Value Proposition framework before the Request for Proposals are issued and engagement is generally viewed as being timely.

ISED works in collaboration with PSPC and DND for the engagement process with industry. Engagement with industry informs these three departments and leads to the overall development of a tailored approach for the RFPs.

Findings from the key informant interviews indicated that the ITB Policy has led to improvements in the engagement process between government and industry prior to issuing a contract. A few respondents explained that under the IRB Policy, engagement typically only occurred after the contract was awarded. Under the ITB Policy, there are multiple touch-points for this dialogue to occur – during industry days, one-on-one meetings, and written solicitations (e.g., via questions and answers prior to RFP issuance). Several ISED officials interviewed noted that early engagement provides industry the opportunity to substantiate the feasibility of the VPs (e.g., investing in R&D, SMEs, etc.), which has improved the RFPs and VPs. The majority of industry respondents interviewed agreed that earlier engagement allows government partners to better understand the context of industry at an earlier stage. Government partners can then use this feedback to gain a general sense of whether a project's requirements are feasible. If industry concerns are found to be valid, RFPs can be adjusted by PSPC to better reflect what the industry can deliver for a given contract.

According to the majority of prime contractors interviewed, earlier engagement activities under the ITB Policy (a joint responsibility with DND and PSPC) are useful and beneficial. However, some projects' timelines were impacted by the transition from the IRB Policy to the ITB Policy. These specific instances occurred when the Defence Procurement Strategy was announced and the ITB Policy went into effect. Some IRB Policy procurement contracts were already underway at the time of this transition; as such, there was a small delay while the procurements were adjusted to include ITB Policy elements. Once these projects resumed, the ITB Policy (including the VP element) was applied to the new contracts and industry engagement occurred well in advance of the final RFP, which was the overall intent of the Defence Procurement Strategy.

3.2.2 To what extent has the VP approach been implemented as planned?

Key Finding: Value Propositions increase benefits for Canada in many areas, including R&D, innovation, skills development, and economic development. Two-thirds of prime contractors interviewed were satisfied with the Value Proposition approach. However, one-third expressed concerns about the lack of clarity of the Value Proposition statements, their complexity and limited flexibility of the approach.

Findings from the interviews and case studies suggests that the VP approach has been implemented as planned – the inclusion of VPs in the evaluation of bids is motivating prime contractors to make greater and higher quality investments earlier, and increasing their willingness to partner with Canadian companies. The VP evaluation criteria was found to be clear, fair and well understood by bidders and encourages high value proposals based on individual firms' strengths.

As noted earlier, the VP approach requires bidders to put forward activities that support the five core objectives of the ITB Policy and provide economic benefits to Canada. In terms of the objective related to boosting innovation through R&D in Canada, a number of positive aspects were noted:

However, according to document review findings stemming from meetings of the Defence Industry Advisory Group in June 2018, the multipliers may not be sufficiently recognizing all forms of R&D. The meetings culminated in industry recommending that the R&D investment portion of the VP approach be made broader to allow for more meaningful investments across different technology areas, and that a wider use of the multiplier would encourage greater R&D investments in Canada.

Overall, two-thirds of prime contractors interviewed were satisfied with the VP approach. However, one-third mentioned that they found the overall VP approach complex and sometimes inflexible, starting at the RFP stage. Some prime contractors noted that there is a lack of clarity of some VP statements in the RFPs which has led to disagreements between some prime contractors and ISED at the contracting stage. Additionally, some prime contractors indicated that RFPs can involve many ITB Policy objectives, which makes it difficult for some firms to meet requirements across the board. Prime contractors understand that the objectives are aligned with ISED and/or government priorities, but sometimes not all of the objectives fit well within a project. Some firms would prefer more flexibility to concentrate on one or two objectives, if such options existed. Further, it was mentioned by some prime contractors that the VP approach was not well adapted for their contract/type of work. For example, some service-related contracts are not good candidates for the conduct of R&D, as they are less likely to involve or benefit from R&D than, for example, manufacturing contracts. A few prime contractors indicated that the requirement to work with SMEs can be challenging, particularly for large-value contracts, as SMEs are not seen as having the capacity to meet the requirements of the contract.

Finally, for the ITB Policy objective related to identifying skills development and training opportunities for Canadians, it was mentioned by interviewees that this objective has led to some investments in training for under-represented groups (e.g., women, visible minorities, Indigenous workers, and those with a disability).

3.2.3 To what extent has the ITB Policy led to the development of new business relationships between prime contractors and recipients?

Key Finding: There is evidence that about one-third of recipients have developed new business relationships with prime contractors as a result of ITB Policy-related work. About half have become qualified suppliers. A majority said that ITB Policy contracts allow them to sell to new clients.

There is evidence from the document review and survey that the ITB Policy enables SMEs to access the global value chains of multi-national firms and create export opportunities in multiple key industrial capability areas. For instance, according to the ITB Policy 2019 Annual Report, the ITB Policy enables SMEs to access the Global Value Chains of multi-national firms and creates export opportunities in multiple key industrial capability areas such as training and simulation, and space systems. Since the launch of the ITB Policy, the Value Proposition has resulted in over $2 billion in export opportunities. SMEs benefiting from these export opportunities include Alta Precision and Gastops.

Slightly more than one-third of survey respondents (35%) reported that they have developed new business relationships with prime contractors and have become qualified suppliers (46%). About one-quarter (26%) said that it increased their participation in their prime contractor's global supply chain (especially among large firms – i.e., 500 employees or more). Further, most (58%) said that the ITB Policy-related contracts helped with selling products to new clients.

According to the sub-contractors and prime contractors interviewed during the case studies, the ITB Policy contracts have allowed many sub-contractors to work for the first time for larger prime contractors. This enabled them to become compliant with the prime contractors' quality assurance system and other standards.

One Canadian company obtained a first-time contract to conduct maintenance work on aircrafts under a major prime contractor. The prime company provided training to the Canadian supplier's staff in order for them to do the work. While some sub-contractors may have worked on these aircrafts for other countries before, it was explained that all sub-contracting firms receive training due to the unique requirements of DND for these types of aircraft (about 30% of the components are unique).

Although their long-term business relationships will ultimately depend on their performance, the ITB contracts allowed them to develop new business relationships (new contracts).

3.2.4 What are the economic and social benefits of the ITB Policy?

Key Finding: The ITB Policy is contributing to the development of a Canadian supplier base, as it enforces sourcing from Canada-based suppliers. Most recipients report that the ITB Policy contracts improved their work processes and contributed to their R&D and innovation capacity. According to data analyses, the ITB Policy was estimated to have contributed to the maintenance/creation of as many as 6,450 jobs per year in the private sector between 2014-15 and 2018-19. The impact on Canada's GDP was estimated to have been as high as $3.3 billion for this period.

In order to measure the impact of the ITB Policy on employment and GDP, an I/O analysis was conducted utilizing data from the results of the survey of industry (conducted specifically for this evaluation) and Statistics Canada's National Accounts and I/O tables (see Appendix A for a description of the methodology).

Overall, evaluation findings suggest that the ITB Policy is contributing to significant economic impacts, through direct expenditures in Canada arising from the ITB Policy contract obligations, as well as through spin-off contract work for firms that received contracts from prime contractors. These two impacts were reported for the 2014-15 to 2018-19 period.

The ITB Policy investments that occurred during this period represent 26% of the ITB obligation valueFootnote 16 ($1.6 billion in investments out of $6.2 billion in ITB obligations). Investments stemming from ITB Policy contracts awarded between 2014-2015 and 2018-2019 are expected to continue until 2037.

According to the I/O analysis, contracts covered by the ITB Policy were estimated to have supported or maintained up to roughly 4,100 jobs per year during the 2014-15 to 2018-19 period (see Appendix B). In addition, jobs created via the spin-offs from these contracts were estimated to potentially add up to roughly 2,350 jobs per year. Therefore, considering the impact of the ITB Policy contracts, as well as the estimated spin-off sales, the total impact on employment was estimated to be as high as approximately 6,450 jobs per year for the 2014-15 to 2018-19 period.

The impact on Canada's GDP from the ITB Policy contracts was also estimated. For the 2014-15 to 2018-19 period, the 32 ITB Policy contracts had an estimated GDP impact of as high as

$2.1 billion (see Appendix B). In addition, the spin-offs from these contracts were expected to contribute as much as $1.2 billion to GDP during this same period. Thus, considering the impact of the ITB Policy contracts, as well as the estimated spin-off sales, the impact on Canada's GDP was estimated to be as high as $3.3 billion for the 2014-15 to 2018-19 period.

The case studies provide multiple examples of projects leading to job creation and/or maintenance, as well as business growth for the prime contractors and sub-contractors. For example:

According to interviewees, the ITB Policy, and the VPs in particular, is contributing to the development of a Canadian supplier base, as it enforces sourcing from Canada-based suppliers. The VPs helped some recipients to acquire new manufacturing techniques and improve their technical capabilities. Some Canadian recipients have moved up the supply chain (e.g., from tier 3 to tier 2 suppliers – i.e., levels of sub-contracting) and many have expanded their client base.

The survey of industry also provides evidence of impacts among recipients in terms of improved internal capacity:

Additionally, most recipients (57%) reported that the ITB contracts contributed to increased sales in existing lines of business and some (35%) indicated that it helped increase their share of international markets.

3.2.5 To what extent does use of the banking mechanism lead to an impact on recipients?

Key Finding: The banking mechanism is generally being utilized as planned and is encouraging prime contractors to make investments in Canada.

ISED officials interviewed generally agreed that the banking mechanism is appropriate and is being used as planned. Many felt it was a great tool within industry, as it provides motivation for contractors to work with Canadian suppliers. It allows prime contractors to "think outside the box" and think longer term. Some ISED respondents mentioned that they have seen improvements in the mechanism over time and one respondent indicated that banking activities have tripled since 2015. Most prime contractors interviewed agreed that, while it is difficult to predict, if the banking mechanism was not an option they likely would not have spent (and banked) these amounts in Canadian companies.

Prime contractors indicated that their banked transactions included a range of activities, such as exporting Canadian products, investing in universities for R&D activities, or putting them towards other areas of business that are not related to the defence sector. These were intended outcomes of the banking mechanism.

In terms of understanding, the majority of the 32 interviewed respondents indicated that the rules and regulations regarding the banking mechanism are clear. Three interview respondents indicated that some of the rules and regulations are unclear. For example, it was suggested that there is a lack of clarity regarding what business activities undertaken prior to a procurement qualify as bankable for credit against their IRB/ITB obligations later.

Among prime contractors interviewed who had banked credits, a few indicated that they have not yet used them due to an industry perception that all ITB/IRB Policy-related activities must be completed first. According to one prime contractor, their company compares various options when deciding where to manufacture and will bank only when it is attractive to do so, while another stated that they will not bank for projects that are less than $100,000 because of the difficulty involved. A third prime contractor explained that the process is complicated because it is difficult to understand what is eligible and what is not, and there is uncertainty about whether the efforts required to apply for credits will be successful.

3.2.6 To what extent is the ITB Branch research and analytics function providing value to ISED and other departments?

Key Finding: The ITB Branch research and analytics function is used for decision-making and reporting purposes and is deemed useful within ISED.

According to ISED interviewees, the ITB Branch research and analytics function plays a valuable supporting role to the ITB Branch and other ISED branches by conducting marketing studies, providing advice about monitoring, and producing data for reporting purposes.

When asked if the services provided by the research function were useful, most ISED respondents agreed that the ITB research function is responsive and that the information has been used for decision-making purposes. The function provides useful information for the purposes of developing RFPs, assessing industry capacity, and reporting:

Outside of ISED, two of three DND respondents interviewed were less likely to require the information from the research and analytics function for their processes (i.e., being able to assess the market's ability to meet the needs of the Department), as DND is more interested in technical information. They observed that information obtained from the prime contractors during the engagement phase often met their requirements, limiting the need for market information provided by the research and analytics function. However, not all projects which leverage the function involved DND.

3.3 Efficiency

3.3.1 To what extent has the implementation of the ITB Policy demonstrated efficiency? Are there alternatives to improve efficiency?

Key Finding: ITB Branch operations were efficient over the 2014-15 to 2018-19 period. There are opportunities to improve the consistency of the crediting process.

Efficiency, or the degree to which the program is efficient in its use of resources, was assessed through a review of the administrative costs against the level of contracts covered by the Policy, as well as via interviews about internal processes.

Over the 2014-15 to 2018-19 period, actual expenditures for the ITB Branch for work involving ITB Policy contracts and the remaining IRB Policy contracts (i.e., salaries plus operations and maintenance costs) totalled $36.3 million (as shown in Table 1). Compared to the 32 contracts initiated during this period (valued at $6.2 billion), ITB administrative costs represent 0.6%.

Interviewees noted that there were no major issues or gaps with respect to the efficiency of the ITB Branch operations. Further there are no redundancies between activities within the ITB Branch, according to respondents. The ITB Branch is taking concrete steps to reduce administrative burden and improve client service. This includes streamlining the annual reporting process for contractors and implementing a six-month service standard for completing verification of annual reports. The goal of the six-month service standard is to award contractor credits in a timely fashion, standardize the annual reporting process, and be accountable and transparent in the verification processes. According to service standards data, verification activities are compliant to the six-month service standard in more than 95% of cases, significantly outpacing the targeted standard of 80%.

Many ISED respondents made suggestions to improve the consistency of the crediting process (i.e., the process whereby prime contractors submit reports to get their expenses credited as per their VPs and obligations). It was noted that the crediting process is based on a set of criteria that is subject to interpretation by ISED program officials – leading to inconsistencies. Having clear criteria ensures a more efficient crediting process, as there is less interaction required with prime contractors submitting the credits.

4.0 Conclusions and recommendations

4.1 Conclusions

Relevance

Performance

Efficiency

4.2 Recommendations

The evaluation findings led to the recommendations noted below.

Recommendation 1: Value Proposition Approach

ISED should continue to work with contractors to ensure that the Value Proposition approach on a procurement-by-procurement basis is as flexible as possible for prime contractors to fulfill requirements, while continuing to meet the policy objectives.

Recommendation 2: Crediting Process

ISED should enhance internal guidance related to the application of crediting guidelines for contractor expenses to improve the consistency of the crediting process.

Recommendation 3: Program Understanding

ISED should continue industry engagement and outreach activities to enhance stakeholder understanding of the ITB Policy, particularly the Value Propositions.

Appendix A – Input-Output Analysis

Input-Output Analysis Methodology

The economic impact analysis undertaken for this evaluation utilized data from the results of the survey of industry undertaken as part of this evaluation. The data obtained was then analyzed using Statistics Canada's National Accounts and Input Output (I/O) Tables. The analysis also includes an I/O analysis of the ITB Policy contracts (as a separate analysis). The I/O analysis was conducted by ISED economists.

The objective of the analysis was to identify how ITB Policy transactions affect the Canadian economy via direct purchases of goods and services and indirect demands inherent in supply chains for inputs into direct goods and services. In addition, throughout the supply chains, contractors may gain both knowledge in producing sophisticated equipment and intellectual property that allows them to gain international accreditation, and expand their offerings in both civilian and non-civilian markets, possibly through the utilization of the prime contractors' marketing channels, for spinoff sales. This analysis also measured the values of such sales.

The economic impacts comprise direct, indirect, and induced impacts:Footnote 17

Process

The first step in this process was to identify the contract and follow-on or spinoff sales revenues generated at the establishment and industry level as a result of the ITB Policy contracts. The North American Industry Classification System (NAICS) codes, available at the firm level on the Industry Canada Business Profiles, or failing that contractors' websites, were used to link ITB Policy establishment data on revenues to Statistics Canada's I/O tables. Where a firm listed more than one NAICS code, the one most applicable was applied based on the description of each ITB Policy contract. Therefore, a firm with several establishments could have several NAICS codes. This process allowed the defining of the economic shock (expenditure), or ITB Policy stimulus, as incremental revenues by NAICS code at purchaser prices.

Contribution to GDP

The impacts on national income at the national level were captured by estimating incremental GDP. GDP is a gross measure of what is produced incrementally domestically in Canada prior to taking depreciation into account. This figure relates not only to the firms receiving the ITB Policy contracts, but also to all suppliers in each contractor's supply chain. Some of these contract recipients and suppliers are likely to have imported materials and supplies, upon which duties would have been paid. These and other taxes have been estimated via the Statistics Canada I/O tables. When they are subtracted from the stimulus, the I/O results yield the direct and indirect GDP impacts at market prices.

As long as there are imports, the GDP multiplier relative to the total economic shock will be fractional. The multiplier approaching or exceeding unity in the analysis comes from induced impacts when the incomes earned from the project are spent. The GDP at market prices is calculated by subtracting indirect taxes and imports on products and adding in any subsidies to the products over and above ITB Policy qualified expenditures. This metric is important since its elements identify various income streams. The direct and indirect impact on GDP and employment by industry indicate the reach of defence procurements (with ITB Policy requirements) in Canada. Since I/O analysis traces supplies back up the supply chain by project, the analysis identifies GDP and employment by industry.

Direct employment estimates

This includes both new full-time equivalents and employees who retain their positions due to ITB Policy defence procurement contracting. In this sense, they are incremental to the counterfactual of, "What would have been the relative level of employment without the funding and follow-on sales, had companies failed to find any other markets?" They may not all be incremental since the companies throughout the supply chain may have found other work or horded Highly-Qualified Persons (HQPs) and other staff in hopes of retaining intellectual capital. Conversely, some prime contractors may not have had a Canadian presence at all without the ITB Policy's impact on defence procurement.

Other assumptions and considerations included the following:

Data Sources

In addition to Statistics Canada I/O tables, the analysis was conducted using ITB Policy program data between 2014 and 2024 and data from a survey of industry that was conducted as part of the evaluation. The I/O analysis includes only the 32 contracts awarded under the ITB Policy. All IRB Policy contracts are excluded from the analysis, as they are outside the scope of this evaluation.

Each of the 32 ITB Policy contracts are listed below, along with their year of award and obligation value.

Year Awarded ITB Policy Project Obligation Value

2018

Chemical, Biological, Radiological and Nuclear Respirator - Acquisition (GSR-A)

$25,946,343

2018

Chemical, Biological, Radiological and Nuclear Respirator (GSR-I)

$2,164,166

2018

Land C4ISR Sustainment Project - Cyber Security Engineering Services (LC4-SC)

$50,000,000

2018

Land C4ISR Sustainment Project - Engineering and Integration (LC4-SE)

$325,000,000

2018

Land C4ISR Sustainment Project - ISTAR (LC4-SI)

$50,000,000

2018

Land C4ISR Sustainment Project - Software Support (LC4-SS)

$175,000,000

2018

Maritime Unmanned Aerial System - Interim Capability (CAF UAS)

$60,437,840

2018

Mercury Global—Strategic Deployable Terminals (MG SDT)

$28,257,799

2018

Underwater Warfare Sensor Suite Upgrade (UWSU)

$97,523,219

2017

Arctic Offshore Patrol Ship and the Joint Support Ship - Maintenance (AJISS)

$4,821,650

2017

Buffalo and Cougar Armoured Vehicles - Maintenance (EROC ISS)

$11,750,000

2017

C130-CP140 Aircraft Engine— Long Term Maintenance (T56-O)

$107,282,053

2017

CF-18 Engines - F404 Long-term Maintenance (F404-O)

$106,434,421

2017

Contracted Airborne Training Services (CAT S)

$407,680,394

2017

Headquarters Shelter Systems—Acquisition (HQSS-A)

$170,031,829

2017

Headquarters Shelter Systems—Maintenance (HQSS-I)

$12,044,381

2017

Phalanx Naval Weapon System - Support (PHALX2-I)

$330,038,170

2017

Victoria Class Command and Control System – Maintenance (VCCCS)

$18,270,000 (USD)

2016

Combat Net Radio Enhanced - Maintenance (CNRE-I)

$33,867,771

2016

CT114 Tutor Fleet - Maintenance (CT114 PAV)

$18,400,000

2016

Fixed-Wing Search and Rescue Aircraft - Acquisition (FWSAR-A)

$1,925,195,879

2016

Fixed-Wing Search and Rescue Aircraft - In-Service Support (FWSAR-I)

$578,793,661

2016

Maritime Satellite Communications Upgrade (MSCU)

$25,012,840

2016

Naval Remote Weapon Station (NRWS)

$37,191,779

2016

Polar Epsilon 2 (PE2)

$57,593,968

2015

C130J Tactical Airlift — Maintenance 2 (C130J-IP2)

$708,676,610

2015

CP-140 Aurora - Data Management System - Block IV (CP140BIV)

$285,111,825

2015

Diesel Generator Sets (DGSet)

$52,465,693

2015

Medium Range Radar - Acquisition (MRR-A)

$138,500,782

2015

Medium Range Radar - Maintenance (MRR-I)

$8,406,404

2014

C17 Strategic Airlift—Acquisition—5th Aircraft (C17-A 5)

$264,500,000

2014

Canadian Coast Guard Helicopter—Medium (CCGHM)

$102,341,743

TOTAL - $6,218,741,220

Appendix B – Economic Impact of ITB Policy

Three scenarios were developed to assess the economic impact of the ITB Policy, including the Value Proposition. The assessment only includes 32 contracts awarded since 2014-15 under the ITB Policy and to which the Value Proposition was applied. This does not include contracts awarded under the National Shipbuilding Strategy, or contracts awarded under the IRB Policy.

2014-15 to 2018-19 Period

Pessimistic Scenario:
Spin-off for sample respondents only and 50% of direct benefits attributable to the ITB Policy

Mid-Range Scenario:
Spin-off for sample respondents and 50% of non-respondents and 75% of direct benefits attributable to the ITB Policy

Optimistic Scenario:
Spin-off impacts of sample extended to population of recipients and 100% of direct benefits attributable to the ITB Policy

Impact on GDP from ITB Policy contracts

$1,040,423,276

$1,560,634,914

$2,080,846,552

Impact on GDP from spin-off revenues

$161,303,877

$681,463,125

$1,201,622,373

Total impact on GDP

$1,201,727,153

$2,242,098,039

$3,282,468,925

Impact on employment from ITB Policy contracts

2,052

3,078

4,104

Impact on employment from spin-off revenues

358

1,355

2,353

Total impact on employment

2,410

4,433

6,457

Explanation of scenarios

As shown in the table, the economic impacts are of two types: direct impacts from the contracts covered by the ITB Policy, and the spinoff contracts (past and projected) for the recipients (sub-contractors). While there are program and survey data that can be used to assess these impacts, there are two factors that prevent the evaluation from producing a single statement/number about the net impacts of the program.

Based on the above, the evaluation provides results under three scenarios.

Scenario 1: Pessimistic

The pessimistic scenario is based on two assumptions: 1) spinoff impacts are limited to the sample of respondents only – no impacts are attributed to those who did not respond to the survey; and 2) impacts attributed to the directly associated ITB Policy contract dollars are estimated at 50% - in other words, the net impact of the policy is set at 50% of the contract dollars (i.e., the other 50% would have happened in the absence of the policy).

Scenario 2: Mid-Range

Under the mid-range scenario, the impacts of the spin-off projects include those reported by the survey respondents, and are extended to half of the remaining sample (that did not respond to the survey). In other words, this scenario assumes that the sample is only representative of about 75% of the recipient population – and that no impacts are attributed to 25% of the remaining non-responding recipients. This remains fairly conservative given that the statistical margin of error of the obtained sample is approximately 12%. It should also be mentioned that there was no evidence (based on the results of the follow-up phone calls during the conduct of the survey) that the non-responding recipients had closed their operations. As for the direct economic impacts of ITB Policy contracts, this scenario attributes a net impact of 75%. In other words, 75% of the impacts associated with the contract dollars are attributed to the ITB Policy. Again, this is considered fairly conservative considering that most of the prime contractors are large international firms, and that interview evidence indicates that they would likely conduct their work through foreign establishments. However, it is still likely that some of the work (e.g., maintenance work) would have been conducted in Canada by Canadians.

Scenario 3: Optimistic

The optimistic scenario basically assumes that the survey sample is representative of all recipients, that is, that the results of the sample (spin-off contracts) can be fully extended to the entire recipient population. It also assumes full attribution of the economic impact to the ITB Policy.

Date modified: