Evaluation of Post-secondary Institutions Strategic Investment Fund

From: Innovation, Science and Economic Development Canada

This publication is available online at www..

To obtain a copy of this publication or an alternate format (Braille, large print, etc.) please fill out the Publication Request Form at www.ic.gc.ca/Publication-Request or contact:

Web Services Centre
Innovation, Science and Economic Development Canada
C.D. Howe Building
235 Queen Street
Ottawa, ON K1A 0H5
Canada

Telephone (toll-free in Canada): 1-800-328-6189
Telephone (Ottawa): 613-954-5031
TTY (for hearing-impaired): 1-866-694-8389
Business hours: 8:30 a.m. to 5:00 p.m. Eastern Standard Time
Email: info@ic.gc.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Innovation, Science and Economic Development Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Innovation, Science and Economic Development Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Innovation, Science and Economic Development Canada.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the Web Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Innovation, Science and Economic Development Canada, 2018.

Cat. No. Iu4-264/2019E-PDF
ISBN 978-0-660-31488-4

Aussi offert en français sous le titre Évaluation du Fonds Technologies du développement durable.

March 2019

Presented to the Performance Measurement and Evaluation Committee on March 20, 2019


List of acronyms

AEB
Audit and Evaluation Branch
CaGBC
Canada Green Building Council
CICan
Colleges and Institutes Canada
GHG
Greenhouse gas
GoC
Government of Canada
ISED
Innovation, Science, and Economic Development
KIP
Knowledge Infrastructure Program
LEED
Leadership in Energy and Environmental Design
NECB
National Energy Code of Canada for Buildings
O&M
Operations and Maintenance
PSI
Post-Secondary Institutions
PSI-SIF
Post-Secondary Institutions Strategic Investment Fund
P/T
Provincial/Territorial
RCGT
Raymond Chabot Grant Thornton
TBS
Treasury Board Secretariat
U15
The Group of Canadian Research Universities
UC
Universities Canada
UPEI
University of Prince Edward Island

List of figures


List of tables

Executive Summary

Program overview

The Post-Secondary Institutions Strategic Investment Fund (PSI-SIF) was launched in 2016-17 with the objective of generating direct economic activity in the Canadian economy and accelerating the strategic construction, repair and maintenance of infrastructure at post-secondary institutions across Canada in order to strengthen capacity for research and innovation and industry-relevant training and to support environmental sustainability.

In the 2016 Budget, the government announced up to $2 billion over three years, starting in 2016–17, for PSI-SIF, a time-limited program that supports up to 50% of the eligible costs of infrastructure projects at post-secondary institutions and affiliated research and commercialization organizations across Canada. PSI-SIF is delivered collaboratively by ISED (headquarters and regional offices) and the provincial/territorial (P/T) governments.

PSI-SIF approved a total of 302Footnote 1 infrastructure projects in 109 colleges and 60 universities across all Canadian provinces and territories (P/Ts).  

Evaluation purpose and methodology

The PSI-SIF evaluation addressed the issues of program relevance, the effectiveness of design and delivery and performance in accordance with the Policy on Results.  The evaluation had the objective of studying the extent to which the program reached its stated objectives and documenting lessons learned to inform organizational learning and program excellence.  The evaluation employed four data collection methods: document review, data review, stakeholders’ interviews and case studies.  It was conducted by ISED’s Audit and Evaluation Branch (AEB) and covered the period from April 2016 to November 2018.

Findings

Relevance

Investment in PSI-SIF helped stimulate the construction sector’s labour market by creating economic activity across Canada at a time when the country was experiencing an economic downturn.

The program also responded to the capital and deferred maintenance investment needs of the Canadian post-secondary institutions and enabled the implementation of many of their infrastructure projects that were shovel-ready and on hold waiting for funding.

Design and Delivery

The federal-provincial program delivery aspect of PSI-SIF made good use of the P/Ts’ knowledge and established relationships with post-secondary institutions.

During its two application phases (initial and extension applications), the program staff provided the information and support needed to facilitate the process for the institutions.  However, the short timeline of the initial application posed challenges for the applicants.  Moreover, the program extension application was perceived by some as time consuming and the information provided in the progress reports could have been better leveraged.  Further, during the extension application assessment phase, the institutions were uncertain of how to proceed with their projects, while awaiting extension decisions.

The program involved the submission of progress reports for the funded projects, which helped identify projects at risk and enabled project monitoring. However, the frequency of reporting and level of detail required posed challenges for some institutions and P/Ts, who considered them time consuming.  Further, in some cases, the submission of the projects’ close-out reports 90 days after “substantial completion” of the projects was reported to be insufficient, more particularly by the institutions, which are accustomed to submitting these reports after full completion of the projects.  The program was responsive to the needs of those institutions and allowed flexibility around the timing of the close-out report submission.

As an economic stimulus program, the PSI-SIF applied an advance payment mechanism that helped projects continue moving forward without delays.  However, in some P/Ts, this mechanism posed cash flow challenges, which impacted smaller institutions more than larger ones.

PSI-SIF’s actual operation costs were in line with the program plan.  The program’s ability to leverage other human capacity during peak operation times allowed the program to efficiently use its operational budget.

Performance

PSI-SIF equipped the funded post-secondary institutions with state-of-the-art infrastructure, which the institutions reported will help them improve their research and training capacity, enable the recruitment of more students and allow collaboration and synergies for networking with their stakeholders and knowledge users.

PSI-SIF’s new and renovated infrastructure enabled some institutions to be certified through the Leadership in Energy and Environmental Design (LEED) and is expected to help others save energy and reduce greenhouse gas emissions.

Some institutions anticipate that, as a result of the PSI-SIF funded projects, they will be able to create more research jobs, faculty positions and internal administrative and technical jobs. Anecdotal evidence showed that some projects were reported to have already shown initial positive impacts by engaging Indigenous and Metis community members and creating job opportunities for students in certain fields.

Lessons Learned

Based on the evaluation findings, the following design and delivery elements should be taken into consideration for any similar future ISED programs.

Lesson Learned 1: Involving subject matter experts in the program design phase

Early consultation with the provinces and infrastructure experts during the program’s design phase is advised in order to mitigate implementation challenges.

Lesson learned 2: Reporting

Making the reporting requirements commensurate with the projects’ risk level and streamlining federal and provincial reporting requirements would decrease reporting burden for P/Ts and institutions.

Lesson learned 3: Program extension

Planning and communicating the extension of infrastructure projects to funded institutions earlier would help reduce uncertainty and related financial burden.

1.0 Introduction

This report presents the results of the evaluation of the Post-Secondary Institutions Strategic Investment Fund (PSI-SIF) program. The purpose of the evaluation is to assess the relevance, effectiveness of the design and delivery model and performance of the PSI-SIF.

The report is organized into four sections:

1.1 Context

Global economic growth slowed in 2015 to its weakest pace since the 2008-09 recession led by slower growth in emerging economies such as China and other commodity producing countries and rapid decline in global crude oil prices.Footnote 2 These challenging external economic conditions had a significant impact on the Canadian economy in 2015 and cast a shadow over the country’s economic future. As the oil prices dropped by about 65 per cent, it led to significant declines in investments in the oil and gas sector and a negative real GDP growth over much of 2015.  This decline in business investment was felt most dramatically in the major oil-producing provinces of Alberta, Saskatchewan, Manitoba and Newfoundland and Labrador, which experienced a sharp increase in their unemployment rates between 2014 and 2016 (an increase of 72%, 66%, 13% and 13% respectively).Footnote 3

To address this economic impact, in Budget 2016, the Government of Canada identified infrastructure investment as a key source of substantial economic activity that would lead to the creation of “good, well-paying jobs that can help the middle class grow and prosper … (and) deliver sustained economic growth for years to come.Footnote 4 Moreover, as outlined in the 2015 Speech from the Throne, infrastructure investments were considered critical to supporting Canada’s climate change objectives.Footnote 5

Budget 2016 also highlighted that investing in infrastructure, in particular at post-secondary institutions, would reinvigorate Canada’s research and science base.  This investment was also expected to address the needs of the Canadian post-secondary institutions,Footnote 6 which experienced a shortfall of an estimated $8.4 billion in deferred maintenance (nearly doubled since 2000) according to a 2014 study by the Canadian Association of University Business Officers.  In addition, a 2015 survey by Colleges and Institutes Canada (CICan)Footnote 7 found that over one third of college and institute programs operate with training facilities that are below the standard of the current workplace equivalent.

Thus, in order to generate direct economic activities, enhance the research and training infrastructure at Canadian post-secondary institutions and improve their environmental sustainability, the Government of Canada allocated $2 billion over three years, starting in 2016–17, for the PSI-SIF (a time-limited program).  PSI-SIF is delivered collaboratively by ISED (headquarters and regional offices) and the provincial/territorial (P/T) governments (See Annex A for more details about the program governance and delivery partners’ roles and responsibilities).

1.2 Program Description

1.2.1 Program Coverage

Under PSI-SIF, the maximum share of funding for eligible projects, from all federal sources, is 50% of total eligible costs.  Funded projects, therefore, required an amount of investment that is at least equivalent to the value of funding provided by the federal government.  The balance of project funding came from post-secondary institutions themselves, P/T governments, or other sources.  The exceptions were those institutions accredited by an Aboriginal Government.  In those cases, 100% of eligible project costs could be covered by the Government of Canada.

PSI-SIF federal funding flowed through two mechanisms:

1.2.2 Projects Eligibility, Application and Selection Process

Program Objectives:

PSI-SIF aimed to:

  • Improve research and innovation-related post-secondary infrastructure;
  • Improve college infrastructure for industry-relevant training; and
  • Improve environmental sustainability of research and innovation related infrastructure at post-secondary institutions and college training infrastructure.

Projects eligible under PSI-SIF included shovel-ready projects that support repair, enhancement, new construction or expansion that would improve the quality of research and development and the ability to deliver advanced knowledge and skills training.

Following the program launch and the call for applications, post-secondary institutions submitted their applications directly to ISED.  ISED’s program team assessed the submitted projects according to the established eligibility criteria.  In each P/T, projects that were a P/T priority were identified and the P/Ts attested to the availability of matched funding and that projects met the readiness criteria.  Prioritized eligible projects were then selected by the Minister of ISED, in consultation with the Minister of Science, and the provinces and territories.

Initially, funded projects were required to be completed by April 30, 2018, which aligned with the expected end date of the program.  In February 2018, the program was extended to November 30, 2018 as a number of projects encountered unforeseen circumstances, such as extreme weather conditions and design complexity that resulted in project delays. Extensions were, therefore, granted to eligible projects following an extension application and review process.

1.2.3 Funding Disbursement and Program Reporting

Under the P/T contribution agreements, the P/Ts received funding directly from ISED and were responsible for dispersing it to the funded projects in their respective jurisdictions.

Under the bilateral agreements, post-secondary institutions received project funding directly from ISED.  Payments to bilateral institutions were made against the achievement of milestones. 

The program developed and implemented a Risk-Based Audit Framework (RBAF), which articulated the independent monitoring, reporting and auditing function supporting the contribution and bilateral agreements, with the help of a third party consultant.  The program reviewed the quarterly reports submitted by the P/Ts and conducted a risk assessment to identify high risk projects in each P/T.  If the overall progress of the projects in a P/T was not proceeding as scheduled, the amount received by the P/T would be less than12.5%, with the balance held back until overall progress was deemed sufficient.  In accordance with a Treasury Board requirement, the P/Ts are required to disburse the 12.5% (or less) to the projects in their jurisdictions within 45 days of receiving the funds; otherwise, a hold back on the subsequent disbursement would apply.

1.3 Program resources

The total annual federal funding disbursement to PSI-SIF projects was as follows:Footnote 8

Description of Pictogram 1.3

Pictogram 1.3 (above) depicts the annual breakdown of ISED’s funding disbursements associated with the Post-Secondary Infrastructure Strategic Investment Fund (PSI-SIF), over the three year duration of the program. Out of the total $1.98 billions spent on the program, in 2016-17, ISED disbursed to the provinces, territories and bilateral institutions $744.3 million in 2016-17, $730.4 million in 2017-18 and $510 million in 2018-19.

In addition to the funds disbursed to the infrastructure projects, P/Ts received funds to offset program administration and delivery costs, which did not exceed 2.5% of the total PSI-SIF contribution to each jurisdiction.Footnote 9

1.4 Program target population and stakeholders

PSI-SIF’s primary direct target population is the post-secondary institutions (i.e., universities, colleges and CEGEPs) and their affiliated research and commercialization organizations across Canada. The program, consequently, serves the needs of the research and training community, including researchers and trainees at these institutions.

PSI-SIF also has a number of stakeholders, which includes the provinces and territories, post-secondary institutions representatives, such as Canadian Association of University Business Officers and CICan, stakeholders operating in the construction sector and program funding partners from the private and non-profit sectors.

1.5 Logic model

The logic model in Figure 1 visually presents the activities undertaken, the outputs and outcomes expected to result from the delivery of the program.  It also shows the logic of how PSI-SIF achieved its objectives.

Figure 1: Logic Model

Description of Figure 1: Logic Model

The logic model in Figure 1 visually presents the activities to be undertaken, the outputs to be produced and the outcomes expected to result from the delivery of the program.  It also shows the logic of how PSI-SIF is expected to achieve its objectives.

PSI-SIF delivers three types of activities and outputs:

  1. Outreach and Communications
  2. Project Selection
  3. Project Management and Monitoring

The outreach and communications activities include consultations with stakeholders, drafting of Ministerial letters, program calls for proposals as well as provincial and territorial calls.

The project selection activities include the program application process, applications received and approval, signature of contributions agreements with the provincial/territorial governments and individual post-secondary institutions.

The project management and monitoring activities include the creation of processes for program management and monitoring, the quarterly review process, payments and compliance audits, as well as the program evaluation and audit.

PSI-SIF three sets of activities lead to two immediate outcomes: the first is the direct economic activities resulting in local economies across Canada and the second is the completion of infrastructure projects in universities and colleges.

Three intermediate outcomes come as a result of the two immediate outcomes:

  1. Improved research and innovation related post-secondary infrastructure
  2. Improved college infrastructure for industry relevant training
  3. Improved environmental sustainability of research and innovation

Ultimately, all three intermediate outcomes lead to the program’s ultimate outcome which is: Canada has world leading research capacity.

2.0 Methodology

This section provides information on the evaluation scope and objective, issues and questions that were addressed, data collection methods, and limitations.

2.1 Evaluation scope and objectives

The evaluation of PSI-SIF was conducted in accordance with the Treasury Board Policy on Results and covered the period from April 2016 to November 2018.

Since PSI-SIF is a time-limited program, the evaluation focused on studying the extent to which the program reached its stated objectives rather than examining the continued need for the program.  As the program was still ongoing during the evaluation, it was difficult to systematically assess the medium and long-term program outcomes.  However, anecdotal evidence of outcomes was collected to assess the program performance.

Therefore, the PSI-SIF evaluation studied in more detail the program’s design and delivery model with the objective of documenting lessons learned to inform organizational learning and program excellence. The design and delivery model was assessed by examining program governance, application processes, monitoring and reporting and funding disbursement mechanism.

2.2 Evaluation issues and questions

PSI-SIF evaluation considered the following evaluation questions under the evaluation issues of relevance, design and delivery and performance:

Relevance

  1. To what extent did PSI-SIF address a demonstrable economic need for Canada?
  2. To what extent did PSI-SIF respond to the needs of Canadian post-secondary institutions?

Design and Delivery

  1. To what extent was PSI-SIF effectively and efficiently implemented?

Performance

  1. To what extent did PSI-SIF contribute to improved research and innovation related infrastructure and industry relevant training infrastructure at post-secondary institutions (universities and colleges)?
  2. To what extent did PSI-SIF contribute to improved environmental sustainability of research and innovation infrastructure?

2.3 Evaluation methods

Multiple lines of evidence were used to address the evaluation questions. As Figure 2 below highlights, data sources included:

Figure 2: Evaluation Issues and Methods of Data Collection

Description of Figure 2: Evaluation Issues and Methods of Data Collection

Figure 2 shows the evaluation issues and methods of data collection associated with the three evaluation issues:

  1. Relevance
  2. Design and delivery
  3. Performance

In order to assess relevance, the program evaluation team reviewed and analysed program documents, administrative and financial data, and conducted interviews with program stakeholders.

For the program design and delivery, the program evaluation team reviewed and analysed program documents, administrative and financial data, conducted interviews with program stakeholders, and undertook case studies on a number of funded projects.

To assess the performance of the program, the program evaluation team reviewed and analysed program documents, administrative and financial data, conducted interviews with program stakeholders, and undertook case studies of a number of funded projects.

Document Review
A document review of PSI-SIF key documents was conducted in order to facilitate an understanding of PSI-SIF, assess the program delivery model and the progress toward program objectives.

Documents analyzed included:

Administrative and Financial Data Analysis
Analysis of PSI-SIF administrative and financial data was conducted in order to provide information regarding the extent to which the delivery model is effective and efficient and assess the extent to which the program is achieving its objectives.

The data analyzed included:

Interviews
The evaluation also included interviews with key PSI-SIF stakeholders (n= 35). Findings from the interviews supported analysis on relevance, and the effectiveness of the program’s design and delivery model.  They also helped in assessing early program outcomes and identifying areas for improvement.

Stakeholders’ interviews included:

Case Studies
The evaluation also included case studies (n=8) in order to capture early evidence of the achievement of expected outcomes and document successful stories linked to the efficient implementation of the program’s design and delivery model.

Each case study included a review of the relevant project files and documents in addition to interviews with a sample of representatives from the projects’ implementers, managers and beneficiaries (researchers and trainees).

Case studies were selected from projects that were deemed successful by the PSI-SIF program.  The success criteria in this respect meant projects of different scope and sizes that were completed as planned.

Cases included representation from P/Ts, Aboriginal institutions, colleges and universities, projects covered under bilateral and P/T agreements.

2.4 Limitations and mitigation strategies

The following were the challenges faced during the PSI-SIF evaluation and the mitigation techniques applied to address them.

Assessment of Intermediate and Ultimate Outcomes
PSI-SIF is still ongoing, thus, the data needed, to systematically assess the program’s intermediate and ultimate outcomes, was not available during the evaluation.  As a mitigation technique, whenever possible, anecdotal evidence of preliminary outcomes was collected through the interviews, case studies and available projects close-out reports, in order to report on intermediate outcomes.

Availability of Cost-Efficiency Data
There was insufficient data to allow for a comprehensive assessment of efficiency. As a mitigation technique, in addition to the data available, questions were raised during the interviews and case studies about whether or not the program delivery model enabled the efficient use of time and money.

Selection Bias
The lists of possible interviewees and cases to be studied for the evaluation were originally suggested by program staff.  To mitigate selection bias, whenever possible, the evaluation team encouraged the interviewees selected from these lists to nominate and invite other members from their organizations (P/Ts or institutions) who participated in the implementation of the program to attend the interviews in order to make sure other points of view were also taken into consideration.  

Gaps in Administrative Data
At the time of the evaluation, only 17 close-out reports for completed projects were available. The evaluation team analyzed the available reports and used the other lines of evidence to fill in the gaps.

Reporting Bias
Since a small sample of post-secondary institutions were interviewed, it is difficult to generalize the results reported by the institutions due to the possibility of reporting bias.  Interviews with the P/T representatives and program management as well as the results of the document review were used to validate the findings from the institutions’ interviews.

3.0 Findings

3.1 Relevance

This section analyzes findings from the interviews and document review to assess the evaluation issue of program relevance. It answers the following evaluation questions:

3.1.1 Addressing Demonstrable Economic Needs for Canada

Investment in PSI-SIF helped stimulate the construction sector’s labour market.

As highlighted above, before the launch of the PSI-SIF there was an economic need to stimulate the economy and create jobs by investing federal money into local Canadian economies. This includes increased government spending on the infrastructure sector. The program estimated that the total number of jobs created over the past three years as a result of investments in PSI-SIF was approximately 37,000 (See Table 1).Footnote 10

Table 1: Estimated Number of Infrastructure Jobs Created as a Result of PSI-SIF

Description of Table 1: Estimated Number of Infrastructure Jobs Created as a Result of PSI-SIF

Table 1 shows the total funding amounts allocated to the PSI-SIF projects from all sources (federal, provincial and institutions) in each province and territory in one column. In the other column, the table shows the number of infrastructure jobs created in each province and territory, as a result of the total PSI-SIF funding allocated to their projects. The table illustrates that the higher the investment the greater the number of jobs created. The total amount of investment in PSI-SIF projects from all sources was $4.97 billion, which created an estimate of 37,290 jobs. It is worth noting that the estimated number of infrastructure jobs created as a result of the PSI-SIF was calculated through the utilization of a Department of Finance job multiplier formula.

Findings from the interviews showed that the program responded to the economic needs by stimulating the construction sector job market in P/Ts.  In Saskatchewan, for example, it was reported that the program helped create momentum in the construction sector, which experienced a sharp decline over the previous six years.

Moreover, the program helped create jobs in a number of Aboriginal communities.  For example, in implementing their project, Ts’zil Learning Centre – Lil’wat Nation had a performance agreement with their construction contractor to select first nation workers in trades like carpentry in order to create jobs in their community.

In provinces, such as Ontario, Quebec, British Colombia, Nova Scotia and Prince Edward Island (PEI), which were not heavily impacted by the oil crisis, the interviews indicated that finding the necessary skilled labour to complete the PSI-SIF projects was quite challenging due to the existing large number of infrastructure projects.  This led to an increased demand for infrastructure labour than the supply available. As a result, some institutions paid higher construction prices than planned for their projects.  Other institutions noted the resulting benefit of enabling some smaller construction firms, which had less presence in the market, to break through.  

3.1.2 Addressing the Needs of Canadian Post-Secondary Institutions

A significant proportion of Canada’s post-secondary infrastructure has exceeded the average service life of educational buildings and are in need of renovation and expansion. PSI-SIF filled a gap in the P/Ts’ capital investment and enabled the implementation of post-secondary institutions’ infrastructure projects that were shovel-ready and in need of funding.

As highlighted in the context section, post-secondary institutions in Canada needed funding to cover their deferred maintenance and expansion plans.Footnote 11  Traditionally, direct federal investments in university and college infrastructure is focused on research and innovation objectives, while “bricks and mortar” investments in post-secondary institutions fall under the responsibility of Provincial/Territorial (P/T) governments, supported by block transfers from the federal government under the Canada Social Transfer. However, the federal government also has had a history of contributing to post-secondary facility renewal and development.Footnote 12 As reported in the interviews and case studies, the P/Ts’ budgets for infrastructure are often insufficient to cover all the projects in need.  As highlighted by many of the institutions interviewed, several projects would not have been undertaken without PSI-SIF.

Moreover, without PSI-SIF funding, an expanding institution, such as Yukon College, would not have found sufficient resources to support its growth. PSI-SIF also provided the opportunity for private institutions, such as St. Mary’s University in Calgary and Concordia University of Edmonton, which are not eligible for provincial capital funding, to receive funding via a direct bilateral agreement in order to implement some of their projects that were waiting on their deferred maintenance list for a long time.

As highlighted in Table 1 above and supported by the interviews, the program enabled funds to be leveraged (every $1 of federal funding leveraged $2.5 from other sources), which consequently led to additional investments in the infrastructure sector. This funding leverage enabled the implementation of the P/Ts’ identified priority infrastructure projects in each jurisdiction.

3.2 Design and delivery

This section analyzes findings from the interviews, case studies and document and administrative data review in order to answer the evaluation question of:

In order to answer this question, the following aspects of PSI-SIF model of design and delivery were assessed:

3.2.1 Governance Structure

The federal-provincial delivery model supported program administration and monitoring by leveraging P/Ts’ knowledge and their established relationship with post-secondary institutions. Roles and responsibilities under PSI-SIF were generally clear, although the role played by ISED’s regional staff initially lacked clarity.

Findings from the interviews and case studies showed that the federal-provincial model of program delivery (See Annex A) was beneficial to the effective implementation of the program.  As reported, the model leveraged the pre-existing relationship between P/Ts and institutions and capitalized on the P/Ts’ knowledge of the institutions’ needs.

Moreover, the P/Ts are well positioned to monitor and report on the progress of PSI-SIF funded projects given their extensive experience funding and participating in the implementation of infrastructure projects at post-secondary institutions.  P/Ts’ involvement in PSI-SIF was particularly helpful in cases where the provinces had a large number of funded projects to manage (e.g., Alberta) and in Quebec where regulation requires that any communications with post-secondary institutions go through the province. Most of the institutions interviewed and the case studies reported that the P/Ts’ involvement facilitated the report submission process since the P/Ts handled the review and, in some cases, vetted the projects’ submissions for compliance with federal reporting requirements.

The regional offices also played an important role in program implementation. Their liaison role with the provinces and the support they provided in implementing the program’s communication plan were key to the program, as reported by program management. Interviews with PSI-SIF management, P/T representatives and institutions showed that the program clearly outlined the roles and responsibilities of key players from the outset and that there was no duplication of responsibilities between ISED, P/Ts and institutions.  That opinion was also validated in the case studies.  It was reported, however, that early in the implementation phase the role of ISED regional staff was not clear to the P/Ts, and in some cases, to the regional staff themselves.  The 2017 Management Control Framework Assessment (MCFA) of PSI-SIF recommended that the role of regional staff be further clarified. As a result, the program’s roles and responsibilities and organizational chart were revised to clarify the roles of program staff in both headquarters and regional offices.

3.2.2 Application Processes

The program provided ongoing support to P/Ts and institutions throughout the initial and extension application phases. However, initial application submission timelines were short and eligible project costs were unclear. Moreover, the program’s extension application process was perceived by some institutions as time consuming and did not leverage available project information.

The program was announced on April 6, 2016 and the application deadline was May 9, 2016.  Institutions, therefore, had approximately one month to prepare their application and supporting documents.  Funded projects were announced in July 2016. 302 (46%) of the 645 project applications were approved (See Annex B).  205 projects aimed to improve research and innovation infrastructure, while 136 projects focused on college infrastructure for industry-relevant training and 225 focused on the environmental sustainability of research and innovation infrastructure (See Annex C).

Application guidelines were posted online and the program team was available to answer institutions’ questions during the initial and the program extension application phases, as confirmed by the institutions interviewed.

Despite the support provided by the program staff during the application phase, most institutions reported facing some challenges with the initial application process.  For example, some highlighted that the application completion and submission period was short, especially for those who submitted applications for more than one project.  Even though all the projects submitted during the application phase were expected to be shovel ready, results from interviews and case studies also showed that, due to the short application phase, some projects submitted unrealistic project plans and timelines, which only became apparent after work had started on the projects.

Moreover, some institutions reported that receiving approval on projects in late summer meant they could not capitalize on the construction summer season.  In some provinces, this delay was attributed to the time taken to finalize and approve the contribution agreements between the federal and provincial governments and the sub-agreements between the provinces and institutions.

Most institutions also reported they struggled, during the initial application phase, to understand whether some project costs were deemed eligible or not.  For example, some institutions indicated that they were unclear about the conditions under which internal project management staff costs would be covered.  Once that concern was raised, the program management responded by communicating and posting further clarifications regarding the eligibility of internal employees’ costs.Footnote 13

With respect to the program extension application (almost 50% of the funded projects were extended beyond April 30, 2018), some P/Ts and institutions indicated that it was very detailed, particularly in light of the extent of reporting provided previously for the projects requesting an extension.  Moreover, during the extension application assessment phase the institutions were uncertain of how to proceed with their projects, while awaiting extension decisions.

3.2.3 Monitoring and Reporting

The program’s reporting requirements helped identify projects at risk and enabled project monitoring. However, the frequency of reporting and the level of detail required were considered time consuming by some institutions and P/Ts. Further, in some cases, the submission of the projects’ close-out reports, 90 days after “substantial completion,” was considered insufficient and not aligned with what the institutions were accustomed to.

The program provided reporting templates and guidelines outlining the reporting requirements for both: projects progress and close-out reports from the program’s outset.  The program staff was also always available to answer questions, as indicated by the interviewees. Moreover, the quarterly reports identified the projects at risk in each P/T to the program management, who reported that those reports helped facilitate project monitoring.

Despite the usefulness of the program reporting system, the interviews and case studies identified some reporting challenges.  First, most of the P/T representatives interviewed noted that under the program, their reporting responsibilities to the federal government were more time consuming than expected and expended considerable resources. Some P/Ts added that the level of detail required in the reporting was disproportionate to the project scope, level of funds provided and risk.  Several institutions (mainly those with more than one PSI-SIF funded project) also mentioned that the frequency of PSI-SIF reporting was very time consuming as they were required to report on projects monthly, as requested by the P/Ts and quarterly, as required under PSI-SIF. 

Some institutions also noted that what the projects’ “substantial completion” phase under PSI-SIF includes is different from what they are accustomed to, which involves completing site cleaning before submitting the close-out report.  Under PSI-SIF the contractors were asked to sign close-out reports before they cleared the sites of the completed projects, a process, which usually takes longer than the duration required for the submission of the PSI-SIF close-out report.  It is worth noting that the program accommodated the circumstances of these institutions and did not penalize the late submission of the close-out reports.  

3.2.4 Funding Disbursment Mechanism

The PSI-SIF advance payment mechanism was designed to ensure the projects continue moving forward without delays. However, during implementation, the mechanism posed some cash flow challenges, which had a greater impact on smaller institutions. Also, the mechanism differs from how some of the P/Ts normally manage infrastructure projects, which led to some challenges for them in processing funding disbursements.

Findings from the interviews highlighted a number of benefits for the program’s funding disbursement mechanism.  As a stimulus program, the rationale of PSI-SIF’s advance payment mechanism was to get the projects up and running as quickly as possible and ensure they continue moving forward without delays.

For some institutions, the program funding mechanism also allowed cost adjustments to account for unanticipated project cost increases.   It is worth noting, however, that projects’ contingencies (i.e., unanticipated costs) were considered ineligible at the application phase.  These costs become eligible only when they are realised and justified through a valid rationale (e.g., effect on project’s benefits), P/T approval, and the availability of freed up/reallocated funding under the signed CA.

Furthermore, the program allowed up to 100% coverage of Aboriginal institution project costs and additional funding was approved for Aboriginal institutions, when needed and as available within existing program resources. 

However, the interviews revealed some challenges regarding the program’s funding disbursement mechanism.  As mentioned previously, PSI-SIF applied an advance payment mechanism, paying 12.5% of total program funding per province quarterly, to help projects progress without delay. Where the overall portfolio of projects in a P/T included a number of at risk projects (i.e., not progressing as planned), the P/T received a percentage less than 12.5%.  In these cases, some projects, that were advancing well, experienced cash flow challenges as they ended up receiving less funding than planned per quarter. According to interviewees, this had a greater impact on smaller institutions.

The program’s advance payment mechanism also posed some challenges to some P/Ts as it differs from the claim-based mechanism that they normally use. As reported in the interviews, this added more administrative work to some P/Ts while processing the PSI-SIF funding disbursement.

3.2.5 Efficiency of Program Operations

Generally, PSI-SIF’s actual operation costs were in line with the program plan. The program’s ability to leverage other human capacity during peak operation times allowed the program to efficiently use its operational budget.

To assess the overall efficiency of program delivery, an analysis of the planned versus actual operating costs was conducted, along with an assessment of the allocation of human resources. Table 2 below shows the program’s Operations and Maintenance (O&M) budget over the program’s duration and the actual program spending.

Table 2: PSI-SIF Planned and Actual Operation and Maintenance Costs

Description of Table 2: PSI-SIF Planned and Actual Operation and Maintenance Costs

Table 2 shows the planned and actual operation and maintenance (O & M) costs of the PSI-SIF program, over the period of the program implementation from 2016-17 to 2018-19. Planned costs in 2016-17 was $3.3 million and the actual costs were $2.4 million. In 2017-18 the planned costs were $2.95 million whereas the actual costs were $3.0 million. Finally, in 2018-19 the planned cost was $2.4 million and the actual costs were $2.3 million. The total planned costs of operations over the three years was $8.8 million and the total actual costs were $7.7 million.

PSI-SIF had a planned O&M budget of $8.8M to manage the program.  The program’s actual spending as of March 8, 2019 was $7.7M (i.e., 12% less than planned).Footnote 17 Generally, the actual costs of operations of the program were in line with the operational costs planned.

The flexible use of human resources for program delivery helped the program efficiently use its operational budget. The total number of FTEs identified in the program plan, from 2016-17 to 2018-19, were 23 positionsFootnote 18 These FTEs were fully utilized for program operations.  However, when there was a need in May 2016 to increase the number of human resources to assist with the initial project assessment, 25 staff from elsewhere in the DepartmentFootnote 19 temporarily joined the program.  After the three-week project assessment period, the program returned to 23 FTEs for the remainder of the program.Footnote 20

3.3 Performance

This section analyzes findings from the interviews, case studies and available close-out reports to assess the following:

It is worth noting that the results reported in relation to medium and long-term outcomes in this section are based on the self-reporting of the institutions and P/Ts interviewed and a review of the available close-out reports. Consequently, this is not a comprehensive assessment of the program’s outcomes. It is rather a list of preliminary program outcomes for a sample of funded projects.

3.3.1 Contribution to Improved Research and Innovation and Training Capacity

PSI-SIF equipped institutions with state-of-the-art infrastructure, which is anticipated to improve the research and training capacity, enable the recruitment of more students and allow collaboration and synergies for networking.

Findings from interviews, case studies and close-out reports showed examples of how the PSI-SIF funded projects actually improved research and training capacity at the funded institutions. Below are some of these examples.

Some institutions, on the other hand, highlighted that even though the program outcomes are still not realized yet, because the program is still running, they are expecting the following positive impacts of their funded projects on their research and training capacity.

3.3.2 Contribution to Improved Environmental Sustainability

Institutions anticipate that their new and renovated infrastructure will enable them to save energy, improve air quality and reduce greenhouse gas emissions, which will ultimately help improve the environmental quality of the research and training space.

In their projects’ close-out reports, institutions highlighted that PSI-SIF’s capital investments enabled some to be certified through the Leadership in Energy and Environmental Design (LEED) and is expected to help others save energy and reduce greenhouse gas emissions.  Below are some of the examples reported by funded institutions:

3.3.3 Other Program Outcomes

Some institutions anticipate that their PSI-SIF funded projects will help them create more research jobs, faculty positions and internal administrative and technical jobs. Other institutions anticipate a positive impact of their completed projects on engaging Indigenous and Metis community members and creating job opportunities for students in certain fields.

In the close-out reports and case studies, some institutions expected that their PSI-SIF funded new/renovated infrastructure will lead to the creation of additional research jobs.  However, since the infrastructure projects were recently completed, it is not currently possible to quantify with accuracy the number of research related jobs that would be created.  For example, St. Thomas College stated that at least ten new faculty and researcher positions in various social science and humanity fields can now be accommodated as a result of their PSI-SIF funded project. Lakeland College also reported that, as a result of their dairy learning facility project, they plan to hire one additional faculty member, a full time research scientist and one associate research technician.

In the case studies and close-out reports some institutions highlighted that the program led (or could lead) to the creation of internal jobs, such as technicians, building maintenance staff and administrative personnel.   For example, Concordia University of Edmonton reported that for its Centre for Science, Research and Innovation project, they staffed two senior management positions for programming and anticipate hiring four additional faculty members.

In their close-out reports, some institutions demonstrated that their funded projects will have positive impacts for different target groups such as Indigenous and Metis community members. Moreover, they mentioned other expected long-term impacts, such as providing professional opportunities for students to meet the labour market demands.  For example, Mount Royal University’s Trico Changemakers Studio project enhanced reciprocal relationships between students and indigenous communities in the region. It also provided support to the Indigenous changemakers (i.e., community involved in social and environmental innovation) in need of funding and skills development.

Furthermore, UPEI reported that their renovated psychology research infrastructure will help the university attract more students in the field of psychology.  In turn, that could lead to an increase in the number of psychologists in the province and consequently fill in the labour market gap in PEI, which has the second-lowest number of psychologists per capita among Canadian provinces.

4.0 Conclusions and lesson learned

4.1 Conclusions

Relevance

Generally, federal investment in PSI-SIF helped stimulate the construction sector’s labor market, during an economic downturn.

PSI-SIF also filled a gap in the P/Ts’ capital investment and enabled the implementation of post-secondary institutions’ infrastructure projects that were shovel-ready and in need of funding.  It contributed to the upgrading and renovation of the Canadian post-secondary institutions’ infrastructure, which is approaching the end of its effective life.

Design and delivery

In general, PSI-SIF’s program design and delivery model contributed to the effective implementation of the program’s objectives.  The model leveraged the P/Ts’ knowledge and their established relationship with post-secondary institutions.  The roles and responsibilities under PSI-SIF were generally clear and the program staff was supportive and responsive to the P/Ts’ and institutions questions and needs.  However, the following challenges were faced during program implementation.

First, the initial application submission timelines were short, which resulted in challenges for some applicants.  More clarity was also needed at the beginning regarding the program’s eligible costs.  Furthermore, the program extension application was perceived as time consuming by some institutions and did not leverage the project information already provided in the progress reports. During the extension application assessment phase the institutions were also uncertain of how to proceed with their projects, while awaiting extension decisions.

Second, although the program’s reporting requirements helped identify projects at risk and enabled project monitoring, the P/Ts and institutions faced challenges regarding the frequency of reporting and the level of detail required in the progress reports.  Further, in some cases, the submission of the projects’ close-out reports 90 days after “substantial completion” of the projects was reported to be insufficient, more particularly by the institutions, which are accustomed to submitting these reports after full completion of the projects.  The program was responsive to the needs of those institutions and allowed flexibility around the timing of the close-out report submission.

Third, though the PSI-SIF advance payment mechanism was designed to ensure the projects continue moving forward without delays, during implementation, the mechanism posed some cash flow challenges, which were more impactful for smaller institutions.  Moreover, the mechanism is different from the claim-based mechanism that the P/Ts normally use to manage infrastructure projects, which led to some inefficiency in processing funding disbursements.

Generally, PSI-SIF’s actual operation costs were in line with the program plan.  The program’s ability to leverage other human capacity during peak operation times allowed the program to efficiently use its operational budget.

Performance

PSI-SIF helped equip institutions with state-of-the-art infrastructure, which is anticipated to improve the research and training capacity, enable the recruitment of more students and allow collaboration and synergies for networking.

Institutions anticipate that their new and renovated infrastructure will enable them to save energy, improve air quality and reduce greenhouse emissions, which will ultimately help improve the environmental quality of the research and training space.

Some institutions anticipate that their PSI-SIF funded projects will help them create more research jobs, faculty positions and internal administrative and technical jobs. Other institutions anticipate a positive impact of their completed projects on engaging Indigenous and Metis community members and creating job opportunities for students in certain fields.

4.2 Lesson learned

Overall, there was a demonstrated need for the PSI-SIF program and it achieved its objectives of creating economic activities in Canadian economies and contributing to the development of the research and training infrastructure at post-secondary institutions.  The program’s design and delivery model was effective enough to enable the achievement of the program objectives. However, based on the evaluation findings, for any similar future ISED programs, the following design and delivery elements should be considered.

Lesson learned 1: Involving subject matter experts in the program design phase

Early consultation with the provinces and infrastructure experts during the program’s design phase could help avoid the challenges related to issues in the application process, program start date and duration, eligible costs and substantial completion during program implementation.

Lesson learned 2: Reporting

Making the reporting requirements commensurate with the projects’ risk level and streamlining federal and provincial reporting requirements would bring greater efficiency by decreasing reporting burden for P/Ts and institutions.

Lesson learned 3: Program extension

Planning and communicating the extension of infrastructure projects to funded institutions earlier would help reduce uncertainty and related financial burden on the institutions.

Annex A

Program Governance and Delivery Partners’ Roles and Responsibilities

Organization Roles and responsibilities
ISED – Headquarters
  • Responsible for the Program’s design and national delivery.
  • Provides direction and oversight to those supporting the Program in ISED regional offices and the P/Ts.
  • Review projects’ progress and close out reports and ensure they contain all required information.
  • Establish CAs with P/Ts (13 CAs) and bilateral partners (3 bilateral agreements).
  • With the help of third-party service provider, develop and implement a Risk-Based Audit Framework (RBAF), which articulates the independent monitoring, reporting and auditing function in support of the contribution and bilateral agreements in place.
ISED – RegionsFootnote 21
  • Participate in the application assessment process.
  • Undertake ongoing liaison with institutions.
  • Assist with the administration of funding agreements.
  • Undertake planning and logistics associated with regional communication events.
Provinces/Territories (P/T)
  • Identify priority projects in support of the Minister’s project selection process.
  • Establish and administer CAs with ultimate recipients (post-secondary institutions) in order to flow the PSI-SIF funding and required P/T matching funds where applicable.
  • Review progress, close-out and accountability reporting from the institutions.
Institutions receiving funding
  • Submit quarterly reports, a final close-out report, an independent or P/T audit reports and a certificate of substantial completion at project end to P/Ts to be forwarded to ISED.
Description of Annex A: Program Governance and Delivery Partners’ Roles and Responsibilities

Annex A outlines the roles and responsibilities of the PSI-SIF governance and delivery partners. There are four types of delivery partners or stakeholders associated with PSI-SIF’s governance and delivery:

  1. ISED Headquarters
  2. ISED regional offices
  3. Canadian Provinces and Territories (P/Ts)
  4. Institutions receiving funding

ISED headquarters are responsible for the overall management and monitoring of the program and subsequent contribution agreements (CA), while ISED regions are in charge of the liaison, communication, planning and logistics of the program. In parallel, the P/Ts work with ISED to identify priority projects to support the Minister’s project selection, administer the CAs with the institutions, disburse the funds to institutions (and where applicable, match the federal PSI-SIF funding), and review project progress and reporting. Finally, beneficiary institutions must submit their quarterly and close-out reports to the P/Ts, so that they will be ultimately given to ISED.

Annex B

Number of Funded Projects in Each Province

Provinces and Bilateral Projects Number of Funded Projects
Quebec 101
Ontario 71
Alberta 44
British Columbia 30
Saskatchewan 11
New Brunswick 9
Manitoba 7
Prince Edward Island 8
Nova Scotia 6
Newfoundland & Labrador 5
Northwest Territories 2
Yukon 2
Nunavut 1
Bilateral Projects 5
Total 302Footnote *

Source: Program application data

Description of Annex B: Number of Funded Projects in Each Province

Annex B shows the number of initially approved projects for PSI-SIF funding in each of the provinces and territories (P/Ts). The number of provincial and territorial approved projects range from a low of one project in Nunavut to a high of 101 projects in Quebec. The total number of initially approved projects for PSI-SIF funding was 302. This number may not reflect the final number of funded projects, as some projects may have been merged or cancelled over the course of the program.

Annex C

Objectives of Funded Projects

Provinces Number of Funded Projects with the Objective of Improving Research and Innovation Infrastructure Number of Funded Projects with the Objective of Improving College Infrastructure for Industry-Relevant Training Number of Funded Projects with the Objective of Improving Environmental Sustainability of Research and Innovation Infrastructure
Alberta 35 23 35
British Columbia 14 18 26
Manitoba 7 2 6
New Brunswick 5 3 6
Newfoundland and Labrador 4 4 4
Nova Scotia 5 3 6
Northwest Territories 0 1 2
Nunavut 1 1 1
Ontario 54 32 64
Prince Edward Island 6 5 8
Quebec 67 37 59
Saskatchewan 6 6 7
Yukon 1 1 1
Total 205 136 225

Source: Program application data
Note: Projects can fall under more than one objective.

Description of Annex C : Objectives of Funded Projects

Annex C shows the number of approved projects for PSI-SIF funding in each of the provinces and territories classified by their objective. The total number of approved projects with the objective of improving research and innovation infrastructure was 205 and range from a low of zero projects in the Northwest Territories to a high of 67 in Quebec. The total number of approved projects with the objective of improving college infrastructure for industry-relevant training was 136 and range from a low of one project in the Northwest Territories, Nunavut, and the Yukon to a high of 37 in Quebec. The total number of approved projects with the objective of improving environmental sustainability of research and innovation infrastructure was 225 and range from a low of one in Nunavut and the Yukon to a high of 64 in Ontario.

Date modified: