Language selection

Search

BAI Performance measurement framework - 2018

ABOUT THIS REPORT

In February 2017, a national dialogue on performance measurement for business accelerators and incubators (BAIs) in Canada was held with a nationally representative group of BAIs and policy leaders representing Canada's Department of Innovation, Science and Economic Development (ISED). It was agreed that a nationally standardized performance measurement framework (PMF) would enable BAIs to benchmark their performance and drive improvement, help companies to choose their best options for support, and assist governments at all levels in increasing the effectiveness of their investments in support programs for innovative growth-oriented firms in Canada.

Following the dialogue, an industry-led BAI Steering Committee was launched to provide leadership in designing a standardized measurement and reporting framework and launching a pilot process that will provide BAIs with an opportunity to test and refine the framework before rolling it out on a national basis. This report summarizes the culmination of 8 months of work directed by the BAI Steering Committee. The report outlines the rationale for establishing a national performance measurement framework, describes the process used to develop it and provides a simple logic model that guides the design of the PMF. The report also provides clear definitions for the metrics that the PMF will draw on and describes the approach for collecting, analyzing and reporting the data, including the methodology that will be used by approved researchers to produce the descriptive statistics and econometric analyses that will illuminate the relationship between BAI programs and the economic performance of client firms. Finally, the report details the operations and administration of the performance measurement platform, including processes for obtaining consent to share information and protecting the confidentiality of data. The report was authored by Anthony Williams, president and co-founder of the DEEP Centre, with input from the BAI Steering Committee and working group members identified below.

The Steering Committee consisted of co-chairs Chris Lumb (President & CEO, TEC Edmonton) and Chris Padfield (Director General, Small Business Branch, ISED) and Natalie Dakers (President, AccelRX), Avvey Peters (VP, Partnerships, Communitech), Serge Bourassa (Président & COO, Centre d'entreprises et d'innovation de Montréal), John Stokes (Co-Founder, Founder Fuel), Patrick White (Managing Director, L-Spark), Karen Greve Young (VP Partnerships, MaRS), Jesse Rodgers (CEO, Volta) and James Maynard (CEO, Wavefront).

In addition to the Steering Committee members, contributors to this report included: Jim Valerio (ISED), Shane Dolan (ISED), Christine McKay (ISED), Warren Clarke, (ISED), Elayne Wandler (AccelRX), Jessica Dupuis (Communitech), Eric Lawlor (CEIM), Raymond Luk (Hockeystick), Clint Sieunarine (Hockeystick), Rich Nicholls (Hockeystick), Nicholas Hsu (Innovate Calgary), Carl Gosselin (Lazaridis Institute), Joe Greenwood (MaRS Data Catalyst), Joseph Lalonde (MaRS Data Catalyst), Chris Diaper (TEC Edmonton), Meghann Coleman (Volta), Brian Roberts (Wavefront), and Nikki Arasaki (Wavefront).

PDF version

Table of contents

  1. Mandate and Progress to Date
  2. Performance Measurement Framework
  3. Performance Measurement Platform
  4. Conclusions and Next Steps
  5. Annexes

1. Mandate and Progress to Date

Over the past number of years, public sector organizations have increasingly recognized the need to develop and deploy performance measurement systems to ensure that they have timely, strategically focused, objective and evidence-based information on their performance, in order to produce better results and remain high-performance organizations. With this goal in mind, the Department of Innovation, Science and Economic Development (ISED) has been working a representative group of policy leaders and stakeholders to develop a performance measurement framework for business accelerators and incubators (BAIs) in Canada. This framework will enable these organizations to benchmark their performance and drive improvement, help companies to choose their best options for support, and assist governments at all levels in increasing the effectiveness of public investments in this area.

The work of the BAI community and its partners in government will result in a standardized reporting framework that establishes consistent definitions for job creation, revenue generation, investment and other outcome-related metrics; a common performance measurement platform that will streamline the collection, analysis and reporting of data; and a pilot process to be launched in early 2018 that will provide a small, representative group of BAIs with an opportunity to test and refine the framework before rolling it out on a national basis. Other deliverables include a set of legal agreements to govern the collection and reporting of client data and an agreed process and methodology for analyzing the economic impact of BAIs.

This report is intended to guide the BAI community and its partners in government as they proceed with the next phase of building a national performance measurement solution. It presents the Performance Measurement Framework (PMF) which will form the basis of the 2018 pilot, and documents the progress achieved to date towards developing the analytics and reporting platform.

More specifically, the report:

1.1 Rationale for a National Performance Measurement Framework

The efforts to forge agreement on a national performance measurement solution are premised on a growing appreciation of the value that will be created for Canada's start-up ecosystem, including BAIs, their clients, partners and funders. Indeed, the majority of business accelerators and incubators (BAIs) participating in the feasibility study already collect performance data and issue annual reports to their funders, partners and stakeholders. However, both a national dialogue among BAIs in February 2017 and a subsequent feasibility study revealed that BAIs are currently measuring their performance using a diverse and (often) inconsistent range of metrics and with widely varying levels of success in obtaining data from their clients. A broad consensus emerged that a national framework for performance reporting would generate several key benefits for the BAIs, their clients and the start-up ecosystem as a whole.

The benefits envisioned by the group include:

With these benefits in mind it is worth clarifying how the PMF will be used to inform policy and funding decisions. The purpose of a standardized national measurement framework is to generate consistent and reliable data about the economic impact of BAIs, for the benefit of BAIs, companies seeking BAI support, and governments that fund BAIs. Analysis performed using the data collected from BAI clients during the pilot period – including the production of descriptive statistics and econometric modelling using linked datasets by ISED, Statistics Canada and/or approved researchers – will not be used to evaluate the performance of individual BAIs. It will, however, be used to inform robust conclusions about the role BAI programs play in firm growth and how to most effectively support innovative growth-oriented firms in Canada. For policymakers in particular, the objective is to use the PMF to evaluate the overall effectiveness of national funding programs, identify policy gaps and frame responses that boost the performance of Canada's business support ecosystem.

With respect to future funding applications for individual BAIs, it is expected that BAIs will present their performance data in a manner consistent with the PMF and, when applicable, use the framework to report their performance against specific program funding they receive. In doing so, BAIs and their partners in government can achieve greater alignment on reporting requirements and eventually reach a point where BAIs can enter data points once for multiple audiences and purposes.

In the interest of enabling fair and effective funding and policy decisions, BAIs will need to work closely with governments and other funding partners to interpret the data collected through the PMF. What constitutes high performance for BAIs will always be subject to variations across regions (e.g., population densities, funding models, and proximity to complementary business support services) sectors and level of ecosystem maturity, among other things. It is incumbent upon all stakeholders to recognize that while performance benchmarks and comparisons across ecosystems are useful, caution should be exercised to ensure that data is interpreted using a sophisticated and nuanced approach thattakes context into account. For this reason, the pilot process is designed to enable a representative group of BAIs and policymakers to incrementally test, evaluate and refine the processes for data collection, aggregation, analysis and reporting to ensure that the PMF informs fair and effective decision-making by all relevant stakeholders.

1.2 Designing The Performance Measurement Framework

The process for developing a performance measurement framework for BAIs in Canada has unfolded over two stages to date, with stage three (the pilot) set to begin in the Spring of 2018. An evaluation of the pilot will take place in Q2/3 of 2019, following two data collection and reporting cycles. However, data collection and reporting activities will continue in parallel, with a national rollout planned for September 2019.

Figure 1: Developing a national performance measurement framework for BAIs

Developing a national performance measurement framework for BAIs (the long description is located below the image)
Description of figure 1

BAI Engagement & Dialogue Feasibility Study Pilot Program Pilot Evaluation & Reporting National Rollout

  1. BAI Engagement (Autumn 2016). During the Autumn 2016, the Department of Innovation, Science and Economic Development (ISED) met with representatives from Canada's entrepreneur support ecosystem at three informal roundtables to discuss how government, entrepreneurs, investors and industry could work together to establish a national performance measurement framework for Business Accelerators and Incubators (BAIs). During the consultations, stakeholders expressed a willingness to collaborate on a national scale and, more specifically, to advance a national discussionon best practices in performance measurement.

    National Dialogue (February 2017). With the support of ISED, an industry-led group of BAIs took the next step toward framing a national solution for data collection and performance reporting by hosting a national discussion on February 10th in Toronto. Representatives from 18 organizations spent the day discussing BAI activities, opportunities and challenges, and exploring the benefits and challenges of creating a national performance measurement framework. The session provided a baseline understanding of what metrics BAIs are currently collecting and for what reasons, and helped to develop a shared understanding of how a national performance measurement framework could provide value to the community and its funders, as well as the challenges and obstacles that would need to be overcome to make industry collaboration successful.

  2. Steering Committee and Feasibility Study (April–December 2017). In April 2017, a BAI Steering Committee consisting of a representative group of BAI leaders and policy makers formed to continue an inclusive national discussion and provide leadership in crafting a national performance measurement solution that works for the BAI community and its partners in government (see Appendix D for a list of Steering Committee and various Working Group members). The Committee's overarching mandate is to work alongside the Government of Canada in partnership to a) establish a performance measurement framework, and b) pilot a performance measurement platform for BAIs. It's first step was to initiate a feasibility study, during which the Committee drafted a list of common performance metrics and selected a platform for data collection and reporting. With the platform selected and metrics defined, the Committee proceeded to upload a set of test data to the platform with the objective of assessing the appropriateness of this metrics list for the pilot, and refining processes for data sharing and analysis (see section 1.3 below for a brief summary of the key findings from the feasibility study).
  3. Pilot Project (March 2018–April 2019). The primary output of the Committee is a Performance Measurement Framework (PMF) and platform that will be in operation as a pilot by March 2018. To date, a provider has been selected for the platform that will be used for the pilot, the measurement framework has been refined, and the Steering Committee is in the process of inviting BAIs to participate in the pilot with the understanding that the pilot group should reflect the diversity of programming models and services offered by BAIs, along with the key economic sectors and regions of Canada. Beginning in March 2018, participating BAIs will complete their organizational profiles and upload their company benchmark and 2017 program data to the data sharing platform (see sections 2 and 3 for details. Depending on their internal data collection processes, participating BAIs will proceed to upload 2018 program data, either in quarterly batches throughout 2018 or for the entire annum during the first quarter of 2019.
  4. Pilot Evaluation and Reporting (April–Sept 2019). In the April to September 2019 period, ISED will produce a report analyzing the data collected from participating BAIs. This report will detail, and where possible demonstrate, the types of descriptive and econometric analyses are possible using the data and other linked datasets from Statistics Canada and other federal partners. In parallel, participating BAIs and government partners will work together to evaluate the pilot process and identify valuable lessons and insights for managing a national performance measurement process with an expanded group of BAI participants. As part of the evaluation, BAIs will be consulted on how best to govern and manage the performance measurement framework and reporting process on an ongoing basis.
  5. National Rollout (Sept 2019 and beyond). Following the pilot and evaluation phase, the Steering Committee and its partners in government will prepare for a national rollout of the PMF. Key activities will include making necessary adjustments to the PMF and platform and further encouraging government programs (federally and provincially) to adopt the metrics defined in the PMF to assess BAI programs. With these foundational steps in place, the BAI Steering Committee and its partners government can begin to invite BAIs from across the country to join in contributing their data to the performance measurement platform. As always, the Steering Committee will continually monitor and make adjustments to the PMF to ensure it remains useful to its stakeholders.
Table 1: Activity Breakdown – BAI Engagement, Feasibility Study, Pilot and Rollout Phases
ACTIVITY DETAILS
1. BAI Engagement & National Dialogue (Fall 2016 to March 2017)
  • Convene a national discussion on best practices in performance measurement.
  • Consult BAIs on their willingness to develop a national performance measurement framework.
  • Host national dialogue to share best practices on BAI performance measurement.
  • Enlist a small, but representative group of BAIs to proceed with a feasibility study.
2. Steering Committee & Feasibility Study (April to Dec 2017)
  • Establish BAI Steering Committee and working groups.
  • Forge agreement on standardized metrics, measurement tools and platform.
  • Test run of data collection and reporting with Hockeystick platform.
  • Report on PMF progress and learnings to date.
3. Pilot Program (March 2018 to April 2019)
  • Enlist a nationally representative group of BAIs to participate in the pilot.
  • Pilot and test the performance measurement framework and data collection platform/process over two collection and reporting cycles (i.e., 2017 and 2018 BAI program cohort/entrant data).
    • March - Dec 2018: BAIs complete organizational profile and upload company benchmark and 2017 and 2018 BAI program cohort/entrant data.
    • Jan-April 2019: BAIs upload 2019 Q1 BAI program cohort/entrant data for those with a quarterly d. BAIs that collect data on an annual cycle will upload their 2018 data in Q1 of 2019.   
4. Pilot Evaluation and Reporting (April to Sept 2019)
  • Produce analysis and BAI performance report.
  • Identify opportunities, challenges and tips for managing the data collection and reporting process with expanded number of BAI participants.
  • Calibrate performance metrics and processes based on insights and lessons learned from the pilot.
  • Gather feedback on the suitability of the data sharing platform for subsequent phases.
5. National Rollout (Sept 2019 and beyond)
  • Make necessary adjustments to the PMF and platform.
  • Further encourage government programs to adopt the metrics defined in the PMF to assess BAI programs
  • Formalize governance/stewardship.
  • Recruit additional BAIs to participate in the national rollout.
  • Continually monitor and make adjustments to the PMF to ensure it remains useful to its stakeholders.

1.3 Key Outputs and Lessons Learned from the Feasibility Study

The feasibility study conducted between April and December 2017 provided the members of the BAI Steering Committee with an opportunity to draft a list of common performance metrics, select a platform for data collection and reporting, and upload a set of test data to the platform with the objective of assessing the appropriateness of this metrics list for the pilot, and refining the processes for data sharing. Along the way, the Committee reflected on the broader purpose and objectives of a national performance measurement framework and problem-solved a variety of technical and operational issues that were deemed likely to arise during the pilot. What follows is a very brief overview of some of the key discussion points and lessons learned during the feasibility study.

Defining a measurement framework. Among the first tasks for the Committee was defining a common set of performance metrics and complementary survey instruments for data collection. The results of this exercise are outlined in section 2 (performance metrics) and in appendixes A and B (the questionnaires for data collection). A consensus was reached that the performance metrics would focus on a core set of financial indicators linked to the annual revenues, employment, capital raised and intellectual property portfolio of client companies. A key challenge was arriving at common definitions for these indicators. While BAIs generally track the same outcomes (e.g., client revenues, employment and investment), they do so using different methods and differing indicators. Considerable time and effort went into crafting acceptable definitions for each metric. For example, how to define a job and which number of hours constitutes full-time employment; how to parse differing types of investment capital; and whether to track indicators using a calendar year or fiscal year. Other challenges included defining an approach to collecting information about founder demographics and establishing a common industry/sector list to ensure that both BAI programs and client firms could be categorized the same way. The effort to reach consensus on these issues had ensured that BAIs participating in the pilot will be tracking the same indicators using the same definitions and methods over the same time period.

Selecting a data sharing platform. With respect data collection and storage, the Committee determined that the pilot data will be aggregated into a secure central platform that complements established platforms/processes used by mature BAIs. For BAIs, the data platform solution had to be low-cost, secure, convenient and useful. For BAIs with existing CRM solutions and data management systems, it was also important that participation in the pilot would not require BAIs to transition to a new platform. For the federal government partners, it was important that the platform provide a secure and trusted environment for hosting and visualizing data on servers located in Canada, as well as a secure data export function to provide approved analysts with convenient access to data. With these criteria in mind, the Committee considered several available data platforms and, for a variety of reasons discussed further in section 3, chose Hockeystick as the data sharing platform for the pilot.

Designing an approach to data collection and analysis. Having defined metrics and selected a platform, the Committee proceeded to work on designing the data collection process and determining how data collected during the pilot will be analyzed. The details of the data collection process are outlined in section 2, however, it was determined that data would be collected from client companies upon entry to a BAI program to establish a benchmark, and for two full reporting cycles covering 2017 and 2018 BAI program cohort/entrant data. With regard to data analysis, the Committee agreed on three fundamental points. First, that the purpose of the data analysis is to draw robust conclusions about the economic impact of BAIs, for the benefit of BAIs, companies seeking BAI support, and governments that fund BAIs. Second, in the interest of enabling longitudinal analysis, strict protocols to protect client confidentiality will be followed to link the data collected to Statistics Canada and other Government of Canada sources using client names and business numbers. Third, that data analysis should be performed by a reliable, committed party capable of consistent interpretation of data. For the purpose of the pilot, it was determined that ISED will manage the data analysis and reporting process and work in partnership with Statistics Canada and approved researchers.

Performing a test run and planning for the pilot. With the platform selected and metrics defined, the Committee proceeded to upload a set of test data to the platform with the objective of assessing the appropriateness of the metrics list for the pilot, and refining the processes for data collection. Following a successful test run, key challenges at this stage included tweaking the metric definitions and survey instruments based on feedback from the participants; working with Hockeystick to clearly define the process for collecting, aggregating and exporting client data with a larger group of BAIs; deciding precisely how and with whom client data will be shared; determining how and when to obtain client consent to aggregate data using Hockeystick and to share data with the federal government for the purpose of research; and communicating the rationale and objectives of the pilot to the broader BAI community, while enlisting a larger group of organizations to participate. The results of these various activities and discussions are detailed in the remainder of the document.

1.4 Desired Outcomes from the Pilot

The Logic Model, shown in Figure 2, identifies the linkages between the activities of the PMF Pilot and the achievement of its results/outcomes. In addition to a number of immediate and longer-term outcomes, the Logic Model highlights four overarching objectives for the PMF Pilot. These include:

Figure 2: Logic model for the PMF pilot

Chart illustrating the Logic model for the PMF pilot (the long description is located below the image)
Description of figure 2
Logic model for the PMF pilot
Input & Participants Activities Outputs Immediate Outcomes Longer-term
Outcomes
Ultimate Outcome
  • Representative group of business accelerators and incubators (BAIs) from across Canada;
  • Federal government partners;
  • BAI client firms.
  • Define performance measurement framework (PMF) with common metrics and definitions;
  • Select data sharing platform for aggregating and reporting data;
  • Enlist BAI participants for pilot of the PMF and data sharing platform;
  • Obtain client consent to store data on the platform and share it with government partners.
  • Collect and upload data for common performance indicators from client firms;
  • Aggregate and collate performance data on data sharing platform;
  • Export data to Statistics Canada for trial run of econometric analysis and preparation of descriptive statistics;
  • Evaluate pilot outcome and consult with pilot participants on lessons learned.
  • Standardized metrics and definitions are calibrated based on BAI/government feedback;
  • Processes for collecting, storing, analyzing and reporting data are refined;
  • Opportunities, challenges and tips for managing the process with an expanded number of BAI participants are identified;
  • BAI Steering Committee and government partners initiate discussions about a national rollout.
  • A national baseline for longitudinal data collection is established;
  • Econometric analysis produces data driven insights into the role of BAIs in firm growth;
  • Data insights help BAIs refine program offerings to better support firms;
  • Ecosystem data highlights opportunities for collaboration and synergy between BAIs and other partners;
  • Performance measurement processes are streamlined across multiple levels of government.

Superior economic performance is enabled by the enhanced growth of accelerated firms as a result of systematic improvements across the ecosystem:

  • BAIs can benchmark their performance and drive improvement;
  • Companies can choose their best options for support;
  • Government as all levels can increase the effectiveness of public investments in BAIs.

As the Logic Model indicates, the inputs and participants for the PMF clients include a representative group of BAIs and the programs and services they currently offer to client firms, federal government partners (particularly ISED) and the data sharing platform supplier. The activities are centred mainly on establishing the PMF, identifying a platform for the collection and aggregation of client data, enlisting BAIs to participate in the pilot, and obtaining consent from client firms to use their data for the pilot. The inputs and activities represent the foundational steps taken by BAI Steering Committee and ISED to create a PMF and prepare for a test run using the data sharing platform.

The logic model indicates that the outputs are linked to the pilot process itself. It is expected that participating BAIs will work with their client firms to collect all of the data specified by the performance measurement framework (or request their client firms to upload the data directly) over two data collection and reporting cycles: one for 2017 BAI program cohort/entrant data and one for 2018 BAI program cohort/entrant data. In both instances, the data sharing platform provider will aggregate and collate the data and prepare it for export. ISED and Statistics Canada will download the data according to the protocols defined in Section 2 and produce a report. Following these steps, all parties will participate in an evaluation of the pilot outcomes and document the lessons learned.

As depicted in the logic model, the pilot phase is expected produce a number of short and long-term outcomes as the BAI community works towards the realization of a national performance measurement solution for Canada. Immediate outcomes will be observable at the conclusion of the pilot in Spring 2019. Medium to long-term outcomes for the pilot are expected to take a year or more to manifest, with some outcomes – such as the use of data-driven insights to refine BAI programming and government investment priorities – taking several years to come to fruition.

The immediate outcomes for the pilot include:

Medium to long-term outcomes for the pilot include:

In summary, a national performance measurement framework will provide BAIs with reliable and comparable data on which to make sound decisions, as well as timely information on the relevance, success and cost-effectiveness of their programs and activities. A common evaluation framework and reporting process will also ease the administrative burden on BAIs, while providing governments and other funders with a rigorous and objective evidence base with which to assess the performance of BAIs and make informed resource allocation decisions. Entrepreneurs across the country will be better equipped to identify the best options for support at various stages of their entrepreneurial journey.

The work Canadian BAIs are doing together at the national level will result in many other tangible benefits for BAIs, their clients and the broader economy. These benefits include the ability to share best practices across institutions and jurisdictions, establish relevant performance benchmarks for different regions and sectors, enhance input into public policy, and position and promote Canada as a destination for start-up activity. Ultimately, this work will inform BAIs and the Government on how to most effectively support innovative growth-oriented firms in Canada and these insights, in turn, will help accelerate the growth of world class companies in a variety of high value sectors. In other words, the collaborative efforts to build a common system for measuring the performance of BAIs could herald the beginning of an exciting new chapter in the growth and evolution of Canada's start-up ecosystem.

Date modified: