Horizontal Evaluation of the Community Futures Program

From: Innovation, Science and Economic Development Canada

June 2019

This publication is available online at http://www.ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03899.html.

To obtain a copy of this publication, or to receive it in an alternate format (Braille, large print, etc.), please fill out the Publication Request Form at www.ic.gc.ca/publication-request or contact:

ISED Citizen Services Centre
Innovation, Science and Economic Development Canada
C.D. Howe Building
235 Queen Street
Ottawa, ON K1A 0H5
Canada

Telephone (toll-free in Canada):
1-800-328-6189
Telephone (international):
613-954-5031
TTY (for hearing impaired):
1-866-694-8389
Business hours:
8:30 a.m. to 5:00 p.m. (Eastern Time)
Email:
ISED@Canada.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from the Department of Industry, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that the Department of Industry is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, or as having been made in affiliation with, or with the endorsement of, the Department of Industry.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the ISED Citizen Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Industry, 2019.

Cat. No. Iu4-269/2019E-PDF
ISBN 978-0-660-32616-0

Aussi offert en français sous le titre Évaluation Horizontale du Programme de Développement des Collectivités.

List of acronyms

ACOA
Atlantic Canada Opportunities Agency
BDC
Business Development Corporation
CBDC
Community Business Development Corporations
CBPP
Cross-Border Partnership Program
CED-Q
Canada Economic Development Agency for Quebec Regions
CFDC
Community Futures Development Corporation
CFO
Community Futures Organization
CFP
Community Futures Program
CFs
Community Futures
FedDev
Federal Economic Development Agency for Southern Ontario
FedNor
Federal Economic Development Initiative for Northern Ontario
GBA+
Gender-Based Analysis Plus
ISED
Innovation, Science and Economic Development
PM Strategy
Performance Measurement Strategy
RDA
Regional Development Agency
SMEs
Small and Medium Enterprises
WD
Western Economic Diversification

List of tables

List of figures


Executive summary

Program overview

The Community Futures Program (CFP) supports economic development in primarily rural areas across Canada through a network of 256 Community Futures Organizations (CFOs). CFOs are independent, not-for-profit organizations governed by a volunteer board comprised of community members. They provide direct support to local communities in four areas: business financing, business support services, community economic development, and strategic planning.

The CFP is part of the ministerial portfolio of Innovation, Science and Economic Development Canada (ISED). It is administered by four Regional Development Agencies (RDAs) (the Atlantic Canada Opportunities Agency (ACOA), the Canada Economic Development Agency for Quebec Regions (CED-Q), the Federal Economic Development Agency for Southern Ontario (FedDev) and Western Economic Diversification Canada (WD)) and the Federal Economic Development Initiative for Northern Ontario (FedNor) (referred to throughout the report as the RDAs).

Evaluation objective and methodology

The evaluation provides a national assessment of the performance of the CFP between 2013-14 and 2017-18, and of the relevance of the program. This is the first national evaluation of the program since the CFP was in its pilot phase in 1990. ISED's Audit and Evaluation Branch led this horizontal evaluation working in close collaboration with the RDAs. The evaluation included six data collection methods: a document and literature review, a data review, key informant interviews, case studies and surveys of CFOs and their clients. Data was collected between May and August 2018.

Findings

Relevance

Economic development in rural communities is complex and depends on a number of interrelated factors. Rural communities are facing a range of challenges. The evaluation determined that there is a strong need for continued federal economic development programming in the communities served by CFOs, particularly smaller and more remote communities. There is also a clear need for the specific services the CFOs provide through CFP funding in rural areas. CFP funding supports initiatives aligned with federal government priorities, including the Innovation and Skills Plan.

Performance: Effectiveness

The CFP has contributed to the achievement of many of its intended outcomes, especially those related to business financing and support services. These results include strengthened business practices, economic stability, growth, job creation, and diversification of rural economies. The program's design, largely its flexibility in allowing CFOs to adapt their activities to respond to local needs, facilitates achievement of results. However, it is difficult to assess the outcomes related to community economic development and strategic planning at a national level (in the regions in which it is implemented), as there is variation in how these activities are implemented, and how performance of these program components are measured across the country, and in how their outcomes are measured.

While considerations for inclusiveness (Gender-based Analysis Plus [ GBA+]), currently required of all federal programs, have yet to be formally integrated into the CFP, the program has a variety of approaches in place to meet the needs of different identity groups such as women, Indigenous people, youth, immigrants and people with disabilities. The RDAs have collaborated to develop a national Performance Measurement Strategy (PM Strategy) for the CFP. Its use of Statistics Canada data to provide a national picture of the program's outcomes for small businesses can be viewed as a best practice. However, there are still gaps in aligning regional performance measures to the national strategy and in capturing the results of community economic development and strategic planning.

At the community level, the evaluation determined that CFOs effectively collaborate with a wide range of organizations and implement a range of services that are adapted to local needs and potential. However, there may be opportunities to increase national consistency in specific areas, such as performance measurement. There is also an opportunity for a deeper examination of other aspects of program implementation that might benefit from greater consistency, such as funding models, and where flexibility should be maintained to respond to regional needs.

Performance: Efficiency

The CFP is generally implemented efficiently. It continues to achieve its primary objectives despite declining real value of its program funding. Some program elements, such as the ongoing contribution of the CFOs' volunteer boards and shared services help to support the cost-efficiency of program delivery. However, limited comparable data across RDAs and CFOs precluded a full assessment of program efficiency.

Recommendations

The recommendations aim to acknowledge the importance of regional delivery of the CFP to address local needs and unique capacities, while identifying opportunities to improve the ability to report its relevance and performance at a national level. The RDAs and FedNor should build upon their efforts to provide a comprehensive national picture of the results of the CFP, to the extent possible, by:

1.0 Introduction

1.1 Report overview

This report presents the results of a national horizontal evaluation of the Community Futures Program (CFP) conducted in 2018. The CFP is part of the ministerial portfolio of Innovation, Science and Economic Development (ISED). It is administered by four Regional Development Agencies (RDAs): the Atlantic Canada Opportunities Agency (ACOA); the Economic Development Agency of Canada for the Regions of Quebec (CED-Q); the Federal Economic Development Agency for Southern Ontario (FedDev); and Western Economic Diversification Canada (WD), as well as the Federal Economic Development Initiative for Northern Ontario (FedNor) (referred to throughout the report as RDAs)Footnote 1. The purpose of the evaluation is to assess the relevance and performance of the CFP. The report is organized into four sections:

1.2 The community futures program

CFP at a glance:

The CFP helps primarily rural communities to build their capacity and self-reliance by diversifying their economic base, and by using local solutions to address their unique challengesFootnote 2. The funding provided through the CFP supports communities in pursuing three primary objectives:

The CFP was established in 1985Footnote 4 under the Department of Human Resources and Development Canada. In 1995, the administration of the program was transferred to the RDAsFootnote 5; and in 2015, ministerial oversight of the RDAs, and by extension the CFP, was transferred to the Minister of ISED. However, the primary responsibility for the CFP remains with the RDAs, which provide funding to 256 Community Futures Organizations (CFOs) to support their economic development programmingFootnote 6. Figure 1 shows how the CFOs are distributed by region.

Figure 1: CFP-supported CFOs, by RegionFootnote 7

Text version

Figure 1 visually maps out all the Community Futures Organizations (CFO) funded by the Community Futures Program (CFP), grouped by their respective Regional Development Agencies (RDA). Overall, there are 256 CFP-funded CFOs across 10 Canadian provinces. The number of local CFOs vary from 89 in the West (includes British-Colombia, Alberta, Saskatchewan and Manitoba), to 24 in Northern Ontario and 37 in Southern Ontario, 57 plus 10 Business Development Centres in Quebec, and 39 in the Atlantic Region (includes New Brunswick, Nova Scotia, Newfoundland and Labrador as well as Prince Edward Island).

CFOs are independent, incorporated, not-for profit organizations that are overseen by a volunteer board of directors composed of community members. They are important actors in their communities for planning and economic development, as well as a source of business financing and support in regions where there might otherwise be limited capacity in these areasFootnote 8.

With the support of CFP funding, CFOs provide direct support to local businesses and communities in four areas outlined in the program's terms and conditions:

While the funding provided through the RDAs supports the provision of all these activities, the actual funding provided by CFOs to SMEs comes from locally controlled investment funds established by each CFO. In addition, access to these programs and services, including offerings and eligibility criteria, vary by CFOFootnote 10.

While the CFOs offer the actual services outlined in the CFP terms and conditions, the responsibilities of the RDAs include:

The RDAs receive CFP annual ongoing (A-base) funding to provide financial support to the CFOs. Since 2010, $78.9 million in CFP funding has been allocated annually to the RDAsFootnote 12. The amount of funding provided to each CFO varies by region, with some RDAs providing additional funding to support activities included in the CFP mandate. Federal funding to the CFOs largely supports their ongoing operating costs. The estimated budget of each RDA (and FedNor) for the CFP, by year of the evaluation period, is summarized in Table 1.

Table 1: Program budget over evaluation period, by RDA/FedNor ($millions)Footnote 13
- ACOA CED-Q FedNor FedDev WD Total
- Total Base funding Other funding Total Total Total Base funding Other funding Total Base funding Other funding Total
2013-14 13.2 21.0 8.0 29.0 8.8 11.3 24.6 5.8 30.4 78.9 13.8 92.7
2014-15 13.2 21.0 8.0 29.0 8.8 11.3 24.6 5.8 30.4 78.9 13.8 92.7
2015-16 13.2 21.0 8.0 29.0 8.8 11.3 24.6 5.8 30.4 78.9 13.8 92.7
2016-17 13.2 21.0 8.0 29.0 8.8 11.3 24.6 5.5 30.1 78.9 13.5 92.4
2017-18 13.2 21.0 8.0 29.0 8.8 11.3 24.6 5.5 30.1 78.9 13.5 92.4

†Totals may not add up due to rounding.

Source: Internal program data.

Note: Budget amounts include funding for Grants & Contributions and Operations & Maintenance

Text version

Table 1 presents all of the CFP funding over the evaluation period (2013-2018), by RDA in millions of dollars. While the Atlantic Canada Opportunities Agency (ACOA), Federal Economic Development Initiative for Northern Ontario (FedNor) and Federal Economic Development Agency for Southern Ontario (FedDev) only provided their total funding amounts, Canada Economic Development Agency for Quebec Regions (CED-Q) and Western Economic Diversification (WD) provided a breakdown of funding by base funding and other funding. It is worth noting that amounts provided in this table include the amounts for Grants and Contributions (Gs&Cs) as well as Operations and Maintenance (O&M). As of fiscal year 2017-18, regional annual funding amounts ranged from $13.2M for ACOA to $29M for CED-Q, $8.8M for FedNor, $11.3M for FedDev and $30.1 M for WD. Moreover, as of 2017-18, the annual funding for all RDAs totalized $92.4M.

1.3 Rural Economic Development Landscape

Economic development in rural communities is complex and depends on a number of interrelated and inseparable issues. To start with, defining the term "rural" presents a challenge. Typically, rural communities are considered municipalities located outside census metropolitan areas (CMAs). According to Statistics Canada, a CMA is formed by one or more adjacent municipalities centred on a population centre of at least 10,000 people (known as the core). All areas inside the CMA that are not population centres are rural areasFootnote 14. Rural Canada is extremely diverse in terms of population size, density and degree of remoteness or proximity to urban centresFootnote 15. A rural township outside of the Greater Toronto Area faces different issues than a similarly sized municipality in northern Alberta or southern New Brunswick. Furthermore, "cities" are defined differently by province. For example, any area that is qualified as a city in the province of Alberta must have a population that exceeds 10,000 people, along with other criteria including the size of land parcels of the area's buildings. On the other hand, in British Columbia, a community can qualify as a city with just a population of over 5,000 residents, even if this number drops in the futureFootnote 16.

The proportion of Canadians who live in rural areas has been dropping and in 2016 remained below one in five Canadians (18.7%)Footnote 17. In addition, rural communities have fewer young adults than the rest of Canada. Young people (15 to 34) represent about one-quarter of the country's population. However, only 15 per cent of Canada's young people live in rural CanadaFootnote 18.

Rural communities are also experiencing high out-migration of youthFootnote 19, and face challenges in attracting newcomers, which impedes the attraction and retention of new talent and causes frequent skilled labour shortages. As a result, labour shortages are considered among the main barriers to investment and innovation. Canada's rural regions generally experience limited economic diversification, in particular those that are single industry or natural resource dependent. Nationally, rural jobs are typically focused in industries with declining employment (e.g., rural employment in the goods-producing sector declined by 8 per cent from 2011-2018)Footnote 20. In addition, a much higher share of rural residents is employed in primary industries and construction. As such, rural areas are significantly impacted by commodity price fluctuations and shocksFootnote 21. Moreover, they are prone to suffer from global economic shifts related to the emergence of new international competitors or to global economic crisisFootnote 22.

The lack of broadband connectivity in rural communities (only 38 per cent of rural households have access to high speed internet)Footnote 23 hampers their technology adoption and competitiveness, impedes their ability to attract businesses, and further limits their access to regional, national and international markets. Rural communities are continuing to experience growing gaps in infrastructure. The more remote a rural community is from an urban centre, the more likely they will experience higher transportation costs and lower access to services.

World and Canadian economies are transforming and this is having an impact on societies, economies, regions and the nature of workFootnote 24. With this changing global and national landscape, Canada's rural communities have been increasingly vulnerable to demographic, economic, technological and infrastructure challenges.

It is clear that rural challenges and opportunities are not uniform across Canada and differ from one region to another. The following section provides an overview of how the CFP is delivered in each region.

ACOA

Since 1995, ACOA has used the CFP to fund most of the activities of Community Business Development Corporations (CBDCs) in the four Atlantic provinces (New Brunswick, Nova Scotia, Prince Edward Island, and Newfoundland and Labrador). There are currently 39 CBDCs funded by the CFPFootnote 25 in the region serving a total population of 1.6 million people in hundreds of communities ranging from small remote villages to large functional economic regions. CFP funding also supports the operations of four CBDC provincial associations and a regional association.

Services

Under the CFP, the 39 CBDCs in the Atlantic region receive operational funding to support core staff and expenses. Each CBDC manages an independent investment fund and they have access to additional capital from a central investment fund administered by the Atlantic Association of CBDCs. CBDCs use investment funds to provide access to capital for SMEs. CBDCs also provide counselling, business management skills training, and advice. Unlike other RDAs, ACOA does not use the CFP to fund community economic development and strategic planning due to the availability of other programming including that of provincial governments, not-for-profit organizations, as well as other ACOA programming (e.g., Business Development Program).

Funding

Over the evaluation period, ACOA received $13.2 million a year in CFP funding. ACOA's funding model bases the amount of support provided to individual CBDCs on each organization's loan activity levels and financial needs. ACOA uses other programs, such as the Business Development Program and the Innovative Communities Fund, to support specific CBDC activities. Most CBDCs also receive support from other federal departments and provincial governments to deliver on special initiatives, such as the Self-Employment Benefit Program (provincial governments) and Workplace Digital Skills training (federal Office of Small and Medium Enterprises). However, the number of CBDCs receiving support through any given initiative varies.

Associations

The CFP supports CBDC associations in the four Atlantic provinces along with an Atlantic association. Such support permits the associations to offer a suite of services, including coordination and liaison, marketing, training, and cost saving initiatives such as group purchasing of information technology supports and benefits programs.

CED-Q

CED-Q has delivered the CFP in the province of Québec since 1995. It uses the program to fund two types of development organizations: Community Futures Development Corporations (CFDCs) and Business Development Centres. They serve 97 communitiesFootnote 26 with a population of up to 4.3 million.

Services

As in other regions, each CFDC manages an independent investment fund that provides local and regional SMEs with access to capital. The CFDCs also provide SMEs with technical assistance such as advice, information services, and consultations. At the community level, CFDCs provide technical and financial assistance to support the creation of local economic development plans and initiatives in partnership with other community stakeholders. CFP support to the Business Development Centres in Québec is only used to fund the business advisory services these centres provide to entrepreneurs.

Funding

Over the evaluation period, CED-Q received $21 million a year from the CFP. It also used $8 million a year from CED-Q's ongoing (A-base) federal allocations to provide additional funding to CFDCs and Business Development Centres, bringing the CFP total budget to $29 million a year. Under CED-Q's funding model, each CFDC received $340,362 in addition to $60,000 from the federal Youth Employment Strategy. The Business Development Centres received $181,785 a year to cover operational costs.

Associations

The Réseau des SADC also receives $2 million a year to implement special initiatives (Student Employment, Shared Communications Services, and Local Economic Development Projects). It coordinates the distribution of this fund to CFDCs and Business Development Centres based on the merit of projects submitted. Some CFDCs and Business Development Centres receive additional funding to ensure that their service delivery complies with Official Languages Act requirements.

FedNor

Since 1995, ISED (formerly Industry Canada) has had responsibility for the delivery of the CFP in Northern Ontario through FedNor. Through the CFP, it supports 24 CFDCs in Northern Ontario. CFDCs supported by FedNor serve a total population of almost 700,000 spread across 149 municipalities and 105 Indigenous communities. CFDC catchment areas in Northern Ontario range in size from about 4,000 to 127,000 residents.

Services

FedNor's support for the operating costs of CFDCs allow them to deliver programming in the four areas of the CFP: providing business counselling and information services to SMEs and social enterprises; providing access to capital through investment funds for new and existing SMEs and social enterprises; fostering strategic community planning and socio-economic development; and supporting the implementation of community-based projects and special initiatives.

Funding

FedNor's annual budget for the CFP over the evaluation period was $8.8 million per year. Under FedNor's funding formula, each CFDC is provided $300,000 annually to fund its base operations. Additional allocations may be added to support remote regions and/or official languages requirements. In addition to the CFP, CFDCs are eligible to receive funding from FedNor's Northern Ontario Development Program, when required, to capitalize local investment funds, and to support a range of local development activities. In addition, CFDCs are also eligible for funding from the Economic Development Initiative to support business and economic development activities in Northern Ontario's francophone communities. As independent, incorporated not-for-profit organizations, CFDCs can also apply for funding from other federal or provincial programs, such as the Northern Ontario Heritage Fund Corporation and the Ontario Trillium Foundation.

Associations

FedNor provides support through the CFP to two regional CFDC networks, as well as the Ontario provincial association. These associations allow for communication and collaboration among members, sharing of services (such as online training), and group purchases to achieve economies of scale.

FedDev

In 2009, responsibility for the CFP in Southern Ontario was transferred from Industry Canada to the newly created Federal Economic Development Agency for Southern Ontario (FedDev). Between 1995 and 2009, the CFDCs in Southern Ontario received their CFP funding through FedNor. FedDev currently provides CFP funding to 37 CFDCs in rural Southern OntarioFootnote 27. These CFDCs serve a total population of just over 3 million people in 288 communities ranging in size from about 16,000 to 145,000.

Services

CFDCs supported by the CFP in Southern Ontario include the full range of CFP services: business financing through local investment funds, business information and services, strategic planning, and community economic development projects and partnerships.

Funding

FedDev had annual CFP funding of $11.3 million over the evaluation period. Ninety percent of the Agency's CFP funding is directly allocated to supporting the operational costs of the CFDCs. Forty percent of this funding is allocated based on the achievements of outcomes over the previous three years, while five percent is allocated based on the achievement of annual Key Performance Indicators. The balance is attributed to core funding which consists of base funding of $150,000 per CFDC; an additional $20,000 for a total of $170,000 to CFDCs that have service areas that include francophone populations, and a portion of funding to CFDCs that serve a large geographical area. The remaining ten percent of CFP funding is allocated to support individual CFDCs with approved community-based projects ($500,000), and $370,000 is allocated to support CF Associations.

The amount of outcome-based funding for each CFDC is determined each year for the upcoming fiscal year based on a rolling three-year average. Individual CFDC performance is assessed against the overall performance of the CFP in Southern Ontario based on a series of indicators that include loan performance, numbers of businesses assisted and jobs created, and the amount of money leveraged through community partnerships.

Southern Ontario CFDCs have also received funding from other FedDev programming, and can apply for various provincial programs including the Ontario Trillium Foundation. CFDCs east of Toronto have also received support under the Agency's Eastern Ontario Development Program (EODP).

To further support operational costs, CFDCs can access funding by transferring a portion of their investment fund into their operational fund, up to a maximum of $100,000, to cover eligible expenses.

Associations

FedDev supports three CFDC associations in Southern Ontario through the CFP. These include the Western Ontario CFDC Association, the Eastern Ontario CFDC Network, and the Ontario Association of CFDCs.

WD

WD has delivered the CFP in Western Canada (Manitoba, Saskatchewan, Alberta, and British Columbia) since 1995. Through the CFP, WD provides core support to 89 CFOs (of the 90 CFOs in the region). These CFOs serve a total population of about 4.5 million people. The population of areas covered by western CFOs ranges from about 4,000 to 220,000Footnote 28.

Services

CFOs supported by WD provide the full range of CFP services, including business financing through local investment funds, business information and services, strategic planning, and community economic development projects and partnerships.

Funding

Over the evaluation period, WD provided on average $28.08 million annually in core operating funding to the 89 CFOs, four CF Associations, and the CF Pan West Association in western Canada. Of this amount, $22.67 million is part of the national program allocation, while the remainder is additional funding from WD to support the program and ensure universal coverage across the region. In the West, CFO funding is based on different models in different provinces, including a formula that considers socio-economic and geographical factors in Manitoba and British Columbia.

WD has also entered into separate agreements with the CFOs and the CF Associations to deliver on other initiatives outside of the CFP. Examples include: the Entrepreneurs with Disabilities Program, Indigenous Business Development Services, and the Churchill Economic Development Fund.

Associations

As indicated, WD supports four provincial CF Associations which provide common services and support to their member CFOs, along with the Community Futures Pan West Association which provides leadership and training to CFOs, and support to the CF Associations in the four western provinces.

1.4 Program logic

The logic model (Figure 2) depicts the activities pursued, the outputs produced, and the expected outcomes resulting from the delivery of the CFP. However, some components may not apply in all regions given the variations in implementation.

Figure 2: CFP Logic ModelFootnote 29

Text version

The CFP logic model (Figure 2) depicts the activities pursued, the outputs produced, and the expected outcomes resulting from the delivery of the CFP. However, some components may not apply in all regions given the variations in implementation. For instance, ACOA does not deliver community economic development (CED), projects and partnerships, or strategic planning program components.

Inputs

From bottom up, the CFP work with inputs such as Operations and Maintenance (O&M), Salaries and Wages as well as Grants and Contributions (Gs&Cs) to deliver on its activities.

Activities

The CFP delivers two different streams of activities: development and maintenance and program delivery.

  • Development and Maintenance (Stream 1): On the development and maintenance side, the program provides different activities such as program development, planning and program management, monitoring and provides non-financial support to CFs organizations and measures CF organization performance and allocates funding.
  • Program Delivery (Stream 2): On the program delivery side, CFP provides funding to CFOs.
Development and Maintenance Outputs (Stream 1)

As a result of the development and maintenance activities, the program outputs are the delivery of policies, plans, reports and program tools, and communications resources, advice information and support as well as funding adjustments. In turn, those three outputs enable the program delivery outputs.

Program Delivery Outputs (Stream 2)
GC Outputs

The Government of Canada (GC) level program delivery output is the capitalization and operating contributions with CF organizations to support community economic development through strategic community planning and implementation, business services and access to capital.

CFO Outputs

Subsequently to GoC outputs, the CFO activities lead to four distinct outputs: business financing, business development knowledge and skills, community strategic plan as well as CED projects and partnerships.

Immediate Outcomes

Program delivery outputs therefore will facilitate the implementation of the four immediate outcomes: improved access to capital and leveraged capital through loans, loan guarantees and equity investments to businesses and social enterprises, enhanced/maintained business development services – information counselling, referrals, training, strengthened community strategic planning and more effective implementation of CED through projects, partnerships and other community economic development initiatives.

Intermediate Outcomes

Likewise, immediate outcomes will enable the intermediate outcomes of improved business practices and increased entrepreneurship, strengthened and expanded businesses and Strengthened community capacity for socio-economic development.

Ultimate Outcomes

Ultimately, the intermediate outcomes are expected to lead to the program's ultimate outcomes of economic stability, growth and job creation, diversified and competitive local rural economies and economically sustainable communities.

2.0 Methodology

This section provides an overview of the methodology used to evaluate the CFP. It defines the scope of the evaluation, as well as the range of methods used to gather relevant data and information that address the evaluation issues and questions.

2.1 Evaluation approach

This evaluation was centrally led and coordinated by ISED's Audit and Evaluation Branch, working in close collaboration with the RDAs. PRA Inc. supported this process, designing data collection instruments, conducting a large part of the data collection, and analyzing and reporting on all lines of evidence. Additional data collection was conducted by ISED, the RDAs, and EKOS Research Associates Inc.

The evaluation included a combination of qualitative and quantitative methods that allowed for triangulation of findings. The goal was to capture a broad national perspective, while acknowledging regional and community diversity. The evaluation was guided by a working group comprised of ISED and RDA evaluators and program staff, as well as internal and external advisory committees which provided feedback at key points in the evaluation.

2.2 Objective and scope

The evaluation provides a national assessment of the performance of the CFP during the period from 2013-14 to 2017-18, as well as of the program's relevance. Regional evaluations have been completed by RDAs every five years, as required by Treasury Board evaluation policies. However, the integration of the RDAs (and thus of the CFP) into a single ministerial portfolio provided an opportunity to undertake an evaluation at the national level, which was last done when the CFP was in a pilot phase in 1990.

ISED led the evaluation in accordance with the Treasury Board of Canada's Directive on ResultsFootnote 30, and in alignment with the objectives set out in the CFP PM StrategyFootnote 31. The Directive on Results requires that all federal evaluations include government-wide policy considerations, where relevant, such as Gender-Based Analysis Plus (GBA+). GBA+ considers how different intersecting identify factors such as gender, Indigeneity, geography, age, disability, and ethnic background influence how diverse groups experience government programsFootnote 32. These considerations informed the overall evaluation approach, the evaluation framework, the design of various methods (especially the case studies), the data collection instruments, and the selection of key informants to be interviewed.

The evaluation scope was developed to look at the issues of relevance and performance of the CFP within the mandate of the program to provide business and lending services and support community economic development in rural Canada. However, given that the relevance of the program has been well established through past evaluations, and the multi-dimensional approaches to its program implementation across the regions, greater emphasis was placed on evaluating the performance of the CFP at the national level in this evaluation. In addition, given that this is a national horizontal evaluation, it has focused on common issues rather than regional specificities.

2.3 Evaluation issues and questions

Based on the program's National PM StrategyFootnote 33 and subsequent consultations with ISED, the RDAs, FedNor, and the Community Futures Network of Canada, the evaluation addressed the following questions:

Relevance

  1. To what extent is the CFP addressing current needs?
  2. To what extent does the CFP align with government priorities?

Performance: Effectiveness

  1. To what extent have the CFP's immediate, intermediate, and ultimate outcomes been achieved in the medium and longer term?
  2. To what extent does the design of the CFP address inclusive growth (including considerations for different identity groups, such as immigrants, and those based on gender, Indigeneity, age, linguistic background, and disability)?
  3. To what extent are the CFP performance measurement and reporting structures effective in reporting on the achievement of the CFP outcomes?
  4. How is the CFP being implemented on an ongoing basis?

Performance: Efficiency

  1. Is the CFP being managed and delivered effectively and efficiently?

The complete evaluation matrix, including indicators and associated data sources, can be found in Appendix A.

2.4 Data Collection methods

Multiple lines of evidence were used to address the evaluation questions. The data collection methods included a document review, a data review, key informant interviews, case studies, and surveys of CFOs and their clients.

PRA conducted the document and data reviews, many of the key informant interviews, and the case studies for one region. Evaluation specialists from ISED and the RDAs conducted the remaining interviews and case studies. PRA designed the surveys, and EKOS conducted them.

Document and literature review

The document review included key documents provided by each region and ISED, such as PM Strategy documents, previous evaluations and reviews, contribution agreements, and manuals and guidelines. The document review primarily supported questions of program relevance, achievement of outcomes, and implementation. In addition, a literature review was undertaken to supplement the document review.

Data review

The data review examined relevant data provided by RDAs, including reports from Statistics Canada. A national analysis of each topic was undertaken where possible, accompanied by regional analysis where feasible and needed. The data review primarily supported the analysis of achievement of program outcomes.

Key Informant Interviews

Semi-structured interviews were conducted by the RDAs, ISED, and PRA. The interviews supported all areas of the evaluation. In total, 69 interviews were conducted with 78 individuals from several stakeholder groups. The number of interviews per group is presented in Table 2.

Table 2: Number of key informant interviews by stakeholder group
Stakeholder Group Number of Interviews Number of Individuals
RDA/FedNor staff 9 13
ISED staff 1 1
Business Development Bank of Canada 1 6
Subject matter experts 3 3
CFO association representatives 8 8
CFO staff 13 13
Provincial government representatives 6 6
Organizations representing different identity groups 10 10
Additional regional partners and stakeholders 9 9
Total 69 78
Text version

Table 2 highlights the breakdown of key informant interviews conducted as part of the Horizontal Evaluation of the CFP, by stakeholder group. The different stakeholders interviewed as part of the research process are: RDA/FedNor staff, ISED staff, Business Development Bank of Canada representatives, subject matter experts, CFO association representatives, CFO staff, provincial government representatives, individuals from organizations representing different identity groups and other additional regional partners and stakeholders. Overall, a total of 69 interviews were conducted with 78 key individuals.

Case studies

A total of 15 case studies, three per region, were conducted (see Appendix B). They focussed on CFO support to community economic development, strategic initiatives, or federal government priorities. Case study research included interviews and document and data review. Evaluators from ACOA, CED-Q, and FedDev conducted the case studies for their respective organizations. Evaluators from ISED conducted the FedNor case studies and PRA conducted the WD case studies.

Client survey

The CFO client survey was conducted by EKOS Research Associates Inc. EKOS received contact information for approximately 27,000 CFO clients. In total, 1,234 clients participated in the survey (616 from Québec, and 618 from the rest of Canada), both by phone (1,194), and online (40). To compensate for the oversampling in QuébecFootnote 34, results were weighted by region to ensure that the sample reflects the broader population of clients. Quotas were set by region and service type. In Québec, loan clients were oversampled to be consistent with previous research. The margin of error associated with the total sample is +/- 2.7%, 19 times out of 20. The telephone response rate for this survey was 14% (19% in Québec and 10% in the rest of Canada). Table 3 presents a summary of the type of client surveyed, by region.

Table 3: Client survey respondents by type of client and region
Type of Client Atlantic Québec Ontario West
Northern Ontario Southern Ontario All of OntarioFootnote *
Loan 58 413 13 19 34 67
Advisory 90 102 39 94 148 169
Community Economic Development NAFootnote 35 101 22 14 38 17
Total 148 616 74 127 220 253
Text version

Table 3 illustrates the breakdown of CFO clients surveyed as part of the CFP evaluation in each of the five Canadian regions covered by the CFP. The numbers are divided by client type: loan, advisory and community economic development clients. Overall, the total number of surveyed clients ranged from 74 for FedNor to 616 for CED-Q. It is worth mentioning, however, that the totals for Northern and Southern Ontario do not sum up to the all of Ontario total, as some survey respondents indicated that they did not know which region they received services in.

CFO survey

The CFO survey was also conducted by EKOS. They contacted representatives of all 256 CFOs receiving CFP funding. In total, 206 respondents (representing 206 CFOs) participated either by telephone (195) or online (11). The respondents represented all regions, with 73 from the Western provinces; 55 from Québec; 49 from Ontario (19 from Northern Ontario and 30 from Southern Ontario); and 29 from the Atlantic provinces. The surveys primarily supported evaluation questions related to relevance and performance.

2.5 Limitations

Changes to the program

Program changes were implemented throughout the evaluation period (nationally and regionally), which may have influenced stakeholder views, and data may not necessarily reflect the current state of the program. During interviews, evaluators explored the impact of these changes, and triangulation of multiple lines of evidence was used to help mitigate this challenge.

Stakeholders have a vested interest in the program

Most key informants and survey respondents have a vested interest in the CFP, which could have led to positive bias in responses. Where possible, this limitation was mitigated by triangulating all lines of evidence and responses across the various groups of stakeholders.

Regional program implementation

The evaluation was conducted nationally in keeping with the program's national terms and conditions and PM Strategy. However, the program is implemented differently in each region and the RDAs each collect program data using their own tools and reporting definitions. As a result, indicators may differ between regions. Furthermore, the context in which the program is implemented varies from community to community, which can impact the types of activities undertaken by each CFO.

To provide a national examination of the program, available evidence was examined to find commonalities across regions (and communities) with regional differences highlighted, where applicable.

Data collected by multiple organizations

Several organizations (PRA, ISED, the RDAs, and EKOS) collected data for the evaluation. To support consistency, PRA held a workshop to discuss the data collection instruments and approach.

Longer term impact

Results from community economic development and strategic planning are typically longer term and are therefore challenging to track on an annual basis, as is possible, with other CFP-related activities. In addition, since they typically involve multiple partners, it can be difficult to attribute results to the efforts of CFOs.

Scope of evaluation

As noted in the overview of the evaluation scope, the exploration of the relevance of the program focused on the relevance of the program's activities within its mandate to provide business and lending services and support community economic development. This evaluation focussed on developing an understanding of the performance of the program at a national level. As such, the program's response to rural emerging needs related to demographic, economic, technological, and infrastructure challenges, was not systematically explored in the evaluation, or integrated into the data collection instruments or approach.

3.0 Findings

3.1 Relevance

3.1.1 To what extent is the CFP addressing current needs?

Key Finding: Economic development in rural communities is complex and depends on a number of factors. Rural communities are facing a range of challenges. There is a strong need for continued federal economic development programming for the communities served by CFOs, particularly smaller or more remote communities. There is also clear need for the unique services the CFP provides in rural areas.

Broad community needs

The economic challenges facing many of the communities served by the CFP are closely tied to broader national and international economic trends. For example, low economic diversification and economic reliance on natural resources leave some communities vulnerable to economic shocks, such as recessions or changes in resource prices or demandFootnote 36. Furthermore, findings from both interviews and the document review indicated that population trends, such as out-migration (particularly of youth) and aging populations, can lead to shortages in labour, knowledge, and expertise; and exacerbate issues related to low population densities (such as small client bases). Interviews further suggest that these challenges are compounded by difficulty in attracting new immigrants to rural areas and can contribute to a contraction in economic growth.

The importance of the CFP in meeting broader community needs is supported by the expected impacts that surveys, interviews, and case studies suggested could arise in the absence of the CFP, including:

Similarly, survey and case study results showed the impact of the absence of the CFP would likely be greater on smaller and more remote communities with higher needs, such as those in Northern Ontario, parts of Québec, Newfoundland and Labrador, or Cape Breton. These communities face limited access to markets, lack of infrastructure, or higher transportation costsFootnote 37. Typically, they also have limited in-person access to government programs and may face connectivity barriers to accessing them online. Case studies showed that some CFOs, such as the Témiscamingue CFDC in Québec, have responded to these challenges by supporting worker retention, market development strategies, and immigration.

Furthermore, interviews and case studies indicated that CFP funding plays a key role in helping communities through economic challenges. For example, many rural communities are single-industry communities, and a lack of economic diversification has made these communities (e.g., Churchill, Manitoba) particularly vulnerable to commodity shocks and changes in global demand. In some communities, community economic development projects have been implemented to address the effects of these types of shocks. A more complete discussion of the community economic development component, including some specific examples, can be found in Section 3.2.1.

In addition, other broader community issues were raised during interviews and corroborated by the literature review. While addressing such challenges has not been an objective of the CFP, these factors may influence the ability of the program to achieve its intended outcomes, and are important components of the challenges faced in some of the communities served by the CFP. As a result, consideration could be given to undertaking a more in-depth review of the policy objectives of the suite of existing federal rural programming, including CFP, to inform any future policy and program changes.

Continued need for business and lending services

Overall, interviews and previous evaluations demonstrate that CFP business services respond to the needs of local entrepreneursFootnote 38. Access to financing is a core need addressed by the CFP across the country. Many entrepreneurs in rural communities do not qualify for financing from traditional banks or credit unions (for various reasons, including a lack of credit, insufficient experience in business, or inadequate property to leverage). Furthermore, traditional lenders may not have a physical presence in some rural communities and may therefore not be considered as a viable option by some prospective clients.

CFO funding to local businesses and entrepreneurs is unique in many ways compared to what is available from traditional lenders and other government programs. CFOs typically base their funding decisions on the potential for long-term growth and community impact, which differs from other lending programs (such as the Canada Small Business Financing Program and the programs offered by the Business Development Bank of Canada). While small businesses can apply to any of these programs (as well as loans offered by conventional lenders), previous researchFootnote 39 suggests that it is unlikely that a given business would be approved by more than one lending program, as the programs have different eligibility criteria, risk tolerance levels, and lending rules. For this reason, these various funding programs have been typically considered to be largely complementaryFootnote 40.

Other potential funding sources for entrepreneurs are provided by provincial governments or organizations, though many of these have narrower focuses than CFO lending, including providing loans to specific target groups, or funding specific activities. As such, these programs are also largely complementary to the CFP. Examples of such programs are included, by region, in Table 4.

Table 4: Examples of provincial business support programs

Atlantic

Newfoundland
Business Retention and Expansion
Prince Edward Island
Economic Growth Program
Nova Scotia
Regional Enterprise Networks
New Brunswick
Miramichi Regional Economic Development and Innovation Fund
New Brunswick
Northern New Brunswick Economic Development and Innovation Fund

West

Pan West
Western Innovation Initiative
Manitoba
Community Economic Development Fund
Saskatchewan
Saskatchewan Economic Development Alliance
British Columbia
Competitiveness Consulting Rebate
British Columbia
Business Retention and Expansion

Québec

  • Québec Economic Development Program

Ontario

  • Indigenous Economic Development Fund
  • Business Retention and Expansion Program
  • Northern Ontario Heritage Fund Corporation
  • Rural Economic Development Program
  • Small Business Enterprise Centres
  • Eastern Ontario Development Fund
  • Ontario Trillium Foundation
  • Southwestern Ontario Development Program
  • Jobs and Prosperity Fund

The extent of the need for each type of service, and the context in which those services are offered, varies by community. The flexibility of the CFP allows for the services offered to be tailored to the needs and characteristics of each community, and is generally considered a key strength of the program. Interviews suggest that the CFP is unique in its extensive presence in rural areas, face-to-face service provision, and ability to take on greater risk in providing loans. CFOs collaborate with other organizations that deliver loans or other economic development programs to ensure complementarity and limit overlap. CFOs also leverage financial and in-kind resources from partners to complement their other sources of funding.

The access to financing provided by the CFOs is an important aspect of the CFP. As previously noted, there are a variety of different funding programs that exist and mechanisms to apply for a loan (i.e., advances in the digital economy and in the financial industry and several institutions now offer loans online). While previous research suggests that businesses are likely to be approved for only one program based on different eligibility criteria, survey results provided additional insights on the positioning of CFOs' lending services relative to other funding sources that might be available in their communities. Some loan recipients indicated that they could have received funding from other sources. Furthermore, CFO clients were asked if they had applied to another funding source. Of those that had (n=332), 43% received funds from both the CFP and another lender, while 7% were approved for a loan from another lender but decided to obtain a loan from their CFO instead. Of those that applied for other funding (n=332), most applied to commercial lenders (90%). Other funders applied to other federal government programs (10%); provincial government programs (10%); or municipal government programs (8%). The gap between funding needs and availability may therefore not be as clear as other lines of evidence would suggest.

Findings from interviews indicate that a number of factors may explain why CFO lending services are preferred to other funding sources. For instance, some entrepreneurs, including those whose businesses have grown enough that they would be eligible to receive bank loans, choose to come back to their CFO for expansion funding, rather than going to other lenders, because they appreciate working with CFOs and have developed positive relationships with CFO staff. Others may have a preference for in-person services that would not be available through other lenders. There are also cases of leveraging effect, whereby other funding options are contingent on receiving a CFO loan. Differences in funding conditions (e.g., interest rate) might also be influencing these decisions.

In other cases, evaluation findings confirm that CFO lending services are the only available option. Results from the survey of CFO clients indicate that, among those who did not apply for other funding (n=379), less than half noted that it was either somewhat or very likely that they would have been approved for a loan from another lender had they applied.

In summary, for some rural communities, having access to capital through a local CFO remains an important component of the CFP. However, the evaluation determined that in some regions there are other complementary programs available.

Interviews and case studies highlighted that CFOs also provide business development services such as business planning, training and mentoring, financial literacy development, and counselling. These services help to fill knowledge gaps that might exist among entrepreneurs. The types of services offered vary by CFO to help address the specific needs of each community. In some cases, business services may be provided that meet the needs of specific groups of entrepreneurs, including youth, Indigenous people, or women, such as the Women in Business Initiative (see box).

The Women in Business Initiative, from the Atlantic region, is designed to improve the growth and competitiveness of women-owned businesses, and to increase their representation in emerging growth sectors. The Initiative offers a variety of programs, including support tailored to the needs of emerging Indigenous entrepreneurs, through collaboration with the Joint Economic Development Initiative, and Ulnooweg.

Unmet needs

In addition to the broader community issue challenges noted previously, interview, case study, and survey findings highlighted some needs that the CFP is not able to adequately meet. Specifically, some CFOs have reduced or stopped engaging in community economic development or strategic planning projects due to limited financial resources. That is not the case in Québec, where CFOs are required to spend a minimum of $22,000 a year on local community development projects. Performance data (presented in Table 7, in Section 3.2.1), further supports that at least some CFOs continue to implement these components. However, it is therefore possible that the scope of these activities has changed, rather than their number, or that only some CFOs have stopped engaging in these activities, though this cannot be determined with existing data on community economic development activities.

More broadly, some CFOs have been limited in their ability to meet community needs due to limited financial resources, whether generally or for specific program components, or due to the capacity of current personnel (given their current workload). This also contributes to outstanding unmet needs for specific identity groups, including women or Indigenous entrepreneursFootnote 41. These limitations were raised in interviews, case studies, and the CFO survey. It reflects the fact that the funding for the CFP has remained at the same level since 2010.

3.1.2 To what extent does the CFP align with government priorities?

Key Finding: The CFP is aligned with federal government priorities set out in the Innovation and Skills Plan, and those related to inclusiveness and clean technology.

Findings from the document review, interviews, and CFO survey showed that the CFP aligns with federal government priorities. It aligns with the Innovation and Skills Plan (introduced as part of Budget 2017), which is "an ambitious effort to make Canada a world-leading centre for innovation; to help create more good, well-paying jobs; and to help strengthen and grow the middle class."Footnote 42 To ensure the Plan's success, the government is mobilizing the RDAs through the development of Regional Growth Strategies, which address four pillars including: Regional Innovation Ecosystems; Investment in and Scale-Up of Firms; Clean Growth; and Community Economic Development and DiversificationFootnote 43.

As one of the RDAs' economic development programs, the CFP's intended outcomes are also aligned with the Plan, including "strengthened and expanded businesses;" "strengthened community capacity for socio-economic development;" "economic stability, growth and job creation;" "diversified and competitive local rural economies;" and "economically sustainable communities."Footnote 44 Some CFOs also implement specific supports for businesses seeking to adopt innovative business practicesFootnote 45.

The flexibility of the CFP also allows CFOs to be mobilized to support evolving federal government priorities. For example, interview findings and document review identified how some CFOs are supporting the federal government's clean technology objectives through CBDC Clean Technology Loans in Atlantic Canada, and an investment target in Southern Ontario to encourage lending in this area. The City of Prince George has collaborated with the Fraser-Fort George CFO in BC and local businesses to identify and enhance clean technology products and services. Further, multiple lines of evidence provide examples of programs or initiatives implemented by a number of CFOs that support inclusiveness (see Section 3.2.2 for further discussion and examples).

Renfrew CFDC (Southern Ontario) worked with Bishop Water Technologies, a company offering environmentally friendly wastewater treatment options. In 2014, the CFDC provided a loan to help the company relocate from Eganville to Renfrew. It also helped the company access funding for business development and marketing plans and identify markets for expansion.

3.2 Performance: Effectiveness

3.2.1 To what extent have the CFP's immediate, intermediate, and ultimate outcomes been achieved in the medium and longer term?

Key Finding: The CFP has contributed to the achievement of many of its intended outcomes, including improved access to capital; strengthened business practices; economic stability, growth, and job creation; and diversification of rural economies. The achievement of these outcomes is facilitated by the program's design, principally its flexibility, which allows the CFOs to adapt their activities to best meet the needs of their communities. However, it is difficult to assess outcomes related to community economic development and strategic planning, as these activities do not translate to easily measurable performance indicators, the outcomes are longer term and there is variation in their implementation across the country.

Findings from all lines of evidence demonstrate that the CFP has been generally successful in achieving its intended outcomes. Evaluation findings indicate that this success is largely due to the overall program approach of developing and implementing local solutions to local problems. The linking of loans and business support services is also considered an important success factor.

Loans

Over the five years covered by the evaluation, CFOs supported by the CFP each year approved, on average, over 5,000 loans worth more than $250 million from their investment funds (CFP funds are used to support loan administration and business services). As shown in Table 5, the number and average size of loans varies by region, likely due to differences in regional contexts. Findings from the client survey and some case studies showed that these loans make an important contribution to improving access to capital. As discussed earlier, half of the CFO clients surveyed said that they would have been unlikely to obtain a loan from another source. When asked about the primary results from CFO services, 28% of clients said that they helped to start a new business, and 14% of clients said that they helped to keep a business openFootnote 46.

Table 5: Number and value of loans approved by CFOs by region (2013-14 to 2017-18)
Region Loans 2013-14 2014-15 2015-16 2016-17 2017-18 Annual Average Annual Average per CFO
ACOA Number 1,330 1,411 1,354 1,333 1,342 1,354 35
Total $ $60,875,761 $69,691,568 $70,696,033 $74,009,250 $70,037,626 $69,062,048 $1,770,821
CED-Q NumberFootnote * 1,130 1,121 1,980 1,408 1,388 1,387 24
Total $ $47,549,332 $39,375,993 $39,347,338 $56,007,973 $41,217,477 $44,709,223 $784,372
FedDev Number 785 726 760 728 693 738 20
Total $ $46,841,221 $43,051,397 $48,935,129 $45,220,290 $50,272,722 $46,864,152 $1,266,599
FedNor Number 381 520 422 431 432 437 18
Total $ $17,066,322 $29,264,601 $20,369,940 $21,915,568 $23,575,884 $22,438,463 $934,935
WD Number 1,425 1,417 1,330 1,396 1,295 1,373 15
Total $ $72,272,000 $75,551,374 $73,304,273 $79,012,821 $67,653,875 $73,588,868 $826,504

Source: Internal program data.

Text version

Table 5 presents the number and value of approved loans by region, by fiscal year (for the evaluation period). The annual breakdown of loans and loan values per region are key program indicators, as they measure one of the program's main line of business (access to capital for small- and medium-sized businesses and social enterprises). The table also illustrates the annual average number of loans and loan amounts by region and by CFO. Annual average of loan numbers range from a low of 437 annual loans in Northern Ontario (FedNor), to a high of 1387 in Quebec (CED-Q), whereas the annual loan average amount by region range from $22.4M in Northern Ontario (FedNor) to $73.6M in the West (WD). In terms of the CFO breakdown average loan number and value per year, it varies from 15 loans per year and $0.826M in annual loan values in the West (WD) to an average of 35 loans per year per CFO and $1.7M in annual loan values per CFO in the Atlantic (ACOA). For analytical purposes, it is important to mention that in Quebec, the number of loans is measured by the number of active business files.

Business services

CFOs also offer business support services such as business planning, training and mentoring, financial literacy development, and counselling. These services are offered to both loan and non-loan clients. Administrative data showed that, over the past five years, CFOs provided support services to more than 47,000 clients each year, ranging from an average of 63 clients per CFO in Québec to 264 in the West, as shown in Table 6. While there is variation in the scope of services offered by CFOs in each region, the variations in the data suggest significant differences in the way business services clients are reported. Since business support services are automatically provided to most loan clients as a fundamental component of the CFO lending approach, these services may be underreported.

Table 6: Number of Business Services Clients (2013-14 to 2017-18) non-loan unless otherwise notedFootnote 47
- 2013-14 2014-15 2015-16 2016-17 2017-18 Annual Average Annual Average per CFO
ACOAFootnote * 6,211 6,888 7,475 7,913 8,792 7,456 191
CED-QFootnote * 3,683 4,539 4,072 4,468 4,235 4,199 63
FedNor 3,142Footnote * 2,876Footnote * 2,758Footnote * 3,472 2,519 2,953 123
FedDev 8,924 10,334 8,894 8,297 8,682 9,026 244
WD 30,906 21,370 22,397 21,745 21,213 23,526 264

Source: Internal program data.

Text version

Table 6 demonstrates the annual breakdown of business services clients that did not receive any CFP loans, for all five Regional Development Agencies (RDA) and for the period covered by the evaluation. It is worth noting, however, that the ACOA, CED-Q and FedNor clients may also have received loans. The table also provides the annual average of business services clients over the evaluation period as well as the annual number of business services clients per CFO for each of the RDA. The annual RDA averages vary from 2,953 advisory clients for FedNor to 23,526 for WD, whereas the annual averages of advisory clients per CFO range from 63 for CED-Q to 264 clients for WD.

According to interviews, this approach of combining loans with business support services is a fundamental strength of the CFP, contributing to the overall success of the program. When asked about the primary results of the CFO services they received, 24% of clients surveyed said they developed their business skills and 14% said they improved their business practices. Furthermore, almost all CFOs surveyed noted that the CFP makes an important contribution to strengthened and expanded businesses (98%), and improved business practices and increased entrepreneurship (91%). Impacts on business survival and employment growth noted through both interviews and the Statistics Canada comparative analysis are also indicative of impact in these areas.

Thus, overall, multiple lines of evidence (interviews, surveys, and Statistics Canada data analysis) indicate that the CFP's support for CFO lending and business services contribute to the program's expected intermediate outcomes of improved business practices and increased entrepreneurship, and strengthened and expanded businesses.

Longer-term results

To assess the longer-term impacts of the CFP, the RDAs engaged Statistics Canada to examine the outcomes of firms that received CFO loans against a comparable group of non-assisted firmsFootnote 48. What the Statistics Canada dataFootnote 49 has consistentlyFootnote 50 shown is that CFP-assisted firms had significantly better outcomes in fostering growth and job creation.

Average annual employment growth for the CFP-assisted businesses was 3.4% greater than non-assisted businesses. There was some variation across regions, ranging from 1.5% in Québec to 5.1% in the West. (Figure 3)Footnote 51.

Figure 3: Average annual employment growth of CFP-assisted and non-assisted businesses by region.
Text version

Figure 3 is a bar diagram that draws a comparison between the average annual employment growth of CFP-assisted and non-assisted businesses by region. Overall, for all of the five regions, the correlation is largely positive, as CFP-assisted businesses have much higher employment growth compared to the non-CFP-assisted Canadian businesses. From East to West, annual employment percentages fluctuate from almost 4% of annual employment growth for CFP-assisted business and close to 0% for non-assisted businesses in the Atlantic region (ACOA), to close to 8% of employment growth for assisted businesses in the West (WD) and less than 3% for non-assisted businesses.

In addition, CFP-assisted businesses stayed in business longer than the non-assisted firms, based on the difference in survival rates. Nationally, the difference in survival rate increased over time, from 8% higher for CFP-assisted firms after one year since start-up, to 26% higher for CFP-assisted firms after nine years (Figure 4). Businesses that survive longer are an important contributor to economic stability and sustainability.

Figure 4: Annual survival rates of CFP-assisted and non-assisted businesses in all regions.
Text version

Similar to Figure 3, Figure 4 is a linear diagram that shows the annual survival rates of CFP-assisted in comparison with the survival rates of non-assisted businesses in all five Canadian regions. On one side, the CFP-assisted firms, survival rates are about 90% during the first start-up year and diminishes to a low of 60%, nine years after the start-up. On the other side, non-assisted start-ups survival rates are just over 80% in the start-up year and decrease to almost 30% after nine years.

Furthermore, CFP-assisted businesses had, nationally, 3.3% higher average annual sales growth than the non-assisted businesses. Regionally, the difference in sales growth varied from 1.8% in Québec to 5.1% in the West, further contributing to economic growth in communities served by a CFO (Figure 5).

Figure 5: Average annual sales growth of CFP-assisted and non-assisted businesses by region.
Text version

Figure 5 is a bar diagram that compares the average annual sales growth of CFP-assisted and non-assisted businesses by region. In all five RDAs, CFP-assisted businesses show significantly higher annual sales growth than for the non-assisted businesses ranging from 5% average annual sale growth for CED-Q, compared to 3% for non-assisted Quebec, to WD's CFP-assisted firms have an average annual sales growth of 12% whereas the annual sales growth for non-assisted firms is 6%.

Interview findings and previous evaluations suggest that local delivery and ownership of the CFP facilitates the broader impact of the program on communities. Decisions on business loans are made by boards composed of community members who take into consideration overall community priorities and interests. For example, a CBDC in the Atlantic region had been approached to fund an economic development initiative on land that the community was in the process of having environmentally protected. In keeping with this community priority, the funding was not approved.

Further, case study and interview findings indicate that the CFP's flexibility facilitates its work in supporting the diversification of rural economies. The flexibility that CFOs typically have in their lending criteria helps to support a broader loan portfolio (e.g., including clients from a broader range of sectors). It also allows the program to take on greater risk, in turn allowing it to provide loans to businesses that pursue new markets or new technologies.

Data from Statistics Canada analyses provide some evidence on the range of sectors supported through CFP funding. In particular, the data demonstrates that a greater proportion of CFP-supported businesses are in the manufacturing and retail sectors than non-assisted firms, while a higher proportion of non-assisted firms are in agriculture, a traditionally rural sector. While further analysis would be required, this could provide an indication that the CFP is contributing to rural diversificationFootnote 52.

As such, the CFP is considered by stakeholders to contribute to the CFP's expected ultimate outcomes of economic stability, growth, and job creation in supported communities; the creation of more economically sustainable communities; and the development of a more diversified and competitive local rural economy.

Community Economic Development

In addition to direct support to businesses through financing and business services, the CFP funding provided to CFOs also helps to support community economic development initiativesFootnote 53. Typically, these activities support community development in a way that promotes self-reliance, based on principles that include the following:

Depending on the region, CFOs can fund community economic development projects, either through their regular CFP funding or by applying for funding for special initiatives. The requirements in these areas differ by region.

As with all components of the CFP, CFOs can tailor community economic development in their region to meet their needs. As such, the scope of activities funded varies significantly, and can include:

The expected outcomes of the community economic development component of the CFP are not as clearly defined as the financing and business services components. As previously noted, some key informants said that CFOs have reduced or largely stopped engaging in community economic development projects due to insufficient funding. However, there are many community economic development projects that have been undertaken across four of the regions (since the community economic development component is not implemented in the Atlantic provinces), as demonstrated in Table 7. There is some variation in the number of projects being undertaken in each of these regions (from an annual average number of new projects per CFO of eight in Northern Ontario to 15 in the West). However, it should be noted that differences between these regions may influence the data, such as community needs, and the additional partnership or funding opportunities available.

Table 7: Number of community economic development activitiesFootnote 55
Annual average (based on number of years of data available) CED-Q
(2013-17)
FedNor
(2013-18)
FedDev
(2016-17)
WD
(2014-18)
Number of new community economic development projects Total 540 202 348 1,350
Per CFO 9 8 9 15
Number of ongoing community economic development projects Total 439 229Footnote 56 348 925
Per CFO 8 10Footnote 57 9 10

Source: Internal program data.

Note: ACOA does not support community economic development activities through CFP funding.

Text version

Table 7 outlines the different community economic development (CED) activities for all of the four RDAs that deliver community economic development activities as part of their mandate (CED-Q, FedNor, FedDev and WD). It is important to note that ACOA does not support community economic development activities through CFP funding. Data is divided in two categories, the number of new community economic development projects (which range between 202 for FedNor to 1,350 for WD) as well as the number of ongoing community economic development projects by each RDA (which vary from 229 to 925). The averages are is also broken down by CFO, ranging from eight community economic development projects per CFO in FedNor to 15 for WD, while the ongoing community economic development projects vary from eight at CED-Q to ten at WD. It is worth noting, however, that the RDAs captured their CED activities in different timeframes so it is not possible to precisely measure all of the CED results.

Annual expenditures on community economic development projects and plans also varied by region (Table 8), but where data was available, it was clear that the CFOs undertaking these projects were effective at coordinating with other funding sources and leveraging additional funds. Community economic development projects leveraged between $7.9 and $40.1 on each CFO dollar committed.

Table 8: Annual investment and leveraging of community economic development projects and plansFootnote 58
Community economic development projects and plans CED-Q
(2013-17)
FedNor
(2013-18)
FedDev
(2015-16)
WD
(2014-18)
Average annual investment $2,337,268 $562,344 $633,326 $2,231,356
Average annual leveraged funds $18,399,530 $22,566,144 $7,257,823 $17,852,638
Amount leveraged per CFO dollar $7.9 $40 $11.5 $8.0

Source: Internal program data.

Note: Leveraged funds are those funds contributed from other organizations to further support these projects.

Text version

Table 8 shows the annual investments and leveraging amounts for the community economic development projects and planning activities for all of the four RDAs which have community economic development as part of their mandate. It is important to note that the data availability varied from one region to another (e.g., CED-Q covered 2013-17, FedNor covered 2013-18, FedDev submitted data for 2015-16 and WD data was available for 2014-18). Thus, it is not possible to compare the data with accuracy. The table presents different items such as the average annual investment in community economic development projects and planning activities (ranging from $.633M for FedDev to $2.3M for CED-Q), the average annual leveraged funds (that ranged from $7.26M for FedDev to $22.6M for FedNor) and the amount leveraged per CFO dollar (which vary from $7.9 for CED-Q to $40 for FedNor).

Based on interviews and the client survey, community economic development projects have aligned with community priorities and offered benefits to the communities in which they took place. Some of the case studies demonstrated those benefits, as well as the variety of projects that are undertaken:

Additional examples emerged through other lines of evidence:

Evidence shows that the scope of what is considered a community economic development project varies greatly. More specifically, it is likely that the scope of projects undertaken and the role that CFOs have played in these projects has evolved over time, a change which is not clearly captured in available documents and data for all regions. Therefore, it is challenging to provide an assessment of the overall impact of these projects and the role that CFP funding plays.

As such, there is an opportunity to re-examine the community economic development component to ensure that it is understood and reported on in a consistent manner reported on in a consistent manner across regions so that it can be more thoroughly evaluated in the future.

Strategic planning

Limited information was available about the strategic planning component of the CFP. However, most CFOs surveyed, as well as many individuals interviewed, consider the CFP to be making an important contribution to strengthening community strategic planning. In particular, as with the community economic development component, CFOs have, through CFP funding, helped to support communities facing economic challenges, such as those related to the collapse or interruption of resource-based industries, or those working to recover following a natural disaster. For example, CFOs in the West have undertaken strategic initiatives to help businesses in communities affected by wildfire (such as those in British Columbia or Fort McMurray).

As with the community economic development component, some limited data is available, which demonstrates that there is variation in both the number of strategic plans developed and the number of partners engaged in these plans. While this variation could indicate a diversity in scope of strategic plans, there are a large number of other regional characteristics that could also impact this variation (such as availability of partners who can offer in-kind or financial support). Overall, findings suggest that this component should be more clearly defined going forward.

Table 9: New or ongoing strategic plans and partners from 2013-14 to 2017-18
- CED-Q FedNor WD
Total Per CFO Total Per CFO Total Per CFO
Strategic plans 1,034 15 230 10 1,190 13
Partners engaged in plans 4,258 63 893 37 1,284 14

Source: Internal program data.

Note: Similar data not available in other regions.

Text version

Table 9 summarizes all of the new or on-going strategic plans and the number of partners engaged in the plans for CED-Q, FedNor and WD, over the evaluation period (2013-2018). The table also shows the average number of new or on-going strategic plans and the number of partners engaged in the plans per CFO in each of the regions. Nonetheless, similar data was not available for FedDev while ACOA's mandate does not include the strategic planning program component.

3.2.2 To what extent does the design of the CFP address inclusive growth (including considerations for different identity groups, such as immigrants, and those based on gender, Indigeneity, age, linguistic background, and disability)?

Key Finding: While considerations for inclusiveness are not yet formally integrated into the CFP, a variety of approaches are already in place across some CFOs to help the CFP meet the needs of different identity groups (as required of all federal programs). However, the extent to which these are implemented differs by region and community.

The following section examines the extent to which the CFP, as implemented in the five regions, promotes growth that is inclusive of all members of the community, using a GBA+ analytical lens. The CFP does not currently formally contain inclusiveness provisions as they were developed prior to the requirements in the Directive on ResultsFootnote 65. Under that Directive, all federal programs, policies, and initiatives must include GBA+ considerations when analyzing their impact on those with different identities, such as differing gender identities, ethnicities, age groups, Indigenous identity, or on persons with disabilities, and work towards addressing any barriers that might exist in their programFootnote 66.

Given the recent policy change, limited reporting is available on the inclusiveness of the CFP during the evaluation period. However, the evaluation provides some baseline data that could be used to facilitate decision-making related to how CFP programming could be used to better meet the needs of diverse groups in the futureFootnote 67. Performance data highlights the current accessibility of the CFP to a broad range of clients.

However, without consistent tracking of these indicators over time, their meaning is limited.

Overall, evidence from the case studies demonstrated that CFP programming provides a significant opportunity to support the needs of vulnerable populations located in rural communities across the country. To date, specific initiatives designed to promote inclusiveness have varied by region and the needs of individual communities. For example, FedNor asks that any community strategic plan developed by a CFDC be "inclusive of relevant communities (e.g., Aboriginal people, youth, visible minorities, etc.)," and Northern Ontario CFDCs must indicate how many Indigenous persons and francophones received counselling and information services in performance reportsFootnote 68. In Québec, CED-Q's 2021 Strategic Plan has adopted a new approach that encourages the participation of diverse groups in economic development initiatives within QuébecFootnote 69. Other RDAs have focussed their inclusion initiatives on specific groupsFootnote 70. For example, WD focusses on women, Indigenous people, and youth.

Some CFOs have developed partnerships or implemented specific programs to overcome the barriers faced by specific identify groups. However, some groups are still facing barriers or challenges (including structural barriers outside of the scope of the program). These themes are explored below by identity group. It should be noted that, across regions and communities, there is varying representation from these identity groups in the local populations. Therefore, the extent to which initiatives meet the needs of given identity groups or to which they are appropriate in each region varies.

Women

Case studies and program documents indicate that some CFOs work with organizations dedicated to female entrepreneurship, such as the PARO Centre for Women's Enterprise in Northern Ontario (see box), the Newfoundland and Labrador Organization of Women Entrepreneurs, or the Women's Enterprise Initiative in the West, to help meet the needs of female entrepreneursFootnote 71.

In other cases, specific programs have been put in place, such as the Women in Business Initiative in Atlantic Canada, which, as previously noted, provides financing and training to women, or the Consulting Advisory Services for SMEs in Québec, which aims to support women-owned businesses, as well as entrepreneurs from other identity groupsFootnote 72.

While these partnerships and programs help increase women's access to financing and support, evaluation findings indicate that some women entrepreneurs continue to face additional barriers to entrepreneurship, such as a lack of professional networks, or from other socio-demographic barriers that have yet to be fully addressed. While these are not specific to the CFP, they could impact the success of the CFOs' work in supporting female entrepreneurs.

PARO is a non-profit charitable organization run by women for women, established in collaboration with the Thunder Bay Ventures CFDC. It supports peer-lending circles in partnership with CFOs that provide access to loan capital and loan administration for circle members. Funding is provided in stages, each with progressively larger loans, with loans only guaranteed by circle members when all members agree. Since 2013, PARO has collaborated with CFDCs to provide 222 loans worth $194,500 to women through these circles.

Indigenous persons

As the demographic distribution of Indigenous communities varies across Canada, so does the range of activities and programs implemented by CFOs that specifically target Indigenous entrepreneurs. For example, interview findings indicated that some CFOs have put in place specific partnerships or programs to support the needs of Indigenous people in their communities. In addition, some CFOs are implementing provisions to ensure that Indigenous people are encouraged to participate in the decision-making of their CFO, including holding specific positions on Boards of Directors for Indigenous representatives.

In the case of a partnership-based approach, several lines of evidence indicate that some CFOs cooperate with Indigenous communities and Aboriginal financial institutions to help ensure that the needs of Indigenous communities are met. In Southern Ontario, many CFOs consult with Indigenous communities to ensure that their services are delivered in a culturally appropriate manner. FedDev has also established specific objectives for the number of Indigenous projects to fundFootnote 73.

In addition, many Indigenous CFOs have been established across Canada, to serve Indigenous clients exclusively. These include the Waubetek Business Development Corporation (see box), which also acts as an Aboriginal Financial InstitutionFootnote 74; the Nishnawbe Aski Development Fund (which was originally established as an Aboriginal Financial Institution, prior to becoming a CFDC), and Wakenagun.

The CFO networks in BC, Saskatchewan, and Manitoba deliver the Indigenous Business Development Services program, which helps Indigenous entrepreneurs to access information and resources on business programs, services, and regulations to start or expand their businessesFootnote 75. The Stó:lō CFO, located in BC, coordinated with various financial partners including BMO and Vancity to support Indigenous business growth and economic development.

However, broader challenges facing Indigenous entrepreneurs were raised during interviews, including different tax rules and structures that are in place for Indigenous people, complications in securing collateral, and transportation issues that might prevent an Indigenous entrepreneur from accessing a CFO. As with broader challenges facing other groups, these are outside of the scope of the CFP, but can impact the effectiveness of the CFOs' initiatives that support Indigenous entrepreneurs.

Waubetek is an Indigenous-owned and controlled CFO in Northern Ontario with a mandate to provide business counselling and investment services to SMEs along with leadership in community strategic planning and socioeconomic development. Its programming is delivered in partnership with a wide range of organizations, including First Nations, CFOs, and business incubators.

To help address the unique challenges and needs in its community, Waubetek has developed industrial strategies for the mining and fisheries sectors; has helped to develop the Indigenous tourism industry around Manitoulin Island; and helped bring together funding for a retail franchise in one community that created jobs and is allowing residents to shop locally.

Youth

CFOs across the country offer programming that targets youth. Interview findings indicated that extensive youth programs are offered by CFDCs in Québec, and a Youth Strategy that has been in place in the province since 1997 has supported 10,000 young entrepreneurs with business training, encouragement, and support. The CFOs in the province also collaborate with schools, non-profit organizations, and businesses on initiatives targeting youth, or have implemented specific initiatives, such as youth mentorship programs. For example, the Réseau des Sociétés d'aide au développement-Nicolet-Bécancour works with the school commission and local high school to introduce students to entrepreneurship and help them develop entrepreneurial skills. They also organize visits to industrial sites and run workshops where students can meet with speakers from a variety of different sectors. With its loans to youth entrepreneurs, the Réseau des Sociétés d'aide au développement-Nicolet-Bécancour also tends to be more flexible in the amount they ask a youth to invest in a project. Further, CFDCs across the province often work alongside the Carrefour Jeunesse-Emploi.

Another example is the Alberta Youth Entrepreneurship Camp organised by the Entre Corp CFO. The is a week-long camp that takes place every August for youth ages 13-15 who are interested in learning about business and entrepreneurship. The focus of this camp is to educate youth about marketing, advertising, preparing a cash-flow statement and writing a business plan.

CFOs in other regions also offer youth internship or student business programs, including:

Newcomers

Interview findings indicated that the needs of newcomers are increasingly being considered by CFOs, and the extent to which initiatives, programs, and partnerships have been put in place typically depends on the number of newcomers in their communities. Initiatives include an immigrant entrepreneur forum established by Nottawasaga Futures, a CFO in Southern Ontario to discuss barriers faced by newcomers.

Some CFOs are also working to attract newcomers to their communities. In Québec, for example, one case study highlighted that work with municipal and provincial governments has been undertaken to help attract and retain individuals, including newcomers, to their community.

However, interview findings suggest that there remain some broader barriers to entrepreneurship for newcomers, including accessing financing, difficulty or costliness in obtaining credit reports, language barriers, or perceived riskiness in lending to newcomers, which may impact the effectiveness of any CFO initiatives.

Persons with disabilities

Interview findings indicated that some CFOs in the Atlantic and Western regions implement programs for entrepreneurs with disabilities, such as the Entrepreneurs with Disabilities Program implemented in the West on behalf of WD, which provides business loans to help reduce the obstacles that people with disabilities or ongoing health issues face in looking for financing.

Linguistic minorities

CFOs that operate in communities that have an Official Language Minority Community are obligated to provide services that meet the needs of linguistic minorities, as a result of requirements for recipients of federal funding in the Official Languages Act, and the Treasury Board Directive on Transfer PaymentsFootnote 77. Several lines of evidence highlighted the approaches used to meet the needs of linguistic minorities (English communities in Québec and French communities outside of Québec). Bilingual CFOs have also been established to better serve the needs of francophone populations (and anglophone populations in Québec), either directly (in their communities) or virtually (in cooperation with other CFOs). These obligations are also sometimes fulfilled through cooperation with other organizations. In addition, in Northern Ontario, an Official Languages Evaluation and Action Plan, which is a template to be completed by CFDCs, has been developed to help guide and support CFDCs in meeting their official language obligationsFootnote 78. Similarly, FedDev provides additional funding to 6 designated bilingual CFDCs which provide all four lines of business in both official languages. However, interview findings suggest that some CFOs still face challenges in offering materials and services in the minority official language.

CFOs may also set up specific initiatives to support linguistic minorities. For example, the Francophone Women Initiative, which both directly supports francophone women, and supports community partners in promoting the economic security and prosperity of francophone women, is in place in a CFO in New Brunswick; and the Entrepreneurs Francophones Plus Initiative in Northern OntarioFootnote 79.

3.2.3 To what extent are the CFP performance measurement and reporting structures effective in reporting on the achievement of the CFP outcomes?

Key Finding: The RDAs and FedNor have developed a national PM Strategy for the CFP. It includes the analysis of Statistics Canada business data to provide consistent national reporting that illustrates outcomes for CFP-assisted businesses, which is a best practice. However, other aspects of the PM Strategy are not uniformly implemented across regions. Overall, the data available supports reporting on the achievement of CFP outcomes at the regional level; further consistency would better support effective reporting at the national level.

In response to previous evaluations' recommendation for clearer and more uniform indicators across regions, the RDAs formed a National PM Strategy Committee to oversee collaborative efforts between organizations with respect to collecting, analyzing, and reporting on performance data. The committee is composed of evaluation and program representatives from each RDA, FedNor, and ISED, who meet on a quarterly basis.

Through the work of the committee, an updated national PM Strategy was developed, which sets out 16 indicators, some of which are measured using Statistics Canada data, and others which use data collected by RDAs and CFOs. However, not all indicators are applicable to all regions (e.g., those that relate to community economic development in the Atlantic region). Targets are also established on a regional basisFootnote 80.

At present, the analysis and reporting that is done using the indicators from Statistics Canada represents a best practice in reporting consistency (since all regions report on the same indicators in the same way, as well as accuracy (since it relies on administrative data rather than self-reported data).

The indicators collected in each region are outlined in RDA-level PM Strategies, Performance Information Profiles, or Departmental Results FrameworksFootnote 81. As such, many indicators are not reported on in some regions and it is unclear whether the indicators used are defined in the same way in all regions. In addition, it is possible that, since CFOs collect data on many of the indicators, they are not consistently measured within regionsFootnote 82. While, as outlined in the current PM Strategy document, the indicators were to be collected and reported on in regional evaluations, which allowed latitude in how the indicators were measured, if the process of evaluating the CFP at a national level is to continue, more uniformity in indicators collected and used will be neededFootnote 83.

In addition, the RDAs have made some efforts to integrate GBA+ considerations in their performance measurement or evaluation practices, but the indicators and groups included vary by regionFootnote 84 as discussed in Section 3.2.2.

To complement existing data and Statistics Canada analysis, a pilot project is being undertaken with the Economic Analysis Division at Statistics Canada, with the objective of determining the feasibility of adding age and gender metrics to the tables compiled for the Community Futures Program.

Based on findings from interviews and the document review, as well as the analysis of regional versus national indicators, common approaches are needed to clearly define and collect performance measurement data. Some work is already underway at the national CFP PM Strategy Committee to analyse and harmonize indicators where possible, but there is more progress to be made.

3.2.4 How is the CFP being implemented on an ongoing basis?

Key Finding: The ability of CFOs to adapt programming to address regional needs and opportunities is a core program strength. CFP-supported CFOs implement programming through high levels of coordination and collaboration with diverse partners. However, there may be opportunities to increase national consistency in specific areas, though overall program flexibility should be maintained as the ability of the program to adapt to regional needs is a program strength.

As previously outlined, the CFP is delivered by four RDAs, who provide financial support (through contribution agreements) to 256 CFOs across Canada. Evidence from documents and interviews showed that the division of responsibilities between RDAs and CFOs is well understood, and relationships between these organizations are well established.

However, evaluation findings point to less consistency in the overall understanding of the role of ISED, due, in part, to the fact that the CFP was more recently (in 2015) moved under ISED's ministerial portfolio. Interview findings suggest that this change has had several positive impacts, notably prompting better communication between the RDAs; promoting greater national consistency in the way that the CFP is delivered across Canada; and helping to renew or promote the program's focus on regional economic development.

The network of relationships between ISED, the RDAs, and the CFOs, as well as the networks that have been established between the CFOs, allow for information sharing, including sharing of best practices between regions to facilitate ongoing improvement. In addition, according to a recent report, the networking of CFOs into provincial and regional associations, as well as a national association, have supported CFOs by:

As outlined in the program profile, each region has its own model for funding the CFP. In the Western provinces, WD's funding formula includes base funding and variable elements based on community characteristicsFootnote 86. FedDev in Southern Ontario and ACOA in the Atlantic region have implemented performance-based funding models. Both include a core funding component, as well as funding based on achievements as measured against outcomes and/or key performance indicatorsFootnote 87. As these funding models are structured differently, they are viewed as having different levels of success.

Based on interview findings and the previous evaluation, the model used in the Atlantic region has received positive reviews. It is implemented in collaboration with the Atlantic Association of CBDCs and includes joint mechanisms, such as a Risk Mitigation Fund. Key informants said that the model has allowed CBDCs in the Atlantic region to plan their finances over a longer time horizon. In Southern Ontario, FedDev's performance-based funding formula provides an annual contribution with additional funding determined based on the CFO's performance over the previous three years in areas such as loan performance, and the number of businesses assisted and jobs created. Key informants, commenting on this model, focussed mainly on the negative impacts, including a higher administrative burden, disincentives to engage in community economic development projects, and the creation of gaps between high and low performing CFDCs.

The management of contribution agreements also varies by region. For instance, the duration of contribution agreements can range between one to five years. Consequently, the expiration and renewal of contribution agreements is not coordinated nationally. This is largely due to varying risk mitigation efforts (such as shorter-term contribution agreements for CFOs that require more frequent RDA follow-up), additional conditions that CFOs may have with RDAs, different fiscal year-end for certain CFOs and local administrative preferencesFootnote 88.

While the CFP's flexibility and adaptability have contributed to its success, interview respondents, documents, and the CFO survey identified a number of areas where greater national consistency may be warranted. This includes clearer national objectives for the CFP's support of local economies, as well as increased consistency of program implementation, notably in performance measurement, as discussed in the previous section. There may also be an opportunity for a deeper examination of other aspects of program implementation, including the funding models used, to determine where greater consistency should be encouraged, as well as where flexibility should be maintained to respond to regional needs.

At the CFO level, the most commonly raised challenge in program implementation, across all lines of evidence, is that funding has not increased since 2010, while program delivery costs have increased. Interview and case study findings and previous research indicate that, to adapt to this reality, CFOs have reduced or modified the services they offer, or have been looking for alternative funding. Funding levels have also had an impact on CFOs' ability to attract and retain staff.

The PEI East CBDC has undergone significant changes in the past 10 years and has been able to take advantage of the flexibility in the CFP's design to adapt. The CBDC previously operated as a Regional Economic Development Organization, providing community economic development services through another line of ACOA funding (since these are not part of the CFP in the Atlantic region). However, that funding ceased in 2013. In response, the CBDC became a Rural Action Centre host, which brought in additional funding and other organizations with which to share space, which has helped to foster partnerships. The CBDC has also increased its loan activity.

3.3 Performance: Efficiency

3.3.1 Is the CFP being managed and delivered effectively and efficiently?

Key Finding: The CFP is generally implemented efficiently and cost-effectively. Key efficiency factors include: delivery of programming by CFOs, high use of volunteers, and multiple partnerships. However, limited quantitative data precludes a full assessment.

Based on interview and survey findings, the evaluation found that the CFP is delivered cost-effectively. It continues to achieve its primary objectives despite the declining real value of program funding (the value of funding adjusted for inflation). In addition, various initiatives and approaches to further improve the efficiency of program delivery have been or are being implemented at the CFO and RDA levels.

The document review, interviews, and case studies highlighted practices that have been put in place by CFOs to improve efficiency, including sharing some services, such as with financial administration and group purchasing through CFO associations, and using technology to help reduce some costs, for example, videoconferencing to help reduce travel costs or to help with content management.

The evaluation found that the ongoing contribution of thousands of volunteers helps reduce the cost of service delivery and supports cost-efficient program delivery, and is considered a key strength of the delivery model, as well as a key contributor to program successFootnote 89. A volunteer board of directors oversees each CFO and makes decisions on granting loans. According to interviews and case studies, those volunteers are one of the key contributors to the CFP's success. However, the quantitative value of that contribution has not been rigorously investigated at the national levelFootnote 90.

CFOs offer a range of resources and training to support the work of volunteers. They have also established procedures to ensure that potential conflicts of interest that might arise are adequately addressed. However, interview findings suggest that it can be challenging to find enough qualified volunteers for boards, given the significant time commitment and the limited talent pool in smaller communities. Reimbursement for expenses incurred by volunteers is also very limited, which dissuades some potential volunteers.

Another important element of the efficient delivery of the CFP is the establishment of partnerships with external organizations. In general, these partnerships have helped to increase the reach of the CFOs and reduced duplication of activities with other economic development organizations. Many different types of partnerships were highlighted during interviews and case studies, including collaboration with:

At the RDA level, the cost of CFP administration, including the management of the contribution agreements signed with each CFO, varies from about 4.3% of program funding in Québec (CED-Q) to 7.6% in western Canada (WD). (Table 10) The number of CFOs found in each region, the range of activities offered by each CFO, and the approach used by each RDA to manage the various programs they deliver, are among some of the factors that may explain the differing administration costs.

Table 10: Administrative costs (O&M) as a percentage of Grants and Contributions (G&C)
RDA 5-year average (2013-14 to 2017-18)
G&C O&M Total % O&M
ACOA 12,642,000 564,000 13,206,000 4.5%
CED-Q 28,754,445 1,226,000 29,980,445 4.3%
FedNor 8,360,000 490,000 8,850,000 5.9%
FedDev 11,282,333 757,713 12,040,046 6.7%
WD 28,082,391 2,146,546 30,228,937 7.6%
Total 89,121,169 5,184,259 94,305,428 5.8%

Source: Internal program data.

Text version

Table 10 shows the administrative costs related to the CFP, divided by the Grants and Contributions (Gs&Cs) amounts and the Operations and Maintenance (O&M) for the period of 2013-2018. It also summarizes the percentage of operating and maintenance costs comparing with the total administrative costs. Generally, the total regional administrative costs fluctuate from $8.9M for FedNor to $30.2M for WD, while the percentages of O&M vary from 4.3% to CED-Q to 7.6% for WD. Moreover, the total administrative costs for all five RDAs are $94.3M and their percentage of O & M costs compared to the Gs&Cs equals 5.8%.

RDAs have implemented, or are developing, approaches to make their activities more efficient. This includes the performance-based funding models (beneficial to varying extents) discussed earlier, simplifying contribution agreements, and re-examining operations to identify possibly efficiency gainsFootnote 91. Another approach has been to conduct research to help target investments in more strategic areas, as highlighted in the Functional Economic Regions case study (see box). By having stronger data to guide decision-making, some CFOs are able to better prioritize and make decisions at the local level about key sectors, needs, and opportunities.

Other approaches that could help to further increase efficiency, suggested during interviews, include consolidation of offices or centralizing some tasks (such as accounting, website maintenance, or marketing). Partnerships should also continue to be leveraged and new partnerships could be pursued to share best practices and resources among different organizations.

Overall, there is limited quantitative evidence to assess the efficiency of the delivery of the CFP at the national level, or the efficiency with which the CFOs contribute to the outcomes of the CFP. As the PM Strategy is further refined, it could be beneficial to conduct a more rigorous analysis of the efficiency of program delivery (looking at elements such as outputs per dollar spent on program administration). This will be particularly valuable as the program continues to implement program modifications that are designed to promote efficiency. However, this type of assessment could only be undertaken in the longer term, as a national-level baseline would have to be established.

The Functional Economic Regions project is delivered by the Newfoundland and Labrador Association of CBDCs, in collaboration with Memorial University's Regional Analytics Lab. The objective is to identify the main drivers of the economy and the intra-regional and inter-provincial industry linkages for priority sectors for each of the Functional Economic Regions in the province. A pilot of the project was conducted with CBDC Trinity Conception. The findings from the study will help in decision-making and planning, since they identify key industries and employers, as well as priorities to address in supporting growth. More specifically, the dashboards that will be produced through the project as it continues will help in identifying which communities are best suited to emerging opportunities.

4.0 Conclusions and recommendations

This section presents the key findings that resulted from the evaluation, and the resultant recommendations.

4.1 Relevance

4.2 Performance: Effectiveness

4.3 Performance: Efficiency

4.4 Recommendations

The recommendations aim to acknowledge the importance of regional delivery of the CFP to address local needs and unique capacities, while identifying opportunities to improve the ability to report its relevance and performance at a national level. The RDAs and FedNor should build upon their efforts to provide a comprehensive national picture of the results of the CFP, to the extent possible, by:

References

Appendix A – Evaluation framework

Evaluation framework

Relevance
Evaluation issues and questions Indicators Data sources
  1. To what extent is the CFP addressing current needs?
  • Needs addressed by the CFP
  • Differences in needs across different groups
  • Differences in needs across regions
  • Role of partnerships in addressing the need(s)
  • Complementarity or duplication with other programs
  • Needs not being met
  • Stakeholder interviews
  • Document review
  • Case studies
  • Client survey
  • CFO survey
  1. To what extent does the CFP align with government priorities?
  • Evidence of alignment of different program components with government priorities (regional innovation ecosystems, investment in and scale-up of firms, clean growth, and community economic development and diversification)
  • Other government priorities supported through the CFP
  • Contribution of the CFP to government priorities related to inclusiveness
  • Document review
  • CFO data system (client data on linguistic minorities, Indigenous people)
  • Interviews with CFO and departmental stakeholders
  • CFO survey
Performance: Effectiveness
Evaluation issues and questions Indicators Data sources
  1. To what extent have the CFP's immediate, intermediate and ultimate outcomes been achieved in the medium and longer term?
  • Measurement against specific indicators outlined in PM Strategy:
    • Immediate outcomes:
      • % of funds in active investments (total $ value of loans/equity/guarantees (net) divided by IF closing balance).
      • number and $ value of CFO investments per year (start-ups and all other eligible loans [by region])
      • Amount leveraged per dollar invested by CFOs
      • Number of partners engaged in community development initiatives
      • CED initiatives/community-based projects implemented (#,value, type and funding sources)
    • Intermediate outcomes:
      • Clients indicating improved business practices as a result of CFO support
      • # of new business start-ups
      • Growth in sales
      • Enhanced community capacity with respect to knowledge skills, services, economic development infrastructure, collaboration and partnerships through CFO support.
    • Ultimate outcomes:
      • Employment growth variance over a five-year period (CFO vs. comparison group)
      • Survival rate variance (% of firms that are still operational after a specific period of time after start-up)
  • Stakeholder perceptions of the impact of the CFP on their businesses and communities
  • Stakeholder perceptions of the contribution of the CFP to the diversification of rural economies
  • Incrementality: Perceived impact of the absence of CFP funding and other non-funding services on communities
  • Incrementality: Perceived impact of absence of CFP funding and other non-funding services on clients and their projects
  • Variations in achievement of outcomes for different target groups (including those with different identities)
  • Achievement of outcomes by region
  • Challenges in achieving outcomes (overall, by group, by region)
  • Examples of mitigation strategies used
  • Perceived impact of status quo funding
  • CFO data system
  • CFO reports
  • StatsCan data analysis
  • Program data
  • CFO survey
  • Client survey
  • Interviews
  • Document review
  • Regional case studies
  1. To what extent does the design of the CFP address inclusive growth (including considerations for different identify groups, such as immigrants, and those based on gender, Indigeneity, age, linguistic background, and disability)?
  • Barriers faced by members of different groups
  • Benefits offered to different groups
  • Policies in place to promote equitable access
  • Approaches in place to address needs of specific groups
  • Needs of different groups
  • Client survey
  • Regional case studies
  • Key informant interviews
  1. To what extent are the CFP performance measurement and reporting structures effective in reporting on the achievement of the CFP outcomes?
  • Indicators in place to measure outcomes
  • Variations between regions (e.g. indicators used, indicator definitions)
  • Regional best practices
  • How performance information is being used
  • Information available to measure impact of program on different target groups (such as women, youth, and Indigenous peoples)
  • Gaps in current information collected
  • Opportunities to improve data collected
  • Data analysis
  • PM Strategy annual reports
  • Audit reports
  • Interviews with CFOs, departments
  1. How is the CFP being implemented on an ongoing basis?
  • Management structure(s) used to implement and oversee the CFP
  • Extent to which roles and responsibilities of various parties are clearly defined and understood
  • Role of volunteers in program implementation
  • Variations in delivery between regions
  • Benefits of current structure
  • Challenges encountered
  • Document review
  • Interviews
  • Case studies
  • CFO survey
Performance: Efficiency
Evaluation issues and questions Indicators Data sources
  1. Is the CFP being managed and delivered effectively and efficiently?
  • Mechanisms currently in place to support efficient/cost-effective program delivery
  • Suggestions of alternative approaches to deliver program more cost-effectively
  • Adaptations put in place by RDAs to improve cost-effectiveness or efficiency of their program delivery
  • Regional best practices
  • Changes in program efficiency with horizontal management
  • Opportunities to streamline management
  • Need for flexibility in program delivery across regions
  • Ratio of administrative costs to project funding by RDA
  • Contribution of volunteers to efficient program deliver
  • Areas for improvement
  • CFO survey
  • Interviews with CFO network representatives, departmental officials
  • Literature / document review

Appendix B – Case studies undertaken by region

Case studies undertaken by region
RDA Case study Rationale
ACOA Evolution of Organization, Approach, and Programming (PEI East CBDC) Demonstrates adaptability of the CBDC to changes in program delivery context
Women in Business Initiative (New Brunswick Association of CBDCs) Demonstrates how CBDCs are meeting the needs of different target groups (primarily women)
Functional Economic Regions (Newfoundland and Labrador Association of CBDCs) Demonstrates importance of partnerships and how data can be used to support the delivery of the CFP
CED-Q Nicolet-Bécancour CFDC Includes a program for youth, to support their retention in the region
Témiscamingue CFDC Demonstrates how CFDCs are meeting the needs of different identity groups (primarily Indigenous people)
Rive-Nord BDC Includes semi-urban areas
FedDev Cross-Border Partnership Program (Cornwall & Counties CFDC) Demonstrates the importance of partnership, how the CFP can meet the needs of different target groups (in this case, Indigenous people) and of surrounding counties, and the skill of CFOs in fostering collaboration
Niagara Cycling Tourism Centre/Network (Venture Niagara CFDC) Example of a CED project supporting tourism
Self-employment training (led by the Oxford CFDC) Example of an initiative undertaken in response to changes in the local economy (i.e., closure of manufacturing plants)
FedNor Waubetek Business Development Corporation Demonstrates how CFDCs are meeting the needs of different identity groups (primarily Indigenous people)
PARO Centre for Women's Enterprise Demonstrates the importance of partnership with CFDC entities, and how the CFP can meet the needs of different target groups (primarily women)
Nord-Aski CFDC (including a bilingual mandate and youth internship program) Demonstrates how CFDCs are meeting the needs of different target groups (primarily francophones and youth)
WD Churchill Region Economic Development Fund (Community Futures Manitoba) Example of an initiative undertaken in response to changes in the local economy (i.e., closure of a port)
Venture Connect (Community Futures Alberni-Clayoquot) Example of a CED project in a remote community
Imagine Kootenay (Community Futures Development Corporation of Central Kootenay) Example of a CED project that includes partnership with local Indigenous communities
Date modified: