Language selection

Search

Archived — Final Evaluation for the Northern Ontario Development Program

Final Report

February 2011

Tabled and approved at the Departmental Evaluation Committee on February 23, 2011

Table of Contents

Annexes (separate document)

(Note: Annexes are available via an Access to Information Request)

  • Annex A Evaluation Advisory Committee members
  • Annex B Evaluation matrix
  • Annex C File review projects
  • Annex D File review template
  • Annex E Interview introductory letters and interview guides
  • Annex F Case study proponent introductory letter and interview guides
  • Annex G Case studies
  • Annex H NODP indicators—expected and actual results
  • Annex I Examples of non-profit organizations NODP funded projects
  • Annex J Concepts from the literature
  • Annex K Northern Ontario economic development programs

top of page

Program Overview

The Northern Ontario Development Program (NODP) supports economic development and diversification in northern Ontario through contributions to a range of organizations both public and private. The NODP supports projects in six program areas: Community Economic Development; Innovation; Information and Communications Technology; Trade and Tourism; Human Capital; and Business Financing Support.

The NODP is delivered by the Federal Economic Development Initiative for Northern Ontario (FedNor), which serves as the main representative for Industry Canada (IC) in northern Ontario.

Evaluation Approach

The purpose of the evaluation was to assess the relevance and performance of the program as required under the Treasury Board Policy on Evaluation. The evaluation covers the period from April 2006 to August 2010. Projects included in the evaluation were those approved between April 2006 and January 2010, representing approximately $148 million to support 858 projects.

The evaluation findings and conclusions are based on the analysis and triangulation of multiple lines of evidence. The methodology included a review of documents and literature, an administrative data and file review, interviews, and case studies.

Findings

Relevance

There is a continued need for the NODP to strengthen and diversify the northern Ontario economy, which has historically relied on primary resources. The isolation of certain communities in northern Ontario, the distance between these communities, and their reliance on one industry make them vulnerable to outside shocks, such as price fluctuations and variations in commodity exports. The NODP aims to foster economic growth and diversify the regional economy by supporting projects in the six thematic areas.

Its scope has been broad, supporting a wide range of projects. However, some interviewees believe the NODP could better address the needs of the region by investing more directly in businesses and targeting its resources to fewer, larger, and strategically positioned projects located in communities that have good links to outside markets. Much of the recent academic literature aligns with this advice.

The NODP is consistent with federal roles and responsibilities and is an important federal presence in northern Ontario. The program promotes economic growth, diversification, job creation, and sustainable communities in northern Ontario, and is the government's response to regional economic development in that area. Interviewees noted that the federal government has a responsibility to lead the way for all orders of government, non-governmental organizations, and other institutions. Recent academic literature supports interviewees' views.

The NODP also supports the federal government's priorities, as outlined in recent Speeches from the Throne and Federal Budgets, and IC's strategic objectives. Although other economic development programs are available in northern Ontario, evidence shows that they are complementary and result in very little overlap and duplication. Many interviewees and case study participants noted that the collaboration of various programs available in northern Ontario is essential for any project to move forward.

Performance

The Mid-term Evaluation of the NODP (2008) identified ways to strengthen the program's Performance Measurement System and data quality. Program management has made some improvements to the program in response to these issues. However, limited progress has been made on the collection of performance measurement information regarding outcomes beyond the project end date. Changes to address the longer-term measurement of project outcomes are being planned and integrated into the revised NODP Performance Measurement System.

The main factor that supports effective delivery of the NODP is the involvement and experience of FedNor regional staff. Regional FedNor offices in northern Ontario enable direct contact with the proponents, and FedNor staff are knowledgeable of the communities' needs. However, the length of time for project approval has impeded the delivery of the program. These approval times are longer compared to both previous FedNor-administered programs and other regional development agencies in Canada.

The NODP has achieved its immediate outcomes. The Final Project Evaluation indicator database captures several indicators under each program area to measure the achievement of the immediate outcomes of the program. These indicators, as well as interviews and case studies, provide direct evidence of the achievement of immediate outcomes.

However, evidence on the achievement of intermediate and ultimate outcomes is mixed. Qualitative evidence from interviewees and case study participants indicates that the program is achieving these outcomes. The lack of continued tracking and measurement of outcomes by the project proponent after the formal relationship with NODP has ceased limits the evidence that could speak to the achievement of long-term outcomes.

Projects funded under the NODP have led to several outcomes and have contributed to the economic development of northern Ontario. According to interviewees, without funding from the program, many projects would not have proceeded or would have proceeded but at a reduced scope.

Administrative costs appear to be reasonable to support program delivery over a vast geographic area, with requisite support for effective communications and core services. Interviewees stated that the experience and knowledge of FedNor staff in the regions and their high level of availability and support are instrumental to the development of projects and success of the program.

It is difficult to assess whether the cost per outcome is reasonable due to data limitations. From a sample of projects, the total cost to the NODP per temporary and permanent job created and maintained was $5,229. However, creating and maintaining jobs were not the only outcomes expected from these projects.

The NODP has been quite successful at securing financial contributions for projects from other sources. For each dollar contributed to a project by the NODP, $2.77 was contributed to the project from other sources, such as the provincial government, municipalities, the private sector, and the proponents.

Although some interviewees noted that the NODP allocates its resources effectively, they also suggested the program could be more effective if it were refocused. In particular, they felt funds should be allocated more strategically to core economic development activities and should be targeted to projects that impact businesses. These suggestions are consistent with the current direction of the literature.

Recommendations

Recommendation 1: To ensure that expected outcomes are being achieved, the program should improve the measurement of medium- and longer-term outcomes.

Recommendation 2: To enhance program delivery success, the program should consider improving the project approval process to decrease approval times.

Recommendation 3: To better achieve the overall program objective of economic development and diversification in northern Ontario, FedNor should be more strategic in its allocation of program resources and target program funding to core economic development activities. Consideration should also be given to how to increase program impacts on for-profit businesses.


top of page

1.0 Introduction

The Northern Ontario Development Program supports economic development and diversification in northern Ontario through contributions to a range of organizations both public and private. NODP staff also coordinate and facilitate consultations among governments, non-governmental organizations, and industry for the purpose of identifying and acting on economic development opportunities.

The Federal Economic Development Initiative for Northern Ontario (FedNor) administers the program. FedNor is Industry Canada's regional development organization in northern Ontario.

The purpose of the evaluation is to assess the relevance and performance of the program under the Treasury Board of Canada Policy on Evaluation. The evaluation covered NODP activities between April 1, 2006 and August 18, 2010.

1.1 Context

Northern Ontario covers over 806,000 square kilometres (almost 90% of the province's total land area) and has a population of 843,853, which accounts for about 7% of the total provincial population.1 The region consists of 9 cities, 26 towns, 93 townships, 18 municipalities, 5 villages, 129 Indian reserves and settlements, and 17 unorganized areas. Five major urban centres—Timmins, Thunder Bay, Sault Ste. Marie, Sudbury, and North Bay—together account for over half (52%) of the population of northern Ontario.2

Historically, the economy of northern Ontario was concentrated in resource-based industries, including agriculture, forestry, and mining.3 These goods-producing activities remain an important source of economic activity. Almost one-quarter of the jobs in northern Ontario are in goods-producing sectors (22%) such as construction (7%), manufacturing and mining (7%), and fishing and forestry (5%). The remaining 78% of jobs are in the service-producing sectors, such as health care and social assistance (16%), wholesale and retail trade (15%), and education (9%).4

Gross Domestic Product (GDP) by sector in northern Ontario has a distribution similar to the employment figures. Goods-producing industries contribute 27% to the northern Ontario GDP, with manufacturing (10%) and construction (7%) contributing the largest amounts. The remaining 73% of the GDP is contributed by services-producing industries, with non-commercial services (23%) and wholesale and retail trade (14%) providing the largest amounts.5 Relative to the province as a whole, the northern economy is much more oriented toward forestry, fishing, mining, oil and gas, and the public sector, and less oriented toward manufacturing, finance, insurance, real estate and leasing, and professional, scientific, and technical services.6

The mining and forestry industries in northern Ontario have changed significantly in recent years. In the mining sector, automation has reduced available employment opportunities, whereas rising commodity prices have tended to have the opposite effect.7 In forestry, cost pressures and turmoil in the US housing market (an important consumer of the region's forestry products) have adversely affected employment in that sector. This has, in turn, seriously affected manufacturing in the region, much of which is tied to the pulp and paper and sawmill industries.8

Southcott (2007) notes that because of the way northern Ontario developed, the region depends on natural resource exploitation and externally induced incomes in the form of public administration. He argues that the implications associated with this pattern of development are economic vulnerability to factors outside the region's control, such as world commodity prices, and low levels of entrepreneurship, which has served as a barrier to "the cultivation of an entrepreneurial culture."9, 10

1.2 Program Profile

1.2.1 Program description

The Federal Economic Development Initiative for Northern Ontario

Established in 1987, FedNor serves as IC's main representative in northern Ontario.11 It "is a regional development organization […] that promotes economic development, diversification and job creation and encourages sustainable, self reliant communities in" the region.12 FedNor delivers two core programs: the NODP and the Community Futures Program. Other programs delivered by FedNor include the Community Adjustment Fund and the Economic Development Initiative.13

FedNor has three main offices in northern Ontario located in Sudbury, Sault Ste. Marie, and Thunder Bay. It also has four satellite offices in strategic locations—Kenora, North Bay, Parry Sound, and Timmins—to serve proponents; officers are located in areas where they can stay in close contact with the communities.

Over 80% of FedNor's resources are dedicated to "supporting a decentralized field network to deliver its programs."14 This provides communities with access to FedNor officers who have established networks of local contacts and working relationships, and can guide strategic project proposals.15

The Northern Ontario Development Program

The NODP provides support in the form of financial contributions to projects in northern Ontario in six program areas:

  • Community Economic Development (CED) – Northern Ontario communities assisted to develop their capacity to undertake strategic planning in response to decreased industry levels and to implement related CED projects.
  • Innovation – Knowledge commercialized; investment in innovation, productivity improvement and value-added product development in key sectors such as biotechnology, mining and forestry.
  • Information and Communications Technology (ICT) – High-speed Internet provided to communities; support for ICT applications, particularly in the areas of health, education and business; access to advanced communications services through central telecommunications hubs referred to as 'Points-of-Presence.'
  • Trade and Tourism – Trade missions sponsored and trade networks assisted to help businesses access U.S., European and interprovincial markets. Northern Ontario's tourism industry (including nature-based tourism) effectively promoted.
  • Human Capital – Youth internships, entrepreneurships and business skills promoted, and more skills development opportunities made available through Northern Ontario educational institutions.
  • Business Financing Support – Enhanced small- and medium-sized enterprise (SME) access to capital, including patient and venture capital, with more capital providers and leveraged capital.16

The contributions in these six areas address the NODP's mission to "promote growth, economic diversification, job creation, and sustainable, self-reliant communities in Northern Ontario, through a range of initiatives aimed at improving business access to capital, information, and markets."17

NODP assistance is limited to SMEs with fewer than 500 employees. Key proponents include:

  • SMEs that are unlikely to obtain financing through a loan/investment fund because the risk is too high;
  • Non-profit organizations engaged in research and development;
  • Aboriginal capital corporations, Aboriginal non-profit organizations, Aboriginal-controlled financial organizations, and similar groups able to increase the economic activity of Aboriginal people;
  • First Nations and legal entities including incorporated not-for-profit organizations such as municipalities and municipal organizations; community development organizations; post-secondary institutions; hospitals and regional health care centres and other not-for-profit consortia of partnerships; sole proprietorships; corporations; or associations;
  • Businesses including corporations, partnerships, co-operatives, proprietorships, trusts, or consortia; and
  • Community Futures organizations.

Table 1 shows the number and dollar value of projects supported under each thematic area of the NODP between April 1, 2006 and January 31, 2010.

Table 1 : NODP support by thematic area (April 1, 2006January 31, 2010)
NODP funding area # of NODP projects NODP funding
CED 272 $39,960,298
Innovation 52 $34,387,450
Trade and Tourism 121 $25,564,089
ICT 70 $21,844,620
Human Capital 312 $13,396,143
Business Financing Support 31 $12,847,497
Total 858 $148,000,097

In total, the NODP approved approximately $148 million to support 858 projects between April 1, 2006 and January 31, 2010. About half of the funding supported CED and Innovation projects. The thematic area that comprised the most projects was Human Capital; however, most of these projects (271) were youth internships, which are generally low-budget projects with contributions below $30,000. Innovation had the highest funding per project.

1.2.2 Program resources

Table 2 sets out the funding profile of the NODP, split between Grants and Contributions (Gs&Cs) and Operating and Maintenance (O&M) expenses.

Table 2 : Initially allocated NODP resources ($million)

* A one-year investment of $6 million went to the Northern Ontario School of Medicine.

** A five-year investment of $7.35 million is going to the Thunder Bay Regional Health Sciences Centre.

Source: FedNor. (2010). Northern Ontario Development Program (NODP) budget. Received from program November 2, 2010.

  2005–06 2006–07 2007–08 2008–09 2009–10 Total
Gs&Cs 44.40* 38.40 39.10** 42.90** 39.90** 204.70
O&M 13.50 13.50 13.50 13.50 13.50 67.50
Total 57.90 51.90 52.60 56.40 53.40 272.20

Actual spending for the program appears in Table 3.

Table 3 : Actual NODP resources ($million)

Source: FedNor. (2010). Northern Ontario Development Program (NODP) budget. Received from program November 2, 2010.

  2005–06 2006–07 2007–08 2008–09 2009–10 Total
Gs&Cs 44.50 36.20 36.50 40.80 38.55 196.55
O&M 11.37 11.10 10.80 10.00 10.95 54.22
Total 55.87 47.30 47.30 50.80 49.50 250.77

Total expenditures were $21 million less than expected between 2005–06 and 2009–10 as a result of government-wide reduction initiatives over the past few years. This reduction in available funding would have been greater had FedNor not received additional one-time funding from the fiscal framework for two specific projects as noted in Table 2.

1.2.3 Program expected results

The NODP has the following expected immediate, intermediate, and ultimate outcomes. The six immediate outcomes align with each of the six program areas (noted in brackets) that, in turn, support a single intermediate and single long-term outcome:

Immediate outcomes

  • Improved community capacity for socio-economic development (CED)
  • Enhanced markets and investment attraction (Trade and Tourism)
  • Enhanced innovation capacity and improved infrastructure (Innovation)
  • Enhanced information and communications technology applications and infrastructure (ICT)
  • Improved access to capital and leveraged capital (Business Financing Support)
  • Attraction, retention, and development of human capital (Human Capital)

Intermediate outcome

  • Improved competitiveness and/or sustainability of communities and key sectors

Ultimate outcome

  • Long-term, sustainable socio-economic development in northern Ontario.

Figure 1 presents the NODP logic model and demonstrates how the activities, outputs, and outcomes of the NODP align.18

Figure 119: NODP Logic Model
Figure 1: Northern Ontario Development Program Logic Model [Description of Figure 1]

1.2.4 Program reporting

Throughout the life of a project, proponents are responsible for providing progress reports (Project Activity Reports) with a financial claim on a monthly or quarterly basis. In addition, a Final Project Evaluation (FPE) report must be completed at the end of the project.

Annually, FedNor is required to report on the NODP for IC's Departmental Performance Report (DPR) and Report on Plans and Priorities (RPP). FedNor also provides information on the NODP for its annual business plan.20

1.3 Report Outline

The final evaluation report is broken down as follows:

  • Section 2 provides a list of evaluation issues and describes the methodologies used to gather information for this evaluation.
  • Section 3 is divided into two subsections: findings on relevance, and performance. It also provides findings of the evaluation by evaluation issue.
  • Section 4 provides conclusions and recommendations.

top of page

2.0 Methodology/Approach

IC engaged Prairie Research Associates (PRA), an independent research company, to conduct a Final Evaluation for the NODP. The purpose of the evaluation was to assess the relevance and performance of the program as required under the Treasury Board Policy on Evaluation.

An Evaluation Advisory Committee (EAC) was established to provide advice and guidance during the evaluation. The committee reviewed and provided input on the methodology, as well as on the findings, conclusions, and recommendations presented in the evaluation report. A list of NODP EAC members is provided in Annex A.

This section lists the issues addressed by the report and describes the methodologies used to gather and analyze the information for this evaluation.

2.1 Evaluation issues

This evaluation meets the requirements of the Treasury Board Policy on Evaluation and addresses the themes of relevance and performance. The five core issues of the Treasury Board Policy on Evaluation and the specific questions for the evaluation included:

Relevance

  • Continued need for the program
    To what extent does the program address a demonstrable need and respond to the needs of Canadians?
  • Alignment with government priorities
    To what extent are NODP objectives linked to (i) federal government priorities and (ii) departmental strategic outcomes?
  • Alignment with federal roles and responsibilities
    To what extent are the roles and responsibilities of the federal government being met in delivering the program?

Performance

  • Achievement of expected outcomes

    To what extent does program management regularly identify and act on required improvements in delivering the program?

    What factors have facilitated or impeded the delivery of the program?

    To what extent has the program achieved progress toward immediate outcomes, intermediate outcomes, and ultimate outcomes in terms of performance targets, program reach, and program design?

    What are the linkages and contributions of outputs to outcomes?

  • Demonstration of efficiency and economy

    What has been the utilization of resources in relation to the production of outputs and progress toward expected outcomes?

    To what extent is the NODP cost-effective?

The complete evaluation matrix appears in Annex B.

2.2 Evaluation methodology

The study methodology specifically ensured that multiple lines of evidence were available for all issues and questions. The methodology included a review of documents and literature, an administrative data and file review, interviews, and case studies. In addition, rather than duplicating the efforts of previous studies, the study team used pre-existing research such as case studies and literature reviews, where feasible, to minimize the burden on individuals consulted.

The evaluation findings and conclusions are based on the analysis and triangulation of multiple lines of evidence (see also Annex B, evaluation matrix). This subsection describes these various methods.

2.2.1 Document and literature review

The document review included materials provided by the program, including past evaluations to minimize duplication of work. The literature review incorporated and extended work undertaken as part of the Northern Ontario Development Fund (NODF)21 summative evaluation. This review focused on rural economic development and included recent studies conducted by the Organization for Economic Co-operation and Development (OECD).

It is important to note the unique situation in northern Ontario and to use the literature as indicative of current trends in regional/community economic development rather than as a policy template.

2.2.2 Administrative data and file review

The administrative data review included excerpts from the FPE/indicator database and the Grants and Contributions Reporting System (GCRS) provided electronically by the program.

The file review included the selection of 40 projects to be analyzed in more detail using Access software. These projects comprised the 20 largest projects in terms of funding plus a random selection of 20 additional projects. Some of the 20 randomly selected projects were replaced with other projects also selected randomly to ensure representation of all program areas as well as representation of the two main regions of northern Ontario (northeast and northwest).

Annex C presents the list of projects included in the file review. See Annex D for a list of the fields that were captured in the Access database.

For each project selected, the program forwarded the application, contribution agreement, final report, and progress reports. These files contained all the information necessary to complete the fields in the Access database.

2.2.3 Interviews

A total of 42 interviews were completed with the following groups:

  • External stakeholders (29)
  • Program managers (6)
  • Project officers (5)
  • Academics (2).

Interviewees received an invitation by email from IC/FedNor in both official languages to inform them that PRA would contact them to participate in an interview. A copy of the letter in both official languages appears in Annex E.

Each interview group received a tailored interview guide (in both official languages) prior to the interview, to support preparation of responses. A pretest with two interviewees in each group was conducted to ensure clarity and comprehension. Only minor changes were necessary. Copies of the finalized interview guides in both official languages also appear in Annex E.

The interviews were conducted by phone, in the official language of the participant's choice. Interviewees received assurance of confidentiality and anonymity prior to the interview. With their permission, interviews were audio-recorded and, to ensure the accuracy of responses, interviewees received notes of their responses.

The findings of the interviews use the following scale:

It is important to note that, in analyzing the findings from the interviews, two situations can occur.

First, there may be agreement across all groups of interviewees, but a single respondent may offer a perspective, based on experience or a unique position, that materially clarifies an aspect of the program or context. A single interviewee may raise a unique point that helps clarify some other aspect of information.

Second, since not all groups of interviewees were asked to respond to each evaluation question, and since different groups may have diverging opinions, findings are linked to the interview group that provided the information. Where these situations arise, they are noted in the findings. Of course, a single view needs corroboration from other information to ensure its validity.

2.2.4 Case studies

The evaluation completed eight case studies. Three were selected from the prior summative evaluation of the NODF (2006) to support a longer follow-up and update. Five were selected (in consultation with FedNor) based on:

  • Size of budget
  • Strategic goal
  • Region
  • NODP supported area
  • Special populations (e.g., Official Languages Minority Communities, Aboriginals)
  • Type of recipient organization (e.g., SME, not-for-profit, municipal).

The projects listed in Table 4 served as the case studies for this evaluation.

Table 4: NODP case studies
Name Proponent NODP area NODP funding Total project cost
New projects
Revitalization of Main Docks in Little Current Town of Northeastern Manitoulin and the Islands Trade and Tourism $500,000 $3,529,887
Assist with Launch and Production of MultiStations OS2 Product Line Three H Furniture Systems Limited Innovation $108,000 $649,000
Extension of Water and Sewer for the North Dryden Commercial Business Park City of Dryden CED $1,300,000 $3,885,000
Establishing a Northern Agriculture and Food Pavilion at the 2009 Royal Agricultural Winter Fair NECO Community Futures Development Corporation CED $348,300 $348,300
Development of a Five-Year Strategic Plan Association des francophones du Nord-Ouest de l'Ontario CED $34,578 $41,700
Updated/follow-up projects
Undertake Work on Ultrasensitive Diagnostics for Early Detection Genesis Genomics Inc. Applied Research and Development (formerly Pre-Commercial Fund) $875,000 $1,718,290
Development of a Marketing Strategy and Related Materials to Attract Potential Businesses to Timmins Timmins Economic Development Corp. NODF – Non-capital $105,000 $334,891
Extend Telemedicine Services to 19 Remote First Nations Keewaytinook Okimakanak/Northern Chiefs Council ICT Applications $450,000 $450,000

Case study data comprised an interview with the project proponent and a review of the NODP project file. In some cases, the project proponents provided additional documentation and information not available in the FedNor files.

Project proponents received an invitation letter from IC/FedNor by email saying PRA wanted to interview them concerning their NODP-supported project. Copies of the letter and the interview guide in both official languages appear in Annex F.

Phone interviews with project proponents were in the official language of the proponent's choice. The information collected from the interviews was combined with the information in the project files to develop case studies of about five pages. Case study interviewees received a draft of the case study for their review to ensure accuracy. Annex G presents the finalized case studies.

2.3 Methodology limitations

The following are the three main limitations of this evaluation.

Outcomes are not independently verified – Information regarding immediate outcomes came from administrative and self-reported data. These data appear in an indicator database. Although the indicator database provides an opportunity to measure the success of the program at achieving its immediate outcomes, the data have not been independently verified and rest on the representations of the project proponent in the FPE. Project officers, who flag issues for clarification, review all these reports. FedNor has now started to conduct edit checks using an automated system to ensure data quality. Any issues identified during the checks are referred to an officer for clarification.

Measured outcomes are short-term – As per the NODP Results-Based Management and Accountability Framework, the program collects quantitative data on short-term outcomes at the end of every project and relies on qualitative data collected at the time of program evaluation to measure intermediate and long-term outcomes. The program does not followup on projects to collect quantitative information on the achievement of intermediate and ultimate outcomes.

Attribution of outcomes to NODP is difficult – The majority of projects supported by the NODP are jointly funded. This makes it difficult to attribute the success of outcomes achieved specifically to the NODP. That said, all evaluations of cost-shared programs would face this issue.


top of page

3.0 Findings

This section provides the evaluation findings by issue, using information generated by the methods described in Section 2.0.

3.1 Relevance

Issue 1: Continued need for the program
To what extent does the program address a demonstrable need and respond to the needs of Canadians?

Summary of findings: There is a continued need for the NODP to strengthen and diversify the northern Ontario economy, which has historically been reliant on primary resources. In addition, evidence shows that available economic development programs in northern Ontario are complementary and result in very little overlap and duplication. Some interviewees believe the NODP could better address the needs of the region by investing more directly in businesses and targeting its resources to fewer, larger, and strategically positioned projects.

Interviewees agreed that northern Ontario's vulnerability to price fluctuations and variations in exports will continue to lead to out-migration and an overall shrinking economy. The isolation of certain communities, the distance between these communities, and their reliance on one industry make them extremely vulnerable to these outside shocks. Interviewees noted that an effort to diversify the economy and create jobs and businesses is essential to building a sustainable economy in northern Ontario.

The NODP aims to foster economic growth and development and diversify the regional economy by supporting projects in six thematic areas. Table 5 shows the NODP support provided to projects by thematic area and provides examples of the types of projects funded under each area.

Table 5: NODP support by thematic area (April 1, 2006 – January 31, 2010)
NODP funding area NODP funding % of total NODP funding Types of projects funded
Source: FedNor. (2010). GCRS database. Provided by program May 18, 2010.
CED $39,960,298 27% Infrastructure (industrial, socio-economic); strategic planning; business plan; local initiative contributions; CED implementation
Innovation $34,387,450 23% Capital (expansion, access to additional equipment); non-capital (studies, research)
Trade and Tourism $25,564,089 17% Expansion/export development (delivery of training and services, marketing); inward investment (readiness studies); capital (construction); non-capital (events, studies, promotion)
ICT $21,844,620 15% Infrastructure (installation of broadband and satellite); applications (e.g., strategic plans, develop multimedia centres)
Human Capital $13,396,143 9% Youth Internship Program; business management skills; labour market skills development
Business Financing Support $12,847,497 9% Loan/investment fund loss reimbursement; capitalization of Community Futures Development Corporations (CFDCs)
Total assistance $148,000,097 100%  

As the table demonstrates, the NODP supports several different areas and project types, such as investments in infrastructure, public services, ICT, Innovation, and the development of local amenities to support tourism and attract high-skill industries. The expectation is that such diverse support will "accelerate Northern Ontario's movement to a knowledge-based economy and build more globally competitive businesses"22 a nd mitigate the effects of the external variations in demand for their primary resources.

In addition, based on the output and outcome indicators in the database, it appears that many of the NODP projects within the thematic areas create and maintain immediate jobs, and about half lead to immediate business support.23 All of the NODP immediate outcomes for each NODP thematic area appear in Annex H.

Table 6 shows that the NODP has invested the majority of its funding in non-profit organizations (57%) and municipalities (25%), whereas only 3% of the funding has gone to for-profit businesses. Some interviewees noted that the NODP could better address the needs of Canadians by investing more directly in for-profit businesses.

Table 6: NODP support by applicant type (April 1, 2006 – January 31, 2010)
NODP applicant type NODP funding % of total NODP funding
Source: FedNor. (2010). GCRS database. Provided by program May 18, 2010.
Non-profit organization $85,137,553 57%
Municipality $37,096,193 25%
Educational institution $10,938,201 7%
Aboriginal community $8,959,171 6%
For-profit organization $4,394,889 3%
Province/provincial Crown corporation $1,345,298 1%
Co-operative $128,792 <1%
Total $148,000,097 100%

However, a closer examination of funding for non-profit organization projects shows that much of the spending classified as "non-profit" really comprises investments in infrastructure such as broadband, health care facilities, and other public capital (see Annex I for more detail). Much of these investments can arguably be linked to support for economic development. Increased broadband connectivity, for example, is often viewed as a critical infrastructure investment that supports the knowledge economy.

In addition, some interviewees suggested the program could better address diversification goals and regional economic performance by targeting investments in fewer, larger, and strategically positioned projects. Much of the recent academic literature aligns with this advice. Some key concepts in the literature included:

  • Place-based policy – "Focus on policy issues as they play out in concrete geographic and community settings, as a means to address challenges and opportunities where the impacts are directly felt. In the context of regional economic development, it means drawing on the assets and opportunities of a 'place' to develop policies/strategies that support economic development."24
  • Comparative versus competitive advantage – Invest based on competitive advantage (e.g., firms and communities that will be successful in market exports outside the region) instead of comparative advantage (e.g., firms and communities that specialize in standard northern Ontario resources).
  • Strategic investment – Focus investments on high potential areas and businesses that will stimulate economic development by building on local strengths and opportunities.25

Many interviewees and case study participants noted that the various programs available in northern Ontario for regional economic development complement each other and that collaboration is essential for any project to move forward. Project data show that the provincial government is often a joint funder on NODP-supported projects. Such joint federal-provincial funding also mitigates the risk to any single funder.

Of the 40 projects included in the file review, the most common funding partner is the Province of Ontario, especially the Northern Ontario Heritage Fund Corporation (NOHFC).26 Just under half (18 projects) were jointly funded by the province, including about one-third (12 projects) through the NOHFC.

Recent consultations with NOHFC management reached the same conclusion, that there is no overlap and duplication between the economic development programs available in northern Ontario. They noted that stacking funds for projects was permissible under the NOHFC, and that complementary programs were necessary to move projects forward in northern Ontario.

Annex K lists some examples of available programs to support economic development projects in northern Ontario.

top of page

Issue 2: Alignment with government priorities
To what extent are NODP objectives linked to (i) federal government priorities and (ii) departmental strategic outcomes?

Summary of findings: The NODP supports federal government priorities and IC's strategic outcomes.

Recent Speeches from the Throne and federal Budgets provide evidence that the NODP has supported and continues to support federal government priorities. From 2006 to 2010, Speeches from the Throne have discussed the importance of maintaining and growing Canada's economy. The 2010 Speech identified the importance of creating new jobs and supporting rural industries.27

Federal Budgets support the Speeches from the Throne by earmarking funding for economic development and job creation. The 2010 Budget Address included a focus on regional economic development titled Leading the Way on Jobs and Growth. It also highlighted the important role that Regional Development Agencies (RDAs) play "by promoting innovation and the commercialization of research in communities throughout Canada." 28

Interviewees concurred that NODP objectives link to and support federal government priorities. Many noted that it is now more the case than ever because of the economic downturn and the importance of economic development in Canada.

Within the IC Program Activity Architecture, the NODP contributes directly to the strategic outcome: Canadian businesses and communities are competitive. Table 7 shows how the NODP advances IC's strategic outcomes.

Table 7: Industry Canada strategic outcomes and NODP action
IC strategic outcomes NODP action
Source: Industry Canada. (2010). Mandate. Retrieved October 3, 2010, from www.ic.gc.ca/eic/site/icgc.nsf/eng/h_00018.html
A fair, efficient, and competitive marketplace The NODP invests in strategic infrastructure in high technology areas, widely believed to be the foundation of a competitive economy.
An innovative economy The NODP has invested in innovation through broadband, and technology ventures.
Canadian businesses and communities are competitive NODP investment increases diversification and competitiveness as a way to promote sustainability in the region.
top of page

Issue 3: Alignment with federal roles and responsibilities
To what extent are the roles and responsibilities of the federal government being met in delivering the program?

Summary of findings: The NODP is consistent with federal roles and responsibilities.

The Government of Canada is committed to promoting equal opportunities for the well-being of Canadians and furthering economic development to reduce disparity, as set out in the Constitution Act, 1982. The Department of Industry Act, particularly section 4(2), sets out how the Minister supports the Government of Canada in its commitment to equal opportunities and reducing regional disparities. Sections 8 and 9 of the Act set out the Minister's responsibility for regional economic development in Ontario. FedNor is the sole federal presence in northern Ontario supporting the Minister in fulfilling these responsibilities.

The NODP promotes economic growth, diversification, job creation, and sustainable communities in northern Ontario, and is the government's response to regional development in that area. The NODP complements the Government of Ontario's initiative known as the NOHFC, which works with northern entrepreneurs, communities, and business organizations to support vital infrastructure and economic development projects to create jobs and stimulate the economy.

There is a general understanding by both levels of government regarding the need to harmonize federal and provincial approaches to northern Ontario economic development to allow for coordination and potential collaboration in areas of mutual interest and where circumstances permit. Ongoing communication, particularly among program officers, ensures duplication does not occur and that initiatives from both levels of government build on each other's successes.

Interviewees noted that the federal government has a clear role and responsibility to deliver a program such as the NODP in northern Ontario. Although the respondents agreed there is a role for all levels of government in this area, they view the federal government as having a responsibility to lead the way for the province, municipalities, non-governmental organizations, and other institutions.

The recent academic literature confirms interviewees' views. The theme of governance figures prominently in the OECD's The New Rural Paradigm and the Canadian Policy Research Networks' (CPRN) review of rural and regional development policies and programs.

The OECD argues that The New Rural Paradigm "requires important changes in how policies are conceived and implemented to include a cross-cutting and multi-level governance approach," adding that "traditional administrative hierarchical structures" are unlikely to suffice for this purpose.29 Developing a governance structure supportive of the new approach to rural policy will require "adjustments … along three key governance dimensions: horizontally at both the central and the local levels and vertically across levels of governance."30

Similarly, the CPRN review finds that novel approaches to policy in the area of rural and regional development incorporate new governance models characterized by:

  • "partnerships among governments, the private sector, communities, voluntary organizations and others;
  • greater devolution to the regional and local level; new roles for the state as an 'enabler' and 'convenor'; and
  • the sharing of power and authority with the community itself."31
top of page

3.2 Performance

Issue 4: Achievement of expected outcomes

Issue 4A: To what extent does program management regularly identify and act on required improvements in delivering the program?

Summary of findings: Program management has made some improvements to the program in response to issues identified in the Mid-Term Evaluation of the NODP. However, limited progress has been made on the collection of performance measurement information on outcomes beyond the project end date.

Previous evaluations have identified the need to improve certain aspects of performance measurement. The most recent of these evaluations—the Mid-term Evaluation of the NODP (2008)—identified ways to strengthen the program's Performance Measurement System and data quality:

  1. FedNor should determine the benefit and cost of collecting performance measurement information on project outcomes beyond the end date of projects. If appropriate, FedNor should extend its Performance Measurement System to capture these longer-term results.
  2. FedNor should strengthen mechanisms and formalize ongoing edit checks to ensure data quality, including:
    1. consistent comparison of intended and actual outcome data for each project
    2. both program officers and proponents report on final results independent of intended outcomes captured at the project launch stage
    3. analyze minimum, maximum, and most frequent values for indicators
    4. create separate sub-indicators based on project grouping with unique outcome characteristics
    5. address concerns and anomalies with data and indicators as required.32

In a follow-up report (January 2010) to the Mid-Term Evaluation of the NODP, FedNor developed several actions to address these recommendations. Table 8 sets out these actions, as well as an assessment of progress.

Table 8: FedNor action and implementation in response to NODP Mid-term Evaluation recommendations
Issue FedNor action Assessment of implementation
Source: FedNor. (2010). Follow-up report to the NODP Mid-term Evaluation: performance measurement considerations and recommendations. Provided by program May 18, 2010.
I FedNor is expanding its Performance Measurement System to collect longer-term impact data through an Internet survey involving students. An implementation plan has been developed for the first year of measurement. Planning is ongoing. No indication that longer-term impact data have been collected yet.
IIa Indicators on the Project Summary Forms (PSF), which are submitted during the application stage, be collected in a separate form using a smart PDF similar to the one used to collect indicator data from the FPE. The smart PDF would be required for all PSFs. Has been implemented. Projects that started later in the file review had separate forms for intended and final outcomes.
IIb Proponents report directly to their project officers for the project officers' review. If there is any disagreement in the reported outcomes, the project officers and proponents would discuss to ensure the data are mutually agreed upon before they can be submitted to the project database. Being implemented.
IId FedNor developed indicator groupings to bring indicators across different groups together. Both the grouped and separate indicators are being captured. Has been implemented. These indicator groupings are being reported in the indicator summary reports.

As is shown in the table, progress is being made on most fronts. For example, a separate form for intended and final outcomes is being used for projects that started later in the file review.

However, limited progress has been made on the collection of performance measurement information on project outcomes beyond the project end date. Changes to address the longer-term measurement of project outcomes are being planned and integrated into the revised NODP Performance Measurement System.

Issue 4B: What factors have facilitated or impeded the delivery of the program?

Summary of findings: FedNor regional staff's expertise is key to the effective delivery of the program. However, the timing of the project approval process is hampering the success of the program.

The main factor that supports effective delivery of the NODP is the involvement and experience of FedNor regional staff. Regional FedNor offices in northern Ontario enable direct contact with the proponents and FedNor staff are knowledgeable of the communities' needs. Interviewees noted that the level of FedNor support and staff's knowledge of northern Ontario facilitates the delivery of the program. The direct involvement of FedNor staff in project design and planning guides investment decisions, mitigates risks associated with projects, and enhances the likelihood of realizing NODP's outcomes.

FedNor's continued presence and experience in northern Ontario appears also to have resulted in linkages with other federal departments, provincial and local governments, and non-governmental organizations. Building such relationships can be important in supporting the delivery of the program since FedNor officers can coordinate support with outside partners to move projects forward.

The main issue impeding program delivery is the time required for project approval, according to interviewees and case study respondents. They noted that project approval times for the NODP are longer compared to previous FedNor-administered programs.

An analysis of the 858 NODP projects approved between April 1, 2006 and January 31, 2010 showed that the average approval time was about 140 working/business days. About one-quarter of the projects were approved in three months or less, and about 60% were approved in six months or less. More than one-third took more than six months for approval and more than 10% took more than a year.

RDAs comparable to FedNor, such as Canada Economic Development-Quebec and Western Economic Diversification Canada (WD), have processing time commitments in working/business days of 25–46 days and 60 days (in 90% of cases), respectively.33 Canada Economic Development-Quebec meets its committed approval time in 50% of its cases.34 Although the Atlantic Canada Opportunities Agency (ACOA) does not have a predetermined commitment time, it has measured its approval times at between 88 days and 8 months, depending on the program.35

top of page

Issue 4C: To what extent has the program achieved progress toward immediate outcomes, intermediate outcomes, and ultimate outcomes in terms of performance targets, program reach, and program design?

Summary of findings: Evidence shows that the NODP has achieved its immediate outcomes. However, evidence on the achievement of intermediate and ultimate outcomes is mixed. Qualitative evidence from interviewees and case study participants indicates that the program is achieving these outcomes. However, intermediate and ultimate outcomes are not systematically tracked and measured by FedNor; thus, there is little quantitative evidence of the achievement of long-term outcomes.

Immediate outcomes

The FPE indicator database captures several indicators under each NODP thematic area to measure the achievement of the immediate outcomes of the program. Indicators from each project are entered into the indicator database. In total, 90 indicators are captured across the 6 thematic areas. A full list of the indicators and the results expected and achieved by the projects can be found in Annex H.

In response to the NODP Mid-term Evaluation, FedNor collapsed certain indicators into 10 indicator groups, which are common across thematic areas. The most commonly reported indicators across thematic areas are businesses supported; jobs created and maintained; studies/plans developed; events held; and number of participants at events.

Table 9 presents the results of these indicator groupings; it includes data for 858 projects ending between April 1, 2006 and March 31, 2010. The table includes three results columns:

  • Count – The number of projects that contributed to the indicator
  • Sum – The total number from the projects that contributed to the indicator (e.g., one project may benefit a number of businesses and organizations)
  • Mean – The average number per project that contributed to the indicator (Sum/Count = Mean).
Table 9: Grouped indicator results (April 1, 2006–March 31, 2010)
# Indicator Count Sum Mean
* Not all indicator data had been received for fiscal year 2009–10 at the time of this report because officers may put projects into a status 618, deferring completion of the final evaluation for up to 12 months. This is normally used for projects where outcomes may not be fully evident by the project end date.
1 Businesses and organizations benefiting from a project 578 12,863 22.25
2 Events (e.g., workshops, trade missions, training, meetings, conventions) 125 1,487 11.90
3 Number of participants at events 378 49,699 131.48
4 Studies or plans 199 327 1.64
5 Full-time direct job equivalents created as a result of the project 132 2,696 20.42
6 Full-time direct job equivalents maintained as a result of the project 79 1,845 23.35
7 Permanent jobs created as a result of the project 130 2,624 20.18
8 Permanent jobs maintained as a result of the project 77 2,524 32.78
9 Temporary jobs created as a result of the project 434 3,083 7.10
10 Temporary jobs (in person years) created as a result of the project 433 3,025 6.99

The indicators provide direct evidence for the achievement of the NODP's immediate outcomes. For example, the number of events held and number of participants at events lend themselves directly to the enhancement of markets and investment attraction under NODP's Trade and Tourism thematic area. The number of studies and plans developed directly supports improved community capacity for socio-economic development, enhanced innovation capacity and improved infrastructure, and enhanced ICT applications and infrastructure.

The case studies provided corroborating evidence that the NODP is achieving its immediate outcomes. Table 10 lists the five case studies examined in this evaluation, the expected NODP outcome, and the quantitative measures that led to the achievement of that outcome.

Table 10: Case studies' achievement of immediate outcomes (case studies included only in the 2010 evaluation)
# Case study project NODP area and expected outcome Achieved results
1

Revitalization of Main Docks in Little Current – The Town of Northeastern Manitoulin and The Islands

Trade and Tourism

  • Enhancement of markets and investment attraction
  • 2 businesses created
  • 30 businesses maintained
  • 5 temporary jobs created
  • 2 permanent jobs created
  • 8 permanent jobs maintained
  • Dockage fees increased from $40K to $80K per year
2

Establishing a Northern Agriculture and Food Pavilion at the 2009 Royal Agricultural Winter Fair – NECO Community Futures Development Corporation

CED

  • Improved community capacity for socio-economic development
  • 41 businesses benefited
3

Development of a Five-Year Strategic Plan – Association des francophones du Nord-Ouest de l'Ontario

CED

  • Improved community capacity for socio-economic development
  • 1 community plan developed and accepted
4

Extension of Water and Sewer for the North Dryden Commercial Business Park – City of Dryden

CED

  • Improved community capacity for socio-economic development
  • 8 businesses benefited
  • 30 temporary jobs created
  • 275 permanent jobs created
  • 10 permanent jobs maintained
5

Assist with Launch and Production of the MultiStations OS2 Product Line – Three H Furniture Systems Limited

Innovation

  • Enhanced innovation capacity and improved infrastructure
  • 1 business benefited
  • 5 permanent jobs created
  • Expected that 5 more permanent jobs would be created

Table 10 shows that there is an overall benefit to businesses, particularly the creation and maintenance of jobs. Other important indicators included the development of a community plan and increased revenue into a community.

In addition, Table 10 highlights the case studies' achievement of their intended outcomes. For example, the Revitalization of Main Docks in Little Current project was expected to enhance markets and investment attraction. According to the case study information, revitalization of the docks resulted in stores opening along the docks and in the community, which has led to additional employment. The Development of a Five-Year Strategic Plan led to the development of a community plan, which improved the region's opportunities for socio-economic development.

Interviewees supported the indicator and case study evidence that the NODP is achieving its immediate outcomes. However, it is difficult to determine whether the program has been as successful as expected, as targets were only set for those indicators reported on in the RPP.

Although proponents are required to identify indicators and estimate their expected contribution to those indicators at the start of projects, these goals only provide targets for the individual projects. With respect to overall targets, some are reported on through IC's RPP, but they are limited in scope.

In particular, the RPP provides annual targets for the number of businesses created, expanded and maintained, and dollars leveraged by NODP investments. However, targets for number of jobs created and maintained and several other outcomes were not developed at all.

FedNor has indicated that although it would be difficult to set targets for several of its measured outcomes, it is refining its performance reporting for 2011–12 and will report on jobs created and maintained in the RPP and DPR.

Intermediate and ultimate outcomes

Most interviewees and all case study proponents agreed that the NODP has achieved progress toward its intermediate and ultimate outcomes. Case study proponents provided direct examples of how their projects are achieving these outcomes. Interviewees mentioned the following indicators to show that the program is progressing toward these outcomes:

  • The development and sustainment of jobs and businesses
    • development of long-term employment for youth interns
    • increase in specialized research positions
  • Support provided to various industry initiatives
  • Innovation and infrastructure work that has contributed to the health sector and the economy.

Intermediate and ultimate outcomes can be problematic to measure. Currently, little data on intermediate and ultimate outcomes are collected and there are few resources allocated to performance measurement. The indicator database provides several "indications" of immediate outcome achievement, but does not capture any quantitative measures in the longer term of jobs continuing to be maintained, increased sales, or other measures of business expansion.

In an effort to measure longer-term outcomes, the evaluation conducted three case studies of projects that had been studied by the preceding NODF. Table 11 provides the longer-term outcomes retrieved from the websites and those mentioned by proponents for the three "updated" case studies.

Table 11: Longer-term results for the "updated" case studies
# Case study project Longer-term results
1

Extend Telemedicine Services to 19 Remote First Nations – Keewaytinook Okimakanak (KO)

  • 26 KO Telehealth Community Telehealth Coordinators completed their certification course, including seven in the communities supported by NODF funding, resulting in permanent and full-time employment
  • Business and economic opportunities in the communities
  • Leveraging of millions of dollars
  • Telehealth services in the communities are an ongoing and sustainable operation
  • Patient and physician satisfaction with the services
2

Development of a Marketing Strategy to Attract Potential Business to Timmins Ontario – Timmins Economic Development Corporation

  • Increased business from the cold weather testing sector—smaller and temporary cold weather testing activities
  • New businesses locating in the area or expanding their presence
  • Creation of 99 full- and 182 part-time jobs in the retail and service sector
  • Hundreds of construction jobs from the development of new retail facilities
  • Use of material by real estate developers to attract business
  • Continued use of the promotional materials developed
3

Development and implementation of ultrasensitive diagnostics to detect cancer at the earliest possible stage – Genesis Genomics Inc.

  • Development and patent of the Prostate Core Mitomic Test™
  • Attraction of educated professionals to the region
  • Creation of jobs for highly qualified personnel

The NODF case studies conducted as part of this evaluation suggest that the NODP is achieving longer-term outcomes. Specifically, they provide examples of jobs, created during a project, that were maintained and of expected future employment that has been realized and could be measured by the program.

For example, the Extend Telemedicine Services to Remote First Nations project, implemented by KO, created seven part-time jobs during the project. As of 2005, those jobs became full-time, and individuals were still employed as of November 2010.

The Development of a Marketing Strategy to Attract Potential Business to Timmins Ontario project, implemented by the Timmins Economic Development Corporation, expected to create approximately 300 new jobs in the retail and service sector by the end of 2007. As of April 2010, a total of 281 new jobs had been created.

Table 11 provides mostly qualitative evidence of success of the proponents in their fields, but the lack of continued measurement of outcomes by the proponent and the program provides limited quantitative evidence of these successes.

top of page

Issue 4D: What are the linkages and contributions of outputs to outcomes?

Summary of findings: Projects funded under the NODP have led to several outcomes and have contributed to the economic development of northern Ontario. Many projects would not have proceeded or would have proceeded at a reduced scope, without funding from the program.

The NODP's "most direct and measurable output is contributions for projects to proponents to support activities related to socio-economic development of Northern Ontario."36

It is evident from the findings presented that the NODP-supported projects have led to several outcomes, including socio-economic development in northern Ontario. Many interviewees and case study proponents indicated that NODP-supported projects would not have proceeded or would have proceeded at a reduced scope if NODP funding had not been received.

Table 12 displays the outputs and achieved results for five case studies.

Table 12: Case study outputs and achievement of immediate outcomes
# Case study project Outputs Achieved results
1

Revitalization of Main Docks in Little Current – The Town of Northeastern Manitoulin and The Islands

  • Improvements and construction to the dock
  • 2 businesses created
  • 30 businesses maintained
  • 5 temporary jobs created
  • 2 permanent jobs created
  • 8 permanent jobs maintained
  • Dockage fees increased from $40K to $80K per year
2

Establishing a Northern Agriculture and Food Pavilion at the 2009 Royal Agricultural Winter Fair – NECO Community Futures Development Corporation

  • Established a Northern Agriculture, Food and Equine Pavilion at the 2009 Royal Agricultural Winter Fair
  • 41 businesses benefited
3

Development of a Five-Year Strategic Plan – Association des francophones du Nord-Ouest de l'Ontario

  • Developed a strategic plan for 2009–14
  • Developed a socio-economic profile of the francophone community of northwestern Ontario
  • 1 community plan developed and accepted
4

Extension of Water and Sewer for the North Dryden Commercial Business Park – City of Dryden

  • Development of eight hectares of investment-ready commercial lots
  • 8 businesses benefited
  • 30 temporary jobs created
  • 275 permanent jobs created
  • 10 permanent jobs maintained
5

Assist with Launch and Production of the MultiStations OS2 Product Line – Three H Furniture Systems Limited

  • Expansion of manufacturing facility
  • Purchase of specialized equipment to develop a new product line and make manufacturing process more effective
  • 1 business benefited
  • 5 permanent jobs created
  • Expected that 5 more permanent jobs would be created

In each case, it is evident that outputs directly contributed to the achievement of results for the project, which contributed to the immediate outcomes of the NODP. For example, the Revitalization of Main Docks in Little Current project improved the community's dock, which led to the opening of businesses along the dock and in the community.

However, the major barrier to measuring outcomes is that follow-up beyond the end date of the projects is not systematically completed to assess whether the intermediate and longer-term outcomes have occurred. For example, the Assist with Launch and Production of the MultiStations OS2 Product Line project was expected to result in the creation of five additional permanent jobs following the completion of the project; however, aside from the data collected at the time of program evaluation, no follow-up was conducted to verify these jobs had been created.

top of page

Issue 5: Demonstration of efficiency and economy

Issue 5A: What has been the utilization of resources in relation to the production of outputs and progress toward expected outcomes?

Summary of findings: The NODP has a cost per job created/maintained of $5,229 and administration costs of about 22%. It is difficult to assess whether the cost per outcome is reasonable due to data limitations and the broad role that FedNor staff play beyond the administration of contributions. Administrative costs appear to be reasonable to support program delivery over a vast geographic area, with requisite support for effective communications and core services.

Between April 1, 2005 and March 31, 2010, FedNor distributed $196.55 million in NODP Gs&Css at a total O&M cost of $54.22 million. Therefore, under the NODP, FedNor has an administrative cost of less than 22%.

The cost to administer a community-based Gs&Css program, such as the NODP, can vary significantly depending on the nature of the program. For this program, these costs appear to be reasonable to support FedNor's capacity to deliver its programming over a vast geographic area, with requisite support for effective communications and core services.

The program's operations include a total of 121 FTEs, with decentralized delivery in 7 offices in northern Ontario and travel to monitor projects, provide a high level of service to proponents, and maintain close contact with local communities. Furthermore, the actual administrative costs for the NODP are in line with program and funding approvals.

Interviewees noted that the experience and knowledge of FedNor staff in the regions and their high level of availability and support are instrumental to the development of projects and success of the program. FedNor staff are also involved in a full range of services beyond the administration of contributions that do not result in projects funded by the NODP (e.g., policy advice, coordination, facilitation, referrals). This suggests the administrative costs to deliver the NODP, as currently defined, are reasonable.

The administration costs of other regional development agencies were reviewed to help determine whether NODP administration costs were reasonable. According to a report produced by Canada Economic Development-Quebec (2010), overall administration costs range from about 16% for Canada Economic Development-Quebec to 26% for the ACOA. 37 The report does not separate the budgets for individual programs delivered by the RDAs. However, this does show that NODP's administrative costs are within the range of other Canadian RDAs.

With respect to NODP costs per outcome, 23 of the 40 projects reviewed as part of the file review resulted in temporary and/or permanent jobs created and/or maintained. Those 23 projects created or maintained a total of 1,637 temporary and permanent jobs at a total NODP funding of $8,560,499, and a total project cost of $30,138,317. Therefore, based on the sample files, the cost to the NODP per temporary and permanent job created and maintained was $5,229.38

It should be noted that creating and maintaining temporary and permanent jobs were not the only outcomes expected from these projects. Many of the projects also expected to achieve other outcomes, such as creating, maintaining, and benefiting businesses; distributing loans; and improving access to capital and leveraged capital. However, since these outcomes are not aligned with specific elements of the contribution, it is not possible to separate the outcomes individually.

Issue 5B: To what extent is the NODP cost-effective?

Summary of findings: The NODP has been successful at securing financial contributions for projects from other sources. However, some interviewees suggested that the program could be more cost-effective if it were refocused. In particular, they felt funds should be allocated more strategically to core economic development activities and should be targeted more to projects that would impact businesses. The literature on regional economic development aligns with the views to focus impact and spending on strategic activities and business.

Between April 1, 2006 and January 31, 2010, the NODP approved Gs&Cs support of $148,000,097 for projects totalling $557,548,777. Therefore, for each dollar contributed to a project by the NODP, $2.77 was contributed to the project from other sources, such as the provincial government, municipalities, private sector investment, and proponents. Table 13 shows the overall leverage effect by thematic area.

Table 13: NODP leverage effect by thematic area (April 1, 2006–January 31, 2010)
NODP funding area NODP funding Total cost NODP % Leverage effect
CED $39,960,298 $166,733,637 23.97% $3.17
Innovation $34,387,450 $176,550,811 19.48% $4.13
Trade and Tourism $25,564,089 $95,587,344 26.74% $2.74
ICT $21,844,620 $ 79,568,014 27.45% $2.64
Human Capital $13,396,143 $25,271,281 53.01% $0.89
Business Financing Support $12,847,497 $13,837,690 92.84% $0.08
Total $148,000,097 $557,548,777 26.54% $2.77

This table shows that Innovation, CED, and Trade and Tourism are the most successful at attracting other funds whereas Business Financing Support and Human Capital are least successful.

Business Financing Support is the least successful at attracting funds because several of the projects funded under this thematic area are Community Futures Development Corporations (CFDC) capitalization projects, which are 100% funded by the NODP. Although the initial NODP contributions to the investment funds of CFDCs do not leverage funds, the NODP contribution enables CFDCs to make loans, which, in turn, leverage funding. This type of leverage is not identified at the project approval stage, (as is the case for other types of projects), and as a result is not shown in Table 13. However, the number of loans and the funding leveraged as a result of capitalization projects is tracked elsewhere, and reported as final project outcomes.

Similarly, the majority of projects funded under the Human Capital thematic area are Youth Internship Program projects, which are normally funded up to 90% by the NODP.

Interviewees and case study proponents indicated that the NODP is a catalyst in attracting outside sources of funding and that projects would not have gone ahead at all or with the same scope without the NODP funding; however, it is difficult to imagine this is the case for projects where NODP funding was minimal.

Although some interviewees noted that the NODP allocates its resources effectively, they also noted the program could be more cost-effective if it were refocused. Two suggestions were made: the program should allocate funds more strategically instead of scattering small amounts of funding across a broad range of projects, and funds should deal more directly with and support businesses. These suggestions are consistent with the literature.

The literature indicates that past approaches to rural policy focused on redistribution of economic activity. However, the OECD (2006) argues that increasing pressure on public expenditure, combined with the general failure of redistributive approaches to reduce inter-regional disparities, suggests new approaches to rural policy are required. Instead of subsidies, the OECD's The New Rural Paradigm focuses on identifying local economic opportunities and making strategic investments to improve regional competitiveness.39

References to this approach are common in the literature. For example, Goldenberg (2008) argues that "regional and rural policy should be based on strategic investments—in people, in communities, in local resources and assets, and in technology—seeking to stimulate economic development by building on local strengths and opportunities."40

Similarly, Bradford (2010) notes that "one of the key evolutions [in federal regional development activity in Canada] has been the progression away from redistributional spending to investments in knowledge and innovation."41 Markey et al. (2008) assert that "the ascendency of place …tells us that abandoning communities and regions to the vagaries of the market are short-sighted …strategic investments are necessary to establish a foundation upon which diversified economic and community development can flourish."42 The alternative approach of spreading support widely is to fund larger projects in communities that have the infrastructure to nurture these investments.

In terms of direct support to business, this approach carries the risk that the business directly supported through the projects may not succeed. One of the factors in the past shift in regional development programming away from direct investment was that business failures were common, which fuelled criticism of government mismanagement. Another impetus to the withdrawal of government from direct investment was the fear that such spending would displace or crowd out private sector lending. This latter concern is less germane in northern Ontario and in the current recession where a shortage of finance capital appears to exist.43

As noted in Section 3.1.1, literature on rural economic development has evolved and increasingly aligns with the view to focus impact and spending on business. Potentially cost-effective regional development concepts mentioned in the literature include:

  • Focus on policy issues as they play out in concrete geographic and community settings, as a means to address challenges and opportunities where the impacts are directly felt44
  • Investment in competitive advantage versus comparative advantage, which simply means identifying activities that are likely to grow through building market share45
  • Focus on strategic investments that link to other industries or create essential support services along the value chain.46, 47

top of page

4.0 Conclusions and Recommendations

This section provides conclusions and recommendations on the evaluation issues of relevance and performance for the NODP.

4.1 Relevance

Evidence shows that there is a continued need for economic development and diversification in northern Ontario. The region's dependency on primary resource exports makes it vulnerable to price fluctuations and variations in exports that lead to periodic economic cycles of out-migration and high unemployment throughout the region. The objective of the NODP is to promote economic development, economic diversification, job creation, and sustainable, self-reliant communities in northern Ontario.

Although other programs to diversify the northern Ontario economy are available, especially from the province, the literature and interviewees agreed that a federal presence is required. The federal government, through IC, offers leadership and financial resources that complement those of the province, municipalities, non-government organizations, and other institutions to support joint investment in economic development in northern Ontario.

Providing a catalyst for joint activity, leadership among federal departments, and coordination of public resources are the roles identified by interviewees as appropriate and essential for the federal government in regional development programming.

The NODP supports the federal government's priorities and IC's strategic objectives and is consistent with federal roles and responsibilities.

4.2 Performance

Overall, the NODP has been successful at achieving its immediate outcomes and has made progress in this area since the Mid-Term Evaluation. However, evidence in the achievement of intermediate and ultimate outcomes is mixed. Interviewees and case study proponents provide some qualitative evidence of achievement of these outcomes. However, these outcomes are not tracked systematically or measured. FedNor recognizes the need to improve measurement of longer-term outcomes as this has been an issue identified in previous evaluations.

Recommendation 1: To ensure that expected outcomes are being achieved, the program should improve the measurement of medium- and longer-term outcomes.

The NODP costs appear to be reasonable to support program delivery over a vast geographic area and to provide support for effective communications and core services. Interviewees and case study proponents noted the importance of the knowledgeable and experienced FedNor staff in the regions, and indicated how the staff has facilitated the overall success of the program. However, they felt the lengthy project approval process is hampering the success of the program.

Recommendation 2: To enhance program delivery success, the program should consider improving the project approval process to decrease project approval times.

Several factors have contributed to the success of the program. However, one theme emerging from the interviews and recent literature suggests that funding a diverse array of projects is not the most effective way of achieving the program objective. Instead, funds should be allocated more strategically to core activities and targeted more to businesses.

Recommendation 3: To better achieve the overall program objective of economic development and diversification in northern Ontario, FedNor should be more strategic in its allocation of program resources and target program funding to core economic development activities. Consideration should also be given to how to increase program impacts on for-profit businesses.


top of page

Management Response and Action Plan

Recommendations Management Response and Planned Action Management Accountability Action Completion Date

Recommendation 1:

To ensure that expected outcomes are being achieved, the program should improve the measurement of medium- and longer-term outcomes.

Agreed. FedNor will implement procedures to systematically track evidence for the achievement of medium- and longer-term outcomes.

   

Action plan: Work to address this issue has already begun (e.g., review of the performance measurement frameworks of Regional Development Agencies, identification of medium- and longer-term performance indicators, and assessment of software options). Measures to track medium- and longer-term project outcomes will be implemented as part of the revised Performance Measurement Strategy (PMS) for NODP. Under the revised PMS there will be an increased emphasis on the need for proponents to track and report on the results of funded projects. Also, FedNor will introduce a follow-up report to be completed annually by the proponent up to five years after the project has ended.

Director, Policy, Planning and Coordination Unit

Actual collection of longer term data to be implemented by April 2012

FedNor will also work with the Audit and Evaluation Branch (AEB) to explore methods to document and report on the longer term impacts of funded projects.

Director, Policy, Planning and Coordination Unit

Methods identified by June 2011

FedNor resources will be reallocated to ensure sufficient capacity to undertake longer-term performance measurement.

Director, Policy, Planning and Coordination Unit

Resource reallocation to be completed by April 2011

Modifications will also be required to contribution agreements reflecting the need for proponents to report on project outcomes, and to track data in order to complete the annual follow-up reports.

Director, Policy, Planning and Coordination Unit

Policies, procedures and training to be implemented by April 2011

New software will be purchased in order to facilitate the collection of performance measurement data. Data collection forms, policies and procedures will also be developed. Staff will be provided with training to familiarize themselves with the new procedures.

 

Software to be purchased by February 2011

Recommendation 2:

To enhance program delivery success, the program should consider improving the project approval process to decrease project approval times.

Agreed. New procedures are being implemented and future improvements will be identified to help decrease project approval times.

   

Action Plan:
Examples of improvements to date, include:

  • better tracking reports and monitoring of applications;

Director, Program Delivery

QA process and monitoring of applications have been implemented

  • a quality assurance process – implemented in October 2009. This process is improving the consistent quality of project evaluations/ recommendations submitted to senior management/MINO for approval, helping to accelerate the approval process. (This process involves a systematic review of feedback received by senior management/MINO on previous project reviews, the communication of lessons learned and best practices to officers, leading to consistent higher quality project reviews/recommendations, in turn, leading to quicker response times);
   
  • Information about the program and direction on how to access funding has recently been posted on FedNor's website. For example, potential proponents now have access to program guidelines that clearly indicate the elements that FedNor considers during the application assessment process. This will help ensure that project proposals are well aligned with NODP criteria, which in turn, will help decrease project approval times.

Director, Communications

Guidelines posted on Internet by January 2011

  • an improved two stage application process has been introduced. This process will allow potential proponents to submit a streamlined Initial Application (Phase One), and receive written confirmation that they may proceed to a Detailed Submission (Phase Two), before they invest in the development of a detailed application; and

Director, Policy, Planning and Coordination

Two stage application process implemented January 2011

  • the introduction of formal service standards and related tracking/monitoring.
 

Service Standards fully operational by June 2011

Recommendation 3:

To better achieve the overall program objective of economic development and diversification in northern Ontario, FedNor should be more strategic in its allocation of program resources and target program funding to core economic development activities. Consideration should also be given to how to increase program impacts on for-profit businesses.

Agreed. FedNor is encouraging stakeholders to focus on core economic development activities to ensure that projects selected for funding provide the greatest economic impact for communities and businesses in Northern Ontario.

   

Action plan:

Program guidelines recently made available to potential proponents will help focus stakeholder efforts on projects that will generate short- to medium-term measurable results leading to economic development and growth in Northern Ontario communities and businesses, within three priority areas:

  • community economic development;
  • business growth and competitiveness; and
  • innovation.

FedNor staff efforts as catalyst to develop projects will also focus on solid projects that have the greatest potential for regional impacts and economic and business growth.

Director General, FedNor

Program guidelines made available to the public January 2011

Further research will be undertaken to explore options to invest in for-profit businesses.

Director, Policy, Planning and Coordination

April 2012


top of page

Footnotes of Final Evaluation for the Northern Ontario Development Program

1 – Industry Canada. (2010). Northern Ontario Economic Overview: Fall 2010, p. 4. Provided by program November 26, 2010. Return to text

2 – In 2006, the populations of North Bay, Sudbury, Sault Ste. Marie, and Thunder Bay were 63,424; 158,258; 80,098; and 122,907, respectively (Fillion. [2009]. Growth and decline in the Canadian urban system: the impact of emerging economic, policy and demographic trends. GeoJournal. doi: 10.1007/s10708-009-9275-8 Return to text

3 – Bollman, R., Beshiri, R., & Mitura, V. (2006). Northern Ontario's communities: economic diversification, specialization and growth (Working Paper No. 82). Agriculture and Rural Working Paper Series. Statistics Canada. Retrieved from http://www.statcan.gc.ca/pub/21-601-m/21-601-m2006082-eng.pdf (PDF Format), p. 7. Return to text

4 – Industry Canada. (2010). Northern Ontario Economic Overview: Fall 2010. Provided by program November 26, 2010. p. 3. Return to text

5 – Industry Canada. (2010). Northern Ontario Economic Overview: Fall 2010. Provided by program November 26, 2010, p. 3. Return to text

6 – Ontario Ministry of Infrastructure, & MNDMF. (2009, October). Proposed growth plan for Northern Ontario. Retrieved from https://www.placestogrow.ca/images/pdfs/Northern_Proposed_Growth_Plan.pdf (PDF Format), p. 3. Return to text

7 – Fiscal Architecture and Economic Competitiveness Working Table. (2008). Partnering to compete: strengthening Ontario's economic competitiveness (Working Paper). Retrieved from http://www.amo.on.ca , pp. 11–12. Return to text

8 – North Superior Workforce Planning Board. (2009). Building a superior workforce: 2009–2010 Labour Market Action Plan. Retrieved from http://www.nswpb.ca/assets/files/nswpb_lmap_2009-2012.pdf (PDF Format), p. 68. Return to text

9 – The North Superior Workforce Planning Board makes very similar observations in its discussion of the changing labour force needs in the Thunder Bay District, arguing that the District "has developed a greater dependence on larger employers and a more limited entrepreneurial culture which is key to developing opportunities," adding that, "in essence, entrepreneurism in Thunder Bay has had fewer opportunities to develop, making economic downturns more damaging as businesses are lost" (2009, p. 15). Return to text

10 – Southcott, S. (2007). Aging population trends in Northern Ontario: 2001 to 2006 (No. 3). Census Research Paper Series. Retrieved from http://www.awic.ca/ (PDF Format), pp. 4–5. Return to text

11 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program, p. 2. Return to text

12 – IC. (2010). Highlights of the 2009–2010 FedNor Business Plan. Retrieved May 21, 2010, from http://fednor.gc.ca/eic/site/fednor-fednor.nsf/eng/h_fn02348.html Return to text

13 – IC. (2010). Highlights of the 2009–2010 FedNor Business Plan. Retrieved May 21, 2010, from http://fednor.gc.ca/eic/site/fednor-fednor.nsf/eng/h_fn02348.html Return to text

14 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program, p. 2. Return to text

15 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program, p. 2. Return to text

16 – NODP Terms and Conditions. Return to text

17 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program, p. 2. Return to text

18 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program, p. 10. Return to text

19 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program, p. 12. Return to text

20 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program, pp. 27–28. Return to text

21 – The NODF was the predecessor to the NODP. The NODP name was adopted in July 2006. Return to text

22 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program. Return to text

23 – The application information prepared by project proponents and recorded in the indicator database itemizes the expected immediate outcomes. Most projects identify job creation as an immediate outcome, the exception being funding for business planning and other "informational" activities. Return to text

24 – Government of Canada. 2010. Policy Research Initiatives – Sustainable Places. 10(4). Retrieved from http://www.policyresearch.gc.ca/doclib/2010-0022-eng.pdf. Return to text

25 – For a full description of these concepts, see Annex J. Return to text

26 – See the table in Annex K for a description of the NOHFC. Return to text

27 – Government of Canada. (2010). Speech from the Throne. Retrieved October 3, 2010. Return to text

28 – Department of Finance Canada. (2010). Budget 2010. Retrieved July 29, 2010, from http://www.budget.gc.ca/2010/pdf/budget-planbudgetaire-eng.pdf. Return to text

29 – OECD. (2006). The new rural paradigm: policies and governance. Paris: OECD. p. 106. Return to text

30 – OECD. (2006). The new rural paradigm: policies and governance. Paris: OECD. p. 106. Return to text

31 – Goldenberg, M. (2008). A review of rural and regional development policies and programs. Canadian Policy Research Networks Inc. Retrieved from http://www.cprn.org/documents/49496_EN.pdf, p. iii. Return to text

32 – Industry Canada. (2008). Mid-term evaluation of the Northern Ontario Development Program (NODP): Final Report. Provided by program May 18, 2010. Return to text

33 – AECOM Tecsult. (2010). Analysis of provincial, national and international regional development organizations and programs: Final Report: Canada Economic Development, p. 57. Provided by program November 26, 2010. Return to text

34 – Information on whether WD was meeting its funding application time commitment was not provided. Return to text

35 – AECOM Tecsult. (2010). Analysis of provincial, national and international regional development organizations and programs: Final Report: Canada Economic Development, p. 57. Provided by program November 26, 2010. Return to text

36 – IC/FedNor. (2007). Results-Based Management and Accountability Framework for the Northern Ontario Development Program. p. 6. Return to text

37 – AECOM Tecsult. (2010). Analysis of provincial, national and international regional development organizations and programs: Final Report: Canada Economic Development, p. 66. Provided by program November 26, 2010. Return to text

38 – Total cost per temporary and permanent job created and maintained = $8,560,499 (total NODP assistance) / 1,637 (number of temporary and permanent jobs created and maintained). Return to text

39 – OECD. (2006). The new rural paradigm: policies and governance. Paris: OECD, p. 56. Return to text

40 – Goldenberg, M. (2008). A review of rural and regional development policies and programs. Canadian Policy Research Networks Inc. Retrieved from http://www.cprn.org/documents/49496_EN.pdf, p. 26. Return to text

41 – Bradford, N. (2010). Regional economic development agencies in Canada: lessons for southern Ontario. Mowat Centre for Policy Innovation. Retrieved from http://www.mowatcentre.ca/pdfs/mowatResearch/15.pdf, p. iii. Return to text

42 – Markey, S., Halseth, G., & Manson, D. (2008). Challenging the inevitability of rural decline: Advancing the policy of place in northern British Columbia. Journal of Rural Studies, 24(4), 409–421. doi:10.1016/j.jrurstud.2008.03.012, p. 419. Return to text

43 – For a review of Canada's regional development programs, see Savoie, D.J. (1992). Regional economic development: Canada's search for solutions [2nd ed.]. Toronto: University of Toronto Press; Polèse, M. & Shearmur, R. (2006). Why some regions will decline: A Canadian case study with thoughts on local development strategies. Papers in Regional Science, 85(1), 24–46; and AECOM Tecsult. (2010). Analysis of provincial, national and international regional development organizations and programs: Final Report: Canada Economic Development. Provided by program November 26, 2010. Return to text

44 – Government of Canada. 2010. Policy Research Initiatives – Sustainable Places. 10(4). Retrieved from http://www.policyresearch.gc.ca/doclib/2010-0022-eng.pdf. Return to text

45 – The standard reference on competitive advantage is Porter, M. (1985). Competitive Advantage: Free Press. Return to text

46 – OECD. (2006). The new rural paradigm: policies and governance. Paris: OECD. Return to text

47 – See Section 3.1.1 and Annex J for additional explanation of these concepts. Return to text

Description of Figure 1: NODP Logic Model

Figure 1 presents the NODP logic model and demonstrates how the activities, outputs, and outcomes of the NODP align. The NODP has two key activities: to provide advice and funding support. The program has one output which is to provide contributions for projects. These contributions are focussed in six key program areas. They are as follows: community economic development; trade and tourism; innovation; information and communications technology; business financing support; and human capital. The immediate outcomes for NODP stem from the outputs and are specific to the six program areas listed above. They are as follows: community economic development leads to improved community capacity for socio-economic development; trade and tourism leads to enhancement of markets and investment attraction; innovation leads to enhanced innovation capacity and improved infrastructure; information and communications technologies leads to enhanced ICT applications and infrastructure; business financing support leads to improved access to capital and leveraged capital; and human capital leads to attraction, retention and development of human capital. Intermediate outcomes result from the culmination of immediate outcomes over time and can be considered as medium-term outcomes. These outcomes are not typically under the complete control of the program, but could be considered to be in the realm of influence, in terms of program effort. NODP has been designed to achieve the intermediate outcome of improved competitiveness and/or sustainability of communities and key sectors. Ultimate outcomes are longer-term and subject to influences beyond the program itself. They are considered to be at a more strategic level. The ultimate outcome of NODP is long-term, sustainable socio-economic development in Northern Ontario.

Return to figure 1

Description of figure 2

The scale is represented by a long, horizontal rectangle. The long horizontal rectangle starts as transparent on the left edge and progressively becomes darker and darker until it is solid black at the end on the right edge. There are five response categories and the five categories are equally spread out across the long, horizontal rectangle. On the left, the rectangle is totally clear for the "Few" (about 20% or less) category. It is mainly clear for the "Some" (about 20% to 40%) category. It has become light grey for the "About half" (about 40% to 60%) category. It is dark grey when it reaches the "Most" (about 60% to 80%) category. By the end, it is solid black for the "Many/all" (about 80%) category.

Return to figure 2

Date modified: