FirstPreviousNextLast

View all pages

Final Evaluation for the Northern Ontario Development Program

3.0 Findings

This section provides the evaluation findings by issue, using information generated by the methods described in Section 2.0.

3.1 Relevance

Issue 1: Continued need for the program
To what extent does the program address a demonstrable need and respond to the needs of Canadians?

Interviewees agreed that northern Ontario's vulnerability to price fluctuations and variations in exports will continue to lead to out-migration and an overall shrinking economy. The isolation of certain communities, the distance between these communities, and their reliance on one industry make them extremely vulnerable to these outside shocks. Interviewees noted that an effort to diversify the economy and create jobs and businesses is essential to building a sustainable economy in northern Ontario.

The NODP aims to foster economic growth and development and diversify the regional economy by supporting projects in six thematic areas. Table 5 shows the NODP support provided to projects by thematic area and provides examples of the types of projects funded under each area.

As the table demonstrates, the NODP supports several different areas and project types, such as investments in infrastructure, public services, ICT, Innovation, and the development of local amenities to support tourism and attract high-skill industries. The expectation is that such diverse support will "accelerate Northern Ontario's movement to a knowledge-based economy and build more globally competitive businesses"²² and mitigate the effects of the external variations in demand for their primary resources.

In addition, based on the output and outcome indicators in the database, it appears that many of the NODP projects within the thematic areas create and maintain immediate jobs, and about half lead to immediate business support.²³ All of the NODP immediate outcomes for each NODP thematic area appear in Annex H.

Table 6 shows that the NODP has invested the majority of its funding in non-profit organizations (57%) and municipalities (25%), whereas only 3% of the funding has gone to for-profit businesses. Some interviewees noted that the NODP could better address the needs of Canadians by investing more directly in for-profit businesses.

However, a closer examination of funding for non-profit organization projects shows that much of the spending classified as "non-profit" really comprises investments in infrastructure such as broadband, health care facilities, and other public capital (see Annex I for more detail). Much of these investments can arguably be linked to support for economic development. Increased broadband connectivity, for example, is often viewed as a critical infrastructure investment that supports the knowledge economy.

In addition, some interviewees suggested the program could better address diversification goals and regional economic performance by targeting investments in fewer, larger, and strategically positioned projects. Much of the recent academic literature aligns with this advice. Some key concepts in the literature included:

• Place-based policy – "Focus on policy issues as they play out in concrete geographic and community settings, as a means to address challenges and opportunities where the impacts are directly felt. In the context of regional economic development, it means drawing on the assets and opportunities of a 'place' to develop policies/strategies that support economic development."²⁴
• Comparative versus competitive advantage – Invest based on competitive advantage (e.g., firms and communities that will be successful in market exports outside the region) instead of comparative advantage (e.g., firms and communities that specialize in standard northern Ontario resources).
• Strategic investment – Focus investments on high potential areas and businesses that will stimulate economic development by building on local strengths and opportunities.²⁵

Many interviewees and case study participants noted that the various programs available in northern Ontario for regional economic development complement each other and that collaboration is essential for any project to move forward. Project data show that the provincial government is often a joint funder on NODP-supported projects. Such joint federal-provincial funding also mitigates the risk to any single funder.

Of the 40 projects included in the file review, the most common funding partner is the Province of Ontario, especially the Northern Ontario Heritage Fund Corporation (NOHFC).²⁶ Just under half (18 projects) were jointly funded by the province, including about one-third (12 projects) through the NOHFC.

Recent consultations with NOHFC management reached the same conclusion, that there is no overlap and duplication between the economic development programs available in northern Ontario. They noted that stacking funds for projects was permissible under the NOHFC, and that complementary programs were necessary to move projects forward in northern Ontario.

Annex K lists some examples of available programs to support economic development projects in northern Ontario.

Issue 2: Alignment with government priorities
To what extent are NODP objectives linked to (i) federal government priorities and (ii) departmental strategic outcomes?

Recent Speeches from the Throne and federal Budgets provide evidence that the NODP has supported and continues to support federal government priorities. From 2006 to 2010, Speeches from the Throne have discussed the importance of maintaining and growing Canada's economy. The 2010 Speech identified the importance of creating new jobs and supporting rural industries.²⁷

Federal Budgets support the Speeches from the Throne by earmarking funding for economic development and job creation. The 2010 Budget Address included a focus on regional economic development titled Leading the Way on Jobs and Growth. It also highlighted the important role that Regional Development Agencies (RDAs) play "by promoting innovation and the commercialization of research in communities throughout Canada." ²⁸

Interviewees concurred that NODP objectives link to and support federal government priorities. Many noted that it is now more the case than ever because of the economic downturn and the importance of economic development in Canada.

Within the IC Program Activity Architecture, the NODP contributes directly to the strategic outcome: Canadian businesses and communities are competitive. Table 7 shows how the NODP advances IC's strategic outcomes.

Issue 3: Alignment with federal roles and responsibilities
To what extent are the roles and responsibilities of the federal government being met in delivering the program?

The Government of Canada is committed to promoting equal opportunities for the well-being of Canadians and furthering economic development to reduce disparity, as set out in the Constitution Act, 1982. The Department of Industry Act, particularly section 4(2), sets out how the Minister supports the Government of Canada in its commitment to equal opportunities and reducing regional disparities. Sections 8 and 9 of the Act set out the Minister's responsibility for regional economic development in Ontario. FedNor is the sole federal presence in northern Ontario supporting the Minister in fulfilling these responsibilities.

The NODP promotes economic growth, diversification, job creation, and sustainable communities in northern Ontario, and is the government's response to regional development in that area. The NODP complements the Government of Ontario's initiative known as the NOHFC, which works with northern entrepreneurs, communities, and business organizations to support vital infrastructure and economic development projects to create jobs and stimulate the economy.

There is a general understanding by both levels of government regarding the need to harmonize federal and provincial approaches to northern Ontario economic development to allow for coordination and potential collaboration in areas of mutual interest and where circumstances permit. Ongoing communication, particularly among program officers, ensures duplication does not occur and that initiatives from both levels of government build on each other's successes.

Interviewees noted that the federal government has a clear role and responsibility to deliver a program such as the NODP in northern Ontario. Although the respondents agreed there is a role for all levels of government in this area, they view the federal government as having a responsibility to lead the way for the province, municipalities, non-governmental organizations, and other institutions.

The recent academic literature confirms interviewees' views. The theme of governance figures prominently in the OECD's The New Rural Paradigm and the Canadian Policy Research Networks' (CPRN) review of rural and regional development policies and programs.

The OECD argues that The New Rural Paradigm "requires important changes in how policies are conceived and implemented to include a cross-cutting and multi-level governance approach," adding that "traditional administrative hierarchical structures" are unlikely to suffice for this purpose.²⁹ Developing a governance structure supportive of the new approach to rural policy will require "adjustments … along three key governance dimensions: horizontally at both the central and the local levels and vertically across levels of governance."³⁰

Similarly, the CPRN review finds that novel approaches to policy in the area of rural and regional development incorporate new governance models characterized by:

• "partnerships among governments, the private sector, communities, voluntary organizations and others;
• greater devolution to the regional and local level; new roles for the state as an 'enabler' and 'convenor'; and
• the sharing of power and authority with the community itself."³¹

3.2 Performance

Issue 4: Achievement of expected outcomes

Issue 4A: To what extent does program management regularly identify and act on required improvements in delivering the program?

Previous evaluations have identified the need to improve certain aspects of performance measurement. The most recent of these evaluations—the Mid-term Evaluation of the NODP (2008)—identified ways to strengthen the program's Performance Measurement System and data quality:

1. FedNor should determine the benefit and cost of collecting performance measurement information on project outcomes beyond the end date of projects. If appropriate, FedNor should extend its Performance Measurement System to capture these longer-term results.
2. FedNor should strengthen mechanisms and formalize ongoing edit checks to ensure data quality, including:
   1. consistent comparison of intended and actual outcome data for each project
   2. both program officers and proponents report on final results independent of intended outcomes captured at the project launch stage
   3. analyze minimum, maximum, and most frequent values for indicators
   4. create separate sub-indicators based on project grouping with unique outcome characteristics
   5. address concerns and anomalies with data and indicators as required.³²

In a follow-up report (January 2010) to the Mid-Term Evaluation of the NODP, FedNor developed several actions to address these recommendations. Table 8 sets out these actions, as well as an assessment of progress.

As is shown in the table, progress is being made on most fronts. For example, a separate form for intended and final outcomes is being used for projects that started later in the file review.

However, limited progress has been made on the collection of performance measurement information on project outcomes beyond the project end date. Changes to address the longer-term measurement of project outcomes are being planned and integrated into the revised NODP Performance Measurement System.

Issue 4B: What factors have facilitated or impeded the delivery of the program?

The main factor that supports effective delivery of the NODP is the involvement and experience of FedNor regional staff. Regional FedNor offices in northern Ontario enable direct contact with the proponents and FedNor staff are knowledgeable of the communities' needs. Interviewees noted that the level of FedNor support and staff's knowledge of northern Ontario facilitates the delivery of the program. The direct involvement of FedNor staff in project design and planning guides investment decisions, mitigates risks associated with projects, and enhances the likelihood of realizing NODP's outcomes.

FedNor's continued presence and experience in northern Ontario appears also to have resulted in linkages with other federal departments, provincial and local governments, and non-governmental organizations. Building such relationships can be important in supporting the delivery of the program since FedNor officers can coordinate support with outside partners to move projects forward.

The main issue impeding program delivery is the time required for project approval, according to interviewees and case study respondents. They noted that project approval times for the NODP are longer compared to previous FedNor-administered programs.

An analysis of the 858 NODP projects approved between April 1, 2006 and January 31, 2010 showed that the average approval time was about 140 working/business days. About one-quarter of the projects were approved in three months or less, and about 60% were approved in six months or less. More than one-third took more than six months for approval and more than 10% took more than a year.

RDAs comparable to FedNor, such as Canada Economic Development-Quebec and Western Economic Diversification Canada (WD), have processing time commitments in working/business days of 25–46 days and 60 days (in 90% of cases), respectively.³³ Canada Economic Development-Quebec meets its committed approval time in 50% of its cases.³⁴ Although the Atlantic Canada Opportunities Agency (ACOA) does not have a predetermined commitment time, it has measured its approval times at between 88 days and 8 months, depending on the program.³⁵

Issue 4C: To what extent has the program achieved progress toward immediate outcomes, intermediate outcomes, and ultimate outcomes in terms of performance targets, program reach, and program design?

Immediate outcomes

The FPE indicator database captures several indicators under each NODP thematic area to measure the achievement of the immediate outcomes of the program. Indicators from each project are entered into the indicator database. In total, 90 indicators are captured across the 6 thematic areas. A full list of the indicators and the results expected and achieved by the projects can be found in Annex H.

In response to the NODP Mid-term Evaluation, FedNor collapsed certain indicators into 10 indicator groups, which are common across thematic areas. The most commonly reported indicators across thematic areas are businesses supported; jobs created and maintained; studies/plans developed; events held; and number of participants at events.

Table 9 presents the results of these indicator groupings; it includes data for 858 projects ending between April 1, 2006 and March 31, 2010. The table includes three results columns:

• Count – The number of projects that contributed to the indicator
• Sum – The total number from the projects that contributed to the indicator (e.g., one project may benefit a number of businesses and organizations)
• Mean – The average number per project that contributed to the indicator (Sum/Count = Mean).

The indicators provide direct evidence for the achievement of the NODP's immediate outcomes. For example, the number of events held and number of participants at events lend themselves directly to the enhancement of markets and investment attraction under NODP's Trade and Tourism thematic area. The number of studies and plans developed directly supports improved community capacity for socio-economic development, enhanced innovation capacity and improved infrastructure, and enhanced ICT applications and infrastructure.

The case studies provided corroborating evidence that the NODP is achieving its immediate outcomes. Table 10 lists the five case studies examined in this evaluation, the expected NODP outcome, and the quantitative measures that led to the achievement of that outcome.

Table 10 shows that there is an overall benefit to businesses, particularly the creation and maintenance of jobs. Other important indicators included the development of a community plan and increased revenue into a community.

In addition, Table 10 highlights the case studies' achievement of their intended outcomes. For example, the Revitalization of Main Docks in Little Current project was expected to enhance markets and investment attraction. According to the case study information, revitalization of the docks resulted in stores opening along the docks and in the community, which has led to additional employment. The Development of a Five-Year Strategic Plan led to the development of a community plan, which improved the region's opportunities for socio-economic development.

Interviewees supported the indicator and case study evidence that the NODP is achieving its immediate outcomes. However, it is difficult to determine whether the program has been as successful as expected, as targets were only set for those indicators reported on in the RPP.

Although proponents are required to identify indicators and estimate their expected contribution to those indicators at the start of projects, these goals only provide targets for the individual projects. With respect to overall targets, some are reported on through IC's RPP, but they are limited in scope.

In particular, the RPP provides annual targets for the number of businesses created, expanded and maintained, and dollars leveraged by NODP investments. However, targets for number of jobs created and maintained and several other outcomes were not developed at all.

FedNor has indicated that although it would be difficult to set targets for several of its measured outcomes, it is refining its performance reporting for 2011–12 and will report on jobs created and maintained in the RPP and DPR.

Intermediate and ultimate outcomes

Most interviewees and all case study proponents agreed that the NODP has achieved progress toward its intermediate and ultimate outcomes. Case study proponents provided direct examples of how their projects are achieving these outcomes. Interviewees mentioned the following indicators to show that the program is progressing toward these outcomes:

• The development and sustainment of jobs and businesses
  • development of long-term employment for youth interns
  • increase in specialized research positions
• Support provided to various industry initiatives
• Innovation and infrastructure work that has contributed to the health sector and the economy.

Intermediate and ultimate outcomes can be problematic to measure. Currently, little data on intermediate and ultimate outcomes are collected and there are few resources allocated to performance measurement. The indicator database provides several "indications" of immediate outcome achievement, but does not capture any quantitative measures in the longer term of jobs continuing to be maintained, increased sales, or other measures of business expansion.

In an effort to measure longer-term outcomes, the evaluation conducted three case studies of projects that had been studied by the preceding NODF. Table 11 provides the longer-term outcomes retrieved from the websites and those mentioned by proponents for the three "updated" case studies.

The NODF case studies conducted as part of this evaluation suggest that the NODP is achieving longer-term outcomes. Specifically, they provide examples of jobs, created during a project, that were maintained and of expected future employment that has been realized and could be measured by the program.

For example, the Extend Telemedicine Services to Remote First Nations project, implemented by KO, created seven part-time jobs during the project. As of 2005, those jobs became full-time, and individuals were still employed as of November 2010.

The Development of a Marketing Strategy to Attract Potential Business to Timmins Ontario project, implemented by the Timmins Economic Development Corporation, expected to create approximately 300 new jobs in the retail and service sector by the end of 2007. As of April 2010, a total of 281 new jobs had been created.

Table 11 provides mostly qualitative evidence of success of the proponents in their fields, but the lack of continued measurement of outcomes by the proponent and the program provides limited quantitative evidence of these successes.

Issue 4D: What are the linkages and contributions of outputs to outcomes?

The NODP's "most direct and measurable output is contributions for projects to proponents to support activities related to socio-economic development of Northern Ontario."³⁶

It is evident from the findings presented that the NODP-supported projects have led to several outcomes, including socio-economic development in northern Ontario. Many interviewees and case study proponents indicated that NODP-supported projects would not have proceeded or would have proceeded at a reduced scope if NODP funding had not been received.

Table 12 displays the outputs and achieved results for five case studies.

In each case, it is evident that outputs directly contributed to the achievement of results for the project, which contributed to the immediate outcomes of the NODP. For example, the Revitalization of Main Docks in Little Current project improved the community's dock, which led to the opening of businesses along the dock and in the community.

However, the major barrier to measuring outcomes is that follow-up beyond the end date of the projects is not systematically completed to assess whether the intermediate and longer-term outcomes have occurred. For example, the Assist with Launch and Production of the MultiStations OS2 Product Line project was expected to result in the creation of five additional permanent jobs following the completion of the project; however, aside from the data collected at the time of program evaluation, no follow-up was conducted to verify these jobs had been created.

Issue 5: Demonstration of efficiency and economy

Issue 5A: What has been the utilization of resources in relation to the production of outputs and progress toward expected outcomes?

Between April 1, 2005 and March 31, 2010, FedNor distributed $196.55 million in NODP Gs&Css at a total O&M cost of $54.22 million. Therefore, under the NODP, FedNor has an administrative cost of less than 22%.

The cost to administer a community-based Gs&Css program, such as the NODP, can vary significantly depending on the nature of the program. For this program, these costs appear to be reasonable to support FedNor's capacity to deliver its programming over a vast geographic area, with requisite support for effective communications and core services.

The program's operations include a total of 121 FTEs, with decentralized delivery in 7 offices in northern Ontario and travel to monitor projects, provide a high level of service to proponents, and maintain close contact with local communities. Furthermore, the actual administrative costs for the NODP are in line with program and funding approvals.

Interviewees noted that the experience and knowledge of FedNor staff in the regions and their high level of availability and support are instrumental to the development of projects and success of the program. FedNor staff are also involved in a full range of services beyond the administration of contributions that do not result in projects funded by the NODP (e.g., policy advice, coordination, facilitation, referrals). This suggests the administrative costs to deliver the NODP, as currently defined, are reasonable.

The administration costs of other regional development agencies were reviewed to help determine whether NODP administration costs were reasonable. According to a report produced by Canada Economic Development-Quebec (2010), overall administration costs range from about 16% for Canada Economic Development-Quebec to 26% for the ACOA. ³⁷ The report does not separate the budgets for individual programs delivered by the RDAs. However, this does show that NODP's administrative costs are within the range of other Canadian RDAs.

With respect to NODP costs per outcome, 23 of the 40 projects reviewed as part of the file review resulted in temporary and/or permanent jobs created and/or maintained. Those 23 projects created or maintained a total of 1,637 temporary and permanent jobs at a total NODP funding of $8,560,499, and a total project cost of $30,138,317. Therefore, based on the sample files, the cost to the NODP per temporary and permanent job created and maintained was $5,229.³⁸

It should be noted that creating and maintaining temporary and permanent jobs were not the only outcomes expected from these projects. Many of the projects also expected to achieve other outcomes, such as creating, maintaining, and benefiting businesses; distributing loans; and improving access to capital and leveraged capital. However, since these outcomes are not aligned with specific elements of the contribution, it is not possible to separate the outcomes individually.

Issue 5B: To what extent is the NODP cost-effective?

Between April 1, 2006 and January 31, 2010, the NODP approved Gs&Cs support of $148,000,097 for projects totalling $557,548,777. Therefore, for each dollar contributed to a project by the NODP, $2.77 was contributed to the project from other sources, such as the provincial government, municipalities, private sector investment, and proponents. Table 13 shows the overall leverage effect by thematic area.

This table shows that Innovation, CED, and Trade and Tourism are the most successful at attracting other funds whereas Business Financing Support and Human Capital are least successful.

Business Financing Support is the least successful at attracting funds because several of the projects funded under this thematic area are Community Futures Development Corporations (CFDC) capitalization projects, which are 100% funded by the NODP. Although the initial NODP contributions to the investment funds of CFDCs do not leverage funds, the NODP contribution enables CFDCs to make loans, which, in turn, leverage funding. This type of leverage is not identified at the project approval stage, (as is the case for other types of projects), and as a result is not shown in Table 13. However, the number of loans and the funding leveraged as a result of capitalization projects is tracked elsewhere, and reported as final project outcomes.

Similarly, the majority of projects funded under the Human Capital thematic area are Youth Internship Program projects, which are normally funded up to 90% by the NODP.

Interviewees and case study proponents indicated that the NODP is a catalyst in attracting outside sources of funding and that projects would not have gone ahead at all or with the same scope without the NODP funding; however, it is difficult to imagine this is the case for projects where NODP funding was minimal.

Although some interviewees noted that the NODP allocates its resources effectively, they also noted the program could be more cost-effective if it were refocused. Two suggestions were made: the program should allocate funds more strategically instead of scattering small amounts of funding across a broad range of projects, and funds should deal more directly with and support businesses. These suggestions are consistent with the literature.

The literature indicates that past approaches to rural policy focused on redistribution of economic activity. However, the OECD (2006) argues that increasing pressure on public expenditure, combined with the general failure of redistributive approaches to reduce inter-regional disparities, suggests new approaches to rural policy are required. Instead of subsidies, the OECD's The New Rural Paradigm focuses on identifying local economic opportunities and making strategic investments to improve regional competitiveness.³⁹

References to this approach are common in the literature. For example, Goldenberg (2008) argues that "regional and rural policy should be based on strategic investments—in people, in communities, in local resources and assets, and in technology—seeking to stimulate economic development by building on local strengths and opportunities."⁴⁰

Similarly, Bradford (2010) notes that "one of the key evolutions [in federal regional development activity in Canada] has been the progression away from redistributional spending to investments in knowledge and innovation."⁴¹ Markey et al. (2008) assert that "the ascendency of place …tells us that abandoning communities and regions to the vagaries of the market are short-sighted …strategic investments are necessary to establish a foundation upon which diversified economic and community development can flourish."⁴² The alternative approach of spreading support widely is to fund larger projects in communities that have the infrastructure to nurture these investments.

In terms of direct support to business, this approach carries the risk that the business directly supported through the projects may not succeed. One of the factors in the past shift in regional development programming away from direct investment was that business failures were common, which fuelled criticism of government mismanagement. Another impetus to the withdrawal of government from direct investment was the fear that such spending would displace or crowd out private sector lending. This latter concern is less germane in northern Ontario and in the current recession where a shortage of finance capital appears to exist.⁴³

As noted in Section 3.1.1, literature on rural economic development has evolved and increasingly aligns with the view to focus impact and spending on business. Potentially cost-effective regional development concepts mentioned in the literature include:

• Focus on policy issues as they play out in concrete geographic and community settings, as a means to address challenges and opportunities where the impacts are directly felt⁴⁴
• Investment in competitive advantage versus comparative advantage, which simply means identifying activities that are likely to grow through building market share⁴⁵
• Focus on strategic investments that link to other industries or create essential support services along the value chain.⁴⁶, ⁴⁷

FirstPreviousNextLast

View all pages