Archived — Summative Evaluation of the Northern Ontario Economic Development Fund (NODF)

Archived Information

Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Executive Summary

Introduction

The Federal Economic Development Initiative in Northern Ontario (FedNor) was established in 1987 to promote business development and economic diversification in Northern Ontario. Currently, FedNor is responsible for the delivery of the Northern Ontario Development Fund (NODF) or the FedNor Program which promotes economic growth, diversification, job creation and sustainable self-reliant communities in Northern Ontario through a range of initiatives aimed at improving small business access to capital, information and markets; the Community Futures (CF) Program which supports the development and growth of rural Ontario communities; the Eastern Ontario Development Fund which supports economic renewal in rural Eastern Ontario; and, a new funding initiative to support the Social Economy throughout Ontario.

Over the past several years the NODF resources have varied from year to year. In 2002-03 fiscal year (FY), the grants and contribution and operating budgets totalled $60.4 million compared to $53.4 million in 2003–04, $49.0 million in 2004–05 and $57.9 million in 2005-06.

The NODF was subject to a formative evaluation in 2002. A commitment was made to complete a summative evaluation of the NODF in accordance with FedNor's Results-based Management and Accountability Framework (RMAF).

This summative evaluation of the NODF covered three fiscal years (2002-03 to 2004-05) based on data collected as at the end of August 2005. The evaluation involved a series of research questions related to the broader issues of relevance, staff support, success, monitoring and accountability, as well as alternatives, cost-effectiveness and lessons learned.

For more details, please refer to Section 1.0 of this report.

Methodology

The evaluation relied on a range of approaches to ensure that each issue was addressed using multiple lines of evidence. The study methodology included:

  • a document review;
  • review / analysis of a wide range of existing NODF data;
  • analysis of the results of the Service Improvement Initiative surveys, conducted in 2002 (with 100 clients) and again in 2005 (with 151 clients) using survey questions from the Common Measurements Tool (CMT);
  • a total of 71 in-depth telephone or in-person interviews with Industry Canada (IC) / FedNor management (5), staff (20), Community Futures Development Corporations (CFDCs) representatives (13) and other stakeholders (33);
  • a telephone survey with 200 project funding recipients; and,
  • nine project-related case studies involving a review of case-related documents and interviews with various individuals involved in the case.

Overall the approaches and sample sizes used for this evaluation resulted in a strong and reliable summative evaluation, which provided evidence to conclude on all issues. However, it should be noted that no interviews were completed with unsuccessful applicants because all had either been approved at some time by FedNor or could not be located.

For more details, please refer to Section 2.0 of this report.

Findings – Relevance

Overall, the evaluation findings demonstrate that there is an ongoing need for NODF programming and that the projects are relevant. There is a wealth of documentary evidence related to the ongoing economic development needs of Northern Ontario and the challenges faced by Northern communities. The NODF project data illustrates that the program is relevant to a wide range of needs (as demonstrated by the wide range of types of projects funded). Additionally, the profile of the funded organizations demonstrates that the NODF is reaching a diverse group of organizations. The program's relevance was also confirmed through the interviews, where FedNor management, staff and stakeholders all agreed that the projects were relevant in particular because of FedNor's presence in the community and its understanding of local needs. Finally, the recipient survey and case studies provided additional evidence of the program's relevance. The survey results showed that NODF's strategic priorities (telecommunications and ICT, innovation and technology, trade and tourism, human capital, business financing support and community economic development) were extremely important and that FedNor was able to address needs related to those priorities through the NODF. The case studies also provided specific examples of how NODF projects are linked to local economic development needs.

For more details, please refer to Section 3.0 of this report.

Findings – Staff Support

The evidence collected during this evaluation shows that clients are satisfied with the counselling, advice, training and workshops provided by FedNor staff. The great majority (86%) of clients have received counselling or advice and report that they have or will use this advice (97% of those who received counselling / advice). Additionally, while a small proportion (21%) of clients have received training or attended a workshop, the great majority of those clients have or will use what they learned (91%).

In general, the evidence from the interviews, survey and case studies shows that staff provide a lot of support at the project development phase. Staff help organizations refine their projects to better meet NODF requirements; they also help these organizations identify other sources of funding. This support is much appreciated, as evidenced by the high levels of client satisfaction with this (average of 8.7 out of 10). Additionally, clients provided direct evidence of having benefited from this support. However, FedNor management and staff interviewees indicated that monitoring and follow-up was fairly limited. Nevertheless, the services provided by Payments and Monitoring were deemed important and useful.

The evidence gathered shows that FedNor has been successful in making significant progress in four of the five priorities outlined in its Service Improvement Initiative plan.Footnote 1 That is, clients are satisfied with the overall application or proposal assessment process, the payment and monitoring process and the FedNor communications with clients. Additionally, the turnaround time on proposals has been reduced by 29% (from 16.1 weeks in 2001/02 to 11.4 weeks in 2004/05) resulting in a 10% increase in client satisfaction (from an average of 3.4 out of 5 in 2002 to 3.8 in 2005). However, FedNor has been less successful in reducing the turnaround time on payments (increase from 45.42 days in 2001/02 to 57.13 days in 2004/05). Nevertheless, client satisfaction with the payment process has increased slightly albeit not statistically significantly (mean of 3.9 out of 5 in 2005 up from 3.6 in 2002).Footnote 2

For more details, please refer to Section 4.0 of this report.

Findings – Success

The evaluation results show that FedNor has achieved all of the NODF intended outcomes. In some cases, FedNor has been extremely successful, in other cases it has achieved success indirectly, while in other cases it has been only slightly successful, mostly due to the fact that it has funded few projects in those areas. The following highlights some of the key NODF results:

  • FedNor has contributed to the development of new businesses and the retention of existing businesses directly through projects and indirectly through the CFDCs capitalization fund, the Credit Union Loan Loss Reserve and the BDC Loan Loss Reserve. Based on CFDCs reports, 1,293 businesses have been assisted (488 new ones) resulting in 6,401 jobs created or maintained. Based on the survey results, at least 124 new businesses were created through direct NODF project assistance resulting in at least 470 jobs created.
  • FedNor has also contributed indirectly to the improved competitiveness of Northern Ontario firms by helping them grow (increased revenues of close to $200 thousand for the private sector organizations receiving NODF funding; 55% indicating some other type of growth, such as employment, new services and new capabilities) or develop innovative products, processes or services (71% of private sector firms surveyed).
  • FedNor has helped attract, retain and develop human capital directly through 347 human capital projects which target youth (98%), Aboriginal people (12%), Francophones (6%), and / or women (1%). While interviewees generally believe that FedNor's contribution is mostly through the Youth Internship Program, the survey of recipients shows that FedNor's contribution is broader than this. That is, 49% of surveyed recipients indicated that their project had resulted in the attraction of new staff, 39% in the retention of existing staff and / or 53% in the development of existing staff. Only 23% did not involve any human capital impacts.
  • Community economic development (CED) is seen to be a key area of NODF success. The data shows that 80% of projects involved community capacity outcomes (close to $100 million in authorized assistance). Interviewees saw FedNor's ability to encourage and build partnerships with a variety of players as a major advantage in strengthening community economic development. The case studies showed direct evidence of CED projects contributing to increasing economic activity in communities. Finally, the survey provided direct evidence of the attraction of more than $1 billion in investments, more than 100 new businesses and dozens of new institutions.Footnote 3
  • FedNor has contributed to the development of business and trade skills as evidenced by 43% of surveyed recipients indicating that their projects has resulted in this. However, interviewees had a difficult time describing FedNor's contribution to this and none of the case studies were directly related to this.
  • FedNor has contributed less extensively to the development of external markets through direct project funding as well as through the trade missions and other work in this area.
  • FedNor has also contributed significantly to increased use of technology and to the development of innovation, as demonstrated through 62 innovation and technology projects (representing close to $25 million in authorized assistance) and other projects resulting in FedNor increasing the number of rural and remote communities with access to high speed Internet and cellular telephone service. Based on the interview, survey and case study evidence, these have resulted in significant socio-economic benefits to the communities involved.

These results are highly attributable to NODF funding. A large proportion of projects would not have gone ahead without FedNor assistance whereas most other projects would have been affected in quality, timing or scope. Additionally, while FedNor cannot necessarily take the full credit for leveraged funds, a contribution of $122 million over the three year period covered by this evaluation has resulted in total project investments of close to $400 million.

For more details, please refer to Section 5.0 of this report.

Findings – Monitoring and Accountability

FedNor has made significant progress in terms of monitoring and accountability. The evidence gathered during this evaluation highlights progress in planning for results and in capturing more complete performance information. However, the interviewees identified some challenges related to monitoring and accountability; most of these challenges were confirmed through the data analysis activities. The key problem lies in the project outcomes and evaluation scoresheets which are completed by the project officers at the end of each project. Project officers are meeting the requirements of completing the forms. However, these forms have not been updated to reflect program changes. Additionally, the forms are completed to varying degrees of accuracy. There are also issues with linking this and other monitoring and accountability tools to the requirements outlined in the program's recent RMAF.

For more details, please refer to Section 6.0 of this report.

Findings – Alternatives, Cost-Effectiveness and Lessons Learned

This evaluation confirmed the findings of the 2002 formative evaluation study of FedNor. Evidence of cost-effectiveness stems from all sources. The data shows that FedNor contributes as little as 1% of the total costs of projects and as much as 100% of the project costs. For every dollar invested by FedNor, another $2.28 are invested by others. There was general consensus among interviewees that there was no other service delivery approach that could work better than the NODF approach. Nevertheless, a range of suggestions were made for increasing cost-effectiveness. The survey results showed that:

  • There are other comparable sources of funding available in Northern Ontario (for example, Northern Ontario Heritage Fund) – 47% of surveyed recipients were aware of comparable programs.
  • The NODF is effective – as reported under success, it has been effective in achieving its intended outcomes and is incremental to the ability of organizations to undertake the projects.
  • The NODF is cost-effective – in its ability to leverage funds as well as in the fact that many (40%) recipients would not have been able to undertake the project if they had received less NODF funding, and those who could (37%) indicated that the lower funding would have affected the effectiveness of their projects in that the timing, quality or scope of the projects would have been affected.

A wide range of lessons learned were provided by the interviewees, survey recipients and case studies. Most of these were suggestions for improvement and are incorporated into the conclusions and recommendations in the next section of this executive summary.

FedNor has been diligent in providing services in the official language of choice. However, it has made less progress in integrating the needs of official language minority communities into its programming. Nevertheless, there is evidence that there has been specific activity in this regard and that progress with respect to this will be evident in the near term.

For more details, please refer to Section 7.0 of this report.

top of page

Conclusions and Recommendations

Conclusions and Recommendations
Conclusions Recommendations
Relevance

Conclusion 1

The economy of Northern Ontario continues to struggle with economic development issues. NODF strategic priorities are relevant and adapted to these economic development issues. Additionally, the projects are relevant in their objectives and meet the criteria established by FedNor.

The key aspects of NODF which make the program particularly relevant are:

  • its flexibility;
  • its broad criteria and broad range of priorities;
  • the range of project size it funds;
  • its broad reach;
  • the fact that it is customized to needs;
  • its partnership approach;
  • its geographic delivery; and,
  • its ever evolving strategic outlook.

Recommendation 1

FedNor should continue to adapt the NODF to the changing economic development needs of Northern Ontario. In particular, it should continue to develop strategic approaches that address pan-Northern and / or sectoral needs.

Staff Support

Conclusion 2

The role of FedNor staff, in particular of program officers, is critical to the successful planning, development and implementation of projects. There are several key aspects with respect to staff support that contribute to this success:

  •  the extensive advice and counselling provided by the officers in the development of proposals;
  • the due diligence exercised by program officers in ensuring, not only that NODF funding does not duplicate or overlap other sources of funding, but also that the client is aware and has access to all other potential sources of funding;
  • the local presence of the program officers;
  • the on-going relationship of staff with some clients; and,
  • the ability to provide services in the official language of choice.

However, there are concerns with some aspects of the support provided in terms of project implementation, monitoring and follow-up. These concerns include:

  • some inconsistency among program officers in applying the criteria as well as in the level and type of support provided during the project; and,
  • workload demands that do not provide program officers with the time to provide an adequate level of monitoring and follow-up that they (and others) believe is required.

Recommendation 2

FedNor should ensure that program officers are aware of the extent of their roles and responsibilities, and that they recognize the areas where flexibility is required versus the areas where consistency is needed. This could be done through more frequent communications via various policy bulletins, training, meetings and other forms of communications amongst program officers as well as between officers and management.

Recommendation 3

FedNor should assess the workload distribution of program officers in terms of the time they spend on NODF project planning activities, project implementation and monitoring activities, and project follow-up activities. This assessment should also examine the time spent on other non-NODF activities and tasks. Based on the result of this assessment, FedNor should make the appropriate adjustments, as required, to ensure an appropriate balance in the time spent on these tasks.

Conclusion 3

All priorities outlined in the Service Improvement Initiative plan have been successfully addressed except with respect to the turnaround time on payments. However, the turnaround time on payments is no longer assessed as being a service improvement priority, from the clients' perspective. This is due to the fact that clients are slightly more satisfied with FedNor's performance on this and they do not believe this to be as important a service feature as it was in earlier years.

Recommendation 4

FedNor should continue to monitor various aspects of its services and to adapt its service improvement priorities as required. The recommendations included in the 2004/05 Service Improvement Initiative report are still relevant in that, not much improvement can be achieved in terms of client satisfaction. Nevertheless, client expectations could be better managed through the establishment of published service standards.

Success

Conclusion 4

A number of NODF projects have contributed to the development and retention of new and existing businesses. Evidence suggests that NODF projects have resulted in the development of new businesses, jobs and growth in existing businesses through retained and new employees as well as increased revenues.  In addition, Northern Ontario CFDCs have indirectly contributed to NODF success in this regard through the impacts of the NODF capitalization assistance. The reported impacts from this assistance include close to 500 new, more than 500 maintained and close to 350 expanded business. Northern CFDC support, resulting at least partially from NODF assistance, have also contributed to the creation of more than 2,100 job and the maintenance of almost 4,300 jobs. The program is therefore contributing to the development and retention of new and existing businesses, both directly and indirectly.

Recommendation 5

FedNor should continue to address the economic development needs of Northern Ontario through its broad range of strategic priorities as these are contributing to the wide ranging impacts of the NODF.

Conclusion 5

The NODF's contribution to the competitiveness of Northern Ontario firms is, in most cases, indirect through the Community Futures top-up and the BDC investment fund. Nevertheless, the NODF also contributes directly to this objective through its trade networks and missions as well as through projects involving the development of new or improved products, services or technologies.

Conclusion 6

The NODF has been successful in attracting, retaining and developing human capital in Northern Ontario. With respect to special target groups, the program has made a significant impact on youth, women, Aboriginal people and Francophones even though the projects did not necessarily target these groups.

Conclusion 7

The NODF has made a significant contribution to economic development in Northern Ontario through community level initiatives as well as, to some extent, sectoral initiatives. FedNor's ability to encourage and build partnerships is a major contributor to the program's success in this area.

Recommendation 6

FedNor should keep the requirement for high involvement of other partners and communities in the NODF funded projects as these are also important to the success of the projects and program.

Recommendation 7

FedNor should also enhance its support of sectoral initiatives as these are important to the economic development of Northern Ontario communities.

Conclusion 8

Based on the evidence available for this evaluation, it is difficult to conclude on the extent to which the NODF has helped develop business and trade skills. However, the survey results and case studies indicate that the NODF has made a contribution in this area.

Conclusion 9

Based on the activities in support of developing external markets, the NODF has made progress in this regard. Progress has been made as a result of project-specific activities such as trade missions. Progress has also been made as a result of support provided by the trade advisors as well as through other non project-specific activities and support.

Conclusion 10

As a result of the projects funded under the Telecommunications and ICT as well as Innovation and Technology strategic priorities, the NODF has contributed to significant investments in this area. These investments have reached a large proportion of communities and businesses in Northern Ontario resulting in increased use of technology and the development of innovations. This has, in turn, contributed to socio-economic benefits in recipient organizations and communities.

Conclusion 11

The NODF has been essential in enabling a large number of organizations to undertake the funded projects. As such, the project impacts are highly attributable to the program. In addition, the design features have contributed to high leveraging of funds from other sources. Over the period of the evaluation, $122 million of NODF funding has resulted in a total of $400 million in investments in the projects. FedNor is therefore recognized as a catalyst for economic development.

Recommendation 8

FedNor should continue its practice of appropriately balancing its requirements that NODF funding be incremental to the recipient organization's ability to undertake the projects as well as requiring an appropriate level of leveraging on projects.

Monitoring and Accountability

Conclusion 12

FedNor has improved its capacity in terms of planning for results. For example, the recent Results-based Management and Accountability Framework (RMAF) is providing management with a better tool for measuring performance. The planning process is more closely linked to the NODF's performance management requirements. FedNor has also invested more resources into this, as evidenced by staff which are dedicated to performance monitoring. Additionally, FedNor has undertaken special studies to review specific aspects of the program (for example, the Youth Internship Program and post evaluations of trade missions). With respect to projects, officers are meeting their requirements in terms of assessing project risk and completing the project outcomes scoresheets. Additionally, with respect to the program as a whole, there is evidence of a lot of analysis and reporting of the data that is currently available.

However, the RMAF does not fully reflect the present design and delivery of the NODF. The existing tools and systems have not been updated to reflect the performance measurement requirements outlined in the recent RMAF. Additionally, it is unclear if the information collected from the project outcomes scoresheets is valid and reliable. It is also unclear if program officers clearly understand how to complete the scoresheets. The departmental systems for capturing and analyzing project information remains poorly aligned with NODF requirements. Finally, the departmental system is not aligned with the project outcomes scoresheets. Therefore, it is difficult to clearly determine the extent to which projects are achieving their intended outcomes.

Recommendation 9

FedNor should further refine the NODF RMAF to more accurately reflect the program priorities. The performance reporting systems and tools should also be refined to meet the requirements outlined in the RMAF. In brief, the NODF RMAF would benefit from the following enhancements:

 1.  A closer integration of the logic model with the expected results, which are priority driven.

 2.  An integrated performance measurement and evaluation strategy which:

  • is directly linked to the logic model;
  • provides assurance that every aspect of the logic model is measured at some point in time (not necessarily ongoing);
  • recognizes the variable level of risk associated with the projects and clients;
  • provides a more direct link between the evaluation success issues and the program's logic model; and,
  • identifies the tools that are in place or needed to meet the performance measurement and evaluation strategy.

The revised RMAF should feed directly into the revisions of existing tools and systems as well as the development of new tools and systems as required. For example, the outcomes captured on NODF projects in the departmental system should match the outcomes in the RMAF. This is not presently the case. For example, the database captures community capacity, connectedness, e-commerce, export, knowledge-based, tourism and trade outcomes. These are currently not directly linked with neither the program priorities nor the outcomes identified in the program logic model. Similarly, what is measured in the project outcomes scoresheet is not reflective of the logic model outcomes nor of the indicators in the performance measurement plan.

Specific instructions on how to specifically complete the project outcomes scoresheets as well as on what is required from the program officer to ensure that the ratings in the scoresheet are reliable must be developed and a means for ensuring data quality and integrity needs to be implemented.

Conclusion 13

The Industry Canada departmental systems (i.e. the Grants and Contribution Reporting System – GCRS which draws upon the information in the Contribution Management Information System – CMIS) to capture information on projects is inadequate to meet the monitoring and accountability requirements of FedNor with respect to the NODF. It currently does not provide the flexibility required to capture the intended and actual results of projects as outlined in the program's RMAF. Additionally, since RMAFs are intended to be “living” documents, program results can change over time, particularly for a program such as the NODF. The Industry Canada departmental system is not currently flexible enough to provide FedNor with the ability to adjust the fields in the database as the program evolves to better reflect the changing needs of Northern Ontario communities.

Recommendation 10

FedNor should consult with Audit and Evaluation Branch (AEB) and the Information Management Branch (IMB) which is responsible for GCRS and CMIS to ensure that the departmental systems provide FedNor management with the flexibility it requires to appropriately capture the performance information it needs to better meet its monitoring and accountability requirements. If the required flexibility cannot be incorporated into the existing / new system, then FedNor should discuss other options with AEB and IMB regarding linking the FedNor system to ensure that all monitoring and accountability information can be captured and linked without requiring duplication.

Alternatives, Cost-effectiveness and Lessons Learned

Conclusion 14

There are currently no alternative service delivery approaches for the NODF that would produce the same results at a lower cost. Currently, a significant proportion of the project costs are leveraged through other sources. FedNor collaborates extensively with other organizations to ensure that NODF funding complements rather than duplicates the contributions of these other sources of funding. The due diligence exercised by the program officers and program management to ensure that NODF cost-effective investments in Northern Ontario (i.e., the right level of funding and the right leveraging for strategically needed projects) is very comprehensive and effective. Additionally, a large proportion of clients would not proceed with their projects if lower levels of funding were provided by FedNor. It is therefore evident that FedNor is managing NODF costs appropriately. The projects are contributing to the achievement of their intended outcomes. It is therefore evident that the NODF is effective in achieving success. Overall, the program is therefore managed in a cost-effective fashion.

See recommendations 5 to 8 under Success.

Conclusion 15

The critical success factors of the NODF are its flexibility, its staff, the partnerships it helps develop and its local presence.

See recommendations 1 to 8 under Relevance, Staff Support and Success.

Conclusion 16

FedNor's strategic focus is key to its continued relevance. This includes the need for specific strategies to address sectoral needs (including natural resources). Additionally, a means of ensuring that clients are aware of the NODF's strategies would also be beneficial.

See recommendations 1 and 7 under Relevance and Success.

Recommendation 11

The new group responsible for gathering market intelligence for FedNor and helping to develop strategies in light of this market intelligence should take due consideration of this lesson learned.

Conclusion 17

Ongoing staff development is critical to the NODF's continued success. This includes training as required, means of ensuring that best practices (and lessons learned) are shared, clear roles and responsibilities. This will lead to more consistency in dealing with clients.

See recommendation 2 under Staff Support.

Recommendation 12

FedNor should continue to have regular organization-wide or strategically driven events aimed at sharing best practices and lessons learned.

Conclusion 18

The Northern Ontario Community Futures Development Corporations believe they should be more involved in the delivery of some of the NODF programming, in particular, in the trade missions that affect their clients and, possibly in other aspects of projects.

Recommendation 13

While there are currently no alternative service delivery approaches for the NODF, FedNor should further explore the possibilities of involving 3rd parties more extensively in the delivery of the NODF to ensure that it continues to maximize its cost-efficiency.

Recommendation 14

FedNor should ensure that, to the extent feasible, all Northern Ontario CFDCs are invited to participate in its NODF trade missions, particularly if clients in their delivery region are participating.

Conclusion 19

The Youth Internship Program may benefit from increased flexibility, such as: permitting the overlap of interns where appropriate; longer term internships; as well as, the possibility of just an internship program rather than a youth internship program.

Recommendation 15

Recognizing that the Youth Internship Program has evolved extensively over time, as a result of ongoing monitoring of the effectiveness of this program, FedNor should further explore the implications of expanding the NODF Youth Internship Program.

Conclusion 20

FedNor has been committed to providing services in both official languages. There is also evidence of significant activity by FedNor in consulting with official language minority communities (OLMC) to ensure that their needs are integrated into NODF programming. However, based on stakeholder perceptions as well as some of the program data, FedNor needs to continue to evolve to ensure that it meets the unique needs of OLMCs.

Recommendation 16

FedNor should continue to work towards integrating the needs of the OLMCs into its NODF programming.


Footnotes

  1. 1 It is important to note that, while the Service Improvement Initiative was an issue in this evaluation, this initiative is not limited to the NODF but rather to all FedNor programming. As such, it involves analysis of more than just NODF data. (back to footnote reference 1)
  2. 2 For further information please see:
    dnor.gc.ca/eic/site/fednor-fednor.nsf/eng/fn02286.html (back to footnote reference 2)
  3. 3 These numbers should be interpreted with care since they are based on 1) a very small number of respondents (n=14) and 2) a wide range of responses (from as little as $1,000 to as high as $1 billion). (back to footnote reference 3)
  • Email
Help us improve
Back to "Help us improve" section.
  
Back to "What's the problem?" section.
Got it, thanks!
Um, you didn't enter anything.
Date modified: