Final Evaluation for the Northern Ontario Development Program

2.0 Methodology/Approach

IC engaged Prairie Research Associates (PRA), an independent research company, to conduct a Final Evaluation for the NODP. The purpose of the evaluation was to assess the relevance and performance of the program as required under the Treasury Board Policy on Evaluation.

An Evaluation Advisory Committee (EAC) was established to provide advice and guidance during the evaluation. The committee reviewed and provided input on the methodology, as well as on the findings, conclusions, and recommendations presented in the evaluation report. A list of NODP EAC members is provided in Annex A.

This section lists the issues addressed by the report and describes the methodologies used to gather and analyze the information for this evaluation.

2.1 Evaluation issues

This evaluation meets the requirements of the Treasury Board Policy on Evaluation and addresses the themes of relevance and performance. The five core issues of the Treasury Board Policy on Evaluation and the specific questions for the evaluation included:

Relevance

• Continued need for the program
  To what extent does the program address a demonstrable need and respond to the needs of Canadians?
• Alignment with government priorities
  To what extent are NODP objectives linked to (i) federal government priorities and (ii) departmental strategic outcomes?
• Alignment with federal roles and responsibilities
  To what extent are the roles and responsibilities of the federal government being met in delivering the program?

Performance

• Achievement of expected outcomes

  To what extent does program management regularly identify and act on required improvements in delivering the program?

  What factors have facilitated or impeded the delivery of the program?

  To what extent has the program achieved progress toward immediate outcomes, intermediate outcomes, and ultimate outcomes in terms of performance targets, program reach, and program design?

  What are the linkages and contributions of outputs to outcomes?

• Demonstration of efficiency and economy

  What has been the utilization of resources in relation to the production of outputs and progress toward expected outcomes?

  To what extent is the NODP cost-effective?

The complete evaluation matrix appears in Annex B.

2.2 Evaluation methodology

The study methodology specifically ensured that multiple lines of evidence were available for all issues and questions. The methodology included a review of documents and literature, an administrative data and file review, interviews, and case studies. In addition, rather than duplicating the efforts of previous studies, the study team used pre-existing research such as case studies and literature reviews, where feasible, to minimize the burden on individuals consulted.

The evaluation findings and conclusions are based on the analysis and triangulation of multiple lines of evidence (see also Annex B, evaluation matrix). This subsection describes these various methods.

2.2.1 Document and literature review

The document review included materials provided by the program, including past evaluations to minimize duplication of work. The literature review incorporated and extended work undertaken as part of the Northern Ontario Development Fund (NODF)²¹ summative evaluation. This review focused on rural economic development and included recent studies conducted by the Organization for Economic Co-operation and Development (OECD).

It is important to note the unique situation in northern Ontario and to use the literature as indicative of current trends in regional/community economic development rather than as a policy template.

2.2.2 Administrative data and file review

The administrative data review included excerpts from the FPE/indicator database and the Grants and Contributions Reporting System (GCRS) provided electronically by the program.

The file review included the selection of 40 projects to be analyzed in more detail using Access software. These projects comprised the 20 largest projects in terms of funding plus a random selection of 20 additional projects. Some of the 20 randomly selected projects were replaced with other projects also selected randomly to ensure representation of all program areas as well as representation of the two main regions of northern Ontario (northeast and northwest).

Annex C presents the list of projects included in the file review. See Annex D for a list of the fields that were captured in the Access database.

For each project selected, the program forwarded the application, contribution agreement, final report, and progress reports. These files contained all the information necessary to complete the fields in the Access database.

2.2.3 Interviews

A total of 42 interviews were completed with the following groups:

• External stakeholders (29)
• Program managers (6)
• Project officers (5)
• Academics (2).

Interviewees received an invitation by email from IC/FedNor in both official languages to inform them that PRA would contact them to participate in an interview. A copy of the letter in both official languages appears in Annex E.

Each interview group received a tailored interview guide (in both official languages) prior to the interview, to support preparation of responses. A pretest with two interviewees in each group was conducted to ensure clarity and comprehension. Only minor changes were necessary. Copies of the finalized interview guides in both official languages also appear in Annex E.

The interviews were conducted by phone, in the official language of the participant's choice. Interviewees received assurance of confidentiality and anonymity prior to the interview. With their permission, interviews were audio-recorded and, to ensure the accuracy of responses, interviewees received notes of their responses.

The findings of the interviews use the following scale:

It is important to note that, in analyzing the findings from the interviews, two situations can occur.

First, there may be agreement across all groups of interviewees, but a single respondent may offer a perspective, based on experience or a unique position, that materially clarifies an aspect of the program or context. A single interviewee may raise a unique point that helps clarify some other aspect of information.

Second, since not all groups of interviewees were asked to respond to each evaluation question, and since different groups may have diverging opinions, findings are linked to the interview group that provided the information. Where these situations arise, they are noted in the findings. Of course, a single view needs corroboration from other information to ensure its validity.

2.2.4 Case studies

The evaluation completed eight case studies. Three were selected from the prior summative evaluation of the NODF (2006) to support a longer follow-up and update. Five were selected (in consultation with FedNor) based on:

• Size of budget
• Strategic goal
• Region
• NODP supported area
• Special populations (e.g., Official Languages Minority Communities, Aboriginals)
• Type of recipient organization (e.g., SME, not-for-profit, municipal).

The projects listed in Table 4 served as the case studies for this evaluation.

Table 4: NODP case studies

Name	Proponent	NODP area	NODP funding	Total project cost
New projects
Revitalization of Main Docks in Little Current	Town of Northeastern Manitoulin and the Islands	Trade and Tourism	$500,000	$3,529,887
Assist with Launch and Production of MultiStations OS2 Product Line	Three H Furniture Systems Limited	Innovation	$108,000	$649,000
Extension of Water and Sewer for the North Dryden Commercial Business Park	City of Dryden	CED	$1,300,000	$3,885,000
Establishing a Northern Agriculture and Food Pavilion at the 2009 Royal Agricultural Winter Fair	NECO Community Futures Development Corporation	CED	$348,300	$348,300
Development of a Five-Year Strategic Plan	Association des francophones du Nord-Ouest de l'Ontario	CED	$34,578	$41,700
Updated/follow-up projects
Undertake Work on Ultrasensitive Diagnostics for Early Detection	Genesis Genomics Inc.	Applied Research and Development (formerly Pre-Commercial Fund)	$875,000	$1,718,290
Development of a Marketing Strategy and Related Materials to Attract Potential Businesses to Timmins	Timmins Economic Development Corp.	NODF – Non-capital	$105,000	$334,891
Extend Telemedicine Services to 19 Remote First Nations	Keewaytinook Okimakanak/Northern Chiefs Council	ICT Applications	$450,000	$450,000

Case study data comprised an interview with the project proponent and a review of the NODP project file. In some cases, the project proponents provided additional documentation and information not available in the FedNor files.

Project proponents received an invitation letter from IC/FedNor by email saying PRA wanted to interview them concerning their NODP-supported project. Copies of the letter and the interview guide in both official languages appear in Annex F.

Phone interviews with project proponents were in the official language of the proponent's choice. The information collected from the interviews was combined with the information in the project files to develop case studies of about five pages. Case study interviewees received a draft of the case study for their review to ensure accuracy. Annex G presents the finalized case studies.

2.3 Methodology limitations

The following are the three main limitations of this evaluation.

Outcomes are not independently verified – Information regarding immediate outcomes came from administrative and self-reported data. These data appear in an indicator database. Although the indicator database provides an opportunity to measure the success of the program at achieving its immediate outcomes, the data have not been independently verified and rest on the representations of the project proponent in the FPE. Project officers, who flag issues for clarification, review all these reports. FedNor has now started to conduct edit checks using an automated system to ensure data quality. Any issues identified during the checks are referred to an officer for clarification.

Measured outcomes are short-term – As per the NODP Results-Based Management and Accountability Framework, the program collects quantitative data on short-term outcomes at the end of every project and relies on qualitative data collected at the time of program evaluation to measure intermediate and long-term outcomes. The program does not followup on projects to collect quantitative information on the achievement of intermediate and ultimate outcomes.

Attribution of outcomes to NODP is difficult – The majority of projects supported by the NODP are jointly funded. This makes it difficult to attribute the success of outcomes achieved specifically to the NODP. That said, all evaluations of cost-shared programs would face this issue.