Evaluation of the FedNor Economic Development Initiative (EDI)

2.0 Methodology

This evaluation was managed by the Evaluation Directorate of Industry Canada. Econotech, an independent research company, was engaged to undertake some of the data collection upon which this report is based. This section sets out the evaluation scope and specific questions that were addressed, describes the data collection methods used to gather and analyze the information for this evaluation, and notes the limitations of the methodology.

2.1 Evaluation Scope

The scope of this evaluation relates solely to the EDI activities of Industry Canada, FedNor. In addition, all Regional Development Agencies will conduct individual evaluations of the EDI program in their respective region or assess results achieved. This will be submitted to Heritage Canada to support a horizontal evaluation of the Roadmap for Canada's Linguistic Duality. The horizontal evaluation will focus on higher level Roadmap outcomes depicted in the logic model.

2.2 Evaluation Issues

This evaluation covers the core issues outlined in the Treasury Board Directive on Evaluation. Specifically, the evaluation examined the following five issues under the themes of relevance and performance:

Relevance:

1. Does the initiative continue to meet a need?
2. Does the initiative meet government priorities?
3. Is the initiative consistent with the federal government's role and responsibilities?

Performance:

4. Have expected outcomes been achieved or are they likely to be achieved?
5. Has the initiative been implemented effectively and economically?

2.3 Evaluation Methodology

The methodology for this evaluation was calibrated to take into consideration the following factors:

• The program is materially small in terms of Industry Canada's expenditures (EDI accounts for less than 0.1% of IC's forecasted spending in 2011–2012.)
• The EDI component of the Roadmap is relatively small (it represents less than 0.5% of the total Roadmap budget) and accordingly is expected to have only a marginal contribution to higher level Roadmap outcomes.

Nevertheless, an evaluation was conducted for the FedNor component of EDI and the methodology employed multiple lines of evidence to cover all evaluation issues.

The methodology included a review of program documents and literature, an administrative data and file review, and key interviews.

2.3.1 Document and literature review

The document review included materials provided by the program, such as general policy and strategy documents that frame the Roadmap and EDI. Consultation records and summary documents relating to the early developmental stages of the project were used to provide additional context. Operational documentation, including guidelines, organizational charts, guides and manuals, and administrative procedures for the review, approval and monitoring of projects were consulted to deepen understanding of the program's functioning. In addition, Final Project Reports (16), completed by recipients, were reviewed as they were a key source of outcome information.

The review of literature focused on research that provided insight into the socio-economic situation of regional OLMCs in Northern Ontario and recent developments within the geographical area. Research reports, as well as conference and symposium reports, were reviewed in conjunction with pertinent Statistics Canada data and analyses. A full listing of all documents and literature consulted can be found in Annex 2 of this report.

2.3.2 Administrative data and file review

The administrative data review included data retrieved from the program's Grants and Contributions Reporting System (GCRS), provided in table format by the program. The file included the bulk of project-related (e.g. recipient names, general project descriptions and contribution amounts) for all 50 projects approved for funding as of September 2011. Additional data was provided by the program in a database.

2.3.3 Interviews

A total of 33 interviews were conducted with the following groups:

• FedNor managers and officers (13);
• Funding recipients (13); and,
• Regional economic development stakeholders (7) (mayors of Northern Ontario cities (3), consultants dealing with economic development of OLMCs (2), Statistics Canada (1), and Réseau de développement économique et d'employabilité (1)).

Interviews were conducted by phone or in-person, depending on the geographical location of the person. Those located in the Sudbury-North Bay area and those located in Ottawa were interviewed mostly in person; those located in other areas were interviewed by phone.

A complete listing of interviewees is available in Annex 3 and interview guides can be found in Annex 4.

2.4 Methodology limitations

In assessing actual outcomes, the evaluators relied largely on the 16 Final Reports that were completed by project recipients. While self-reporting generally results in some level of bias, FedNor staff report that they were reviewed by Project Officers responsible for the projects and by Payment and Monitoring Officers to ensure that they were accurate portrayals of project outcomes. While the evaluators were unable to verify this validity, this limitation was judged to be acceptable in relation to the level of calibration set out for this evaluation.

A further limitation is that the report relies on 2006 Census data for demographic analysis. While the most recent Census was conducted in 2011, major releases of data are only scheduled to commence in February 2012. While the situation in Northern Ontario may have changed somewhat from 2006, at the time of the evaluation, there was no alternate comparable source of data.