Summative Evaluation of the Northern Ontario Economic Development Fund (NODF)

6.0 Findings – Monitoring and Accountability

6.1 Is reporting timely, reliable and complete?

Document Review

A review of FedNor Business Plans for the previous three years indicates that FedNor has enhanced its capacity in the area of performance-based planning, monitoring and reporting. For example, some noteworthy examples evident in the most recent annual Business Plans are as follows:

• planning pressures facing the program are described (e.g., funding renewal, advertising expenditures, social economy, etc.);
• outcomes, activities / outputs and for each strategic priority area are clearly identified along with budget allocations for each of the priorities;
• in most instances, performance targets are identified; and,
• a corporate management renewal action plan that addresses IC priorities in areas such as the Service Improvement Initiative, Government On-line as well as other internal priorities related to human resources and financial management and communications.

Although a review of the Business Plans for the previous years shows an increased level of sophistication in planning and analysis of FedNor's operating environment, the organization and format of the Business Plans has changed each year making it difficult to easily track progress against specific performance measures on a year-to-year basis.

Data Review

A review of the FedNor data found that, for most completed projects, the project officer had completed the required Project Outcomes and Evaluation Scoresheets. However, further analysis of this data showed that the scoresheets were completed inconsistently and incorrectly. For example:

• Questions 1 to 12 of the score sheet require yes or no answers; the instructions in the user manual specifies that the only symbols allowed in entry are "Y" or "N" and that N/A or no response are not correct entries; however, analysis of the data reveals the following problems with the responses to these questions:
  • numeric values (2 or 3 in each of the 12 questions);
  • ND (no data) (between 3 and 14 depending on the question);
  • N/A (as many as 95 for question 2);
  • the word "Excerpts" was written in one case at question 2;
  • there were 36 blanks for each question; 37 blanks for question 9; and,
  • therefore the correct Y/N answers were between 534 and 642 out of 683 entries.
• For questions 15 to 35, a 1 to 5 score was to be provided; the instructions in the user manual specified that no zeroes should be used except for question 33 where a "0" indicates that the questions is not applicable; however, analysis of the data reveals a range of zeroes (for all questions except question 31 – as many as 47 for question 24), blanks (between 39 and 56 blanks for all questions), 80 N/As and 4 NDs for question 33. The number of correct 1 to 5 answers (and 0 to 5 for question 33) therefore ranges between 573 and 643 out of 683.
• Another problem involves overall scores above 1.0 for those areas where computations are required.

Interviews

FedNor staff were somewhat satisfied with timeliness, reliability and completeness of reporting from clients. As would be expected, there are some clients who are very good at meeting all reporting requirements and others who are not. Program officers are very pleased with the support they receive from Payments and Monitoring and the systems in place for monitoring reporting requirements. Staff indicated that the assessment approach for ranking client risk has been useful. Officers also discussed that for new clients they make an effort to ensure that the reporting requirements are clearly understood and that they will often include a representative from Payments and Monitoring in discussions with clients at project outset to discuss procedures and answer client questions. Program officers have found the electronic reminders of when reports are due to be very useful. It was suggested by a few officers that FedNor could benefit from a full scale client relationship management system that would serve as a central database for tracking interactions with clients. Current systems are organized by project. Generally, interviewees indicated that reporting has improved but further enhancements to administrative practices should continue to be developed (e.g., an electronic payment system is under development and testing).

Interviews with CFDCs and other stakeholders revealed that there are differences in opinion with respect to how easy or difficult it is to meet FedNor reporting requirements as it relates to NODF projects. While some were generally satisfied, many felt that reporting is onerous and disproportionate to the funding received and that reporting requirements have increased exponentially in the past few years. The most significant concern expressed was that processing of payments takes too long particularly for smaller organizations that have cash flow challenges.

Recipient Survey

In total, 93% of surveyed recipients indicated that they fully understood when they were supposed to submit reports and 91% understood what information they needed to include in those reports.

6.2 Are program data reports providing FedNor useful information for management purposes and to guide future program decisions?

Interviews

FedNor management noted that, when they needed specific information for management purposes, they asked for this information. They noted that, in some cases, it was a lot of work for the people who had to provide the information, because the information was not readily available. However, management generally agreed that, if they needed information, they could get it.

6.3 Is the reporting system allowing FedNor to demonstrate the outputs and outcomes of its programs and thereby to meet accountability requirements?

Interviews

FedNor management noted that performance monitoring was still in various stages of development for a range of valid reasons. These included: recent changes to the FedNor RMAF which has implications related to the performance measurement system; changes to the departmental system; new staff / changes in the staff involved in the further development of performance measurement tools; and conflicting priorities.

While all managers interviewed agreed that a lot of progress had been made over the last several years, the key issues raised by some managers were:

• Project Outcomes and Evaluation Scoresheets : There is no link between the project objectives, as stated in the applications, and the project outcomes reports that are completed by the project officers. Additionally, these are completed to varying degrees of accuracy depending on the officers. Finally, the scoresheet has not been updated in quite some time and is not linked to the program's outcomes as outlined in the NODF's logic model.
• Departmental System : There are noted difficulties in working with the Industry Canada departmental system for capturing and analysis of information on projects. The system does not provide the flexibility required to easily capture outputs and outcomes that are specific to NODF. A separate NODF database may be helpful if this issue cannot be appropriately addressed.
• Lack of Baseline Data : At the project level, the tools are not available to collect baseline data and to thus be able to measure the impacts of the projects (e.g., socio-economic impacts of tourism projects). Additionally, many clients do not have the capacity to do this appropriately. Even though it may be ineffective to attempt to do this for all areas, it could be strategically important to do it for key program areas and to thus be able to show program progress (not necessarily project progress) in these key areas.
• Ad Hoc Information : while it is important to ensure that FedNor has quantitative performance information, this information is not necessarily required on an ongoing basis and it may not be feasible to collect it on an ongoing basis. Ad hoc information is extremely useful for ongoing management and should not be ignored.

FedNor staff were less familiar with the performance measurement system currently in place for the NODF. Most seemed to be unaware of how project specific results are integrated or interpreted on a more macro level. Some felt that the current score sheet that is completed by officers at the end of projects needs to be revamped and should include room for anecdotal and interpretative information. Some officers also indicated that more work should be done with project proponents at the outset to discuss the importance of performance data to instill awareness and emphasize FedNor's commitment to results-based management.

6.4 Conclusions – Monitoring and Accountability

12. FedNor has improved its capacity in terms of planning for results. For example, the recent Results-based Management and Accountability Framework (RMAF) is providing management with a better tool for measuring performance. The planning process is more closely linked to the NODF's performance management requirements. FedNor has also invested more resources into this, as evidenced by staff which are dedicated to performance monitoring. Additionally, FedNor has undertaken special studies to review specific aspects of the program (for example, the Youth Internship Program and post evaluations of trade missions). With respect to projects, officers are meeting their requirements in terms of assessing project risk and completing the project outcomes scoresheets. Additionally, with respect to the program as a whole, there is evidence of a lot of analysis and reporting of the data that is currently available.

However, the RMAF does not fully reflect the present design and delivery of the NODF. The existing tools and systems have not been updated to reflect the performance measurement requirements outlined in the recent RMAF. Additionally, it is unclear if the information collected from the project outcomes scoresheets is valid and reliable. It is also unclear if program officers clearly understand how to complete the scoresheets. The departmental systems for capturing and analyzing project information remains poorly aligned with NODF requirements. Finally, the departmental system is not aligned with the project outcomes scoresheets. Therefore, it is difficult to clearly determine the extent to which projects are achieving their intended outcomes.

13. The Industry Canada departmental systems (i.e. the Grants and Contribution Reporting System – GCRS which draws upon the information in the Contribution Management Information System – CMIS) to capture information on projects is inadequate to meet the monitoring and accountability requirements of FedNor with respect to the NODF. It currently does not provide the flexibility required to capture the intended and actual results of projects as outlined in the program's RMAF. Additionally, since RMAFs are intended to be "living" documents, program results can change over time, particularly for a program such as the NODF. The Industry Canada departmental system is not currently flexible enough to provide FedNor with the ability to adjust the fields in the database as the program evolves to better reflect the changing needs of Northern Ontario communities.