Summative Evaluation of the Northern Ontario Economic Development Fund (NODF)

4.0 Findings – Staff Support

4.1 Are project clients satisfied with counselling, advice, training or workshops provided?

Interviews

FedNor management and staff felt that for the most part clients were satisfied with counselling and advice provided. Several described their close relationships with clients and stressed the importance of early involvement with proponents as project ideas are developing. Several were unaware of specific training and workshops provided by FedNor through the NODF apart from some workshops related to trade and tourism.

CFDC interviewees provided a mixed reaction to the question of satisfaction with counselling, advice, training and workshops. Some were very satisfied with support and advice from FedNor. Many were unaware of training or workshops that had been provided. A concern with the cost to travel long distances to attend training sessions was identified as an impediment. Some also expressed concern that there appears to be inconsistencies in the advice provided by FedNor staff with respect to project selection criteria and matters pertaining to reporting and payment claims.

Recipient Survey

In total, 86% of surveyed recipients indicated that they had received counselling or advice from FedNor. Of those who did receive counselling or advice, 97% indicated that they had used or intended to use the advice. The key ways of using this advice were:

• for project applications / proposals (32%);
• for project planning purposes (17%);
• to find partners or other sources of funding or other contacts (15%);
• for general knowledge on their services and how to use or access them (13%);
• to help in filling out various forms (11%); and,
• to help in project management or project implementation (11%).

All other responses were provided by less than 10% of the respondents.

Only 21% of surveyed recipients indicated that they had received training or attended a workshop offered by FedNor. However, of those, the great majority (91%) indicated that they had used or intended to use what they had learned. The key ways of applying what they learned were:

• for day-to-day activities (23%);
• for project management or project implementation (15%);
• for planning purposes (15%); and,
• for project applications / proposals (10%).

All other responses were provided by less than 10% of the respondents.

Case Studies

None of the projects examined in the case studies involved formal counselling, training or workshops. Therefore the issue was interpreted to involve the advice provided by project managers to applicants in developing the project application.

A number of the organizations applying for project funding have long standing relationships with FedNor and have received funding for several projects. In one case examined, the Sault Ste. Marie Innovation Centre has had eight youth interns funded through FedNor. In many of these cases, project applicants are generally familiar with the criteria for funding and require little additional advice or support from FedNor staff. However, in other cases, advice from the FedNor officer has had a major effect on the scope of the project. An example of this is the project involving development of a GIS plan for seven municipalities in the District of Parry Sound. Initially the Township of the Archipelago submitted an application to develop a plan for their Township alone. The FedNor officer encouraged a broadening of the application to include seven local municipalities, all of which could benefit from a GIS plan. Through the revised project, a number of rural municipalities were able to pool resources to undertake a project that took advantage of the GIS expertise of one staff member of the Township of the Archipelago to benefit all seven municipalities. The revised project was also consistent with the FedNor objective of developing larger regionally based projects, where appropriate, to encourage networking and partnerships among small Northern communities.

top of page

4.2 Is FedNor providing appropriate support and follow-up to project representatives?

Service Improvement Initiative⁹

There were several questions in the Service Improvement Initiative survey which dealt with staff support and follow-up. Table 11 summarizes the satisfaction with relevant aspects of FedNor support and follow-up as well as the importance of these features.

Table 11: Satisfaction with and Importance of FedNor Support and Follow-Up

Feature	Mean Satisfaction (out of 5)	Mean Importance (out of 5)	Service Gap (Satisfaction minus Importance)
Courteous	46	45	1
Helpful	45	48	-3
Competent	45	48	-3
Have up-to-date information	43	47	-4
Respectful	46	45	1
Flexible	42	46	-4
Protect privacy / confidentiality	46	46	0
Keep clients informed of the status of their requests	41	47	-6
Offer suggestions about other services offered by other organizations	36	41	-5
On-going business relationship with clients	44	46	-2
Appointments easy to make	42	44	-2
Overall, with staff service	44	47	-3
Strategic advice and support to strengthen proposal	42	46	-4
Helpful advice and support during the payment process	43	46	-3
Regular and timely monitoring of projects	41	43	-2
Help in approving logo placement and design of promotional materials	37	4	-3
Questions are answered	43	48	-5
Consistent information / advice	41	47	-6
Written and verbal language is clear	42	46	-4
Overall satisfaction with communications	44	46	-4
Staff did an excellent job	45	n.a.	n.a.
Overall FedNor service delivery	44	n.a.	n.a.

The table shows that, while there are gaps between the satisfaction of recipients with support features and the importance of these features, overall, surveyed recipients were highly satisfied with all aspects of staff support. The only features with average ratings below 4 (satisfied) were related to offering suggestions about other services offered by other organizations and help by the Communication Officer in approving logo placement and design of promotional materials.

The table also shows that the largest staff support gaps when comparing satisfaction to important are in keeping clients informed of the status of their requests, providing consistent information / advice, offering suggestions about other services offered by other organizations, and answering questions.

Interviews

FedNor management noted that staff support was more intensive at the project development phase where the officers worked intensively with the clients to ensure that they had the tools they needed to meet FedNor requirements. This included assistance in refining goals and objectives to better meet FedNor strategic objectives; assistance in identifying other sources of funding and working with these sources, as applicable, to ensure that they complemented rather than duplicated each other; and helping throughout the proposal writing stages. FedNor management believed that staff did an outstanding job in this regard. However, management felt that monitoring and follow-up, once a project was approved, was somewhat limited.

FedNor officers also described support to project representatives as being most intensive during the project development phase. Many officers also have on-going and long-term relationships with clients. Support and follow-up is carried out on specific projects as well as through the participation of officers at meetings, steering committees and various networks and organizations within communities. Several officers indicated that they would like to provide more support and follow-up on projects but that workload demands and in some cases geographic considerations placed limitations on time available. Some indicated that an increased focus on 'after-care' would be beneficial.

Officers also spoke positively about the FedNor's risk mitigation approach to the assessment of project proponents. This approach helps to determine the level of follow-up and support that should be provided to clients in order to ensure success of projects. Program officers also expressed satisfaction with services provided by Payments and Monitoring and described the importance of their role in project monitoring.

Again the response of CFDC interviewees was mixed with respect to the level of support and follow-up provided by FedNor. About half were highly satisfied, appreciative of the constructive role played by FedNor officers. Specific areas of concerns that were noted by about half of the interviewees include frustration with slow turnaround time on project approvals and payments and onerous reporting requirements. A few commented that they would like FedNor to have a greater presence in their community.

Recipient Survey

The surveyed recipients were asked what support they had received from FedNor staff during and in follow-up to the project. Only 8% indicated that they had received no support and 2% did not know what support they had received. However, 4% indicated that the staff had been "supportive in every way". The most frequently mentioned responses were:

• reporting assistance and oversight; ensuring that we report; assistance with claims (25%);
• assistance in proposal preparation; how to get funding; how to fill out the application form; how to comply with the application requirements (25%);
• provision of advice; answering our questions; providing suggestions (24%); and,
• attending meetings; regular contacts; site visits; other personal communications (22%).

All other responses were provided by less than 10% of the respondents.

When asked how satisfied they were with the support and follow-up provided, 39% indicated that they were extremely satisfied and an additional 41% gave a rating of 8 or 9 out of 10. Only 5% gave this a rating of 5 or less (i.e., dissatisfaction). Overall, the mean satisfaction was 8.7 out of 10, which is very high.

When asked how they had benefitted from the staff support and follow-up, only 5% said they had not benefitted and 2% said they did not know how they had benefitted. The most frequently benefits gained from the staff support and follow-up were:

• the project was successful, met their goals, it was a better project as a result of the support (20%);
• the organization was able to do the project (20%);
• the organization accessed the funds (19%); and,
• this resulted in employment (10%).

All other responses were mentioned by less than 10% of the respondents.

When asked if they had received enough support and follow-up, 84% indicated that they had. For the few who indicated that they had not received enough support, the reported negative impacts included:

• delays / missed deadlines (33% of those who indicated that they had not received enough support);
• added confusion / complexity to the process (26%);
• resulted in more work for us (22%); and,
• unable to complete the project (11%).

Case Studies

Evidence from the case studies shows that FedNor officials often provide ongoing monitoring and support to organizations receiving funding. At times, this support includes helping develop applications for additional funding to support the organization's overall objective and through them the achievement of the overall FedNor mandate. A prime example is the provision of funding to Keewaytinook Okimakanak (K-Net) to install tele-medicine equipment for seven remote First Nations. This is part of a much larger initiative of FedNor and Bell Canada involving the coordination of 20 million in regional telecommunications infrastructure. This particular project is supportive of KNet's 6 million Health Canada Primary Health Care Transition Fund to establish permanent tele-medicine services to the remaining 18 remote First Nations in the Sioux Lookout Zone that do not have this service. Another example is the provision of funding through several projects to the Sault Ste. Marie Innovation Centre for a total of eight youth interns. Almost all of these interns have remained in the North. In the case of the construction of the museum complex in Gore Bay, this is just the latest in a number of FedNor funded projects developing museum facilities and interpretation centres in communities on Manitoulin Island, as a means to develop the Island as a destination tourist attraction.

top of page

4.3 How many objectives have been achieved in the Service Improvement Plan?

Overview

The FedNor Service Improvement Initiative (SII) Plan covers the services provided by FedNor for all of its programming. Therefore, while this issue was included in this evaluation of the NODF, it is important to note that the evidence presented in this section is not limited to NODF performance, but rather on organizational performance. For example, the data on turnaround time on payments includes all FedNor payments, be they NODF projects, payments on other FedNor programs, such as the Community Futures Program, or even payments for contracting for services.

Additionally, the survey data presented is based on two surveys conducted with FedNor clients and was therefore not limited to NODF clients.

Nevertheless, the majority of FedNor activities in this regard involves NODF services.

Service Improvement Initiative¹⁰

The FedNor Service Improvement Initiative (SII) Plan identifies five priorities. Performance against these priorities is summarized in Table 12.

All relevant results from the Service Improvement Initiative survey are presented in summary form in Table 13 which follows:

Table 13: 2002 and 2005 Satisfaction with FedNor Service Features Related to Priorities

Feature	2002 Mean Satisfaction (out of 5)	2005 Mean Satisfaction (out of 5)	Statistically Significant Change
Priority 1: Reduce the turnaround time on proposals
The turnaround time on applications was acceptable	3.4	3.8	Yes
Priority 2: Improve overall application / proposal assessment process
FedNor / IC staff provided strategic advice and support to strengthen their proposal	4.1	4.2	No
The FedNor / IC application and evaluation process was fair	4.3	4.2	No
The information that they required in order to submit an application was available	4.2	4.1	No
Application documents were easy to understand	3.9	4	No
Overall, with the application and assessment process	4.1	4	No
Priority 3: Reduce turnaround time on payments
FedNor's payment process was timely	3.6	3.9	No
Priority 4: Improve overall payment and monitoring process
FedNor staff provided helpful advice and support during the payment process	4.3	4.3	No
The approved FedNor logos were easily located on the FedNor website	4.2	4.3	No
The information required in order to request a payment was available	4.2	4.1	No
FedNor's monitoring of projects was regular and timely	4.1	4.1	No
Overall with the payment and monitoring phase of the project	3.9	4	No
The requirements regarding project announcements / communications were clear	4.1	4	No
Forms were easy to understand and fill out	3.6	3.7	No
The Communication Officer was helpful in approving logo placement and design of promotional materials	3.8	3.7	No
The public announcement of projects was well planned	3.7	3.4	No
Priority 5: Enhance communications with clients
They had a choice of English or French language	4.6	4.7	No
Questions were answered	4.3	4.3	No
It was easy to find out how to get FedNor's service	4.3	4.3	No
Overall with communications	4.3	4.3	No
Written and verbal language was clear	4.1	4.2	No
They received consistent information / advice	4.1	4.1	No

Interviews

The majority of management and staff interviewed had some general knowledge of FedNor's Service Improvement Plan, indicating that a major focus was on improving turnaround time on project approvals and payments. Most were aware that changes to program codes had been established to more accurately capture the life cycle of project proposals and reflect the situation where submitted proposals are incomplete and require additional information from the proponent.

Managers were of the opinion that significant progress had been made and that some of the satisfaction levels were so high that things could not improve much. However, there was general agreement that work was still required with the payment and monitoring process.

Officers indicated that process improvements have been made and described a good working relationship between program officers and payments staff. Interviewees commented that the organization is committed to continually finding ways to improve administrative processes. Program officers also make an effort to meet directly with new clients at project outset to ensure reporting requirements are clear and often include a representative from the Payments and Monitoring group in these discussions. Program officers were also pleased with the electronic messages they receive that serve as reminders when client reports are due.

top of page

4.4 Is staff support being provided in the official language of choice?

Database Review

The FedNor recipient database identifies 91 French clients and 984 English clients. Therefore, according to the data, 8.5% of clients are French. The FedNor staff list identifies 29 unilingual (English) officers and 16 bilingual ones. Therefore 35.5% of officers can provide services to the 8.5% of French clients. It is noteworthy that bilingual resources are readily available at all levels of management and at all staff levels.

Service Improvement Initiative

The two SII surveys revealed that clients were very satisfied that they had a choice of English or French languages. In 2001/02, 72% of clients were very satisfied with this and in 2004/05, 74% of clients were very satisfied.

Interviews

FedNor management and staff were asked if they were able to provide clients with services in both official languages. While less than half those interviewed indicated that they were bilingual, all of the interviewees stated that this had never presented a problem and that, if a Francophone client did require services in French, staff resources would be available to meet these needs.

Recipient Survey

In total, 86% of surveyed recipients indicated that they preferred to deal with FedNor staff in English, 11% preferred French and 3% had no preference. When asked if they were able to get support in the official language of choice, an overwhelming 97% said yes. However, those who preferred French were statistically significantly less likely to be able to get support in their language of choice than their English counterparts. Nevertheless, as shown in Table 14, the great majority of French and English clients are able to get staff support in the language of their choice.

Table 14: Ability to Obtain Staff Support in Language of Choice

* denotes less than 1%
Able to get Staff Support in Language of Choice	Language of Choice
	French	English
Yes	91%	99%
Sometimes	5%	*
No	4%	0%
Don't know	0%	*

top of page

4.5 Conclusions – Staff Support

2. The role of FedNor staff, in particular of program officers, is critical to the successful planning, development and implementation of projects. There are several key aspects with respect to staff support that contribute to this success:

• the extensive advice and counselling provided by the officers in the development of proposals;
• the due diligence exercised by program officers in ensuring, not only that NODF funding does not duplicate or overlap other sources of funding, but also that the client is aware and has access to all other potential sources of funding;
• the local presence of the program officers;
• the on-going relationship of staff with some clients; and,
• the ability to provide services in the official language of choice.

However, there are concerns with some aspects of the support provided in terms of project implementation, monitoring and follow-up. These concerns include:

• some inconsistency among program officers in applying the criteria as well as in the level and type of support provided during the project; and,
• workload demands that do not provide program officers with the time to provide an adequate level of monitoring and follow-up that they (and others) believe is required.

3. All priorities outlined in the Service Improvement Initiative plan have been successfully addressed except with respect to the turnaround time on payments. However, the turnaround time on payments is no longer assessed as being a service improvement priority, from the clients' perspective. This is due to the fact that clients are slightly more satisfied with FedNor's performance on this and they do not believe this to be as important a service feature as it was in earlier years.

---

⁹ As a part of the Government of Canada Service Improvement Initiative, FedNor / Industry Canada surveys of clients and staff were undertaken in 2002. The purpose of the surveys was to compare client and staff perceptions, determine where success had been realized and areas for FedNor / Industry Canada service improvement. In follow-up to the 2002 surveys, FedNor / Industry Canada contracted a second survey of clients in 2005. The 2005 approach involved a survey of 151 FedNor / Industry Canada clients who had active projects in 2004. For further information please see: http://www.ic.gc.ca/eic/site/ae-ve.nsf/eng/03067.html (Return to reference 9)

¹⁰ The Service Improvement Initiative has, to date, involved two surveys resulted in two separate reports (in 2002 and 2005). Basic information on the sample sizes is provided in the methodology section. More details on the studies themselves is available in the two reports produced at the time of the surveys.(Return to reference 10)