Final Evaluation of the Community Futures Program in Ontario

Executive Summary

This report presents the findings of a final evaluation of the Community Futures (CF) program in Ontario. An evaluation of the CF Program is required by the Transfer Payment Policy and was undertaken to respond to the requirements outlined in the program's Results-Based Management and Accountability Framework (RMAF).Government Consulting Services (GCS) was engaged to undertake the evaluation of the CF program, and more specifically, the delivery of this program through the four Regional Development Agencies (RDAs): the Atlantic Canada Opportunities Agency (ACOA), Canada Economic Development for Québec Regions (CED-Q), Western Economic Diversification Canada (WD), and the Federal Economic Development Initiative for Northern Ontario (FedNor) under Industry Canada (IC).

The findings presented in this report focus specifically on the delivery of the CF program through Industry Canada and FedNor. The evaluation was conducted between November 2007 and September 2008.

The aim of the Community Futures program is to: support local rural communities and small and medium-sized enterprises (SMEs) in meeting their economic needs; help rural communities to develop and implement long-term community strategic plans leading to the sustainable development of their local economies; and provide resources to local CF organizations to build community capacity to adapt to and manage change. In Ontario, the CF program is delivered through a network of 61 non-profit Community Futures Development Corporations (CFDC) in seven regions (East, Southeast, South Central, Southwest, West, Northwest and Northeast).

According to budget information provided by FedNor, over the six-year study period, a total of $131.25 million in contributions was allocated to the CFDCs. With respect to operating costs, FedNor does not maintain separate budgets for each of its programs, therefore figures for O&M and capital included costs for the CF Program as well as the Eastern Ontario Development Program (EODP) and the Northern Ontario Development Program (NODP). Each CFDC receives roughly the same amount of operating dollars based upon base funding agreements, as proposed in the CFDC business plans.

Methodology

The evaluation methodology integrates the use of multiple lines of evidence and complementary quantitative and qualitative research methods as a means to ensure the reliability of results being reported, and the validity of information and data collected. The research methods included:

• document review;
• stakeholder interviews;
• surveys of CFDCs and clients;
• review of administrative and labour market data;
• comparative analysis; and
• case studies.

The evaluation focused on four main areas: program relevance, design and delivery, success, cost-effectiveness and alternatives. Data reliability varied across the methodologies employed in the evaluation. Consequently, reliability of findings related to Program Success and Cost-effectiveness is low.

Relevance of Program

The evaluation found that there is a continuing need for the CF program, primarily because there are limited sources of funding available for SMEs, and CFDCs are supporting businesses that may not have otherwise been able to start, expand or survive. This need for funding is more pronounced in smaller communities, which are predominantly located in Northern Ontario, many of which have higher rates of unemployment and lower earnings than the provincial averages. In recent years these communities have been affected by fluctuations in commodity prices and the CFDCs are helping to bring economic stability to the communities. The biggest impacts of CED work have been that it mobilized businesses or community groups and improved or developed business knowledge.

The objectives of the CFDCs are in-line with the objectives of the CF Program as they are working towards fostering economic growth and stability, fostering creation and maintenance of jobs, helping to create diversified and competitive local economies, and helping to build sustainable communities. Similarly, the objectives of the CF Program are aligned to Industry Canada's strategic objective of a competitive industry and sustainable communities. Based on interviews with FedNor senior managers and external stakeholders their view is that the Federal Government has a legitimate, continuing role to play in delivering the CF Program.

In addition to providing funding for SMEs, the CF Program offers a range of business services (e.g., counselling, business information, referrals) and provides assistance to communities with planning and implementing their economic development priorities. CFDCs have been successful in working with other programs and organizations to ensure that their services compliment, rather than duplicate one another. The evaluation raised a few issues with respect to potential duplication of administrative efforts and reporting of outcomes with the EODP and NODP programs.

Design and Delivery

The CF Program has been in operation for over 20 years and changes have been made to its design and delivery as issues have been identified. Therefore, the Program appears to function well and the evaluation identified few issues impacting on the success of the program. Based upon stakeholder interviews and surveys, funding was the biggest issue cited by stakeholders who felt there was a particular need for additional funding for operations, community economic development (CED) activities, and investment funds.

Overall, the investment funds of the CFDCs appear well-managed with the overall portfolio experiencing positive growth in the past four years. The growth of the investment fund is influenced by the level of loan activity of CFDCs, including the proportion of those funds that are in active investment, as well as the loan loss rates. There are large variations across CFDCs with respect to the level of funds in active investment and a large range in loan loss rates. Without targets for desired performance for loan portfolios, it is difficult to assess whether there should be concerns regarding the performance of investment funds for certain CFDCs.

Recommendation #1: FedNor should work with CFDCs to establish general target ranges related to desirable levels of funds in active investment and loan loss rates based upon local realities. This would allow FedNor to assess a CFDC's loan portfolio performance and assist with improvements where necessary.

top of page

Currently, the biggest challenge for the CF Program is in the collection and management of its performance information. While a structure is in place for collecting data related to CFDC performance and outcomes, the data is rarely used for the management of the program and does not provide appropriate information to measure the success of the Program. The current system in use for the tracking of performance information (i.e., The Exceptional Assistant, also known as TEA) is not used by all CFDCs and not all CFDCs are meeting the reporting requirements (i.e., submitting annual reports). TEA is also of limited use because it does not capture all performance indicators required for measurement purposes as per the RMAF. In addition, information to measure the longer-term outcomes of the Program is currently lacking, particularly with respect to Community Economic Development (CED) activities.

The evaluation found that the data entered into TEA is unreliable, making the resulting Quarterly Reports also unreliable. The outcome information currently reported by CFDCs (e.g., jobs created) is based on estimates at the time of the application and no verification is done to ensure that those estimates were actually achieved. There also seems to be a lack of verification of data entered by CFDCs, as there were many variables in the TEA data that appeared incorrect.

Recommendation #2: FedNor should complete a review of the performance data collected for the CF Program to ensure reliable and meaningful reporting and establish additional indicators to provide information for assessing the longer-term impacts of the program, particularly for CED activities. In addition:

• FedNor needs to ensure that TEA is capable of capturing and reporting on all required performance information. It is possible that FedNor could reduce reporting requirements for CFDCs by maximizing its use of TEA and thereby improving its ability to manage the Program as well as make it more useful for CFDCs.
• FedNor should ensure that all CFDCs are fulfilling all reporting requirements and should establish consequences for those CFDCs that do not comply with the requirements.
• FedNor should ensure that CFDCs are not reporting activities undertaken and outcomes achieved through the use of EODP and NODP funds in CF Program reports (i.e., in the annual reports).
• To ensure data reliability, FedNor should implement a system whereby TEA data captured and submitted in Quarterly reports would be reviewed on a regular basis; FedNor needs to work with the CFDCs to ensure data is corrected should issues be identified.

Program Success

Due to concerns with respect to the reliability of Quarterly Report data, especially with regards to reporting of business services provided, findings on program success are primarily based on soft data. It appears that CFDCs have provided a consistent level of business information, counselling and referral services to clients over a four-year period. The services provided by CFDCs have been deemed necessary by clients and there is a very high level of satisfaction with the services provided by CFDCs. The CF Program is also meeting the needs of OLMCs by providing advice, support and services in both official languages where required. There are however, on-going challenges related to maintaining skilled, bilingual staff and costs associated with providing bilingual information materials.

Through its loan portfolio, business services, and CED activities, the CF Program has been successful in improving clients' business skills and knowledge, creating new business start-ups, maintaining and strengthening existing businesses, increasing community capacity, and assisting communities to diversify their economies. The impacts of some of these activities, particularly CED work and the longer-term impacts of the program are more difficult to measure as indicators to measure these impacts have yet to be established.

Cost-Effectiveness and Alternatives

The current delivery model in place for the CF Program integrates a number of components that are important to successful community economic development which includes a grassroots approach where decisions are made at a local level; the involvement of local community volunteers; and a focus on partnership development and cooperation between community organizations. No other delivery alternatives were identified.

It was not possible to determine whether FedNor is cost-effective in administering the CF program compared to EODP and NODP, as program costs² are not available separately for these programs.

Recommendation #3: FedNor should undertake a costing exercise to identify the costs for the operation of NODP, EODP, and the CF Program. This would be particularly useful because different delivery models are used to deliver these programs and it would allow FedNor to assess the extent to which the different delivery models are effective in achieving program outcomes.

Overall the CFDCs are cost-effective in administering the program. For the amount of funds received, CFDCs are providing a fairly high level of investment into the community, with approximately $42 million lent and $9 million contributed to CED annually province-wide, as well as offering a number of other services. The extent to which all CFDCs are equally effective in delivering the program raises a question about cost-effectiveness, as there are great ranges in the level of activity and outcomes of the program across CFDCs. While there may be reasons for some CFDCs to have varying levels of activity and outcomes equal to other CFDCs, such as geographic location, demographic composition of the community, the economy of the community and the level of demand for services, it may also reflect a lower level of effectiveness.

It seems that as the economic conditions of an area improve, the demand for CFDC services increases to the point where the private sector begins to compete for "bankable" clients. In these generally more populated areas, the CFDCs are serving more start-ups who do not have long credit histories. As there is more credit available in these regions, the loan clients are able to leverage the CFDC loan dollars to a greater extent and thus impact more jobs and businesses per investment fund dollar. However, each of these jobs in the larger communities represents a smaller percentage of the jobs in the regions. So while smaller communities may not demonstrate the same volume of job impact, percentage wise, they are impacting their community more profoundly. There may also be differences in cost-effectiveness related to the performance of the CFDCs. FedNor in concert with the CFDCs would be in the best position to determine whether certain CFDCs could be more effective in delivering the program given the many variables which influence effectiveness.

Recommendation #4: FedNor should work with CFDCs to establish target ranges for the level of activity for CFDCs (e.g., number of loans, investment fund growth). These ranges should be developed in consideration of factors that may influence the activities of CFDC (e.g., geographic location, demand for services). This would provide a guide for program offices and assist them in understanding whether CFDCs could be more effective in delivering the program or achieving program results.

---

² Program costs are defined as the cost to FedNor to administer the program, not including funds provided to the CFDCs (i.e. FedNor salaries and O&M)