Archived — National Evaluation of the Softwood Industry and Community Economic Adjustment Initiative (SICEAI)

Final Report

March 2006

Prepared for
Industry Canada

Prepared by
Bearing Point

Tabled and approved by DAEC on

This publication is available upon request in accessible formats.

Contact:  
Multimedia Services Section  
Communications and Marketing Branch

Industry Canada 
Room 264D, West Tower 
235 Queen Street 
Ottawa ON K1A 0H5

Tel.: 613-948-1554  
Fax: 613-947-7155  
Email: ic.cmb-creative.ic@canada.ca


Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Industry Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Industry Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Industry Canada.

For permission to reproduce the information in this publication for commercial redistribution, please email: copyright.droitdauteur@pwgsc.gc.ca

Aussi offert en français sous le titre : Évaluation de l'Initiative nationale d'adaptation économique, industrielle et communautaire pour le secteur du bois d'œuvre résineux (INAEICB).


Table of Contents

Annexes (Separate document)

(Note: Annexes are available via an Access to Information)

Appendix A: Glossary of Acronyms

Appendix B: Case Studies

Appendix C: Data Collection Instruments

Minor editorial changes were made to this report in order to prepare the document for posting to the Industry Canada's Website (including removal of standard Appendices such as list of interviewees and questionnaires). Readers wishing to receive a copy of the original version of this report should contact the Audit and Evaluation Branch at Industry Canada.

Executive Summary

Introduction

This document is the Final Report of the formative evaluation of the federal government's Softwood Industry and Community Economic Adjustment Initiative (SICEAI). It also is intended to help Industry Canada (IC), the Regional Development Agencies (RDAs), and FedNor to help design future adjustment initiatives, especially those directed towards the forest industry and rural communities. This evaluation report is formative in the sense that—although the SICEAI program has already ended—the program existed only for a short time and many of the projects it supported had barely been finished at the time this study was conducted. The findings must also be taken in the context that program officials in federal, regional, and economic development agencies had a relatively short period to "ramp up" activities from essentially nothing to full operations. Thus the program was addressing a very significant adjustment situation (see below) in the affected communities with little lead time and modest resources.

The U.S. Softwood Tariffs

On March 21, 2002, the U.S. Department of Commerce first announced subsidy and dumping tariffs with respect to Canadian softwood lumber. By November 4, 2002, the number of duty-related, direct job losses across in sawmills Canada totalled 3,940, and by September 31, 2004 were estimated at about 5,800. It was expected that production could decline by 8–10 percent (as high-cost producers dropped out) and that Canadian exports would decline by up to 20 percent, or up to $1.8 billion.

The SICEAI Program

To mitigate impacts of the tariffs, the government initiated a package of measures, of which the key response was the $110 million SICEAI program, announced October 8, 2002. The program's objective was to create long-term, sustainable economic benefits in regions and communities affected by the tariffs. (Note that the program was not intended to directly provide assistance to laid-off or displaced workers, or to directly replace these forestry-related jobs.) SICEAI was originally intended to expire at the end of the 2003–04 fiscal year. Due to delays in initiating the program, no money was released to proponents in 2002–03 and very little in 2003–04, requiring that the program be extended to the end of the 2004/05 fiscal year.

The program provided both repayable and non-repayable contributions, depending on the region and nature of the project. Two types of projects were eligible:

  • (1) Diversification, productivity improvement and new economic opportunities, which would create tangible and sustainable long-term jobs; and
  • (2) Community capacity-building, including developing community economic action plans, strengthening local developmental and leadership capacity, and exploiting opportunities in other local industries for sustainable economic development. The program was directed towards hard-hit communities.

SICEAI was funded federally by Industry Canada (IC), and delivered regionally by Regional Development Agencies (RDAs): Western Economic Diversification Canada (WD), Canada Economic Development for Quebec Regions (CED) and the Federal Economic Development Initiative for Northern Ontario (FedNor), which is under Industry Canada ADM Operations, as well as other regional and private sector partners. Delivery in B.C. for nonrepayable contributions was through a third-party delivery model: WD delivered non-repayables through an agreement with the Community Futures Development Association of BC (CFDABC) and its individual Community Futures Development Corporations (CFDCs).

top of page

Evaluation Issues

The study issues were adapted from the SICEAI Results-Based Management and Accountability Framework (RMAF), but are not identical to the RMAF questions. Main issue topics included:

  • (1) Program relevance;
  • (2) Design and Delivery;
  • (3) Preliminary Results and Success;
  • (4) Cost-effectiveness; and
  • (5) Lessons Learned.

Methodologies

The main methodologies included:

  • (1) Review of SICEAI program documents (e.g., program guidelines, eligibility and review criteria, performance data, statistical information);
  • (2) Review of project files for case study projects;
  • (3) Interviews with program officials and staff in both "headquarters" and in RDAs in the three regions;
  • (4) A survey of the 33 participating CFDCs in BC;
  • (5) A survey of as many program clients as could be contacted within study resources (n = 252); and
  • (6) Fifteen mini-case studies of projects that were selected to best illustrate lessons learned and/or best practices.
top of page

Finding on Program Relevance

Appropriateness. As part of a larger government initiative, the SICEAI program was very appropriate in that it provided the resources required in order to respond quickly to the layoffs in the forestry industry, and to start planning for community adjustment measures. Although certain factors (e.g., which communities were affected) and specific project funding requirements changed over the lifetime of the program, the overall needs addressed by the program did not change. At the time the program was announced, there were no reasonable alternatives to SICEAI. The community-based delivery model was appropriate, because the RDAs and CFDCs could best respond to the needs of their communities, while allowing Industry Canada to ensure consistency between regions and to remain accountable for the program overall. At least 72 percent of the projects would not have proceeded at all, or only after a significant delay, without the program, meaning SICEAI had good incrementality (also see below).

Program Design and Delivery. The roles, responsibilities and respective spending authorities between IC and the RDAs under SICEAI were unclear to all concerned early in the program's lifetime, and were one cause of the delay in implementing the program. However, these difficulties were eventually resolved.

Project review and selection. A review of the Terms and Conditions indicates that the program eligibility criteria were appropriate and, in general the file review and other data indicate the criteria were appropriately applied to individual projects. All three regions were very flexible in terms of the nature of projects supported, so long as they clearly fit the intent of the program. However, the program's detailed objectives (e.g., community vs. industry focus), rules, and operating procedures were unclear to all concerned at the beginning of the program, and communications from Industry Canada on these topics were often confusing and conflicting.

Targeting. There were three types of marketing and outreach targets set:

  • (1) Towards communities—those that had suffered significant impacts because of the tariffs;
  • (2) On potential clients and projects—especially forestry companies that were SMEs, First Nations communities, and/or projects were value-added; and
  • (3) Strategic targets—for example for outreach to specific regions, or to attempt to generate interest for projects in certain sectors (e.g., tourism). Targeting on First Nations communities or SMEs in value-added wood industries was especially done in BC, based on internal WD strategies.

However, there were no rigidly enforced targets set on the allocation of contributions—project selection was done in an entirely "bottom-up" manner based on the eligibility and quality of the project application. Program targets overall (e.g., jobs by sector, or by region) were not set by IC or by the regions, although an attempt was made in BC to market and encourage client groups to propose projects which fit provincial sector priorities. Targets for job creation and/or maintenance, timelines, deliverables, and matching funding were set for each individual project. These were set in conjunction with proponents, and were generally seen as reasonable. However, the very short timeframe available for project implementation and completion caused problems for many clients.

Client satisfaction. Clients were generally satisfied or very satisfied with all aspects of service delivery, especially from regional offices and their community-based delivery networks (including CFDCs), and very few complaints were noted. Many clients noted that the assistance of the local agencies was essential in allowing them to succeed.

Reach. The program made every effort to target communities affected by the tariffs, although in practice not all were served. Small, remote, and First Nations communities were especially targeted for outreach, as were communities known not to have much economic development capacity. Some aspects of eligibility criteria reduced the program's ability to serve needy communities; e.g., if the timing of layoffs didn't quite match the program's application period, or if communities did not suffer "direct" sawmill job losses, or if local governments could not secure the 10 percent private sector funding required.

top of page

Findings on Preliminary Results and Success.

Program success. The program has had a modest success on mitigating the direct impact on displaced forestry sector workers as a result of the tariffs, although not surprisingly it has not been able to dramatically diminish these effects (nor was the program intended to directly assist these displaced workers or replace these jobs). It has been rather more successful in helping communities diversify their economies, in creating new jobs, and increasing community capacity for responding to future change and the need for adjustment and diversification.

Project success. About 65 percent of projects are currently complete and operational, with another 11 percent complete but not yet operational, and 14 percent still in the construction phase but expected to be completed. Only 1 percent of the projects are not expected to be completed, and another 2 percent were never started (the remaining 12 percent of respondents to the client survey either answered "not applicable" or did not answer at all). Both RDAs and clients noted that the projects have created and/or maintained jobs, and that the projects have often helped communities think "outside the forestry box" in diversifying their economies.

Community Impacts. About 85 percent of projects were successful in creating jobs or maintaining jobs to date. Just over half the projects did both. A conservative extrapolation of job creation and maintenance is shown below.

Findings on Preliminary Results and Success
  No. jobs expected No. jobs actual 
to date
No. jobs estimated in 3 years' time
(Totals may not add due to rounding errors.)
Direct job creation 3,100 2,600 3,900
Direct job maintenance 2,200 4,000 2,800
Indirect job creation 1,100 850 1,400
Indirect job maintenance 1,100 1,500 750
Total all jobs 7,600 9,000 8,900
top of page

It was noted that the jobs created through SICEAI are not direct replacements (either in terms of the individuals affected, or in the wages replaced) for those lost in the forestry industry, which tend to be higher paying jobs. (The program was not, of course, intended to directly replace these jobs.)

A conservative estimate is that the program to date has created about $147 million in annual gross revenues, and maintained about $436 million. These figures are expected to rise to about $646 million and $490 million, respectively, in three years' time.

Capacity building projects. Capacity building projects were not common in any of the three regions. It is too soon to judge the impacts of those that were funded.

Increasing Investments. The program was successful in obtaining about $247 million in external (non-program) funding, representing a leverage of about 225 percent of the $110 million federal funding. The "gross (i.e., total) leveraging" of the average project in our client survey was about $516k, or roughly 230 percent of the program's contribution to the project. The "net leveraging" for the average project in our client survey was about $208k, or about 157 percent of the program's contribution. Net leveraging is defined as the additional investment that occurred because of the program's existence; i.e., non-program funding that would not have been committed to these projects in the absence of SICEAI. That the net leveraging figure is relatively large helps confirm that SICEAI was indeed incremental—the program support acted as a critical seed to help project proponents find funding and implement the project.

Unexpected Impacts. Virtually all of the unexpected impacts for clients were positive, and included additional jobs being created, new business opportunities being identified, considerable media attention, revitalization of neglected business areas, and a generally greater sense of community participation and collaboration. Negative impacts were mainly related to the burden of preparing applications and dealing with delays. For RDAs, similar positive impacts were noted. Some additional negative impacts were related to the initial confusion regarding program goals and procedures, and (in BC) the workload associated with the very high number of applications for funding.

top of page

Findings on Cost-effectiveness

Cost-effectiveness. First, note that this study was not intended to definitively answer whether the program was cost-effective. Instead, the issue is the standard program evaluation question of whether there are improvements in the program design or delivery that could be made that would increase the cost-effectiveness of this program. A full accounting of cost-effectiveness is one issue that might be addressed in a summative evaluation, which would not be appropriate in this situation.

The general answer is that the program appears to be cost-effective. There is little "hard" data available on this point (although it is always a hard question to answer). Most respondents in the RDAs believe the program was cost-effective because the program could "piggy back" on existing local and regional knowledge and programs. The cost per job to date (i.e., all jobs, either directly or indirectly created, and including both job creation and maintenance) was roughly $12,000. If one only considers direct job creation and maintenance, then the cost per job to date is about $16,000. Both figures will remain relatively constant over the next three years—although the number of newly-created jobs are expected to increase, the number of maintained jobs is expected to diminish. The major mechanism suggested that might improve program efficiency was elimination of the Other Government Department (OGD) suspense accounts mechanism.

Due diligence. The review of Terms and Conditions and the review of program files conducted for the case studies both indicate that extensive due diligence was conducted within this program. External review of the application procedure in BC supports this conclusion.

top of page

Findings on Factors that Influenced Success

Key success factors include the local knowledge of CED, FedNor, and the CFDCs of their communities' needs, in some cases supplemented by additional staff hired specifically to help with program outreach and client support. This local expertise also helped to extend the reach of the program to many smaller and more remote communities, develop high quality proposals, and help clients with issues arising (e.g., dealing with Environmental Assessments). Negative factors included the initial delays in implementing the program, and the lack of appropriate promotional materials.

top of page

Findings on Lessons Learned for Future Similar Programs

Lessons learned. A longer time frame is needed for adjustment programs to be fully effective. This will allow better outreach and marketing, better quality proposals to be developed, more time for initiation and completion of complex projects, and better retention of community expertise.

Having program goals, objectives, authorities, and procedures worked out between IC and other program partners in advance of the program's public announcement will save considerable time and frustration for all parties.

Terms and Conditions must be flexible enough to respond to individual community circumstances, but there must be an oversight mechanism to ensure the policies regarding eligibility and support levels are applied reasonably consistently and equitably across similar clients in all regions and communities across the country.

A simpler program structure than the Other Government Department (OGD) suspense accounts mechanism used by SICEAI between IC and RDAs would be beneficial.

A mechanism for identifying community and industry needs in advance of crisis situations is preferable, and would allow better identification of exactly what negative impacts need to be addressed by government programs.

Supporting larger projects offers the opportunity for significant collaboration among stakeholders (including those across multiple communities), as well as significant and broadly-spread benefits. There may additionally be long-term "knock-on" benefits not directly associated with the project. Of course, the number of smaller projects that could be supported may diminish, and a very strong business case would have to be made for a large project.

Sharing of lessons learned. Best practices were frequently shared among the RDA offices. Best practices were also shared with at least a third of clients, but it is unclear to what extent these reflect best practices for SICEAI specifically, as opposed to more "general" lessons learned through other economic development initiatives. (Of course, both are valuable.)

top of page

Recommendations for Future Similar Programs

  1. The program scope should be widened to include all communities, which are deemed to be heavily dependent on the forestry industry and are being negatively affected by restructuring in the forest sector (i.e., not just those linked directly to the sawmilling industry, as was SICEAI).
  2. A decentralized community-driven model using local delivery networks (including CFDCs) is appropriate for government initiatives intended to benefit broadly-dispersed clients who will use the funding in highly-divergent projects. This is especially true when these clients require a substantial community outreach and assistance program to help all of them develop appropriate proposals, meet complex eligibility criteria (e.g., environmental assessments), and to ensure small, remote, and First Nations communities benefit.
  3. Program Terms and Conditions should be kept flexible. However, an effort should be made to standardize the degree of (necessary) flexibility that is exercised by the regions in interpreting program guidelines.
  4. Set a longer program duration from the beginning. Note that if a longer-term program is employed, the design criteria should consider that it may require additional financial and (perhaps) human resources.
  5. IC needs to be able to anticipate problems within the industry, and begin a process of adjustment sooner, rather than having to wait for a crisis and large numbers of layoffs to be the trigger to initiate a program.
  6. Industry Canada should consult extensively with FedNor and the Regional Development Agencies at the early design stage of such programs. In particular, the details of goals and objectives, outreach and communications, project and applicant eligibility, co-funding, the application and review process (including selection criteria), delivery roles and accountabilities, signing authorities, and implementation guidelines should be carefully thought out ahead of time.
  7. Similar early consultation with the provinces, and increased efforts to engage the provinces by demonstrating the joint aspects of the issues, may result in greater federal/provincial collaboration and an increased ability to obtain co-funding.
  8. While strategic targeting is entirely reasonable with respect to marketing and outreach (e.g., to encourage and/or help specific types of needy clients to apply, or encourage particular types of projects), project selection and funding allocation should be made "bottom-up", entirely on the basis of fit to eligibility criteria and quality of the application.
  9. A simpler model than the OGD (e.g., one that would assign more responsibility and accountability within the RDAs) would offer reduced effort by clients and government alike, more transparency, and easier communications.
  10. A simple, long-term performance monitoring system should be instituted which follows-up with 100 percent of clients on an annual basis for at least five years following completion of each project to determine actual job creation and maintenance and the degree to which diversification of the local economic base has been achieved and sustained. While an added cost, collecting such long-term data would greatly help in designing future initiatives.
  11. A dedicated cost-effectiveness study of the economic benefits and costs resulting from SICEAI and other similar initiatives should be conducted. The data collected in the monitoring program discussed in recommendation #10 should be of a type to allow such a study to be carried out. Such a study would be of great interest to Industry Canada, RDAs, FedNor, the CFDAs, and other economic development organizations for any individual program. It would also allow far better comparison of the benefits and costs of different program delivery models, so long as the exact same methodologies were used to compare different programs.

1.0 Introduction

This document is the Final Report of the evaluation of the federal government's Softwood Industry and Community Economic Adjustment Initiative (SICEAI), which was intended to help forestry-dependent communities adjust to the impacts of the U.S. softwood tariffs.

This report provides information related to all evaluation study issues. It also is intended to help Industry Canada (IC) and the Regional Development Agencies (RDAs) help design future adjustment initiatives, especially those directed towards the forest industry and rural communities.

This evaluation report is formative in the sense that—although the SICEAI program has already ended—the program existed only for a short time and many of the projects it supported had barely been finished at the time this study was conducted. The findings must also be taken in the context that program officials in federal, regional, and economic development agencies had a relatively short period to "ramp up" activities from essentially nothing to full operations. Further, the program was addressing a very significant adjustment situation in the affected communities, again with little lead time and modest resources. SICEAI had about $110 million at its disposal, while the impact of the U.S. tariffs was estimated to potentially be as high as $1.8 billion as of September 2004, and to have cost over 3,900 Canadian jobs by November of 2002, and over 5,800 jobs (the latter being sawmill jobs only) by October, 2004.

Finally, although the sustainable long-term impacts of SICEAI on job creation and maintenance in the communities is ultimately of great interest, this was not an economic impact study and it did not attempt to rigorously document these impacts, many of which are expected to manifest themselves over the mid- to long-term. Thus the study's conclusions about long-term job creation and/or maintenance impacts must be regarded as tentative at this time.

2.0 Evaluation Work Plan and Methodologies

2.1 Overview of the Work Plan

The study took place from October 2005 through April 2006. The work plan for the evaluation of the Softwood Industry and Community Economic Adjustment Initiative (SICEAI) is illustrated in Figure 1.

Figure 1: Work Plan for the SICEAI Formative Evaluation

Work Plan for the SICEAI Formative Evaluation

Description of Figure 1

2.2 Evaluation Issues

The evaluation issues were developed based on the issues found in the Results-Based Management and Accountability Framework (RMAF) for the program. However, they are not identical, as some changes were made to RMAF issues specifically for this evaluation. Past and current environments were also taken into consideration so that lessons learned could more easily be developed for future similar programs.

The evaluation issues for this study are presented in the following table.

  1. Relevance
    • 1.1. Appropriate program given the needs? (a) Was SICEAI an appropriate response to the needs at the time? (b) Have the needs changed? If so, how?
    • 1.3. Alternatives. What are alternatives to the SICEAI in attempting to meet the stated objectives?
  2. Design and Delivery
    • 2.1. Program clarity. Are the objectives and desired outcomes of the SICEAI clearly identified and agreed upon by the various stakeholders?
    • 2.2. Project selection. Were eligibility criteria
      • (a) appropriate, and
      • (b) properly applied by the SICEAI?
    • 2.3. Appropriate targets. To what extent have realistic targets/performance measures been established?
    • 2.4. Client satisfaction. How satisfied are clients; e.g.,
      • (a) program benefits;
      • (b) accessibility;
      • (c) fairness;
      • (d) ability to respond to client needs;
      • (e) response time;
      • (f) guidance provided during the process;
      • (g) quality of service delivery
    • 2.5. Reach. (a) To what extent did the RDAs/CFDABC/CFDCs reach their intended target groups? (b) To what extent did RDAs/CFDABC/CFDCs provide appropriate geographic coverage and equitable distribution of funding?
  3. Preliminary Results/ Success
    • 3.1. Program success. How successful has SICEAI been in achieving its overall objectives? For example, (a) What has been the impact of SICEAI in terms of mitigating the socio-economic effects of U.S. duties on softwood lumber exports to the U.S.? (b) To what extent has the SICEAI supported community economic development and assisted communities to develop and diversify through: implementation of capacity building; increased employment and changes in employment distribution, stability and sustainability inside and outside of softwood lumber production?
    • 3.2. Community impacts. To what extent were there:
      • (a) new businesses created;
      • (b) jobs created and/or maintained;
      • (c) new innovative projects leading to diversification in softwood-dependent communities;
      • (d) improved sustainability and/or self-reliance of communities?
    • 3.3. Project success. How successful have individual projects funded by SICEAI been in achieving their stated objectives?
    • 3.4. Capacity building. For the Capacity Building component, what has been the impact of SICEAI in helping communities to identify, plan, and manage the transition required to adjust to the socio-economic consequences of the U.S. duties?
    • 3.5. Investment. What has been the impact of SICEAI in terms of increasing investments (internal and external) in affected communities?
    • 3.6. Unanticipated impacts. Have there been unexpected positive or negative impacts as a result of the SICEAI program?
  4. Cost Effectiveness
    • 4.1. Cost effectiveness. Is SICEAI the most cost-effective way to achieve the stated objectives? If not, what could be improved?
    • 4.2. Due diligence. Were the elements of due diligence applied by the SICEAI?
  5. Lessons Learned
    • 5.1. Success factors. What factors have facilitated/impeded the implementation of the SICEAIe.g.,
      • (a) reach/awareness/promotion;
      • (b) accessibility;
      • (c) planning and coordination;
      • (d) targeting of and programming for communities;
      • (e) resources;
      • (f) partnering;
      • (g) visibility;
      • (h) project monitoring and data collection (frequency and type of tools/mechanisms)?
    • 5.2. Lessons learned. (a) What specific best practices have been generated with respect to the SICEAI? (b) How and to what extent were best practices shared at the SICEAI delivery level with communities; associations (provincial and regional); coordinating/RDA partners; and CFDABC/CFDCs?
top of page

2.3 Study Activities

2.3.1 Preliminary Interviews

The study team conducted preliminary interviews with knowledgeable individuals in the three Regional Development Agencies (RDAs): Western Economic Diversification Canada (WD) in BC, Canada Economic Development for Quebec Regions (CED) for Quebec, and the Federal Economic Development Initiative for Northern Ontario (FedNor). These discussions were mainly to discuss data availability, although come early evaluation information was also obtained.

2.3.2 Document Review

The study team reviewed a considerable amount of background material, which was obtained from Industry Canada (IC) and/or the RDAs. Note that in BC, non-repayable contributions were administered through the Community Futures Development Association of British Columbia (CFDABC) and its rural network of 33 Community Futures Development Corporations (CFDCs)Footnote 1, while repayable contributions required approval from Industry Canada and were administered directly by WD. The involvement in delivery in BC precipitated further data collection from the CFDABC and CFDCs.

The available documents were reviewed to:

  • Understand the program and its operations (e.g., methods and criteria used by the RDAs in each region for project review and approval, funding allocation mechanisms, methods of repayment where applicable, information available to project proponents, etc.);
  • Obtain existing data on activities and outputs (e.g., number and type of contributions, nature of projects, reach by community and region, etc.);
  • Obtain existing data on outcomes and anticipated impacts such as job creation or maintenance, businesses created or maintained, etc. (if reported).
  • Identify lessons learned and best practices (if reported);
  • Identify interesting potential case studies.

2.3.3 Project File Review

Because of the very extensive nature of many files, and because only the BC files were centrally located at WD headquarters (FedNor and CED project files were located within individual RDA offices), project files were reviewed for the case studies only. The file review was generally used to obtain information to help prepare the case studies, and to report on four factors:

  1. Due diligence procedures—there is considerable information available on how projects were proposed, assessed, and selected.
  2. Funding structures.
  3. Intended and actual outcomes and impacts.
  4. Success factors, lessons learned, and best practices—there is considerable data on issues and challenges that arose during each project, and the solutions proposed and undertaken.
top of page

2.3.4 Interviews of SICEAI Officials

SICEAI officials were contacted using two different methods, as described below, but the same interview guide was used. SICEAI representatives include those in IC and the RDAs and their delivery partner organizations where appropriate. Although not all officials were well-informed on all evaluation issues (e.g., IC Headquarters had little information regarding client satisfaction), and for some issues they may not be completely unbiased (e.g., with respect to the success of the program), they were nonetheless able to provide an "official" perspective which could be compared to that of clients and other stakeholders.

Note: The interview guide for SICEAI officials includes questions related to all evaluation issues. Some information was available to the study team from other sources (e.g., the CED client satisfaction surveys, or the manuals describing the due diligence process). However, this gave respondents an opportunity to comment on all issues if desired.

"Headquarters" SICEAI Interviews

These included interviews with representatives within Industry Canada, as well as in WD, CFDABC, CED, and FedNor. Every attempt was made to contact all relevant individuals. Note that CED site visits were conducted by bilingual members of the study team.

Regional SICEAI Interviews

These included interviews with two Regional Delivery Managers (FedNor), one Regional Director (CED), two project officers (FedNor), five Economic Development Advisors (CED), and members of the BC Ministers Advisory Group (MAG). For CED, all Regional Directors were also interviewed as a group. For BC, a group interview was held with the Quality Assurance Review Officers (informally called "two older guys", or TOGs), and some regional CFDC representatives (in addition to the web survey of all CFDC regional managers, discussed in section 2.3.5).

top of page

2.3.5 CFDC Survey

The sample selected for this survey was the managers of 33 CFDCsFootnote 2 in BC.

Invitations to participate were emailed to potential respondents along with the project numbers for the SICEAI projects in their region and a link to the survey website. Respondents were asked to enter their project numbers in the survey, which made it possible to determine which participants had not yet responded to the survey. Some respondents initially experienced technical problems when submitting the survey, although this problem was resolved quickly, and a second email explaining this fact was sent out. An additional e-mail reminder was sent to survey non-respondents in an attempt to increase the response rate, and the CFDC representative on the evaluation steering committee also encouraged CFDC managers to respond during a managers' meeting. The survey remained on-line for approximately one month.

The sample size for this survey was 33. A total of 24 respondents (73 percent) returned completed survey questionnaires.

top of page

2.3.6 Proponent (Client) Survey

The sample selected for this survey was all SICEAI project proponents (also called "clients" in this report) for whom email addresses were available.

A survey pre-test was conducted with 6 SICEAI clients (two per region), to ensure that the survey website functioned properly and that all questions were clear. Following the pre-test, invitations to participate in the survey were emailed to all potential respondents along with the project number(s) for their SICEAI projects and a link to the survey websites (English or French). Respondents were asked to enter their project numbers in the survey, which made it possible to determine which participants had not yet responded to the survey. Multiple attempts were made to increase the response rate including: an e-mail reminder to all survey non-respondents; telephone calls to about 80 percent of survey non-respondents in all three regions (although not all of them were reachable); and emails from the RDAs to clients encouraging them to participate in the survey. The survey remained on-line for approximately one month.

The sample size for this survey was 252, or about 46 percent of the total number of projects supported by SICEAI. Those not contacted represented client organizations for which e-mail information provided by the RDAs was either missing or proved to be incorrect. In our follow-up phone calls, the latter situation was usually the result of routine changes to contact information (e.g., firms using a different Internet Service Provider) rather than substantive changes to the client organization (e.g., suspension of operations). A total of 122 respondents (48 percent of our sample) returned completed survey questionnaires. The breakdown of these figures by region is as follows, and shows no response bias by region:

Proponent (Client) Survey
  Total Number of ClientsFootnote * Sample SizeFootnote ** Number of Respondents % of Respondents
  1. * The total number of clients does not necessarily correspond to the total number of SICEAI projects, since a number of clients had multiple projects. (back to footnote reference *)
  2. ** CED did not have email addresses for all Quebec clients, although the study team was able to augment the list of emails provided by calling clients directly. In addition, some email addresses in all three regions were no longer in service. (back to footnote reference **)
Quebec 311 101 47 47%
Ontario 60 58 32 55%
BC 140 93 43 46%
Totals 511 252 122 48%

Clients were asked about all evaluation issues with the exception of cost effectiveness and due diligence, for which it was anticipated they would not have much knowledge, or might be biased, respectively.

The surveys were divided into those for BC and Ontario, and that for Quebec (which does not include the client satisfaction questions, as these were asked in separate CED surveys). Note that for comparability's sake, in the client satisfaction questions for BC and Ontario, the same rating scales were used as in the CED client surveys.

top of page

2.3.7 Mini Case-Studies

Overall, we conducted 15 very detailed mini case-studies; five from each region. Detailed case studies are found in Appendix B. The mini-case studies were used to investigate some issues in detail, in particular the clients' perception of the delivery process, the benefits expected and gained from the projects, incrementality, and lessons learned. The findings from the mini-case studies were used to help reinforce the main study findings and add detail that could not be obtained through survey techniques.

An individual "mini-case study" consisted of a file review plus telephone interviews with the main project contact and other stakeholders. (Where appropriate, these interviews were conducted by bilingual interviewers.) A short write-up was prepared to illustrate the highlights of what was done, likely results, and lessons learned.

In order to select projects that would illustrate success factors, lessons learned and best practices, the RDAs were initially asked to provide examples of cases to represent both significant successes and significant failures. The study team then selected the final case study projects from this list, while attempting to have a representative sample according to size and type of project. Note: Although some of the projects suggested encountered certain difficulties during execution, they were generally all perceived as being successful by the RDAs. Projects that would have represented significant failures generally never got started, and therefore would not have been appropriate as a case study. Thus in the end the case studies all represented projects with at least some degree of success.


Footnotes

  1. 1 There are 34 CFDCs, but one (Revelstoke) was ineligible because it had no related job losses. (back to footnote reference 1)
  2. 2 There are 34 CFDCs, but one (Revelstoke) was ineligible because it had no U.S. tariff-related job losses. (back to footnote reference 2)

3.0 Program Profile

3.1 Rationale

SICEAI was created to mitigate adjustment impacts arising from the softwood lumber dispute with the U.S. The $110 million SICEAI program, announced October 8, 2002, was aimed at meeting the medium and long-term economic adjustment and transition needs of forest-dependent communities and the forest industry. The SICEAI's objective was to create long-term, sustainable economic benefits in regions and communities affected by U.S. tariffs on Canadian softwood products. (Note that the program was not intended to directly provide assistance to laid-off or displaced workers, or to directly replace these forestry-related jobs.) SICEAI was originally intended to expire at the end of the 2003–04 fiscal year. Due to delays in initiating the program, no money was released to proponents in 2002–03 and very little in 2003–04, requiring that the program be extended to the end of the 2004/05 fiscal year.

On March 21, 2002, the U.S. Department of Commerce (DOC) announced final subsidy and dumping determinations with respect to Canadian softwood lumber. A countervailing duty rate of 18.79 percent was applied to exports from all provinces except Atlantic Canada. An antidumping duty of 8.43 percent was also applied to Canadian softwood lumber, for a total of 27.22 percent in duties.

As of December 31, 2004, Canadian lumber producers were liable for an average of 20.15 percent in countervailing and antidumping duties. The Canadian industry is heavily dependant on exports and $9.2 billion (or 85 percent of annual shipments) go to the U.S., where they account for about 33 percent of that market. It was expected that production could decline by 8–10 percent (as high-cost producers dropped out) and that Canadian exports would decline by up to 20 percent, or up to $1.8 billion (as of September 2004).

Many affected softwood Canadian lumber producers are in small, forest-dependent communities in rural areas and the duties created a serious threat to their economic base. Mill closures not only resulted in direct job losses but also negatively affected other related parts of the local forest sector (logging, re-manufacturing, etc.) threatening the economic viability of the host communities. By November 4, 2002, the number of duty-related, direct job losses across Canada's sawmills totalled 3,940, with Quebec and British Columbia accounting for the large majority of jobs lost. By September 31, 2004, indefinite direct sawmill layoffs were estimated at about 5,800, of which about 5,300 were in BC, Quebec, and Ontario3, and with BC showing the greatest increase in layoffs since 2002.

Canada pursued a two-track strategy for resolving the softwood lumber dispute: (i) litigation, involving challenging the U.S. duties under the North American Free Trade Agreement (NAFTA), at the World Trade Organization (WTO), and most recently, in the U.S. Court of International Trade (CIT), and (ii) negotiations towards a durable resolution of the dispute.

To mitigate impacts, on October 8, 2002, the federal government announced funding totalling $336.5M to respond to the softwood industry crisis. Of this, the allocations were: $110M for community adjustment (i.e., SICEAI); $71M for worker assistance; $40M over five years for the pine beetle infestation; $25M for industry research and development; $75M for softwood lumber marketing and lobbying in the U.S.; and, $15M towards legal costs of key forestry associations.

top of page

3.2 Eligible projects

SICEAI activities were to be focused on communities within forest dependent economic regions experiencing difficulty in adjusting to loss of employment due to the softwood dispute. It was intended to support forest industry, community stakeholder, aboriginal community and other local entity projects aimed at:

  • Diversification, productivity improvement and new economic opportunities, which would create tangible and sustainable long-term jobs; and
  • Community capacity-building, including developing community economic action plans, strengthening local developmental and leadership capacity, and exploiting opportunities in other local industries for sustainable economic development.

Note that the program was not directed at the specific individuals who may have lost their jobs because of the U.S. tariffs—it was directed towards hard-hit communities. Within Diversification, projects could also access and develop any necessary economic infrastructure such as visitors' centres, utilities, communications, transport, information technologies, marketing, etc., so long as they clearly contributed to community economic development and diversity. Within these projects, direct job creation (i.e., job creation within the proponent organization) was an important criterion, although job maintenance was also taken into consideration. (BC was the most stringent on this point, requiring all projects, including those related to Capacity Building or from First Nations clients, to demonstrate direct job creation—i.e., to follow a business model.)

top of page

3.3 Program contributions

SICEAI was funded federally by Industry Canada (IC), and delivered regionally by Regional Development Agencies (RDAs): Western Economic Diversification Canada (WD), Canada Economic Development for Quebec Regions (CED) and the Federal Economic Development Initiative for Northern Ontario (FedNor), as well as other regional and private sector partners4. SICEAI offered both repayable and non-repayable contributions to projects in the two program categories noted above5. Federal funding could not exceed 50 percent of the total project funding for all projects approved within a region over the life of the program.

top of page

3.4 Program delivery

The Minister of Industry was accountable for the administration of SICEAI resources, and for coordinating, monitoring and reporting on it to Parliament.

Western Economic Diversification Canada (WD), Canada Economic Development for Quebec Regions (CED) and the Federal Economic Development Initiative for Northern Ontario (FedNor) delivered the program to clients. Funds were allocated among British Columbia, Quebec and Ontario on the basis of the job losses in those regions arising from the softwood lumber dispute. Atlantic Canada Opportunities Agency chose not to take part in the program.

The RDAs reviewed all proposed projects in relation to their own sector knowledge, IC's Forest Industry Policy on Investment Incentives, and the program's eligibility criteria.

A unique attribute of SICEAI was the tailoring of the program in each region to the existing delivery networks and delivery capacity of each RDA in its rural areas. Delivery in B.C. was through a third-party delivery model, which differs from the direct delivery model used in Ontario and Quebec. WD delivered non-repayable contributions through an agreement with the Community Futures Development Association (CFDABC) and the individual CFDCs. The repayable projects, which are structured like loans, were delivered directly by WD (and IC has final approval authority for these projects). The non-repayable contributions were delivered through individual CFDCs.

top of page

3.5 Project approval criteria

Project approval criteria were stipulated as follows:

  • Diversification, productivity improvement and new economic opportunities projects must have created tangible and sustainable long-term jobs and benefits in affected regions and have targeted the improvement of the economic base and the investment climate in affected communities.
  • Community capacity building projects must have resulted in the development and implementation of action plans, strengthening the capacity to maximize communities' developmental potential, exploiting opportunities, developing leadership, promoting sustainable development, and providing local qualified human resources for the implementation of action plans.
top of page

3.6 Nature of projects

There were about 540 approved projects in total, which were broken down by region as follows:

  • Ontario—72
  • BC—145
  • Quebec—325.

A summary of funded projects is found in the table below.

SICEAI Projects as of April 30, 2005
  FedNor WD CED-Q Total

* These are "hoped-for" projects (i.e., targets). See also section 6.3. (Return to Table").  
Source: SICEAI Monthly Update, May 27, 2005.

2002–2003
Expenditures 100,000 NIL NIL 100,000
2003–2004
Number Projects Approved 12 122 175 309
Additional investment leveraged $6,606,950 $94,700,000 $82,892,259  
Expenditures $1,972,000 $6,934,557 $15,508,839 $24,415,396
Jobs created or maintained* 24 2,207 2,671 4,902
2004–2005
Number Projects Approved 60 23 150 233
Additional investment leveraged $31,090,875   $32,539,974  
Expenditures $12,513,425 $36,820,181 $17,586,944 $66,920,550
Jobs created or maintained* 675 293 1,551 2,519
Total for SICEAI Program
Number Projects Approved 72 145 325 542
Additional investment leveraged $37,697,825 $95,005,654 $115,046,233 $247,749,712
Expenditures $14,585,425 $43,754,738 $33,095,783 $91,435,946
Jobs created or maintained* 699 2,500 4,222 7,421
SICEAI cost per job       $12,321
top of page

CED funded a larger number of projects than the other two regions, however they tended to be at lower SICEAI contribution levels (<$200K). On the contrary, WD in BC funded a number of very large (>$1M) projects. Figure 2 provides a distribution of projects, by contribution level and by region.

Figure 2: Distribution of Projects by Contribution Level

Distribution of Projects by Contribution Level

Description of Figure 2

In all three regions, the majority of projects were diversification projects outside of the forestry sector. These mostly included tourism and manufacturing-related projects. Capacity building projects represent approximately 11 percent of projects funded in Quebec, approximately 13 percent of projects funded in BC, and approximately 28 percent of projects that were funded in Ontario. However, contribution levels for capacity building projects tend to be lower in comparison to other types of projects (e.g. capacity building projects make up almost 14 percent of total projects, but less than 6 percent of total funding). Figure 3 provides a distribution of projects, by type and by region, Figure 4 illustrates the distribution of funding, by project type and by region, and Figure 5 shows number and proportion of projects by province.

Figure 3: Distribution of Projects by Project Type

Distribution of Projects by Project Type

Description of Figure 3

Figure 4: Distribution of Funding by Project Type

Distribution of Funding by Project Type

Description of Figure 4

Figure 5a: Distribution of Projects by Province
  Ontario Quebec C.-B.
Distribution of Projects 72 325 145
Figure 5b: Distribution of SICEAI Funding, by Province
  Ontario Quebec C.-B.
Distribution of SICEAI Funding 16 % 40 % 44 %

3 Softwood Lumber Monitoring Report, Manufacturing Industries Branch, Industry Canada. October 14, 04. (Return to Reference 3).

4 The Atlantic Canada Opportunities Agency (ACOA) was eligible to be part of SICEAI, but opted out. (Return to Reference 4).

5 An exception was obtained to the Treasury Board Transfer Payment Policy (which restricts non-repayable contributions to private companies to a maximum of $100,000) there by enabling CED to raise this level to $200,000 without Treasury Board approval. (Return to Reference 5).

4.0 Findings on Program Relevance

4.1 Appropriateness of the program

Evaluation questions

  1. Was SICEAI an appropriate response to the needs at the time?
  2. Have the needs changed? If so, how?

Summary

  • As part of a larger government initiative, the SICEAI program was very appropriate in that it provided the resources required in order respond quickly to the layoffs in the forestry industry, and to start planning and implementing community adjustment measures.
  • Although certain factors (e.g., affected communities) and specific project funding requirements changed over the lifetime of the program, the overall needs addressed by the program did not change.

Discussion

RDA perspective. Forest-dependent communities were being affected by their key industry being hit. SICEAI officials at IC, and within the RDAs felt that the government needed to:

  • develop a quick response to the layoffs (an up-front plan)
  • support the communities and forest industry (crisis management)
  • start to plan for adjustment measures (hopefully for communities to be able to diversify)
  • find other opportunities for people (replace jobs lost and see new sources of future growth)

From that standpoint, and as part of a larger government initiative (which also strengthened trade negotiations and forestry-related research), the SICEAI program was very appropriate in that it provided the resources to do this. Also, the final Terms & Conditions were sufficiently flexible to allow a variety of relevant activities to be supported.

Client perspective. The survey data support the view that the program was an appropriate response to the needs of the communities.

Appropriateness of Program to Meet Client Needs
 
Very inappropriate
2
Neutral
4
Very Appropriate
Don't Know N
Source: Survey of project proponents (clients). Table may not add to 100% due to rounding errors.
BC 13% 3% 13% 23% 48% 3% 40
ON 10% 7% 7% 17% 52% 7% 29
QC 19% 4% 2% 15% 60% - 47
Region unknown - - - - 100% - 3
Total 14% 4% 7% 18% 55% 3% 119
top of page

Although certain factors changed over the lifetime of the program, the overall needs did not change:

Did Client Needs Change?
  Yes No N
Source: Survey of project proponents (clients). Table may not add to 100% due to rounding errors.
BC 21% 79% 39
ON 14% 86% 29
QC 48% 52% 44
Region unknown - 100% 3
Total 29% 71% 115
top of page

There was always the need for more (and different) jobs, and more community adjustment, although different communities suffered layoffs, and there were continuously new communities in need. Clients reported that over time, specific project requirements might have changed (e.g., with respect to funding). In other words, this changed specific project needs as opposed to overall program needs: as long as the dispute was not resolved, the original need remained. In fact, the situation is still ongoing and impacts are still manifesting themselves. There continue to be layoffs and companies continue to struggle. If anything, because of the integrated nature of forest industries the situation in many communities has worsened.

However, program officials brought up certain limitations, with respect to program relevance:

More help required for larger sawmill companies. Some FedNor and CED officials noted that because of its focus on diversification, the program was not designed to help the larger sawmill companies, where many of the layoffs were occurring. This was primarily due to the fact that funding to such companies might have resembled subsidies, and would have conflicted with international trade laws, and indeed might have reinforced the U.S. rationale for the tariffs in the first place. Moreover, public investments in sawmilling capacity are prohibited under Canada's Forest Sector Investment Policy. (In BC some support was available to small and medium sized private firms that had operations in the sawmill sector but SICEAI funds were directed exclusively towards investments in new high value-added wood product processing operations (e.g. engineered wood products) as these projects represented true diversificationFootnote 6). Still, some officials hoped that any new program might better provide assistance to the companies that were hardest hit by the dispute.

Jobs created could not directly replace jobs lost. Another limitation of the program was that the jobs created as part of the program were frequently lower paying jobs in sectors such as tourism, and could not directly replace unionized, higher-paying jobs in the forestry sector. Further, the individuals employed in the new jobs were often not those who had lost their jobs through the tariffs. Communities placed the greatest importance on diversification projects that offered to replace lost forestry jobs within the community as these generally offer the highest wages and the potential for skills transferability of locally displaced workers.

Need for expanded scope. Many officials noted that the trigger that created the program (i.e., layoffs as a result of the U.S. duties) was too narrow. Many communities are forest-dependent, not necessarily only those having lumber mills. Many other inter-connected industries (for example, pulp and paper or wood harvesting) were also affected by the softwood lumber dispute. The communities that housed these industries also needed to diversify, although they were not eligible in the program.

Program too short term. Many officials and clients alike pointed out that the program was too short-term—true diversification takes more than 2–3 years. The program was also slow in terms of implementation, which added to the problem since projects started late, but had to finish spending all program funds within a very short period. This caused substantial difficulties for proponents who in many cases had to complete a detailed environmental impact assessment (and have it reviewed by the RDA), and then initiate construction projects during winter weather conditions in order to have spent their funds within the very short time frame. The short time frame for the program also meant that RDAs, which had built up expertise in the needs of individual communities, lost that expertise when the program terminated. (Most of the BC CFDCs, for example, hired coordination and/or outreach officers to interact directly with communities to identify projects, assisted by the two Quality Assurance Review Officers hired by the programFootnote 7. Most of these individuals are now long gone.)

top of page

4.2 Alternatives to SICEAI

Evaluation questions

  1. What were the alternatives to the SICEAI in attempting to meet the stated objectives?

Summary

At the time the program was announced, there were no reasonable alternatives to SICEAI.

The community-based delivery model was appropriate, because the RDAs and CFDCs could best respond to the needs of their communities, while allowing Industry Canada to ensure consistency between regions and to remain accountable for the program overall.

Many of the projects that were funded through SICEAI in Ontario and Quebec could have been funded through regular Agency programming, but the program helped expand the scope of what could be funded. (This was however not the case in BC where alternative core funding was not available as a possible replacement to SICEAI funding.)

Discussion

RDA and FedNor perspective. The majority of Program Officials interviewed and 100 percent of surveyed CFDC representatives in BC noted that, at the time the program was announced, there were no reasonable alternatives to SICEAI (i.e., no other programs or sources of funding that could have adequately addressed the needs of the communities). Sixty-nine percent of clients noted that there were no reasonable alternatives to funding their project, while about 30 percent noted that there were other sources of support available, including:

  • Other programs of the CFDCsFootnote 8, FedNor, and CED, although these programs had limitations that made many projects ineligible (e.g., FedNor is currently restricted to helping not-for profit organizations and SMEs);
  • Federal/provincial infrastructure programs (for which the municipality would have had to be the applicant)
  • Financial institutions.

However, virtually every respondent noted that these sources were more restrictive in the nature of projects they would support, harder to access, more costly to use, or would have required downsizing of the project.

Assessment of the Community-based Delivery Model. Most respondents were supportive of the basic model in which the program was "led" by Industry Canada and delivered through the regional agencies. The RDAs and CFDCs are very familiar with the needs of their communities, and maintain a good dialog with community groups in order to stay informed on potential projects. This is evidenced in the case study projects. For at least half of the case studies, the local delivery organization (whether it be FedNor, CED or the CFDCs) was described by the client as being instrumental in the success of the project (e.g., the delivery organization was responsible for bringing the various partners together, in preparing the application, or in providing critical advice). As well, almost all proponents for the case study projects noted that the level of service and support they received was excellent or very good.

The FedNor project officers are very responsive to the proponents' needs. It is much easier to make a case to someone local, rather than to deal with somebody long-distance. FedNor has people out in the field, who know the realities of certain regions.

At the same time, it was seen as important for Industry Canada to maintain accountability for public funds (in this case, $110M) and to ensure consistency between regions. This is especially true when delivering a national program, which addresses a problem that is not unique to a particular region.

However, some regional representatives either did not fully understand Industry Canada's role, or suggested that the SICEAI model was needlessly complex and inefficientFootnote 9. They believed that if Industry Canada gives the mandate of running the program to the agencies, the RDAs should also be given funding directly, and have more responsibility and accountability. Several respondents suggested following earlier approaches, in which federal resources were delivered directly to the RDAs, which could then use their own Terms and Conditions (T&Cs) to implement appropriate regional programs. The BC Fisheries Adjustment Programs10 used this basic model and a similar mining adjustment program was previously used in Ontario. In that case, FedNor had received resources directly for an adjustment initiative related to a downturn in the mining industry. The T&Cs of the main FedNor program are very flexible, and the adjustment initiative could be implemented through FedNor without designing a new program.

[The study team notes that an evaluation of the third-party delivery structure used for the BC Fisheries Adjustment Programs, which expended roughly $45M in federal funds, was very positive, and uncovered no evidence of poor accountabilityFootnote 11. Further, a roll-up evaluation of the Community Futures programs delivered by WD, FedNor, CED, and the Atlantic Canada Opportunities Agency was also very positive, including that the delivery and governance structures were very effectiveFootnote 12. Finally, WD also conducted both an audit of claims, which uncovered no problems, and an independent audit of the BC program's review process which concluded that the process resulted in a consistent, efficient, and effective utilization of the SICEAI fundsFootnote 13.]

Client perspective. Projects supported by this program had moderate incrementality, as shown by client responses in the table below.

Client perspective
If SICEAI had not provided funding BC ON QC Region unknown Total N
Source: Survey of proponents (clients). Table may not add to 100%, as multiple responses were possible.
No real change—Project(s) would have gone ahead anyway, with similar funding obtained from other sources 5% - 2% - 3% 3
Project(s) would have gone ahead, but with different size or scope 30% 28% 21% - 25% 28
Project(s) would have gone ahead, but after a delay 22% 21% 17% - 19% 21
Project(s) probably would not have happened at all 43% 62% 52% 100% 53% 59
Other (e.g., cash flow problems,) 14%   12%   9% 10

Incrementality was taken into account by all RDAs in their review of project applications, and clients stated that only 3 percent of projects would have gone ahead completely unchanged without SICEAI support. Many clients commented that SICEAI funding "kick-started" the project (for example, by making it easier to decide to take on the project, or to find other investors), or they had already approached other possible sources of funding without success (or finding too many restrictions), or that they could not take on all the risk on their own, or that the program's support allowed the project to begin earning revenues earlier. Several commented that they would be bankrupt by now without program support.

While it is clear that a number of SICEAI projects would likely have gone ahead in some modified form or timeline, this is not entirely unexpected, as a project that fully met SICEAI's criteria of being a reasonable "business proposition" would likely meet similar criteria of other funding sources.

For comparative purposes, if we call projects "incremental" if they either would not have gone ahead at all, or only after a delay, then about 72 percent of SICEAI projects are "incremental". This compares quite well with the Community Economic Adjustment Initiative (CEAI) program within the BC Fisheries Adjustment programs, in which 75 percent of projects were considered incremental using a similar criterionFootnote 14.

Similar moderately high incrementality was found in the case studies. Of the 15 case study projects, six were found to be totally incremental (that is, the project would not have proceeded in the absence or SICEAI) and six of the projects would likely have proceeded, but only after a delay or with a reduced scope (partially incremental). As one proponent noted,

Because we are a young company, we have limited borrowing power. SICEAI funding allowed the company to expand our operations sooner than we otherwise would have.

Three of the case studies likely would have gone ahead with other funding sources (for two of these projects, FedNor likely would have funded the project regardless).

Many FedNor and CED officials noted that the projects that were funded through SICEAI often could have been funded through regular Agency programming, but the program helped expand the scope of what could be funded. In fact, both FedNor and CED have fairly broad Terms & Conditions, and those for SICEAI were found to be very similar. This is especially true of capacity building projects, where the Agencies have historically done a lot of work. However, SICEAI was able to fund larger projects, and was able to provide more funding to the private sector (e.g., it could get involved with the purchase of capital equipment, etc.). The program was also more generous in helping expanding and start-up businesses and (contrary to normal FedNor practice) SICEAI had a provision to deal with larger private sector firmsFootnote 15. In BC, the CFDCs have relatively small loan funds, and can only provide repayable contributions to private sector companies up to $125,000. The SICEAI program was able to support a number of very large projects involving repayable contributions made to private sector companies (25 BC projects had total projec t costs of $1M or more, of which eight had SICEAI contributions of $1M or more). The CFDCs could not have supported repayable projects of this magnitude, not to mention projects to non-profit organizations.

In addition, both FedNor and CED officials also noted that there was not enough funding available under their regular budget to assist businesses to the extent that they did under SICEAI. However, if they had been provided with a similar amount of additional resources, and had been able to "tweak" the program slightly to be able to get around the private sector issue, direct delivery through the RDAs would have been a good alternative mechanism. On this matter, WD and CFDABC agreed, noting that the Fisheries Adjustment Programs referred to earlier had supported very similar economic adjustment projects for fisheries-dependent communities, only having CFDCs directly making both the repayable and non-repayable contributions through a dedicated loan trust. This model was seen as far less bureaucratic, cost-efficient and highly responsive to community needs while still fully capable of meeting all of government's accountability requirements.


Footnotes

  1. 6 The tariffs are quite complex and the rules changed from time to time. To greatly simplify, relatively "raw" softwood products from large companies such as studs, heavy timbers, siding, decking, interior panelling, boards, flooring, and the like are subject to tariffs. The higher up the "value chain" one goes, or the smaller the firm involved, the less likely that tariffs will be imposed. Thus products such as box spring frame kits, truss kits, edge-glued wood, and such are generally not subject to tariffs. Reviews were done by the RDAs to ensure that any value-added SICEAI products would not be subject to tariffs. Projects had to clearly represent something not subject to the tariffs (e.g., modernization of sawmill capacity using traditional technologies was clearly ineligible under SICEAI). (back to footnote reference 6)
  2. 7 Three were originally hired, but one passed away and was not replaced. (back to footnote reference 7)
  3. 8 Clients may have been mistaken on this point. CFDCs do not have access to programs that could support the types of nonprofit projects that SICEAI funded, and the very small internal loan funds of the CFDCs can only support projects to a maximum of $125,000, far lower than some SICEAI project figures. (back to footnote reference 8)
  4. 9 Within the RDAs, handling program funds through the Other Government Department (OGD) suspense account created difficulties in establishing spending authorities, reporting, and budget estimations. (back to footnote reference 9)
  5. 10 There were actually four types of BC coastal fisheries adjustment programs, namely (1) the Fisheries Legacy Trust Loan (FLTL) Fund, which was operated by coastal CFDCs through their non-profit Fisheries Legacy Trust (FLT); (2) the Community Economic Adjustment Initiative (CEAI) program (which was most like SICEAI in its goals and decision-making, although funding came through the FLTL Fund); (3) the Recreational Fishery Loan Program, which was to help recreational fishing lodges and charter operations; and (4) various outreach and coordination activities. (back to footnote reference 10)
  6. 11 Western Economic Diversification & Pacific Fisheries Adjustment, an Evaluation. GSGislason & Associates. January, 2002. (back to footnote reference 11)
  7. 12 Roll-Up of CFP Formative Evaluation Reports from ACOA, CED, WD and FedNor—Revised Final Report. Performance Management Network. September 19, 2003. (back to footnote reference 12)
  8. 13 Western Economic Diversification Canada Review of the Application Process for the Softwood Industry Community Economic Adjustment Initiative, PriceWaterhouseCoopers. February 9, 2005. (back to footnote reference 13)
  9. 14 GSGislason & Associates. January, 2002, p. 54. Op. cit . This report estimated what proportion of projects would not have gone ahead within two years, as well as those that would not have gone ahead at all. (back to footnote reference 14)
  10. 15 FedNor is currently restricted to helping not-for profit organizations and SMEs (fewer than 250 employees, and less than $20M in sales). (back to footnote reference 15)

5.0 Findings on Program Design and Delivery

5.1 Program clarity

Evaluation Question

  1. Are the objectives and desired outcomes of the SICEAI clearly identified and agreed upon by the various stakeholders?

Summary

  • The roles, responsibilities and respective spending authorities of SICEAI were unclear to all concerned early in the program's lifetime, and were one cause of the delay in implementing the program.
  • These difficulties were eventually resolved and the program was rolled out quite smoothly.

Discussion

Generally, the objectives and the outcomes of the program were clear once the program's T&Cs were finalized. However, this was far from the case early on, and both RDAs and clients were unclear as to overall goals and objectives, eligibility of projects and proponents, and the criteria and processes that would be used to select projects. This was particularly true in BC, which suffered from conflicting communications from various sources, including confusion about roles and responsibilities.

Some CED and WD officials indicated that the name of the program led to some confusion. Because the program title contained the words "Economic" and "Industry", some clients thought it was aimed to help the softwood industry, and/or the specific individuals who had lost their jobs because of the U.S. tariffs. In fact, the aim of the SICEAI in particular (as one component of a broader federal support package) was to help the affected forest dependent communities, not necessarily the affected companies and their employees suffering from job losses.

top of page

5.2 Project review and selection

Evaluation Question

  1. Were eligibility criteria appropriate, and properly applied by the SICEAI?

Summary

  • A review of the Terms and Conditions indicates that the program eligibility criteria were appropriate and, in general, appropriately applied to project applications.
  • All three regions were very flexible in terms of the nature of projects supported, so long as they clearly fit the intent of the program.
  • However, the program's detailed objectives, rules, and operating procedures were very unclear to all concerned at the beginning of the program, and communications on these topics were often confusing and conflicting (in particular owning to the appearance of a focus on the "forest industry" whereas in fact the program actually targeted "communities" and not "companies")

Discussion

Quebec and Ontario. For FedNor and CED, project selection was not a competitive process, and it followed the same process as for regular project funding. Field officers begin by looking at the projects that were in their "pipeline", and by determining which projects could be completed within the required timeframe. Once they took on a case, they would work with clients from the start to develop project proposals and provide continuous feedback and recommendations. If a project was not likely to be supported, clients were given an indication early on, and were given the opportunity to improve their proposals. Project Officers also encouraged partnership and collaboration between groups interested in working on similar projects (i.e. collaboration between municipalities, or with industry). Once the file was prepared, it was presented to the Regional Manager (FedNor) or Regional Director (CED), who reviewed it and signed off a recommendation. In the case of FedNor, it then went to the Director General of FedNor. If a proposal was under $500,000, she could have the final approval; if it was over $500,000, the final approval had to go to Ottawa. In the case of Quebec, if a proposal was under $100,000, only the Regional Director's approval was required. If a proposal was between $100,000 and $199,000, the final approval had to go to the Operations ADM of CED. A proposal over $200,000 had to go to the Minister.

The eligibility criteria are outlined in the terms and conditions of the program, and include job creation and/or building community capacity. FedNor officials reported that, similarly to their other projects, officials also looked for incrementality and cost sharing wherever possible, either with the proponent or with other government bodies. They also sought to avoid long-term dependency situations. The due diligence process was seen by FedNor and CED program officials as being effective.

British Columbia. In BC, the process was a competitive one, with about 1,100 letters of intent ("stage one" applications) being received. There was a thorough and rigorous review process, which varied according to the complexity of the project. The process (simplified) was as follows16:

SICEAI applications underwent a local (community-based) review that started with a two-stage assessment of eligibility by a CFDC. The CFDABC worked with each CFDC to make decisions regarding project approvals. Proponents were asked to provide information to the CFDC including data, plans, anticipated impacts, schedules and reports to enable the determination of eligibility. Initial and detailed project assessments led to a completed Due Diligence Report (DDR), which was used to monitor the progress of the project, to complete monitoring of the final project, and to evaluate the effectiveness and benefits of the project. The assessment and approval process was as follows.

  1. Stage I—Preliminary review of "Expressions of Interest" against basic eligibility criteria. This includes verifying community support, gauging environmental implications, reviewing the structure of the funding profile and confirming the potential for sustainable economic benefits to the local economy. Eligible projects that passed this initial review then moved to a detailed stage-two review.

  2. Stage II—Detailed Review of Full Project Plan for non-profit organizations, or of a full business plan for commercial for-profit ventures, was done by the CFDC. Note that Stage II applications were reviewed on a "business case" basis—i.e., using similar criteria that the CFDCs would use if thinking of providing a loan to a new small business or WD would apply to any enterprise development project with a non-profit community group or industry association. More than 400 projects moved to this detailed assessment stage. At this stage, the CFDC conducted a detailed due-diligence assessment of a full fledged plan including an assessment of the impact of layoffs on the community, a detailed review of project budgets and eligible costs, a review of the project to determine that it qualifies for a non-repayable contribution, a review for Canadian Environmental Assessment Act requirements, a review of expected long-term sustainable economic benefits (especially job creation impacts17) and an assessment of risks associated with the project. The assessment includes consideration of SICEAI program eligibility criteria and strategic program goals, and expected results and outcomes in the form of a detailed project assessment. Projects meeting these criteria moved to an independent quality assurance review.
  3. Stage III—Quality Assurance Review. The CFDABC and the Regional CFDC managers examined all recommendations for funding from each CFDC to ensure consistency with SICEAI program criteria and strategic goals. This process was a collaborative exercise that involved the CFDABC's Quality Review Officers and the CFDC regional managers (one for each of the eight geographic regions in BC) who formed an independent review panel comprised of these community economic development (CED) experts. The quality assurance review ensured completeness and consistency of the assessment and consistency in decisions and recommendations across CFDCs, as well as consistency in the contribution agreements with the ultimate recipients.
  4. In the case of repayable projects, there was an additional final approval by Industry Canada (that served to endorse the local decision processes) prior to WD moving to put into place a funding contract (contribution agreement) with the proponent.

    The Ministers' Advisory Group (MAG) was responsible for providing advice and making recommendations to Ministers on strategic goals and targets for the program. Bundles of projects were reviewed by the MAG, which made observations about projects approved by the CFDCs relative to the achievement of strategic goals and targets for the program, such as sectoral and geographic distribution, Aboriginal participation, and need to offer tangible and sustainable economic benefits. However, the MAG did not make formal funding decisions on specific SICEAI projects.

In BC, all diversification applications (including those from non-profits) had to demonstrate direct job creation and/or maintenance; e.g., applications for "core funding", strategic planning studies, or studies to develop economic development options, were not eligible. (Commercially-sensitive repayable contributions were confidentially reviewed by the local CFDC's loan committee.)

The level at which authority for funding approval depended upon whether the project was repayable or nonrepayable (non-repayables were delivered through a contribution agreement between the CFDC and the proponent), while repayables were submitted to IC, approved, and then WD engaged in a contribution agreement directly with the client. These two parallel streams and different responsibilities for delivering projects were frequently confusing to clients especially once projects were referred to the repayment and collection phases.

Client perspective. Clients indicated that most aspects of the project application and review process were seen as good or very good, although the roughly 20–25 percent who rated these as "neutral" to "very poor" frequently commented on the shifting and conflicting messages they received about these topics. Many commented that their RDA office had been very helpful in interpreting the criteria and helping proponents prepare a successful application and helping them through the often long and arduous process. A small number of clients also commented that there appeared to be some "favoured friends" of the program who received preferential (and large) funding, but within the scope of this study it is impossible to say whether, or how often, this may have occurred.

Project review and selection process overall
  1  
Very poor
2 3  
Neutral
4 5  
Very good
Don't Know N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 5% 15% 10% 28% 38% 3% 39
ON   7% 7% 17% 66% 3% 29
QC     15% 32% 51% 2% 47
Region unknown       33% 67%   3
Total 2% 7% 11% 27% 51% 3% 118
top of page
Communication & Understanding of Review Criteria
  1  
Very poor
2 3  
Neutral
4 5  
Very good
Don't Know N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 13% 15% 8% 33% 28% 3% 39
ON   7% 10% 17% 62% 3% 29
QC   4% 13% 32% 49% 2% 47
Region unknown       33% 67%   3
Total 4% 8% 10% 29% 46% 3% 118
top of page
Appropriateness of the Eligibility Criteria
  1  
Very poor
2 3  
Neutral
4 5  
Very good
Don't Know N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 8% 10% 13% 41% 26% 3% 39
ON   3% 14% 17% 62% 3% 29
QC   2% 23% 32% 40% 2% 47
Region unknown       67% 33%   3
Total 3% 5% 17% 32% 41% 3% 118

The majority of clients interviewed as part of the case studies reported that the application process was appropriate and timely18, although some noted that there was a lot of paperwork involved (in one case, the project required three different submissions before the project was approved). In addition, almost all case study clients specifically praised the local delivery organizations, with respect to the support they provided during the process. As one client noted, "The application process was incredibly complicated. We couldn't have delivered our submission without local staff support."

top of page

5.3 Appropriate targets

Evaluation Question

  1. To what extent were realistic targets/performance measures established?

Summary

  • There were three types of marketing and outreach targets set: (1) Towards communities—those that had suffered significant impacts because of the tariffs; (2) On potential clients and projects—especially forestry companies that were SMEs, First Nations communities, and/or projects were value-added; and (3) Strategic tar gets—for example, for outreach to specific regions, or to attempt to generate interest for projects in certain sectors (e.g., tourism). Targeting on First Nations communities or SMEs was especially done in BC, based on internal WD strategies.
  • However, there were no rigid targets set on the allocation of contributions—project selection was done in an entirely "bottom-up" manner based on the eligibility and quality of the project application. Program targets overall (e.g., jobs by sector, or by region) were not set by IC or by the regions, although an attempt was made in BC to encourage clients to propose projects which fit provincial sector priorities.
  • Targets for job creation and/or maintenance, timelines, deliverables, and matching funding were set for each individual project. These were set in conjunction with proponents, and were generally seen as reasonable. However, the very short timeframe available for project implementation and completion caused problems for many clients.

Discussion

General. Although no specific targets were set for the regions by IC, the objective was to create and maintain as many jobs as possible in the affected areas. Because of the many communities affected, the intent of the program was usually to fund a large number of small projects, rather than commit all the money to only a few larger projects. This was definitely the case in all three provinces. Although there were a few large projects in BC, these were only approved if it could be argued that the impacts would affect a large region of the province and many communities (e.g., in the Prince Rupert container port expansion project, it was estimated that impacts would extend as far as Terrace and affect the local region's transport cluster).

Quebec. CED project planning, monitoring, and automated systems were designed to set and track the specific number of direct jobs created and maintained for each project. These statistics were then available to report back to key stakeholders within and outside the government. In some regions of Quebec, projects were limited to the immediate communities where sawmills had suffered layoffs (this was a complaint mentioned by some CED officials). However, this was not a set rule, and it appears that most regions approved projects in surrounding communities where workers of the sawmills lived.

For CED, each of the agreements (when written up with letters of offers) had a set of deliverables with specific expected outcomes. The field officer was responsible for working with the proponent to make sure these deliverables were achieved, and that reporting on them during and after the project was completed.

Ontario. FedNor exercised as much flexibility as possible in terms of which projects could be funded. They allowed for a broader interpretation of the criteria, providing support to all communities that were deemed to be forestry-dependent because they felt that all these communities were at risk, even if they did not have a sawmill.

For FedNor, each of the agreements (when written up with letters of offers) had a set of deliverables with specific expected outcomes. The field officer was responsible for working with the proponent to make sure these deliverables were achieved, and that reporting on them during and after the project was completed. In Northern Ontario, officials held broad discussions when the program was first introduced to see what the possibilities would be, which communities were most affected, which should be targeted, and how they could best approach the private sector. They generally tried to support value-added projects and aboriginal communities, and tried to shy away from open-ended capacity-building projects, because those projects were normally funded under FedNor regardless (i.e., the SICEAI program was not really needed).

British Columbia. In BC, the Minister's Advisory Group (MAG) at first set targets by sector (e.g., x percent of funding for forest sector, tourism, film-related projects) which were applied as a guide to the CFDC outreach, promotion and marketing activities in their local communities. Job creation and/or maintenance and milestone completion targets for individual projects were set by the project proponents jointly with their CFDCs. Monitoring was done by the CFDCs and reported to WD. CFDCs became involved in any cases where important milestones were not met, and therefore milestone payments were affected. The projects targets, as mentioned earlier, always involved direct job creation and/or maintenance. In some larger projects, regional targets that crossed the boundaries of several CFDCs had been set. The Quality Assurance Review Officers considered that about 90 percent of the individual project targets were reasonable.

BC was quite flexible in terms of the nature of projects supported. There was a policy to not normally fund certain types of projects because they usually did not meet the intent of the program; e.g., airport (stand-alone) infrastructure improvements such as new runways were usually deemed ineligible as they did not meet some of the program criteria, such as "direct" creation of sustainable new jobs19. However, even these "rules" were flexibly applied as appropriate. The Prince George airport expansion project, for example (which put Canadian customs and inspection agencies on-site for direct access to international tourism and cargo customers, rather than having to go through Vancouver) was deemed eligible, as the airport would be able to better support local tourism, the film industry, etc., all of which provide sustainable, diversified employment. Within this context, BC funded some 20 Capacity Building projects, and only to a cap of $50,000 each, as it was believed it was difficult to meet the diversification and sustainable employment goals of the program. (Many of the capacity building projects were in First Nations communities, and it is thought that it will take several years to know how successful they will be.)

Client perspective. Roughly 80 percent of project proponents found the targets set for their individual projects with respect to job creation and/or maintenance, milestones, timeframes, deliverables, and matching funding were realistic or very realistic, as shown in the tables below.

Nonetheless, a number of clients commented on the very short timeframe given them for completing complex construction projects in inclement conditions, and the difficulty in obtaining matching funding. (Projects, which did not succeed in obtaining matching funding did not, in fact, receive SICEAI funding and did not proceed. Most projects, which were considered "failures" by the RDAs were in this category.)

With respect to the case studies, eight of the 15 projects reported difficulties associated with tight project or program timelines. In three of these cases, deadline extensions were required.

The program's tight timelines offered very little flexibility. Because the project had to be completed by March 31, 2004 we were required to complete construction over the winter, which led to additional expenses.

However, none of the case study project proponents complained of difficulties in obtaining matching funding.

Appropriateness of targets for job creation and/or maintenance
  1  
Very unrealistic
2 3  
Neutral
4 5  
Very realistic
Don't Know N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 5% 3% 15% 44% 28% 5% 39
ON   3% 21% 28% 41% 7% 29
QC     13% 24% 62%   45
Region unknown       33% 67%   3
Total 2% 2% 16% 32% 46% 3% 116
top of page
Appropriateness of targets for project milestones, timeframes and deliverables
  1  
Very unrealistic
2 3  
Neutral
4 5  
Very realistic
Don't Know N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 11% 5% 16% 39% 29%   38
ON     3% 45% 52%   29
QC   4% 16% 38% 42%   45
Region unknown       33% 67%   3
Total 3% 3% 12% 40% 41%   115
top of page
Appropriateness of targets for matching funding
  1  
Very unrealistic
2 3  
Neutral
4 5  
Very realistic
Don't Know N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 5% 15% 13% 21% 46%   39
ON   4% 11% 36% 46% 4% 28
QC   2% 11% 31% 56%   45
Region unknown       33% 67%   3
Total 2% 7% 11% 29% 50% 1% 115
top of page

5.4 Client satisfaction

Evaluation Question

  1. How satisfied are clients; e.g., program benefits; accessibility; fairness; ability to respond to client needs; response time; guidance provided during the process; quality of service delivery

Summary

  • Clients were generally satisfied or very satisfied with all aspects of service delivery, especially from regional offices, and very few complaints were noted.

Discussion

RDA perspective. Because the project officers continue to work closely within the communities even now that the program has ended, they often receive feedback from clients. Officials noted that clients generally seem satisfied with the outcome of their projects. There are isolated cases where members of the private sector were unhappy with repayable contributions, and felt that they would be better off getting a bank loan. The timeframe might also have been an issue with some clients, who often were hard-pressed to get the project done within the allocated timeframe. It is worth noting that in BC, where only 145 projects were funded out of 1,100 letters of intent, there were a negligible number of complaints from the applicants who were not selected. The only substantive complaints made to BC program officials were about the extensive review process (especially the environmental assessments, or EAs), the time lag to obtaining funding, and the short period in which projects had to be completed.

Client perspective. Clients were very positive about the services offered by CED, FedNor, and the CFDCs, as seen in the tables on the following pages showing results from the client survey. Still, a few clients complained about paperwork and reporting.

Almost all clients interviewed as part of the case studies were also extremely pleased with the quality of service delivered through the program. In fact, in all three regions many clients noted that the local delivery organization went above and beyond what they would have expected, and they specifically praised certain members of the local staff as being extremely knowledgeable and supportive.

The economic development officer that was assigned to the project was very good. In fact, the project likely never would have happened without his support. It's too bad he's no longer there; I could still use his help.

The CFDC staff had a good understanding of the political climate and provided very good advice. We could not have put together the submission without local staff support. They were phenomenal.

The application process was incredibly complicated. We couldn't have delivered our submission without local staff support.

Overall quality of service delivery
  1  
Very dissatisfied
2 3  
Neutral
4 5  
Very satisfied
N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 8% 10% 5% 33% 44% 39
ON - 3% 3% 31% 62% 29
QC - - 25% 25% 50% 4
Region unknown - - - 33% 67% 3
Total 4% 7% 5% 32% 52% 75
top of page
Amount of funding from SICEAI
  1  
Very dissatisfied
2 3  
Neutral
4 5  
Very satisfied
N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC   8% 13% 26% 53% 38
ON 3%   3% 21% 72% 29
QC   25%     75% 4
Region unknown       33% 67% 3
Total

1%

5%

8%

23%

62%

74

top of page
Accessibility of services
  1  
Very dissatisfied
2 3  
Neutral
4 5  
Very satisfied
N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 8% 8% 11% 37% 37% 38
ON   3% 10% 31% 55% 29
QC     25% 25% 50% 4
Region unknown       33% 67% 3
Total 4% 5% 11% 34% 46% 74
top of page
Fairness
  1  
Very dissatisfied
2 3  
Neutral
4 5  
Very satisfied
N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 5% 16% 8% 29% 42% 38
ON     3% 31% 66% 29
QC     25% 25% 50% 4
Region unknown - - - 33% 67% 3
Total 3% 8% 7% 30% 53% 74
top of page
Ability to respond to client needs
  1  
Very dissatisfied
2 3  
Neutral
4 5  
Very satisfied
N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 11% 5% 16% 39% 29% 38
ON   4% 4% 25% 68% 28
QC     25% 25% 50% 4
Region unknown       33% 67% 3
Total 5% 4% 11% 33% 47% 73
top of page
Guidance and support provided
  1  
Very dissatisfied
2 3  
Neutral
4 5  
Very satisfied
N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 5% 5% 11% 32% 47% 38
ON 3%   7% 14% 76% 29
QC   25%   25% 50% 4
Region unknown       50% 50% 2
Total 4% 4% 8% 25% 59% 73
top of page
Project monitoring and supervision
  1  
Very dissatisfied
2 3  
Neutral
4 5  
Very satisfied
N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 5% 5% 11% 32% 47% 38
ON 3%   7% 14% 76% 29
QC   25%   25% 50% 4
Region unknown       50% 50% 2
Total 4% 4% 8% 25% 59% 73
top of page

5.5 Reach

Evaluation questions

  1. To what extent did the RDAs/CFDABC/CFDCs reach their intended target groups?
  2. To what extent did RDAs/CFDABC/CFDCs provide appropriate geographic coverage and equitable distribution of funding?

Summary

  • The program did make every effort to target communities affected by the tariffs, although in practice not all were served.
  • With the exception of emphasis placed on supporting SMEs with value-added wood business opportunities, the program did not attempt to direct program funding to specific industries (and the strategic targets set in BC), or to specific regions, except in the sense that projects had to be consistent with program goals.
  • Some aspects of eligibility criteria reduced the program's ability to serve needy communities; e.g., if the timing of layoffs didn't quite match the program's application period, or if communities did not suffer "direct" sawmill job losses and the requirement for a minimum 10 percent private sector funding against any project often impeded the eligibility of projects led by local municipalities.

Discussion

Some program officials believed that because of certain eligibility criteria (i.e., the need for permanent layoffs, the definition of softwood lumber, exclusions from the taxation/duties), some regions were excluded from the program, or were placed at a disadvantage.

In Ontario, there was no deliberate distribution of projects. FedNor did not purposely favour larger communities to the detriment of smaller communities, although the geography did make it difficult to justify funding large projects in small remote communities. Overall, the project officers tried to be reasonable with the projects that were selected.

Likewise in Quebec, project distribution between regions was pro-rated based on the number of jobs lost, although officials noted that the distribution of projects is also largely based on the regional office's ability to mobilize its region, to find projects that meet the criteria, and to allocate its budget.

This was also true in BC, where WD, CFDABC, and the CFDCs conducted an initial "outreach" survey to all their communities to ascertain needs and inform them of the program, with some special emphasis on First Nations communities as well. Thus "targeting" was done in the sense of knowing community impacts and needs (and thus knowing which CFDCs should put extra effort into "lighting a fire" in the community and providing assistance to proponents), rather than through any internal priorities of the agencies—in all other respects, the project application and review process was responsive, rather than directed. WD noted that about 180 communities across the province suffered job losses because of the tariffs, and the program assisted about 140 of these communities. Furthermore, WD estimated that roughly 100 communities (or 60 percent) of BC communities were "hard-hit" (i.e., the job losses were significant given the size of the community), and the program directed support to 90 of these. A small number of BC clients complained that the relatively large number of BC projects with large budgets left less for the "little guys". It should be said, however, that all the large projects were funded only on the condition that they had a broad geographic impact, and that there was substantial support for each large project from these outlying communities.


16 This is a short description of a complex process. Details of eligibility criteria, proposal review methods, instructions to applicants and reviewers, etc., along with appropriate form, are found in SICEAI Program Guidelines for the Community Futures Development Corporations of British Columbia, November 2003. There were several supporting documents, such as one describing the Due Diligence and approval process for CFDCs, WD, and IC. (Return to Reference 16).

17 Job maintenance was regarded as an additional, but not sufficient, benefit. (Return to Reference 17).

18 Only two on the 15 case study projects reported timeliness issues associated with the SICEAI application process, and three others specifically praised the timeliness of the application process, as compared to other programs. (Return to Reference 18).

19 A proposed project which only created short-term construction employment, for example, would be ineligible. (Return to Reference 19).

6.0 Preliminary Results and Success

6.1 Program success

Evaluation questions

  1. How successful has SICEAI been in achieving its overall objectives? For example,
    1. What has been the impact of SICEAI in terms of mitigating the socio-economic effects of U.S. duties on softwood lumber exports to the U.S.?
    2. To what extent has the SICEAI supported community economic development and assisted communities to develop and diversify through: implementation of capacity building; increased employment and changes in employment distribution, stability and sustainability inside and outside of softwood lumber production?

Summary

  • The program has had a modest success in mitigating the direct impacts of forest sector job losses due to the U.S. tariffs, although not surprisingly it has not been able to dramatically diminish these effects, especially amongst laid-off workers. (Note that the program was not intended to do so, as it was not intended to directly provide assistance to laid-off or displaced workers, or to directly replace these forestry-related jobs.)
  • It has been rather more successful in helping communities diversify their economies and increase their capacity for change.

Discussion

RDA perspective. Program officials generally felt that it is not realistic to expect that the problems facing the softwood lumber industry can be solved within a 3-year program. Economic adjustment and diversification requires a longer-term process, especially if capacity-building is involved. However, the program certainly met the government's expectations, in that it helped to meet the needs of the communities going through a significant adjustment.

Client perspective. Clients essentially agreed with the RDA officials on this topic, believing that SICEAI had had a modest success in mitigating the tariffs' socio-economic impacts on displaced forest sector workers. The program was not intended to directly provide assistance to laid-off or displaced workers, or to directly replace these forestry-related jobs but had a rather greater success in supporting community diversification and capacity building efforts. Several clients commented that the project had helped the community to think more seriously about diversification and had brought people and organizations together, but there simply hadn't yet been enough time elapsed to know what the true impact of the program and their project will be.

Success in mitigating socio-economic impacts due to tariffs
  1  
Very unsuccessful
2 3  
Neutral
4 5  
Very successful
Don't Know N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 3% 5% 37% 29% 21% 5% 38
ON 8% 15% 38% 27% 8% 4% 26
QC   12% 38% 29% 12% 10% 42
Region unknown       100%     2
Total 3% 10% 37% 30% 14% 6% 108
top of page
Success in supporting community's diversification and adjustment efforts
  1  
Very unsuccessful
2 3  
Neutral
4 5  
Very successful
Don't Know N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
BC 3% 3% 13% 32% 50%   38
ON 7%   30% 33% 30%   27
QC 2% 5% 11% 32% 45% 5% 44
Region unknown       67%   33% 3
Total 4% 3% 16% 33% 42% 3% 112
top of page

6.2 Project Success

Evaluation Question

  1. How successful have individual projects funded by SICEAI been in achieving their stated objectives?

Summary

  • About 61 percent of projects are currently complete and operational, with another 10 percent complete but not yet operational (e.g., in cases where new facilities have been constructed, but staffing has not yet been completed and/or all activities are not in full operation), and 14 percent still in the construction phase but expected to be completed. Only 1 percent of the projects are not expected to be completed, and another 2 percent were never started (the remaining 12 percent of respondents either answered "not applicable" or did not answer at all).
  • Both RDAs and clients mentioned that the projects have created and/or maintained jobs, and that the projects have often helped communities to think "outside the forestry box" in diversifying their economies.

Discussion

RDA perspective. Because there was a deliberate effort in all three provinces to fund projects with long-term, sustainable diversification impacts, officials are hopeful that the businesses and jobs created will be sustainable. Many "success stories" were mentioned in which the projects were apparently very successful (e.g., the McBride project in BC now supports about 70 percent of the community's workforce). Many project officers and RDA officials believed that the bulk of projects had been successful. In fact, virtually all "failures" that came to mind were projects that never succeeded in getting off the ground (usually because a good application could not be prepared, or because matching funds could not be obtained) rather than projects that were initiated but failed.

Client perspective. About 65 percent of SICEAI projects have been completed and, as of our survey, were operational. Another 11 percent were complete but not operational. Only 1 percent of the projects actually initiated are not expected to ever be operational. Given the complexity of many of these projects and the short timeframe for the program, this is a high success rate.

Project Status
  BC ON QC Region unknown Total N 
(Projects)
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
Never started - 9% - - 2% 3
Partially implemented, but not expected to be completed - 3% 2% - 1% 2
Partially implemented, but will be completed 17% 26% 7% - 15% 20
Complete but not operational 7% 17% 10% - 11% 15
Complete and operational 71% 40% 74% 100% 65% 89
Not applicable 5% 6% 7%   6% 8
Total 100% 100% 100% 100% 100% 146

The percentage of projects that were complete and operational was highest for "first projects", with 66 percent being in this state. For second and third projects (as some proponents had more than one project) this percentage dropped to 44 percent (18 respondents) and 33 percent (3 respondents), respectively. It is unknown whether this declining success rate represents less time or energy to complete later projects, or a "learning curve" with earlier, unsuccessful, projects.

Clients usually noted that the most important impacts of their projects for their communities were (1) diversification of the local economy into production value-added wood products and/or entirely new industries, and generally "thinking outside the forestry box"; and (2) local job creation and/or maintenance, along with associated revenues. Both types of impacts were mentioned roughly equally often. A number of clients mentioned long-term and/or indirect knock-on benefits as other businesses and suppliers grew to accommodate new opportunities. Some impacts on local community collaboration, spirit, and pride (especially where cultural projects such as historical centres had been implemented) were also mentioned. It was notable that only a small number of respondents (roughly 10 percent) did not mention any positive impacts to date at all.

The case studies also demonstrate the success of some of the SICEAI projects. Of the 15 case study projects, 13 are now complete and operational, and the remaining two are complete but not yet operational. Thirteen of the case studies also demonstrate a clear success in diversifying the local economy, into areas such as tourism, technology, manufacturing, power generation, and value-added forestry operations.

The Pontiac region was historically heavily dependent on the forest industry. The project clearly helped to diversify the local economy, as [the company], a leader in the turf grass industry, is the largest single employer in the MRC of the Pontiac. Therefore, it was very important to maintain this level of employment.

The ___ region was previously 100 percent reliant on the lumber industry. This project will serve as a catalyst to demonstrate the potential of a new value-added wood-manufacturing sector in [our region].

A good project like this makes people realize that we don't have to rely strictly on primary resources.

In addition, one of the remaining two projects is a successful capacity building project.

top of page

6.3 Community impacts

Evaluation Question

  1. To what extent were there: new businesses created; jobs created and/or maintained; new innovative projects leading to diversification in softwood-dependent communities; improved sustainability and/or self-reliance of communities?

Summary

  • About 85 percent of projects were successful in creating jobs or maintaining jobs to date, as shown below. Just over half the projects did both.
  • A conservative extrapolation is that roughly 2,500 jobs have been directly created to date, increasing to about 4,000 jobs expected in three years time as projects come fully on-line.
  • A conservative extrapolation is that about 4,000 jobs have been maintained to date, and this is actually almost double what was originally anticipated. However, these are somewhat less sustainable than the newly-created jobs, and only about 2,800 are expected to remain in three years time.
  • Indirect job creation and maintenance (i.e., in the surrounding community to help provides goods and services to the proponent organization) were also important, and together have added about another 2,300 jobs to date. These are also nearly all sustainable, although again the newly-created jobs are expected to increase in future as projects mature, while maintained jobs will probably slowly decrease.
  • However, it was noted that the jobs created through SICEAI are not direct replacements (either in terms of the individuals affected, or in the wages replaced) for those lost in the forestry industry.
  • A conservative estimate is that the program to date has created about $147 million in annual gross revenues, and maintained about $436 million. These figures are expected to rise to about $646 million and $490 million, respectively, in three years' time.
  • A general conclusion is that new jobs and revenues that are created appear to be more sustainable over the long term than maintenance of existing jobs.

Discussion

RDA perspective. Field officers in all three provinces have noticed the positive effects of the program within the communities (e.g., companies and jobs created and maintained) although the full impacts of the program within the communities may not be felt for many years. Many program officials viewed the projects that were funded as a starting point for many entrepreneurs and SMEs; it gave them the push they needed to take their business a step further and improve productivity. Further, many program officials felt that these businesses and jobs were critical for the affected communities ("10 on a scale of 1 to 10").

Because there was a deliberate effort in all three provinces to fund projects with long-term, sustainable diversification impacts, officials are hopeful that the businesses and jobs created will be sustainable. Many "success stories" were mentioned in which the projects were apparently very successful (e.g., the McBride project in BC now supports about 70 percent of the community's workforce). However, it was also noted that the jobs created through SICEAI are not direct replacements (either in terms of the individuals affected, or in the wages replaced) for those lost in the forestry industry, which tend to be higher-paying jobs.

Client perspective. About 85 percent of projects were successful in creating jobs or maintaining jobs to date, as shown below. Just over half the projects did both.

Job creation and Maintenance
Jobs created or maintained BC ON QC Region unknown Total N
Source: Survey of proponents (clients). Table may not add to 100% due to rounding errors.
No jobs created or maintained 8% 41% 6%   15% 17
Created new jobs only 34% 11% 6%   17% 19
Maintained jobs only 5% 15% 21% 33% 15% 17
Both created new & maintained jobs 53% 33% 66% 67% 54% 62
Total 100% 100% 100% 100% 100% 115
top of page

Project proponents were asked directly in the client survey about the number of jobs created and/or maintained in their project(s). This was asked in terms of the numbers originally hoped-for (i.e., at the application stage, and in project targets), what had actually resulted to date, and what was expected in three years' time (since many of the larger projects are only just coming on-line). The results are encouraging, if somewhat difficult to interpret with rigour. The interpretation problem stems because:

  • We were unable to contact about half of the clients, and the response rate for those we did contact is only about 50 percent; and
  • Not all respondents answered all three parts of the question (i.e., hoped-for, to date, and expected in three years)

As a result of these analytic difficulties, we present the results for job creation and maintenance in two parts:

  • "Minimum" figures. These represent the absolute minimum figures based on the results for our survey respondents only. They are not extrapolated up to the client sample (i.e., we did not extrapolate to clients we were able to contact, but who did not respond to the survey), or to the total client population as a whole (i.e., we did not extrapolate to clients we could not contact at all). These figures are almost certainly a tremendous under-estimate of the true job creation and maintenance figures, but they are a solid lower bound.
  • Conservative "Extrapolated" figures. These represent an extrapolation of the survey data up to the client population as a whole.

    Total # jobs extrapolated for a given province = (Average # jobs per project in survey sample for that province) x (Total # projects in that province).

    The "average jobs per project" figure used in this extrapolation conservatively assumes that all non-responses for a given province, in a given response category, represent "zeros". This almost certainly results in an under-estimate of the true figure (there are several instances in the survey data where respondents provide, say, "hoped-for" and "to date" figures, but no "expected in three years" figures, but it is unlikely that no jobs are actually expected to remain in future), but there is no easy way to determine the value of the missing data. Note that the number of respondents varied by response category (e.g., direct job creation to date, vs. indirect job maintenance in three years' time), so extrapolations were done individually for each response categoryFootnote 20.

We would normally be quite uncomfortable making extrapolations from a sample of only about 22 percent of the total number of projects. However, remember that only about 46 percent of client could be contacted in the first place, and the response rate for those proponents that could be contacted was about 48 percent, which (given the program is over) was excellent. The main reason that some proponents could not be contacted in the first place was lack of time and resources within the study—i.e., it was not because of some feature of the proponent organizations themselves. As a result, we believe that the final respondent sample is random enoughFootnote 21 that extrapolation is justifiable for discussion purposes, so long as the reader is aware that these figures are necessarily very rough approximations, and we have additionally used a conservative extrapolation methodologyFootnote 22. See also the Recommendations section. Note that "direct" job creation and maintenance are within the proponent (client) organization, while "indirect" job creation and maintenance are in the surrounding community (e.g., to help provide goods and services to the proponent organization).

Number of jobs created and maintained (FTEs)
  Hoped-for Actual to Date Expected in 3 Years
Minimum Conservative Extrapolation Minimum Conservative Extrapolation Minimum Conservative Extrapolation
Source: Survey of proponents (clients).
Jobs directly created 718 3,146 500 2,552 1,044 3,926
Jobs directly maintained 423 2,230 1,033 4,036 719 2,845
Sub-total Direct Jobs 1,140 5,376 1,533 6,638 1,763 6,770
Jobs indirectly created 308 1,089 232 850 389 1,427
Jobs indirectly maintained 239 1,109 572 1,500 163 740
Sub-total Indirect Jobs 548 2,199 804 2,350 551 2,167
Total all jobs 1,688 7,575 2,337 8,988 2,314 8,937

Notwithstanding that (for reasons detailed above) it is impossible to say exactly how many jobs have been created and/or maintained, a conservative extrapolation is that roughly 2,500 jobs have been directly created to date. Furthermore, these are expected to be sustainable, with this figure actually increasing to about 4,000 new directly created jobs expected in three years time as projects come fully on-line. The fact that the number of these jobs is, to date, somewhat less than hoped-for is not unexpected given that for many projects entirely new businesses are being created.

On the job maintenance side, a conservative extrapolation is that about 4,000 jobs have been maintained to date, and this is actually almost double what was originally anticipated. However, these are somewhat less sustainable than the newly-created jobs, and only about 2,800 are expected to remain in three years time, no doubt reflecting further changes expected in the forestry industries.

Indirect job creation and maintenance were also important, and together have added about another 2,300 jobs to date. These are also nearly all sustainable, although again the newly-created jobs are expected to increase in future as projects mature, while maintained jobs will slowly decrease.

The jobs created and maintained will result in revenues being created and/or maintained for the project proponents, as shown in the following table. (As for job creation, a conservative extrapolation method was used where missing data were treated as "zeros".)

Annual Gross Revenue Generation ($ millions)
  Hoped-for Actual to Date Expected in 3 Years
Minimum Conservative Extrapolation Minimum Conservative Extrapolation Minimum Conservative Extrapolation
Source: Survey of proponents (clients).
Revenues directly created 73.9 306 31.1 147 174 646
Revenues directly maintained 17.1 75 129 436 116 490
Total all Revenues 90.9 382 160 583 290 1,137

Note that similar comments regarding extrapolation applies to this revenue generation and/or maintenance table as for that showing job creation and/or maintenance.

Case study data. Of the 15 case studies, only one project did not lead to the creation or maintenance of any sustainable jobs (although some temporary jobs were created, and the organization hopes that the results of the project will lead to other jobs in the future). The following table provides a breakdown of the case study data on job creation and maintenance:

Case study data
  Number of projects
No sustainable jobs created or maintained 1
1 to 5 sustainable jobs created or maintained 5
6 to 10 sustainable jobs created or maintained 1
11 to 25 sustainable jobs created or maintained 4
26 to 50 sustainable jobs created or maintained 2
Over 50 sustainable jobs created or maintained 2

The case study projects also illustrate how in some cases, SICEAI projects have already acted as a catalyst for other community diversification initiatives, for example within the tourism industry:

The project has allowed Red Lake to branch into the tourism sector, and will act as a catalyst for future tourism development projects. For example, one of the local canoe outfitters is now planning to develop canoe tours, based on the history of Red Lake, and has approached [the Centre] for historical information and photographs.

The SICEAI initiative, and focus on economic diversification away from the forest industry, has assisted with the thinking process toward [a new recreation and entertainment] facility. The community is now planning initiatives it may not have considered before.

top of page

6.4 Capacity building

Evaluation Question

  1. For the Capacity Building component, what has been the impact of SICEAI in helping communities to identify, plan, and manage the transition required to adjust to the socio-economic consequences of the U.S. duties?

Summary

  • Capacity building projects were not common in any of the three regions. It is too soon to judge the impacts of those that were funded.

Discussion

In all three regions, capacity building projects represented a small percentage of total projects funded through SICEAI, as well as a small percentage of the overall funding allocation (e.g. capacity building projects make up approximately 14 percent of all projects funded, but less than 6 percent of total funding). These numbers are slightly higher for Ontario, where approximately 28 percent of projects were capacity building projects and these projects received 11.6 percent of the funding allocation.

Generally, regional agency representatives felt that capacity building projects (i.e., building leadership, community networking, economic development plans) could be funded relatively easily through regular agency programming, since they normally require less funding than diversification projects.

In addition, the long-term effects of these projects are difficult to identify and may only become apparent after several years. Also, despite all the planning, the funding and resources required to implement the projects are not always available in the long term.

One capacity building project was included in the case studies. This project has allowed members of a First Nations community to acquire specialized training, as is probably unique in that it has led to the creation of five jobs for these people, to date.

top of page

6.5 Investment

Evaluation Question

  1. What has been the impact of SICEAI in terms of increasing investments (internal and external) in affected communities?

Summary

  • The program was successful in obtaining about $247 million in external (non-program) funding, representing a leverage of about 225 percent of the $110 million federal funding.
  • The "gross (i.e., total) leveraging" of the average project in our client survey was about $516k, or roughly 230 percent of the program's contribution to the project.
  • The "net leveraging" for the average project in our client survey was about $208k, or about 157 percent of the program's contribution. This represents the additional leveraging that occurred because of the program's existence; i.e., funding that would not have been committed to these projects in the absence of SICEAI.

Discussion

RDA perspective. Finding matching funding was problematical in all three provinces. The rule of thumb was that SICEAI funding had to be less than 50 percent of the overall investment. However, many of the municipalities and First Nations Communities did not have the funds available to contribute to the projects. Especially in such small communities with few resources, FedNor at times increased its contribution slightly. FedNor officials felt that it would have been easier to leverage funds if the provincial government had been more involved at the time.

Generally, the Agencies might have been able to fund some of the projects through their existing funding (especially in Ontario and Quebec), but not as many, and there would have been more limitations. Because the BC projects tended to be larger, it was believed that many would not have been possible to support without SICEAI; e.g., the CFDCs might each typically have $150k–$200k per year for such projects, whereas SICEAI provided about $44 million to the network of 33 CFDCs.

Client perspective. Of interest is the amount of "net leveraging" that the projects created. This is the amount of additional funding contributed to the project that would not have been obtained if SICEAI had not contributed to it.

Investments—Average per project
Investment by Actual With Program Estimated Without Program Net Leveraging
SICEAI program $224,000 0 N/A
Client organization $260,000 $165,000 $95,000
Other project partners $280,000 $167,000 $113,000
Total $846,000 $329,000 $208,000

The "gross leveraging" is simply how much money in total was leveraged from other organizations. In this case, the average SICEAI project in our survey sample leveraged about $516k, or roughly 230 percent of the program's contribution. Overall, the program was successful in obtaining about $247 million in external (non-program) funding.Footnote 23

In terms of net leveraging, of course, these figures are lower since some project partners would have contributed towards the project even in the absence of SICEAI (although often a lower amount, or even zero). The average "net leveraging" is about $208k per project (the difference between $260k+$280k and $165k+$167k), or about 157 percent of the program's contribution.

top of page

6.6 Unanticipated impacts

Evaluation Question

  1. Have there been unexpected or negative impacts as a result of the SICEAI program?

Summary

  • Virtually all of the unexpected impacts for clients were positive, and included additional jobs being created, new business opportunities being identified, considerable media attention, revitalization of neglected business areas, and a generally greater sense of community participation and collaboration. Negative impacts were mainly related to the burden of preparing applications and dealing with delays.
  • For RDAs, similar positive impacts were noted. Some additional negative impacts were related to the initial confusion mentioned earlier regarding goals, procedures, etc., and (in BC) the workload associated with the very high number of initial project applications.

Discussion

RDA Perspective.

Positive impacts:

  • Collaboration/partnerships between the various parties involved
  • Increased knowledge and awareness of the problems facing each community
  • Created a model for future initiatives (learning experience)
  • Positive federal government/agency image and reputation; perception of being a leader in providing community assistance
  • Encouraged some clients to carry their projects further, even once the program funding had ended.

Negative impacts:

  • There was confusion (by proponents) with other FedNor programs (proponents often did not know about SICEAI; just knew it came from FedNor).
  • The program created certain expectations within some communities for future support.
  • The process created a very substantial additional workload for WD and the CFDCs, without much (if any) return for them. (The CFDABC would have been willing to take on responsibility for collection o f repayable contributions, for example, for a modest percentage of the revenues, perhaps 2 percent.)
  • In BC, unrealistic expectations created early on because of uncertainties about the program's intent (although these problems disappeared once a transparent application review process was initiated).

Client perspective. About a third of the projects reported unexpected positive impacts. Of these, the most significant was diffusion of impacts into the broader community, for example through additional jobs being created, new business opportunities being identified, considerable media attention, revitalization of neglected business areas, and a generally greater sense of community participation and collaboration. Almost no clients reported unexpected negative impacts, but those mentioned were mainly related to the lengthy delays experienced in getting the projects approved and monies paid.

Other Impacts
  BC ON QC Region Unknown Total
Source: Survey of proponents (clients). Table may not add to 100%—multiple responses are possible.
No unexpected impacts 43% 61% 55% 33% 52%
Unexpected positive impacts 46% 21% 21% 33% 30%
Unexpected negative impacts 6% 4%     3%
Both unexpected positive and 
negative impacts
3%   2%   2%
Don't know 3% 14% 21% 33% 14%
Total 100% 100% 100% 100% 100%

Footnotes

  1. 20 We did not attempt to make the small correction for the fact that some clients had more than one project. (back to footnote reference 20)
  2. 21 We did not, for example, attempt to only contact the larger client organizations, or only those known to have successful projects. (back to footnote reference 21)
  3. 22 We checked our extrapolations against detailed figures available from WD for BC. WD figures—which are based on "hoped-for" jobs—state that about 2,500 jobs were directly created in BC. This compares with our extrapolation of "hoped-for" directly created jobs in BC of 1,641, suggesting that the extrapolations are not entirely unreasonable, but are indeed conservative. (back to footnote reference 22)
  4. 23 SICEAI Monthly Update, May 27 2005. (back to footnote reference 23)

7.0 Findings on Cost-Effectiveness

7.1 Cost effectiveness

Evaluation Question

  1. Is SICEAI the most cost-effective way to achieve the stated objectives? If not, what could be improved?

Summary

  • The cost per job (i.e., all jobs, either directly or indirectly created, and including both job creation and maintenance) is roughly $12,000. If one only considers direct job creation and maintenance, then the cost per job is about $16,000. On this basis, the program appears to have been very cost-effective. Most respondents in the RDAs believe the program was cost-effective.
  • The major mechanism suggested that might improve program efficiency was elimination of the Other Government Department (OGD) suspense accounts mechanism, discussed in detail in Section 9. No other significant changes that bear directly on this issue were suggested by respondents.
  • There is little "hard" data available on this point (although it is always a hard question to answer), and the study was not intended to definitively determine whether the program was costs effective or not. Instead the study determined whether there are any improvements to design and delivery that would improve cost-effectiveness. A full accounting of incremental benefits and costs would be a great help to Industry Canada, FedNor, and the RDAs in understanding the impacts (including the true cost-effectiveness) of SICEAI. If applied to other similar programs, this would also help identify the impacts of different program models.

Discussion

First, note that this study was not intended to definitively answer whether the program was cost-effective or not. Instead, the issue is the standard program evaluation question of whether there are improvements in the program design or delivery that could be made that would increase the cost-effectiveness of this program. A full accounting of cost-effectiveness is one issue that might be addressed in a summative evaluation, which would not be appropriate in this situation.

The majority of program officials believe that the program was cost-effective, mostly because it was delivered through the regional agencies, which already have the staff and expertise, the information resources, and the infrastructure in place to deliver the program. In BC, the regional CFDC network has experience working collaboratively on a variety of programs and projects, and also has strong communications networks in place within the communities. With only a limited number of additional operating resources, the agencies were able to augment their capabilities to handle the program quite easily. Many officials stressed the importance of using the locally-based agencies (and CFDCs in BC) when delivering future community-based programs.

Some program officials commented increased cost effectiveness could have been achieved by de-centralizing the program to the agencies, as for the mining and fisheries adjustment programs referred to earlier. This would have reduced delays, and therefore saved money. One official suggested that a competitive, timed intake for projects may have required less staff (thus being slightly more cost-effective), but felt that the overall value would be less.

The approximate cost of job creation is shown in the table below. The cost per job (i.e., all jobs, either directly or indirectly created, and including both job creation and maintenance) is roughly $12,000. If one only considers direct job creation and maintenance, then the cost per job is about $16,00024. (Not shown in the table is that if one only considers direct job creation—i.e., ignoring all jobs maintained, and all indirect job impacts—then the cost per job created to date is about $43,000, falling to about $28,000 in three years' time as the projects become fully operational. However, considering only direct job creation would obviously ignore a substantial portion of the program's impacts.)

Job Creation and/or Maintenance Costs
  Cost per Job 
to Date
Cost per Job 
Est. in 3 Year's Time
The figures in the table above are estimated from the "conservative extrapolations" discussed in Section 6.3.
Jobs directly 
created & maintained only
$16,571 $16,248
Total all jobs 
(direct & indirect, created & maintained)
$12,239 $12,308

The study team comments that the SICEAI program, like many community-based economic development programs, had a number of client-assistance features that have implications for administrative and oversight costs. These include the extensive outreach program to ensure affected communities knew about the program, the coordination and outreach officers hired both for communications and to assist communities and proponents to prepare applications, the multiple steps often taken to ensure clients met the complex eligibility requirements, the extensive review process employed in BC, the "extra mile" often taken to create the flexibility needed to support an important project that was somewhat outside the norm, and the sometimes remote nature of (and high cost of getting to) the communities in all three regions (especially those of First Nations). In addition, WD, CED, CFDCs, and FedNor were often intimately involved in the sometimes onerous and lengthy environmental assessments required. Many of the larger projects have literally hundreds (and probably thousands) of pages of paperwork and correspondence associated with them. Finally there were costs (probably mostly hidden) of the lengthy process of simply designing and negotiating the program in the first place.

These activities mean higher administrative and oversight costs than would probably be the case in a highly-centralized and more rigid program delivery model. The other side of the coin, however, is that this model allows projects that are custom tailored to each individual situation and thus—while administrative costs may be higher—it is likely that the resulting benefits are as well.

The question then arises as to exactly what the cost-effectiveness of SICEAI (and similar programs) is. A defensible analysis of this issue requires two things: (1) accurate data on the true total costs of the program, including any "hidden" costs; and (2) accurate data on the true long-term impacts of the program, including indirect benefits (this being the "effectiveness" portion of the equation). Each is non-trivial to obtain:

Benefits. Data required include:

  • Data on all projects. That is, contact as close to a census of projects as possible, rather than relying on an extrapolation of data from a random sample of projects. Often it is the case that only a small proportion of projects provide the "lion's share" of the benefits—missing just one or two of these projects may result in a dramatic under-estimation of the true total benefits. If a sample must be used, it should be done by picking as many of the most successful projects as possible and using the sum of their benefits as a lower bound, without extrapolation 25a.
  • The true number of sustainable jobs created and/or maintained. This first requires defining what "sustainable" means—i.e., over what time period. This ideally requires understanding trends over time, such as job numbers from project completion through to, say, ten years afterwards. However, even information collected two or three years following project completion would be of great help, as it would identify projects that are still in operation vs. terminated, and which are expanding or contracting.
  • Direct and indirect jobs created and/or maintained. Many SICEAI projects were said to have been responsible for indirect job creation, and some of these impacts were expected to become quite large in future years. This should be documented.
  • Analysis of incrementality. This is especially important for indirect benefits, as typically many other factors have been partially responsible for these impacts.
  • Analysis of any displacement and/or substitution effects. This is necessary for a complete understanding of job impacts.

Costs. The data required include:

  • Total direct project contributions.
  • Direct costs to project proponents (including any costs of borrowing, legal fees, etc.);
  • Administrative and other non-project costs for all community, government, and regional delivery agencies. These data should include any hidden costs that may not be directly accounted for in "line items" associated with the program, such as administrative and oversight costs for agency "headquarters", unpaid overtime, etc.
  • Any other "hidden" costs associated with, for example, volunteer effort expended by individuals and communities.

A full accounting of hidden costs is especially important for decentralized, community-based programs, for which many of these costs are "hidden". Note that many analyses of cost-effectiveness do not take into account all such factors, or assess them in different ways. Thus a fair comparative analysis of different programs should use the same methodology for each program25b. The study team found that there was great interest in Industry Canada, WD, FedNor, and CED for such information. See recommendations #10 and #11.

top of page

7.2 Due Diligence

Evaluation Question

  1. Were the elements of due diligence applied by the SICEAI?

Summary

  • The review of Terms and Conditions, and the review of program files conducted for the case studies, indicate that extensive due diligence was conducted within this program. External review of the application procedure in BC supports this conclusion26.

Discussion

The review of program files conducted for the case studies indicate that extensive due diligence was conducted within this program. The following elements were present in the file information for the projects that were examined.

Ontario and Quebec. The project files contained, at a minimum, the initial project proposal, as well as specific deliverables, which were to be provided in order to funding to be released. These included: payment confirmations from other funding sources, if applicable; project budgets and/or business plans; progress reports, photographs and notes of site visits; contracts, purchase orders, receipts and other supporting documentation for expenditures; environmental assessments, if applicable.

British Columbia. All BC projects were subject to an extensive application review process, including a Stage One information and checklist assessment (which included review of eligibility, anticipated benefits, budgets, timelines, community support, potential environmental impacts, market assessment, and management ability), a very detailed assessment of a full project or business plan, a risk assessment, a Due Diligence Report (DDR, which includes assessment of incrementality, viability and sustainability), a Screening Report, an Environmental Assessment, and review of federal and provincial regulatory requirements. Following this, a separate review by the CFDC Quality Review Panel (QRP) was conducted to approve funding and a review by the Minister's Advisory Group (MAG) for fit with the strategic goals of the program. For projects in receipt of repayable funding, projects were also reviewed and approved by Industry Canada. Evidence of these elements was present in the file information for the projects that were examined.


24 This figure is quite similar to the $12,000 estimate calculated by program staff, as shown in the table in section 3.6. (Return to Reference 24).

25a This is analogous to "partial benefit-cost analysis" used in the review of science and technology programs. See Partial Benefit/Cost in the Evaluation of the Canadian Networks of Centres of Excellence Program, A. Dennis Rank and Douglas Williams. Evaluation and Program Planning, Volume 22, No. 1, February, 1999. (Return to Reference 25a).

25b Ibid.; By collecting information on all these variables, the results can often be compared to those in other reports that used different analytic methods. For example, one could compare results to another analysis that did not include hidden costs simply by ignoring the hidden costs in the program under review. (Return to Reference 25b).

26 PriceWaterhouseCoopers. February 9, 2005. Op. cit.. (Return to Reference 26).

8.0 Findings on Success Factors

Evaluation Question

  1. What factors have facilitated/impeded the implementation of the SICEAI; e.g., reach/ awareness/ promotion; accessibility; planning and coordination; targeting of and programming for communities; resources; partnering; visibility; project monitoring and data collection (frequency and type of tools/mechanisms)?

Summary

  • Key success factors include the local knowledge of CED, FedNor, and the CFDCs of their communities' needs (in some cases supplemented by additional staff hired specifically to help with program marketing, outreach, and client support). This local expertise also helped to ensure a broad reach to small, remote, and First Nations communities could participate, and helped develop high quality proposals.
  • Negative factors included the initial delays in implementing the program, and the lack of appropriate promotional materials.

Discussion

The principal success factor mentioned by the program officials that were interviewed was the fact that the program was delivered through the regional agencies, which already had a strong presence in the regions given the existence of their local community networks (including CFDCs). They largely attributed the success of the program to the staff's knowledge of the issues, and of the communities. This affected almost all areas of program delivery, including reach (including reach to many smaller, more remote, and First Nations communities, as well as to SMEs), awareness, accessibility, partnering, and resources. However, this delivery model did complicate the planning and coordination of the program, as well as project monitoring. In essence, there was an extra layer of bureaucracy in this program.

Program officials noted that another key success factor of the program was the quality of projects selected and their focus on creating sustained economic opportunity, as well as the motivation and experience of the project proponents.

A significant factor impeding the success of the program was the delays encountered during program set-up, for reasons discussed earlier. As one official noted, "The program could have benefited from better timing".

Another factor which FedNor and CED officials listed as impeding the reach and promotion of the program was the lack of promotional materials and program documentation. This made it more difficult for staff to understand the particulars of the program, and to get the word out to the communities. (Probably this was less of a problem in BC, which conducted its own outreach program and community needs survey.)

9.0 Lessons Learned

9.1 Best Practices

Evaluation Question

  1. What specific best practices have been generated with respect to the SICEAI?

Summary

  • A longer time frame is needed for adjustment programs to be fully effective. This will allow better quality proposals to be developed, and more time for initiation and completion of complex projects, and better retention of community expertise.
  • Having program goals, objectives, authorities, and procedures worked out between IC and other program partners in advance of the program's public announcement will save considerable time and frustration for all parties.
  • Terms and Conditions must be flexible enough to respond to individual community circumstances, but there must be an oversight mechanism to ensure they are applied reasonably consistently across regions and communities.
  • A simpler program structure than the use of OGD suspense accounts would be beneficial.
  • A mechanism for identifying industry needs in advance of crisis situations is preferable, and allow better identification of exactly what impacts need to be addressed.
  • The case studies showed that larger projects have the potential to create benefits that are wide-spread throughout the community, and to foster significant "knock-on" benefits. (Given fixed program funding, of course, supporting a few large projects may prevent supporting many small ones.)

Discussion

Need for longer time frame. Many officials and project proponents mentioned that the program's narrow timeframe was a problem. Communities have a certain "percolation period"; start-up and planning can easily take one and a half to two years in themselves, especially if an Environmental Assessment (EA) is required. Therefore the program was not available to all communities. The communities that were able to take advantage of the program often already had projects "in the hopper". Considerable effort was made by the RDAs to help communities identify appropriate projects given SICEAI's eligibility requirements and the time available (e.g., in BC by the coordination and/or outreach officers, and the Quality Assurance Review Officers). It was also noted that the short timeframe meant that non-governmental stakeholders were not "at the table" during development of the program. Program officials noted, however, that a longer time frame would have resource implications (see the Recommendations section).

Community-based delivery. The local knowledge and expertise of the CFDCs, RDAs and FedNor were cited as significant advantages of this program. The community-based system offered the advantages of better community knowledge and stakeholder input, more flexible response to changing needs, better fit of projects to needs, and faster response time.

Initial delays and confusion. Industry Canada officials agreed that the structure was too complex, and that it took too long to get the administrative structures in place, but felt that once they were working within an approved delivery structure, with the proper accountability and spending authority structures in place, the program ran quite smoothly. They further noted that initial program set-up was particularly difficult in British Columbia, where agreements had to be negotiated not only with Western Economic Diversification Canada but also with the CFDABC and the CFDCs. In addition, the Minister's Advisory Group (MAG; made up of BC community leaders, company and sector representatives) had to be created and made operational.

Because of the initial program delay, the funding was eventually pushed back another year. However, this was not confirmed until the last minute, which made it difficult for the RDAs to plan for projects until they knew how much money they could commit. This was especially true in working with the private sector, where some lead-time is required. By the time the SICEAI allocation amounts were clearly known, in the private sector it was starting to be too late. Application review was difficult because of this. In BC, for example, a comprehensive review of about 1,100 applications had to be completed within a very short timeframe.

Overall, it was noted that the glitches in the program were driven by the need for the federal government to respond quickly. Whatever the method, it was felt that a long-term system should be in place, in advance, in order to better respond to acute crises.

The BC situation was worsened by competing visions among different stakeholders early on in the program's lifetime. This did not delay implementation (since the delay was due to the program funding not being authorized right away), and were eventually resolved between the MAG, WD, and the Quality Review Panel (QRP27; which had the mandate to review applications and approve projects for funding), but did cause early tension.

Allocation of resources among the provinces. The program trigger—indefinite layoffs—was seen to be very narrow and inflexible, since communities where companies had responded to the crisis with temporary layoffs or job-sharing were not eligible to the program. Also, some officials suggested that the quality of the data collected may have been inconsistent among the provinces (due to varying data sources). Initially, few current and projected layoffs were attributed to the softwood dispute in Ontario. FedNor argued that the full effect on jobs had not yet been realized, and it felt it was disadvantaged in negotiating its allocation of program funds. This was also the case in Quebec's Gaspé and Laval regions, where layoffs only occurred in the later years of the program. As future year layoffs became known, Ontario's allocation was augmented (in part from transfers from other RDAs); however, the fact that the amount was not known early on created difficulties in planning projects and managing for the allocation. In developing future programs, it was believed to be important for Industry Canada to base allocation decisions on historical data on job losses, which would allow better analysis of long-term trends and promote a more proactive approach to industry problems28. [The study team notes that a longer-term program would go a long ways towards alleviating this problem, and adjustments to allocations might be made over time.]

Initial planning. More initial planning by Industry Canada in consultation with the Regional Development Agencies would be very useful. This will allow IC and the agencies to address program design and delivery issues early on, preventing implementation delays. This should be supplemented by a longer-term program approach that will allow for more front-end development within the communities, better application preparation, a more reasonable timeframe for application review, a more appropriate time period for completion of projects, more time to build community capacity and diversity, and more ability of the RDAs to continue to use the expertise they build up within the communities.

For future initiatives, it would be important to look at how the program is addressing the needs of the private sector. For example, some program officials suggested that similar future programs might allow for the funding of more industry R&D, which was not eligible under this program.

Nature of Guidelines. The three provinces took quite different approaches to this program. For example, BC funded a number of large projects that affected many communities (and would have been impossible to support with normal CFDC funding), while this was uncommon in the other provinces, which typically funded smaller projects (many of which would fall into their "normal" project envelopes, although SICEAI increased how many could be supported). Quebec and Ontario commonly provided multiple contributions (up to four) to individual proponents, whereas this was rare in BC. The application and review processes were also quite different in each province. Such variability is typical of decentralized programs—it can be good if it represents a response to regional differences, but bad if it means that program goals are being somewhat misunderstood or mis-applied. Industry Canada should carefully consider the implications of this variability in terms of the intent of, and processes used in, any new adjustment initiative. [The study team explicitly does not mean that variability and flexibility should be discouraged, only that the pros and cons of these different approaches should be reviewed.]

On a related topic, many officials saw the program flexibility as being excellent, and felt that this flexibility contributed to program success (i.e., the Terms and Conditions, as well as regional program guidelines, were well framed while still being flexible enough to provide for direct assistance to businesses as well as to community projects). However, some officials believed that this flexibility needed to be standardized to some degree across the provinces and regions. For example, FedNor exercised a great deal of flexibility in interpreting program guidelines, whereas some areas of Quebec felt limited by the same criteria.

Program complexity. Although the program model was successful overall, many officials (including those within Industry Canada) noted that the structure, which utilized an instrument known as an Other Government Department (OGD), was too complex, and that it took too long to get the required spending authorities, decision making delegation and administrative structures in place. A model which transfers these responsibilities and accountabilities entirely to the RDAs has been seen to work very effectively in other initiatives. Using the RDAs (and the CFDCs in BC) in a community-based delivery model has the advantages of better community knowledge and stakeholder input, more flexible response to changing needs, better fit of projects to needs, and faster response time. It also helps build and maintain community economic development capacity outside of internal project impacts (e.g., within the RDA offices). Examples include HRDC's Softwood Sector Employer program, the BC Fisheries adjustment programs, and the Ontario mining industry adjustment program. There are potential difficulties with maintaining consistency with program goals and rules, but these can be overcome through appropriate governance mechanisms. Long-term data collection on impacts may also be somewhat more difficult, but this is a problem no matter what model is employed.

Provincial support. Another suggestion made was to elicit greater support from the provinces up-front. In Ontario, the Heritage Fund (the provincial equivalent to FedNor) had a lot of discretion with their funding, but a forestry initiative never materialized. FedNor and WD officials noted that it was difficult to get provincial support for the projects, and that in smaller communities especially it was difficult to mobilize the communities and to obtain the 50 percent co-funding without this provincial support. It is important to garner federal/provincial/municipal collaboration up front, in order to get everyone working on the same project; funding is key for small communities.

Large projects. A number of large projects were investigated in the case studies—e.g., the Prince George Airport Authority, the Penticton & Wine Country Visitors Centre, Premium Pellet, and Hupacasath First Nations. Each of these required extensive collaboration among many players and very detailed business planning. However, most of the resulting projects demonstrated broad impacts within the region, and in Prince George these are likely to occur across many communities. (Hupacasath did not exhibit broad impacts, but is likely to provide a strong and sustainable income stream for the proponents.) Further, there are likely to be strong "knock-on" benefits—i.e., long-term benefits to other stakeholders that are not directly tied to the project, but which are linked to them, such as those to all firms which rely upon efficient international air access to Prince George, to wineries and tour operators in the Penticton area, to other wood pellet manufacturers in the case of Premium Pellet, or other economic development ventures which Hupacasath First Nations can pursue with the project revenues. There were other similar cases (e.g., BC's Rails to Trails) which were not investigated in the case studies, but which were also noted by program officials to have had similar impacts. Thus applications for large projects, if supported by strong business cases, appear to be viable mechanisms through which significant economic impacts may occur. Of course, within a fixed budget, program officials will have to make difficult trade-off decisions in terms of how many large versus small projects might be supported.

top of page

9.2 Sharing of Lessons Learned

Evaluation Question

  1. How and to what extent were best practices shared at the SICEAI delivery level with communities; associations (provincial and regional); coordinating/RDA partners; and CFDABC/CFDCs?

Summary

  • Best practices were frequently shared among the RDA offices.
  • Best practices were also shared with at least a third of clients, but it is unclear to what extent these reflect best practices for SICEAI, as opposed to more "general" lessons learned through other economic development initiatives. (Of course, both are valuable.)

Discussion

RDA perspective. The most detailed information is available from BC through the CFDC survey. Sixty-five percent of respondents said there was much, or very much sharing of best practices among the CFDCs via conference calls, meetings, and informal "mentoring". The RDA officers in the other two regions also noted considerable sharing.

Client perspective. Roughly a third of clients commented that their local RDA office had provided considerable support, encouragement, and insight about what was required during the application stage and/or during implementation. However, it is not clear how much of this assistance is specifically regarding the sharing of "lessons learned" from other SICEAI projects—most comments reflect assistance with detailed proposal preparation, who to contact, communications, accounting, partnerships, and the like, all of which are important and may reflect "general" lessons learned through other economic development initiatives. Another third of clients said that little or no information on best practices were shared, while the remainder made no comment at all (which probably means "no"). However, the wording of the question ("Did the SICEAI program officials provide you with any tips about "best practices" from other projects?") is such that a respondent might have been assisted in a "general" way without having had lessons shared from other SICEAI projects.


27 Consisting of about a dozen CFDC representatives, in a similar model to that used in the Fisheries Adjustment Program. (Return to Reference 27).

28 The data in Industry Canada's Softwood Lumber Monitoring Reports were, in fact, used to help SICEAI set its two-year allocations among regions. (Return to Reference 28).

10.0 Conclusions

Program relevance

The SICEAI program was found to be appropriate, as part of a larger government initiative, in that it provided the resources required in order to respond quickly to the layoffs in the forestry industry caused by the U.S. softwood tariffs, and to start planning for community adjustment measures. The community-based delivery model was appropriate, because the RDAs and CFDCs could best respond to the needs of their communities, while allowing Industry Canada to ensure consistency between regions and to remain accountable for the program overall. At the time the program was announced, there were no reasonable alternatives to SICEAI, and the majority of projects showed good incrementality—program funding was a critical seed in allowing most projects to proceed.

Design and Delivery

Although the roles, responsibilities and respective spending authorities between IC and the RDAs under SICEAI were unclear to all concerned early in the program's lifetime, and were one cause of the delay in implementing the program, these difficulties were eventually resolved.

A review of the Terms and Conditions indicated that the program eligibility criteria were appropriate and, in general, appropriately applied. All three regions were very flexible in terms of the nature of projects supported, so long as they clearly fit the intent of the program. However, the program's detailed objectives (e.g., community vs. industry focus), rules, and operating procedures were very unclear to all concerned at the beginning of the program, and communications from Industry Canada on these topics were often confusing and conflicting.

There were three types of marketing and outreach targets set: (1) Towards communities—those that had suffered significant impacts because of the tariffs; (2) On potential clients and projects—especially forestry companies that were SMEs, First Nations communities, and/or value-added projects; and (3) Strategic targets—For example, for outreach to specific regions, or to attempt to generate interest for projects in certain sectors (>e.g., tourism). However, there were no rigidly enforced targets set on the allocation of contributions—project selection was done in an entirely "bottom-up" manner based on the eligibility and quality of the project application. Program targets overall (e.g., jobs by sector, or by region) were not set by IC or by the regions, although an attempt was made in BC to market and encourage client groups to propose projects which fit provincial sector priorities.

Targets for job creation and/or maintenance, timelines, deliverables, and matching funding were set for each individual project. These were set in conjunction with proponents, and were generally seen as reasonable. However, the very short timeframe available for project implementation and completion caused problems for many clients.

With respect to due diligence, a review of the program's Terms and Conditions, and the review of program files conducted for the case studies, indicate that extensive due diligence was conducted within this program. External review of the application procedure in BC supports this conclusion.

Overall, clients were either satisfied or very satisfied with all aspects of service delivery, especially from regional offices and their community-based delivery networks (including CFDCs), and very few complaints were noted.

Preliminary Results and Success

Overall, it appears that the program has had a modest success on mitigating the direct impact of displaced forestry sector workers as a result of the tariffs, although not surprisingly it has not been able to dramatically diminish these effects. (Note that the program was not intended to directly provide assistance to laid-off or displaced workers, or to directly replace these forestry-related jobs). It has been rather more successful in helping communities diversify their economies and increase their capacity for responding to future change and the need for adjustment and diversification.

A reasonable majority of projects are complete and are currently, or soon to be, operational. An important impact of the projects was job creation. About 85 percent of projects were successful in creating jobs or maintaining jobs to date, and about 6,600 jobs in total have been directly created or maintained to date as a result of the program. Newly created jobs are expected to be sustainable over the next three years, actually increasing in number over time as projects come fully on-line. Many other benefits of the projects were seen within the communities including indirect job creation and maintenance, stimulation of new business opportunities, revitalization of neglected business areas, and increased community participation in economic diversification.

Capacity building projects were not common in any of the three regions. It is too soon to judge the impacts of those that were funded.

Cost-effectiveness

This study was not intended to definitively answer whether the program was cost-effective. Instead, the issue is the standard program evaluation question of whether there are improvements in the program design or delivery that could be made that would increase the cost-effectiveness of this program. A full accounting of cost-effectiveness is one issue that might be addressed in a summative evaluation, which would not be appropriate in this situation.

The general answer is that the program appears to be cost-effective. There is little "hard" data available on this point (although it is always a hard question to answer). Most respondents in the RDAs believe the program was cost-effective. However, the cost per job (either directly or indirectly created, and including both job creation and maintenance) is roughly $12,000. If one only considers direct job creation and maintenance, then the cost per job is about $16,000.

The program was very successful in leveraging program funding. Overall, an additional $247 million in external (non-program) funding was raised, representing a leverage of about 225 percent of the $110 million federal funding. The "gross leveraging" of the average project in our client survey was about 230 percent of the program's contribution to the project, while "net leveraging" for the average projects was about 157 percent of the program's contribution. The net leveraging figure represents the additional leveraging that occurred because of the program's existence; i.e., funding that would not have been committed to these projects in the absence of SICEAI. That the net leveraging figure is relatively large helps confirm that the program support acted as a critical seed to help project proponents find funding and implement the project.

Lessons Learned

The study highlighted certain key success factors of the program, mainly the local knowledge of CED, FedNor, and the CFDCs of their communities' needs (in some cases supplemented by additional staff hired specifically to help with program outreach and client support). This local expertise also helped to develop high quality proposals and extend the reach of the program to many smaller and more remote communities. Negative factors included the initial delays in implementing the program, and the lack of appropriate promotional materials.

The evaluation report contains a number of lessons learned, most of which relate to design and delivery of the program. These lessons learned have helped to develop the recommendations for future similar programs, presented in the following section.

11.0 Recommendations for Future Similar Programs

Based on the findings and analysis, which stem from this study, the study team advises that Industry Canada, the RDAs, and FedNor consider the following recommendations when developing future forest industry and rural community adjustment initiatives.

  1. The program scope should be widened to include all communities, which are deemed to be heavily dependent on the forestry industry, and not only those which produce a certain "basket of products". (For example, "forestry-dependent" communities would include those involved in wood harvesting, pulp and paper, etc.)

  2. A decentralized community-driven model using local delivery networks (including CFDCs) is appropriate for government initiatives intended to benefit broadly dispersed clients who will use the funding in highly-divergent projects, especially when these require a substantial community outreach program. Such models have the advantages of better community knowledge and stakeholder input, more flexible response to changing needs, better fit of projects to needs, and faster response time. It also helps build and maintain community economic development capacity (e.g., within community economic development offices, including those within RDAs). There are potential difficulties with maintaining consistency with program goals and rules, but these can be overcome through appropriate governance mechanisms. Long-term data collection on impacts may also be somewhat more difficult, but this is a problem no matter what model is employed. Within such considerations, and where appropriate, consideration of third party delivery models should be made; these may allow government to "piggy-back" onto existing community expertise. Mechanisms and resources should be in place for providing appropriate program administration and oversight related to outreach (e.g., to SMEs, First Nations, smaller and/or more remote communities), and ongoing assistance to clients in matters such as identifying economic development opportunities, preparing appropriate applications, fulfilling eligibility requirements (e.g., environmental assessments), etc.

  3. Program Terms and Conditions should be kept flexible. However, an effort should be made to standardize the degree of flexibility that is exercised by the regions in interpreting program guidelines. This may require greater clarity of program guidelines or more education of these guidelines to the regional offices. In particular, the program's criteria for project and proponent eligibility, as well as the processes for project application, review, selection, and budget allocation, must be clearly tied to the program goals and objectives and additionally must be thorough, objective, and transparent. In particular, there must be no opportunity for contributions to be made which might potentially be seen as subsidies by U.S. or other international interests.

  4. Set a longer program duration from the beginning. This would allow better identification of community needs, better preparation of applications, an easier review process, and a more realistic time frame both for completion of individual projects (especially those requiring Environmental Assessments, which for larger projects may take eight to twelve months), and in the ability to respond to changing community needs over time. It would also help the RDAs and any other delivery agents to maintain in-house program expertise, especially where additional staff need to be hired on a term basis. (Adding year after a year to the program forces the agencies and proponents to do last-minute planning and project allocations.)

    Note that if a longer-term program is employed, the design criteria should consider that it may require additional financial and (perhaps) human resources. For example there may be more outreach required, a higher load of projects, larger and/or longer timeframe projects, follow-on projects which build upon the success of earlier work, additional "hands-on" assistance for integrated community economic development and/or to assist in developing complex projects to meet federal program guidelines (as well as other requirements such as Environmental Assessments, if applicable), etc. This would be especially true if the program eligibility requirements were less restricted than SICEAI's were.

  5. IC needs to be able to anticipate problems within the industry, and begin a process of adjustment sooner, rather than having to wait for a crisis and huge number of layoffs to be the trigger to initiate a program.

  6. Industry Canada should consult extensively with FedNor and Regional Development Agencies at the early design stage of such programs. In particular, the details of goals and objectives, outreach and communications, project and applicant eligibility, co-funding, the application and review process (including selection criteria), delivery roles and accountabilities, signing authorities, and implementation guidelines should be carefully thought out before attempting to roll out the program in the communities. (Not doing so can result in unrealistic expectations, substantial delays, and a huge application review burden for the RDAs.)

  7. Similar early consultation with the provinces, and increased efforts to engage the provinces by demonstrating the joint aspects of the issues, may result in greater federal/provincial collaboration and an increased ability to obtain co-funding.

  8. The allocation of contributions should respond to community needs, as they currently exist, rather than preconceived ideas or targets. That is, while strategic targeting is entirely reasonable with respect to outreach (e.g., to encourage and/or help specific types of needy clients to apply, or encourage particular types of clients or projects29), the processes for project selection and funding allocation should be made "bottom-up", entirely on the basis of fit to eligibility criteria and the quality of the application.

  9. A simpler model would offer reduced effort by clients and government alike, more transparency, and easier communications. Industry Canada should investigate simpler models than the use of OGD suspense accounts; e.g., those which assig n more responsibility and accountability within the RDAs, including responsibility for repayables.

  10. A simple, long-term performance monitoring system should be instituted which follows-up with 100 percent of clients on an annual basis for at least five years following completion of each project. This will allow accurate, robust, and defensible data to be obtained on the true long-term impacts on job creation and maintenance. The follow-up could be done by the RDAs and FedNor.

    Such long-term data would greatly help in designing future initiatives. For example, the SICEAI data suggest that newly-created jobs and revenues appear to be more sustainable over the long term than existing jobs which have been maintained by the program. However, the data are not robust enough to be certain. If this finding were confirmed (or found false) through long-term monitoring, this would have important implications for future adjustment programs. See also recommendation #11.

  11. A dedicated cost-effectiveness study of the economic benefits resulting from SICEAI and other similar initiatives should be conducted. This would be of great interest to Industry Canada, RDAs, FedNor, the CFDAs, and other economic development organizations for any individual program. It would also allow far better comparison of the benefits and costs of different program delivery models, so long as the exact same methodologies were used to compare different programs.

    The data collected in the monitoring program discussed in recommendation #10 should be of a type to allow such a study to be carried out. For example, if one wishes to understand sustainable benefits versus costs, then on the benefits side there should be firm data on the incrementality of the jobs created, their longevity, indirect impacts occurring more broadly through the communities, and any displacement effects. On the cost side there should be thorough analysis of all costs; i.e., not just including direct contributions, but also real administrative and oversight costs (including "headquarters" costs, effort expended by individuals and communities on a voluntary basis, etc.)


29 And, of course, "targeting" in terms of formal eligibility criteria (e.g., communities hard-hit by the U.S. softwood tariffs, in SICEAI's case) is entirely appropriate. (Return to Reference 29).

Management Response to Program Evaluation of SICEAI

March 30, 2007

Management Response to Program Evaluation of SICEAI
Recommendations Management Response Timing Lead

Recommendation #1: 
The program scope should be widened to include all communities, which are deemed to be heavily dependent on the forestry industry, and not only those which produce a certain 'basket of products'.

Agreed. 
To be considered in future program design decisions where recognition of other needs to address community assistance is required.

When required

Resource Processing Industries Branch (RPIB)

Recommendation #2: 
Consideration of community-driven, third party delivery models should be made; these may allow government to 'piggy-back' onto existing community expertise. Mechanisms and resources should be in place for providing appropriate program administration and oversight related to outreach (e.g., to SMEs, First Nations, smaller and/or more remote communities), and ongoing assistance to clients.

Agreed. 
FedNor employs the Community Futures Development Corporation (CFDC) organizations to deliver its programs routinely. FedNor did not use the CFDC network to deliver SICEAI. Western Economic Diversification (WD) opted to utilize the CFDC network for the SICEAI program as a means of establishing an on-the-ground delivery capacity with direct representation throughout rural British Columbia where funding recipients are located. The use of CFDCs and other local, non-government delivery partners must entail the provision of funds for outreach, client assistance, administration activities to ensure community participation, and local oversight and reporting consistency in project delivery and in the measuring/monitoring of impacts of projects.

When required

RPIB

Recommendation #3: 
Program Terms and Conditions should be kept flexible. In particular, the program's criteria for project and proponent eligibility, as well as the processes for project application, review, selection, and budget allocation must be clearly tied to the program goals and objectives and additionally must be thorough, objective, and transparent. As well, there must be no opportunity for contributions to be made which might potentially be seen as subsidies by U.S. or other international interests.

Agreed. 
In the case of the SICEAI program, Industry Canada's only function was to administer funds which were distributed to Regional Development Agencies (RDAs) and FedNor for program delivery. The RDAs and FedNor selected and managed projects within the fairly general guidelines that were provided, based on their extensive experience in the sector and their region. Future program delivery would likely benefit from more clarity in program guidelines.

When required.

RPIB

Recommendation #4: 
Set a longer program duration from the outset. This would allow better identification of community needs, better preparation of applications, an easier review process, and a more realistic time frame for completion of individual projects (especially those requiring Environmental Assessments.)

Agreed. 
Length of program duration is a key aspect which will be taken into consideration in future program design decisions for this sector.

When required

RPIB

Recommendation #5: 
Industry Canada (IC) needs to be able to anticipate problems within the industry, and begin a process of adjustment sooner, rather than having to wait for a crisis and huge number of layoffs to be the trigger to initiate a program.

Agreed. 
The Forest Industry Directorate (FID) of RPIB is working to improve its level of industry knowledge, including developing provincial and sub-sector profiles for the forest sector as well as increasing liaison within the federal system so as to provide more informed advice to senior departmental managers.

Ongoing

FID/RPIB

Recommendation #6: 
IC should consult extensively with FedNor and Regional Development Agencies at the early design stage of such programs.

Agreed. 
In any future program design IC will consult extensively with FedNor and Regional Development Agencies at the early design stage.

When required

RPIB

Recommendation #7: 
Similar early consultation with the provinces, and increased efforts to engage the provinces by demonstrating the joint aspects of the issues, may result in greater federal/provincial collaboration and an increased ability to obtain co-funding.

Agreed. 
The provinces and territories are consulted regarding the issues confronting the forest sector and their views would be taken into consideration in new programming design decisions.

When required

RPIB

Recommendation #8: 
The allocation of contributions should respond to community needs, as they currently exist, rather than pre-conceived ideas or targets.

Agreed. 
Decisions as to which projects to support would be made by the RDAs and FedNor that reflected community-driven needs and priorities that met program and due diligence requirements.

When required

RPIB

Recommendation #9: 
A simpler model would offer reduced effort by clients and government alike, more transparency, and easier communications. Industry Canada should investigate simpler models than the use of OGD suspense accounts; e.g., those which assign more responsibility and accountability within the RDAs, including responsibility for repayables.

Agreed. 
Design of future programs would take a simpler model into consideration.

When required

RPIB

Recommendation #10: 
A simple, long-term performance monitoring system should be instituted which follows-up with 100 percent of clients on an annual basis for at least five years following completion of each project. This will allow accurate, robust, and defensible data to be obtained on the true long-term impacts on job creation and maintenance. The follow-up could be done by the RDAs and FedNor.

Agreed. 
The five year post-project completion monitoring period would likely exceed the life of the program. Funds for this type of monitoring would have to be identified from other sources within each RDAs, FedNor or department. The monitoring period should be examined and determined during the design stage of future programs.

When required

RPIB

Recommendation #11: 
A dedicated cost-effectiveness study of the economic benefits resulting from SICEAI and other similar initiatives should be conducted. This would be of great interest to Industry Canada, RDAs, FedNor, the CFDA, and other economic development organizations for any individual program.

Agreed. 
Assuming that funding is made available for post-completion monitoring of projects as well as funding for the necessary analysis involved in the conduct of a cost-effectiveness study, such a study could be feasible

When required

RPIB

Long Description

Figure 1: Work Plan for the SICEAI Formative Evaluation

At the top of the image is a box containing the words "1. Steering Committee Meeting" An arrow leads down from that to the next box that reads "2. Preliminary Document Review." An arrow leads down from that to the next box that reads "3. Design Review." An arrow leads down from that to the next box that reads "4. Steering Committee Meeting (Design Report Review)." An arrow leads down from that to the next box that reads "5. Detailed Document Review." A series of arrows then leads to 5 boxes, each beside each other. They read as follows (from left to right): "6. Project File Review", "7. interviews of SICEAI Officials.", "8. CFDC Survey", "9. Client Survey." and "10. Mini Case-Studies." These five then have arrows going down to the next box that reads "11. Preliminary Findings Report." An arrow leads down from that to the next box that reads "12. Steering Committee Meeting (Preliminary Findings Review)." An arrow leads down from that to the next box that reads "13. Draft Final Report." An arrow leads down from that to the next box that reads "14. Steering Committee Meeting (Draft Final Report Review)." From that, an arrow leads to the final box that reads "15. Final Report."

Return to Figure 1

top of page

Figure 2: Distribution of Projects by Contribution Level

The image shows a Bar Graph with the heading "Distribution of Projects by Contribution level." It depicts what percentage of projects received what level of funding, and the chart shows the levels for BC, Ontario, and Quebec. The first level of funding was Less than $100,000. Ontario is at approximately 62%, Quebec has approximately 65% and BC has approximately 35%. The next level of funding is $100,000 to $999,999. Ontario is at approximately 20%, Quebec has approximately 32% and BC has approximately 49%. The next level of funding is $500,000 to $499,999. Ontario is at approximately 16%, Quebec has approximately 2% and BC has approximately 8%. The final level of funding is $1 Million or more. Ontario is at approximately 2%, Quebec has approximately 1% and BC has approximately 8%.

Return to Figure 2

top of page

Figure 3: Distribution of Projects by Project Type

The image shows a Bar Graph with the heading "Distribution of Projects by Project Type." It depicts what percentages of projects were in place based on the type of the project, and the chart shows the levels for BC, Ontario, and Quebec. The first project type was "Capacity Building" Ontario was approximately 28%, Quebec was approximately 11%, and BC was approximately 14%. The next project type was "Diversify within forestry sector" Ontario was approximately 18%, Quebec was approximately 21%, and BC was approximately 11%. The final project type was "Diversify outside forestry sector" Ontario was approximately 54%, Quebec was approximately 68%, and BC was approximately 75%.

Return to Figure 3

top of page

Figure 4: Distribution of Funding by Project Type

The image shows a Bar Graph with the heading "Distribution of Funding by Project Type." It depicts what percentage of funding was given based on the type of the project, and the chart shows the levels for BC, Ontario, and Quebec. The first project type was "Capacity Building" Ontario was approximately 12%, Quebec was approximately 7%, and BC was approximately 4%. The next project type was "Diversify within forestry sector" Ontario was approximately 44%, Quebec was approximately 17%, and BC was approximately 28%. The final project type was "Diversify outside forestry sector" Ontario was approximately 44%, Quebec was approximately 76%, and BC was approximately 68%.

Return to Figure 4

Date modified: