National Evaluation of the Softwood Industry and Community Economic Adjustment Initiative (SICEAI)

7.0 Findings on Cost-Effectiveness

7.1 Cost effectiveness

Evaluation Question

1. Is SICEAI the most cost-effective way to achieve the stated objectives? If not, what could be improved?

Summary

• The cost per job (i.e., all jobs, either directly or indirectly created, and including both job creation and maintenance) is roughly $12,000. If one only considers direct job creation and maintenance, then the cost per job is about $16,000. On this basis, the program appears to have been very cost-effective. Most respondents in the RDAs believe the program was cost-effective.
• The major mechanism suggested that might improve program efficiency was elimination of the Other Government Department (OGD) suspense accounts mechanism, discussed in detail in Section 9. No other significant changes that bear directly on this issue were suggested by respondents.
• There is little "hard" data available on this point (although it is always a hard question to answer), and the study was not intended to definitively determine whether the program was costs effective or not. Instead the study determined whether there are any improvements to design and delivery that would improve cost-effectiveness. A full accounting of incremental benefits and costs would be a great help to Industry Canada, FedNor, and the RDAs in understanding the impacts (including the true cost-effectiveness) of SICEAI. If applied to other similar programs, this would also help identify the impacts of different program models.

Discussion

First, note that this study was not intended to definitively answer whether the program was cost-effective or not. Instead, the issue is the standard program evaluation question of whether there are improvements in the program design or delivery that could be made that would increase the cost-effectiveness of this program. A full accounting of cost-effectiveness is one issue that might be addressed in a summative evaluation, which would not be appropriate in this situation.

The majority of program officials believe that the program was cost-effective, mostly because it was delivered through the regional agencies, which already have the staff and expertise, the information resources, and the infrastructure in place to deliver the program. In BC, the regional CFDC network has experience working collaboratively on a variety of programs and projects, and also has strong communications networks in place within the communities. With only a limited number of additional operating resources, the agencies were able to augment their capabilities to handle the program quite easily. Many officials stressed the importance of using the locally-based agencies (and CFDCs in BC) when delivering future community-based programs.

Some program officials commented increased cost effectiveness could have been achieved by de-centralizing the program to the agencies, as for the mining and fisheries adjustment programs referred to earlier. This would have reduced delays, and therefore saved money. One official suggested that a competitive, timed intake for projects may have required less staff (thus being slightly more cost-effective), but felt that the overall value would be less.

The approximate cost of job creation is shown in the table below. The cost per job (i.e., all jobs, either directly or indirectly created, and including both job creation and maintenance) is roughly $12,000. If one only considers direct job creation and maintenance, then the cost per job is about $16,000²⁴. (Not shown in the table is that if one only considers direct job creation — i.e., ignoring all jobs maintained, and all indirect job impacts — then the cost per job created to date is about $43,000, falling to about $28,000 in three years' time as the projects become fully operational. However, considering only direct job creation would obviously ignore a substantial portion of the program's impacts.)

Job Creation and/or Maintenance Costs

	Cost per Job  to Date	Cost per Job  Est. in 3 Year's Time
The figures in the table above are estimated from the "conservative extrapolations" discussed in Section 6.3.
Jobs directly  created & maintained only	$16,571	$16,248
Total all jobs  (direct & indirect, created & maintained)	$12,239	$12,308

The study team comments that the SICEAI program, like many community-based economic development programs, had a number of client-assistance features that have implications for administrative and oversight costs. These include the extensive outreach program to ensure affected communities knew about the program, the coordination and outreach officers hired both for communications and to assist communities and proponents to prepare applications, the multiple steps often taken to ensure clients met the complex eligibility requirements, the extensive review process employed in BC, the "extra mile" often taken to create the flexibility needed to support an important project that was somewhat outside the norm, and the sometimes remote nature of (and high cost of getting to) the communities in all three regions (especially those of First Nations). In addition, WD, CED, CFDCs, and FedNor were often intimately involved in the sometimes onerous and lengthy environmental assessments required. Many of the larger projects have literally hundreds (and probably thousands) of pages of paperwork and correspondence associated with them. Finally there were costs (probably mostly hidden) of the lengthy process of simply designing and negotiating the program in the first place.

These activities mean higher administrative and oversight costs than would probably be the case in a highly-centralized and more rigid program delivery model. The other side of the coin, however, is that this model allows projects that are custom tailored to each individual situation and thus — while administrative costs may be higher — it is likely that the resulting benefits are as well.

The question then arises as to exactly what the cost-effectiveness of SICEAI (and similar programs) is. A defensible analysis of this issue requires two things: (1) accurate data on the true total costs of the program, including any "hidden" costs; and (2) accurate data on the true long-term impacts of the program, including indirect benefits (this being the "effectiveness" portion of the equation). Each is non-trivial to obtain:

Benefits. Data required include:

• Data on all projects. That is, contact as close to a census of projects as possible, rather than relying on an extrapolation of data from a random sample of projects. Often it is the case that only a small proportion of projects provide the "lion's share" of the benefits — missing just one or two of these projects may result in a dramatic under-estimation of the true total benefits. If a sample must be used, it should be done by picking as many of the most successful projects as possible and using the sum of their benefits as a lower bound, without extrapolation ^25a.
• The true number of sustainable jobs created and/or maintained. This first requires defining what "sustainable" means — i.e., over what time period. This ideally requires understanding trends over time, such as job numbers from project completion through to, say, ten years afterwards. However, even information collected two or three years following project completion would be of great help, as it would identify projects that are still in operation vs. terminated, and which are expanding or contracting.
• Direct and indirect jobs created and/or maintained. Many SICEAI projects were said to have been responsible for indirect job creation, and some of these impacts were expected to become quite large in future years. This should be documented.
• Analysis of incrementality. This is especially important for indirect benefits, as typically many other factors have been partially responsible for these impacts.
• Analysis of any displacement and/or substitution effects. This is necessary for a complete understanding of job impacts.

Costs. The data required include:

• Total direct project contributions.
• Direct costs to project proponents (including any costs of borrowing, legal fees, etc.);
• Administrative and other non-project costs for all community, government, and regional delivery agencies. These data should include any hidden costs that may not be directly accounted for in "line items" associated with the program, such as administrative and oversight costs for agency "headquarters", unpaid overtime, etc.
• Any other "hidden" costs associated with, for example, volunteer effort expended by individuals and communities.

A full accounting of hidden costs is especially important for decentralized, community-based programs, for which many of these costs are "hidden". Note that many analyses of cost-effectiveness do not take into account all such factors, or assess them in different ways. Thus a fair comparative analysis of different programs should use the same methodology for each program^25b. The study team found that there was great interest in Industry Canada, WD, FedNor, and CED for such information. See recommendations #10 and #11.

top of page

7.2 Due Diligence

Evaluation Question

1. Were the elements of due diligence applied by the SICEAI?

Summary

• The review of Terms and Conditions, and the review of program files conducted for the case studies, indicate that extensive due diligence was conducted within this program. External review of the application procedure in BC supports this conclusion²⁶.

Discussion

The review of program files conducted for the case studies indicate that extensive due diligence was conducted within this program. The following elements were present in the file information for the projects that were examined.

Ontario and Quebec. The project files contained, at a minimum, the initial project proposal, as well as specific deliverables, which were to be provided in order to funding to be released. These included: payment confirmations from other funding sources, if applicable; project budgets and/or business plans; progress reports, photographs and notes of site visits; contracts, purchase orders, receipts and other supporting documentation for expenditures; environmental assessments, if applicable.

British Columbia. All BC projects were subject to an extensive application review process, including a Stage One information and checklist assessment (which included review of eligibility, anticipated benefits, budgets, timelines, community support, potential environmental impacts, market assessment, and management ability), a very detailed assessment of a full project or business plan, a risk assessment, a Due Diligence Report (DDR, which includes assessment of incrementality, viability and sustainability), a Screening Report, an Environmental Assessment, and review of federal and provincial regulatory requirements. Following this, a separate review by the CFDC Quality Review Panel (QRP) was conducted to approve funding and a review by the Minister's Advisory Group (MAG) for fit with the strategic goals of the program. For projects in receipt of repayable funding, projects were also reviewed and approved by Industry Canada. Evidence of these elements was present in the file information for the projects that were examined.

---

²⁴ This figure is quite similar to the $12,000 estimate calculated by program staff, as shown in the table in section 3.6. (Return to Reference 24).

^25a This is analogous to "partial benefit-cost analysis" used in the review of science and technology programs. See Partial Benefit/Cost in the Evaluation of the Canadian Networks of Centres of Excellence Program, A. Dennis Rank and Douglas Williams. Evaluation and Program Planning, Volume 22, No. 1, February, 1999. (Return to Reference 25a).

^25b Ibid.; By collecting information on all these variables, the results can often be compared to those in other reports that used different analytic methods. For example, one could compare results to another analysis that did not include hidden costs simply by ignoring the hidden costs in the program under review. (Return to Reference 25b).

²⁶ PriceWaterhouseCoopers. February 9, 2005. Op. cit.. (Return to Reference 26).