Mid-Term Evaluation of Industry Canada's Sustainable Development Strategy, 2003-2006 (SDS III)

VI. Effectiveness of SDS III

The focus of this chapter of the report is on the cost-effectiveness of the approach that Industry Canada has followed in delivering the SD action items; the effectiveness of the reporting system for SDS III; and constraints encountered in implementing the action items.

6.1 Cost-Effectiveness

This section deals with the following evaluation question:

Are the SDS III initiatives cost-effective?

The operating costs of SDS III action items were mostly rolled into existing budgets of branches. Some responsibility centres provided estimates of level of resources (person years and funds) that would be needed to deliver their respective action items. The following table summarizes this information for twelve action items.

The average person-years from this sample is just over 1 person-year per action item. Estimated funds range in the $100+K, per action item.

Exhibit 16: Sample of Estimated Resources

Action Items	2004	2005	2006	2004	2005	2006
	PYs	PYs	PYs	$'000	$'000	$'000
Bio-based Industrial Products	2.0	3.0	3.0	250	250	200
Climate Change Technologies	1.5	1.5	1.5	267	267	267
CSR Management Guide	0.5	0.5	0.5	45	45	45
Electronic Products Recycling	0.5	0.3	0.3	15	15	15
Environmental Stewardship in SMEs	1.5	1.5	1.5	100	100	100
Canadian Green Chemistry Network	1.0	1.0	1.0	25	25	25
Nanotechnology	1.0	2.0	2.0	150	150	150
Research Studies on SD	0.5	0.5	0.5	30	45	45
SD Policy Frameworks	2.0	2.0	2.0	20	20	20
Supply Chain Management Tools	0.5	0.5	–	250	200	–
Sustainable Manufacturing	3.0	3.0	3.0	80	60	60
Technology Roadmaps	–	–	–	150	150	150
Total	14	15.8	15.3	$1,383	$1,329	$1,080
Average	1.2	1.3	1.3	$115.25	$110.75	$90.00

These figures do not include larger action items associated with major program expenditures such as those of Precarn and the Sustainable Cities Initiative (SCI), or the Canadian Foundation for Innovation. Precarn estimated about $3 million a year worth of R&D funding going to SD-related projects that it supports, administered through 1 person-year. SCI estimated in the range between $1 million and $4 million a year of its funding going to SD-related activities relevant to SDS III, for over 16 designated sustainable cities.

It is generally concluded, for example from previous separate evaluations of Precarn and Canadian Foundation for Innovation, that these programs compare very-well with other federal grants and contributions programs in terms of cost-effectiveness. This is essentially because, for a relatively small number of person-years, accounted for in operating budgets, a large amount of program funds are disbursed on R&D with a significant amount of leverage for matching funds from the private sector, and from other levels of government. SDS III is in a sense "piggy-backed" onto these significant programs, and the benefits accrue from steering some of the funded project activities to sustainability issues and related technology development.

Interviews with responsibility centres, for the above action items in Exhibit 16, suggest that SDS III has been a very cost-effective delivery model for IC. For example, one responsibility centre remarked "… all of our initiatives are cost-shared and we work closely with our partners. For that reason, I can not think of any more effective approaches to be taken." Moreover, several interviewees have suggested that the roughly 1.2 to 1.3 person-year average per action item, estimated in Exhibit 16 above, may even have been too high an estimate, in the actual course of implementation of the action items. Many of the responsibility centres have multiple responsibilities, their SDS III projects being only a part of their full complement of roles and duties. The real PYs per action item, on average, may be well less than 1 PY.

6.2 Assessment of Reporting System for SDS III

Accountability requirements — Industry Canada’s accountability reporting system for SDS III has substantially improved from previous SD strategies. The new monitoring system implemented during SDS III has allowed for a more systematic approach, enabling IC officials to effectively roll-up results and provide both succinct and comprehensive performance reports on targets and activities to the Deputy Minister. Unlike previous evaluations (SDS I and SDS II), none of the SDS III responsibility centres that were interviewed for this evaluation identified any substantial issues with the reporting system. One observation concerned the frequency of reporting (i.e., "make it annual"), and one observation suggested that each action item responsibility centre provide a final "roll-up report" for the three-year period, once the action item is deemed completed. Another suggestion was that IC should consider more "proactive ways to disseminate more information on benefits of SD, and as a result receive more applications to fund SD."

The department also reports on its sustainable development performance in the Departmental Performance Report (DPR), which presents sustainable development accomplishments against planned performance expectations set out in the Report on Plans and Priorities (RPP). IC was able to effectively report on SDS III progress in its most recent DPR for the period ending March 31, 2005. It was further able to provide more detailed SDS III results on performance information by posting the progress of its 32 action items on the web (strategis.gc.ca/sd), thereby fulfilling a commitment it made in the 2002–03 DPR to provide more comprehensive performance information to the public, as recommended by CESD and Treasury Board Secretariat.

SD Website — The SD website has evolved into a robust vehicle for information dissemination on sustainable development, both for the public, for industry, and for updates and information sharing within the department itself. Considerable progress has taken place during SDS III in making the website relevant and timely. The department has also committed to continuing to improve this important vehicle as an enabler for advancing the SD agenda in Canada. As part of the reporting mechanism for SDS IV, website "hits" were monitored by the department. Exhibit 17 shows that from 2004 to 2005, the SD website hits increased from 23,379 to 25,670.

Measuring results — In a 2004 assessment of the quality of departmental strategies by the Commissioner of the Environment and Development, Industry Canada was deemed to be "meeting expectations" in terms of linking its SD goals and objectives with targets and actions.^{Footnote 23} In terms of having clear and measurable targets it was ranked as "meeting expectations to some degree." In the latter case, IC is among the majority of departments facing the challenge of reporting on results of SD-related actions on high-level indicators such as "impacts on reduction of GHG emissions", "level of influence in providing policy advice", "level of awareness created through information dissemination and outreach efforts", and "strengthened market positioning of renewable energies in Canadian and international markets". These are indicators associated with some of the SDS III action items (see Exhibit 17) that are difficult to report on credibly, on an ongoing basis, without an appropriate monitoring budget and operational approach, at the action item level, to gather relevant evaluative information as activities and deliverables evolve and mature. Moreover, some of these indicators are typically longer-term in nature and difficult to assess in terms of attribution to specific IC SD-related action items, which are mostly implemented with multiple partnerships. During SDS III, out of 32 action items 28 were implemented with several external partners — for example, partners in industry, academia, non-profit organizations, other government departments, provinces and municipalities. The remaining 4 action items were implemented through partnerships within IC, across different branches.

In spite of the challenge of measuring results, as described above, it would be appropriate for the department, and SDS III responsibility centres, to address this issue by attempting to provide more systematic information (including substitute or alternative information if appropriate) on the indicators set-out in Exhibit 18, during the remaining reporting periods of the strategy — or give an explanation why this is not being provided.

As a general comment for developing SDS IV indicators, more care and attention should be paid by responsibility centres in choosing indicators to measure outcomes, and the availability and reliability of data sources for these indicators.

Monitoring templates — A new template for reporting on SDS III action items was introduced in 2004. Officers from responsibility centres across the department had previously experienced technical difficulties with the Intranet-based SD monitoring and reporting system. To ensure a more effective and efficient reporting system for the new SDS, the Strategic Policy Branch (SPB) developed a WordPerfect reporting template based on the structure used in the two-page action items, which underpin each of the strategy's action items. This new template provided some pertinent performance progress indicators, and by most accounts was appreciated by responsibility centres in that it eased the burden and allowed them to enhance the information they provided. Essentially, this system worked because there was a manageable number of action items — 32 in SDS III compared to 58 in SDS II.

Challenge — There is no longer a requirement to incorporate sustainable development planning into the RPP — it is now optional. This has the potential to make it difficult to engage the department on the importance of sustainable development, and on adequately reporting on substantial results being achieved through SD action items.

6.3 Constraints

Out of the thirty-two SDS III action items twenty-two responsibility centres reported no significant barriers in delivering their SD initiatives. Only two responsibility centres indicated staff-turnover was a constraint. Altogether, ten responsibility centres reported the following noteworthy other constraints.

The main constraint indicated by interviewees within Industry Canada is that their SDS III action items are mainly delivered from within existing branch-level budgets, and from funding mechanisms available to them that have to respond to competing priorities and requirements. In other words, there is not a dedicated SDS III budget, earmarked for most of the projects undertaken, directly under the sponsorship of the strategy. Seven of the thirty-two action items indicated resources as a constraint.

One of the strengths of most of the SDS III projects is the partnership arrangements that they have developed over a long period of time. Twenty-eight of the action item responsibility centres reported partnerships with external stakeholders (other federal government departments, provincial governments, private and non-government organizations, academia, and other institutions and associations); and four reported internal partnerships within the department, across IC branches. These partnerships have been very effective in all the action items in engendering SD initiatives. On the other hand, the process of collaborating with partners does require a commitment to dialogue and consultation; and this often takes lengthy periods of time, to initiate and launch projects and activities involving multiplicities of interests and interest groups. While this is accepted as part of "doing business" in addressing sustainable development, it does need to be managed in a way that effectively advances the SD agenda. The challenge is that responsibility centres often need to juggle priorities within their individual mandates to address requirements for coordinating partnership activities in an effective way. This is recognized as not always an easy thing to do.

Another constraint suggested by some of the responsibility centres that were interviewed, is that in situations where broad and cross-cutting initiatives are involved, senior federal officials need to participate in the process. Garnering the support of the senior federal officials is often seen as a major challenge, to provide the appropriate support and profile for the SD work they are pursuing, and that IC is committed to.

In the action items involving support to R&D activities (e.g., Precarn, granting councils, NCEs), there is a complexity of bringing research and policy communities across government together, to focus on SD-related concerns and issues. For the project officer managing a related SDS III action item, such as nanotechnology, this is a constraint. For example, the challenge of identifying stakeholders in industry becomes significant, because nanotechnology is a technology, not a recognizable industry from a policy perspective. The potential benefits of nanotechnology on the other hand, as a cross-cutting technology is enormous. It has been estimated that the global nanotechnology market will reach USD 29 billion by 2008.^{Footnote 24}

In action items where negotiations are involved with private sector interests (e.g., negotiating the MOU for GHG emissions reductions in the auto-industry, negotiating contribution agreements for R&D initiatives, negotiating Broadband implementation agreements), slow progress in the initial phases of this process requires long-term commitments to allow for the results to mature. IC has demonstrated great strength in persevering with its contribution to these initiatives, in a way that leads to positive results. Nonetheless, the process in some of the action items involving private sector interests is seen as a constraint by some of the IC officials interviewed.

Finally, in action items where there are split responsibilities between two or more branches, this is seen by at least two responsibility centres as potentially a constraint, in that it could slow down progress of initiatives due to competing priorities. However, for the most part, interviewees suggested that this is just considered as a challenge to overcome than a constraint per se that hampers progress.

---

Footnotes

1. ²³ 2005 Report of the Commissioner of the Environment and Sustainable Development, Chapter 7, Exhibit 7.3. (back to footnote reference 23)
2. ²⁴ ISO Focus: The Magazine of the International Organization for Standardization, Vol. 3, No. 1 January 2006, page 44. (back to footnote reference 24)