Mid-Term Evaluation of Industry Canada's Sustainable Development Strategy, 2003-2006 (SDS III)
VI. Effectiveness of SDS III
The focus of this chapter of the report is on the cost-effectiveness of the approach that Industry Canada has followed in delivering the SD action items; the effectiveness of the reporting system for SDS III; and constraints encountered in implementing the action items.
This section deals with the following evaluation question:
Are the SDS III initiatives cost-effective?
The operating costs of SDS III action items were mostly rolled into existing budgets of branches. Some responsibility centres provided estimates of level of resources (person years and funds) that would be needed to deliver their respective action items. The following table summarizes this information for twelve action items.
The average person-years from this sample is just over 1 person-year per action item. Estimated funds range in the $100+K, per action item.
|Bio-based Industrial Products||2.0||3.0||3.0||250||250||200|
|Climate Change Technologies||1.5||1.5||1.5||267||267||267|
|CSR Management Guide||0.5||0.5||0.5||45||45||45|
|Electronic Products Recycling||0.5||0.3||0.3||15||15||15|
|Environmental Stewardship in SMEs||1.5||1.5||1.5||100||100||100|
|Canadian Green Chemistry Network||1.0||1.0||1.0||25||25||25|
|Research Studies on SD||0.5||0.5||0.5||30||45||45|
|SD Policy Frameworks||2.0||2.0||2.0||20||20||20|
|Supply Chain Management Tools||0.5||0.5||–||250||200||–|
These figures do not include larger action items associated with major program expenditures such as those of Precarn and the Sustainable Cities Initiative (SCI), or the Canadian Foundation for Innovation. Precarn estimated about $3 million a year worth of R&D funding going to SD-related projects that it supports, administered through 1 person-year. SCI estimated in the range between $1 million and $4 million a year of its funding going to SD-related activities relevant to SDS III, for over 16 designated sustainable cities.
It is generally concluded, for example from previous separate evaluations of Precarn and Canadian Foundation for Innovation, that these programs compare very-well with other federal grants and contributions programs in terms of cost-effectiveness. This is essentially because, for a relatively small number of person-years, accounted for in operating budgets, a large amount of program funds are disbursed on R&D with a significant amount of leverage for matching funds from the private sector, and from other levels of government. SDS III is in a sense "piggy-backed" onto these significant programs, and the benefits accrue from steering some of the funded project activities to sustainability issues and related technology development.
Interviews with responsibility centres, for the above action items in Exhibit 16, suggest that SDS III has been a very cost-effective delivery model for IC. For example, one responsibility centre remarked "… all of our initiatives are cost-shared and we work closely with our partners. For that reason, I can not think of any more effective approaches to be taken." Moreover, several interviewees have suggested that the roughly 1.2 to 1.3 person-year average per action item, estimated in Exhibit 16 above, may even have been too high an estimate, in the actual course of implementation of the action items. Many of the responsibility centres have multiple responsibilities, their SDS III projects being only a part of their full complement of roles and duties. The real PYs per action item, on average, may be well less than 1 PY.
6.2 Assessment of Reporting System for SDS III
Accountability requirements — Industry Canada’s accountability reporting system for SDS III has substantially improved from previous SD strategies. The new monitoring system implemented during SDS III has allowed for a more systematic approach, enabling IC officials to effectively roll-up results and provide both succinct and comprehensive performance reports on targets and activities to the Deputy Minister. Unlike previous evaluations (SDS I and SDS II), none of the SDS III responsibility centres that were interviewed for this evaluation identified any substantial issues with the reporting system. One observation concerned the frequency of reporting (i.e., "make it annual"), and one observation suggested that each action item responsibility centre provide a final "roll-up report" for the three-year period, once the action item is deemed completed. Another suggestion was that IC should consider more "proactive ways to disseminate more information on benefits of SD, and as a result receive more applications to fund SD."
The department also reports on its sustainable development performance in the Departmental Performance Report (DPR), which presents sustainable development accomplishments against planned performance expectations set out in the Report on Plans and Priorities (RPP). IC was able to effectively report on SDS III progress in its most recent DPR for the period ending March 31, 2005. It was further able to provide more detailed SDS III results on performance information by posting the progress of its 32 action items on the web (strategis.gc.ca/sd), thereby fulfilling a commitment it made in the 2002–03 DPR to provide more comprehensive performance information to the public, as recommended by CESD and Treasury Board Secretariat.
SD Website — The SD website has evolved into a robust vehicle for information dissemination on sustainable development, both for the public, for industry, and for updates and information sharing within the department itself. Considerable progress has taken place during SDS III in making the website relevant and timely. The department has also committed to continuing to improve this important vehicle as an enabler for advancing the SD agenda in Canada. As part of the reporting mechanism for SDS IV, website "hits" were monitored by the department. Exhibit 17 shows that from 2004 to 2005, the SD website hits increased from 23,379 to 25,670.
Measuring results — In a 2004 assessment of the quality of departmental strategies by the Commissioner of the Environment and Development, Industry Canada was deemed to be "meeting expectations" in terms of linking its SD goals and objectives with targets and actions.Footnote 23 In terms of having clear and measurable targets it was ranked as "meeting expectations to some degree." In the latter case, IC is among the majority of departments facing the challenge of reporting on results of SD-related actions on high-level indicators such as "impacts on reduction of GHG emissions", "level of influence in providing policy advice", "level of awareness created through information dissemination and outreach efforts", and "strengthened market positioning of renewable energies in Canadian and international markets". These are indicators associated with some of the SDS III action items (see Exhibit 17) that are difficult to report on credibly, on an ongoing basis, without an appropriate monitoring budget and operational approach, at the action item level, to gather relevant evaluative information as activities and deliverables evolve and mature. Moreover, some of these indicators are typically longer-term in nature and difficult to assess in terms of attribution to specific IC SD-related action items, which are mostly implemented with multiple partnerships. During SDS III, out of 32 action items 28 were implemented with several external partners — for example, partners in industry, academia, non-profit organizations, other government departments, provinces and municipalities. The remaining 4 action items were implemented through partnerships within IC, across different branches.
In spite of the challenge of measuring results, as described above, it would be appropriate for the department, and SDS III responsibility centres, to address this issue by attempting to provide more systematic information (including substitute or alternative information if appropriate) on the indicators set-out in Exhibit 18, during the remaining reporting periods of the strategy — or give an explanation why this is not being provided.
As a general comment for developing SDS IV indicators, more care and attention should be paid by responsibility centres in choosing indicators to measure outcomes, and the availability and reliability of data sources for these indicators.
Monitoring templates — A new template for reporting on SDS III action items was introduced in 2004. Officers from responsibility centres across the department had previously experienced technical difficulties with the Intranet-based SD monitoring and reporting system. To ensure a more effective and efficient reporting system for the new SDS, the Strategic Policy Branch (SPB) developed a WordPerfect reporting template based on the structure used in the two-page action items, which underpin each of the strategy's action items. This new template provided some pertinent performance progress indicators, and by most accounts was appreciated by responsibility centres in that it eased the burden and allowed them to enhance the information they provided. Essentially, this system worked because there was a manageable number of action items — 32 in SDS III compared to 58 in SDS II.
Challenge — There is no longer a requirement to incorporate sustainable development planning into the RPP — it is now optional. This has the potential to make it difficult to engage the department on the importance of sustainable development, and on adequately reporting on substantial results being achieved through SD action items.
Out of the thirty-two SDS III action items twenty-two responsibility centres reported no significant barriers in delivering their SD initiatives. Only two responsibility centres indicated staff-turnover was a constraint. Altogether, ten responsibility centres reported the following noteworthy other constraints.
The main constraint indicated by interviewees within Industry Canada is that their SDS III action items are mainly delivered from within existing branch-level budgets, and from funding mechanisms available to them that have to respond to competing priorities and requirements. In other words, there is not a dedicated SDS III budget, earmarked for most of the projects undertaken, directly under the sponsorship of the strategy. Seven of the thirty-two action items indicated resources as a constraint.
Exhibit 18: Indicators for SDS III Action Items
Innovation Towards Sustainable Development *
Capacity building in R&D skills
- Level of influence in providing policy advice and support to CFI project funding decisions
- Perceived influence of support to NCE project applicant assessment and selection process
- Level of awareness created through information dissemination and outreach efforts
- Level of support for the development, attraction and retention of highly qualified people
Promoting technology innovation
- Number of sustainable development projects funded by TPC
- Number of sustainable development projects funded per annum (PRECARN)
- Percent of new light vehicles with improved fuel efficiency
- Increased hydrogen and fuel cell commercialization and early adoption
- Strengthened market positioning of renewable energies in Canadian and international markets
Applying the tools in the marketplace
- Number of technologies identified, and number of industry commitments to technology development and commercialization-related actions arising from a TRM
- Extent of greenhouse gas emissions reductions in buildings
- Number of companies that have implemented innovative manufacturing practices
- Number of companies that have implemented environmental stewardship practices
- Number of companies that have implemented environmental supply chain management tools
- Number of Canadian companies awarded contracts in overseas markets for environmental technologies and services
Corporate and Community Sustainability *
Promoting corporate responsibility and sustainability
- Effective and efficient regulatory regime for environmental protection
- Enhanced knowledge and awareness by Canadian industry of CSR
- Number of Canadian companies implementing CSR management tools and/or practices
- Number of new reporters per annum
- Implementation of stewardship programs across Canada for electronic products
- Level of participation and support for the Media Awareness Network’s Web Awareness Canada initiative and Cybertip
Advancing local and global sustainability
- Number of computers distributed to schools and libraries, and metric tons diverted from landfills to CFS re-use activities
- Number of communities served, equitable access to the Internet
- Number of Aboriginal businesses receiving Aboriginal Business Canada contribution funding
- Development of roadmaps with SCI partner cities, identification and implementation of projects
Sustainable Development Capacity Building Within Industry Canada *
Improving sustainable development planning, performance measurement and evaluation
- Extent of further integration of sustainable development language and commitments in corporate planning documents
- Data on procurement, automotive fleet operations, non-hazardous materials and solid waste
- Completion of Mid-term Evaluation of SDS III
- Completion of SDS IV internal scan
- Development of SDS IV evaluation framework
- Number of participants in courses, lectures and workshops
- Level of senior management support
- Number of participants registered in sustainable development challenges
- Calculated emission reductions
- Fitness level of employees
* These performance indicators were identified by SDS III responsibility centres for their respective action items. These indicators were also published in the SDS 2003-06 strategy document, and the IC 2005-2006 Report on Plans and Priorities.
One of the strengths of most of the SDS III projects is the partnership arrangements that they have developed over a long period of time. Twenty-eight of the action item responsibility centres reported partnerships with external stakeholders (other federal government departments, provincial governments, private and non-government organizations, academia, and other institutions and associations); and four reported internal partnerships within the department, across IC branches. These partnerships have been very effective in all the action items in engendering SD initiatives. On the other hand, the process of collaborating with partners does require a commitment to dialogue and consultation; and this often takes lengthy periods of time, to initiate and launch projects and activities involving multiplicities of interests and interest groups. While this is accepted as part of "doing business" in addressing sustainable development, it does need to be managed in a way that effectively advances the SD agenda. The challenge is that responsibility centres often need to juggle priorities within their individual mandates to address requirements for coordinating partnership activities in an effective way. This is recognized as not always an easy thing to do.
Another constraint suggested by some of the responsibility centres that were interviewed, is that in situations where broad and cross-cutting initiatives are involved, senior federal officials need to participate in the process. Garnering the support of the senior federal officials is often seen as a major challenge, to provide the appropriate support and profile for the SD work they are pursuing, and that IC is committed to.
In the action items involving support to R&D activities (e.g., Precarn, granting councils, NCEs), there is a complexity of bringing research and policy communities across government together, to focus on SD-related concerns and issues. For the project officer managing a related SDS III action item, such as nanotechnology, this is a constraint. For example, the challenge of identifying stakeholders in industry becomes significant, because nanotechnology is a technology, not a recognizable industry from a policy perspective. The potential benefits of nanotechnology on the other hand, as a cross-cutting technology is enormous. It has been estimated that the global nanotechnology market will reach USD 29 billion by 2008.Footnote 24
In action items where negotiations are involved with private sector interests (e.g., negotiating the MOU for GHG emissions reductions in the auto-industry, negotiating contribution agreements for R&D initiatives, negotiating Broadband implementation agreements), slow progress in the initial phases of this process requires long-term commitments to allow for the results to mature. IC has demonstrated great strength in persevering with its contribution to these initiatives, in a way that leads to positive results. Nonetheless, the process in some of the action items involving private sector interests is seen as a constraint by some of the IC officials interviewed.
Finally, in action items where there are split responsibilities between two or more branches, this is seen by at least two responsibility centres as potentially a constraint, in that it could slow down progress of initiatives due to competing priorities. However, for the most part, interviewees suggested that this is just considered as a challenge to overcome than a constraint per se that hampers progress.
- Date modified: