First
Previous
Final Evaluation of the Hydrogen Early Adopters Program
1. Introduction
1.1 Background
The Hydrogen Early Adopters (H2EA) program was launched in late fall of 2003 with the goal of addressing the need to accelerate market adoption of hydrogen technologies and other hydrogen-compatible technologies that facilitate the transition to a hydrogen economy. Through Technology Partnerships Canada (TPC) (since renamed the Industrial Technologies Office (ITO)), support has been targeted on consortia comprised of public and private sector partners to validate these technologies through demonstrations, and showcase Canadian capabilities. These partnerships have involved integrating hydrogen-compatible technologies in hydrogen production, storage and distribution, demonstrated by end users of stationary and mobile applications.
top of page1.2 Study Objective
The objective of the study is to undertake a final evaluation of the H2EA program based on the evaluation issues and research questions, and performance measures identified in the H2EA Results-Based Management and Accountability Framework (RMAF), 2003. The evaluation is responding to a Treasury Board Secretariat requirement and was conducted in accordance with Treasury Board policies.
As the program will be sunsetting on March 31, 2008, the report does not include recommendations. The focus of the evaluation is on program level results determined through an examination of program experience at the project level.
top of page1.3 Study Approach
The study approach involved consideration of the evaluation issue areas of relevance, program design and delivery, program success, and cost effectiveness. The approach used three lines of enquiry: document review; interviews with representatives of TPC/ITO, Industry Canada, Natural Resources Canada (NRCan), consortium members and industry associations; and case studies of the four funded H2EA projects.
The evaluation was managed by the Audit and Evaluation Branch (AEB) of Industry Canada. Hickling Arthurs Low (HAL) Corporation was engaged by Industry Canada (IC) to conduct this evaluation of the H2EA program. The study was conducted in two phases, Phase 1 covering two funded projects (Hydrogen Solution for Utility Vehicles, and Stationary Fuels for Residential Heating and Power Generation) completed or nearing completion in FY 2006–2007, and Phase 2 covering the remaining two funded projects (Hydrogen Internal Combustion Engine (H2ICE) Shuttle Bus Demonstration, and Integrated Waste Hydrogen Utilization Project) ongoing through FY 2007–2008.
Progress was reported to an Evaluation Study Steering Committee for feedback during the course of the study (see Appendix A for a list of Steering Committee members). This final report on the H2EA evaluation incorporates the results of the Phase 1 evaluation, submitted in a preliminary report on June 8, 2007, as well as the results of the Phase 2 evaluation.
top of page1.4 Study Methodologies
The lines of enquiry are as follows:
1.4.1 Document review
Documents reviewed included:
- H2EA RMAF and Risk-Based Audit Framework (RBAF)
- H2EA Six Month Progress Report
- H2EA Terms and Conditions
- Program Audit of H2EA
- Key Performance Indicators for H2EA
- Project documents and files: contribution agreements, proposals, progress reports, annual reports, communication documents
- Press articles and releases
1.4.2 Consultations
Interviews were held with representatives of the interview groups listed in Table 2; the list of interviewees is given in Appendix B. Completed interviews numbered 42 for Phases 1 and 2 covering TPC/ITO staff, Industry Canada and NRCan, project lead firms, consortia members, suppliers, supporters, and unfunded projects. The companies and organizations interviewed included those given in Chapter 3 (Project Information) with the addition of the lead companies for four unfunded projects, Ballard (Vancouver), Solar Hydrogen Energy Corporation (Saskatoon), University of British Columbia (Vancouver) and Air Liquide (Montreal), as well as representatives of the Canadian Hydrogen Association, Hydrogen & Fuel Cells Canada, and the Hydrogen Village Project.
The questions included in the interview guides were based on the RMAF and, together with the list of proposed interviewees, were reviewed with AEB and program representatives beforehand. Consultations with representatives of TPC/ITO, Industry Canada and NRCan, located in Ottawa, were conducted in-person, as were interviews with project participants contacted as part of the site visits supporting the case studies; other interviews were by telephone. The consultations were semi-structured, allowing for effective probing of questions set out in interview guides. Respondents were assured that their responses would be aggregated and that individual comments would not be included in the report without their prior consent.
| Phase 1 | ||
|---|---|---|
| Interview Groups | Number of Interviews | |
| Project | Fuel Cell Technologies |
Hydrogenics |
| TPC/ITO Staff | 1 | 1 |
| Regional Agencies/Other Government Departments | 1 | 1 |
| Project Lead Firms | 1 | 1 |
| Consortia Members | 1 | 3 |
| Suppliers | 1 | 1 |
| Supporters | 3 | 3 |
| Project Totals | 8 | 10 |
| Unfunded Projects | 4 | |
| Phase 1 Total | 22 | |
| Phase 2 | ||
|---|---|---|
| Interview Groups | Number of Interviews | |
| Project | Sacré Davey |
Ford |
| TPC/ITO Staff | 2 | 1 |
| Other Government Departments | 2 | 1 |
| Project Lead Firms | 1 | 1 |
| Consortia Members | 2 | 1 |
| Suppliers | 4 | 2 |
| Supporters | 2 | 1 |
| Project Totals | 13 | 7 |
| Phase 2 Total | 20 | |
1.4.3 Case Studies
Case studies were prepared on the two Phase 1 and two Phase 2 projects (i.e. the universe of funded projects) to provide a more in-depth understanding of the rationale for the projects, roles and responsibilities of participants, implementation issues, objectives achievement (results), and cost-effectiveness considerations. As noted, site visits were undertaken as part of the case studies to provide a fuller picture of how the program has been implemented, including issues of program design, how well the projects moved towards planned objectives, and what barriers had been encountered. The four case studies are included as Appendix C.
top of page1.5 Study Limitations and Mitigation Measures
The four H2EA projects that we examined were not all at the same stage of implementation at the start of the study and therefore could not all be evaluated at the same time. To mitigate this limitation, the study was divided into two phases with Phase 1 dealing with the two projects that were to be completed in FY 2006–2007 and Phase 2 with the two projects to be completed in FY 2007–2008. The result of the two phased approach was a lengthening of the study; to maintain continuity through the gap in the two phases, a preliminary report was produced on the first phase results.
Another broad concern for the study was the small number of projects and therefore the possible limitations on generalizing on the results for the whole Canadian hydrogen and fuel cell industry. That said, the companies involved in the projects are of varying size representing a reasonable cross section of the industry including fuel supply, fuel storage, and fuel usage both stationary and mobile in the public and private sectors. As well, the projects were all studied in detail through the case study approach. Therefore, we believe that the results are indicative of how the industry as a whole would have responded to the H2EA program including the issues faced and the lessons learned.
A more specific concern relates to the timing of the evaluation. The demonstrations of the two Phase 2 projects have not been running long enough to allow for a full assessment of project results. The evaluation, however, revealed no evidence of diminishing performance in the projects. It is expected that the evaluation of the projects at this stage is a fair representation of the situation of the projects at the end of the demonstration period which is expected to be approximately one year after the H2EA program ends.
A final potential limitation of the study was the winding down of a project partner in Phase 1 while the evaluation was underway. While this difficulty complicated the preparation of the project case study, the necessary information and interview responses were eventually obtained to enable an assessment to be made. No other difficulties in accessing interviewees or data were encountered.
Overall, it is believed that the evaluation findings and conclusion are reliable and represent an accurate assessment of the program.
- Date modified:
