Evaluation of the Sustainable Cities Initiative

3.0 Methodology/Approach

3.1 Evaluation Issues

Four issues were established for this evaluation:

• to what extent is there a need/relevance for the Sustainable Cities Initiative?
• to what extent has SCI been successful in reaching its planned objectives?
• to what extent is the program cost-effective?
• what lessons have been learned?

Specific evaluation questions were drawn from the draft Results-based Management and Accountability Framework (RMAF) that was developed for the program and are listed in Annex A.

3.2 Methodology

The information required to address the evaluation issues was obtained from a variety of sources. These included:

• a review of documentation including SCI program information and reports, federal government policy documents and reports, studies undertaken of the SCI pilot or similar initiatives by international organizations, reports from international forums, annual reports of international organizations, etc. A detailed listing of all the documents reviewed in the course of this evaluation is provided in Annex B, Bibliography. A summary of projected and realized results for seven SCI cities is provided in Annex C. The information for this summary was drawn from Project Tracking Sheets.
• interviews with key stakeholders: SCI current and previous staff, members of the SCI Business Advisory Committee, representatives from other federal government departments including staff from Canadian embassies in countries with a SCI city, partners from Canadian companies and NGOs, and local representatives from SCI cities provided input to the evaluation. E-mails were sent to one hundred and thirty-three prospective interviewees identified by SCI management and staff, inviting them to participate. Forty-one individuals self-selected themselves through this process. Twelve current and former SCI staff members were also interviewed. (See Annex D for a list of those interviewed and Annex E for the interview matrix employed).
• a web based survey of Canadian partners. Emails were sent to approximately 1,500 individuals whose name is on a database maintained by SCI. Two hundred and eight responses were received. (See Annex F for a copy of the survey employed.)
• the results of an audit that was undertaken concurrently with the evaluation.

3.3 Limits of Methodology

It is recognized that the methodology utilized has limitations. It is important in an evaluation to obtain the perspective first-hand from the beneficiaries of the program, e.g., the Canadian organizations who were exporting Canadian sustainable development technologies and services, and the participating cities who were endeavouring to improve the quality of life and promote sustainable development within their city. Direct evidence is considered more reliable than second hand information (e.g., the perspective of program management and staff). Despite several attempts to solicit a greater participation in the interview process by local representatives of the participating cities, the level of participation was too low to make any definitive statement about that stakeholder group.

While input from participants is required, it is generally considered less reliable than other potential measures such as the dollar value of increased Canadian exports, number of trade opportunities for participating organizations in comparison to the number of opportunities for comparable organizations over the same period, or the number of business partnerships formed among participating organizations in comparison to those formed by comparable organizations over the same period. People who self-select to participate in interviews or surveys of programs such as SCI, tend to be generally positive about the experience, especially when they are provided with funding to carry out a desired activity. Notwithstanding these limitations in stakeholder input, it is often the only source of information that can be readily obtained within a reasonable timeframe or at a reasonable cost.

Such was the case with this evaluation. Hard data on elements such as the number of identified proposals, number of Canadian organizations involved in projects, type of activities, number of Canadian partners at initial and subsequent visits to the cities, and the number of meetings with International Funding Institutions, is available in the technical mission books, trip reports, project tracking sheets and project budget tracking sheets. A significant effort beyond the scope of this evaluation, however, is required to extract the relevant information so that it would be in a useful format for management decision making.

Conclusions could also not be drawn from the size of the database maintained by SCI. At the time of the evaluation, in January 2005 it contained approximately 1,500 names. While the Departmental standard was implemented, Microsoft Access' file structure is not conducive to tracking when records were added or the nature of changes made. This information is necessary to track growth over time. The growth in the number of organizations represented on the database would have been more meaningful information than the number of names per se. Based on the information we were provided with, there were at most 900 different organizations represented on the database. Organizations represented included private sector companies, NGOs, universities, municipalities, provincial government departments and federal government departments and agencies.

Approximately 15% of the names on the database are Government of Canada employees with CIDA, Canada Mortgage & Housing Corporation, Canadian Heritage, Environment Canada, Foreign Affairs, Health Canada, Indian & Northern Affairs Canada, Industry Canada, International Trade, Natural Resources Canada, or Public Works & Government Services Canada. SCI management and staff were included on the database. Many of these individuals are key partners in the delivery of SCI and played a critical role in its delivery but the program was not targeted at them. To include them in a measure that is attempting to demonstrate increasing exports of Canadian sustainable development technologies and services or the promotion of market development and trade opportunities for an increasing number of Canadian firms, is in our opinion, inappropriate.

Further, more than 15% of the emails that were sent out using the database as part of the evaluation, were returned as undeliverable. Keeping addresses up-to-date is the bane of anyone who has to maintain any form of mailing list. Nonetheless, various methods need to be employed to keep mailing lists as accurate as possible to maximize the number of individuals/organizations who will receive messages on program initiatives and activities. A name that messages do not reach is of no value. Others responded to the survey indicating that they didn't know why it had been sent to them as they have no involvement with SCI.

Despite these limitations with the methodology, the evaluation is considered to provide a good assessment of the stated evaluation issues. The tendency of participants to be naturally positive about a program they are participating in is tempered by the fact that Canadian partners, in particular those from the private sector, had to pay to participate. They had to cover their own travel costs for the Roadmap sessions and the salaries of the staff who participated. If the private sector doesn't see a value in what they are doing, they are unlikely to continue for very long.