Evaluation of the Sustainable Cities Initiative

Final Report

Audit and Evaluation Branch, Industry Canada

May 24, 2005

Tabled and approved by DAEC on September 21, 2006

Table of Contents

• 1.0 Executive Summary
• 2.0 Background
• 3.0 Methodology/Approach
  • 3.1 Evaluation Issues
  • 3.2 Methodology
  • 3.3 Limits of Methodology
• 4.0 Findings
  • 4.1 To what Extent is there a need/rationale for SCI?
  • 4.2 To what Extent has SCI been Successful in Reaching its Planned Objectives?
  • 4.3 To What Extent is the Program Cost-effective?
  • 4.4 What Lessons have been Learned?
• 5.0 Conclusions
• 6.0 Recommendations
• Management Response or
  PDF version (10 KB, 2 pages)

Annexes (Separate document)

(Note: Annexes are available via an Access to Information)

• Annex A: SCI Evaluation Issues and Questions
• Annex B: Bibliography
• Annex C: Projected and Realized Results for Selected Cities
• Annex D: Individuals Interviewed
• Annex E: Interview Matrix
• Annex F: Survey

Note:
Minor editorial changes were made to this report in order to prepare the document for posting to the Industry Canada's Website (including removal of standard Appendices such as list of interviewees and questionnaires). Readers wishing to receive a copy of the original version of this report should contact the Audit and Evaluation Branch at Industry Canada.

1.0 Executive Summary

Background

Under the Sustainable Cities Initiative (SCI), Industry Canada (IC) facilitates a process whereby Canadian companies, non governmental organizations (NGOs) and municipalities, work with cities in developing countries or with an economy in transition, to address a range of urban problems generated by rapid urbanization. Nine million dollars in funding was provided in 2002-03 for a three year initiative to build upon an earlier pilot. The target was to engage 17 cities in total in Africa, Asia, Central America, Eastern Europe and South America.

SCI's overall objective is to increase exports of Canadian sustainable development technologies and services. It also:

• brands Canada as a pre-eminent source of technology, expertise and products for urban sustainable development;
• promotes market development and trade opportunities for Canadian firms;
• enhances the quality of life in developing and emerging economies as well as promotes sustainable development;
• facilitates business partnerships among firms, nongovernmental organizations (NGOs), and government to access market opportunities.

Key Findings

SCI remains relevant to Government of Canada and Industry Canada priorities, and is viewed as making an important contribution to addressing a serious and urgent global challenge. Its approach to addressing this challenge is widely regarded as reflecting the best of current theory and practice. It has demonstrated that its process can be successfully replicated under a wide range of situations that may have applicability for other initiatives. It does not overlap or duplicate other Canadian international development and trade promotion programs and if anything, it does not lever as much as it could from them.

SCI's effectiveness has been hampered since its inception by:

• a confusing mandate that has led to an unproductive debate (amongst stakeholders interviewed for the evaluation) concerning whether the program should include funding for project implementation. Is it a trade development program or a development assistance program? If it is the former, it does not require funding for implementation, but may require specialists who can identify funding sources and arrange financing. If it the latter, then it does require funding, as well as specialists who can oversee high dollar value/high risk projects). The Department of Industry Act provides the Minister with a mandate for industry and technology, trade and commerce and small businesses. There is no mandate for development assistance;
• inadequate staffing levels;
• the extent of reliance that was placed on interpreters due to the language skills of the City Team members both from Canada and the participating cities;
• inconsistent application of its own city selection criteria;
• an inability to effectively vet Canadian partners;
• discontinuities in staffing and in organizational placement within IC.

Notwithstanding these impediments, the program has generated a great deal of goodwill, and is highly regarded by most who have had extended contact with it. Businesses and NGOs that have established ongoing relationships with the program remain optimistic about its potential to benefit themselves and the cities they are working actively with. The results were inconclusive on the extent to which SCI has met its other stated objectives. While the foundations may have been laid, it will take additional time to see significant results.

The program should be continued, provided its design flaws (especially the mandate) and operating flaws (staffing, selection criteria, vetting) are corrected. To give SCI a mandate that encompasses both trade and development assistance, it may be necessary to formally make it a horizontal initiative with program responsibility assigned to more than one department.

Recommendations

It is recommended that the Director General, Sustainable Technologies and Services Industries Branch consider implementing the following if SCI is renewed:

• clarification and better communication of SCI's objectives so that all stakeholders know whether it is a trade development or a development assistance program. If it is to be a combination of trade and development assistance, then a horizontal initiative involving departments and/or agencies with the appropriate mandate needs to be established;
• more rigorous application of the existing city selection criteria;
• better matching of the language skills of IC SCI staff with the requirements of the cities with which they are working;
• better alignment of the number of cities SCI is active in with SCI's actual staff levels.

2.0 Background

The idea of the Sustainable Cities Initiative (SCI) was first put forward by the Foreign Policy and Sustainability Committee of Canada's National Round Table on the Environment and the Economy (NTREE) in advance of and in the context of the 1997 AsiaPacific Economic Forum Environment Ministers' meeting. There was a recognition that the growing number of people living in the cities of the developing world was straining the natural environment and negatively impacting the local economies.

A pilot of the concept was launched in the fall of 1999 and involved five cities in Argentina, Brazil, Costa Rica, China, and Poland. A review of the Pilot Project published in October 2001 concluded that SCI had demonstrated its effectiveness and should be expanded. During the World Summit on Sustainable Development (WSSD) in Johannesburg, South Africa, the Government of Canada announced on September 1, 2002 funding of $9 million over a period of three years as shown in Table 1 to expand SCI. In the intervening period, SCI has expanded to an additional 11 cities. It is now active in Africa (Algeria, Senecal, South Africa and Tanzania), Asia (China and Mongolia), Central America (Costa Rica, Honduras and Mexico), Eastern Europe (Poland and Romania), and South America (Argentina, Brazil and Chile).

SCI results in the development of integrated strategies and solutions, through cooperation and partnerships, to the range of urban problems generated by rapid urbanization. This is achieved through a four step process:

• city identification. SCI staff in consultation with key stakeholders (e.g., the Department of Foreign Affairs and International Trade (DFAIT), the Canadian International Development Agency (CIDA), other government agencies, the private sector, non governmental organizations (NGOs) and the candidate cities ) identify potential cities for participation in the initiative. Key criteria for selecting cities include:
  • city is located in a developing country or an economy in transition;
  • the city is ideally between one and five million in population;
  • it has progressive governance and a commitment at the most senior levels to sustainable commitment;
  • a Canadian presence already exists in the city;
  • the city has at least some technology and knowledge needs that are available from Canadian sources;
  • realistic opportunities exist for financing identified projects.

  In addition to the city selection criteria, the selection process has also tried to ensure that:

  • there is a good distribution of SCI cities among the continents of the world;
  • selected cities address expected future international priorities, both with respect to disaster relief and with respect to availability of financing to enable Canadian organizations to meet international priorities within the objectives and principles of the SCI; and
  • world-class Canadian expertise is available to meet the needs of the nominated cities.
• team formation. SCI staff identify companies, municipalities, provinces, non-governmental organizations and others who have an interest and who are able to make a contribution, to participate in the city team. The city team focuses on the SCI city and works with the Industry Canada SCI team (City Team Manager and Project Officer) to prepare a roadmap. The design of individual projects, which together comprise the SCI within a specific city, is entirely the work of the city team members in partnership with the municipal government and other organizations from the SCI city.
• roadmap building. The objective of this phase is to identify and design projects in a manner that best meets the city's needs. Discussion is focussed on key pillars such as solid waste, water/wastewater, sustainable economic development, energy, transportation and governance¹. SCI staff facilitate a series of workshops and small group discussions to ensure that Canadian participants have a complete understanding of the city's problems and preferred solutions. The workshops also serve to build the relationship between the potential client and Canadian suppliers.

  There are generally two steps to the roadmap building process. The first focuses on the presentations by the City officials and group discussions involving City and Canadian partners. The second step is the presentation of the draft understanding by the Canadian Team and verification by the City officials. This normally takes place in two separate visits. Outcomes of the process are a working document that is accepted by City officials that outlines the opportunities for future collaboration and puts forward a sustainable vision for the future.

• implementation. Using the working document accepted by City officials, project teams, made up of the private sector with NGO and public sector partners, take lead responsibility for working with the city's project teams to achieve the specific goals described in each pillar of the Roadmap. As part of this process, more detailed objectives and project proposals will be developed. The speed at which proposals move forward is dependent on the business case, the extent to which it can be self-financing over a reasonable period of time or the extent to which financing can be provided from local, national or international sources.

  The IC SCI city team manager continues to provide a coordination role by taking the lead in monitoring the overall progress towards the city's sustainable development objectives. SCI team members are often involved in exploring sources of funding such as the international funding institutions and Canadian or international investors.

  Not all the needs of participating cities are dependent on the sale of technologies and services. Some aspects of capacity building for example , are best achieved through technical exchanges with municipalities in Canada and/or technical training provided by a Canadian educational or professional organization.

  The approved Roadmap document may also require expertise beyond that already available on the SCI city team. If this is the case, city team members will seek out appropriate expertise from the Canadian private, NGO, municipal or academic sectors. This is coordinated by the IC SCI city team manager.

The government's role (Industry Canada/SCI) is to develop and maintain an official framework for the relationship with selected cities and to facilitate projects. It does not finance projects. Some funding was provided during the Roadmap phase to help defray some of the participation costs.

In achieving its overall objective of increasing exports of Canadian sustainable development technologies and services, SCI also:

• brands Canada as a pre-eminent source of technology, expertise and products for urban sustainable development;
• promotes market development and trade opportunities for Canadian firms;
• enhances the quality of life in developing and emerging economies as well as promotes sustainable development;
• facilitates business partnerships among firms, nongovernmental organizations (NGOs), and government to access market opportunities.

As part of the approval process for SCI in 2002, a commitment was made to complete an evaluation of the SCI. This report fulfills this commitment. There is also a desire on the part of the SCI Business Advisory Committee to see the program continued. A recent evaluation is required as part of the renewal process.

In making a decision about renewal, Industry Canada will need to take into account the government's decision to move much of its international activity to International Trade Canada (ITCan) which was created in December 2003 and the mandate of the Ministry of Industry. Legislation to create ITCan has not, however, been passed as yet and is not expected to be passed in the near term.

¹ While certain pillars are common to a number of cities, the key pillars for a specific city vary and are determined in consultation with senior municipal officials usually at the city identification stage. The program approval documents indicated that the "key areas of focus are: clean water, sustainable waste management, clean energy, urban transportation, housing, capacity-building, urban planning and telecommunications." (Return to Reference 1)

3.0 Methodology/Approach

3.1 Evaluation Issues

Four issues were established for this evaluation:

• to what extent is there a need/relevance for the Sustainable Cities Initiative?
• to what extent has SCI been successful in reaching its planned objectives?
• to what extent is the program cost-effective?
• what lessons have been learned?

Specific evaluation questions were drawn from the draft Results-based Management and Accountability Framework (RMAF) that was developed for the program and are listed in Annex A.

3.2 Methodology

The information required to address the evaluation issues was obtained from a variety of sources. These included:

• a review of documentation including SCI program information and reports, federal government policy documents and reports, studies undertaken of the SCI pilot or similar initiatives by international organizations, reports from international forums, annual reports of international organizations, etc. A detailed listing of all the documents reviewed in the course of this evaluation is provided in Annex B, Bibliography. A summary of projected and realized results for seven SCI cities is provided in Annex C. The information for this summary was drawn from Project Tracking Sheets.
• interviews with key stakeholders: SCI current and previous staff, members of the SCI Business Advisory Committee, representatives from other federal government departments including staff from Canadian embassies in countries with a SCI city, partners from Canadian companies and NGOs, and local representatives from SCI cities provided input to the evaluation. E-mails were sent to one hundred and thirty-three prospective interviewees identified by SCI management and staff, inviting them to participate. Forty-one individuals self-selected themselves through this process. Twelve current and former SCI staff members were also interviewed. (See Annex D for a list of those interviewed and Annex E for the interview matrix employed).
• a web based survey of Canadian partners. Emails were sent to approximately 1,500 individuals whose name is on a database maintained by SCI. Two hundred and eight responses were received. (See Annex F for a copy of the survey employed.)
• the results of an audit that was undertaken concurrently with the evaluation.

3.3 Limits of Methodology

It is recognized that the methodology utilized has limitations. It is important in an evaluation to obtain the perspective first-hand from the beneficiaries of the program, e.g., the Canadian organizations who were exporting Canadian sustainable development technologies and services, and the participating cities who were endeavouring to improve the quality of life and promote sustainable development within their city. Direct evidence is considered more reliable than second hand information (e.g., the perspective of program management and staff). Despite several attempts to solicit a greater participation in the interview process by local representatives of the participating cities, the level of participation was too low to make any definitive statement about that stakeholder group.

While input from participants is required, it is generally considered less reliable than other potential measures such as the dollar value of increased Canadian exports, number of trade opportunities for participating organizations in comparison to the number of opportunities for comparable organizations over the same period, or the number of business partnerships formed among participating organizations in comparison to those formed by comparable organizations over the same period. People who self-select to participate in interviews or surveys of programs such as SCI, tend to be generally positive about the experience, especially when they are provided with funding to carry out a desired activity. Notwithstanding these limitations in stakeholder input, it is often the only source of information that can be readily obtained within a reasonable timeframe or at a reasonable cost.

Such was the case with this evaluation. Hard data on elements such as the number of identified proposals, number of Canadian organizations involved in projects, type of activities, number of Canadian partners at initial and subsequent visits to the cities, and the number of meetings with International Funding Institutions, is available in the technical mission books, trip reports, project tracking sheets and project budget tracking sheets. A significant effort beyond the scope of this evaluation, however, is required to extract the relevant information so that it would be in a useful format for management decision making.

Conclusions could also not be drawn from the size of the database maintained by SCI. At the time of the evaluation, in January 2005 it contained approximately 1,500 names. While the Departmental standard was implemented, Microsoft Access' file structure is not conducive to tracking when records were added or the nature of changes made. This information is necessary to track growth over time. The growth in the number of organizations represented on the database would have been more meaningful information than the number of names per se. Based on the information we were provided with, there were at most 900 different organizations represented on the database. Organizations represented included private sector companies, NGOs, universities, municipalities, provincial government departments and federal government departments and agencies.

Approximately 15% of the names on the database are Government of Canada employees with CIDA, Canada Mortgage & Housing Corporation, Canadian Heritage, Environment Canada, Foreign Affairs, Health Canada, Indian & Northern Affairs Canada, Industry Canada, International Trade, Natural Resources Canada, or Public Works & Government Services Canada. SCI management and staff were included on the database. Many of these individuals are key partners in the delivery of SCI and played a critical role in its delivery but the program was not targeted at them. To include them in a measure that is attempting to demonstrate increasing exports of Canadian sustainable development technologies and services or the promotion of market development and trade opportunities for an increasing number of Canadian firms, is in our opinion, inappropriate.

Further, more than 15% of the emails that were sent out using the database as part of the evaluation, were returned as undeliverable. Keeping addresses up-to-date is the bane of anyone who has to maintain any form of mailing list. Nonetheless, various methods need to be employed to keep mailing lists as accurate as possible to maximize the number of individuals/organizations who will receive messages on program initiatives and activities. A name that messages do not reach is of no value. Others responded to the survey indicating that they didn't know why it had been sent to them as they have no involvement with SCI.

Despite these limitations with the methodology, the evaluation is considered to provide a good assessment of the stated evaluation issues. The tendency of participants to be naturally positive about a program they are participating in is tempered by the fact that Canadian partners, in particular those from the private sector, had to pay to participate. They had to cover their own travel costs for the Roadmap sessions and the salaries of the staff who participated. If the private sector doesn't see a value in what they are doing, they are unlikely to continue for very long.

4.0 Findings

The findings have been grouped by issue and specific question using the framework established for this evaluation.

4.1 To what Extent is there a need/rationale for SCI?

SCI is consistent with current Government of Canada policy objectives, and is of continuing relevance to the government's and Industry Canada's priorities.

In the Speech from the Throne of October 5, 2004, the Government committed to pursuing "a five-point strategy to build an even more globally competitive and sustainable economy". The promotion of trade and investment was put forward as the fifth pillar of this strategy, and the Government pledged to enhance its capacity to expand international trade and commerce, with a particular focus on North America and emerging markets. It also committed to working "with its partners to build sustainable development systematically into decision making", as well as to working with the private sector to improve the commercialization of the best new environmental technologies. SCI, by showcasing Canadian companies' expertise in sustainable development, technologies and services during its missions to and when partnering with its selected cities, is a direct reflection of these commitments.

Industry Canada's most recent Report on Plans and Priorities (RPP) makes clear that the department's mandate is to make Canadians more productive and competitive. It therefore concentrates its efforts on three strategic outcomes, including competitive industry and sustainable communities. A commitment is made in the RPP to building on past accomplishments in sustainable development and addressing emerging challenges and opportunities to ensure a more sustainable future for Canadians. This commitment is made in support of a vision of Canada as a leader in the development, commercialization and adoption of sustainable development tools, practices and technologies. The department's investment priority sectors therefore include energy and environmental technologies, and it commits to identifying actions that can promote Canada's advantages. The SCI is identified in the RPP as example of factoring sustainable development into investment and trade decisions. Moreover, SCI's continued implementation and expansion is identified as an action plan item for the outcome area: advancing local and global sustainability.

In addition to this document-based evidence of SCI's relevance to the government's and the department's priorities, the interviews revealed a broad-based agreement that SCI is consistent with current federal policy objectives. Its managers and staff had no doubt about this. Staff from other government departments also concurred that the program fit current priorities, as did members of the Business Advisory Committee, who were often quite positive about this. SCI's Canadian Partners/NGO expressed the same views.

SCI's stakeholders (program management and staff, other government departments, NGOs, Canadian partners, members of the Business Advisory Committee) believe that the program has enduring relevance.

SCI's program management and staff were unanimous in their belief that not only is SCI still relevant today, but also that it will be for the foreseeable future. They pointed out that there is now a recognition at the political level that cities are the engines of the economy, and that there remain many cities that fit the profile for SCI that have not yet been included in the program. They also stated that there is continuing interest from the developing world in the program, and that they have thus far had more requests to participate than they have been able to deal with. In addition, they reported that there is regularly positive feedback from the Canadian companies that participate in SCI, and that they hear from companies that SCI is opening doors for Canadian organizations to non-SCI projects as a result of the relationships that have been developed through SCI. This view was echoed by some of the companies that responded to the survey.

Interviews with staff from other government departments (e.g., CIDA, DFAIT—including Canadian embassy officials in "SCI cities", NRCan), SCI's Canadian Partners/NGO², and members of the Business Advisory Committee yielded similar results. As with SCI personnel, they foresee a continuing need for SCI and its premise that trade has an important role to play in international development. Indeed, there was not a single interview that questioned the continuing relevance of program's underlying concepts.

SCI's business model and modus operandi fits current thinking among international institutions and agencies concerning how to address today's major global challenges.

It has been estimated that, within a generation, the world's urban population will have swelled by a further two billion, and that 98% of this expansion will take place in developing countries. Many cities in Asia and Africa will nearly double their population in the next 15 years. The need to address the challenges posed by this vast migration of rural poor to urban environments throughout the developing world is increasingly recognized as urgent among international institutions and agencies.

Launched by the World Bank and United Nations Centre for Human Settlements (UN-Habitat) in 1999, Cities Alliance was created to foster new tools, practical approaches and knowledge sharing to promote local economic development. It is committed to improve the living conditions of the urban poor through action in two key areas:

• City development strategies (CDS), which link the process by which local stakeholders define their vision for their city, analyse its economic prospects and establish clear priorities for actions and investments, and,
• County-wide and nation-wide slum upgrading to improve the living conditions of slum dwellers.

Cities Alliance, in other words, has determined that, like SCI, the first step to developing sustainable cities is the development of a "roadmap" that lays out a vision for the future and targeted initiatives. Cities Alliance also directly engages local stakeholders in the development of this roadmap. Similarly, and as noted below (see "The basic premises for the program have not changed"), UN-Habitat's Sustainable Cities Program is founded on the same premises as SCI.

The basic premises for the program have not changed.

In the view of all of those interviewed, SCI's multi-stakeholder, consultative approach to develop city sustainable development plans, and to implement them by using private sector participation as the vehicle for putting in place the basic infrastructure cities need to maintain and increase their prosperity, is as valid today as it was at the program's inception. Indeed, the members of the international organizations³ interviewed indicated that the trend was towards greater partnership building with the private sector, and toward implementing comprehensive consultation processes with multiple stakeholders.

A review of current reports from organizations committed to improving the living conditions of the urban poor (e.g., Cities Alliance, UN-Habitat) confirmed the perspective of the individuals interviewed.For example, the Second Session of the World Urban Forum (UN-Habitat) emphasized promoting partnerships and fostering broad-based participation of stakeholders in addressing urban poverty. UN-Habitat itself has a Sustainable Cities Program that is founded on broad-based, cross-sectoral and stakeholder participatory approaches. As well, the Cities Alliance 2004 Annual Report notes that the international system is increasingly recognizing the need to work with all the actors when dealing with cities and local governments (emphasis added).

Questions were raised about whether SCI was in the right cities, and in the right number of cities.

SCI's program management and staff commented that some of the cities selected for participation in the program were inappropriate (e.g., Qingdao, China, and Ulaanbaatar, Mongolia—the former because it was relatively well developed and prosperous, the latter because it was very poor and could not attract funding for projects. The difficulty and cost of travel to Ulaanbaatar also discouraged participation by potential Canadian partners). They believed that the city selection criteria had not always been consistently applied, and that decisions had been made to engage certain cities that reflected primarily the other considerations that were to be taken into account, rather than those stipulated by the selection criteria, (e.g., geographic location, active presence of another government department). They also commented that other departments wanted to become more active in the selection process, and that these organizations' priorities for selecting cities could be at variance with IC's.

The views of the other stakeholders in SCI concerning this were mixed. While none challenged the validity of the selection criteria, their comments on the suitability of specific cities tended to reflect their views of the primary orientation of the program. Those who saw this as sustainable development thought that SCI was in the right cities, noting that it is difficult to exclude or disqualify cities and that there are hundreds to choose from that could benefit from the SCI. Those who saw it as promoting the development of trade or of providing business opportunities for Canadian companies questioned the selection of some of the cities. Concerns included:

• lack of funding and/or investment partners for projects, whether from the city itself or other organizations;
• political instability/corruption at the local level; and,
• lack of motivation to pursue sustainable development projects at the city level.

In expressing these concerns, some stakeholders noted that it may not have been obvious that these problems existed at the time the cities were selected.

The cities themselves, of course, regarded their participation as appropriate.

Views on whether the SCI was in the right number of cities were similarly mixed. There was widespread agreement that the number of cities depended on the level of resources, particularly staff, available to operate the program. Some believed that the number of cities was appropriate to the number of staff currently in place. Current program management and staff, as well as some members of other government departments and the Business Advisory Committee, thought that the number of cities was higher than could be adequately supported by the number of resources currently in place⁴. The former DG responsible for SCI commented that the staffing levels created workplace stress, and that "in all his years in government, he never saw such a dedicated group of staff working long hours".

There was little perceived overlap or duplication of SCI with other Canadian international development and trade promotion programs.

The stakeholders interviewed viewed SCI as quite distinctive from other Canadian international development and trade promotion programs, as well as differing from other international agencies and programs. Among the differences noted were:

• SCI's missions are more targeted/focussed than typical trade missions, and seek to establish an ongoing relationship with a multi-year commitment with the selected city and other partners;
• SCI starts with the identification of city needs, and builds from these;
• its focus is on topical issues (i.e, clean water, waste management), unlike a generic trade mission;
• its orientation is to Canadian capabilities and organizations, unlike other international organizations that may be doing some similar things as SCI, but would not be trying to promote Canadian interests;
• its willingness to include small and medium enterprises (SMEs) as well as major organizations;
• its orientation to both planning and implementation, at all scales of projects, large and small.

Rather than viewing SCI as overlapping with or duplicating other Canadian programs, there was some concern that SCI was not levering as much as it could from the government's other programs for trade and international development, and that potential efficiencies were not being realized. Greater coordination was seen as desirable—for example, it was ventured that SCI might be a good way to develop projects that CIDA might want to invest in or implement. However, it was acknowledged that CIDA's focus was primarily rural, and that it was not very receptive to the feasibility projects that have predominated to date in SCI. It was suggested that the integration of a program such as CIDA-INC⁵ with SCI or a replication of the SCI model in other sectors could be a very effective tool for the development of international trade opportunities for Canadian SMEs.

Less than 20% of survey respondents agreed/strongly agreed that there was duplication, while almost 75% agreed/strongly agreed that SCI complements other Canadian international development and trade promotion programs. This pattern of findings held across all types of organizations surveyed (incorporated business, sole proprietorship, partnership, not-for-profit, and government) and level of participation in the program (considered but did not participate, participated in roadmaps and pulled out, participated in two or less cities, participated in three to five cities, participated in six or more cities).

4.2 To what Extent has SCI been Successful in Reaching its Planned Objectives?

Views on whether SCI has succeeded in reaching its target audiences differed depending on whether one was directly responsible for delivering the program, or was one of the multiple other stakeholders in SCI.

SCI's program management and staff were non-committal on whether the program had reached its targeted cities—those having between 1 and 5 million in population in developing and emerging economies. They clearly believed that there are many more cities where the program could be positioned, but they were somewhat disappointed in the capacity of Canadian businesses to respond, and felt that their success with this audience (Canadian organizations for whom market development and trade opportunities can be promoted) had been "hit and miss". The approach, of first identifying a city and its needs, was questioned, and it was suggested that it might be better to first identify Canadian strengths and then determine where these are needed.

The other stakeholders (other government departments, Canadian Partners/NGO and members of the Business Advisory Committee) were much more positive about SCI's ability to reach its "cities" target audience. When they expressed an opinion on this, is was invariably that SCI had reached this audience.

There were also divergent views on whether the initiative has succeeded in branding Canada as a pre-eminent source of technology, expertise and products for urban sustainable development.

The only group of stakeholders that consistently, if not unanimously, expressed a view in interviews that SCI had succeeded in branding Canada as such a source was its Canadian Partners/NGO. Other stakeholders, and SCI's program management and staff, expressed more divergent views. Some stated that it had "missed the mark" and that more effort was needed to accomplish this and distinguish Canada and Canadian companies from "competitors". Others believed that the process had started but that, since few projects have been implemented, and most work to date had involved small service companies, additional time and effort was required. Still others felt that SCI had been successful in this regard.

This range of views was present in every stakeholder group, except, as noted, with Canadian Partners/NGO, two-thirds of whom responded to the survey by indicating that the agreed/strongly agreed that Canada is recognized as such a source. In fact, the percentage agreeing/strongly agreeing tended to rise with increased level of participation in the program (though 61% who "considered the program but did not participate" also agreed/strongly agreed, suggesting a high level of pre-existing conviction concerning this amongst respondents).

Most stakeholders believed that it was unrealistic to expect SCI to be able to measurably enhance the quality of life in the developing world and emerging economy cities as well as promoting sustainable development over the life span of the program to date.

One of SCI's goals is to help ensure sustainable urban growth and thus enhance the quality of life of citizens in the developing world. This was widely regarded by both program management and staff, and all of SCI's stakeholders, as a long-term goal, one that cannot be measured at this stage of the implementation of the SCI's projects, particularly given the historical and cultural barriers present in many cities. Several, however, believed that SCI is laying a foundation for this through its work, and that the process of meeting with diverse local groups and building awareness of sustainable development was, itself, an important step towards the goal.

There was widespread agreement that SCI has promoted opportunities for trade development, even though relatively few instances of tangible trading arrangements in new markets have arisen since program inception.

Program management and staff, members of the Business Advisory Committee, representatives from other government departments, and Canadian Partners/NGO all endorsed the view that SCI promotes opportunities for trade development. The program was viewed as particularly valuable in this regard to SME, who gained access to local officials and political leaders who would not, in the view of almost everyone interviewed, be accessible to these organizations otherwise. As of April 2005, SCI management estimated that $2 billion in potential projects had been identified as a result of SCI for which $44 million of secured funding was reported⁶.

Questions were raised concerning whether these opportunities will mature into trading arrangements. Program management and staff believed that they would, contending that many project opportunities were at early stages. Others saw obstacles to this that would not easily be overcome. These included lack of funding for following through on project opportunities, competition from organizations from other developed countries (which may have funding support from their governments), and the capabilities of the SME that were involved to undertake the substantial infrastructure projects that feature in some of the roadmaps.

An analysis of Project Tracking sheets for seven of the 16 cities⁷ (see Annex C) showed that while the value of the potential projects was estimated by staff at over $600 million, few projects had moved beyond the planning phase. The total value of the completed projects during the period 2002-2005⁸ is approximately $22 million. This total includes one project which was valued at $15.3 million. It is not clear, however, how the project is directly related to SCI as it involved the installation of hydro-meteorological monitoring stations along major rivers and their tributaries throughout the country.

Notwithstanding questions surrounding how many of the potential projects will be realized, the survey revealed that Canadian Partners/NGO remained optimistic about the opportunities provided through SCI:

• over 66% of respondents agreed/strongly agreed that SCI creates international business opportunities for Canadian businesses that would not otherwise occur. This finding applied regardless of organization size (e.g., agreement or strong agreement ranged from a "low" of 61% in organizations with 1000 or more employees, to a "high" of 82% in organizations with between 250 and 999 employees). It also tended to rise with increased levels of participation in SCI cities, though here as well, organizations that had considered SCI but had not participated were also predisposed to this view (60% agreed/strongly agreed);
• a clear majority (44.6%, with 26% neutral) agreed/strongly agreed that participation in SCI had positioned their organization for future growth. This also applied regardless of organization size, and was related to the level of participation in SCI cities, with unanimous agreement among respondents who had participated in 6 or more cities; and,
• a clear majority of respondents that participated in SCI in three or more cities agreed/strongly agreed that participation in SCI had had a positive financial impact on their organization. Not surprisingly, this view was not shared by respondents that had considered but not participated or that had participated only in roadmap sessions: they were either neutral or disagreed/strongly disagreed (considered but not participated, participated only in roadmap sessions, respectively).

There was also agreement among most stakeholders interviewed that SCI had facilitated business partnerships among firms, NGOs and governments.

All stakeholders interviewed (program management and staff, other government departments, members of the Business Advisory Committee and Canadian Partners/NGO) agreed that SCI facilitated the formation of partnerships by bringing diverse groups together in the same place at the same time, with a common interest. Many cautioned, however, that the extent to which partnerships were actually formed depended on the interest and willingness of the participants to share information and expertise, and to form partnerships.

The results of the survey bore this out. Views on whether, at this point in the program's life, SCI had led to partnerships depended on the level of respondent participation in SCI cities:

• while all respondents that had participated in six or more cities agreed/strongly agreed that SCI participation had significantly increased their number of business partnerships, this percentage fell steadily with decreasing participation in SCI (to 7% for those that had considered but not participated in SCI); and,
• about two thirds of respondents that had participated in at least one SCI city agreed/strongly agreed that they had pursued other, non-SCI business opportunities with partners they met through their participation in SCI. None of those that had participated but pulled out after the roadmap agreed with this.

The was no support amongst stakeholders interviewed for the termination of SCI.

All stakeholders interviewed (program management and staff, members of international organizations and the Business Advisory Committee, Canadian Partners/NGO, cities, and representatives from other government departments) wanted to see SCI continue in some form. Several ventured that it was a program unlike any other in the Canadian government, and many stated that it should be expanded (provided additional resources were made available to support expanded operations). A few believed that SCI should be contracted, under the assumption that existing resource levels were not going to change, so that its staff could focus their efforts.

In endorsing the continuation of SCI, there were calls for changes. Indeed, almost every group that wanted SCI to continue also thought that changes were necessary. These included:

• providing funding to support implementation of projects;
• ensuring that the program is sized (expanded or contracted) to match its staff levels;
• clarifying and sharpening the program's raison d'etre (is it a development assistance program or an industry support program?) and results;
• reexamining how firms are selected for participation in missions so that these reflect Canadian strengths;
• extending its orientation beyond cities to regions; and,
• improving coordination with other departments, such as CIDA.

The results of the stakeholder interviews were matched by those from the survey. Two-thirds of respondents disagreed/strongly disagreed that the government should wind the program down. In fact, the only respondents that thought that the program should be terminated were those that had participated in the roadmap but had then pulled out of SCI. Even amongst these, the margin in favour of termination was narrow (i.e., 43% versus 39%, with 18% neutral). At all other levels of participation, there was a clear majority in favour of SCI.

Further support for continuing the program came from the survey question that asked whether, knowing what they now know about SCI, would respondents recommend that their organization participate in it. Almost two-thirds (62%) agreed or strongly agreed that they would. This was true even with respondents that had considered SCI but had not participated, with 57% recommending participation, and rose to 100% endorsement for respondents who participated in 6 or more cities.

4.3 To What Extent is the Program Cost-effective?

SCI's program management and staff, as well as its other stakeholders, believe that the program has exhibited considerable accomplishments despite the challenges that have faced it since its inception. They also believe that there is room for improvement in several areas.

When it first started operations, SCI needed to make itself visible and credible both within the federal government and the international community. Its first Director General is viewed as having been tireless in promoting the program, and together with a staff that was widely regarded as very hard working and dedicated, succeeded in convincing other federal government departments and partners to become engaged with and support the program. The planned roll out of the initiative to 17 cities in two years was in large measure accomplished⁹, and a database of contacts was established. Several stakeholders commented on how much SCI had been able to achieve on a "shoe string" budget.

SCI has evolved into a replicable process for the development of trade and bilateral exchanges with international counterparts. It has shown that the process of the development of the Roadmap can be replicated under varied conditions and circumstances. SCI has developed and field tested a process of gaining understanding, developing relations and establishing a protocol for the development of long standing partnerships both with foreign counterparts as well as with Canadian partners. Presumably, this process can be applied to trade and bilateral exchanges in any number of other sectors such as agriculture, environment, infrastructure, etc. and in most parts of the world.

If the SCI's management and staff was considered one of its greatest strengths, staff, or rather lack of sufficient staff, was also seen as a serious weakness. According to its current staff, the program has never been resourced to the levels planned. A comparison of actual expenditures to planned expenditures confirmed that it did not occur. Only one of two planned Regional Manager positions was staffed and the Outreach, Financial, Evaluation Officer also had Project Officer responsibilities for specific cities. In-place staff were over-extended such that the number responsible for each city was lower than planned. This was seen as having left insufficient time to take on activities regarded as important, such as researching funding/financing for projects identified in the roadmaps. The workload required from the staff that were recruited/seconded to the program has contributed to turnover that disrupted continuity in delivery, as did organizational changes within IC. As well, the travel schedule of the DG and the Director/Team Leader, combined with an absence of delegated authority (in part, because the required positions were not filled), contributed to perceived delays and bottlenecks in decision making.

Communications concerning SCI could be improved. As well as better promoting the program in Canada, requirements were perceived to improve understanding of the initiative among cities, and to better manage their expectations of what the program does, and does not, provide (i.e., that it does not entail Canadian government financing for the projects identified in the roadmaps).

Language problems have been experienced, both with respect to getting translations of documents from cities where some rarer languages are spoken, and in terms of the capabilities of mission participants. In several cities, very few members of the local government or administration spoke either English or French. IC City Team members and Canadian partners did not always speak the local language.

As noted earlier, the lack of funding to take projects from the conceptual phase to implementation was viewed as a hindrance to its ability to go beyond its success at opening doors to opportunities.

The views of SCI's stakeholders on whether the funding mix for SCI was appropriate and adequate were mixed, and complicated by the fact that the program was not able to use all its planned salary allotment.

Most stakeholders interviewed were in agreement that the money that was available to SCI was spent well, even if they could not provide an informed opinion on the funding mix (i.e., whether there should have been funding only for salary and operating costs, as was the case, or whether there should have also been funding for grants and contributions)

As to the question of whether more of the available funds should have been used for salaries, or for operating costs such as travel and translation, this was viewed as moot because it was not possible to convert salary funding to operations. As well, several individuals noted that the program was unable to use all its salary allotment. Some of it lapsed, and some was used to cover shortfalls elsewhere in Industry Sector that stemmed from lower than planned levels of retirement and hiring designed to replace employees retiring at planned levels.

Views of whether there should have been funding for grants and contributions were mixed. Some believed that it would have been easier to carry out project feasibility studies within such a funding mechanism (such studies were done using contracts, which were viewed as imposing onerous and time consuming processing requirements). Others acknowledged that grants and contribution, themselves, would have had significant overheads.

More significantly, there was concern that, if grants and contributions funding was available, it would be a disincentive to companies committing their own resources to the pursuit of opportunities. At the levels of support that were available to support companies pursuing opportunities, many SME would discount this, pointing out that the support they did receive did not fully cover their costs.

In sum, the only aspect of the enquiry in funding that generated widespread, but not unanimous, agreement was that there should be more of it, and that whatever mechanism would lead to this should be pursued.

Most stakeholders interviewed were satisfied with the program's basic design, but offered suggestions for refinements.

The program's underlying concepts—focus on cities in developing and emerging economies having from one to five million inhabitants, eschewing capital cities, engaging multiple stakeholders in the development of a city roadmap, forging partnerships to deliver projects in the roadmap—were all seen as key strengths of the program that should be preserved. Refinements suggested included:

• adjusting city selection criteria to include an assessment of the stability of local governments and administrations, and the ability of the city to secure financing. Program management and staff expressed a view that, in many cities, governments and administrations change faster than projects can be rolled out, which may take five or more years. When elections are held every three years, as they are in many cities, a change in administration may mean that SCI has to start from scratch. Assessments should therefore either be made of the likelihood of continuity of officials over elections or, where this is unlikely, the time frame for SCI to remain involved in a city should be extended to five or six years. As well, the city's creditworthiness and its ability to secure credit should be examined—there seems to be little point in engaging with a city than cannot finance the roadmap, particularly since SCI cannot do it either. To address this requirement for assessing city credit/financing, the program should have a permanent officer devoted to this function;
• exercising greater prudence in screening its Canadian partners to try to ensure that they have the technical and financial strengths for meaningful participation¹⁰;
• ensuring that SCI itself has continuity in staffing. A significant number of the current staff were reported to be on temporary assignment to SCI, and, come March 31, 2005, they are scheduled to depart;
• increasing the amount of coordination between SCI and other departments such as CIDA and DFAIT so that SCI has better intelligence concerning a city before it sets up a mission to it. This intelligence would include the city's credit worthiness and access to financing, its culture and attitudes to sustainable development, its receptivity to Canada and Canadian companies, and other Canadian development activities in proximity to the city. Several individuals saw CIDA as a key department for SCI to coordinate with, though it was acknowledged that CIDA's focus and strengths appear to be rural rather than urban;
• providing for a sustained SCI presence in the cities selected for SCI so that there was a capability to provide local coordination and collaboration on roadmap projects.

As well, managing the diverse cultures, priorities and reward systems in a multi stakeholder environment was seen as a major challenge, but a necessary and unavoidable one.

4.4 What Lessons have been Learned?

Interviewees were asked to comment on what factors have facilitated the delivery of SCI and what factors have impeded its delivery.

The factors identified that facilitated delivery included:

• the structure for delivering program to cities
  • the ability of Canadian partners to work with mayors and municipalities to identify needs. Cities don't always know at the outset what they need.
  • having host country groups in the same room talking to each other, in some instances for the first time
  • the team approach
  • active involvement of the Canadian embassy
  • ability to tailor the delivery to the specific needs of cities
  • multiple stakeholders (industry, universities and government) thinking about the future of cities. This is only happening in a few jurisdictions (Australia, United States and Canada);
  • the choice of an appropriate market niche. The big markets were already occupied and SCI for the most part, avoided them.
• role of the Canadian government
  • the backing of the Canadian government provided a counterweight to the host countries' historical and political ties
  • program could lever Canada's good reputation abroad
• management and staff who developed and delivered program
  • the very entrepreneurial focus of those who created the program
  • SCI management and staff were not overly bureaucratic
  • talent and commitment of SCI staff
• Structure of the missions
  • SCI partners come back more than once. When they leave, they are already talking about the next visit. Many trade missions only come once and there is no followup.
  • good identification of strategic partners
• the selected cities
  • local champions have been well selected
  • presence of local help (e.g., Canadian embassy or Chamber of Commerce) to set up meetings

Identified factors that impeded program delivery included:

• funding
  • a source of funding is needed to produce results;
  • expectation on the part of cities that funding to implement projects comes with this program
  • lack of sufficient funding for travel. SCI needs to get resources into place overseas
  • SCI needs to set the expectations straight, and then provide financing mechanisms or help secure financing afterwards
• SCI program management issues
  • program is too dispersed;
  • lack of technical knowledge of SCI
  • bouncing of SCI officers around multiple cities impeded the ability to build trust and develop relationships
  • there is a lack of professionalism
  • apparent strained relationship with CIDA
  • program timeframe of 2-3 years is too short. Need to think in terms of 5-7 years
  • call for staffing in early 2003 limited to the Industry Sector
  • lack of IC staff with skills in the native language of the host cities
• issues with selected city
  • political issues within the SCI city, changing local situation;
  • physical facilities in host countries
  • lack of understanding of local culture, e.g., no one in City may say "no" but on the other hand, they will not commit themselves
  • lack of suitable people to develop projects in certain parts of the world
  • lack of understanding of their role. The city may be accustomed to receiving aid as opposed to developing projects that address needs
  • poor fit with the program. The city may have been selected to be an "announceable"
• partners
  • need inter-cultural effectiveness training
  • some of the participants do not have strong technical or financial credentials. It is difficult to exclude them from participating
• uncertainty about whether the program will continue. Some may have questioned putting in the necessary effort for relationship building when they perceived that the initiative was potentially coming to an end.

² Fifteen were interviewed, and span a range, from representatives of companies that had participated in SCI projects, through provincial government agency officials, development bankers and academics, to a member of the Prime Minister's Advisory Committee on Cities and Communities. (Return to Reference 2)

³ The Inter-American Development Bank, Cities Alliance, UN-Habitat, the World Bank. (Return to Reference 3)

⁴ It was noted that SCI called for 20 FTE at its peak in 2003-2004, but that this level of staffing was never reached in part as a result of a decision to extend the program to 2005-2006, and to ensure that a portion of the approved $9 million would be available for a fourth year. Senior management decisions at the time that required positions be staffed on an assignment basis from within the Sector also had a direct impact on SCI's ability to staff to planned levels. (Return to Reference 4)

⁵ CIDA's Industrial Cooperation Program (CIDA-INC) provides funding to Canadian firms planning sustainable business activities in developing or transition countries. (Return to Reference 5)

⁶ The figures reported are based on estimates prepared by SCI City Team Leaders. They have not been independently validated. (Return to Reference 6)

⁷ The seven cities selected had, as of April 2005 according to SCI management, $1.56 billion in potential projects. Secured funding of $42.3 million had been obtained for these projects. (Return to Reference 7)

⁸ The Tracking Sheets included projects that predated the September 1, 2002 announcement of the initiative that is being evaluated in this study. (Return to Reference 8)

⁹ As of March 2005, SCI is active in sixteen cities. (Return to Reference 9)

¹⁰ From a practical perspective, implementation of this suggestion is difficult. If it was implemented, it could create legal liabilities. Other efforts by governments to rate the performance of suppliers has met with very limited success. Public Works & Government Services Canada for example, promulgated a Vendor Performance Policy effective September 1, 1996 to track the performance of vendors across contracts in response to observations made by the Auditor General of Canada in 1991 and 1994. Very few firms have been identified as poor performers under this policy. In response to recent problems, the VPP was to have been strengthened and a revised draft issued by March 31, 2005. If participating organizations were formally "endorsed", there is a risk that Canada could be held liable if any problems arose. The easiest way to mitigate this risk is to not formally endorse any Canadian partners. (Return to Reference 10)

5.0 Conclusions

SCI remains relevant to Government of Canada and Industry Canada priorities, and is viewed as making an important contribution to addressing a serious and urgent global challenge. Its approach to addressing this challenge is widely regarded as reflecting the best of current theory and practice. It has demonstrated that the Roadmap process can be successfully replicated under a wide range of situations that may have applicability for other initiatives. It does not overlap or duplicate other Canadian international development and trade promotion programs and if anything, it does not lever as much as it could from them.

SCI's effectiveness has been hampered since its inception by:

• a confusing mandate that has led to an unproductive debate (amongst stakeholders interviewed for the evaluation) concerning whether the program should include funding for project implementation. Is it a trade development program or a development assistance program? If it is the former, it does not require funding for implementation, but may require specialists who can identify funding sources and arrange financing. If it the latter, then it does require funding, as well as specialists who can oversee high dollar value/high risk projects). The Department of Industry Act provides the Minister with a mandate for industry and technology, trade and commerce and small businesses. There is no mandate for development assistance;
• inadequate staffing levels;
• the extent of reliance that was placed on interpreters due to the language skills of the City Team members both from Canada and the participating cities;
• inconsistent application of its own city selection criteria;
• an inability to effectively vet Canadian partners;
• discontinuities in staffing and in organizational placement within IC.

Notwithstanding these impediments, the program has generated a great deal of goodwill, and is highly regarded by most who have had extended contact with it. Businesses and NGOs that have established ongoing relationships with the program remain optimistic about its potential to benefit themselves and the cities they are working actively with. The results were inconclusive on the extent to which SCI has met its other stated objectives. While the foundations may have been laid, it will take additional time to see significant results.

The program should be continued, provided its design flaws (especially the mandate) and operating flaws (staffing, selection criteria, vetting) are corrected. To give SCI a mandate that encompasses both trade and development assistance it may be necessary to formally make it a horizontal initiative with program responsibility assigned to more than one department.

6.0 Recommendations

It is recommended that the Director General, Sustainable Technologies and Services Industries Branch consider implementing the following if SCI is renewed:

• clarification and better communication of SCI's objectives so that all stakeholders know whether it is a trade development or a development assistance program. If it is to be a combination of trade and development assistance, then a horizontal initiative involving departments and/or agencies with the appropriate mandate needs to be established;
• more rigorous application of the existing city selection criteria;
• better matching of the language skills of IC SCI staff with the requirements of the cities with which they are working;
• better alignment of the number of cities SCI is active in with SCI's actual staff levels.

Management Response to Program Evaluation of Sustainable Cities Initiative