Evaluation of the Contributions Program for Non-Profit Consumer and Voluntary Organizations

4. Success Related to Support

A. Introduction

This chapter explores topics related to the achievement of success through the Contributions Program. Success is assessed separately for Research Projects, Development Projects and for the Program as a whole.

B. Success related to Research

1. Impact on Recipient Organization

We measure the success of Research Project Contributions through the effects on recipient organizations in terms of:

• Consumer research capacity.
• Intellectual property/capacity.
• Solicitations for research, analysis, and consultative work.
• Demand for input by marketplace stakeholders.

a) Impact on Consumer Research Capacity

There appear to be two general models for the research conducted through a Research Project Contribution. The first involves researchers within the recipient organization conducting all aspects of the research. The second involves the recipient organization contracting out some or all of the research activities. The first model appears to offer the greatest opportunity for an impact on the consumer research capacity of the organization. In the second model, staff of the organization have less hands-on experience related to the research. However, there can still be impacts through the development of the research application, interactions with the researchers and use of the research results. Learning may also occur through interaction with a methodologist hired to assist in the design stage, to evaluate the final product, or to provide other input.

Key informants who were able to comment on the research capacity change for recipient organizations felt that it had increased through the research projects. A number of organizations identified their ability to take on larger or more complex research assignments with successive projects. Others suggested that being successful on projects each year had allowed them to retain staff they otherwise could not have kept. This stability of trained staff resulted in enhanced research ability for their organization. A number identified improvement that had taken place in their research competence through interactions with the external methodologist required by the Program to review methodology and final reports.

One key informant suggested that universities were starting to acknowledge the research contribution of consumer groups and their role in society. The individual noted that the Université du Québec à Montréal recently gave a prize to the director of a consumer organization in recognition of the group's research activities.

b) Impact on intellectual property/capacity

A generally held view by key informants was that recipient organizations are developing expertise in particular areas through their involvement with Research Project Contributions. They report being invited to conferences or being asked for input related to government initiatives. They receive more media attention related to their expertise. A number of recipient organizations identify their credibility increasing through their involvement with the Research Project Contributions.

The body of work over a number of years for a single organization covered by our case studies is an illustration of the intellectual property that has been vested in the organizations. A minority view was that expertise developed through the Program then made it harder for others to compete against this expertise for future Research Project Contributions. Those groups who are successful in past applications are then more likely to be successful in subsequent applications. This may be an indication of the development of intellectual property in the recipient organization.

Another minority view was that the competence gained through past projects did not translate into success on future projects in the same area, suggesting that no intellectual property had been created.

c) Solicitations for research, analysis, and consultative work

A few examples were put forward by recipient organizations of cases where they had participated in a follow-on activity, including providing policy input, as a result of the expertise they had developed through a research project.

However, this participation was not always funded, particularly if it involved a federal or provincial government. More typical in activities involving a non-profit consumer or volunteer organization and government is for the latter to cover out-of-pocket costs only (typically travel but sometimes time) of the organization. This practice likely evolved when organizations received operating grants from the federal government but no longer fits with the current fiscal reality. Organizations are typically paid for their time and expenses if they are granted intervenor status (for example in a rate hearing of a regulated monopoly).

Some, albeit few, related activities have involved private sector work. Policies of some consumer groups prevent organizations taking on such work due to concern over conflict of interest. Some research activities have involved support from the private sector conducted on a non-directed basis.

In general, organizations have achieved only limited leverage of the expertise they have developed through the Program in paid activities for others.

d) Demand for input by marketplace stakeholders

Demand for input on consumer-related matters by non-profit consumer and volunteer organizations was judged to be high by recipient organizations especially after the release of a project report. Demand has two main sources—government and the media.

Demand occurs in areas even outside organizations' area of expertise. One possible interpretation is that those who need consumer input and are unable to obtain it, subsequently try to obtain it from any group, not just those with expertise in the area.

Of those who commented, organizations appear to be able to respond to such demands in only a small percentage of cases. One identified the fraction at 1 in 10 cases. Organizations report that they are not funded for these activities and to spend time on them ultimately takes away from other activities. They need to be very selective about what demands they meet and to understand that to meet one unpaid demand, they must take away from all other activities.

e) Summary of Impacts on Recipient Organizations

Recipient organizations increase their capacity to do research and may do better research through their involvement with the Research Project Contributions. Many are developing expertise in particular consumer issue areas. Their credibility is also enhanced. However, few recipient organizations appear to have capitalized on these impacts through other funded activities. Although demand for their input on consumer issues is high, consumer groups are able to meet very little of this unfunded demand. To do so would take away from other activities of the organization.

2. Impact on Organizations' Employees

As previously mentioned two models exist related to the funded research. Those groups that conduct the research internally are likely to see the biggest improvement in the human capital/expertise of their employees. However, human capital/expertise increases are also possible for employees of organizations that tend to contract out the research activities although the skills are likely different between the two models.

Recipient organizations report an improvement in the human capital of their employees through activities involving the research projects. As previously noted, some suggested the string of successful projects had provided a continuity in the work of their researchers and subsequent increase in their expertise that the organization would not have been able to achieve on its own.

However, this increase in expertise is a double-edged sword. A number of organizations report that trained staff gets hired away by other employers. Examples cited included a crown corporation, federal government, health research organization, university-based organization, private research, and a consumer organization (Consumers International). Most were said to be still active in the public policy area. As a result, there is some support for the view that the research environment provided by supported consumer organizations acts as an incubator for developing consumer-related researchers who ultimately are hired by outside organizations.

The challenge as reported by one organization was to develop staff but also to retain them. We suspect the issue is one of higher pay scales outside the non-profit consumer and voluntary sector. Staff that remain, place higher value on non-pecuniary aspects of their job (have a missionary spirit according to one organization).

A number of organizations also report that summer/co-op students are exposed to a richer training environment in a broader range of research activities than they would likely experience in any other organizational setting.

3. Impact on Those Outside Supported Organizations

An impact of the supported research may also occur beyond the recipient organization or its employees. This led us to investigate who receives study results and how these results affect the work recipients do or actions they take. In the process, we assessed the timeliness and dissemination of study results. This section discusses findings on these issues.

Projects are typically one year or less in duration. Reports are to be completed by fiscal year end (March 31) to meet requirements of the government funding cycle. However, many are not. Reasons for the lateness of reports are also many. These are grouped within themes and presented by frequency of response, from most to least frequent, below:

• Lack of core depth in organizations/ everyone over-extended.
• Project reliant on input of others and this is sometimes delayed.
• Vagaries of research/ cannot always predict how things will go.
• More time-sensitive issue comes up/ research can be set aside while other issue cannot/ OCA is the least pressing demand on time.
• Groups don't understand the size of the research or have unrealistic timelines.
• Organizations live off cash flow from project and the money has been spent by report time.

Summarizing across these reasons about one-half reflect a lack of capacity in the organizations, about one-third indicate the nature of the work and the balance reflect some lack of experience or competence in the management of the project.

Once reports are produced, the extent and form of dissemination of information varies considerably across organization and likely across projects within an organization. Some identified that some research results turn out not to be worth disseminating.

Most organizations identify posting a summary of the report on their web site and providing the capability to download the report from their site. Most also report holding a press conference or providing a media release. Many develop and send out reports to a list of key stakeholders (including industry) they think should get the information. A few identify making the report available to their membership. A few also mention sending the report to politicians or meeting with politicians to review the study findings.

One organization mentioned doing all of the above but also sending reports to public libraries, universities, and industry, as well as using the information in a weekly television program. One organization sends summaries of their research reports to all non-profit consumer and volunteer organizations.

Some identify doing little with the reports. Some suggested more could be done.

A number identified problems with dissemination that they linked to the Program. Before the research began they found it difficult to identify how much dissemination of results would be required. Given the competitive nature of the application for support and the uncertain surrounding the weight attached to dissemination in the analysis of proposals, organizations may under-allocate resources to dissemination in their application.

The OCA has begun posting summaries of research projects on their website and providing links to the website of the organizations that produced them. However, OCA does not own the report (ownership resides with the organization under a contribution agreement). As a result, OCA's role in distribution is limited although they may be involved in notifying a government department of a research report of potential interest to them.

Key informants interviewed in the study who should have been in positions where they could make use of research reports said they did not receive them routinely or were only able to find them with some difficulty. However, they only knew to look for them if they found a quotation of the study. Some key informants suggested they should, but do not, receive notice of reports produced under the Contributions Program. Recipient organizations identify they should but do not know the outcomes of other organizations' projects.

We have limited information on the use of research results. Organizations do not record contact information for those who obtain their reports, frustrating attempts to follow-up to assess an impact and neither the OCA nor supported organizations systematically record mentions of reports in the media or quotes in peer journals or articles. A few organizations mentioned keeping a clippings file but not compiling any information from them.

Information we have collected on the use of reports is indicative but not definitive:

• A number of studies (automobile retailing and repair) have been featured in television programs. The media has covered issues such as problems with the Drive Clean program in Ontario.
• The Drive Clean study results were discussed at a conference in San Diego.
• Public Interest Advocacy Centre (PIAC) provided useful input through public process and parliamentary committees related to the Personal Information Protection and Electronic Documents Act. Amendments that were put forward by a Member of Parliament were based on briefings by PIAC.
• Research conducted by Option Consommateurs and PIAC on compliance was cited numerous times by other groups.
• Studies on e-commerce were quoted at the 1998 OECD Ministerial Conference on Emerging Market Economy Forum on Electronic Commerce.
• The consumer representative on the Working Group looking into Electronic Funds Transfers is having a dramatic impact on the laws/regulations that will be developed. This consumer involvement would not occur without the past Program support.
• The new Ontario Consumer Protection Act (likely to be passed in 2005) includes a section on disclosure of information on commissions to workers doing repairs on vehicles as a result of the funded research by the Automobile Protection Agency (APA). The APA study was identified as the principal reason for making the commission receive as part of a workers' pay a component of Ontario's proposed regulation.

The variety and diffusion of stakeholder impacts noted above suggest the challenge involved in trying to assess the overall impact of the Contributions Program. Our study was limited to assessing use of research results by stakeholders. Assessing the overall impacts was beyond our evaluation mandate.

C. Success related to Development

Relatively few of the organizations included in key informant interviews had been involved in Development Project Contributions. As a result, our findings are more tentative related to development support. However, the few cases provide a number of examples of positive impact.

1. Impact on Recipient Organization

The following are examples of impact on organizations through the Development Project Contributions:

• An organization was provided funding to translate an educational tool in the energy field that they had developed previously. As a result, the Quebec-based group was able to access the English market with their product. The organization received a contract for 10,000 units that were distributed throughout Canada. Without funding, the English product would not have proceeded. The group would not have received the revenue from this broader distribution and the resulting publicity.
• Funding allowed an organization to further develop a family budgeting software package they expected to market after copyright problems had been overcome.
• Funding was provided to develop a business plan to merge two consumer organizations. As a result, the merger was more harmonious and efficient, leading to a much stronger combined group.
• Through funding one organization developed an on-line virtual member pool of 5,000 individuals. This allows them to do quick consumer surveys of their members. This facility has become an important research tool for them. It has increased their status and recognition with the media and other groups.
• The Canadian Consumer Initiative (supported through contributions to a few member organizations) is a forum for the development of a shared perspective by its members. This initiative has identified opportunities for mutual benefit and for joint action to improve their collective benefit. One such action was to speak to the Minister responsible for Industry Canada.
• The APA has identified class action suits as an opportunity to build on its expertise and to establish a potential revenue stream.

2. Impact on Those Outside Organizations

Development Project Contributions have as their principal goal the strengthening of the consumer organization. The Development Projects, noted above, appear to achieve this goal. Although this may ultimately result in benefits that extend beyond the organization, most of the impact of development support is likely to reside with the organization. The only direct impacts on groups, outside of organizations supported through Development Projects, were the closer more effective link between one organization and its members through the on-line membership initiative and potential awards to consumers as a result of class action suits.

D. Success related to Consumer Movement

In addition to the impacts noted separately for Research Project Contribution and Development Project Contribution, we asked more generally about the impacts of the Contributions Program on the consumer movement in Canada and particularly on the organizational capacity of organizations within the consumer movement.

The most frequent impact identified was the added credibility the Program has brought to non-profit consumer and volunteer organizations. Organizations are now better able to present research-backed arguments that are the equal of arguments by industry. As a result they are more effective lobbyists and better able to protect the interests of consumers.