Evaluation of Industry Canada's Organisation for Economic Co-operation and Development (OECD) Grants Program
Recommended for approval to the Deputy Minister by the Departmental Evaluation Committee on
Approved by the Deputy Minister on
Table of contents
- Executive Summary
- 1.0 Introduction
- 2.0 Evaluation Methodology
- 3.0 Findings
- 4.0 Conclusions
List of acronyms used in this report
|AEB||Audit and Evaluation Branch|
|CAS||Comptrollership and Administration Sector|
|MOU||Memorandum of Understanding|
|OECD||Organisation for Economic Co-operation and Development|
|PSF||Project Summary Form|
|RMAF||Results-Based Management and Accountability Framework|
|SPB||Strategic Policy Branch|
|SPS||Strategic Policy Sector|
List of tables
|Table #||Table title|
|Table 1||List of IC Funded Projects|
|Table 2||Alignment of IC Funded Projects with Government Priorities and Departmental Strategic Outcomes|
List of Figures
|Figure #||Figure title|
|Figure 1||Logic Model of the OECD Grants Program|
Introduced in 2009, the OECD Grants Program provides Industry Canada with the necessary grant authority to fund OECD research projects that meet departmental objectives and advance the departmental policy and research agenda. Since its inception, seven projects have been funded through the program, totalling $269,000.
Evaluation Purpose and Methodology
In accordance with the Treasury Board Policy on Evaluation and the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of the OECD Grants Program. The methodology for this evaluation was calibrated to take into consideration the materiality of the program. The methodology included a review of documents, review of project files, and interviews.
There is a continued need for the OECD Grants Program as it affords the department the latitude to participate in OECD projects that are related to its policy and research agenda. It also provides opportunity for international comparisons between Canada and other countries through the development and maintenance of common economic indicators.
The program aligns with the Department of Industry Act of 1995, and also responds to Canada's role as a member country of the OECD. The projects funded through the OECD Grants Program are aligned with federal government priorities and Industry Canada's strategic outcomes.
The OECD Grants Program has increased Canada's influence on OECD committees and projects. Voluntary contributions to the OECD allow IC to firmly establish its policy priorities within the OECD research agenda. Contributions are instrumental in informing and advancing sponsoring branches' policy and research interests. Additionally, the collaborative aspect between national representatives and experts at the OECD helps to ensure that projects also advance common data development and information sharing.
The program is being administered efficiently as it uses existing reference levels and does not present a significant time burden on program staff or the program DG committee. The project approval process is clear and straightforward. When compared to alternative methods (i.e. Treasury Board submission process), the program is the most efficient approach for the department.
Overall, the evaluation did not find any major issues with the OECD Grants Program and as a result makes no recommendations.
This report presents the findings of an evaluation of Industry Canada's Organisation for Economic Cooperation and Development (OECD) Grants Program conducted between May and July 2012. The purpose of the evaluation was to assess the relevance and performance of the OECD Grants Program.
The report is organized as follows:
- Section 1 provides background on the OECD and a program profile;
- Section 2 sets out the evaluation methodology;
- Section 3 presents the key findings related to the evaluation issues of relevance and performance; and,
- Section 4 summarizes the evaluation's conclusions.
1.1 OECD Background
Founded in 1961, the OECD is a 34-member international forum whose mission is to promote policies and conduct research that will improve the economic and social well-being of people around the world. The OECD is a key source of comparative economic data and analysis at the international level. Activities at the OECD include: policy research and analysis; data and model development; discussions among recognized experts; consultations with industry, other private sector stakeholders and international bodies; and dissemination of research findings through the publication of reports and other mechanisms.
The OECD focuses on a wide range of economic issues, many of which fall under Industry Canada's mandate. Some of these include innovation, electronic commerce, science and technology, intellectual property, and competition and consumer protection policies. The analyses conducted by the OECD as a whole enable Canadian policy to be benchmarked against global best practices, allow Canada to learn from other experiences, and also highlight Canada's strengths on the global stage.
The work and research of the OECD is conducted by subject-specific committees and working parties, which are comprised of Canadian and other member country representatives and experts. Industry Canada (IC) represents Canada on a number of OECD's permanent committees and working parties as well as on temporary project-specific advisory bodies. Increasingly, the work of the OECD is funded by voluntary contributions from OECD members rather than from annual contributions from OECD member countriesFootnote 1. This type of funding now accounts for one-third of overall OECD resources.
1.2 Program Profile
The OECD Grants Program was created in 2009 and provides IC with the necessary grant authority and source of funds to share in funding OECD research projects which advance the departmental policy and research agenda. The objective of the program is to use the results of supported projects to assist the department in fulfilling its mandate and in meeting its strategic outcomes.
The program enables the department to share the costs of specific projects with other OECD member countries through voluntary contributions (i.e. grants). Such projects involve activities relating to policy research, analysis and development. These include data collection and model development; policy analysis and assessment; exchange of information among recognized experts; consultation with representatives of private sector stakeholders and other international bodies; and dissemination of research findings through the publication of reports and other mechanisms.
Activities of an ongoing nature which are unrelated to a specific OECD project are not considered eligible projects under the Grants Program.
1.3 Program Resources
The program's funding authority is $500,000 per year over 5 years (2009–10 to 2013–2014), for a total of $2.5M, where IC's contribution to any OECD project may not exceed $300,000. To date, the program has funded or co-funded seven projects at the OECD (four of which are completed) with grants totalling $269,000, ranging from $15,000 to $91,000 per project (see Table 1).
|OECD Project Title||Project Description||Grant Amount|
|OECD Scoreboard on SME and Entrepreneurship Financing Data and Policies
|The Scoreboard is aimed at discussing, monitoring, and reporting on SME and entrepreneurship financing trends as well as judging the effectiveness of current SME financing policies.||$15,000|
|Business Innovation Policies – Selected Country Comparison
|Study to gain a better understanding of the policies and programs put in place by various OECD member countries to stimulate demand-side (or "pull") innovation.||$91,000|
|The Economics of Privacy
|Project will provide policy makers with the information they need to establish economic-based arguments and rationale for the protection of personal information in the digital age.||$25,000|
|Progress in the Governance of Biomedicine and other Health Innovations
|Workshop of experts with the goal of analyzing how regulatory systems will need to adapt to emerging technologies in the worldwide biomedical R&D environment.||$13,000|
|Development of a Global Consumer Product Recalls Database
|Database to increase information about recalled products which will help to foster consumer confidence in the marketplace.||$50,000|
|Selected Country Comparisons on Policies to Support the Financing, Transfer and Commercialization of Knowledge between Public Research and Industry
|Review of industry-science relations which will build on the work already performed by the OECD in terms of profiling the innovation policies of various countries.||$50,000|
|Measuring the Economics of 'Big Data'
|Assessment of the methods used by different Web Analytics to estimate and provide better measurement of online advertising expenditures.||$25,000|
1.4 Program Delivery
The program is managed by the Strategic Policy Sector, Strategic Policy Branch (SPS-SPB). The primary responsibility of SPB is to guide the sponsoring sector or branch through the processes, guidelines, and procedures of the program. The OECD is the recipient of grants under this program. Departmental sectors and branches that want to sponsor OECD research initiatives must ensure that the supported projects are aligned with IC's objectives and priorities.
A branch and the Strategic Policy Branch must also ensure program funding is internally available to pursue a potential grant to the OECD. Next, they must complete a document, called a Project Summary Form (PSF), which describes the project, the rationale behind the project, and related financial details. The initial review of this document is done by the OECD Grants Program Coordinator.
Subsequently, the document is reviewed by the DG Committee for the OECD Grants Program. This committee is comprised of various departmental DGs and is chaired by the DG of the Strategic Policy Branch. The proposal is reviewed based on the project's relevance to IC policy and research priorities, as well as its prospective advancement of the sponsoring branch's responsibilities and the program's objectives. Proposals are currently reviewed electronically and allow each committee member a forum to provide comments and feedback on the project proposal.
Once approved by the Committee, the sponsoring branch completes a Memorandum of Understanding (MOU) with the OECD. This MOU sets out the grant amount, the payment conditions, reporting requirements, and a brief description of the project.
1.5 Expected Results
The purpose of the program is to enable Industry Canada to more fully participate in OECD projects and to take better advantage of the potential to advance its objectives through its participation in OECD activities.
The following logic model for the OECD Grants Program was developed in 2009 and was updated in 2012 prior to the evaluation. The logic model outlines the program's activities and outputs, as well as the intended immediate, intermediate, and ultimate outcomes.
2.0 Evaluation Methodology
This section describes the overall approach of the evaluation, the objective and scope, the specific evaluation questions that were addressed, the data collection methods, and data limitations.
2.1 Evaluation Approach
The evaluation was conducted in-house by Industry Canada's Audit and Evaluation Branch.
2.2 Objective and Scope
The evaluation of the OECD Grants Program is required under section 42.1 of the Financial Administration Act. In accordance with the TB Policy on Evaluation and the Directive on the Evaluation Function, the purpose of the evaluation was to assess the core issues of relevance and performance.
Given that the program has only been in existence for three years, the evaluation focused on the achievement of immediate outcomes and examined intermediate outcomes to the extent possible.
2.3 Evaluation Issues and Questions
Based on the program's logic model and the Performance Measurement Strategy developed in 2009 (and revised in 2012), the evaluation addressed the following questions:
- To what extent does the OECD Grants Program address a demonstrable need?
- To what extent does the OECD Grants Program align with federal roles and responsibilities?
- To what extent do the objectives of the OECD Grants Program align with priorities of the federal government and the strategic outcomes of Industry Canada?
- To what extent have OECD funded projects contributed to common approaches to data development and information sharing among OECD member countries?
- To what extent has the OECD Grants Program increased Canada's influence at the OECD? Have the OECD funded projects informed and advanced IC's policy and research agenda?
- Has the grant program been administered efficiently and economically?
2.4 Data Collection Methods
The methodology for this evaluation was calibrated to take into consideration the low materiality of the program. This program accounts for less than 0.1% of IC's forecasted spending for fiscal year 2012-2013. Nevertheless, the methodology employed a few lines of evidence to cover all evaluation issues.
The data collection methods included a review of documents, review of OECD project files, and interviews.
2.4.1 Document review
A number of documents were reviewed for the purpose of this evaluation. These documents included Federal Budgets and Speeches from the Throne, IC reports, Program Treasury Board Submission (and Terms and Conditions), Program Results-Based Management and Accountability Framework (RMAF), and a sample of OECD reports.
A list of documents reviewed is provided in Appendix A.
2.4.2 Review of OECD Project Files
The OECD projects funded through the Grants Program were assessed. As of June 2012, there were four projects that had been completed and three on-going. The purpose of this review was to better understand the impact of the funding and describe in detail the extent to which the projects have contributed to the main objectives of the program.
Each project review included reviewing PSFs, program documents (i.e. MOUs), and final project reports and deliverables when applicable.
The objective of the interviews was to gather in-depth information for evaluation purposes, including views, explanations, and factual information that address the evaluation questions. Interviews allowed evaluators to gain insight into the performance of the OECD program from the perspective of program recipients and the OECD. Interviews were conducted in-person or by phone.
A total of 15 interviews with 19 individuals were conducted, as follows:
- Industry Canada program management and staff (3 with 4 individuals);
- Industry Canada program recipients, program DG Committee members and/or departmental OECD committee representatives (11 with 14 individuals); and
- Canada's Permanent Delegation to the OECD (1).
Interview guides were provided to each interviewee prior to the interview. Interviews were conducted in the official language identified by the interviewee and each interview was documented.
The generic interview guide is provided in Appendix B.
The primary limitation to the methodology is the small number of projects that have been funded under this program (7 in total). It is difficult to fully assess some of the program outcomes outlined in the logic model. However, interviews helped to mitigate this limitation, particularly as the key focus was to gain insight into the performance of the program. Further, this limitation was judged to be acceptable in relation to the level of calibration set out for this evaluation.
3.1.1 To what extent does the OECD Grants Program address a demonstrable need?
OECD's strengths as an organization include a multi-disciplinary capacity for analysis and policy dialogue; its sharing of best practices; and monitoring of its 34-member countries through peer review, extensive policy dialogue, and capacity building activities with non-member economies, international organizations and other stakeholders.
The work of the OECD regularly intersects with Canadian interests. Since the start of the OECD Grants Program, the department has benefited from projects ranging from collaborative work on the development of a global consumer product recalls database to privacy protocols and international comparison work on business innovation policies. The OECD Grants Program provides a mechanism for the department to sponsor special projects at the OECD using existing funds.
The work of the OECD is conducted by committees and the department has exclusive or shared responsibility for Canada's participation in seven plenary committees involving more than fifteen formalized sub-committees and a handful of additional informal groups. Membership in these groups, however, is insufficient to fully participate in forming the OECD research agenda or scoping out individual projects. Increasingly, committees are funded by project-specific voluntary contributions (grants) from OECD member countries rather than from fixed annual contributionsFootnote 3.
The department's representatives at OECD committees are regularly presented with opportunities for Canada to participate as a co-funder, alongside other OECD countries, in projects that would effectively advance the department's priority policy and research interests. Prior to the program, Canadian delegates found that the inability to fund individual projects was undermining the effectiveness of IC's participants.
The capacity to make grants using existing departmental program funds has allowed the department to influence the selection of projects as well as their definition, ensuring that the department's interests are fully reflected in the work of OECD committees. The grants have also been used to ensure access to reliable, comparative data on key economic issues so that departmental activities can be measured against other OECD economies.
It should be noted that uptake for the program has been somewhat low, however, this may be due in part to budgetary constraints across the department rather than an indication of interest or program requirements going forward. Interviews unanimously indicated that there is a definite need for the program. A number of interviewees also noted that some OECD projects sponsored through this initiative would not have gone forward without Canadian involvement, or else the scope would not have included Canada in projects involving international comparisons.
3.1.2 To what extent does the OECD Grants Program align with federal roles and responsibilities?
In the global economy, international collaboration has become key in understanding economic development, fostering growth and in advancing national interests. Under the Department of Industry Act of 1995, the Minister is tasked with performing the duties and functions assigned by the Act in a manner that advances Canadian interests within the international sphere.
Specifically, Section 6 of the Act states that the Minister shall:
(b) collect, gather, by survey or otherwise, compile, analyse, coordinate and disseminate information in respect of matters under the Minister's administration, as well as in relation to trends and developments, both within and outside Canada, in respect of those matters; and
(d) promote cooperation with the governments of provinces and their agencies and non-governmental entities in Canada and participate, as appropriate, in the promotion of cooperation with agencies of other nations and international agencies.
Canada's status as a member-nation of the OECD enables the department to more fully discharge its duties via participation in the OECD's committees and working parties. However, it is the voluntary contributions (grants) through the OECD Grants Program that enable Canadian delegates to further shape activities. The department's policy and research agenda, as well as its international data requirements, are supported by the latitude afforded by the OECD Grants Program as it provides the nation with the opportunity to advance Canadian interests within the organization.
3.1.3 To what extent do the objectives of the OECD Grants Program align with priorities of the federal government and the strategic outcomes of Industry Canada?
The program's outcomes are consistent with the Speech from the Throne and Federal Budgets. In fact, in order to make use of the program, proposals for potential projects must show how the project will align with federal government and departmental priorities. Additionally, the alignment is verified by the Program DG Committee.
Table 2 demonstrates how the seven projects funded to date by IC align with the priorities of the federal government as well as the strategic outcomes of IC. While the program's projects actually respond to all three of the department's strategic outcomes, the program is housed within the Science, Technology and Innovation Capacity (2.1) program activity and the Science and Technology Partnerships sub-activity (2.1.2) of the Program Activity Architecture.
|OECD Project||Federal Government Priorities||Industry Canada Strategic Outcomes|
Business Innovation Policies – Selected Country Comparison
"The Government is committed to a new approach to supporting innovation in Canada."
OECD Scoreboard on SME and Entrepreneurship Financing Data and Policies
"Canada's small and medium-sized companies are an important engine of our economy, driving innovation, productivity, job creation and economic growth."
(Economic Action Plan 2012)
The Economics of Privacy
"Recent Federal Initiatives in Support of Canada's Digital Economy: Modernizing policies to build confidence in e-commerce through the passage of new anti-spam legislation and tabling privacy and copyright legislation."
Progress in the Governance of Biomedicine and other Health Innovations
"The Government is committed to promoting safe communities and protecting the health of individual Canadians."
Selected Country Comparisons on Policies to Support the Financing, Transfer and Commercialization of Knowledge between Public Research and Industry
"The Government will continue to provide support for research and Canada's public research infrastructure, while seeking to increase the impact of its investments by levering contributions from the private sector and other levels of government in national projects and commercialization partnerships."
Development of a Global Consumer Product Recalls Database
"Our Government will follow through with legislation providing better oversight of food, drug and consumer products. It will strengthen the power to recall products and increase penalties for violators."
(Speech from the Throne 2008)
Measuring the Economics of 'Big Data'
"In order to be a world leader in knowledge and innovation, Canada must attract and develop talented people, increase our capacity for world-leading research and development, improve the commercialization of research, and promote education and skills development."
Of particular note is the Business Innovations Policies project—an OECD Grants Program recipient that has had a noteworthy impact on the review of Federal Support to Research and Development. The OECD project report was used as input into the review and is cited throughout the final Canadian report, known as the Jenkins Report.
3.2.1 To what extent have OECD funded projects contributed to common approaches to data development and information sharing among OECD member countries?
The OECD offers a unique forum for the development of common economic indicators via its work in global standards and the principles of economic development policies. Common approaches to data development are prevalent within the organization as country representatives are required to work together through committees and working parties on universal issues. As the OECD Grants Program allows the provision of voluntary contributions (grants), Canada has leveraged its own experience in facilitating common approaches to data development and information sharing.
At the time of the global economic recession in 2008, there existed no common international database for the comparison of countries' SME financing policies. Canada was identified as a leader as the nation was equipped with the most robust dataset among OECD member nations. The program funded a portion of a pilot project entitled the OECD Scoreboard on SME Financing and Data and Policies, which created a global scoreboard under Canada's leadership. After the funded pilot project was completed, it was very well received and led to a fully funded project that involved many other OECD member countries. This scoreboard has and will be a valuable resource for policy decisions as more countries are added to the Scoreboard.
Another example of a common approach to data development involves the Development of a Global Consumer Product Recalls Database, developed by the Working Party on Consumer Product Safety. Prior to this project, a standardized global database for recalled consumer goods did not exist. With the increasing prominence of international trade, consumer protection of international products became an issue of greater importance. This project targeted a global gap by collecting and translating information from various countries in order to create a valuable source of information for consumers around the globe. A beta version of this database is set to launch in October 2012, and will be publicly available as well as used by various member countries, including Canada.
3.2.2 To what extent has the OECD Grants Program increased Canada's influence at the OECD? Have the OECD funded projects informed and advanced IC's policy and research agenda?
The work and research plan of the OECD is in part shaped by voluntary contributions (i.e. grants) of member countries. Those grants made via the OECD Grants Program allow the department to advance its policy and research agenda parallel to the collaborative work of the organization. Additionally, the funded projects allow IC to further benefit from the expertise that is housed within the OECD as a way to inform and advance the department's objectives.
Every two years, OECD Counselors on the permanent delegation form a Programme of Work and Budget in order to define projects that will be undertaken by working parties within each committee. Interviewees stated that Canada has been proactive in this stage to ensure that Canadian interests are well reflected in OECD projects. Additionally, many IC program recipients are actively involved in OECD committees and working parties and have direct influence on both the planning and the implementation of OECD projects.
Interviews with program recipients also indicated that the OECD Grants Program is instrumental in leveraging the expertise that is concentrated at the OECD. For a relatively low grant amount, branches are able to harness knowledge from OECD experts, foreign government officials, and private sector representatives. Staff of sponsoring branches stated in interviews that the facilitative role of the OECD Grants Program has improved Canada's image and allowed them to actively participate in new initiatives and occasionally lead projects at the OECD.
One particular projectFootnote 4 that compared business innovation policies among various nations demonstrates the benefit of the program in increasing Canada's influence at the OECD. Industry Canada was the sole contributor in this project and was able to guide it in the way that best suited the sponsoring branch's mandate. IC's contribution also increased the scope of the project to include additional countries, furthering Canada's ability to compare domestic innovation policies to other similar economies, and expanding the project's potential influence on IC policy development. During interviews, it was stated that it is unlikely that this project would have been undertaken without Canada's contribution. Further, references to the project in the Jenkins Report demonstrate its contribution to informing and advancing the policy and research development of both the department and more broadly, the federal government. In this case, IC's grant funneled directly into an initiative that has informed government decisions on improving policy on productivity and innovation.
3.2.3 Has the grant program been administered efficiently and economically?
The OECD Grants Program is sourced from existing departmental budgets. There is very minimal cost and time burden associated to administering the program and any associated administrative costs are absorbed by SPB.
The project approval process is seen by staff and recipients as being straightforward and the OECD DG Committee comments provided good value-added to the project proposals. Committee approval is conducted by email, and hence does not require a substantial time commitment from participating DGs. Additionally, the low number of projects that have been approved through the Grants Program does not pose a time constraint to either program staff or DG Committee members. Overall, the approval process is seen as efficient.
After project proposals have been approved, many recipients stated that the financial transfer process with CAS and the OECD took longer than expected. Program staff, as well as earlier recipients of the program, noted the absence of a clearly outlined procedure on the mechanisms of the payment process. Evidence suggests that this process was unclear which had a negative impact on the timeliness of the transfer and the start-up of projects. Many recipients stated that a document outlining the entire process would have been very helpful in planning and implementing the transfer. However, some recent program recipients articulated that the administrative process has improved and instructions for users had been developed by SPB.
Without the OECD Grants Program, an alternative for the Department would be to pursue a separate authority through a Treasury Board submission for providing financial support to each individual OECD project. The time, effort, and expense of developing a Treasury Board submission for minor initiatives with relative low materialityFootnote 5 would be more onerous and discourage branches from funding OECD undertakings that would otherwise contribute to achieving departmental objectives. Interviewees also noted that the level of effort to distribute contributions through the Treasury Board process would be an inefficient use of internal resources. Thus, the program provides a more streamlined approval process for individual projects while maintaining the proper accountabilities.
Alternatively, it was suggested by some interviewees that a sole-source contract may, for smaller projects, be another option for funding an OECD initiative. However, these interviewees stated that this situation is not optimal as they would have to contract out a specific piece of work within the OECD project. Using this method, projects funded by IC would be limited to a low dollar value threshold. Therefore, when assessing alternative methods, the OECD Grants Program presents an option that is more efficient and economic when funding individual OECD projects.
When asked about awareness levels around the OECD Grants Program, many recipients as well as program staff stated that due to the structure of the approval committee, branches that may be in need of a mechanism to fund OECD projects would find out about this program through their respective DG, CAS or departmental OECD committee representative.
Regarding relevance and performance of the OECD Grants Program, the following conclusions can be reached.
- There is a need for the OECD Grants Program as it allows branches within Industry Canada to fund OECD projects that will contribute to their mandated policy and research areas.
- The program fits within federal roles and responsibilities as it aligns with the Department of Industry Act of 1995.
- Projects funded through the program are in line with federal government priorities and the strategic outcomes of Industry Canada.
- OECD projects funded through the program have contributed to common approaches to data development and information sharing between Canada and other OECD member countries.
- As a result of the program, Canada's influence on OECD committees and projects has increased through the provision of voluntary contributions. Contributions also allow sponsoring branches to further inform and advance departmental policy and research priorities.
- The program has been administered efficiently and economically as it uses existing reference levels and does not impose a significant time burden on program staff or the program DG committee.
Overall, the evaluation did not find any major issues with the OECD Grants Program and as a result makes no recommendations.
- Date modified: