Evaluation of the Council of Canadian Academies

From: Innovation, Science and Economic Development Canada

March 16, 2018

Table of contents

Evaluation of the Council of Canadian Academies PDF - 1.35 MB


top of page

Executive summary

Summary of Recommendations

The findings from the evaluation led to the following recommendations to ISED.

  1. Develop, in consultation with the CCA, a coordinated approach to improve the dissemination of CCA assessments.
  2. In collaboration with the sponsoring federal department/agency, strengthen its tracking of the results and impacts of ISED-funded CCA assessments.
  3. ISED should review its submission and approval process to simplify and better respond to the timelines and needs for scientific knowledge by the federal government. Going forward, consideration should be given to a process which would increase predictability for both the CCA and the federal government.
  4. Develop service standards for key steps in the proposal submission and approval process.

The evaluation assessed the relevance and performance of the CCA from 2005-06 to 2016-17 based on qualitative and quantitative research.

The evaluation found that the CCA responds to a need for independent, objective, and transparent scientific knowledge to support evidence-based decision-making. Requests for CCA assessments have been steady since its inception.

In terms of performance, evidence suggests that the CCA has achieved its expected results. The CCA has produced credible, independent and evidence-based assessments for the federal government. Although the CCA conducts some awareness activities to highlight its assessment findings, more can be done in terms of dissemination of its assessments. There is evidence that the CCA assessments funded by ISED have supported decision-making. However, the results and impacts of assessments are difficult to measure and are not tracked by ISED.

ISED's proposal submission and approval process is effective at supporting the steady conduct of CCA assessments on behalf of the federal government.
However, it is both lengthy and unpredictable and federal departments/agencies have recently been submitting more proposals directly to the CCA for urgent assessments using their own funding.

Overall, the evaluation found that CCA program delivery is efficient and effective.
The CCA's use of volunteer experts helps drive down the cost of its assessments.  Assessment and administration costs are in line with expectations, and the organization has improved the leveraging of funding from other sources in recent years. Timelines to complete an assessment are largely consistent with targets.

In terms of the CCA's governance and operations, it has made improvements in recent years, which have contributed to efficiency and effectiveness gains in the production of assessments.

The Council of Canadian Academies (CCA) is a not-for-profit organization created in 2005 to produce assessments that cover a broad range of science. Innovation, Science and Economic Development Canada (ISED) has been supporting the CCA on behalf of the federal government since its inception.

top of page

Background

Program Description and Objectives

To achieve its objectives, the CCA reviews assessment requests to ensure that:

Convening experts. Assessing evidence. Informing decisions.

Council of Canadian Academies

Objective of the CCA

The Council of Canadian Academies (CCA) is a not-for-profit organization that was created in 2005 to produce assessment reports that cover a broad range of science, incorporating the natural, social, and health sciences, as well as engineering and the humanities.

The objective of the CCA is to enhance Canadians' access to the best available scientific knowledge on pressing issues by providing credible, independent assessments that can inform debate, discussion, and decision-making.Footnote 1 The assessments are conducted by multidisciplinary panels of volunteer experts from across Canada and abroad.

About the CCA Assessments

The overarching goal of the CCA assessments is to evaluate and present the best available evidence on complex issues where the science may be challenging to understand, conflicting, or difficult to gather. The assessments strive to identify emerging issues, gaps in knowledge, Canadian strengths, and international trends and practices. The assessments do not provide recommendations and instead, formulate conclusions based on existing scientific knowledge.

Stakeholders who request assessments do not have a role or influence in the conduct of these assessments, and do not have access to them until they are completed and made publicly available.

Previously, the CCA offered three types of assessments: standard, streamlined and workshops. However, in 2016-17, the streamlined and standard assessments were collapsed into one overarching category, as it was found that streamlined assessments require similar resources in terms of volunteer experts, meetings, and time to gather the information and assemble the final product.

CCA Organizational Structure

The CCA is led by a 12-member Board of Directors, which is responsible for setting the strategic direction of the CCA. It is supported in its governance by its three founding Academies. Each Academy appoints two members to the Board; these appointments can be Fellows or senior decision-makers of the Academies. These six members then together appoint two Directors from the general public. The remaining four Directors are proposed by the Minister of Innovation, Science and Economic Development and are formally appointed by the Board. A representative from ISED participates on the Board as an observer.Footnote 3

The Board also appoints a Scientific Advisory Committee (SAC) that is composed of experts who have earned credibility among their peers and who have developed a national or international reputation in their field of research. The SAC provides expert advice on the suitability of potential subjects for assessments and input on the selection of expert panelists and reviewers.

CCA Organizational Structure Chart

Description of the CCA Organizational Structure Chart

The CCA Organizational Structure chart depicts the hierarchical structure and reporting relationships at the CCA. At the highest level of the organization chart is the CCA Board of Directors which is comprised of: the Canadian Academy of Engineering, Canadian Academy of Health Science, and the Royal Society of Canada. The Board of Directors is supported by the Scientific Advisory Committee. The second level of the organization chart is the CCA President. The third level is comprised of the Vice President and Chief Financial Officer, Director of Assessments, and Director of Communications and Publishing. Lastly, the Assessment Teams (which include Volunteer Expert Panels that are comprised of Canadian and International volunteer experts) report to the Director of Assessments while the Publishing and Communications Team reports to the Director of Communications and Publishing.

Requests and Funding for CCA Assessments

ISED Funding for CCA assessments

To support the CCA's assessment work, the CCA received an upfront grant of $30M from ISED in Budget 2005 for 10 years.Footnote 5 The CCA's funding was renewed in Budget 2015 for five years with an annualcontribution of $3M from ISED.Footnote 6 Budget 2018 proposed to provide the CCA with renewed funding of $9M over three years, beginning in 2020-21.Footnote 7

Between 2005-06 and 2016-17, the CCA completed 36 assessmentsFootnote 8 of which 30 were ISED-funded (see Appendix A):

  • 26 of 27 standard were ISED-funded
  • 3 of 6 streamlined were ISED-funded
  • 1 of 3 total workshops were ISED-funded

To date, ISED has launched a total of 14 calls for proposals.

Requests for CCA Assessments

The CCA receives requests for assessments from federal departments and agencies either through ISED's call for proposals or through requests made directly to the CCA. It is also encouraged to seek assessment requests from other levels of government, non-governmental organizations, and the private sector.

ISED's Submission and Approval Process

ISED funds CCA assessments on behalf of the entire federal government. Assessments are selected through a process that begins with a call for proposals. To date, 40% of the completed assessments funded through ISED have been for departments and agencies other than ISED.

ISED's Science Programs and Partnerships Branch is responsible for the implementation of the contribution to the CCA, the ongoing management and oversight of the funding for assessments, the launch of the call for proposals, and the proposal submission and approval process.

Proposals submitted to ISED by federal departments and agencies are first reviewed and recommended for approval by ISED based on the following criteria before referral to the CCA:

At least 3 assessment requests were submitted to ISED by departments and agencies during each call for proposal.

top of page

Methodology

The evaluation addressed the following questions:

Evaluation Objectives, Scope and Approach

The evaluation covered the period from 2005-06 to 2016-17.

Evaluation Objectives and Scope

The objectives of the evaluation were to examine the relevance, performance, efficiency and effectiveness of the CCA over time.

ISED's Audit and Evaluation Branch (AEB) undertook an in-house evaluation of the CCA. The evaluation covered key evaluation issues of relevance and performance.Footnote 9 The evaluation focused on:

Evaluation Approach

The relevance of the CCA was assessed based on the extent to which there is a continued need for the CCA and its assessment work. The evaluation assessed the performance of the CCA against its logic model (see Appendix B). In assessing the efficiency and effectiveness of ISED's funding, the evaluation focused on program delivery and ISED's oversight. It also assessed the efficiency of the CCA's operations in terms of timeliness and the cost of assessments, as well as recent improvements to the CCA's governance structure.

External Evaluation in 2013

As part of its Funding Agreement with ISED, the CCA was required to conduct an external evaluation in 2013.Footnote 10 The CCA Board of Governors assembled an external evaluation panel to undertake the evaluation. The panel produced findings in relation to the relevance, performance and efficiency of the CCA.Footnote 11

Data Collection Methods

The evaluation was based on qualitative and quantitative research methods. Data limitations are described in Appendix C.

Data Analysis

An analysis of the CCA's performance, administrative and financial data was performed. ISED's program data was also included in the analysis.

Bibliometric Analysis

Bibliometric analysis included an analysis of citations (e.g., direct and indirect citation counts based on Scopus and Google Scholar databases). It also included an analysis of authors (e.g., geography and affiliations) and publication sources (e.g., journals, books, patents, conferences).

Literature and Document Review

A review of academic literature related to the mechanisms and approaches used to deliver evidence-based scientific knowledge to support government decision-making was performed. In addition, federal government priority setting documents, ISED program and foundational documents, and CCA corporate and governance documents were also reviewed.

Interviews

A total of fourteen group and individual interviews were conducted with ISED program staff, federal departments and agencies that have requested assessments directly from the CCA, CCA staff , the Academies, members of the Board of Governors, and the Science Advisory Committee.

Case Studies

Seven CCA assessments were selected for the case studies (State of S&T 2006, State of S&T 2012, Conducted Energy Weapons 2013, Shale Gas 2013, Science Culture 2014, STEM Skills 2015, and Regenerative Medicine 2017). Goss Gilroy Inc. was contracted to conduct these case studies, which included a document review (e.g., sponsor surveys, lessons learned documents), an analysis of CCA performance data (e.g., citations in federal documents, media and academia), and twenty-three interviews.

International Comparative Analysis

Four international organizations were selected for the comparative analysis (US National Academy of Sciences, UK Royal Society, Australian Council of Learned Academies, and the German National Academy of Sciences Leopoldina). Goss Gilroy Inc. was contracted to perform a comparison of the CCA's operations and governance to international organizations with similar mandates (i.e., as independent sources of expert scientific assessments). As part of this analysis, three interviews were conducted with international organizations.

top of page

Findings: Relevance

Continued need for scientific knowledge and CCA assessments

International principles adopted by the CCA for delivering scientific knowledge

The United Kingdom, New Zealand, and Japan have each developed principles to strengthen the mechanisms for delivering scientific knowledge to support evidence-based decision-making. All three jurisdictions identified independence, objectivity, and transparency as key factors for effective delivery of scientific knowledge.Footnote 16

The CCA has adopted several international principles for delivering scientific knowledge to support evidence-based decision-making, such as independence, objectivity, and transparency

Finding: The CCA addresses a need for independent, objective, and transparent scientific knowledge to support evidence- based decision-making. Requests for CCA assessments have been steady since its inception.

Need for independent, objective, and transparent scientific knowledge

A large body of literature has identified a global need for independent, objective, and transparent mechanisms to deliver scientific knowledge that can be used to support governments in evidence-based decision-making.Footnote 12Footnote 13 The body of literature also emphasized the roles and responsibilities of expert bodies, such as national academies, which is the model used by the CCA, in evidence-based decision- making. A scan of the CCA assessments, coupled with interviews, found that the CCA provides publicly available, independent, objective assessments of the current state of scientific knowledge on a wide range of topics to support evidence-based decision-making and educate Canadians.

Need for scientific knowledge in federal decision-making

The mandate letter to ISED's Minister of Science emphasizes that the federal government believes that good scientific knowledge should inform decision-making. It also sets out an overarching goal for the Minister of Science to support scientific research and the integration of scientific considerations in the federal government's investment and policy choices.Footnote 14 Budget 2017 highlighted the Government of Canada's commitment to an evidence-based approach to decision-making. It also cited the CCA, further substantiating the fact that its assessments are supporting federal government decision-making.Footnote 15

Need for CCA assessments by federal departments and agencies

A review of the total number of proposals for CCA assessments submitted by federal departments and agencies since 2005-06 revealed that there has been steady demand across the federal government. Specifically, 32 of the 36 completed CCA assessments, or 89%, were funded by the Government of Canada. Of this total, 30 assessments were funded through ISED's submission and approval process and two were funded by departments/agencies through a direct request to the CCA. Most interviewees suggested that the demand for CCA assessments will continue to grow given the federal government's priority for credible scientific knowledge to support evidence-based decision-making.

top of page

Findings: Achievement of Expected Outcomes

Producing credible, independent, and evidence-based assessments

Credible and independent volunteer experts

The CCA has diverse representation of volunteer experts on its assessment panels in terms of gender, discipline and geography. Further, the CCA's Expert Panel Exit SurveysFootnote 19 found that close to 90% of the respondents accepted the CCA's invitation to volunteer on an assessment panel because they wanted to contribute to science advice in the public interest.

Volunteer experts are recruited from the Academies, the general public, and other countries. The CALM requires that each volunteer expert undergo a thorough vetting process to ensure that experts not only have the expertise, but also the ability to prepare objective report content.

Finding: The CCA uses a comprehensive assessment methodology to ensure that its assessments are credible, independent, and evidence-based. The credibility of the CCA is also demonstrated by the volume of citations of its assessments across the world.

The Council's Assessment Lifecycle Methodology

The CCA uses a rigorous assessment methodology to ensure credibility. This process is referred to as the Council's Assessment Lifecycle Methodology (CALM) and it includes requirements and guidelines for all phases of an assessment:

It was stated in several interviews that the CALM is reviewed and revised regularly to ensure that the assessment process remains rigorous and credible.

Volume of direct and indirect citations of CCA assessments

The credibility of the CCA and its assessments is also demonstrated by the number of citations of its assessments. Bibliometric analysisFootnote 18 found that 31 of 36, or 86%, of all CCA assessments completed as of 2016-17 were cited by authors in their respective scientific fields using at least one publishing avenue (e.g., journal, book, patent, or conference). This translates to over 500 total direct citations of CCA assessments (i.e., citation counts of publications that had referenced a CCA assessment) and over 4000 indirect citations (i.e., citation counts of publications that had referenced a publication that cited a CCA assessment).

Citations across the world

All 26 of the ISED-funded standard assessments between 2005-06 and 2016-17 have been cited by authors in their respective scientific fields across 43 countries using at least one publishing avenue.

Increasing visibility and awareness

Experts' awareness of the CCA

Respondents from the CCA's Expert Panel Exit Surveys indicated that they had heard about the CCA either from: a previous assessment; colleagues and word of mouth; references in the media; the CCA's website; or the Academies. No respondents specified that they had heard of the CCA through federal government outreach.

Interviews and a review of the CCA's Expert Panel Exit Survey results also suggested that there is an opportunity for the CCA to improve its activities related to visibility and awareness.

Finding: While the CCA conducts some awareness activities to highlight its assessment findings, the dissemination of its assessments is limited.

Visibility and awareness of the CCA across the federal government

An analysis of program data found that at least 25 federal departments and agencies have submitted at least one request for a CCA assessment through ISED's call for proposals since its inception, suggesting awareness of the CCA. Given that the level of awareness was measured by the total number of proposals submitted by individual federal departments and agencies through ISED's call for proposals, the actual number of departments and agencies that are aware of the CCA may be higher. Interviews also revealed that, within the federal government, the CCA is largely known through word-of-mouth of scientific staff.

Visibility, awareness and dissemination activities of the CCA

Interviews, along with a review of CCA documents, revealed that the CCA's visibility and awareness activities include planning a release for each completed assessment, identifying stakeholders that would be interested in the assessment and disseminating it to a targeted distribution list.Footnote 20 The CCA produces videos, infographics, and snapshot summaries to communicate the findings in its assessments. It also does some media and social media outreach, but the extent to which this is done is inconsistent between assessments as interviews indicated that these activities are dependent on resource availability. It was also found from both the review of documents and interviews, that there were instances when the CCA was approached to disseminate its findings and engage in discussions with federal government officials, but this has been inconsistent between assessments. It should be noted that deliverables related to knowledge dissemination are not a requirement in the 2015 funding agreement for CCA assessments.

Recommendation: ISED, in consultation with the CCA, should develop a coordinated approach to improve the dissemination of CCA assessments.

Supporting discussion, debate, public policy development and evidence-based decision-making (1 of 2)

Finding: There is evidence that the CCA assessments have provided decision-makers with independent scientific information. However, the direct results and impacts of CCA assessments are difficult to measure and are not tracked by ISED.

The case studies found that assessments have supported discussion, debate, public policy development and evidence-based decision-making, as shown in the illustrative examples below.Footnote 21

Assessments have supported the federal government in its funding and priority decision- making.

State of S&T (2006/2012): The assessment identified funding priorities for research, which supported the update of the federal government's S&T Strategy in 2014. These funding priorities supported investment decision-making related to the tri-agency's initiative, Canada First Research Excellence Fund. The assessment was also cited in Budget 2013 and Budget 2015.

STEM Skills (2015): The assessment supported briefing material to senior federal government officials on the current situation of STEM skills in Canada. It also advanced discussions on how to support innovation, education and training in Canada. It was cited in Budget 2017 with regards to STEM skill distribution across gender, immigration status and Aboriginal identity.

Assessments have supported industry and the provincial government in their decision-making.

Conducted Energy Weapons (2013): The assessment supported provincial decision-making related to the operationalizing of conducted energy weapons (CEWs) for front line police officers in Ontario. At the national level, it supported the update of the national guidelines for the use of CEWs, which were originally developed in 2010.

Shale Gas (2013): The assessment supported provincial decision-making related to fracking in Nova Scotia, as well as provincial review panels on the use of hydraulic fracturing in Newfoundland and Labrador and New Brunswick. The assessment also supported the creation of a federal task team to discuss issues on shale gas. It also supported the Canadian Association of Petroleum Producers in the development of their 2015 risk registry.

Assessments have supported stakeholders in addressing their information and collaboration gaps.

Science Culture (2014): The assessment was used to create science curriculum packages for grades 3 to 7 teachers, which were posted on the federal government website. The assessment provided baseline data, where none previously existed, and supported teachers in the development of curricula in schools.

Regenerative Medicine (2016): The direct result that stemmed from the workshop assessment was the creation of a national organization in 2017, known as the Regenerative Medicine Alliance of Canada, an idea that was developed by the workshop participants to address a collaboration gap in the field of regenerative medicine.

Supporting discussion, debate, public policy development and evidence-based decision-making (2 of 2)

Challenges in measuring the direct impacts of CCA assessments

It was found that there is a challenge in measuring this type of impact given that the CCA does not formulate recommendations or policy advice that could be tracked and attributed directly to its assessments. In addition, policy development and decision-making are rarely attributable to one single factor.

Although many interviewees from the case studies were confident that these assessments provided credible and independent scientific information, they also recognized that challenges exist in measuring the direct impact of CCA assessments.

The comparative analysisFootnote 22 found that all international respondents also emphasized the challenges in measuring the impacts of their assessments, as outcomes of projects are often intangible and difficult to quantify. Instead, the international respondents indicated that measuring the impacts of their assessments were more focused on bibliometrics and the outcomes of the assessments for researchers.

Documenting the results and impacts of CCA assessments

ISED improved its proposal submission template in 2015-16 to include a description of the relevance and expected outcomes/impacts of the assessment. Specifically, federal departments and agencies are required to explain how the proposal is relevant to the federal government's and sponsoring department/agency's policy agenda; the expected socio- economic benefits; and how the assessment could inform future priorities and activities.

However, ISED does not require federal departments/agencies to report back on the results and impacts of CCA assessments, nor does it track the results and impacts of completed assessments that it funds through its call for proposals (though it is not required under the 2015 funding agreement). The results and impacts of assessments are reported only through an optional Sponsor Feedback Survey that is conducted by the CCA one year after an assessment has been completed.

Recommendation: ISED should, in collaboration with the sponsoring federal department/agency, strengthen its tracking of the results and impacts of ISED-funded CCA assessments.

top of page

Findings: Efficiency and Effectiveness

Efficiency and Effectiveness: ISED Program Oversight (1 of 2)

Key steps in ISED's proposal submission and approval process

ISED tracks proposal approval at two key stages: Director General (DG)-level Working Group, and the Assistant Deputy Minister Committee on Science and Technology (ADM-CST).

The DG-Working Group meets with the CCA's Science Advisory Committee (SAC) to discuss and select the proposals to refer to the ADM- CST. Upon ADM-CST approval of these proposals, Ministers from sponsoring departments prepare their letters to the Minister of Science requesting approval and referral of their assessment request to the CCA.

However, the program does not track and monitor the rationales to support decision- making at these two stages (i.e., reasons why a proposal was successful or unsuccessful).

Finding: ISED has established a proposal submission and approval process. However, the process is lengthy and unpredictable. Further, federal departments/agencies are submitting more proposals directly to the CCA for urgent and time-sensitive assessments using their own funding.

ISED's proposal submission and approval process

ISED has a 12-step proposal submission and approval process in place to fund CCA assessments on behalf of the federal government (see Appendix D). It has developed a guidance documentFootnote 23 for federal departments and agencies on the proposal submission and approval process through which ISED refers assessment topics to the CCA. This guidance document describes the:

The primary advantage of ISED's approach is that it allows for consistent and centralized oversight of the funding program and the proposal submission and approval process.

Timelines and length of the proposal submission and approval process

Interviews found that ISED's proposal submission and approval process for CCA assessment requests is lengthy, lasting approximately 12 months.

Further, the timelines associated with individual proposals at each step of the submission and approval process are not tracked by ISED. This makes it difficult to identify bottlenecks and improve the efficiency of the proposal submission and approval process.

Efficiency and Effectiveness: ISED Program Oversight (2 of 2)

CCA standing capacity as a result of ISED's calls for proposals

Interviews indicated that although the proposal submission and approval process is lengthy, ISED's continued support over the years has enabled the CCA to:

Predictability of the calls for proposals and the urgency of assessments

 A review of CCA documentsFootnote 24 and interviews confirmed that since 2016- 17 at least eight federal departments and agencies have approached the CCA expressing interest in submitting an assessment request directly to them, up significantly from the previous ten-year period (2005-06 to 2015-16) where only two federal departments had funded their own assessments. Further, ISED has not launched a call for proposals since 2015-16 given that the CCA has already reached its target of 18-20 assessments funded by ISED under the 2015 funding agreement.Footnote 25

Other than the lack of a call for proposals, interviews confirmed that reasons for submitting requests directly to the CCA include the urgency of an assessment, and the length and certainty of the approval and planning process. Where there is an immediate need for CCA assessments, federal departments require certainty that their topic will be approved. In these cases, they interact with the CCA directly to refine the question and scope, and plan the assessment. It was also found from interviews that the sign-off process for assessment requests that were submitted directly to the CCA were less onerous, with the planning and approval process being as short as a few months. Interviews also noted that certainty on the topic enables the CCA to take immediate action on resourcing needs and operational planning. These interviews emphasized the need for flexibility in the process for time-sensitive assessment requests.

Recommendations:

ISED should review its submission and approval process to simplify and better respond to the timelines and needs for scientific knowledge by the federal government. Going forward, consideration should be given to a process which would increase predictability for both the CCA and the federal government.

ISED should develop service standards for key steps in the proposal submission and approval process.

Efficiency and Effectiveness: CCA Program Delivery (1 of 2)

In assessing the efficiency and effectiveness of program delivery, the analysis focused on:

Finding: The CCA's use of volunteer experts helps drive down the cost of its assessments. Assessment and administration costs are in line with expectations, and the organization has improved the leveraging of funding from other sources in recent years. Timelines to complete an assessment are largely consistent with targets.

Allocation of expenditures

An analysis of CCA financial data found that total CCA expenditures from 2005-06 to 2016-17 was $46.5M, with an annual average of 3.9M.Footnote 26 Roughly 66% ($30.6M) of total spending was spent on the conduct, production and dissemination of assessments. This is in line with the funding agreement which specifies that at least 64% of ISED's funding should be spent directly on assessment-related activities.Footnote 27 Indirect expenditures, such as overhead and administration, took up the remaining 34% ($16.0M), consistent with the funding agreement.Footnote 28

Use of volunteer experts in the conduct of assessments

The CCA relies on volunteer experts for its assessment work, which is consistent with similar international organizations. An analysis of the CCA's volunteer data found that there are, on average, 13 panelists and 11 reviewers per assessment. The CCA has estimated that the average time that each panelist spends on assessment work is 24 days, while each reviewer spends 2.5 days. The CCA also estimated that the compensation for an expert volunteer would be $1200/day. With these estimates, the average contribution of volunteers is equivalent to $407,400 per assessment.Footnote 29

As such, the use of volunteer experts help drive down the cost of CCA assessments. With the average cost of an ISED-funded standard assessment calculated at $1.43M, the cost would be about $1.8M in the absence of volunteer experts.Footnote 30

Leveraging of funding from other sources

Since its inception, the CCA has been encouraged to attract funding from other sources to undertake additional assessments on a cost-recovery basis.Footnote 31 Over the last ten years (2005-06 to 2014-15), funding from other sources made up 5.8% ($2.2M). From 2015-16 to 2016-17, other sources made up 19.1% ($1.4M), representing a significant increase, which in part could be explained by the lack of call for proposals in recent years and the need for urgent and time- sensitive assessments.

Efficiency and Effectiveness: CCA Program Delivery (2 of 2)

Completion times for CCA assessments

Completion time
Type of CCA Assessment Average Completion (months) Target Completion (months)
Standard 27 18-24
Streamlined 18 12-18
Workshop 6 Up to 6

Overall, completion times for ISED-funded CCA assessments are largely in line with targets. For standard assessments, the CCA took an average of 27 months to complete, slightly more than the completion target of 24 months. For streamlined assessments and workshops, the average completion time fell within the expected timeframe of 12-18 months and 6 months, respectively.

As Figure 1 shows, the average time needed to complete an ISED-funded standard assessment varies, from a minimum of 19 months to a maximum of 34 months.Footnote 32 These factors are impacted by the complexity and scope of an assessment, which can further increase the completion time. A review of assessments, along with interviews, found that the completion times vary based on complexity of topic and scope. Factors that contribute to the length of time to complete an assessment are consensus building among multidisciplinary panelists, and the review and consideration of all the comments provided to the panel from the reviewers. As the CCA improved the rigour of its assessment methodology over the years, it took longer, on average, to complete ISED-funded standard assessments with complex topics and scopes. However, some factors that impact the completion time are outside of the CCA's control (i.e., delays in finalizing an assessment request).

Figure 1: Average Completion Time of ISED-Funded Standard Assessment

Description of Figure 1

Figure 1 is a bar graph that presents the average completion time of a CCA assessment in months categorized by fiscal year. The average completion time in months is located on the Y-axis while the fiscal year of completion is located on the X-axis. The bar graph covers the time period from 2006-07 to 2013-14. The trend shows that the average completion time of an assessment fluctuates between fiscal years. The average completion time of an assessment between 2006-07 and 2013-14 was 27 months.

The table below presents the data in the bar graph:

Fiscal Year Average Completion Time (in months)
2006-07 19
2007-08 23
2008-09 31
2009-10 26
2010-11 27
2011-12 34
2012-13 29
2013-14 21

Efficiency and Effectiveness: CCA Governance and Operations

Three founding Academies of the CCA

Finding: The CCA has made improvements to its governance and operations in recent years, which have contributed to efficiency and effectiveness gains in the production of assessments.

The role of the Academies in CCA governance

Effective and efficient working relationships between the CCA and the Academies are critical to the production of assessments. In recent years, the CCA has improved its governance by ensuring that the President of each Academy is one of the two Directors appointed. Interviews confirmed that this has improved dialogue between the Directors and simplified internal communication when reporting back to the individual Academies.

In 2016-17, a Collaboration Agreement was established between the CCA and the Academies to further improve the efficiency and effectiveness of the CCA governance.Footnote 33 The Agreement serves to clarify and formalize the roles and responsibilities of the Academies and lay out the terms and conditions for collaboration. It also serves to act as a mechanism to recognize the services of the Academies and remunerate them for documented expenses related to eligible activities under the 2015 funding agreement.Footnote 34 Eligible activities are activities directly related to supporting, conducting, and producing assessments on topics which are referred by the Minister of Science. The comparative analysis found that national academies within other jurisdictions also play similar roles in the governance of their respective CCA-like organization.

Horizontal oversight of CCA assessments and volunteer expert panels

The CCA has made recent changes to its organizational structure to improve the efficiency and effectiveness of its operations. A Director of Assessments position was created in 2016-17 to allow for consistent and horizontal oversight of all active assessments. This position is responsible for resource allocation and coordination across assessments, overseeing quality standards and report writing. Interviews confirmed that the staffing of this position hascontributed to better resource management and has improved the quality of assessments.

It was also found from both the document reviewFootnote 35 and interviews that the CCA has recently improved the selection process for its volunteer expert panels to ensure that it continues to produce high quality assessments. The vetting process for a Panel Chair was revised to include a review of research and social media profiles to ensure that Chairs not only have the expertise and proven chairing abilities, but also the ability to prepare an objective report. Co-chairs have also been recently introduced which has proven to be an effective and efficient approach when a subject has two well established positions or when a Chair has limited availability.

Improved relationship with the three founding Academies

It was noted from interviews that the relationship with the Academies has improved in the last two years, and has been further strengthened as a result of the signed Collaboration Agreement.

top of page

Conclusions

Overall, the CCA continues to be relevant and has achieved its expected outcomes. The program delivery model and CCA governance/operations are effective and efficient. However, there are opportunities to improve the dissemination of CCA assessments, as well as the efficiency and effectiveness of ISED's oversight of the CCA.

Relevance: The CCA addresses a need for independent, objective, and transparent scientific knowledge to support evidence-based decision-making. Requests for CCA assessments have been steady since its inception.

Achievement of Expected Outcomes: The CCA uses a comprehensive assessment methodology to ensure that its assessments are credible, independent, and evidence-based. The credibility of the CCA is also demonstrated by the volume of citations of its assessments across the world. While the CCA conducts some awareness activities to highlight its assessment findings, the dissemination of its assessments is limited. There is evidence that the CCA assessments have provided decision-makers with independent scientific information. However, the results and impacts of CCA assessments are difficult to measure and are not tracked by ISED.

Efficiency and Effectiveness of ISED Program Oversight: ISED has established a proposal submission and approval process. However, the process is lengthy and unpredictable. ISED has established a proposal submission and approval process.
However, the process is lengthy and unpredictable. Further, federal departments/agencies are submitting more proposals directly to the CCA for urgent and time-sensitive assessments using their own funding.

Efficiency and Effectiveness of CCA Program Delivery: The CCA's use of volunteer experts helps drive down the cost of its assessments. Assessment and administration costs are in line with expectations, and the organization has improved the leveraging of funding from other sources in recent years. Timelines to complete an assessment are largely consistent with targets.

Efficiency and Effectiveness of CCA Governance and Operations: The CCA has made improvements to its governance and operations in recent years, which have contributed to efficiency and effectiveness gains in the production of assessments.

top of page

Recommendations

The findings of the evaluation led to the following recommendations.

It is recommended that ISED:

  1. Develop, in consultation with the CCA, a coordinated approach to improve the dissemination of CCA assessments.
  2. In collaboration with the sponsoring federal department/agency, strengthen its tracking of the results and impacts of ISED-funded CCA assessments.
  3. ISED should review its submission and approval process to simplify and better respond to the timelines and needs for scientific knowledge by the federal government. Going forward, consideration should be given to a process which would increase predictability for both the CCA and the federal government.
  4. Develop service standards for key steps in the proposal submission and approval process.

top of page

Appendices

Appendix A: Assessments from 2005-06 to 2016-17 (1 of 2)

Assessments Funded by ISED
Assessments Funded by ISED Fiscal Year of Completion
1. The State of Science and Technology in Canada, 2006 2006-07
2. Influenza Transmission and Personal Protective Respiratory Equipment 2007-08
3. Energy from Gas Hydrates 2008-09
4. Nanotechnology: Small is Different 2008-09
5. Sustainable Management of Groundwater in Canada 2009-10
6. Better Research for Better Business 2009-10
7. Innovation and Business Strategy: Why Canada Falls Short 2009-10
8. Honesty, Accountability and Trust: Fostering Research Integrity in Canada 2010-11
9. Canadian Taxonomy: Exploring Biodiversity, Creating Opportunity 2010-11
10. Healthy Animals, Healthy Canada 2011-12
11. Integrating Emerging Technologies into Chemical Safety Assessment 2011-12
12. Informing Research Choices: Indicators and Judgment 2012-13
13. The State of Science and Technology in Canada, 2012 2012-13
14. Strengthening Canada's Research Capacity: the Gender Dimension 2012-13
15. Water and Agriculture in Canada: Towards Sustainable Management of Water Resources 2012-13
16. The State of Industrial R&D in Canada 2013-14
17. Aboriginal Food Security in Northern Canada 2013-14
18. Environmental Impacts of Shale Gas Extraction in Canada 2014-15
19. Enabling Sustainability in an Interconnected World 2014-15
20. Improving Medicines for Children in Canada 2014-15
21. Energy Prices and Business Decision Making in Canada: Preparing for the Energy Future 2014-15
22. Leading in the Digital World: Opportunities for Canada's Memory Institutions 2014-15
23. Science Culture: Where Canada Stands 2014-15
24. Policing Canada in the 21st Century: New Policing for New Challenges 2014-15
25. Accessing Health and Health Related Data in Canada 2014-15
26. Some Assembly Required: STEM Skills and Canada's Economic Productivity 2015-16
27. Technological Prospects for Reducing the Environmental Footprint of the Oil Sands 2015-16
28. Health Product Risk Communication: Is the Message Getting Through? 2015-16
29. Understanding the Evidence: Wind Turbine Noise 2015-16
30. Building on Canada's Strengths in Regenerative Medicine 2016-17
Assessments Funded by Other Departments/Agencies
Assessments Funded by Other Departments/Agencies Fiscal Year of Completion
31. Vision for the Canadian Arctic Research Initiative 2008-09
32. Health Effects of Conducted Energy Weapons 2013-14
Assessments Funded by Other Departments/Agencies
Assessments Funded by Other Departments/Agencies Fiscal Year of Completion
33. Innovation Impacts: Measurement and Assessment 2013-14
34. Ocean Science in Canada: Meeting the Challenge, Seizing the Opportunity 2013-14
35. Technology and Policy Options for a Low-Emission Energy System in Canada 2015-16
36. Commercial Marine Shipping Accidents: Understanding the Risks in Canada 2016-17

Appendix B: Program Logic Model

Figure 2: Program Logic Model

Description of Figure 2

Figure 2 depicts a logic model for the CCA. The logic model shows the relationship between inputs, activities, outputs and different levels of outcomes.

The logic model is comprised of six inputs:

  1. Expert Volunteers,
  2. Professional Staff,
  3. Federal Funding,
  4. Call for Proposals,
  5. Academies, and
  6. Other Funding.

These inputs are used to produce three types of Activities:

  1. Assessments which includes: preliminary scoping, panel and chair recruitment, panel meetings and logistical support, research and content development, peer review, sponsor briefing, and impact measurement;
  2. Communications which includes: outreach, dissemination, media releases, conference presentations, CCA newsletters, online presence, outreach plans; and
  3. Collaboration which includes: exchange scientific knowledge with other national academies around the world.

These program activities lead to two outputs:

  1. Expert assessment reports, and
  2. Assessment-specific communication products.

The outputs produce two immediate outcomes:

  1. Credible, independent, evidence-based scientific knowledge, and
  2. Visibility and awareness of the CCA's work.

These immediate outcomes lead to the intermediate outcome of "Informed discussion, debate and public policy development", which in turn leads to the ultimate outcome of "Science to support evidence-based decision-making".

Appendix C: Data Limitations

Data Availability: A standard CCA assessment has a target completion timeframe of 24 months and the CCA conducts a voluntary Sponsor Feedback Survey 12 months after the release of an assessment to collect data on assessment impacts. Often, the impacts of an assessment are either difficult to measure or are not observed until several years after its release. For the case studies, between this time period, sponsoring departments and agencies experienced employee turnover and retirements, particularly of those who were initially involved in the proposal submissions for assessments. As a result, the list of potential interview contacts for the case studies was limited.

Potential Bias in Interview Data: Interview participants holding the broadest knowledge of the CCA and its operations are individuals who were active or previous members of the CCA's Board of Governors and/or the CCA's Science Advisory Committee. The opinions and perspectives of these individuals may be slightly biased due to their involvement in the CCA's governance structure, however, this limitation was mitigated through triangulation of findings from other lines of evidence.

 Strength of International Comparative Analysis: Various factors limited the comparative analysis with international organizations. First, there was no organization fully comparable to the CCA in terms of objectives, funding model, operations, governance structure, and size. Second, the data necessary to compare the cost per assessments and overhead costs between the selected international organizations and the CCA was either insufficient or unavailable.

Appendix D: ISED Proposal Submission and Approval Process

Proposal Submission and Approval Process

Description of Proposal Submission and Approval Process
  1. ISED launches a call for proposals
  2. Departments and agencies submit a proposal summary for each assessment topic being proposed
  3. ISED assesses the proposals based on selection criteria and identifies eligible assessment topics
  4. Sponsors develop and submit full draft proposals for the assessment topics deemed eligible by ISED
  5. ISED provides feedback on the draft proposals for assessment topics and sponsors submit revised draft proposals
  6. ISED submits the revised draft proposals to the CCA's Scientific Advisory Committee (CCA-SAC) for its review
  7. The Director General (DG) Working Group, CCA-SAC, and Sponsors meet to discuss and address questions and comments pertaining to the proposals
  8. Final proposals and Assistant Deputy Minister (ADM) letters of support are submitted to ISED for a second round of review
  9. CCA-SAC and the DG Working Group select proposals to refer to the ADM Committee on Science and Technology (ADM-CST) for approval
  10. Upon ADM-CST approval, sponsoring Ministers prepare letters to the Minister of Science requesting the referral of the proposed topics to the CCA
  11. Minister of Science selects the proposals and requests the conduct of the assessments from the CCA on behalf of the Government of Canada
  12. CCA begins the proposed assessments

top of page

Endnotes

top of page

Management Response and Action Plan

Management Response and Action Plan—Evaluation of The Council of Canadian Academies PDF - 59.8 KB

A - For inclusion in the report

The findings and recommendations of the Council of Canadian Academies Evaluation Report were presented to program management. Management has agreed with the findings included in this report and will take action to address all applicable recommendations by May 31, 2019

B - For follow-up purposes - Detailed actions to address the recommendations in the report

Recommendation Planned Action on the Recommendation Responsible Official (position) Target completion date

Recommendation 1

Develop, in consultation with the CCA, a coordinated approach to improve the dissemination of CCA assessments.

Agreed.

SRS will work with CCA to develop and implement a coordinated approach to improve the dissemination of assessments within the federal government community.
This could include:

  • using the ADM Committee on Science and Technology to support the dissemination of assessment findings; and
  • requiring through a new funding agreement that the CCA deliver presentations on all assessments to sponsoring departments.
DG, Science Program and Partnerships, SRS March 31, 2019

Recommendation 2

In collaboration with the sponsoring federal department/agency, strengthen its tracking of the results and impacts of ISED-funded CCA assessments.

Agreed.

ISED will request federal departments/ agencies to report back on the impact and use of CCA assessments funded through its Calls for Proposals to inform government decision-making, potentially through the ADM Committee on Science and Technology. Given that there can be a lag between the provision of evidence and its influence on public policy, this effort is intended to significantly expand what is currently captured by the CCA through annual Sponsor Feedback Surveys.  

The CCA will be required to continue to report on assessment report bibliometrics and the outcomes of the assessments for researchers.
DG, Science Program and Partnerships, SRS and the CCA March 31, 2019

Recommendation 3

Review its submission and approval process to simplify and better respond to the timelines and needs for scientific knowledge by the federal government. Going forward, consideration should be given to a process which would increase predictability for both the CCA and the federal government.

Agreed.

A new process is needed that is not only streamlined, but also better aligns the resources available to support CCA assessments and the demand for such assessments from federal departments.  Demand can be managed by taking steps to ensure assessment topics being considered address government priorities.  Regarding the availability of resources to expand the CCA's capacity, various revenue-generating options could be considered, such as having departments pay a share of the costs even for assessments supported under the ISED contribution agreement.

  • SRS will work with the CCA and other bodies in the federal government as necessary, to redesign the assessment proposal development and approval process, streamlining and standardizing it where possible, while also considering the incorporation of new requirements to focus on assessments that are high-priority and/or co-funded by federal departments.
DG, Science Program and Partnerships May 31, 2019

Recommendation 4

Develop service standards for key steps in the proposal submission and approval process.

Agreed.

SRS will work with CMS and CCA to develop service standards to include in the proposal submission and approval process for key steps.

DG, Science Program and Partnerships May 31, 2019
Date modified: