Evaluation of the Spectrum Management Business Plan Investments (BPI)

2.0 Evaluation Framework

A formal evaluation framework was not required in 2001 when the program received funding. However, the TB Submission did contain expected results and indicated how those results would be measured. Based on the TB Submission, TB Evaluation Policy, a document review, and selected strategic interviews, the consultants developed the following components of the evaluation framework:

• BPI Logic Model;
• Outcomes Measurement Strategy, including Indicators and Data Sources;
• List of Evaluation Issues; and
• Evaluation Methodology, including Indicators and Data Sources.

Subsequently, a workshop with key members of the Evaluation Steering Committee (for a list of members, see Annex A) was held to discuss, modify and validate the evaluation framework. The final evaluation framework is detailed in Annex B. At that time, a prioritization exercise was conducted to provide input on the relative level of effort that the evaluators should place on the evaluation issues. The results of this prioritization exercise are contained in Annex C. This feedback was incorporated into the evaluation process.

top of page

2.1 Evaluation Objective

This evaluation focused on the areas of relevance, success, and cost effectiveness of the program and provides input into programming decisions regarding the ongoing funding of these types of activities. The evaluation assessed the effectiveness of the program in achieving its objectives, impacts, intended and unintended, continued relevance and alternative ways of achieving expected results.

2.2 Evaluation Issues

The list of evaluation issues outlined below was developed based on the TB Submission, TB Evaluation Policy and existing documentation. The issues were then confirmed and refined through consultations and Steering Committee direction.

Relevance

The continued relevance of the BPI, including whether or not it is consistent with IC and government priorities:

• What need does the BPI satisfy?
• Are the investments identified appropriate for satisfying this need?
• Is there a clear and relevant vision and objectives for Spectrum/Telecom Program activities under BPI?
• Does this program continue to be consistent with Spectrum/Telecom Program, Industry Canada and government-wide priorities?

Success

An assessment of the success of the BPI; to what extent has BPI achieved its objectives:

• Were the planned activities implemented and did they produce the expected outputs?
• Have the expected outcomes been achieved through the activities identified?
• Have there been any unexpected outcomes?

Cost-Effectiveness

• Are there appropriate management and decision-making structures in place to meet the objectives?
• Has there been an assessment and use of lessons learned? Where, when and how were lessons used?
• For identifiable groups of activities, has there been consideration of options/alternatives to increase cost-effectiveness or efficiency in delivery? Were any options found? Which were tried and what was the impact?

Funding

• Was the funding sufficient?
• Is the use of program, sunset funding the most appropriate way to fund these activities?

Design and Delivery

The extent to which the design and delivery of the investment initiatives facilitates established objectives.

• Were the programs and activities designed and delivered with the appropriate management accountability?
• Could these needs be satisfied through a mechanism other than BPI?

Lessons Learned and Recommendations

• What are the relevant lessons learned from the BPI and what recommendations can be made for future policy and programming needs?

top of page

2.3 Evaluation Methodologies

The evaluation was conducted using a multi-method approach, including:

• Document and File Review;
• Review of Web Publications;
• Key Management Interviews;
• Central Agency Interviews; and
• Key Stakeholder Interviews.

How each of these methods was used is discussed in more detail below.

Document and File Review

The document and file review provided a solid understanding of the purpose of the Business Plan Investments in the Spectrum / Telecom Program, including the activities that were undertaken. The information contained in these documents assisted in measuring planned versus actual outputs and outcomes. The BPI documents also were used to provide information on management practices, and lessons learned. Two key documents were an interim and final performance assessment of the BPI program, conducted in 2002 and 2004 respectively. A full list of the documents that were supplied by the Spectrum / Telecom Program and reviewed is listed in Annex D.

Review of Web Publications

A variety of the outputs related to the BPI fund have been published on the IC website. A review of the published documents was conducted. The information contained in these documents assisted in measuring planned versus actual outputs and outcomes.

Central Agency Interviews

A total of 14 key management interviews (see Annex E) were conducted with IC representatives, from Headquarters and the Regions. These interviews focused on the overall program, including the achievement of objectives and results. The interview guides for these interviews is provided in Annex F.

Key Stakeholder Interviews

A total of ten key stakeholder interviews were conducted with key stakeholders external to Industry Canada. This included representatives from federal central agencies, industry associations, private sector clients and partners, manufacturers and testing laboratories. These interviews focused on the achievement of outputs and outcomes. The list of interviewees is included in Annex E and the interview guides are provided in Annex F.