Evaluation of the Spectrum Management Business Plan Investments (BPI)

5.0 Cost-Effectiveness

This section examines the management and decision-making structures, the use of lessons learned and consideration of alternatives. Data was collected from key management interviews and file review of planning and reporting documents and meeting minutes.

5.1 Are there appropriate management and decision-making structures in place to meet the objectives?

The primary forum for planning and management of the BPI funds is the Directors General Spectrum / Telecom Committee which includes the DGs of the three Headquarters Branches and representation from each Region. Meetings were held approximately three times a year for overall management of the Spectrum / Telecom Program and include a standing item on the BPI funds. In addition, monthly DG teleconferences included BPI items on an as-and-when required basis. These reviews are described as very active discussions and require substantial time.

The DG Committee uses a consensus model and focuses primarily on planning, funding allocations, shortfalls and reallocations, and the funding formula. An Office of Primary Interest (OPI) was established within DGRB in 2002 in response to recommendations in the mid-term performance assessment to coordinate administration.

Improvements in the planning and review functions have been observed overt the period under review in terms of records of decision, supporting documentation, and developing criteria for setting priorities and accessing funds. Examples of critical issues that were brought forward at this forum for discussion and decision include:

• Record of discussion in 2001 on reallocating funds to succession planning;
• Reallocation from BPI to DGRB for a salary and O&M shortfall. Carried through to 2002 and 2003;
• Record of discussion in 2002 for high priority items: succession planning, security related initiatives, SEED/ALS/SOTT initiatives;
• Record of decision to reallocate funds to DTH 2003; and
• Record of decision to reallocate some funds to DGSE for salary shortfalls.

Day-to-day project management occurs at the DG and Regional levels for individual projects and activities. This approach is considered appropriate given the breadth of projects and individual requirements, particularly in the Regions. Each Region and Branch appears to have established internal management processes which vary significantly across the country. Co-ordination between Ottawa and the Regions appears good at the project level. However, the lines of authority to senior management are less clear for the Regions. The Regions have a direct line of reporting on operational activities to the ADM of Operations and for the BPI activities, they also report to ADM of SITT.

Flexibility has been identified as critical to the management of the Spectrum / Telecom Program given the dynamic nature of the environment. The Program strove to balance stability with flexibility through a funding formula. Roughly 80% of the funds were allocated to the Branches and Regions based on an agreed-upon formula. The remaining 20% was then made available for new pressures and priorities. The program did change with the environment and responded to new requirements, with examples already noted in cyber security, succession planning, DTH and antennae issues. Some improvements could be made in the rigor of the process used to establish priorities.

In summary, the primary forum for management and decision-making was administered by consensus through the DG Level Committee. This DG committee focus was on financial management, with output and outcomes managed individually by the DGs. Individual DGs and Regions have independence of action within the bounds of BPI objectives. At the project level, there were clear lines of authority for planning, managing and implementing projects.

Management was characterized by a balance of stability and an appropriate degree of flexibility to allow the Program to respond to the dynamic environment. Improvements have been noted in terms of management process and documentation. Possible enhancements to management and decision-making include more rigorous processes for priority setting and more timely identification of free balances as year-end approaches to facilitate reallocation.

top of page

5.2 Has there been an assessment and use of lessons learned?

A comprehensive mid-term review of the BPI program was completed in November 2002. The objective of the review was to determine if the projects and activities being implemented were effectively managed in accordance with the objectives and expected results set out in the Treasury Board Submission. The review was presented at DG Management meetings and recommendations were reviewed. Key recommendations and the actions taken are outlined below.

Mid-Term Review of the BPI Program

Recommendation	Action Taken
The establishment of an office of primary interest.	Completed. DGRB assumed role and assigned management and administrative leads.
The inclusion of regional representation in the management of the program.	Completed. Regional participation at quarterly management reviews and monthly teleconferences.
Revision of the format of the presentation of the financial information to communicate impact of BPI	Financial information presented and reviewed regularly at DG meetings by Branch. Not well linked to activities, outputs or outcomes.
Revisit the funding allocation formula to assess need to increase regional allocation.	No evidence of discussion of increasing regional allocation. Some revisiting of formula in 2003 to discuss if high priority funds should be increased.
Bring together the available results information and improve reporting with respect to the 6 measures outlined in the TB Submission.	IC has met its commitments for five of the six results measurement undertakings. Information remains limited for reporting on the success or failure of individual spectrum recipient organizations.
Consider whether a separate reporting document is required to communicate achievements of BPI activities.	Final performance assessment completed. Separate reporting document not implemented.

In addition, a final performance assessment was completed in 2004 with the following aim: "To obtain and analyze performance information and provide a performance assessment of initiatives implemented as Business Plan Investments (BPI) under the five-year funding approved by Treasury Board. The assessment will focus on the achievement of the expected results and the related means of measurement as set out in the Treasury Board Submission prepared to obtain the funding." The results of this assessment were shared within the Program.

Mid-term and final performance assessments were conducted for the BPI funds and the majority of the lessons learned from the mid-term assessment were discussed and implemented/rejected.

top of page

5.3 Alternatives

The nature of the BPI activities creates special challenges in assessing the consideration and use of alternatives. First, the BPI activities are broad categories of activities and a diverse number and type of actual projects can be funded under a category. Second, the funding is often incremental to A-Base funding rendering alternatives assessment very difficult without examining the entire Spectrum / Telecom Program.

Notably, an estimated 36% of the funds were directed to the activity grouping "improving service delivery" which is itself targeted at increasing cost-effectiveness and efficiency. In addition, there were isolated instances of structured process improvement initiatives. For example, two Regions conducted a structured review of service delivery compared to best practices based on Baldridge principles. Improvements were targeted and client satisfaction tracked. The other Region has implemented Process Improvement Teams.

5.4 Conclusions

Conclusion #5-1: The primary forum for management and decision-making was the joint DG Level Committee including the three Branches and the Regions. This DG committee focus was on financial management, with output and outcomes managed individually by the DGs. Individual Branches and Regions have independence of action within bounds of BPI objectives. At the project level, there were clear lines of authority for planning, managing and implementing projects.

Conclusion #5-2: Management was characterized by a balance of stability and an appropriate degree of flexibility to allow the Program to respond to the dynamic environment.

Conclusion #5-3: Improvements have been noted in terms of management process and documentation.

Conclusion #5-4: Possible enhancements to management and decision making include more rigorous processes for priority setting and more timely identification of free balances by project as year-end approaches to facilitate reallocation.

Conclusion #5-5: Mid-term and final performance assessments were conducted for the BPI funds and the majority of the lessons learned from the mid-term assessment were discussed and implemented or rejected.