First
PreviousMid-Term Evaluation — First Nations SchoolNet Program
Management Response
September 2005
Background
This formative evaluation of the First Nations SchoolNet Program (FNS) was undertaken by R.A. Malatest & Associates Ltd., on behalf of the Audit and Evaluation Branch and the Information Highway Applications Branch (IHAB) of Industry Canada.
The purpose was to provide Industry Canada with information pertaining to the management, delivery, progress and challenges of the program.
The evaluation methodology included 55 key informant interviews comprised of representatives from the following stakeholder groups: Industry Canada national and regional staff, Regional Management Organizations (RMOs) service providers and program staff, school administrators, and other stakeholders involved with the FNS program. In addition, two case studies were undertaken at the RMOs in Manitoba and Ontario, and a series of document reviews took place.
The following is a brief summary of the findings of the evaluation which are provided in more detail in the evaluation report:
- The FNS program continues to be relevant and necessary;
- Substantial progress has been made towards reaching the goals and objectives of the FNS program;
- FNS has had the intended impact among schools, teachers, and learners;
- RMOs continue to be an efficient and effective method of program delivery;
- Building local capacity was regarded as one of the most important factors governing the success of the FNS program;
- Case studies revealed several innovative programs have been created and supported through the FNS program.
First Nations SchoolNet seeks to provide First Nations schools on federal reserves with computers and other information and communications technologies (ICT) equipment, Internet access, enhanced broadband connectivity (1.5 mb), technical support and training to enable First Nations learners to pursue new opportunities for learning and skills development within their own communities.
Although FNS currently sunsets on March 31, 2006, this evaluation study will be very helpful in making recommendations to guide decisions for the future design of the program.
Management is committed to addressing each of the recommendations found within the evaluation study as set out in the detailed Management Response below, as applicable now and if the program continues.
Management Response to the Recommendations
Recommendation A: Increased communication between the schools and RMOs is required to support program delivery and partnership development.
Currently the schools and the RMOs experience barriers to communication including insufficient staff at schools to facilitate program delivery, partly due to high staff turnover, school staff unfamiliarity with RMO representatives, lack of regular e-mail use and general unresponsiveness of schools to communication efforts of the RMOs.
The Consultant believes that by overcoming these barriers and enhancing communication, the program will better support local skill and capacity building. Further, schools may gain a greater appreciation of the program and partners with more frequent communication resulting in a better working relationship during the tenure of this program. More communication with the schools will also increase the understanding of the program goals and objectives as well as the potential applications of the FNS resources provided to First Nations schools.
Since on-site visits require a substantial commitment of time and money due to the remoteness of the schools, videoconferencing may represent a plausible alternative for increased communication with the schools, where such technology exists. Other avenues to be considered to support communication include training seminars delivered on-line, videoconferences, or in-person workshops similar to those offered in the Youth Employment program.
Management Response:
Agreed. Management accepts the recommendation that increased communication between the schools and the RMOs is required to support program delivery and partnership delivery.
However, it is important to ensure that this requirement is addressed within the context of the parameters of program delivery. Specifically, Industry Canada delivers FNS through Contributions funding and as such must comply with the Treasury Board Policy on Transfer Payments. As stated in this policy, transfer payments are made to further approved federal government policy and program objectives, and funding is provided on the basis of achievement of performance objectives as set out in a contribution agreement. To ensure that specific accountabilities, milestones and program objectives are addressed, the instrument within which the program can influence these outcomes is through the contribution agreement and ongoing monitoring of the agreement. Once a contribution agreement is in place, any addition or modification of existing activities, milestones, or accountabilities may require an amendment to the existing agreement or the drafting of a new agreement.
Therefore, the recommendation to increase communications between the RMOs and the schools at this point in the project year requires an examination of the contribution agreements that are already in force.
With respect to the current contribution agreement requirements, management is committed to ensuring that the prescribed communication-related activities between RMOs and the schools continue to be undertaken in as efficient and effective manner as possible to support program delivery and partnership development. Within the parameters of the current agreements, FNS is in a position to encourage more communication via videoconferencing, where available, and increased web-based communications as part of the existing requirement for RMOs to provide ongoing service delivery to schools over the 2005-06 fiscal year.
With respect to enhancing and augmenting current communication-related activities between RMOs and the schools, management will take into consideration the increased cost and budget requirements related to on-site visits, expanded video-conferencing and other recommended forms of communication and monitoring in the future design of its program and by extension during the development of future request for proposals and contribution agreement documents.
Timeline: Fall/Winter 2005-2006
top of pageRecommendation B: The identification of how schools are using the resources provided by the First Nations SchoolNet program needs to be improved.
As the target audience of the program, First Nations schools receive connectivity and other ICT resources from the FNS program to assist students with their learning. Resources typically provided by the FNS program include connectivity upgrades, computer hardware and software, related equipment (including digital cameras and video cameras), and technical support through the Help Desk.
Although schools differ in their ability to integrate available technology to enhance learning opportunities, it was evident through school administrator interviews and case studies that several schools are leaders in resource application. Divergent skill levels among schools highlights an opportunity for knowledge transfer between schools. Dissemination of lessons learned and best practices among the schools could be directed by the RMOs, facilitated by the existing channels of communication, namely the national meetings that occur with FNS stakeholders.
Since each RMO signs a contract with the First Nations schools on an annual basis, the Consultant recommends that Industry Canada include a requirement for schools to report on how both students and teachers are using the FNS resources. This brief report would outline the applications of the resources sent to the school used for educational purposes and would also share challenges encountered by the school. The report would replace the annual surveys that have been typically affected by poor response rates. By having the schools detail the uses of the FNS resources, the RMOs would gain a better understanding of usage patterns, potential need for other technology, and any challenges impacting the school's ability to utilize the resources to their potential. Having the onus of reporting falling with the schools would also serve to assist the RMOs by reducing the administrative costs of trying to contact unresponsive schools. Ideally, the reports submitted by the schools to the RMOs would then serve to identify best practices, which could then be shared with other RMOs and hence other First Nations schools served by the program.
Management Response:
Agreed. Management supports the recommendation that it would be beneficial to have an improved method of identifying how both students and teachers are using FNS resources.
In support of this result, the current contribution agreement between the RMOs and Industry Canada requires the RMOs to draft and submit an ICT Strategic Plan. This Strategic Plan has been designed to be a multi-year plan spanning a three-year period. The implementation of this Strategic Plan will require schools to provide information not only on the usage of FNS resources by students and teachers within the school, but also on the impact the usage of these resources have had in the context of a First Nations community. However, the implementation of the Strategic Plan is subject to the program receiving authority beyond 2005/06.
Should this authority be granted, in preparation of future program contribution agreements and the subsequent articulation of project objectives, milestones and accountabilities, the program will undertake awareness sessions on the necessity of receiving information on usage from the schools. RMOs and the IC Regional Representatives will be engaged to identify and share solutions on how best to collect key metrics, such as usage. Program management will work closely with the RMOs to identify standardized reporting requirements for schools so that the program is better able to obtain information to assist with project monitoring and program policy purposes.
Management also supports the recommendation to encourage knowledge transfer and sharing of best practices between First Nations schools. The FNS program will take a more active role in facilitating this knowledge transfer through various means: encouraging RMOs to share known leading practices and invite schools in their jurisdiction to post their own experiences on currently available web-based communication systems; sharing leading practices at the semiannual national FNS stakeholder meetings and monthly RMO meetings for further dissemination by RMOs to the schools; working with RMOs to encourage and facilitate school events whereby they invite other schools across Canada to participate via videoconferencing; and in collaboration with the RMOs, produce video clips of leading practices in action for dissemination to other schools and other parties of interest.
Timeline: Winter 2006.
top of pageRecommendation C: Partnerships with other departments and organizations should be established to enhance the efficiency and sustainability of the program.
The goals and objectives of First Nations SchoolNet are closely aligned to the mandate of other programs and represents opportunities for partnerships. Thus, respondents noted that development of partnerships might help to facilitate increased connectivity as well as increasing learning opportunities in First Nations schools. Programs noted by respondents include the following Industry Canada programs: Broadband for Rural and Northern Development (BRAND), Computers for Schools, and the Community Access Program (CAP). Other potential partnering opportunities include building or strengthening relationships with Indian and Northern Affairs Canada (INAC), Health Canada, Atlantic Canada Opportunities Agency (ACOA), Western Economic Diversification Canada (WD), and educational partners to facilitate and expand on the services provided by the FNS program while leveraging funds and building sustain ability. It was also felt that partnering could help to reduce the "silo effect" which limits the overall success of programs and represents an opportunity to stream-line resources to the First Nations communities within a broader mandate.
The significance of partnerships has been demonstrated in the Ontario region with the Federal Economic Development Initiative for Northern Ontario (FedNor), the only economic agency involved with the FNS program. In total, FedNor has invested over ten million dollars into building and supporting connectivity and infrastructure in North-western Ontario. The investment by FedNor, coordinated with FNS resources, has led to other partnerships including the involvement of Health Canada who is currently purchasing connectivity from the Ontario RMO to deliver Telehealth to 24 First Nations communities.
In addition to FedNor's involvement with the program, five RMOs have implemented a Youth Employment program designed to provide youth with computer skills, thereby helping youth build local capacity while gaining employable skills. The funding for the youth program resulted in part from an investment from Human Resources Skills Development Canada (HRSDC). The partnerships with other programs as well as other organizations is mutually rewarding and helps to create a sustainable approach to providing connectivity and ICT resources to First Nations communities in Canada.
Management Response:
Agreed. Management supports the view that partnerships with other federal departments and organizations can enhance the efficiency and sustainability of the program.
In recent years, First Nations SchoolNet has been collaborating more closely with fellow IHAB programs, including the Broadband for Rural and Northern Development (BRAND) initiative. Both programs have been coordinating efforts towards providing broadband connectivity to First Nations communities on federal reserves. Specifically, these two programs worked together on a project in Québec in collaboration with the regional economic development agency (CED) and the Province of Québec, to bring connectivity to unserved communities in that province, including First Nations communities.
In support of information sharing and knowledge transfer between the two programs as it pertains to connectivity delivery for First Nations reserves, BRAND has been taking an active part in attending the FNS semi-annual national stakeholder meetings; the monthly technical meetings with RMOs; and an interdepartmental working group with INAC which seeks to provide a better understanding of this unique client group.
Over the current year, in co-operation with IHAB programs and other government departments (OGDs), FNS will continue to increase the adoption of ICT in First Nations communities through various means, including actively participating in INAC's interdepartmental Aboriginal Connectivity and Portal Working Group. In addition, FNS will explore and encourage aboriginal representation on existing consultation networks related to broadband connectivity. FNS will also continue to work with HRSDC via the Youth Employment Strategy to ensure effective program delivery as it pertains to engaging First Nations youth for the purposes of building local technical capacity in these communities.
In support of these partnerships, FNS continues to track data on the impact these partnerships are having on connectivity progress in First Nations schools. RMOs report on a monthly basis the degree of enhanced connectivity and the partnerships they are able to leverage at the local level to reduce/share costs for both government and end-users.
Timeline: Winter 2006
top of pageRecommendation D: The annual Contribution Agreement could be extended to secure funding to the RMOs for a longer time period.
A longer term funding commitment from Industry Canada would allow for better RMO planning and would facilitate the development of new partnerships or strengthen existing partnerships. Findings suggest that there are several drawbacks of the current annual funding agreement. Challenges include difficulty in leveraging funds from partners, demonstrating commitment to the schools through a sustainable approach, and forecasting budgets at both the RMO and the school level for the next several years. While annual funding agreements allow monitoring of the RMO progress and added flexibility to individualize their services, many RMO coordinators felt that the challenges far outweighed the benefits of this funding arrangement with special mention of reporting and paperwork burdens. It was felt that secure funding over several years would allow the RMOs to plan additional training opportunities for community members and youth, as well as, develop long-range hardware and connectivity strategies for First Nations schools. Additionally, RMO staff noted that a reduction of, or assistance with both progress and financial reporting would be appreciated. This assistance may be partly accomplished through greater promotion of existing templates, an extension of the annual contracts, or a reduction in reporting duties.
Extended funding arrangement were also felt to facilitate stronger partnerships with the schools, and provide a better opportunity to establish funding relationships with other government sponsored programs (CAP, BRAND), as well as initiatives from other organizations (Health Canada, INAC, WD, ACOA). Finally, with respect to the Contribution Agreement, respondents felt that it should be revised to account for the discrepancy between fiscal years of the RMOs (fiscal year) and the schools (academic year).
Management Response:
Agreed. Management supports this recommendation, with qualifications.
First Nations SchoolNet is a sunsetting program. In the event funding is extended beyond March 31, 2006, program design will take into consideration the need for multi-year funding and address the need to streamline the reporting requirements of First Nation non-profit organizations. These needs will be assessed against the context, parameters and conditions of a renewed program authority, Treasury Board's Policy on Transfer Payments and the assessment of risk.
Timeline: Fall/Winter 2005-2006
top of pageRecommendation E: Annual short-term goals and objectives of the RMOs should be aligned with the longer-term objectives of the FNS program.
While the overall program goals and objectives have remained the same, each RMO prepares a list of goals and activities to be completed for each annual Contribution Agreement, resulting in the short-term goals of the RMOs sometimes being noticeably different from overall program goals that have a longer-term focus. Given this discrepancy, the Consultant suggests that more work should be done to align the goals of the RMOs with the program. By working to better articulate the short-term and long-term goals of the program a more accurate assessment of the progress of the program could be obtained.
Management Response:
Agreed. Management accepts this recommendation, with qualifications.
An underlying principle of the Treasury Board Policy on Transfer Payments pertains to providing contributions to organizations that further approved government policy and program objectives. Therefore, it may be more representative to ensure that the short-term goals and objectives of the RMOs support the longer term objectives of the program, rather than to suggest that the current goals of these recipients be adjusted or 'aligned' to fit those of a government program. That is to say, FNS needs to determine that this alignment pre-exists as part of the assessment of project proposals from applicants seeking funding support.
In the event funding is extended beyond March 31, 2006, the approval and selection process of applicants (RMOs) will be reviewed to ensure a clear alignment exists between the shorter term goals and objectives of the RMOs with the longer term objectives of the FNS program.
Timeline: Fall/Winter 2005-2006
top of pageRecommendation F: RMOs should continue to be the primary method of program delivery.
Although the First Nations SchoolNet program was developed, implemented, and managed directly from Industry Canada prior to the establishment of the RMOs, the program has become appreciably more efficient and successful as a result of RMO program delivery. In conducting the Mid-Term Evaluation of the First Nations SchoolNet program, the Consultant found that respondents from both the Industry Canada regional and national office were fully supportive of the RMO program delivery method, as well as the progress achieved to date. The success of the RMOs was attributed to each organization's understanding of the local realities impacting the community as well as the school, and the RMOs ability to maximize program success through a flexible approach to program delivery. The credibility of the RMOs along with the established relationships with community members provided a forum to engage the program in an efficient and meaningful manner, thereby encouraging the success of the FNS program. In addition, the Consultant recommends the continuation of frequent communication between the RMOs and Industry Canada at the national and regional level to discuss best practices and share knowledge.
Management Response:
Agreed. Management supports this recommendation.
In the event program authority is granted beyond March 31, 2006, program design will take into consideration the RMO delivery model.
Management also supports the recommendation of continued frequent communication between RMOs and Industry Canada at the national and regional level to discuss best practices and share knowledge. As referenced earlier, FNS will take a more active role in facilitating this knowledge transfer through various means: sharing leading practices at the semi-annual national FNS face-to-face stakeholder meetings; holding monthly technical meetings with RMOs and regions; undertaking monthly program operational meetings with RMOs and regions whereby guest speakers on relevant topics of interest are also invited to participate; conducting monthly regional meetings with the regional office and the RMO to discuss issues as they pertain to contribution agreement implementation; and beginning Fall 2005, implementing quarterly governance meetings between the regional offices and the national office.
Timeline: Fall/Winter 2005-2006
- Date modified:
