Mid-Term Evaluation — First Nations SchoolNet Program

Section 7: Synthesis of Findings and Consultant's Recommendations

The following section outlines the overall conclusions of the Mid-Term Evaluation of the First Nations SchoolNet program, as well as recommendations for improvement based on lessons learned and best practices offered by the respondents. This section is structured by the questions that guided the evaluation.

7.1 Synthesis of Findings

Is the First Nations SchoolNet program relevant and needed?

First Nations schools have typically lagged behind other Canadian schools with ICT due to limited resources and geographical barriers. The need for programs and initiatives designed to assist First Nations learners in their education was highlighted in the 2004 Auditor General's Report which stated that "a significant education gap exists between First Nations people living on reserves and the Canadian population as a whole and that the time estimated to close this gap has increased slightly, from about 27 to 28 years". FNS has significantly improved the connectivity of First Nations schools and the program continues to be relevant and needed as further demonstrated by the discrepancy in computer student ratios and connectivity speed between First Nations and non First Nations schools in Canada.

One of the most revealing statistics emphasizing the need for the FNS program is the ratio of computers to students. Data from RMOs demonstrates that the computer to student ratio is now between 8:1 and 5:1 in all of the regions served by the FNS program. Anecdotal evidence from the key informant interviews suggested that this ratio was considerably worse prior to the implementation of the program and that there is still opportunity for improvement.

While the FNS program has achieved connectivity in all of the First Nations schools served by the program, the schools locations in remote areas have created barriers to accessing high speed connectivity. Chart 7-1 demonstrates that the First Nations schools have typically slower connectivity than other Canadian schools. The majority of First Nations schools are still dependent on connectivity delivered through a satellite connection, which is typically slower than a broadband connection (i.e. T-1, DSL, ISDN) and may limit potential for educational opportunities.

The relevancy of the program is also emphasized by findings from the key informant interviews and case studies conducted through this evaluation. All RMO coordinators (100%) rated the program as "extremely important" in fostering connectedness amongst First Nations schools. Key informants also stated that to date no other program has been able to provide ICT in First Nations communities to the degree that FNS has. Further, key informants noted that the increase in connectivity has led to greater acceptance and use of FNS provided technology.

To further support comments by key informants, case studies showed that First Nations schools generally lack sufficient library resources and thus have greater dependence on the Internet to provide educational information to students.

top of page

To what extent have the goals and objectives of the First Nations SchoolNet program been achieved?

Respondents felt that significant progress has been made towards the overall goals and objectives of the program, however the goals were felt to be long-term outcomes that require ongoing work to achieve. In addition, it was mentioned that while RMOs can provide the necessary infrastructure, other goals of the program were thought to be largely dependent on the awareness and acceptance of the program by the First Nations schools for the utilization of the resources in learning opportunities. Several RMO staff members suggested that the goals should be more specific to allow for more accurate assessments of the progress of the program and the degree to which it has met the overall objectives. This may be due to the longer-term vision of Industry Canada's goals and objectives of the program compared with annual targets for each RMO set forth in their respective Contribution Agreement. In completing the research activities associated with this evaluation, the Consultant found that Industry Canada and RMO staff either agreed or strongly agreed that the FNS program was implemented as intended, has produced positive impacts, and has met some significant short-term outcomes. Overall, respondents noted that the program is headed in the right direction, and with continued support and increased funding to build local capacity, the objectives will be reached in the next several years.

Information collected from the case studies echoed the results of the key informant interviews, however one RMO cautioned that the program should address certain needs of the student population before advancing new technology and applications, specifically e-learning. The rationale provided for this view was that a significant proportion of students in First Nations schools are affected by learning disabilities and sometimes personal issues that have resulted in drop-out rates higher than the Canadian average. It was felt that placing higher demands on students would be counterproductive to the goals of the program. It was also stated that with only two years of involvement in delivering the FNS program to the schools, connectivity challenges remain, and therefore the priorities of the program need to be clear and allow for sufficient time to pass to reach the goals and objectives of the program.

top of page

To what extent has FNS had the intended impacts among schools, teachers, and learners?

First Nations SchoolNet has impacted schools, teachers and learners as intended through providing connectivity, hardware, software, and technical support to First Nations schools across the country. Specifically, key informants commented that the technology provided through FNS has:

• afforded many students in First Nations schools new opportunities and an avenue to acquire skills necessary to become competitive in the knowledge-based economy.
• provided students living in First Nations communities the opportunity to advance their knowledge through the Internet and to communicate with people from outside their community allowing for sharing of knowledge and culture.
• served as a mechanism to provide an informational resource to students, thereby supporting them in their learning and compensating for a lack of resourceful libraries, typical of many First Nations communities.

Through the key informant interviews, especially with RMO staff, it was noted that determining the applications related to FNS in the schools was difficult. Several RMO coordinators reported the use of the technology in the schools being used primarily as an informational tool to assist with classroom study, although in some schools students were using the provided resources for multimedia applications such as creating videos and personal web pages. Perhaps the most direct educational application of the technology was the emergence of Internet high schools operating in several regions. Although all of the respondents recognized the value and potential of e-learning opportunities, many schools cannot actualize this potential due to a lack of local skills among students and teachers, as well as insufficient connectivity. The school administrators were appreciative of the resources provided by the program and are currently provided the flexibility to integrate the resources into the classroom at their discretion.

FNS has also impacted schools by providing a way in which the schools can overcome major barriers to the acquisition of ICT. As demonstrated in Chart 7-2, a primary challenge for the First Nations schools in regards to ICT is acquiring the funding to obtain the technology.

Through the FNS program, First Nations schools are provided with or subsidized to obtain computer hardware, software, and connectivity.

While the First Nations SchoolNet program has positively impacted schools, teachers and learners with the introduction of technology and connectivity, this has not been without challenges. Case studies revealed that a major problem with the introduction of new technology, namely connectivity, is that once the communities receive connectivity in their region, the risk of contracting computer viruses and other malware becomes a significant problem. From key informant interviews and the case studies it became clear that a lack of knowledge of the threats to Internet security constantly affects the performance of the computers in the schools. Further, findings from key informant interviews also suggest that a lack of knowledge of general computer maintenance skills is evident in the schools. This indicated a need for training and potential capacity building at the community level. This lack of sufficient training opportunities was also found in the recent Statistics Canada study with respect to First Nations schools.

In response to these challenges, five of the six RMOs have developed a Youth Employment program that provides valuable technical training to local youth and students. The program educates the students on maintaining the function of the computers through anti-virus software and other technical related skills. Students are also versed in multimedia applications including web page design and video streaming. While not the intention of the program, it was found that the skills provided to youth have enabled them to acquire employment in some cases. The youth program is thought to represent a practical solution to building relevant skills and fostering an ICT culture in First Nations schools and communities.

Both of the case study sites had developed and implemented youth training programs, which are described in greater detail in the case study reports.

top of page

Does the current delivery model represent the most effective and efficient way to deliver the program?

The use of RMOs appears to be an effective way to deliver the program. Overall, key informants agreed that use of RMOs was efficient and effective, even those who had been involved with the program prior to the implementation of the RMO delivery system. In addition, the Ontario RMO had been involved with the FNS program prior to the implementation of the six RMOs, and during the case study conducted by the Consultant RMO staff stated that the changes to program delivery have led directly to greater results in a more efficient manner and timely manner.

The Consultant investigated whether the RMOs were implemented as intended, and whether program delivery had improved as compared to the central management of the FNS through Industry Canada. The Industry Canada National Staff, IC Regional Staff, RMO staff, and school administrators altogether endorsed the current management structure and provided insight into reasons governing the success of the six organizations in delivering the program. The success of the RMOs in delivering the FNS program was felt to be due to the following factors:

• RMOs have existing relationships with the schools and communities in their region leading to a more personal approach;
• The ICT needs of the schools were more carefully analyzed and met in a more responsive manner through the RMOs;
• RMOs operate with less bureaucracy than the federal government, therefore allowing greater flexibility and responsiveness;
• Technical support to the schools was found to be provided better due to the RMO's working knowledge of each school's capabilities and staff involved in assisting the FNS program; and
• Training opportunities were regarded as a successful element of program delivery through the active engagement of the RMOs in educating the students and teaching staff on related skills to maintain and apply the ICT resources.

Key informants agreed that RMOs were successfully satisfying both Industry Canada and First Nations communities by identifying the needs and intentions of both parties. It was believed that RMOs successfully address the different barriers facing communities and regions served by the program through independent annual proposals and Contribution Agreements of each RMO, that assist by accounting for the challenges impacting success.

top of page

What are the lessons learned and best practices with respect to program design, delivery and implementation?

Program structure and reporting were areas in which the program could improve or learn from current operations. RMO coordinators felt that the paperwork of monthly reporting and financial claims was a burden that impacted their ability to deliver the program. To help with the reporting required it was suggested that a standardized report structure be developed, along with a listing of required data to facilitate the generation of both monthly and annual reports. Currently, such a template exists in the Annex of each Contribution Agreement signed with the RMOs. The lack of awareness of this document among RMOs signals a need for improved communication.

The life span of the FNS program was commonly cited as a challenge to effective program delivery to long-term planning and the sustainability of the program overall. RMO staff indicated that they had difficulty creating long range ICT plans with a school when funding is uncertain (on yearly renewal). They felt it would be more efficient to provide services on a multi-year plan and that the multi-year plan would assist in building effective and extended relationships with First Nations schools and communities to support program delivery.

Best practices identified through the key informant interviews included:

• Locating individuals in each of the communities to educate with technical support and other computer-related issues. Teaching staff and students were also afforded training opportunities. However by training community members, it was found that the technical skills would endure at the community level and counter the high turnover rates among teaching staff.
• Relationships with the schools were seen as one of the most essential elements of program delivery. Communication efforts through e-mail were unsuccessful in many cases, therefore faxing and telephone follow-ups were cited as a more effective way to reach the schools. The value of on-site visits with the schools was recognized but the remote locations of the schools and the current budget for the RMOs were seen as preventing more visits. One solution presented by several key informants was the use of videoconferencing to communicate with the schools, assess their ICT needs, and to determine how the schools were integrating the technology with learning opportunities for the students.

Several best practices recognized through the case studies include the following initiatives:

• The Youth Employment program provides a learning opportunity for students while building local capacity. Through this program, students are educated on technical support that includes the development of multimedia applications. In many cases, the trained youth have gained employment from their community as a result of their skills.
• The Grade 8 (G8) supplementary program operating in Ontario represents a method of facilitating the transition from grade 8 to high school by reducing any knowledge disparity. The program was designed primarily for First Nations students to serve as an additional educational resource.
• The Keewaytinook Internet High School (KiHS) represents an innovative response to a community need linking students from various communities to educational opportunities and credit. The Internet high school allows students the opportunity to remain in their communities while pursuing their education.

top of page

7.2 Consultant's Recommendations

1. Increased communication between the schools and RMOs is required to support program delivery and partnership development.

   Currently the schools and the RMOs experience barriers to communication including insufficient staff at schools to facilitate program delivery, partly due to high staff turnover, school staff unfamiliarity with RMO representatives, lack of regular e-mail use and general unresponsiveness of schools to communication efforts of the RMOs.

   The Consultant believes that by overcoming these barriers and enhancing communication, the program will better support local skill and capacity building. Further, schools may gain a greater appreciation of the program and partners with more frequent communication resulting in a better working relationship during the tenure of this program. More communication with the schools will also increase the understanding of the program goals and objectives as well as the potential applications of the FNS resources provided to First Nations schools.

   Since on-site visits require a substantial commitment of time and money due to the remoteness of the schools, videoconferencing may represent a plausible alternative for increased communication with the schools, where such technology exists. Other avenues to be considered to support communication include training seminars delivered on-line, videoconferences, or in-person workshops similar to those offered in the Youth Employment program.

2. The identification of how schools are using the resources provided by the First Nations SchoolNet program needs to be improved.

   As the target audience of the program, First Nations schools receive connectivity and other ICT resources from the FNS program to assist students with their learning. Resources typically provided by the FNS program include connectivity upgrades, computer hardware and software, related equipment (including digital cameras and video cameras), and technical support through the Help Desk.

   Although schools differ in their ability to integrate available technology to enhance learning opportunities, it was evident through school administrator interviews and case studies that several schools are leaders in resource application. Divergent skill levels among schools highlights an opportunity for knowledge transfer between schools. Dissemination of lessons learned and best practices among the schools could be directed by the RMOs, facilitated by the existing channels of communication, namely the national meetings that occur with FNS stakeholders.

   Since each RMO signs a contract with the First Nations schools on an annual basis, the Consultant recommends that Industry Canada include a requirement for schools to report on how both students and teachers are using the FNS resources. This brief report would outline the applications of the resources sent to the school used for educational purposes and would also share challenges encountered by the school. The report would replace the annual surveys that have been typically affected by poor response rates. By having the schools detail the uses of the FNS resources, the RMOs would gain a better understanding of usage patterns, potential need for other technology, and any challenges impacting the school's ability to utilize the resources to their potential. Having the onus of reporting falling with the schools would also serve to assist the RMOs by reducing the administrative costs of trying to contact unresponsive schools. Ideally, the reports submitted by the schools to the RMOs would then serve to identify best practices, which could then be shared with other RMOs and hence other First Nations schools served by the program.

3. Partnerships with other departments and organizations should be established to enhance the efficiency and sustainability of the program.

   The goals and objectives of First Nations SchoolNet are closely aligned to the mandate of other programs and represents opportunities for partnerships. Thus, respondents noted that development of partnerships might help to facilitate increased connectivity as well as increasing learning opportunities in First Nations schools. Programs noted by respondents include the following Industry Canada programs: Broadband for Rural and Northern Development (BRAND), Computers for Schools, and the Community Access Program (CAP). Other potential partnering opportunities include building or strengthening relationships with Indian and Northern Affairs Canada (INAC), Health Canada, Atlantic Canada Opportunities Agency (ACOA), Western Economic Diversification Canada (WD), and educational partners to facilitate and expand on the services provided by the FNS program while leveraging funds and building sustainability. It was also felt that partnering could help to reduce the "silo effect" which limits the overall success of programs and represents an opportunity to stream-line resources to the First Nations communities within a broader mandate.

   The significance of partnerships has been demonstrated in the Ontario region with the Federal Economic Development Initiative for Northern Ontario (FedNor), the only economic agency involved with the FNS program. In total, FedNor has invested over ten million dollars into building and supporting connectivity and infrastructure in Northwestern Ontario. The investment by FedNor, coordinated with FNS resources, has led to other partnerships including the involvement of Health Canada who is currently purchasing connectivity from the Ontario RMO to deliver Telehealth to 24 First Nations communities.

   In addition to FedNor's involvement with the program, five RMOs have implemented a Youth Employment program designed to provide youth with computer skills, thereby helping youth build local capacity while gaining employable skills. The funding for the youth program resulted in part from an investment from Human Resources Skills Development Canada (HRSDC). The partnerships with other programs as well as other organizations is mutually rewarding and helps to create a sustainable approach to providing connectivity and ICT resources to First Nations communities in Canada.

4. The annual Contribution Agreement could be extended to secure funding to the RMOs for a longer time period.

   A longer term funding commitment from Industry Canada would allow for better RMO planning and would facilitate the development of new partnerships or strengthen existing partnerships. Findings suggest that there are several drawbacks of the current annual funding agreement. Challenges include difficultly in leveraging funds from partners, demonstrating commitment to the schools through a sustainable approach, and forecasting budgets at both the RMO and the school level for the next several years. While annual funding agreements allow monitoring of the RMO progress and added flexibility to individualize their services, many RMO coordinators felt that the challenges far outweighed the benefits of this funding arrangement with special mention of reporting and paperwork burdens. It was felt that secure funding over several years would allow the RMOs to plan additional training opportunities for community members and youth, as well as, develop long-range hardware and connectivity strategies for First Nations schools. Additionally, RMO staff noted that a reduction of, or assistance with both progress and financial reporting would be appreciated. This assistance may be partly accomplished through greater promotion of existing templates, an extension of the annual contracts, or a reduction in reporting duties.

   Extended funding arrangement were also felt to facilitate stronger partnerships with the schools, and provide a better opportunity to establish funding relationships with other government sponsored programs (CAP, BRAND), as well as initiatives from other organizations (Health Canada, INAC, WD, ACOA). Finally, with respect to the Contribution Agreement, respondents felt that it should be revised to account for the discrepancy between fiscal years of the RMOs (fiscal year) and the schools (academic year).

5. Annual short-term goals and objectives of the RMOs should be aligned with the longer-term objectives of the FNS program.

   While the overall program goals and objectives have remained the same, each RMO prepares a list of goals and activities to be completed for each annual Contribution Agreement, resulting in the short-term goals of the RMOs sometimes being noticeably different from overall program goals that have a longer-term focus. Given this discrepancy, the Consultant suggests that more work should be done to align the goals of the RMOs with the program. By working to better articulate the short-term and long-term goals of the program a more accurate assessments of the progress of the program could be obtained.

6. RMOs should continue to be the primary method of program delivery.

   Although the First Nations SchoolNet program was developed, implemented, and managed directly from Industry Canada prior to the establishment of the RMOs, the program has become appreciably more efficient and successful as a result of RMO program delivery. In conducting the Mid-Term Evaluation of the First Nations SchoolNet program, the Consultant found that respondents from both the Industry Canada regional and national office were fully supportive of the RMO program delivery method, as well as the progress achieved to date. The success of the RMOs was attributed to each organization's understanding of the local realities impacting the community as well as the school, and the RMOs ability to maximize program success through a flexible approach to program delivery. The credibility of the RMOs along with the established relationships with community members provided a forum to engage the program in an efficient and meaningful manner, thereby encouraging the success of the FNS program. In addition, the Consultant recommends the continuation of frequent communication between the RMOs and Industry Canada at the national and regional level to discuss best practices and share knowledge.