Mid-Term Evaluation—First Nations SchoolNet Program

Final Report

Audit and Evaluation Branch

March 2005

Tabled and approved by DAEC on December 8, 2005

Table of Contents

• Executive Summary
• Section 1: Background and Project Rationale
  • 1.1 Background
  • 1.2 Evaluation Objectives and Rationale
• Section 2: Methodology
  • 2.1 Phase One—Evaluation Design
    • 2.1.1 Review of Background Material
    • 2.1.2 Development and Pre-Testing of Interview Guides
  • 2.2 Phase Two—Evaluation Implementation
    • 2.2.1 Key Informant Interviews
    • 2.2.2 Case Studies
  • 2.3 Limitations of the Evaluation
• Section 3: Project Findings—Industry Canada Staff
  • 3.1 Program Participation
  • 3.2 Program Management
    • 3.2.1 Communication and Roles/Responsibilities
    • 3.2.2 Monitoring/Reporting on the Program
    • 3.2.3 Resource Availability and Levels
    • 3.2.4 Challenges to Program Management
  • 3.3 Program Delivery
    • 3.3.1 Challenges to Program Delivery
    • 3.3.2 Opportunities for Program Delivery
  • 3.4 Program Impact
• Section 4: Project Findings—RMO Coordinators
  • 4.1 Program Participation
  • 4.2 Program Management
  • 4.3 Program Delivery
    • 4.3.1 Services Delivered
    • 4.3.2 Degree to Which Program Implemented as Intended
    • 4.3.3 Barriers to Program Implementation
    • 4.3.4 Emerging Connectivity and Innovation Themes
  • 4.4 Program Impact and Challenges
• Section 5: Project Findings—School Administrators
  • 5.1 Program Participation
  • 5.2 Program Delivery
  • 5.3 Program Impact
• Section 6: Project Findings—Overview of the Case Studies
• Section 7: Synthesis of Findings and Consultant's Recommendations
  • 7.1 Synthesis of Findings
  • 7.2 Consultant's Recommendations
• Management Response or
  Version PDF (28 Ko, 9 pages)
• Appendix A: Steering Committee Members
• Appendix B: List of Materials Reviewed
• Appendix C: List of Respondents Interviewed
• Appendix D: Interview Guides
• Appendix E: Case Study—Keewatin Tribal Council (Manitoba RMO)
• Appendix F: Case Study—Keewaytinook Okimakanak (Ontario RMO)

Acronym List, Mid-Term Evaluation of the First Nations SchoolNet Program

Executive Summary

The Mid-Term Evaluation of the First Nations SchoolNet (FNS) program was undertaken to provide Industry Canada with information pertaining to the management, delivery, progress and challenges of the program. The research activities included 55 key informant interviews comprised of representatives from the following stakeholder groups: Industry Canada national and regional staff, Regional Management Organizations (RMOs) service providers and program staff, school administrators, and other stakeholders involved with the FNS program.

The following provides a brief summary of the findings of the key informant interviews, case studies, and document reviews conducted by R.A. Malatest & Associates as part of the formative evaluation. These findings are presented in more detail in this report.

• The FNS program continues to be relevant and necessary. The program has made significant progress in delivering connectivity and ICT resources to First Nations schools. The program has also helped to compensate for a lack of informational resources in the schools while providing a forum for communication with individuals from outside of the community. Since the FNS program is the only federal program designed to provide connectivity and ICT resources to First Nations schools, the FNS program was regarded as essential in helping First Nations youth acquire computer related skills to be competitive with students in other regions in Canada.
• Substantial progress has been made towards reaching the goals and objectives of the FNS program. Respondents felt that significant progress towards meeting the overall goals and objectives has been made although many felt that the goals represent long-term outcomes requiring ongoing work from the RMOs as well as acceptance of the program and utilization of the resources by the First Nations schools.
• FNS has had the intended impact among schools, teachers, and learners. The technology provided through the program has afforded many students in First Nations schools new opportunities and an avenue to acquire necessary computer skills to become competitive in the knowledge-based economy. Students were reported as having the ability to comprehend the technology rapidly with some students progressing to skills that include web design and the production of multimedia applications.
• RMOs continue to be an efficient and effective method of program delivery. A great degree of the success enjoyed by the program can be attributed to the management structure that includes national and regional representatives of Industry Canada as well as non-profit First Nations and Aboriginal organizations. The use of RMOs to deliver the program was endorsed as an improvement over the previous central management of the program from Industry Canada.
• Building local capacity was regarded as one of the most important factors governing the success of the FNS program. Technical support is available through each of the RMO's Help Desk, although support at the local level was felt to be necessary to maintain the operation of the ICT resources and to create enhanced skills among students, teachers, and members of the community. Training opportunities identified by RMO staff include workshops during on-site visits to the schools, conferences, and through the Youth Employment program.
• Case studies revealed several innovative programs have been created and supported through the FNS program. The Keewaytinook Internet High School (KiHS) links students from various communities to educational opportunities thereby allowing students the opportunity to stay in their community while gaining educational credits. The Grade Eight (G8) supplementary program is an educational resource designed to help students enter higher levels of education with greater ease. Both programs were developed and supported with assistance from the FNS program.

Through undertaking the research activities associated with the Mid-Term Evaluation of the First Nations SchoolNet program, the Consultant was able to identify the following recommendations:

• Increased communication between the schools and RMOs is required to support program delivery and partnership development.
• The identification of how schools are using the resources provided by the First Nations SchoolNet program needs to be improved.
• Partnerships with other departments and organizations should be established to enhance the efficiency and sustainability of the program.
• The annual Contribution Agreement could be extended to secure funding to the RMOs for a longer time period.
• Annual short-term goals and objectives of the RMOs should be aligned with the longer-term objectives of the FNS program.
• RMOs should continue to be the primary method of program delivery.

The recommendations are described in greater detail in Section 7.2 of this report.

Section 1: Background and Project Rationale

1.1 Background

Launched in 1996, the First Nations SchoolNet (FNS) program is one of Industry Canada's SchoolNet initiatives. The First Nations SchoolNet (FNS) program was designed to connect all First Nations schools under federal jurisdiction to the Internet. The program was established as a key element in the agenda of "Building a More Innovative Economy" with a mission "to work with Canadian learning partners to increase access to and integration of Information and Communications Technologies (ICT) into the learning environment in order to develop an ICT-skilled population, capable of participating in the Knowledge Economy."

The program's first mandate was to connect and maintain all First Nations schools under federal jurisdiction to the Internet by providing them with telecommunications infrastructure and promoting ICT. This target was achieved in March 2001. The second mandate of the program, as included in the 1999 Speech of the Throne and reinforced in the 2001 Speech of the Throne, was to provide high speed Internet access to First Nations schools.

The Government of Canada reiterated its commitment to the First Nations SchoolNet program in a budget speech on March 23, 2004 by allocating $15 million per year for 2004–2005 and 2005–2006 to the program to improve the connectivity and technical capacities of First Nations schools by providing them with high speed Internet, computers, technical support and ICT skills development workshops in order to bridge the digital divide in First Nations schools located in remote and rural communities. The program will continue to assist First Nations schools in their efforts to connect with e-learning resources in order to give First Nations learners on reserve the opportunity to acquire the skills and knowledge they need to participate in the growing knowledge-based economy.

From its inception, FNS dealt with each First Nation school separately through a central management structure within Industry Canada. However, this method of program delivery shifted to Contribution Agreements with six locally based not-for-profit organizations to create Regional Management Organizations (RMOs) in December 2002, representing a dramatically different method of program delivery. Each RMO, selected through a rigorous regionalized selection process involving representatives from national and/or regional offices of the Industry Canada—Information Highway Applications Branch (IHAB), Industry Canada -Aboriginal Business Canada (ABC), Human Resources Skills Development Canada (HRSDC), Atlantic Canada Opportunities Agency (ACOA), Canada Economic Development for Quebec Regions (CED-Q), and Indian and Northern Affairs Canada (INAC). The six RMOs were selected to utilize their existing knowledge and experience of First Nations connectivity issues and their ability to facilitate the delivery of the First Nations SchoolNet program regionally. There are also responsible for aggregating demand, delivering the program regionally, and putting in place any local service contracts required to meet the connectivity-related requirements of the schools. To facilitate the change in program delivery, funding was increased approximately seven times the funding amount of the previous year.

All of the RMOs work toward specific activities outlined in their annual Contribution Agreement with Industry Canada, and while the RMOs perform similar activities, as organizations they differ in their history, main clientele, mandate, and the duration of their relationship to Industry Canada's FNS program. For example, five of the organizations are First Nations based with the remaining organization focused towards those of Aboriginal descent. More specifically, there are two RMOs that are Tribal Councils, two regional Education Councils, one territorial Chiefs organization, and an Aboriginal community service agency. Further, RMOs varied in terms of their previous experience working with delivering connectivity and other ICT to the schools in their regions. This previous experience resulted in some RMOs progressing at a much more rapid pace than others. For example, the case studies conducted by the Consultant found that the Ontario RMO has been involved with FNS program-related activities since 1996 while the Manitoba RMO became involved in 2002. To account for these and other regional variations, annual contracts are signed with each RMO that outline the activities to be completed with consideration to specific regional challenges. Although the RMO staff felt the level of reporting was cumbersome, the process allows for flexibility and recognition of barriers to program delivery.

The current structure of the First Nations SchoolNet program is detailed in the following chart.

Figure 1.1 Background - First Nations SchoolNet National Office Org chart

 [Description of Figure 1.1]

1.2 Evaluation Objectives and Rationale

This evaluation addressed a number of evaluation issue areas and research questions including lessons learned identified in the Results-based Management and Accountability Framework (RMAF). The evaluation focused on current management issues related to program implementation using RMOs in relation to the following objectives of the program:

• To promote innovation in First Nations schools by providing them with connectivity infrastructure, related support services, and a new pedagogical approach addressed to teachers;
• To support the production of First Nations—based relevant online educational resources;
• To contribute to the establishment of an ICT culture in First Nations schools through investments in the ICT skills training of First Nations learners and teaching staff;
• To promote the use and integration of ICT into First Nations classrooms for learning purposes; and
• To help First Nations learners acquire the skills and knowledge to be able to better access the job market.

The evaluation sought to understand whether adjustments are necessary to the current method of program implementation and whether progress toward the achievement of the outcomes is occurring. Four primary issue areas for evaluation were considered: program rationale, program design and delivery, program success and lessons learned.

Section 2: Methodology

This section summarizes the research activities completed as part of the First Nations SchoolNet Mid-Term Evaluation.

2.1 Phase One—Evaluation Design

The design phase of the Mid-Term Evaluation encompassed the following activities:

• Review of Background Material Program Files, Policy Documents and Other Data Sources which included all relevant materials and sources provided by the project authority and other stakeholders; and
• Development of Interview Guide/Protocols and Draft Surveys in consultation with the Project Authority.

2.1.1 Review of Background Material

The Consultant began the evaluation with a review of background material on the SchoolNet program provided by Industry Canada. A comprehensive list of the materials reviewed can be found in Appendix C.

2.1.2 Development and Pre-Testing of Interview Guides

The Consultant developed four research instruments for the Mid-Term Evaluation. These included three key stakeholder interview guides and a case study interview guide.

The key informant interview guides were designed by the Consultant to capture information from RMO program staff, IC national and regional staff, and school administrators. Each key informant interview guide was developed with questions that addressed specific stakeholder's involvement and perspective. The key informant interview guides were submitted to the Steering Committee for approval prior to the commencement of the interviews. The final version of each survey was pre-tested. The field test gauged the relevance, clarity, and flow of the questions. The interview guides were finalized without any changes and the Consultant translated the IC national/regional staff questionnaire into French.

In addition to the key informant interview guides, the Consultant developed an introductory letter that introduced the evaluation and the issues covered during the interviews and which also provided contact information for an Industry Canada representative and the R.A. Malatest & Associates Ltd. project manager to field any questions.

The case study interview guide was developed with additional questions pertaining to the program. The case study interview guide shared a number of common questions with the key informant questionnaires.

2.2 Phase Two—Evaluation Implementation

2.2.1 Key Informant Interviews

Table 2-1 outlines the key informant interviews undertaken by the Consultant. A detailed list of respondents interviewed can be found in Appendix B of this report.

School administrators were e-mailed a copy of the interview guide, along with an introductory letter and Frequently Asked Questions (FAQ) document outlining the purpose of the study, information about the confidentiality of the research, contact information including the Consultant's toll-free number, and instructions on completing the questionnaire. All school administrators were contacted via telephone to complete the interviews. The case study interviews were conducted in-person and the rest of the sample completed the surveys via telephone.

2.2.2 Case Studies

The Consultant undertook case studies at the RMO office in Manitoba and Ontario. For each case study the Consultant completed and reviewed:

• Background documentation specific to the site;
• Available administrative data collected at the site;
• One focus group; and
• Key informant interviews with representatives of the site and other associated stakeholders.

The results of the case studies have been appended to this final report as Appendix E and Appendix F. Findings from the case studies have been incorporated into this report and are presented in the Consultant's synthesis of findings and recommendations in Section 7.

2.3 Limitations of the Evaluation

It should be noted that this study was intended to serve as a general assessment of the operation of the First Nations SchoolNet program. The study was designed to provide Industry Canada with qualitative oriented information regarding the program participation, management, delivery, basic impact and challenges, as well as best practices and lessons learned. The findings of this evaluation are aimed at assisting the program rather than demonstrating the value, return-on-investment or the ability to meet long-term outcomes as those are more related to an evaluation of a summative nature.

In this context, the number of survey responses suggests that the information gathered should be regarded as preliminary. In total, 55 individuals were contacted to participate in the study.

It should be noted that many of the school administrators indicated that they had only limited contact or familiarity with the First Nations SchoolNet program. During survey administration, the Consultant found that the knowledge and familiarity of the First Nations SchoolNet program varied considerably within this group as did their overall experience and tenure at their school. As such, the ability of the respondents to complete the questionnaire also varied and this may be an interceding variable or influence.

Section 3: Project Findings—Industry Canada Staff

The Consultant completed eleven key informant interviews with Industry Canada National and Regional staff including the Program Director, Program Manager, Project Officers as well as Regional Office (RO) staff. The responses of the key informants are detailed in the following sections in aggregate form.

3.1 Program Participation

The FNS program involves direct management and support from Industry Canada as well as representatives in each of the six defined regions. The Industry Canada Regional Office (RO) staff typically act as a liaison between the Industry Canada national office and the RMO staff in the delivery of the program. This role plays a vital link in the communication and management of the program. The role is supported through participation in monthly conference calls and videoconferences. In most cases, the RO also assists the RMO in their region with reporting duties, specifically preparation with the RMO's proposal and resulting Contribution Agreement with Industry Canada national office. The RO is also in a position to oversee and assist with program delivery and related activities of the RMO.

Prior to the implementation of RMOs, the FNS program was centralized and managed exclusively by Industry Canada who worked directly with First Nations schools. Of the respondents that had experience with the FNS program both before and after the implementation of RMOs, all stated that the establishment of the RMOs has afforded a better program delivery approach and has surpassed the results achieved when the program was centrally managed. The benefits of using RMOs were stated as providing a more personal approach by having not-for-profit organizations deliver the program directly to the residents of the community through established relationships with community members. Of the six organizations, five are First Nations oriented with the remaining organization focused to an Aboriginal clientele. These relationships were thought to allow for a more responsive and tailored approach to the 567 First Nations schools served by the program.

3.2 Program Management

3.2.1 Communication and Roles/Responsibilities

To assess the degree of communication between the levels of management of the FNS program, respondents were asked questions pertaining to communication as well as the roles and responsibilities of the RMOs, Industry Canada Regional Offices, and the Industry Canada national office. There was consensus that sufficient processes were in place to facilitate communication among all the levels of staff involved in the program. It was felt that through the regular videoconferences, teleconferences and e-mail correspondence, the lines of communication were clear and people were responsive to concerns. The amount of communication between the RO and the RMO tended to vary across the regions, but in most cases this relationship proved to be valuable. Overall, the roles and responsibilities of the RMOs, Industry Canada Regional Offices, and the Industry Canada national office were found to be clearly outlined through the Contribution Agreement and all of those interviewed agreed that duplication of roles or responsibilities was non-existent.

3.2.2 Monitoring/Reporting on the Program

The results of RMO program delivery of First Nations SchoolNet were delivered through a number of different methods. The monthly reports on activities and financial claims were noted as important for providing useful information on the progress of each RMO. Other respondents noted that research has been conducted through an RBAF and RMAF, the current mid-term evaluation, and a recent audit which have all helped to detail the activities undertaken by each RMO. It was noted that evaluating the progress made by the RMOs in helping First Nations schools is often difficult due to the challenge of knowing how the services provided by the program have been utilized by each school. Accurate assessments are also influenced by the high turnover rates among teachers in First Nations schools. To identify and collect performance measurements and the impact of the program, a more complicated and labour intensive process would need to be performed through either assigning additional resources to allow for more on-site visits by IC RO staff or by having more videoconferencing where available.

3.2.3 Resource Availability and Levels

For the First Nations SchoolNet program to be delivered as intended, the level of resources would need to match the objectives of the program. Respondents were questioned about the level of staff, infrastructure, and funding associated with the program and whether they would modify the current resources in amount, distribution, or allocation. It was found that the majority of respondents noted that the Industry Canada national office is currently short-staffed by two positions, resulting in more tasks for the current staff. Generally, it was felt that RMOs and ROs are properly staffed and although there is a lot for each office to accomplish, they are managing the best they can to stay within their budget. The remote locations of the First Nations schools limited the amount of direct contact due to the expenses associated with traveling to the communities. Several respondents noted that with increased resources, RMO staff would be in the position to train more teachers with technology associated with FNS and build on existing relationships with community members. Overall, respondents felt the program requires secure funding for the next several years to ensure broadband connectivity to all First Nations schools and to reach the objectives of the program (described in Section 1.2).

Further to general inquiries of the resources of the FNS program, respondents were asked whether the current resources were sufficient to meet the demand in the First Nations community and whether the resources are keeping pace with what students require. There was considerable variance in the responses with several respondents stating that current connectivity requires additional resources to meet demand in the First Nations communities and to offset increasing connectivity costs. Other responses included the need for additional support for training youth as well as language and culture, curriculum development, and e-learning. One suggestion provided noted that to increase the resources for the program, it would be beneficial to establish partnerships with other programs and government departments to obtain additional resources.

3.2.4 Challenges to Program Management

Overall, respondents were positive about the mandate and implementation of the FNS program, although a number of challenges persist. A common issue that was found to be difficult was identifying how First Nations schools are using FNS services and whether students are developing skills. Due to the remoteness of the schools, travel is difficult and expensive for RMO staff indicating a need for additional resources. Another issue pertaining to RMO program delivery was the recognition of the need to simplify the reporting process as it currently takes a lot of resources to provide the monthly and annual reports. With a simplified reporting process with standardized reports, it was stated that resources would be made available for other RMO tasks. Other challenges to program delivery mentioned include the shortage of staff at the national level, and the lack of a national database of the First Nations schools served that would provide an inventory of what they have received through the program.

In addition to the aforementioned challenges, most of the respondents cited a lack of a long-term focus of the FNS program. It was stated that a long-term focus and funding security would allow more time to deliver the program and reach the intended goals. With short timeframes, it was felt that a proper analysis of the impact of the program would be difficult. Further, the need for partnerships to ensure sustainability was noted. Through establishing partnerships, many felt that Industry Canada would be able to utilize other similar programs to reach the objectives and benefit the clients. Specifically, these partnerships could help with funding levels and possibly subsidize the increased costs of connectivity.

3.3 Program Delivery

In December of 2002, Industry Canada selected six not-for-profit organizations to become RMOs to work directly with First Nations schools in their respective region. Respondents were asked a number of questions pertaining to the implementation and effectiveness of using RMOs to deliver the First Nations SchoolNet program. The general consensus was that RMOs represent a positive change in program delivery from being centrally managed by Industry Canada. Respondents felt that through RMOs, the needs of the First Nations schools and communities were better understood, resulting in program delivery that was tailored and more responsive. It was stated that due to the success of the RMOs delivering the FNS program, additional resources including staff and funding were then allocated to the program. Further, the awareness of local realities and the cultural awareness also helped establish credibility of the program resulting in greater acceptance and appreciation of the program.

3.3.1 Challenges to Program Delivery

Overall, the satisfaction with the progress made by the RMOs was overwhelming, therefore most challenges outlined by respondents were potential, rather than actual. Several of the respondents could not identify any challenges associated with using RMOs to deliver the FNS program, however, comments regarding the ability of the RMOs to correctly identify the needs of the First Nations schools was mentioned. Further risks include having enough qualified organizations to select one that would be able to meet the mandate of the program, although the respondents who noted this risk qualified the remark by stating that all of the current RMOs were meeting their targets. It was generally felt that with the management processes in place to identify the activities of the RMOs, the goals of the program were being met and there was consensus that the program was implemented as intended with no evident gaps in program delivery.

3.3.2 Opportunities for Program Delivery

While progress continues to be made with the FNS program, there remain opportunities that could benefit the program. Of the opportunities listed by the respondents, the most commonly cited opportunity lies in building partnerships with other departments or agencies as well as making more connections with other programs. Educational software and distance learning were provided as innovative opportunities to be explored and expanded in the FNS program although a few respondents stated that their focus remains on connectivity. Others noted that the unique needs and challenges of the First Nations schools in their region need to be addressed more carefully prior to any plans for e-learning, or other innovative avenues. The lack of local capacity to implement other opportunities should be viewed as an indication that additional training or other resources may be required prior to these opportunities being employed.

3.4 Program Impact

There were a number of positive outcomes cited by respondents due to the success of the FNS program. One of the most frequently cited positive element of the program noted was the strength of the national network involving the RMO, Industry Canada Regional Office, and Industry Canada national office staff in terms of program management and delivery. The efficiency of this structure and the frequent communication were thought to be of significant importance to the overall success. The primary objective of the program of providing high speed connectivity to the First Nations schools was also noted due to the educational benefits, future applications (i.e. distance education, e-learning, and e-health), and the ability of the program to help students in First Nations schools to be better prepared for employment opportunities. The Internet was seen as a tool for students that could support First Nations students in becoming competitive in a knowledge-based community.

Respondents were questioned about the progress made towards the overall goals of the FNS program. The objectives of the First Nations SchoolNet program are:

• To promote innovation in First Nations schools by providing them with connectivity infrastructure, related support services, and a new pedagogical approach addressed to teachers;
• To support the production of First Nations—based relevant online educational resources;
• To contribute to the establishment of an ICT culture in First Nations schools through investments in the ICT skills training of First Nations learners and teaching staff;
• To promote the use and integration of ICT into First Nations classrooms for learning purposes; and
• To help First Nations learners acquire the skills and knowledge to be better able to access the job market.

Respondents either agreed or strongly agreed that the FNS program is meeting its objectives, however, many cautioned that the aforementioned objectives are long-term and difficult to gauge at this point in time. All respondents stated that FNS was implemented as intended, has produced positive impacts, and most were more comfortable speaking on the program's ability to meet shorter-term outcomes. Overall, respondents noted that the program is headed in the right direction, and with continued support and increased funding to build local capacity, the objectives will be reached in the next several years.

The perceived importance of the FNS program in fostering connectedness amongst First Nations schools was evident with all of the RMO coordinators rating the program as either very or extremely important. Industry Canada respondents elaborated on this importance by stating that the FNS program provides opportunities not provided by any other program through connectivity, training, and a tailored approach to account for local needs. The program was also seen as essential in the education of First Nations students, and is viewed with the potential to create employment opportunities thereby positively influencing the economy while providing needed skills to First Nations students.

Section 4: Project Findings—RMO Coordinators

The Consultant completed key informant interviews with six RMO coordinators. Their responses are categorized by theme in the following sections.

4.1 Program Participation

All of the RMO coordinators interviewed were well aware of the funding structure associated with First Nations SchoolNet. This is likely due to the Contribution Agreements signed annually detailing the activities of each RMO along with the timelines and funding levels. The issue of funding is elaborated on in Section 4.2 of this report.

Respondents were asked to describe of the number of people in their organization that are devoted to the management and/or delivery of the program as well as the roles and responsibilities of each member. The number of staff dedicated to the First Nations SchoolNet program ranged from 3 full time staff to 14, although the smaller offices tended to use sub-contractors to fulfill their objectives. While many of the job titles and duties were considered to overlap, RMO staff most commonly includes the following personnel and associated job duties:

• RMO Coordinator—general supervision, management, planning and vision of the RMO. Other duties include reporting on activities and attending national meetings.
• Program Manager(s)—reporting duties, supervising the distribution of resources to schools, developing partnerships for leveraging funding, and other management related tasks.
• IT Systems Analyst(s) / Technician(s)—hardware and software assistance, identification of technical requirements, web design and general technical support.
• Finance Clerk / Accountant(s)—processing of invoices and application forms, generation of activity reports and general accounting tasks.
• Administrative staff—common office duties and general assistance.

Other positions that are often sub-contracted include multimedia technicians, human resource consultant, and persons with specialized duties that include training school representatives on computer-related topics associated with First Nations SchoolNet (e-learning applications).

While most RMOs tend to be achieving their targets with current staffing levels, the ability to sub-contract some tasks has helped in reaching their objectives. Although the RMOs are capable of meeting their targets, several of the RMO coordinators believe that either an increase in staff or in the ability to hire more contract staff would be beneficial to support reporting duties associated with the program as well as the need to interact with schools known for high teacher turnover rates.

The RMOs are commonly located close to the communities they serve and organizations with RMO designation are commonly involved in initiatives outside of the First Nations SchoolNet program. Most of the RMOs have contracts with other programs typically related to computers and education.

4.2 Program Management

Respondents were questioned about common management issues that may affect the delivery of the First Nations SchoolNet program. The responses pertaining to program management were consistent among the informants. The clarity and extent of the guidelines from Industry Canada was investigated and all of the RMO staff responded similarly. All of the respondents felt they received sufficient and clear guidelines from Industry Canada to deliver the First Nations SchoolNet program. Many felt that the Contribution Agreement clearly outlines the roles and responsibilities of all stakeholders, and the annual proposals submitted by the RMOs detail the activities to be accomplished during the following year. Suggestions for improvement include standardizing annual reporting through a report template. It is important to note that standardized reporting templates were created by Industry Canada, however this finding indicates a need to promote the available templates.

All respondents were satisfied with the support they received from the Regional Office staff as well as the national office staff from Industry Canada. The Regional Office staff were found to be supportive and adept at facilitating partnerships through frequent communication and knowledge transfer with the RMO staff. Similarly, First Nations SchoolNet national office staff from Industry Canada were portrayed as responsive, helpful in generating funding partnerships, flexible, and beneficial for raising issues and developing solutions encountered with the program. In addition, there was consensus on the clarity of the roles of the various partners of the First Nations SchoolNet program, including RMO, Industry Canada Regional Office, and Industry Canada national office staff. Respondents agreed that roles were well-known with little to no duplication of roles.

Each RMO stated that they follow a business plan in the delivery of the First Nations SchoolNet program and had mechanisms in place to ensure the plan is being followed by their organization. The strategic planning mentioned by respondents is typically based on the Contribution Agreement that includes certain milestones. The business plan is closely monitored and managed with recognition of changing goals and priorities.

To ensure the effectiveness of the First Nations SchoolNet program, RMOs are required to be responsive to the needs of the First Nations schools they serve. Respondents acknowledged that feedback from schools is essential to providing tailored services that recognize local challenges. All of the respondents stated that their RMO has a process to assess schools' ICT situation that include annual ICT surveys of the schools to identify the level of equipment the school possesses, on-site visits to schools, and collecting feedback from FNS staff and community residents. Several of those interviewed stated that a web-based tool would be beneficial to identify the needs of the school, with the results entered into a searchable database. Respondents noted that through frequent communication with school administrators and members of the community and feedback from Help Desk inquiries help to identify and assess the needs of the schools. The overall flexibility of the program was regarded as an essential element to individualize support and services for the First Nations schools served by the program.

Respondents were asked about how they ensure the service they provide to the school support them in meeting their learning objectives. Most explained that they had difficulty finding resources to determine how First Nations schools are using the services provided by the program. Respondents suggested numerous methods that might be used to assess how the services offered by the program are utilized by the school that included frequent communication with school principals, on-site visits that include training sessions with school staff, and a survey of the applications of the technology provided by the program.

Each RMO is involved in evaluating and reporting on the results of the FNS program in the schools they serve. As detailed in the contractual Contribution Agreement, RMOs are required to submit monthly activity reports detailing performance indicators as well as financial claims to keep Industry Canada informed of their progress in reaching their goals for the year. Annual reports and proposals for funding also facilitate evaluating and reporting on the results achieved by the RMO. Other evaluation mechanisms mentioned by RMO coordinators to evaluate and report on results include the collection of information through the Help Desk, sub-contracting an external assessor for general research duties, and management committee meetings. While RMO coordinators understood the function and importance of the reporting affiliated with the program, they frequently stated that current reporting duties account for a considerable amount of staff time. Respondents typically stated that assistance with the reporting tasks would be appreciated, and the most popular solution presented was the establishment of a standardized reporting template and a listing of data required for the reports. Currently, a template exists in the Annex of each Contribution Agreement signed with the RMOs, however, the lack of awareness indicates that greater promotion of the template would be beneficial to the RMO staff.

To effectively reach their objectives, it is essential that each RMO have sufficient resources to meet the demand of First Nations schools. Respondents were asked specifically about the level of staff, infrastructure and monetary resources and whether the current resources are keeping pace with what students require. The need for additional staff was evident, since most of the schools served are in remote locations making on-site visits difficult and expensive. By having more staff at each RMO to visit the First Nations schools, respondents stated that their relationship with the community would improve and they would be more aware of the needs and applications of the support from FNS. Increased on-site visits would also encourage relationships with school administrators resulting in an improved awareness of the school's needs and potential challenges. Staff turnover rates were noted as being quite high in the First Nations schools, leading to reduced efficiency of on-site visits by RMO staff.

Another factor influencing the resources available to meet their objectives was the increase in costs associated with upgrading the existing connections to broadband. The majority of respondents stated that having more schools with improved connectivity represents greater resources spent from their annual budget, thereby sacrificing other areas of the program including on-site visits, training opportunities for school officials, and upgraded computers. Conversely, respondents were aware that the improved connectivity is an integral part of the program and that broadband connectivity increases the number of opportunities for the schools. Overall, most individuals interviewed pointed out that the connectivity infrastructure has improved through the last several years and each RMO is working to their capacity to deliver programming within their budgets although their progress has not been as rapid as they may have hoped for.

4.3 Program Delivery

4.3.1 Services Delivered

Each RMO was asked which services they provide through the First Nations SchoolNet program. Responses were relatively consistent on a national level among the RMOs. The main services offered include the following:

• Community-based provincial connectivity;
• Help Desk support;
• Computer hardware and software;
• Equipment upgrades; and
• ICT technical support and training aimed at capacity building at the local level.

In addition to the aforementioned services, the RMOs also provided other benefits to the First Nations schools in their region. Most respondents noted that they are involved with a Youth Employment initiative or youth programming that enables students in the First Nations schools to help support the technical assistance required in running the FNS program. Several of the RMO coordinators mentioned that they help support the creation and maintenance of the school's web site. A few respondents also noted that they supply security software to the schools in their region to protect against malware and ensure that the computers are operational.

4.3.2 Degree to Which Program Implemented as Intended

All of those interviewed agreed that the FNS program has been implemented as intended, although several respondents cited key challenges that have affected the implementation of the program.

4.3.3 Barriers to Program Implementation

While all of those interviewed agreed that the FNS program has been implemented as intended respondents cited key challenges that have affected the implementation of the program. Respondents mentioned that there remain persistent challenges to achieving connectivity. These challenges generally include:

• geography;
• general lack of responsiveness from the schools;
• school staff turnover; and
• lack of ability/skills in the school and community.

The general lack of responsiveness in the schools was said to make on-going communication with the schools difficult. While high rates of staff turnover among teachers in the First Nations schools served by the program was mentioned as an impeding factor, nearly all of those interviewed also stated that a lack of skills and overall capacity among community members has made the targets difficult to reach.

While establishing effective partnerships is one possibility of overcoming certain challenges with the delivery of the program, respondents noted that partnering with other programs is often difficult. Potential relationships/partnerships were thought to be possible with other Industry Canada programs including the Broadband for Rural and Northern Development (BRAND) program and the Community Access Program (CAP) since each program deals with connectivity to some degree, however, several respondents noted that these programs typically operate independently from one another. Efforts have been made to promote the program and invite partnerships through participation in national Aboriginal roundtables that sought to determine current issues facing Canada's Aboriginal population.

Although the aforementioned challenges surfaced as barriers to program implementation, considerable variance was found in the degree to which each barrier affected RMOs, further demonstrating the regional differences.

To further explore barriers to program implementation, respondents were asked if three factors had impacted program delivery in their area. The three factors being: lack of qualified contractors/partners; lack of interest/ability; difficulties working with the dominant telecommunication carrier. The following chart details the responses pertaining to these challenges of program delivery.

Chart 4–1 Barriers to the implementation and promotion of ICT

 [Description of Chart 4–1]

As demonstrated in Chart 4-1, for half of the RMOs, finding qualified contractors had been either a barrier or a major barrier. When asked specifically about these problems the respondents stated that the skills of the contractors varied greatly and many contractors were found to be somewhat unresponsive to helping schools. When qualified contractors were not available from the community, many RMOs were forced to spend more of their budget on acquiring skilled contractors and sending them to the remote sites that required assistance. Since the RMOs generally established relationships with qualified contractors through trial and error, several respondents stated that they are in the process of building capacity at each school to eliminate the need for sub-contractors.

Through teaching the necessary skills to school officials and community members, RMOs would benefit from not having to hire separate contractors, however, all but one of the RMOs had experienced a lack of interest or ability of the school staff to assist with the FNS program. This reluctance has hampered training efforts. Several respondents noted that the school staff already has a full compliment of tasks, and additional resources or staff are necessary to properly maintain the FNS resources in each school.

In addition to the challenge of engaging school staff, nearly all of the RMOs also expressed problems working with telephone companies or other ISPs to provide service to the First Nations schools in their region. Several respondents stated that although relationships with dominant telecommunication carriers are a challenge, progress has been made and this barrier is becoming less of a factor in successful program delivery.

Respondents also mentioned that there are persistent challenges to achieving broadband connectivity due to geographical barriers in many areas, which in turn affect the potential to implement e-learning applications and multimedia opportunities.

4.3.4 Emerging Connectivity and Innovation Themes

The next set of questions addressed emerging connectivity and innovation themes pertaining to the FNS program. The potential of the available technology supplied by FNS was thought to allow for other learning applications that include the following:

• e-learning and other educational software;
• language programs and other cultural applications;
• distance education allowing people to remain in their community while pursuing further education; and
• multimedia applications including videoconferencing.

Respondents noted that most of the aforementioned innovative applications require broadband connectivity which is not currently available in most of the participating schools. It was also cautioned that many of the First Nations communities involved with FNS lack the skills and general capacity to adopting these applications at the current time. Many of the respondents noted that these opportunities could be implemented more rapidly if effective partnerships were created between the various government connectivity programs and among government departments including Indian and Northern Affairs Canada (INAC) and Health Canada.

Even though RMO coordinators listed a number of impeding factors and a lack or resources affecting program delivery, there was consensus that they are reaching their objectives and continue to make progress with the First Nations school they serve. The RMO coordinators also mentioned many positive benefits to First Nations communities served by the FNS program as detailed in the next section.

4.4 Program Impact and Challenges

Respondents noted many positive elements and outcomes as a result of the FNS program. The most commonly cited response was the increase in connectivity that typically yields greater acceptance and use of the technology provided. Increased technology use resulted in students learning valuable skills that allow them to be better prepared for the job market while providing a forum for enhanced communication outside of their community. Other positive elements of the program include the building of linkages and increased communication between First Nations communities, allowing opportunities for cultural exchanges and further communication. Overall, the program was thought to have a strong impact on youth, providing beneficial training for students and teachers of the First Nations schools it serves, and promoting the use and understanding of ICT. The perceived importance of the FNS program in fostering connectedness among First Nations schools was evident with all of the RMO coordinators rating the program as extremely important.

The benefits of the program were numerous, and many of the respondents noted success stories of the students served by the program, however the RMO coordinators identified several constraints and challenges of the program. These challenges include the following:

• Sustainability of the program—a general lack of capacity among the First Nations communities and the challenges in forming partnerships to leverage funding and other resources was thought to represent a threat to the sustainability of the FNS program.
• Annual application process for each RMO—believed to represent additional paperwork and be a threat to the longevity of the program. A long-term funding solution was seen as more of a commitment to the program and as a method of securing the relationship built with the First Nations schools.
• Increased proportion of budget towards connectivity—the costs of connectivity have increased due to the increased number of schools on broadband connectivity although these additional costs were not met with a similar increase in the RMO's budget. Further, the RMOs stated that increased funding would allow for enhanced program delivery and closer relationships with the clients served by the program.

While not a specific gap of the FNS program, respondents acknowledged that schools that have a high proportion of Aboriginal students as well as First Nations schools located in Canada's territories have not been provided with the same opportunities as the schools served by the FNS program.

Although the RMOs are responsible for achieving the goals set forth in their Contribution Agreement, external factors affecting program delivery were noted. In some cases, the terrain of the region constituted a major barrier to providing broadband connectivity to First Nations schools. This dependence on the environment may skew the results of certain RMOs. Other external factors mentioned were a delay in implementing SuperNet in Alberta and a labour strike in Atlantic Canada directly affecting the ability of the RMO staff to meet their objectives. While similar issues may exist in other regions, the intention of listing these variables was to provide a realistic context for evaluating the effort of the RMOs.

Section 5: Project Findings—School Administrators

The following section outlines the findings from interviews conducted with a random sample of school administrators from First Nations schools served by the program. As noted in the methodology, there were 19 schools contacted for this study, and schools were drawn randomly from each of the First Nations SchoolNet regions.

5.1 Program Participation

The Consultant interviewed personnel that were familiar with the services and support provided by FNS in each school. Of the school administrators interviewed, eleven were principals, four were computer/network administrators, and four were directors of education. Respondents were involved in the management and/or implementation of connectivity and technology training in their school. Additional responsibilities of those interviewed include:

• general technical support, teaching students;
• professional development with teachers;
• acquisition of hardware and software through requests to the RMO; and
• related human resource issues.

Although all of the school administrators seemed to be aware of the resources acquired by the school, there were only ten respondents who could correctly identify the funding source and the lead organization directing the FNS program. These respondents were also unaware of the relationship between the RMOs and Industry Canada, and many stated that they did not realize that RMOs were a third party in program delivery. The lack of understanding of the program structure did not seem to influence later responses since all of the respondents interviewed had mentioned they had contact with their RMO even though they may have been unfamiliar with the name of the organization.

Six respondents stated that their school was involved in the FNS program prior to the implementation of RMOs. Respondents were then asked about their experience with FNS when they dealt with Industry Canada directly as well as with program delivery through the RMOs. Four of the six school administrators stated that RMOs represent an improvement in the availability of resources, personalized service, and information sharing. The remaining two respondents stated that RMOs were less effective, with one comment directed at consistent problems with obtaining reliable connectivity and another comment that expressed the desire to return to having funds from the program directly available to the school from Industry Canada.

5.2 Program Delivery

The next set of questions were aimed at school administrators' satisfaction with the level of support and assistance they received through the FNS program.

Almost all (18) of the respondents were cognizant of the services provided by the RMOs through the FNS program. All of the school administrators noted that they had received connectivity as a result of the program, however many respondents noted that the connectivity was less than a high speed line, with ten of the connections listed as a satellite connection. Even though many of the respondents stated they did not have a high speed Internet connection, over half (11) of the school administrators agreed that the services they received have helped them meet connectivity challenges at their school. Along with connectivity, the school administrators reported receiving the following supports:

• Computers;
• Digital cameras;
• Digital video cameras;
• Educational software;
• Projectors;
• Scanners;
• Printers;
• Wireless receiver; and
• Videoconferencing hardware.

Five school administrators were not satisfied with FNS program's ability to meet the connectivity challenges of their school and with the resources provided by their RMO. Those who felt that the resources were not sufficient in meeting their connectivity challenges cited the need for a high speed connection and more up-to-date computers. Respondents that stated the resources provided by their RMO did not adequately address their school's ICT situation stated that there needs to be more frequent communication between the RMO and the school to identify the appropriate resources based on the needs of the school.

Most (13) had also received training on applications or on the use of technology facilitated by their RMO. The training mentioned by the school administrators included on-site professional development training for teachers and training on the use of the digital camera and multimedia applications. All of the respondents that had received training agreed that it was valuable. The importance of the training was highlighted by the demand for more training among staff and students. In addition to the training that some respondents received, nearly all (15) of the school administrators noted that they had contacted the Help Desk in their region to assist them with technical problems demonstrating the awareness and utility of the Help Desk.

School administrators were asked of their objectives for connectivity, e-learning, and ICT as well as any existing challenges that limit their ability to meet their objectives. The objective most commonly cited was to provide current technology to students through high speed connectivity in order to enhance their skills and learning. Existing challenges identified by the school administrators include the following, listed in order of frequency:

• High speed connectivity;
• Protection against viruses, spam, and spyware;
• Lack of a technician to assist with common computer problems;
• Staff training on Internet capabilities and applications;
• Having up-to-date computers; and
• Identifying applicable software programs for learning.

Despite these challenges, the vast majority of respondents felt that the services offered by their RMO through the FNS program were keeping pace with what students in their school require.

5.3 Program Impact

Overall, school administrators felt that using a third party organization such as the RMO to deliver the FNS program was beneficial. Respondents noted that the RMO's knowledge of available resources allows them to objectively determine the school's needs. Other respondents stated that the use of RMOs leads to responsive and personalized service through facilitating the acquisition of resources and by providing technical support. Several respondents cited challenges of using RMOs to deliver the program. The most common challenge reported was the distance of the RMO from the school, preventing more on-site visits for assessing the needs of the school and for training. A few respondents suggested that they require additional assistance and proposed that the RMOs should acquire more funding from Industry Canada to allow for more communication and training opportunities.

Respondents were asked their opinion of the relative impact of the FNS program through a series of questions. As demonstrated in Chart 5-1, the most important function of the services provided by the program was the ability to provide both teachers and students access to up-to date resources. Results were mixed when considering the degree to which the program encourages students to stay in school or transition to higher grades as well as allowing schools to deliver programs in local communities.

Chart 5-1 Perceived Impact of the FNS Program on Staff and Students

 [Description of Chart 5-1]

The majority of school administrators stated that resources provided by their RMO through FNS are incorporated into teaching practices and utilized in the classroom. The resources were reported as allowing students to conduct basic research for school projects and allowing teachers to explore other learning resources through the Internet. Other applications of the FNS resources provided by respondents included:

• utilization of learning software;
• computer skill training workshops;
• communication with other schools;
• assisting special education classes;
• multimedia applications including the use of the digital camera;
• video editing and photo courses; and
• language development.

Several respondents understood the value of using the available resources for teaching purposes, however there were factors that prevented their participation. These barriers were reported as issues with connectivity, literacy issues among students, and a general lack of computer skills of the teachers hampering their ability to utilize the technology in the classroom.

The potential for e-learning to assist students was well recognized by all of the school administrators. The main benefits of e-learning were acknowledged as the ability of making more courses available to students in remote areas that may not have access to an education, thereby providing students with more learning opportunities. Although the positive outcomes of e-learning was noted, most respondents were not involved with delivering this type of education in their schools.

Section 6: Project Findings—Overview of the Case Studies

In addition to the key informant interviews, the Consultant conducted two case studies of RMOs, namely Manitoba and Ontario. The case studies included key informant interviews and a focus group of RMO staff as well as a review of administrative documents and other research activities intended on highlighting each of the organizations as it applies to the FNS program. The Consultant developed independent case study reports for each site location included with this report as Appendix E and Appendix F. The following represents an overview of the findings from the case studies.

• Through conducting the case studies, the Consultant found that the FNS program was implemented as intended using RMOs. The roles of the RMO staff were found to be clear and current resource levels were satisfactory although additional funding was stated as a factor that would allow the goals and objectives to be achieved faster. Overall, case study respondents felt that the goals of the program were being met.
• Respondents interviewed through the case studies revealed that on-site visits are strongly encouraged to establish and support partnerships with the First Nations schools in their region. Additionally, on-site visits were stated as opportunities to train teaching staff and students necessary skills. The main challenge associated with conducting on-site visits with the schools is the substantial level of resources in terms of time and funding necessary, due to the geographical remoteness of many of the schools. This challenge has led to other methods of communication including a reliance on telephoning, mailing, faxing, and e-mailing which were all found to be generally unsuccessful in contacting the schools. The communication with the schools represents a major barrier in identifying the needs of the schools as well as how the schools are using the resources provided through the program. Both of the RMOs have formalized communication plans, however due to the high staff turnover and workloads in the First Nations schools, the efforts of the RMOs were stated as hampered by the general communication barriers.
• Respondents from both case studies noted that there is a lack of computer and related technical skills in the First Nations schools served by the program. In response, RMO staff implemented Youth Employment programs designed to build local capacity through training First Nations students in a range of computer skills.
• Due to the tenure of the Ontario RMO and additional partnerships, they were able to make significant progress towards the overall goals of the program. Examples of their success include fostering an ICT culture through the provision of free web hosting for personal and school web sites, whose popularity among the First Nations communities in Northwestern Ontario is significant. Partnerships with FedNor and Health Canada have also assisted the Ontario RMO in reaching their goals.
• The Manitoba RMO, having been established in late 2002, has progressed rapidly in delivering the program to the schools in their region. The Manitoba RMO developed and piloted the Youth Employment program which was replicated by other RMOs due to its success. The Manitoba RMO faces major geographical barriers to providing connectivity to the First Nations schools in the province although it was stated that with additional time and resources, the RMO will reach the goals and objectives of the FNS within the next several years.

Section 7: Synthesis of Findings and Consultant's Recommendations

The following section outlines the overall conclusions of the Mid-Term Evaluation of the First Nations SchoolNet program, as well as recommendations for improvement based on lessons learned and best practices offered by the respondents. This section is structured by the questions that guided the evaluation.

7.1 Synthesis of Findings

Is the First Nations SchoolNet program relevant and needed?

First Nations schools have typically lagged behind other Canadian schools with ICT due to limited resources and geographical barriers. The need for programs and initiatives designed to assist First Nations learners in their education was highlighted in the 2004 Auditor General's Report which stated that "a significant education gap exists between First Nations people living on reserves and the Canadian population as a whole and that the time estimated to close this gap has increased slightly, from about 27 to 28 years". FNS has significantly improved the connectivity of First Nations schools and the program continues to be relevant and needed as further demonstrated by the discrepancy in computer student ratios and connectivity speed between First Nations and non First Nations schools in Canada.

One of the most revealing statistics emphasizing the need for the FNS program is the ratio of computers to students. Data from RMOs demonstrates that the computer to student ratio is now between 8:1 and 5:1 in all of the regions served by the FNS program. Anecdotal evidence from the key informant interviews suggested that this ratio was considerably worse prior to the implementation of the program and that there is still opportunity for improvement.

While the FNS program has achieved connectivity in all of the First Nations schools served by the program, the schools locations in remote areas have created barriers to accessing high speed connectivity. Chart 7-1 demonstrates that the First Nations schools have typically slower connectivity than other Canadian schools. The majority of First Nations schools are still dependent on connectivity delivered through a satellite connection, which is typically slower than a broadband connection (i.e. T-1, DSL, ISDN) and may limit potential for educational opportunities.

The relevancy of the program is also emphasized by findings from the key informant interviews and case studies conducted through this evaluation. All RMO coordinators (100%) rated the program as "extremely important" in fostering connectedness amongst First Nations schools. Key informants also stated that to date no other program has been able to provide ICT in First Nations communities to the degree that FNS has. Further, key informants noted that the increase in connectivity has led to greater acceptance and use of FNS provided technology.

Chart 7-1 Comparison of Connectedness (School Year 2003–04)
Internet Access Among First Nations Schools and Other Canadian Schools

 [Description of Chart 7-1]

To further support comments by key informants, case studies showed that First Nations schools generally lack sufficient library resources and thus have greater dependence on the Internet to provide educational information to students.

To what extent have the goals and objectives of the First Nations SchoolNet program been achieved?

Respondents felt that significant progress has been made towards the overall goals and objectives of the program, however the goals were felt to be long-term outcomes that require ongoing work to achieve. In addition, it was mentioned that while RMOs can provide the necessary infrastructure, other goals of the program were thought to be largely dependent on the awareness and acceptance of the program by the First Nations schools for the utilization of the resources in learning opportunities. Several RMO staff members suggested that the goals should be more specific to allow for more accurate assessments of the progress of the program and the degree to which it has met the overall objectives. This may be due to the longer-term vision of Industry Canada's goals and objectives of the program compared with annual targets for each RMO set forth in their respective Contribution Agreement. In completing the research activities associated with this evaluation, the Consultant found that Industry Canada and RMO staff either agreed or strongly agreed that the FNS program was implemented as intended, has produced positive impacts, and has met some significant short-term outcomes. Overall, respondents noted that the program is headed in the right direction, and with continued support and increased funding to build local capacity, the objectives will be reached in the next several years.

Information collected from the case studies echoed the results of the key informant interviews, however one RMO cautioned that the program should address certain needs of the student population before advancing new technology and applications, specifically e-learning. The rationale provided for this view was that a significant proportion of students in First Nations schools are affected by learning disabilities and sometimes personal issues that have resulted in drop-out rates higher than the Canadian average. It was felt that placing higher demands on students would be counterproductive to the goals of the program. It was also stated that with only two years of involvement in delivering the FNS program to the schools, connectivity challenges remain, and therefore the priorities of the program need to be clear and allow for sufficient time to pass to reach the goals and objectives of the program.

To what extent has FNS had the intended impacts among schools, teachers, and learners?

First Nations SchoolNet has impacted schools, teachers and learners as intended through providing connectivity, hardware, software, and technical support to First Nations schools across the country. Specifically, key informants commented that the technology provided through FNS has:

• afforded many students in First Nations schools new opportunities and an avenue to acquire skills necessary to become competitive in the knowledge-based economy.
• provided students living in First Nations communities the opportunity to advance their knowledge through the Internet and to communicate with people from outside their community allowing for sharing of knowledge and culture.
• served as a mechanism to provide an informational resource to students, thereby supporting them in their learning and compensating for a lack of resourceful libraries, typical of many First Nations communities.

Through the key informant interviews, especially with RMO staff, it was noted that determining the applications related to FNS in the schools was difficult. Several RMO coordinators reported the use of the technology in the schools being used primarily as an informational tool to assist with classroom study, although in some schools students were using the provided resources for multimedia applications such as creating videos and personal web pages. Perhaps the most direct educational application of the technology was the emergence of Internet high schools operating in several regions. Although all of the respondents recognized the value and potential of e-learning opportunities, many schools cannot actualize this potential due to a lack of local skills among students and teachers, as well as insufficient connectivity. The school administrators were appreciative of the resources provided by the program and are currently provided the flexibility to integrate the resources into the classroom at their discretion.

FNS has also impacted schools by providing a way in which the schools can overcome major barriers to the acquisition of ICT. As demonstrated in Chart 7-2, a primary challenge for the First Nations schools in regards to ICT is acquiring the funding to obtain the technology.

Through the FNS program, First Nations schools are provided with or subsidized to obtain computer hardware, software, and connectivity.

Chart 7-2 Perceived Challenges of Using ICT by School Type (School Year 2003–04)

 [Description of Chart 7-2]

While the First Nations SchoolNet program has positively impacted schools, teachers and learners with the introduction of technology and connectivity, this has not been without challenges. Case studies revealed that a major problem with the introduction of new technology, namely connectivity, is that once the communities receive connectivity in their region, the risk of contracting computer viruses and other malware becomes a significant problem. From key informant interviews and the case studies it became clear that a lack of knowledge of the threats to Internet security constantly affects the performance of the computers in the schools. Further, findings from key informant interviews also suggest that a lack of knowledge of general computer maintenance skills is evident in the schools. This indicated a need for training and potential capacity building at the community level. This lack of sufficient training opportunities was also found in the recent Statistics Canada study with respect to First Nations schools.

In response to these challenges, five of the six RMOs have developed a Youth Employment program that provides valuable technical training to local youth and students. The program educates the students on maintaining the function of the computers through anti-virus software and other technical related skills. Students are also versed in multimedia applications including web page design and video streaming. While not the intention of the program, it was found that the skills provided to youth have enabled them to acquire employment in some cases. The youth program is thought to represent a practical solution to building relevant skills and fostering an ICT culture in First Nations schools and communities.

Both of the case study sites had developed and implemented youth training programs, which are described in greater detail in the case study reports.

Does the current delivery model represent the most effective and efficient way to deliver the program?

The use of RMOs appears to be an effective way to deliver the program. Overall, key informants agreed that use of RMOs was efficient and effective, even those who had been involved with the program prior to the implementation of the RMO delivery system. In addition, the Ontario RMO had been involved with the FNS program prior to the implementation of the six RMOs, and during the case study conducted by the Consultant RMO staff stated that the changes to program delivery have led directly to greater results in a more efficient manner and timely manner.

The Consultant investigated whether the RMOs were implemented as intended, and whether program delivery had improved as compared to the central management of the FNS through Industry Canada. The Industry Canada National Staff, IC Regional Staff, RMO staff, and school administrators altogether endorsed the current management structure and provided insight into reasons governing the success of the six organizations in delivering the program. The success of the RMOs in delivering the FNS program was felt to be due to the following factors:

• RMOs have existing relationships with the schools and communities in their region leading to a more personal approach;
• The ICT needs of the schools were more carefully analyzed and met in a more responsive manner through the RMOs;
• RMOs operate with less bureaucracy than the federal government, therefore allowing greater flexibility and responsiveness;
• Technical support to the schools was found to be provided better due to the RMO's working knowledge of each school's capabilities and staff involved in assisting the FNS program; and
• Training opportunities were regarded as a successful element of program delivery through the active engagement of the RMOs in educating the students and teaching staff on related skills to maintain and apply the ICT resources.

Key informants agreed that RMOs were successfully satisfying both Industry Canada and First Nations communities by identifying the needs and intentions of both parties. It was believed that RMOs successfully address the different barriers facing communities and regions served by the program through independent annual proposals and Contribution Agreements of each RMO, that assist by accounting for the challenges impacting success.

What are the lessons learned and best practices with respect to program design, delivery and implementation?

Program structure and reporting were areas in which the program could improve or learn from current operations. RMO coordinators felt that the paperwork of monthly reporting and financial claims was a burden that impacted their ability to deliver the program. To help with the reporting required it was suggested that a standardized report structure be developed, along with a listing of required data to facilitate the generation of both monthly and annual reports. Currently, such a template exists in the Annex of each Contribution Agreement signed with the RMOs. The lack of awareness of this document among RMOs signals a need for improved communication.

The life span of the FNS program was commonly cited as a challenge to effective program delivery to long-term planning and the sustainability of the program overall. RMO staff indicated that they had difficulty creating long range ICT plans with a school when funding is uncertain (on yearly renewal). They felt it would be more efficient to provide services on a multi-year plan and that the multi-year plan would assist in building effective and extended relationships with First Nations schools and communities to support program delivery.

Best practices identified through the key informant interviews included:

• Locating individuals in each of the communities to educate with technical support and other computer-related issues. Teaching staff and students were also afforded training opportunities. However by training community members, it was found that the technical skills would endure at the community level and counter the high turnover rates among teaching staff.
• Relationships with the schools were seen as one of the most essential elements of program delivery. Communication efforts through e-mail were unsuccessful in many cases, therefore faxing and telephone follow-ups were cited as a more effective way to reach the schools. The value of on-site visits with the schools was recognized but the remote locations of the schools and the current budget for the RMOs were seen as preventing more visits. One solution presented by several key informants was the use of videoconferencing to communicate with the schools, assess their ICT needs, and to determine how the schools were integrating the technology with learning opportunities for the students.

Several best practices recognized through the case studies include the following initiatives:

• The Youth Employment program provides a learning opportunity for students while building local capacity. Through this program, students are educated on technical support that includes the development of multimedia applications. In many cases, the trained youth have gained employment from their community as a result of their skills.
• The Grade 8 (G8) supplementary program operating in Ontario represents a method of facilitating the transition from grade 8 to high school by reducing any knowledge disparity. The program was designed primarily for First Nations students to serve as an additional educational resource.
• The Keewaytinook Internet High School (KiHS) represents an innovative response to a community need linking students from various communities to educational opportunities and credit. The Internet high school allows students the opportunity to remain in their communities while pursuing their education.

7.2 Consultant's Recommendations

1. Increased communication between the schools and RMOs is required to support program delivery and partnership development.

   Currently the schools and the RMOs experience barriers to communication including insufficient staff at schools to facilitate program delivery, partly due to high staff turnover, school staff unfamiliarity with RMO representatives, lack of regular e-mail use and general unresponsiveness of schools to communication efforts of the RMOs.

   The Consultant believes that by overcoming these barriers and enhancing communication, the program will better support local skill and capacity building. Further, schools may gain a greater appreciation of the program and partners with more frequent communication resulting in a better working relationship during the tenure of this program. More communication with the schools will also increase the understanding of the program goals and objectives as well as the potential applications of the FNS resources provided to First Nations schools.

   Since on-site visits require a substantial commitment of time and money due to the remoteness of the schools, videoconferencing may represent a plausible alternative for increased communication with the schools, where such technology exists. Other avenues to be considered to support communication include training seminars delivered on-line, videoconferences, or in-person workshops similar to those offered in the Youth Employment program.

2. The identification of how schools are using the resources provided by the First Nations SchoolNet program needs to be improved.

   As the target audience of the program, First Nations schools receive connectivity and other ICT resources from the FNS program to assist students with their learning. Resources typically provided by the FNS program include connectivity upgrades, computer hardware and software, related equipment (including digital cameras and video cameras), and technical support through the Help Desk.

   Although schools differ in their ability to integrate available technology to enhance learning opportunities, it was evident through school administrator interviews and case studies that several schools are leaders in resource application. Divergent skill levels among schools highlights an opportunity for knowledge transfer between schools. Dissemination of lessons learned and best practices among the schools could be directed by the RMOs, facilitated by the existing channels of communication, namely the national meetings that occur with FNS stakeholders.

   Since each RMO signs a contract with the First Nations schools on an annual basis, the Consultant recommends that Industry Canada include a requirement for schools to report on how both students and teachers are using the FNS resources. This brief report would outline the applications of the resources sent to the school used for educational purposes and would also share challenges encountered by the school. The report would replace the annual surveys that have been typically affected by poor response rates. By having the schools detail the uses of the FNS resources, the RMOs would gain a better understanding of usage patterns, potential need for other technology, and any challenges impacting the school's ability to utilize the resources to their potential. Having the onus of reporting falling with the schools would also serve to assist the RMOs by reducing the administrative costs of trying to contact unresponsive schools. Ideally, the reports submitted by the schools to the RMOs would then serve to identify best practices, which could then be shared with other RMOs and hence other First Nations schools served by the program.

3. Partnerships with other departments and organizations should be established to enhance the efficiency and sustainability of the program.

   The goals and objectives of First Nations SchoolNet are closely aligned to the mandate of other programs and represents opportunities for partnerships. Thus, respondents noted that development of partnerships might help to facilitate increased connectivity as well as increasing learning opportunities in First Nations schools. Programs noted by respondents include the following Industry Canada programs: Broadband for Rural and Northern Development (BRAND), Computers for Schools, and the Community Access Program (CAP). Other potential partnering opportunities include building or strengthening relationships with Indian and Northern Affairs Canada (INAC), Health Canada, Atlantic Canada Opportunities Agency (ACOA), Western Economic Diversification Canada (WD), and educational partners to facilitate and expand on the services provided by the FNS program while leveraging funds and building sustainability. It was also felt that partnering could help to reduce the "silo effect" which limits the overall success of programs and represents an opportunity to stream-line resources to the First Nations communities within a broader mandate.

   The significance of partnerships has been demonstrated in the Ontario region with the Federal Economic Development Initiative for Northern Ontario (FedNor), the only economic agency involved with the FNS program. In total, FedNor has invested over ten million dollars into building and supporting connectivity and infrastructure in Northwestern Ontario. The investment by FedNor, coordinated with FNS resources, has led to other partnerships including the involvement of Health Canada who is currently purchasing connectivity from the Ontario RMO to deliver Telehealth to 24 First Nations communities.

   In addition to FedNor's involvement with the program, five RMOs have implemented a Youth Employment program designed to provide youth with computer skills, thereby helping youth build local capacity while gaining employable skills. The funding for the youth program resulted in part from an investment from Human Resources Skills Development Canada (HRSDC). The partnerships with other programs as well as other organizations is mutually rewarding and helps to create a sustainable approach to providing connectivity and ICT resources to First Nations communities in Canada.

4. The annual Contribution Agreement could be extended to secure funding to the RMOs for a longer time period.

   A longer term funding commitment from Industry Canada would allow for better RMO planning and would facilitate the development of new partnerships or strengthen existing partnerships. Findings suggest that there are several drawbacks of the current annual funding agreement. Challenges include difficultly in leveraging funds from partners, demonstrating commitment to the schools through a sustainable approach, and forecasting budgets at both the RMO and the school level for the next several years. While annual funding agreements allow monitoring of the RMO progress and added flexibility to individualize their services, many RMO coordinators felt that the challenges far outweighed the benefits of this funding arrangement with special mention of reporting and paperwork burdens. It was felt that secure funding over several years would allow the RMOs to plan additional training opportunities for community members and youth, as well as, develop long-range hardware and connectivity strategies for First Nations schools. Additionally, RMO staff noted that a reduction of, or assistance with both progress and financial reporting would be appreciated. This assistance may be partly accomplished through greater promotion of existing templates, an extension of the annual contracts, or a reduction in reporting duties.

   Extended funding arrangement were also felt to facilitate stronger partnerships with the schools, and provide a better opportunity to establish funding relationships with other government sponsored programs (CAP, BRAND), as well as initiatives from other organizations (Health Canada, INAC, WD, ACOA). Finally, with respect to the Contribution Agreement, respondents felt that it should be revised to account for the discrepancy between fiscal years of the RMOs (fiscal year) and the schools (academic year).

5. Annual short-term goals and objectives of the RMOs should be aligned with the longer-term objectives of the FNS program.

   While the overall program goals and objectives have remained the same, each RMO prepares a list of goals and activities to be completed for each annual Contribution Agreement, resulting in the short-term goals of the RMOs sometimes being noticeably different from overall program goals that have a longer-term focus. Given this discrepancy, the Consultant suggests that more work should be done to align the goals of the RMOs with the program. By working to better articulate the short-term and long-term goals of the program a more accurate assessments of the progress of the program could be obtained.

6. RMOs should continue to be the primary method of program delivery.

   Although the First Nations SchoolNet program was developed, implemented, and managed directly from Industry Canada prior to the establishment of the RMOs, the program has become appreciably more efficient and successful as a result of RMO program delivery. In conducting the Mid-Term Evaluation of the First Nations SchoolNet program, the Consultant found that respondents from both the Industry Canada regional and national office were fully supportive of the RMO program delivery method, as well as the progress achieved to date. The success of the RMOs was attributed to each organization's understanding of the local realities impacting the community as well as the school, and the RMOs ability to maximize program success through a flexible approach to program delivery. The credibility of the RMOs along with the established relationships with community members provided a forum to engage the program in an efficient and meaningful manner, thereby encouraging the success of the FNS program. In addition, the Consultant recommends the continuation of frequent communication between the RMOs and Industry Canada at the national and regional level to discuss best practices and share knowledge.

Management Response

September 2005

Background

This formative evaluation of the First Nations SchoolNet Program (FNS) was undertaken by R.A. Malatest & Associates Ltd., on behalf of the Audit and Evaluation Branch and the Information Highway Applications Branch (IHAB) of Industry Canada.

The purpose was to provide Industry Canada with information pertaining to the management, delivery, progress and challenges of the program.

The evaluation methodology included 55 key informant interviews comprised of representatives from the following stakeholder groups: Industry Canada national and regional staff, Regional Management Organizations (RMOs) service providers and program staff, school administrators, and other stakeholders involved with the FNS program. In addition, two case studies were undertaken at the RMOs in Manitoba and Ontario, and a series of document reviews took place.

The following is a brief summary of the findings of the evaluation which are provided in more detail in the evaluation report:

1. The FNS program continues to be relevant and necessary;
2. Substantial progress has been made towards reaching the goals and objectives of the FNS program;
3. FNS has had the intended impact among schools, teachers, and learners;
4. RMOs continue to be an efficient and effective method of program delivery;
5. Building local capacity was regarded as one of the most important factors governing the success of the FNS program;
6. Case studies revealed several innovative programs have been created and supported through the FNS program.

First Nations SchoolNet seeks to provide First Nations schools on federal reserves with computers and other information and communications technologies (ICT) equipment, Internet access, enhanced broadband connectivity (1.5 mb), technical support and training to enable First Nations learners to pursue new opportunities for learning and skills development within their own communities.

Although FNS currently sunsets on March 31, 2006, this evaluation study will be very helpful in making recommendations to guide decisions for the future design of the program.

Management is committed to addressing each of the recommendations found within the evaluation study as set out in the detailed Management Response below, as applicable now and if the program continues.

Management Response to the Recommendations

Recommendation A: Increased communication between the schools and RMOs is required to support program delivery and partnership development.

Management Response:

Agreed. Management accepts the recommendation that increased communication between the schools and the RMOs is required to support program delivery and partnership delivery.

However, it is important to ensure that this requirement is addressed within the context of the parameters of program delivery. Specifically, Industry Canada delivers FNS through Contributions funding and as such must comply with the Treasury Board Policy on Transfer Payments. As stated in this policy, transfer payments are made to further approved federal government policy and program objectives, and funding is provided on the basis of achievement of performance objectives as set out in a contribution agreement. To ensure that specific accountabilities, milestones and program objectives are addressed, the instrument within which the program can influence these outcomes is through the contribution agreement and ongoing monitoring of the agreement. Once a contribution agreement is in place, any addition or modification of existing activities, milestones, or accountabilities may require an amendment to the existing agreement or the drafting of a new agreement.

Therefore, the recommendation to increase communications between the RMOs and the schools at this point in the project year requires an examination of the contribution agreements that are already in force.

With respect to the current contribution agreement requirements, management is committed to ensuring that the prescribed communication-related activities between RMOs and the schools continue to be undertaken in as efficient and effective manner as possible to support program delivery and partnership development. Within the parameters of the current agreements, FNS is in a position to encourage more communication via videoconferencing, where available, and increased web-based communications as part of the existing requirement for RMOs to provide ongoing service delivery to schools over the 2005-06 fiscal year.

With respect to enhancing and augmenting current communication-related activities between RMOs and the schools, management will take into consideration the increased cost and budget requirements related to on-site visits, expanded video-conferencing and other recommended forms of communication and monitoring in the future design of its program and by extension during the development of future request for proposals and contribution agreement documents.

Timeline: Fall/Winter 2005-2006

Recommendation B: The identification of how schools are using the resources provided by the First Nations SchoolNet program needs to be improved.

Management Response:

Agreed. Management supports the recommendation that it would be beneficial to have an improved method of identifying how both students and teachers are using FNS resources.

In support of this result, the current contribution agreement between the RMOs and Industry Canada requires the RMOs to draft and submit an ICT Strategic Plan. This Strategic Plan has been designed to be a multi-year plan spanning a three-year period. The implementation of this Strategic Plan will require schools to provide information not only on the usage of FNS resources by students and teachers within the school, but also on the impact the usage of these resources have had in the context of a First Nations community. However, the implementation of the Strategic Plan is subject to the program receiving authority beyond 2005/06.

Should this authority be granted, in preparation of future program contribution agreements and the subsequent articulation of project objectives, milestones and accountabilities, the program will undertake awareness sessions on the necessity of receiving information on usage from the schools. RMOs and the IC Regional Representatives will be engaged to identify and share solutions on how best to collect key metrics, such as usage. Program management will work closely with the RMOs to identify standardized reporting requirements for schools so that the program is better able to obtain information to assist with project monitoring and program policy purposes.

Management also supports the recommendation to encourage knowledge transfer and sharing of best practices between First Nations schools. The FNS program will take a more active role in facilitating this knowledge transfer through various means: encouraging RMOs to share known leading practices and invite schools in their jurisdiction to post their own experiences on currently available web-based communication systems; sharing leading practices at the semiannual national FNS stakeholder meetings and monthly RMO meetings for further dissemination by RMOs to the schools; working with RMOs to encourage and facilitate school events whereby they invite other schools across Canada to participate via videoconferencing; and in collaboration with the RMOs, produce video clips of leading practices in action for dissemination to other schools and other parties of interest.

Timeline: Winter 2006.

Recommendation C: Partnerships with other departments and organizations should be established to enhance the efficiency and sustainability of the program.

Management Response:

Agreed. Management supports the view that partnerships with other federal departments and organizations can enhance the efficiency and sustainability of the program.

In recent years, First Nations SchoolNet has been collaborating more closely with fellow IHAB programs, including the Broadband for Rural and Northern Development (BRAND) initiative. Both programs have been coordinating efforts towards providing broadband connectivity to First Nations communities on federal reserves. Specifically, these two programs worked together on a project in Québec in collaboration with the regional economic development agency (CED) and the Province of Québec, to bring connectivity to unserved communities in that province, including First Nations communities.

In support of information sharing and knowledge transfer between the two programs as it pertains to connectivity delivery for First Nations reserves, BRAND has been taking an active part in attending the FNS semi-annual national stakeholder meetings; the monthly technical meetings with RMOs; and an interdepartmental working group with INAC which seeks to provide a better understanding of this unique client group.

Over the current year, in co-operation with IHAB programs and other government departments (OGDs), FNS will continue to increase the adoption of ICT in First Nations communities through various means, including actively participating in INAC's interdepartmental Aboriginal Connectivity and Portal Working Group. In addition, FNS will explore and encourage aboriginal representation on existing consultation networks related to broadband connectivity. FNS will also continue to work with HRSDC via the Youth Employment Strategy to ensure effective program delivery as it pertains to engaging First Nations youth for the purposes of building local technical capacity in these communities.

In support of these partnerships, FNS continues to track data on the impact these partnerships are having on connectivity progress in First Nations schools. RMOs report on a monthly basis the degree of enhanced connectivity and the partnerships they are able to leverage at the local level to reduce/share costs for both government and end-users.

Timeline: Winter 2006

Recommendation D: The annual Contribution Agreement could be extended to secure funding to the RMOs for a longer time period.

Management Response:

Agreed. Management supports this recommendation, with qualifications.

First Nations SchoolNet is a sunsetting program. In the event funding is extended beyond March 31, 2006, program design will take into consideration the need for multi-year funding and address the need to streamline the reporting requirements of First Nation non-profit organizations. These needs will be assessed against the context, parameters and conditions of a renewed program authority, Treasury Board's Policy on Transfer Payments and the assessment of risk.

Timeline: Fall/Winter 2005-2006

Recommendation E: Annual short-term goals and objectives of the RMOs should be aligned with the longer-term objectives of the FNS program.

Management Response:

Agreed. Management accepts this recommendation, with qualifications.

An underlying principle of the Treasury Board Policy on Transfer Payments pertains to providing contributions to organizations that further approved government policy and program objectives. Therefore, it may be more representative to ensure that the short-term goals and objectives of the RMOs support the longer term objectives of the program, rather than to suggest that the current goals of these recipients be adjusted or 'aligned' to fit those of a government program. That is to say, FNS needs to determine that this alignment pre-exists as part of the assessment of project proposals from applicants seeking funding support.

In the event funding is extended beyond March 31, 2006, the approval and selection process of applicants (RMOs) will be reviewed to ensure a clear alignment exists between the shorter term goals and objectives of the RMOs with the longer term objectives of the FNS program.

Timeline: Fall/Winter 2005-2006

Recommendation F: RMOs should continue to be the primary method of program delivery.

Management Response:

Agreed. Management supports this recommendation.

In the event program authority is granted beyond March 31, 2006, program design will take into consideration the RMO delivery model.

Management also supports the recommendation of continued frequent communication between RMOs and Industry Canada at the national and regional level to discuss best practices and share knowledge. As referenced earlier, FNS will take a more active role in facilitating this knowledge transfer through various means: sharing leading practices at the semi-annual national FNS face-to-face stakeholder meetings; holding monthly technical meetings with RMOs and regions; undertaking monthly program operational meetings with RMOs and regions whereby guest speakers on relevant topics of interest are also invited to participate; conducting monthly regional meetings with the regional office and the RMO to discuss issues as they pertain to contribution agreement implementation; and beginning Fall 2005, implementing quarterly governance meetings between the regional offices and the national office.

Timeline: Fall/Winter 2005-2006

Long descriptions

Figure 1.1: Long description—Background—First Nations SchoolNet National Office Organizational Chart

Organizational chart depicting the current structure of the First Nations SchoolNet program and the number of schools per Region Management Organization

Return to page

Long description of Chart 4-1—Barriers to the implementation and promotion of Information and Communications Technology (ICT)

Return to page

Long description of Chart 5-1—Perceived Impact of the FNS Program on Staff and Students

A clustered column bar chart depicting RMO responses on the barriers to implementation and promotion of ICT.

Return to page

Long description of Chart 7-1—Comparison of Connectedness (School Year 2003–04)

A clustered column bar chart comparing the level of connectedness of internet access between First Nation Schools and other Canadian Schools (School Year 2003–04).

Return to page

Long description of Chart 7-2—Perceived Challenges of Using ICT by School Type (School Year 2003–04)

A clustered column bar chart depicting the perceived challenges of using ICT by School Type (School Year 2003–04).

Return to page