Formative Evaluation of Student Connections

Final Report

May 17, 2006

Prepared for:
for Student Connection Program, Industry Canada

Prepared by:
Hallux Consulting Inc.

Tabled and approved by DAEC on September 21, 2006

Table of Contents

• Executive summary
• 1.0 Introduction
• 2.0 Program Description
• 3.0 Methodology/Approach
• 4.0 Findings
  • 4.1 Program Design and Implementation
  • 4.2 Continuous Improvement
  • 4.3 Performance Measurement Systems
• 5.0 Conclusions and Recommendations
• Management Response or
  PDF version (40 KB, 3 pages)

Annexes (Separate document)

(Note: Annexes are available via an Access to Information)

• Appendix A – Interviews and Guides
• Appendix B – Documents Reviewed

Note:

Minor editorial changes were made to this report in order to prepare the document for posting to the Industry Canada's Website (including removal of standard Appendices such as list of interviewees and questionnaires). Readers wishing to receive a copy of the original version of this report should contact the Audit and Evaluation Branch at Industry Canada.

Executive Summary

Hallux Consulting Inc. (Hallux) was engaged by Industry Canada (IC) to conduct a formative evaluation of Student Connections (SC). The evaluation was undertaken in order to:

• Evaluate the program's design and implementation;
• Identify opportunities for improving the effectiveness or efficiency of program delivery; and,
• Assess the adequacy of performance measurement systems.

The key steps in the evaluation were:

• Reviewing program documentation provided by IC and summarizing it in relation to the evaluation issues; and,
• Key informant interviews with IC program management, the staff of the Association of Canadian Community Colleges (ACCC)—the third party delivery agent for SC, and with Administrative Centre (AC) Coordinators at the colleges and universities contracted by the ACCC to deliver SC.

The findings of this study are presented under the three evaluation issues:

1. Program design and implementation
2. Continuous improvement
3. Performance measurement systems

Evaluation Issue No. 1—Program Design and Implementation

• SC's program objectives require a design that balances reaching as many Canadian small and medium enterprises (SME) and seniors as possible to deliver information and communications technologies (ICT) training while providing meaningful, short-term work experiences to as many youth as possible. Under its current design SC has a national scope, with a presence in every province and most major cities. It trains over 15,000 clients and employs 300 youth each year.
• The program's design is viewed as offering numerous benefits, and generates a great deal of enthusiasm and support among all involved with its delivery. SC is also valued by the SME (and seniors) that take SC training and by the youth who provide the training as Student Business Advisors (SBA). Semi-annual surveys done of SME, seniors and SBA by an independent company consistently show high levels of positive responses to questions concerning the program's value and benefits.
• SC remains an unique program in most of the regions in which it is delivered. Its focus on technology training for SMEs and seniors distinguishes it from other initiatives aimed at the general population. SC's use of youth trainers is also distinctive, as is its use of them in a capacity that is directly related to their educational focus.
• All AC stated that they would provide training services in the minority official language if requested Three of them experience sufficient demand for such training that they offer a fully bilingual program.
• SC's governance and accountability structure has a variety of mechanisms that support decision-making for ongoing planning, management and resolution of issues. The structure combines mechanisms such as mandatory reporting and a formal committee structure with a close working relationship among the three parties responsible for delivery of SC (IC, the ACCC, and the individual AC). These allow for timely exchanges of information and views concerning the program, its design and evolution.
• SC is satisfying IC's accountabilities to the approved program terms and conditions and of the Youth Employment Strategy (YES), which is the source of funding for SC. The extent to which SC is meeting its objectives and generating its results are systematically monitored by surveys. These have consistently shown that it is.

Evaluation Issue No. 2—Continuous Improvement

• SC's location within its host post-secondary institution has a significant influence on the program's success, as does the presence of either a full-time coordinator or a coordination team that has the support of the institution's senior management. Emphasis is therefore placed on the positioning of ACs within their host institutions, and on trying to ensure that those appointed as AC Coordinators possess the appropriate combination of personal and professional qualities.
• Starting in 2003–2004, SC has been delivered through a contribution agreement with the ACCC. This change was accompanied by several measures designed to improve the accountability, results measurement and transparency of the program. Since then, the ACCC has implemented other measures to support new ACs that have joined SC.
• Notwithstanding these improvements, significant challenges have arisen since 2004. These challenges, which were identified as risks in the 2004 evaluation of SC, stem from changes in the program's operating parameters brought about by the contribution agreement and by certain Youth Employment Strategy restrictions. They are threatening SC's ability to maintain its national scope.
• Funding is widely regarded as inadequate. It also threatens the program's ability to maintain its national scope.
• SC has a very limited marketing budget. Its experience has been that national promotions are ineffective. Consequently it focuses on networked and grassroots approaches assisted by ACCC and IC. These are labour-intensive, but they work, particularly in established centres.
• There are mixed views on the value of the ACCC's contribution to the delivery of the program. While new centres and program management at IC endorse the ACCC, its added value is questioned by a significant number of established AC.
• SC's operating parameters and funding level are not its only challenges. The persistent issue with the program's name being confusing for some clients, the difficulties than can arise if the AC is not positioned appropriately within its host institution, turnover of AC, the short-term nature of the program's contracts, and a perceived unwillingness to share impede delivery of the program.
• Outdated products are another risk that threatens the program's ability to maintain its national scope. The ACCC has therefore developed a product renewal strategy to address this issue. The strategy is based on distributed product development among AC, coupled with contractual and/or in-kind contributions for development and quality assurance.
• The ACCC's product renewal strategy is unproven.

Evaluation Issue No. 3—Performance Measurement Systems

• SC has a robust set of measurement systems that provide a rich data set for monitoring the program's performance. The ability of AC to meet quantitative performance targets and benchmarks is monitored on a monthly, quarterly and annual basis. Failure to meet targets quickly becomes evident, and when it does the ACCC intervenes directly to provide advice and assistance.
• In addition to the scheduled monitoring, the ACCC has instituted rotating audits of the AC that review procedures, data quality and compliance with government and program policies. The ACCC would like to supplement the audits with spot checks and formal lessons learned analyses, but lacks the funding to do so.
• The data quality assurance processes used by the ACCC and IC are complemented by local procedures.
• SC periodically refines its performance measurement processes to ensure that they continue to generate useful information concerning the program.
• The 2004 SC Risk-based Audit Framework (RBAF) identified four key risks to SC. Measures to mitigate these risks are in place, but they are not all effective. In addition, a new risk, stemming from the changes in SC's operating parameters, has since arisen for which there may be no effective mitigation.

Conclusions and Recommendations

SC is an unique initiative that consistently draws very positive responses from both its participants and its target beneficiaries as well as meeting its national targets and benchmarks. Because it is delivered by IC but funded by the YES, it must balance delivering practical, relevant ICT training to SME and seniors with providing youth with meaningful work experiences that enhance their employability. Recent changes to the program's operating parameters, combined with the lack of an increase in funding since 1999 and outdated training products and services are compromising the program's ability to maintain that balance. They also pose a risk to the program's ability to maintain its national scope. SC needs to take action to address to these issues. It should:

• develop a contingency plan for product renewal in the event that the ACCC's current strategy fails;
• explore options for securing additional funding and getting greater flexibility in the terms and conditions governing youth employment in the program;
• examine its reporting requirements to determine if the reporting burden on AC, which increased when SC began to be delivered through a contribution agreement, can be reduced;
• explore options for easing the pressures new AC face when they join the program and must establish themselves in local markets for SC products and services.

1.0 Introduction

HALLUX Consulting Inc. (Hallux) was engaged by Industry Canada (IC) to conduct an evaluation of Student Connections (SC).

SC provides practical Internet and e-commerce training to Canadian small and medium enterprises (SMEs). Its suite of training services in information and communications technologies (ICT – the Internet and e-commerce applications), helps SMEs accelerate their e-business adoption and thereby become more competitive in the knowledge-based economy. SC also supports Canadian seniors who want to take advantage of the benefits of the Internet.

Since it was launched in 1996, SC has undergone a number of changes, most recently in how it is funded. For many years the program was delivered through a contract for services with the Association of Canadian Community Colleges (ACCC). Effective with the 2003–2004 fiscal year, the program was delivered through a contribution program with the ACCC. Separate terms and conditions for SC were established effective April 1, 2004. They will remain in effect until March 31, 2008 when the existing funding under the Youth Employment Strategy (YES) is scheduled to expire.

As part of the ongoing management of the SC program, a formative evaluation is required as the program is now approaching the midpoint of its authorized four years. This study responds to that requirement.

The objectives of the current study are to evaluate:

• Program Design and Implementation. Is the program being delivered/implemented as it was designed? Is service available to minority language groups (e.g., in English in Quebec and in French in the rest of Canada)? Is there an effective governance/accountability structure? Does it support Industry Canada's accountabilities to the YES?
• Opportunities for Continuous Improvement. Are there ways to improve program delivery from either an effectiveness or efficiency perspective? What factors have facilitated/impeded the delivery of the SC program since April 2004? What are the strengths and weaknesses of the delivery approaches taken at individual sites? Have there been any unexpected or unwanted outcomes associated with the delivery of SC since April 2004? What should change? Could efficiency or effectiveness be improved?
• Performance Measurement Systems. Is appropriate performance information being collected, captured, safeguarded and used? Is data quality assured? Is the performance information described in the Contribution Agreements and the Results-based Management and Accountability Framework (RMAF) being collected as frequently and as timely as prescribed? Is there any information being collected that is not used or any information not being collected that would be beneficial to have? Is information being collected and used on a regular basis to reassess key program risks? Are key program risks being adequately mitigated? Are processes in place for validating the data quality (e.g., checking for reasonableness, followup questions, audited financial statements, recipient audits, monitoring visits)?

2.0 Program Description

SC was established in 1996 under the YES to:

• Provide Canadian SME with assistance in understanding the benefits of using ICT in their business practices; and,
• Provide youth between the ages of 15 and 30 with short-term work experience.

SC also supports Canadian senior citizens who want to take advantage of the Internet.

Operating out of selected universities and colleges across Canada, youth between the ages of 15 and 30 pursuing post-secondary education in IT-related fields provide short, practical, hands-on ICT training to SME and seniors. The experience that the youth get by working in SC as Student Business Advisors (SBA) helps prepare them for the labour force. SC's suite of training services helps SMEs accelerate their e-business adoption and thereby become more competitive in the knowledge-based economy. SC also supports Canadian seniors who want to take advantage of the benefits of the Internet.

SC draws on academia, business and government to pursue its objectives. It is delivered through a third-party transfer agreement where the ACCC, a not-for-profit, ISO-certified association, is the initial recipient that provides assistance to selected Canadian universities and colleges according to nationally-specified requirements for training, development of training modules, recruitment, hiring, reporting and evaluation.

The ACCC is the national membership organization for more than 150 post secondary colleges, institutes and university-colleges, with locations in more than 900 communities. It is governed by a Board of Directors composed of 12 Presidents/Directors-General of its member institutions from across the country.

Up to 15 universities and colleges participate in SC at any one time. They are responsible for the recruitment, selection and training of the SC SBA, as well as training clients (SME and seniors). Current participants are:

• Collège Boréal (Ontario)
• CEGEP Jonquière (Quebec)
• CEGEP Saint Hyacinthe (Quebec)
• École des Hautes Études Commerciales (Quebec)
• Holland College (Prince Edward Island and Nova Scotia)
• Humber College (Ontario)
• La cité collégiale (Ontario)
• Langara College (British Columbia)
• Memorial University (Newfoundland and Labrador)
• Collège communautaire du Nouveau-Brunswick (CCNB) (New Brunswick)
• Red River College (Manitoba)
• University of Calgary (Alberta)
• University of Regina (Saskatchewan)
• Wilfred Laurier University (Ontario)
• University of Alberta (Alberta)

The total estimated annual cost of SC to IC is $3,500,000. This includes an operating budget of $600,000 within IC that pays the salary and benefits of the SC management team at IC headquarters, and for national marketing. The Contribution Agreement with the ACCC provides for the reimbursement of its eligible costs up to a maximum of $2,900,000. Of this, $2,141,610 flows through the Contribution Agreement to pay for the salaries and benefits of the youth hired by the AC as SBA, and to offset the costs of employing an AC Coordinator at each participating educational institution. The balance ($758,390) is retained by the ACCC to pay the salaries and benefits of the ACCC's project officers, for outreach and communications, training development and delivery, product module development and project management at ACCC. The ACCC also uses the balance to perform evaluation and monitoring activities, including contracting semi-annual surveys of SME clients, seniors clients and youth SBAs through an independent company. These surveys provide SC with feedback on extent to which SC is meeting its objectives and generating its results, as well as supporting improvements to program design and implementation.

Revenues from the sale of training services are retained by the ACs. These may be combined with in-kind contributions from the ACs' host institutions to help offset the costs of delivering SC at the local level. IC estimates that retained revenues and in-kind contributions provide between $1.0 and $1.5 million additional funding to support delivery of the program.

3.0 Methodology/Approach

The evaluation objectives identified in Chapter 1 were addressed by:

• Interviews with stakeholders and delivery agents. Three individuals responsible for SC in IC were interviewed, including its manager. Two representatives from the ACCC were interviewed, as were 16 representatives from universities and colleges currently participating in the program. Two institutions that no longer participate in SC were also contacted, and their representatives were interviewed. A list of the individuals interviewed is provided in Appendix A, as are copies of the interview guides;
• Reviewing program documentation provided by IC and summarizing it in relation to the evaluation issues. Appendix B lists the documentation that was reviewed.

The risks in using testimonial evidence are:

• individuals will feel compelled to participate and this will colour their responses. This risk was controlled by making it clear to all individuals contacted that participation was voluntary and neither their responses nor their refusal to participate would affect their dealings with Industry Canada;
• individuals may not be frank with the interviewer. This risk was controlled by assuring all individuals that all comments would be summarized and that no comments would be attributed to any individual in any reports resulting from the evaluation;
• non representativeness of the sample of individuals interviewed. This risk was controlled by ensuring that representatives of all currently participating institutions were interviewed (i.e,the sample encompassed the entire relevant AC population), interviewing the entire SC management team at IC HQ, and the two officials at the ACCC directly responsible for the program.

Assertions made by individuals (e.g., concerning reports used for tracking performance) were corroborated by inspecting current samples of the reports.

Based on these controls, Hallux believes that the conclusions reached in this report are appropriate.

4.0 Findings

This chapter presents the overall study findings of SC presented by the evaluation issues.

4.1 Program Design and Implementation

In order to reach as many SME and seniors as possible, SC needs to have a presence in as many provinces and cities as possible. One way to do this would be through the Internet – to have a virtual presence everywhere by offering on-line training. The obvious flaws in this approach are that it is unlikely to reach its target SME and seniors (i.e., those who need training in the benefits of the Internet and who are probably not connected to it) and it would not provide work to very many youth.

By offering training through colleges and universities throughout Canada, SC is able to have a presence in every province (and in most major cities) as well as in several rural areas. By offering practical, hands-on training, SC is able to reach businesses and seniors that may be so new to the Internet that the prospect of working with it without a live "coach" is overly daunting. By offering training in either a classroom/computer lab setting or, at the client's option, at the client's business or home, SC is able to reach both novice users who may not have a computer as well as those that have one and some basic skills and who want to acquire greater expertise.

Offering training with these characteristics is labour-intensive, and provides work opportunities for students everywhere the training is offered. It also requires a substantial support infrastructure to develop and refresh SC's training products; market the program and sell specific product offerings to potential clients; recruit, hire, train and manage the student workforce; administer SC at each institution; as well as to satisfy the monitoring and reporting requirements associated with the delivery of a federal government program.

SC's program design provides the required infrastructure by dividing the work required to create and maintain it among program management at departmental headquarters, the ACCC, and each local university or college Administrative Centre (AC).

Among other things, the program's three-person management team in IC concentrates its efforts on policy/funding proposals, audit and evaluation, performance reporting, partnership development and national marketing. The ACCC, which has a five person team devoted to the program, manages national delivery structures, develops training modules to train SMEs and seniors in a nationally consistent fashion, develop targets, monitors and reports on performance, and provides guidelines on such matters as privacy, hiring, networking, and local marketing. The AC Coordinators or the coordinating team, at participating universities and colleges recruit, select, hire, train, and mentor youth; provide training to SMEs/Seniors; and develop/perform regional marketing and sales.

Underpinning this division of responsibilities are yearly contribution agreements with the ACCC, which signs yearly contracts with participating colleges and universities. AC performance is closely monitored by the ACCC, and contracts with ACs that consistently under-perform may not be renewed (if the centres themselves do not decide to withdraw from the program). When this occurs, the ACCC uses a rigorous request-for-proposal process to select a new AC. The RFP selection committee includes a representative from the private sector, and a representative from an AC, as well as the ACCC and IC.

An unique feature of SC is that AC are permitted to retain all the revenues they generate from providing training to SMEs/Seniors. This not only helps them to offset their local delivery costs, which are only partially funded under the contract with the ACCC, it is an incentive to delivering as much training as possible to SMEs/Seniors. The opportunity for leverage provided by this feature not only increases the potential reach of the program to SME, it also increases the number of youth who can be given work experience in delivering training.

The contribution agreement that IC has struck with the ACCC offers many benefits from IC's perspective:

• the ACCC has a strong association with the Canadian post-secondary environment and has significant experience in working with youth;
• it manages other federal programs and therefore has good understanding of program delivery requirements;
• its membership (150 post-secondary institutions with locations in more than 900 communities) allows for greater regional access;
• the ACCC is more in tune with operations of academic institutions than IC. This enables it to generate better rapport with these institution's senior management, as well as making it easier to meet with the institution's senior management to promote and support the program;
• delivery through a third-party allows the AC to voice concerns with program structure/delivery that they might be unwilling to raise to IC directly;
• the ACCC has an excellent record in delivering the program, and has been very strong in its monitoring of the AC. In addition to monitoring performance on a monthly, quarterly and annual basis, it recently (since 2004) started rotating audits of AC;
• it also provides regular support to the AC. A wide range of mechanisms, some time and labour intensive, are used to deliver support. There are monthly conference calls, national meetings, on site visits and follow up, and orientation and training sessions for coordinators at new AC (or for new coordinators at established AC) . The ACCC reviews and provides advice concerning each AC's annual business plan. It has also instituted a mentoring program that pairs coordinators at new AC with experienced ones, and is implementing a new content management system that the AC will be able to use to customize and extend the SC product offerings, and that the SBA will be able to use to learn about contents of these offerings so that they can subsequently deliver training on them.

In sum, IC views using the ACCC as an effective delivery mechanism as it allows the department to concentrate its efforts on policy, evaluation and measurement while ACCC takes care of most of the resource-intensive administrative and logistic matters.

Using universities and colleges to deliver training is beneficial because:

• it enables SC to have a national presence;
• it provides a ready source of qualified youth to serve as SBA in delivering training (and the institutions have a vested interest in providing students with practical experience/skills);
• it enables the program to leverage these institutions' ties to the business community in their area, as well as their access to seniors (many institutions have continuing education/lifelong learning programs) to funnel potential clients for its training services into the program.

From the AC's perspective, SC:

• provides students with a direct link with entrepreneurship, with the realities of the world of business and work in general. It allows them to live a real experience in a business ("Textbook learning comes alive", in the words of one coordinator). Students who serve as SBA grow as individuals, develop self-confidence and self-esteem, and increase a wide range of skills (technology, professional, presentational, sales, team work, and project management). Surveys of SBA consistently bear out just how beneficial SC is for the youth who serve as SBA;
• forces the student to develop a methodical approach. In order to train others, they must develop a planning dimension, which they may not get in their studies;
• improves the visibility of institution within its community. Many universities and colleges have outreach programs to make them less of an "ivory tower", and SC is a natural complement or extension to these programs;
• supports the host institution's efforts to develop collaborative relationships with local businesses;
• enables the host institution to offer services that SMEs cannot get from private-sector providers because they are too expensive. SC training is often offered at very low prices in comparison to those typically charged by the private sector. Some institutions are able to use SC training as a "foot in the door" for SME that may eventually require other, more expensive, professional services offered by the institution (or by the private sector);
• allows the institution to discover exceptionally-talented students who can go on to assist in other parts of it;
• gives their students who have served as SBA an "edge" in the labour market when they graduate and start looking for full-time work.

The AC's perspective, that SC is both good for their students who serve as SBA and for the SME (and seniors) that take SC training is corroborated by the surveys that are done of SBA, SME and seniors who have participated, by Prairie Research Associates on behalf of the ACCC. As reported in the 2004 Evaluation of Student Connections, these surveys have consistently shown high levels of perceived value of and support for SC among its clients. In the most recent survey, conducted of clients who received SC training between October 1, 2004 and March 31, 2005, 91% of respondents agreed that SC was a good value. Such high levels of positive responses are typical. The surveys are designed with a five-point rating scales (e.g., strongly agree, agree, neutral, disagree, strongly disagree), and ask respondents to indicate their level of agreement with a statement about the program (e.g., "I received practical instruction that improved my use of technology"). A wide range of program attributes is assessed, from the overall experience, through the quality and relevance of the course material and the knowledge and competence of the student instructor, to the overall value of the training and whether they would recommend it to others. Negative responses (e.g., disagree, strongly disagree) to positive statements concerning these attributes are rare, often lower than 10%.

Even AC who had decided to no longer participate in the program stated that the program was beneficial to their students and valued by their clients. These AC, however, had found that SC was a financial strain on the host institution, which had come to regard it as unaffordable.

The 2004 Evaluation of Student Connections found it to have an unique combination of features. This remains the case. SC's training is provided by post-secondary students who are in the midst of their education and who have been certified on the SC training modules. They provide short, hands-on training sessions to businesses and seniors at times and locations of their choice (i.e., at the workplace as well as in conventional "classroom" settings at the host institution). Because SC's training is modularized, it can be customized to the recipient's interests or to local market requirements.

IC's Canada Business Service Centres (CBSC) also offer nationwide programs for SME, but their focus is on business information and processes (e.g., developing a business plan, finding new markets, getting ready to export). SC, in contrast, focuses on the Internet and e-commerce applications, as well as on basic computer literacy skills, making it complementary to the services offered by CBSC. Several AC coordinators stated that they station SBA in their local CBSC to take advantage of the marketing opportunities offered by the fit between the two programs.

The Community Access Program (CAP) is another nation-wide initiative that aims to provide Canadians with affordable public access to the Internet and the skills they need to use it effectively, but it is oriented to the general population rather than to the specific needs of SME that want to use the Internet to grow their businesses. An offshoot of CAP, CAP YI (Youth Initiative), hires youth between the ages of 15 to 30, typically students, to work as interns at CAP Sites across Canada. They are employed to help individuals, community organizations and small businesses improve their knowledge and effective use of the Internet and related information technologies. The training, however, is only offered at the CAP site, and the youth employed there may perform a range of tasks in addition to providing training on the Internet and e-commerce applications.

There are, of course, many private sector-providers of ICT training, but they are considerably more expensive than SC and/or tend to focus on material that SC excludes (e.g., MicroSoft's office products such as Access, Excel, and Word). Private-sector trainers often offer courses that require a full- or half-day commitment of time, something many SME are unwilling to do.

While SC's focus on the practical benefits to SME of the Internet and e-commerce applications is distinctive, several coordinators commented that they were seeing increased competition in their market related to seniors training, and that there seemed to be an increasing number of programs aimed at seniors provided by both provincial and municipal governments. At present this is not a widespread phenomenon, but it may represent the start of a trend that could see SC's seniors market decline, making it important to ensure that the program retains its appeal and relevance to SME.

In sum, SC remains unique. Its focus on technology training for SME and seniors distinguishes it from other initiatives aimed at the general population. SC's use of youth as trainers for SME is also distinctive, as is its use of them in a capacity that is directly related to their educational focus (many participating youth are IT or business students).

For the most part, SC's training is offered predominantly in the majority official language spoken where the AC is located. Only three of the current AC reported having sufficient demand to offer a fully bilingual training program. All other AC stated that requests for training in the minority official language in their communities were so small as to be counted on one hand. The typical contingency plan for servicing a request for training in the minority official language was to try to always have at least one bilingual SBA on staff, though many admitted that this was difficult.

The SC governance and accountability structure combines mechanisms such as mandatory reporting and reviews and a formal committee structure with a close working relationship among the three parties responsible for the delivery of SC (IC, the ACCC and the individual AC). These allow for timely exchanges of information and views concerning the program in support of ongoing planning, management and the resolution of issues.

The ACCC closely monitors each institution's AC activities, with ACs reporting to the ACCC on a monthly basis via the Intranet and following this up with quarterly written reports. The ACCC, in turn, provides IC with monthly statistical and financial reports, as well as detailed quarterly reports. The ACCC also reviews and provides advice concerning each AC's business plan, which must be submitted every year. IC and ACCC both participate in the RFP selection committee when selecting new centres.

There is a formal structure of three committees which focus on marketing, product development, and training. The structure includes regional representation on every committee, with a reporting strategy to bring back information to each AC individually.

IC and the ACCC meet regularly, and IC is invited to national SC meetings organized by the ACCC. Both participate in monthly conference calls with the AC.

Both the ACCC and the AC reported that they have ample opportunities to contribute their perspectives and concerns about the program's design and evolution. Some AC noted that change seems to sometimes be slow in coming, but added that they understood that the program's terms and conditions limited what could be changed and how fast this could occur.

The approved terms and conditions for SC set out, among other things, its objectives and expected results. The extent to which SC is meeting its objectives and generating its results are systematically monitored by surveys conducted by Prairie Research Associated on behalf of ACCC and reported to SC management. As noted earlier, these surveys assess a wide range of program attributes related to SC's objectives and results. The surveys have consistently shown that SC is consistently meeting its objectives and generating its results.

In January 2005, a formative evaluation was completed of the redesigned YES, of which SC is a component program. This evaluation found that SC is meeting all the YES program requirements. SC is also systematically collecting data on 8 of the 10 YES outcome indicators for the common performance measurement strategy of YES. The two not being collected were:

• employers' views on enhanced employability skills of participating youth; and,
• the number of youth returning to post-secondary studies.

While these may not be systematically collected by the SC performance measurement regime, it is clear from the interviews conducted during this evaluation with the AC (the employer of the participating youth) that the AC coordinators regard SC as having a significant positive impact on youth employability skills. This view is corroborated by the surveys of SBA conducted by Prairie Research Associates. These have consistently shown that youth believe that the program increases their marketability and potential for employment in the knowledge economy.

A commitment to returning to post-secondary studies is a prerequisite for youth to participate in SC as SBA.

4.2 Continuous Improvement

IC and the ACCC believe that the ideal location for an AC within a post-secondary institution is within a department or program that is:

• entrepreneurial in philosophy and orientation;
• outward looking; and,
• connected to or networking with the local business community.

It is also important to establish a working relationship with an executive at the vice-presidential level or higher to ensure that there is an appropriate level of institutional support for and commitment to SC, and to its objectives.

The ACCC understands this and what needs to be accomplished to maximize the opportunities for success of new centres. It actively exploits its links to its members to try to ensure that host institutions locate the AC appropriately and employ dynamic, full-time coordinators (or teams) who have (or can establish) an extensive network of contacts within their institutions, their communities and among local SME.

Having the right placement within the host institution also helps SC overcome the lack of national marketing. An institution that is well-connected to the local business community can impart awareness of SC's offerings by word of mouth and local chambers of commerce or industry associations.

Situations have arisen in which the AC has only a part-time coordinator who is responsible for other programs as well as SC, an indifferent or laissez-faire host-institution management, and academic programs that are inward looking or operate in silos from each other. The result is usually poor all-round performance, and mutual disappointment.

As a result of such experiences, the ACCC now places great emphasis on the proposed positioning of the AC within the institution and on trying to ensure that new centres appoint individuals or teams with the appropriate combination of personal and professional qualities as AC Coordinators. Since AC are independent contractors to the ACCC, however, there are limits to its ability to influence institutional decision-making about this.

The contribution agreement between IC and the ACCC, which was established for the first time in 2003–2004, contains explicit reporting requirements. These requirements have been flowed down to the AC through the contracts it signs with them following the RFP process discussed earlier in this report. The requirements include monthly, quarterly and annual reports:

• AC must make monthly updates, via the Internet, to the SC database, concerning all eligible SBA on staff and clients trained;
• AC must also confirm, in writing, the information provided in the updates;
• AC must provide a quarterly payment tracking report that details the financial and in kind contributions for interns and core staff salary and benefits. This report must be signed by a senior financial authority at the host institution;
• AC must submit an annual operational plan and budget for the centre.

The ACCC supplements this reporting regime with rotating audits of ACs.

In the view of IC and the ACCC, the program's reporting and its associated controls have increased accountability, and improved audit trails. The program is also regarded as more transparent, both with respect to how AC are selected, and with respect to its operations at the institutional level. IC and the ACCC believe that their abilities to detect problems at the AC level have been improved.

When problems occur, the ACCC will intervene to provide advice and assistance. It has also instituted a mentoring program that pairs new AC Coordinators with experienced ones. Both measures are intended to help AC, particularly new AC, to overcome any start-up problems they may experience.

The change to delivering SC though a contribution agreement not only led to an increase in monitoring and reporting requirements, it also entailed a change in the funding formula. Both changes have created problems.

When the program's additional reporting requirements were introduced it was recognized that they would increase the reporting burden being placed on the AC.

As expected, the burden has increased perceptibly among AC Coordinators. In fact, it was one of the most frequently-cited problems of the program¹, even if it is viewed philosophically by almost all of them. One AC, which ceased operations with the program in March 2006, estimated that it spent about one week of every month dealing with the reporting requirements. Another, a well-established and successful AC, is developing its own system and database to support reporting, believing that it would otherwise "drown".

As noted earlier, the additional reporting has improved the visibility of the program to host institutions' management, but this has not always been beneficial. Several AC Coordinators explained that they must now spend time explaining the reports to university signing officers. There are two aspects to this. First, some must account for their centre's performance in relation to SC's benchmarks to these officers. Second, due to differences in reporting cycles, some have to spend time reconciling the reports produced for the ACCC with those produced by the host institution's systems so that they can account for the differences. In addition, in institutions that have poorly performing AC, the increased scrutiny by the senior financial authority has made it clear that SC can be a financial strain on the institution.

The ACCC, in particular the Vice President Partnership Programs, tries to counter this latter perception by visiting senior officials at current and prospective host institutions and explaining to them that they should not view participation in SC as a money-making opportunity. Rather, they should consider the other benefits of the program. These include maintaining links with businesses and organizations in their community, diversifying their activities, and providing opportunities for their students to experiment with and integrate into the labour market. For some institutions, these benefits have limited appeal — the two institutions contacted that have recently withdrawn from the program cited the cost to the institution of continuing the program as the major factor in their withdrawal from it. Other coordinators expressed concern that their host institutions would discontinue the program in the face of rising costs.

Prior to the Contribution Agreement, funding to the AC was on the basis of clients trained up to a maximum per AC based on a formula for FTE allocation. With the advent of the contribution agreement, funding pays the salary and benefits of the youth hired as SBA, with each SBA paid a set rate of $11.50 per hour (plus 20% for benefits). Originally, all SBA were to be paid for 12 weeks (420 hours) of work at this rate. This has now been changed to an average of 420 hours/SBA, with a minimum of 210 hours and a maximum of 1200 hours for any single SBA. There are also limits on the number of the number of SBA who can be hired for 210 and 1200 hours, as well as a stipulation that no SBA can work for SC for more than 12 months.

The change from funding AC via a formula to paying the salary and benefits of the SBA hired by the AC, while placing controls on the hours and months they can work, was introduced both to ensure better accountability for monies received (employee payroll records provide particularly good audit trails) and to ensure that the program reaches the maximum number of students.

The change, combined with other performance measures instituted concurrently, posed challenges for the ACs. Several subsequently left the program The change was also viewed as entailing several risks when it was instituted. These risks have now become problems – problems that raise questions about the viability of the business model underlying using AC as delivery points for the program.

AC Coordinators contacted for the 2004 evaluation of the program stated that using an uniform rate for all SBA could result in a rate that was out-of-line with local labour market rates in some regions. At that time, the concern was that $11.50 per hour would be too high for an entry-level, inexperienced SBA. To this concern, which was expressed again by some coordinators, must be added a new one: in regions where the economy is booming, $11.50 per hour compares unfavourably to rates paid for less demanding (but admittedly less fulfilling) work by the private sector. In some regions, the SC rate also does not compare favourably with rates paid by other government departments for skilled labour.

The uniform rate has made it more difficult to attract youth to SC and to retain them in the program. Highly-qualified and talented youth cannot be offered the prospect of better pay, unless they are paid outside the provisions of the program. Indifferent performers are compensated the same as excellent performers, a fact that can create human resource management problems for the AC (and that the AC coordinator must devote time and energy to solving).

These problems can be overcome by stressing to prospective hires the other benefits of SC: its long term advantages in terms of facilitating their careers, the fulfilling nature of the work itself, contribution they are making to the local community, even the prestige of being associated with a national program. It is important to note, however, that few people are indifferent to money, so that selling the program to a prospective hire who may be considering a higher-paid job will take time and energy on the part of the coordinator.

None of the coordinators contacted for the current evaluation stated that they were unable to recruit their target number of SBA, but many noted that recruitment was a very time consuming component of the job that ate into the time available for marketing and sales activities — activities that are vital if the AC is to generate sufficient revenues to cover the shortfall in program funding to the host institution (see next sub-section for a discussion of this).

The uniform rate of pay, while troublesome, has not, however, had as big an impact on AC as the SBA hour and time restrictions introduced in 2004. The 12-month maximum duration of a youth's participation in SC has, by far, been the most significant change. It has meant that ACs experience almost 100% turnover in staff every term. Recruitment and training of SBA has become a major cost, in terms of the percentage of an AC's time (and hence salary cost) that must be devoted to it. Some try to smooth the 100% turnover by staggering their recruitment, but this can mean that in an AC with an allocation of 28 SBA, the coordinator can have over 40 direct reports. Because these are all youth, not all of them will be low maintenance employees from a human resources management perspective, further eroding the time AC can devote to marketing and sales.

Coordinators also expressed concerns about their ability to deliver consistently high-quality training services to SME under a business model that entails such high staff turnover. SBA may need to develop self-confidence and presence as well as improving their communications and presentation skills in addition to becoming certified on the SC training products before they can interact effectively with SME. Until this has taken place, they either need to be paired with other, more experienced SBA or mentored by the coordinator. Either reduces the overall productivity of the AC.

The negative impacts of the changes in the program's operating parameters are being felt most acutely in newer centres – those that have joined the program since 2003–2004. Coordinators at the established centres are, for the most part, coping with the changes, but noted that they have made the program significantly more challenging and that they would not want to be trying to set up a new centre under the restrictions of the current regime.

If the changes that have been made to the program's operating parameters are examined from business model perspective, it is clear that they present problems. They have significantly increased the opportunity costs of operating an AC. They have also had a domino effect that compromises a key assumption underlying the viability of the business model – that revenues from the sale of training services, which are retained by the host institution, can make up for any shortfalls in program funding and/or any increases in operating costs. Such revenues only occur if there are sales, which require marketing activities and a workforce qualified to deliver services after the sale is made. AC, which are the primary sales force in the program, may not have a qualified workforce until several weeks into a term (when SBA could remain with the program for more than a year, they could be developed in their first term as sales associates and return in a subsequent term fully-qualified and ready-to-go upon being hired. Those without an interest in or aptitude for sales could come back as trainers in their second and subsequent years. They too would be ready to go upon being hired). ACs' time for sales activities are now squeezed by the program's reporting requirements, and by their own recruitment, training, human resource management, and service delivery activities. In new centres, sales are not occurring in sufficient quantity to generate the revenue needed to make u p for the shortfall in contribution agreement funding. For example, in 2004–2005, 70% of all program revenues were generated by six AC (and almost one-third were generated by two AC), all of which joined the program before the SBA hour and time restrictions were introduced in 2003–2004.

As noted, these problems are most acute in newer centres, which comprise 8 of the current 15 participants, but they are not confined to them. One centre, which has been with the program since its inception, and which is regarded by other centres as having enormous potential, is struggling this year, a fact that is regarded as "deplorable" by other centres.

If ways cannot be found to improve the operating performance of the new AC and sustain the performance of established ones, there is a risk that their host institutions will decide that the funding strain it places on them, which was the second most frequently-cited problem with the program, is unacceptable and will withdraw from it. SC could be left with too few delivery points to be considered a national program.

Funding for SC has not increased in real terms since the program was founded. In addition, despite the fact that YES has been renewed until 2008, funding for SC is reviewed annually and cannot be committed for a period of longer than one fiscal year. As well, and as noted earlier, SC used to fund ACs on the basis of a formula, but now funds them on the basis of the SBA employed. This change, which has reduced AC flexibility in how they deploy their funds, was driven by YES terms and conditions.

The problems that limited funding creates for the program were noted in the 2004 Evaluation of SC, and included:

• Some institutions have been reluctant to participate in or strongly support an initiative that can provide no assurance of a long-term commitment;
• It is difficult to recruit and retain high-quality AC Coordinators. Some of the better-quality individuals who have joined SC have done so as a developmental experience, and have moved to more secure positions as these became available;
• The ACCC found it difficult to build a core, permanent team to support the initiative
• The level of funding means that SC has a very limited presence in many parts of Canada. With more money, it could expand its services to un- or under-served regions and groups (e.g., Aboriginal people). As well, there are certain fixed costs borne by ACCC in delivering the program. ACCC believes that it could serve more institutions with no significant increase in these fixed costs, so that if more funding was available, most of it would enable expanded regional delivery.

Some of these problems remain, though with the change to funding the ACCC though a Contribution Agreement, the ACCC has recruited and put in place a core team of five full-time individuals. This has permitted it to:

• Provide support to the AC coordinators, including conducting operational reviews and audits to detect emerging problems and to mount interventions;
• Make and maintain direct contact with institutional executives, ensuring that they understand SC's objectives and the contribution it can make to their institution.

The ACCC remains convinced that, with additional funding, it could expand the program's reach significantly without commensurate increases in overhead costs. It recognizes, however, that a more pressing funding issue relates to the sustainability of current centres.

In addition to paying the salaries of the SBA, SC provides a $15,000 stipend to offset the cost of the Coordinator's salary and benefits. Aside from a small amount ($750) that is available for promotion and marketing costs, all other direct and indirect costs are to be recovered from training revenues and/or cash and in-kind contributions from the host institution.

The amounts provided for the coordinator's salary and local marketing are almost universally regarded as inadequate within the program. The stipend will not pay for a full-time coordinator. A full-time coordinator (or coordination team) is, however, required for the program to have any chance of succeeding. The ACCC is trying to stress this in its meetings with host institutions, but some are reluctant to take the ACCC's advice.

The combination of the high operating costs entailed by the SC business model and the low level of direct program funding though the Contribution Agreement for the coordination function and for local marketing has become a serious threat to SC. It has created a structural deficit that, at present, can only be overcome by sales revenues or cash contributions from the host institution. With the exception of a several well-established institutions, AC reported that the required sales revenues are not present.

For example, in 2004–2005 only the following institutions, all well established, achieved or exceeded the "break-even" target of sales equalling 40% of the total funding they received through the ACCC: University of Regina, Collège Boréal, Wilfred Laurier University, École des Hautes Études Commerciales, Jonquière and Saint Hyacinthe.

As well, when SBA hours spent training as a percentage of total SBA hours is compared year over year across all institutions, it shows about a 5% annual drop. This trend, for SBA to be doing less and less training each year, is evident since 2001/02. Training, of course, is the primary revenue generating activity of the SBA.

Now, no institution is likely to be a high performer in its first year of operations, but the ones that are now doing well have been with the program for several years, and set themselves up under a different business model. There is a risk that the 8 new institutions, as well as the established ones that are not doing well (one of the above six reported that it is now also running a deficit and may not continue with SC) will not be able to withstand the financial strain. Their hosts may then decide, as did the two institutions that withdrew from SC, that it is unacceptable and withdraw. SC could be left with too few delivery points to be considered a national program.

Alternately, the program could be reduced to having a stable core of established and successful AC, and an ever-changing periphery of AC that join for short time and then withdraw because they cannot accept the financial strain or are terminated because they cannot meet the required benchmarks and/or the objectives of the program.

SC's funding places limitations on its ability to launch significant marketing initiatives. There are no current national media campaigns that are targeted at SME or seniors and, other than promotions done through YES (e.g., its web site, the Summer Work Experience brochure), there are no national campaigns targeted to youth. As a result, a common response from businesses learning about SC is, "Why haven't we heard about this before?"

AC Coordinators cited the lack of a national marketing campaign as a serious challenge that results in low awareness of the SC "brand" among SME. Because there is low awareness of the SC brand, coordinators reported that they have to overcome misunderstandings, attributable to the program's name, of what SC is and does. For example, SME tend to believe that the program is not even aimed at them because its name contains no reference to businesses, or even to ICT (except, perhaps, indirectly through "Connections"). Lack of clear branding and the resulting misunderstanding of the program increase the time and effort required to get a "foot in the door" with SME.

Notwithstanding the AC Coordinators' desire for a national marketing program, IC program management's experience with past media campaigns is that they have not been effective. Consequently, current efforts focus on networked and grassroots marketing approaches. At the national level, the ACCC focusses on promoting SC to industry associations, sector councils and business organizations (e.g., the Canadian Collision Industry Forum (CCIF), CARSTAR Automotive Canada, Chambers of Commerce, Canada Business Service Centres (CBSCs) and other intermediaries), and building relationships with them in order to develop contacts that can be exploited by the AC. For its part, IC has developed success stories and e-newsletters for the ACs' use, and has placed articles in some trade/business publications (e.g., Canadian Collision Magazine, BodyShop magazine, Enterprise Magazine).

At the AC level, coordinators have considerable freedom in how they market the program and how they capitalize on the support provided by ACCC and IC. Most of them use a variety of grassroots approaches, seeking "free" publicity when they can get it from the local press or media, and relying on networks of contacts obtained either personally, through the host institution, or via the ACCC's national contacts. Such networked marketing, particularly if preceded or supported by word of mouth referrals, can be very effective. But it is labour intensive: a lot of time has to be spent on the phone or visiting prospective clients." One AC has developed a telemarketing program that it has presented to the other AC. Several have been able to apply it to gain additional clients.

Coordinators in established centres are able to rely a network of contacts that they have built up over time to reach prospective clients and generate sales. In new centres, the coordinator's challenge is to find the time to first build the network, and then to exploit it to generate sales. There are limits to the extent to which either IC or the ACCC can help a coordinator with the first task — a coordinator simply has to devote the required time, something that is increasingly difficult to do under the program's current operating parameters. There is a risk that, given all the other parts of the AC Coordinator's job, which is now very large, coordinators in newer centres will not have the time required to develop the network of contacts required to generate the necessary revenues from training clients.

Data collected by the program suggest that this may be happening. The AC that were most successful in meeting or exceeding the benchmark for number of clients trained in 2004–2005 were, with one exception (that has since left the program) established before the change to the current operating parameters. Newer centres fell (and continue to fall) short.

Program management at IC clearly sees the ACCC's involvement in the delivery of the program as beneficial. This view is not shared by all AC, with 42% of those contacted expressing doubts about the value of its contribution. Failure to understand the "realities" of operating an AC, and stalled product development were the two most frequently expressed concerns. Established centres were more likely to be critical of the ACCC's contribution than new ones. In new centres, the ACCC was praised for its willingness and efforts to help AC Coordinators get though their startup difficulties.

As noted earlier, several coordinators stated that the program's name made it more difficult than necessary to sell the program to SME, who tend to initially see it as a program aimed exclusively at students, or who have doubts about the credibility of students as individuals who can provide instruction and advice that is relevant to their businesses. The program's name appears to be particularly troublesome in francophone regions, where there is an unwittingly negative colloquial connotation associated with the name. We were advised that "Bien branché" is a seen as a bit of a joke in Quebec because when you say to someone "branche-toi!" it is usually because you consider the person way off base.

Several of the established AC have overcome these problems, but in new ones they exacerbate the difficulties of launching the product and selling its merits to SME.

Turnover of AC is problematic. When an AC is replaced, there are requirements for transitional support from the ACCC to help the start-up. It usually takes several months for a new centre and coordinator to fully understand the program and to be able to deliver it independently of ongoing support from the ACCC.

While the ACCC is very "hands on" with new coordinators and AC, the time it spends getting them established is time it cannot spend on other activities such as national marketing and outreach to business associations. With high turnover AC, the ACCC risks facing a problem at a national level similar to that faced at local levels by AC. This is that it must spend so much time on addressing pressing current problems that it has insufficient time to spend on the activities necessary to ensure the program's future.

The ACCC is trying to address this problem by pairing new coordinators and AC with established ones so that the latter can mentor the former, through monthly conference calls where, amongst other things, best practices are discussed, and by fostering a sharing culture across the AC participating in the program.

Both the ACCC and some AC coordinators noted that there is, at present, insufficient sharing. One aspect of this concerns locally-developed training materials, whether extensions to existing SC products or new products.

The contract that the ACCC requires AC to sign may be contributing to centres' unwillingness to share such materials. Its section on proprietary rights not only retains for the ACCC all right, title and interest in all programs, systems, data and materials supplied by the ACCC to the AC, but also states that all right, title and interest in all programs, systems, data and materials supplied by the AC to the ACCC are the property of the ACCC. Some centres are unwilling to hand over their intellectual property on such a basis.

The fact that each contract terminates within 12 months of its being struck, contains a clause allowing the ACCC to terminate the contract if the AC is not, after 9 months, achieving a minimum of 50% on any of the required targets included in the contract, and applies the same targets to all AC, whether just joining the program or having been with it since inception may also encourage AC to view themselves as independent contractors rather than as "partners" in a national enterprise.

The 2004 SC Risk-based Audit Framework (RBAF) noted that SC delivers training in a highly volatile and rapidly changing industrial sector – ICT – and that ensuring that its products, services and support materials remained relevant to and valued by its clients in such an environment would be challenging. It therefore identified the risk that SC's training products and services would become outdated as a key risk that had to be managed.

The Contribution Agreement makes new product development the clear responsibility of the ACCC. It relies a combination of techniques to ensure relevance and quality in the products for which it is responsible. These include consultations with ACs about needs for new or enhanced products, sharing of product content amongst AC, contractual arrangements with AC to develop products, and in-kind arrangements with IT experts both within and outside the SC program both with respect to development/updating of products and with respect to quality assurance.

The ACCC stated that product development is expensive, and that, with the decline in private-sector sponsorship that it recently experienced, it does not have sufficient funds to renew products the way it did it in the past. For example, about 2 ½ years ago , the ACCC contracted renewal out to ACs. A new Internet Security module was created in 2004, the E-Business Blueprint, Web-site Planning and Web-site Evaluation modules were combined, and Online Promotion was updated to consolidate the original module with Selling on the Internet and Distributing e-Newsletters. The other modules in the E-Commerce First Step, Group Presentations and Basic Internet Training (for business and seniors), however, were not modified. As well, while there have been additional, more recent updates (e.g., an Online Search for Businesses module was created by the ACCC in March 2005), many coordinators stated that outdated products that were hard to sell was one of their greatest challenges (it was the third most frequently-cited problem among AC coordinators).

Both IC and the ACCC recognize that the success of the program depends a lot on the quality of its products and that SC's current suite of products needs to be updated. The ACCC has therefore developed a product development strategy that it hopes will address the AC Coordinators' concerns. Two key elements in the strategy are distributed product development among AC, and in-kind contributions from corporate partners who will provide quality assurance of the products developed at the AC.

The first part of the strategy is being implemented through the introduction of a new eLearning System from Thinkingcap. This system, which is replacing an outdated and expensive product training and SBA certification platform, has two products (Studio – for content creation, and Campus – for delivering eLearning courses to learners and for tracking learner and class performance). The ACCC intends to use these products to distribute content, refresh it, train SBA and, eventually, evaluate and certify the competence of SBA on the SC modules.

AC are being taught how to use Thinkingcap products now. In parallel, the current SC modules are being disaggregated into self-contained lessons of varying length (say from 15 minutes to two hours). These will be loaded into Studio and distributed (downloaded) to the AC. AC may use the lessons as they wish – they can mix and match them, extend them, add to them, customise them so that they best meet local requirements – all using Studio. Once they have done this, they can make the results available for use and reuse by the rest of the SC community by uploading them.

Because the AC are not subject matter experts in ICT, any material developed by them within Studio will have to be subjected to a quality assurance review process before it is made available for use by the SC community. Such a process is necessary to ensure that modules do not make unwarranted assumptions about a learner's knowledge, and to prevent errors and ambiguities in the training materials, such as those found in Web Site Planning. The ACCC does not have subject matter experts in ICT. It therefore hopes to secure contractual and/or in-kind support from faculty and corporate partners who will be willing to do this on its behalf.

The ACCC adopted this product development strategy, in part, because it was aware that some AC are developing or have developed products for their own use. The strategy seeks to capitalize on this phenomenon.

At this point, there are a number of risks with the strategy:

• First, AC must be willing and able to master and use the Thinking cap products. While the company's web-site promotes its products as easy to use, whether busy coordinators will find them so, and find the time to use them is unknown;
• There will be a need for ongoing technical support. According to Thinking cap, the ideal layout for content differs between how it should be presented for content developers and students. The product therefore allows for different templates, with those for students defined by a user interface designer. It is not clear that the ACCC's current Research, Development and Training Officer has the required expertise in user-interface design;
• It assumes that the proprietary rights issue can be solved and that AC will be willing to transfer their intellectual property to the ACCC;
• It assumes that corporate partners, institutions and faculty will be found to take on the time-consuming quality assurance function.

Any one of these risks could subvert the strategy, further delaying the much needed renewal of SC's products. In the absence of continuously renewed products, the program's primary clients, SME, may find SC's offerings of declining relevance to their needs. This would exacerbate the difficulties that ACs face generating the sales needed to overcome their deficits.

4.3 Performance Measurement Systems

The Contribution Agreement between IC and the ACCC calls for it to provide performance and status reports and summaries, with appropriate confirmation by AC, to SC Headquarters on a monthly and quarterly basis. There are also monthly conference calls and reports on request. As well, national performance expectations are established annually by IC in the first quarter of each fiscal year. Performance against these is tracked through monthly reporting, and results are reviewed at national meetings.

The contracts that the ACCC signs with the AC flow down to the AC all of the Contribution Agreement's reporting requirements. Each contract has a Reporting Protocol that describes the AC's reporting responsibilities in detail, including submission deadlines. The ACCC employs a full-time Finance and Contracts Officer to manage, monitor and support the reporting responsibilities of the AC. AC reported that if they miss a reporting deadline or if there is an error or omission in a submitted report, they are promptly contacted by the ACCC.

There are core targets and other measures, both of which are quantitative. The core targets are used to track overall progress, while the other measures enable SC management to assess the contribution of individual AC to SC. SC's core targets are the number of clients to be trained and number of youth hired as SBA each year. Other measures evolve as management assesses various aspects of AC contributions to SC. Recently, most of the other measures have been ratios. For example, SC now has a benchmark that a minimum of 35% of all clients should receive training on an E-commerce product and that each AC should generate a minimum of 40% sales revenue based on its total IC contribution.

In addition to tracking quantitative measures, at the end of each term, twice a year surveys are conducted by Prairie Research Associates (PRA) of youth who have served as interns in SC and of SME and seniors who have received ICT training through SC. The surveys are done through the ACCC and provided to IC.

Since 2004, the ACCC has instituted rotating program audits of AC that review their procedures, data quality, and compliance with government and program policies.. It tries to complete two each year. It would like to be able to complete more audits each year, and to conduct spot checks of AC activities to ensure that these continue to fall within the mandate of the program. With more resources, it would also undertake formal analyses of best practices to generate lessons learned that could be shared with all AC, as well as conducting in depth assessments among its clients of SC's products.

IC conducts recipient audits of AC as deemed necessary using a risk-based approach.

Under the term of the contract each AC signs with the ACCC, both the summary and detailed financial reports regarding monetary and in-kind contributions and the payment tracking reports submitted by AC must be signed off by an official senior financial signing authority at the host institution before payment is made.

In the Spring of 2005, an analysis was done of the surveys of SBA, SME and Seniors. Each survey's questions were mapped to:

• The Student Connections RMAF's Performance Measures, which were derived from the RMAF's Immediate, Intermediate or Final Outcomes;
• The RMAF's Outcomes as documented in its Logic Model

As well, the 2004 SC Evaluation Report was reviewed to determine which survey questions had figured prominently in that evaluation.

The analysis showed that there was considerable scope to refine the each survey by eliminating/replacing questions not directly related to collecting data to support planned evaluations of SC. Recommendations were made to eliminate several questions, as well as to refine the retained questions to ensure that they were tied to SC performance measures and outcomes. The resulting surveys, first used in September 2005, contained 25% less questions, as well as being better tied to planned evaluations of SC.

The key risks identified in the 2004 RBAF included:

• The AC selected by ACCC may not be able to satisfy the requirements of government program delivery (e.g., official languages policy, privacy policy, Federal Identity Program);
• The AC selected by ACCC may hire ineligible youth to be SBA;
• The AC selected by ACCC may not be able to attain the quantitative performance targets and benchmarks established by SC; and,
• SC ICT training products and services may become outdated, so that they are no longer relevant to or valued by SC's target clients.

The first risk is mitigated through the ongoing monitoring conducted by IC and the ACCC, and by using the national meetings to ensure that all AC are aware of the practical implications of the requirements.

Eligibility of the youth hired to be SBA is assured through Participant Information Forms that detail the criteria for participating in the program and that require a signature confirming that the youth meets the criteria.

The ability of AC to meet quantitative performance targets and benchmarks is monitored on a monthly, quarterly and annual basis though the SC Reporting Protocol. An AC's failure to meet a target or targets quickly becomes evident, and when it does, the ACCC intervenes directly with the centre to provide advice and assistance. Common problems with benchmarks are also discussed at monthly teleconferences. Recently, the ACCC has been pairing new and established centres so that mentoring can take place to help the new centres, which are usually the ones that have the greatest difficulty with respect to the targets and benchmarks.

If all of the above fails, the contract with the ACCC gives it the right to terminate the AC's contract if it is not, after 9 months, achieving a minimum of 50% on any of the contracts' required targets. We were advised by the ACCC that this clause is usually not invoked in an AC's first year of operation – its practice is to intervene as described earlier to help the AC to correct performance problems. Nevertheless, there has been turnover of AC (8 of the current 15 AC are were not part of SC when it was evaluated two years ago), both as a result of the ACCC not renewing contracts and of decisions by host institutions to withdraw from the program.

The risk that training products and services may become outdated was to be mitigated by having a product development committee that includes AC representation and that meets regularly, and by employing dedicated staff at ACCC who are responsible for product development. These measures have not mitigated the risk. They have confirmed the need to develop new products, but new product development has stalled, and the risk that training products and services may become outdated is now a problem. The ACCC's response to the problem has been discussed above, as have the risks it entails.

The new risk that has emerged stems from the restrictions on SBA hours and term of employment that have been imposed by the YES terms and conditions. IC advised us that YES renewal came very late in 2003 and provided very little flexibility, making it difficult to devise a business model that would provide AC the flexibility they had when SC funded them on the basis of clients trained.

In devising the current model, input from the AC has been considered and changes were made (originally every SBA was to be employed for a set number of hours and there was no flexibility to have some work more or less than this set number). Despite these changes, the constant staff turnover that is required by the YES terms and conditions imposes high administrative costs on the AC. To these costs must be added the costs of fulfilling the program's reporting requirements and the costs of marketing and sales. Most of these costs are direct-labour costs borne personally by the AC Coordinator, who now has a very large job. The problems that this creates for new AC, and the risks that these problems pose for SC as a national program, have been discussed above.

One way to deal with these problems would be to try to reduce the size of the coordinator's job. Reducing the reporting burden might be a part of this, as would ensuring that easier-to-sell new products are developed as quickly as possible. Revamping human resource management practices to reduce staff turnover would also be important. It may also be useful to examine whether a new AC should have the same targets as established AC, or should be expected to build up to them. Flexible SBA salary rates tied to local labour market conditions may also be worthwhile.

At this juncture, other than reducing reporting burden and introducing progressive targets and flexible salary rates, it is not clear that other actions to deal with SC's problems are within its control. It may not be possible to secure additional funding to offset the cost pressures on AC, and to accelerate product development. Securing greater flexibility in the terms and conditions governing youth employment in the program may have to wait until YES is renewed, two years from now.

5.0 Conclusions and Recommendations

SC is an unique initiative that consistently draws very positive responses from both its participants and its target beneficiaries. Because it is delivered by IC but funded by the YES, it must balance delivering practical, relevant ICT training to SME and seniors with providing youth with meaningful work experiences that enhanced their employability. Recent changes to the program’s operating parameters, combined with outdated training products and services are compromising the program’s ability to maintain that balance. They also pose a risk to the program’s ability to maintain its national scope. SC needs to take action to address these issues. It should:

• Develop a contingency plan for product renewal in the event that the ACCC's current strategy fails;
• Explore options for securing additional funding and getting greater flexibility in the terms and conditions governing youth employment in the program;
• Examine its reporting requirements to determine if the reporting burden on AC, which increased when SC began to be delivered through a contribution agreement, can be reduced;
• Explore options for easing the pressures new AC face when they join the program and must establish themselves in local markets for SC products and services.

Management Response to Recommendations of Formative Evaluation

The Student Connections Program (SC) has consistently met or exceeded its national targets and has fulfilled the requirements of both its own terms and conditions, as well as those of the Youth Employment Strategy (YES). A rich data set, combined with semi-annual surveys of clients and youth, facilitate performance measurement and reporting on an ongoing basis and allow for adjustments to certain program parameters. These data will also be beneficial in developing a summative evaluation in 2007–08.

Continuous discussions and open consultations amongst Industry Canada program staff, the contribution recipient (Association of Canadian Community Colleges or ACCC), and the 15 universities and colleges delivering the program (SC Administrative Centres or ACs) also ensure that any changes made can respond quickly to opportunities or issues raised. In particular, considerable progress has already been made since the Formative Evaluation report was written.

Given that information technology is such a fast-moving field, it is essential that the product modules used to train business and senior clients are continuously updated. ACCC therefore developed a product renewal strategy to leverage the extensive knowledge that exists within centres. In response to the Formative Evaluation, ACCC moved quickly to develop a complete plan to accelerate the process of updating product modules, in consultation with the ACs. All products will have been reviewed, updated and available in both official languages by August 2006. The plan will be monitored closely and will be supplemented by contracting for product updates, if necessary, as a contingency plan. ACCC also immediately rectified a clause on intellectual property which may have inhibited sharing of product by some ACs in their 2006–07 agreements with colleges and universities.

The product renewal strategy includes product development by individual ACs, supplemented by review and quality assurance by other ACs, faculty members outside the ACs and private sector partners and contractors, as necessary. Products are incorporated into a new content management system which will allow for continuous updating, collaboration and sharing of products amongst ACs. The content management system replaced an outdated and expensive training and certification platform; problems with the platform which had been encountered at the time of interviews for the evaluation have since been resolved. Measures are in place to monitor progress on development, which has been excellent to date, and a product development committee has been formed with ACs.

It should be noted that client surveys carried out two times per year by an independent firm consistently show that clients believe that the training they receive is practical, relevant, up to date and that the youth who deliver the training are capable. Despite the issues raised by ACs, many products (particularly those which are within the e-commerce suite of products and which assist in meeting IC's objectives with respect to SMEs) have been updated over the past two years. Six out of 7 E-Commerce First Step national products have been updated for use by all ACs. Innovation in product development is actively encouraged. Additional products or additions to existing modules have been developed by individual ACs, tested in one market and later become part of the national suite of products (e.g. Search Engine Optimization and On-Line Business Research modules added in the past two years).

Student Connections derives its funding entirely from the Youth Employment Strategy (YES, $2.9M G&C, $0.6M, O&M). At the time of the last YES renewal (2003–04 to 2007–08), additional terms and conditions were imposed that youth must return to studies following an internship, that internships must be short term and that contribution agreements should be no longer than one year. Little can be done immediately to change these parameters. However, in the event of YES renewal, recommendations will be made to the interdepartmental YES community for changes with respect to these and other parameters, as well as with respect to additional funding. Notwithstanding the YES requirements, the Student Connections terms and conditions and contribution agreement with ACCC, as well as the excellent collaborative relations with ACCC and with the ACs, do provide flexibility to make some changes in the program; these have already been made in agreements with ACs for 2006–07.

Additional funding would alleviate most concerns expressed in the Formative Evaluation without commensurate increases in overhead costs; such additional funding would allow for greater efficiencies, expanded regional outreach particularly to official language minority communities, and somewhat greater contribution to salaries of Coordinators at ACs. Already, revenues retained by ACs and in-kind contributions contribute another $1 to $1.5 million to the program and are used for hiring more youth, training more clients, local marketing and coordinator salaries and overhead. ACs have stated that they had no difficulty recruiting adequate numbers of youth; in 2004–05 there would have been demand for another 60 internships.

More rigorous performance and reporting measures were introduced in 2003–04 as part of the move to contribution agreements to ensure better accountability for monies received; these are not considered to be an unreasonable burden. Benefits which have been achieved are clear audit trails, clear adherence to the spirit of the program's objectives, engaged senior management in the academic institutions delivering the program, and an ACCC-instituted rotation of audit and evaluation activities. These are considered to be a very positive factor for program design, even though it posed challenges for the ACs.

ACCC is however exploring with ACs how very specific individual reports they currently require could be streamlined to reduce the time required by ACs for reporting. Other reporting requirements imposed by an AC's host institution may also add to the reporting burden but cannot be influenced by ACCC.

It should be noted that the measures instituted in 2003 did result in the loss of some ACs which may have been unwilling or unable to subject themselves to a more rigorous approach and more rigorous financial accountability. ACCC has had little difficulty in replacing these ACs with new ones, emphasizing to senior management at host institutions that this is a contribution program, and that other valuable benefits derive from it, including greater involvement with their local business community and relevant work experience gained by their students.

ACCC has already moved to assist new ACs to ease pressures. These include formal orientation sessions for new coordinators, mentoring by Coordinators in established centres, rigorous business planning and evaluation of plans and advice by ACCC; and assistance in marketing. ACCC also place particular emphasis on appropriate placement of Student Connections within the host institution, having found from experience that placement in a business-oriented environment produces the best success. Activities around celebration of the 10 year anniversary of the program should also help to build awareness and attract revenue-producing clients.

Early indications are that these measures are resulting in quick establishment of centres. For example, two newer centres established in 2005–06 exceeded their client targets (La Cité Collégiale in Ottawa by 129%, Memorial University in Newfoundland and Labrador by 106%). Another centre established only in January 2006 (at University of Alberta in Edmonton) had by mid-March already hired all their interns and established key marketing initiatives including seminars at an Open House and e-newsletters, which resulted in numerous client leads.