Formative Evaluation of the Canadian Apparel and Textile Industries Program (CATIP) — Textile Production Efficiency Component (TPEC/CANtex)

3.0 Evaluation Methodology

This section of the report sets out the methodology used for the Formative Evaluation. The methodological approach used reflected the standards of Program Evaluation and the expectations set out in Canada's Treasury Board Policy on Evaluation.

The Evaluation was guided by Industry Canada's Evaluation Plan and was based on the contents of the CANtex Results-Based Management Accountability Framework. As is the standard practice of Industry Canada's Evaluation function, a Steering Committee was established and included representatives from the industry and the Department. The list of Steering Committee members is included as Appendix A.

3.1 Evaluation Issues

The selection of Evaluation issues was guided by the TPEC/CANtex Formative Evaluation Plan developed in 2006. The Evaluation addressed the following areas:

• Rationale — whether the design of the program and its activities are sound and whether the program is meeting the needs of the industry targeted,
• Program Design/Delivery — whether the current approach to implementing the program is appropriate, consistent and effective,
• Progress on results to date and whether the program is likely to meet expected results, and
• Lessons Learned — focusing on best practices to identify factors that lead to effective delivery and the types of projects most likely to yield good results.

The list of Evaluation questions is attached as Appendix B.

3.2 Evaluation Design and Methodology

The CANtex Formative Evaluation Plan provided the main design elements of the Evaluation methodologies. The methodologies and research instruments were developed in collaboration with representatives at Industry Canada to ensure they reflected the program and project characteristics.

As required by Evaluation standards, the methodology for this study included multiple lines of inquiry. These included the review of documents, interviews with multiple stakeholder groups, survey of program recipients, case studies, and horizontal analysis across all lines of inquiry. Also, as part of a study on a separate component of CATIP, independent of this Formative Evaluation, interviews were conducted with national textile and apparel associations. This provided another source of information regarding the delivery and design of the program that has been included in this Formative Evaluation. Details of these methods are provided below.

Review of Documentation

CANtex program activities generated several types of documents that contributed to the Evaluation research. These included:

• Treasury Board Submissions for CATIP and the subsequent two CANtex program elements,
• Terms and conditions of the program,
• The application form,
• The contribution agreement,
• Files outlining the projects conducted by recipient companies,
• Files outlining rejected projects,
• Quantitative summaries of projects' characteristics,
• Performance data collected by the program, and
• General correspondence.

These documents were reviewed for information relevant to the implementation of the CANtex program. Where possible, additional context on the information contained in the documentation was obtained during interviews.

Interviews

Interviews were conducted with representatives from Industry Canada, CED-Q, recipient companies, and companies identified as having rejected projects. The interviews followed a structured process using an interview guideline. Questions were open-ended and the interviews were interactive, focusing on the interviewees' particular knowledge and perspective of CANtex. In all, 47 individuals were interviewed. The interview guidelines are included in Appendix C.

The interview sample was developed at the outset of the Evaluation, based on program data from July 2006. As a result, the universe of recipients and projects used for sampling reflects the population as of July 2006 (although the population has increased as the program continued since the time the sample was developed). Furthermore, the statistical results presented in Section 4.0 of this report are based on the recipient and project population as of July 2006, even though the views expressed by interview respondents reflect their experience up to the time of their interview.

Interviews with representatives from Industry Canada and CED-Q

These interviewees were identified by CANtex program personnel. In all, 12 persons were interviewed. They are listed in Appendix D.

Interviews with Recipient Companies

Documents provided by officials from Industry Canada and CED-Q indicated that, at the time of planning the Evaluation, there was a total of 200 CANtex projects involving 135 companies. A sample of 49 projects from 49 separate companies was identified, and the principal CANtex contact from each of these companies was selected as an interviewee. The sample took into account the size of the project, the project type, and the company's geographic location. Also, of the 49 recipient companies selected, 10 of them also had a project that was identified as "rejected" for the program by either Industry Canada or CED-Q. Representatives from those 49 companies were interviewed about their accepted projects, and, where applicable, about their rejected projects.

To encourage response from the sample companies, they were contacted five times, as follows: representatives from the selected companies were contacted by email three times requesting an interview (twice by the Evaluation team and once by the IC or CED-Q program officer), and, if they had not yet responded to the request for interview, they were contacted twice by telephone by the Evaluation team. Of the sample of 49 companies, 28 responded to the request for an interview and were interviewed. External interviewees are listed in Appendix D.

To provide an understanding of the coverage of these interviews, the tables below present a breakdown of the total projects undertaken through CANtex (i.e., the universe of projects as of July 2006), the sample of recipient projects selected, and the actual projects that were covered by interviews. The breakdown is by project size, project type, and geographic location of the company.

Table 1

	Project Size
	Under $30K	Over $30K to $50K	Over $50K to $100K	Over $100K
Universe of CANtex Projects	21	29	139	11
Number of Those Projects in the Sample	11	10	24	4
Number of Projects in Actual Interviews	4	7	14	3

Table 2

	Project Type
	Management System / Automation	People Focused	Process Improvement
Universe of CANtex Projects	137	2	61
Number of Those Projects in the Sample	35	2	12
Number of Projects in Actual Interviews	17	2	9

Table 3

	Geographic Location of Company
	B.C.	Prairies	Ontario	Quebec	Maritimes
Universe of CANtex Projects	6	4	89	98	3
Number of Those Projects in the Sample	4	4	19	19	3
Number of Projects in Actual Interviews	2	2	12	11	1

The distribution illustrated in the above tables indicates that the Evaluation findings provide a good coverage of these different characteristics of CANtex recipients.

Although the 28 companies were interviewed about specific projects, collectively the companies have undertaken 55 of the 200 total CANtex projects (as of July 2006), and they were asked their views on the design and delivery of the program through all of these projects.

Also, of the 10 recipient companies selected that also had projects rejected, 8 were interviewed. As a result, the overall interviews provide a good coverage of the CANtex recipient companies, and a good overall coverage of the CANtex projects.

Interviews with Rejected Companies

Documents provided by officials from Industry Canada and CED-Q indicated that there has been 44 projects from 39 companies rejected by the program. In addition to the 10 recipient companies that also had a rejected project (described above), a sample of 10 rejected projects from 10 other (non-recipient) companies was selected. Of those 10 rejected companies selected, none also had a project accepted by the CANtex program. A representative from each of these 10 companies was selected to be interviewed.

Representatives from the selected rejected companies were contacted by e-mail once requesting for an interview, and, if they had not yet responded, were contacted twice by telephone. Of the 10 selected rejected companies, 5 responded to the request for an interview and were interviewed.

The response from the selected rejected companies was very slow at the beginning of the Evaluation process, and it seemed that the percentage of respondents would be low. In response, representatives from an additional 3 companies identified as having a rejected project and having no accepted projects were also selected for an interview. These representatives were contacted twice by telephone requesting for an interview. Of these 3 additional representatives from rejected companies, 2 responded to the request for an interview and were interviewed. This brought the total number of rejected companies contacted to 13 and the total number of rejected companies interviewed to 7.

Therefore, of the 23 companies identified as having a rejected project (10 of these also had an accepted project, and 13 had only a rejected project), interviews were held with 15. These interviewees are also listed in Appendix D.

Survey

All recipient companies who were not selected for an interview were asked to complete a survey. Of the 135 companies identified in July 2006, 86 were asked to complete a survey.

The survey was developed as an internet site and delivered to the companies by a link embedded in an email message. Each company was contacted twice by email requesting they complete the survey. Of the 86 companies asked to complete the survey, 25 companies completed the survey. The survey is included in Appendix E.

Case Studies of Projects

To complement the above methodologies and provide a greater understanding of CANtex projects, 3 case studies were conducted. The selection of projects attempted to focus a range of the types of activities undertaken within CANtex projects. The case studies that were ultimately completed addressed a number of different vehicles used by recipients to increase productivity. The completed case studies are included as Appendix F.

Analysis

The results from the review of documentation, interviews, survey results, and case studies were examined and synthesized to determine conclusions regarding the issues addressed by this Formative Evaluation. This horizontal analysis enabled a compilation of qualitative and quantitative information.

Level of Confidence

Overall, the research methods used in this Formative Evaluation provides a good level of confidence that the conclusions reached are valid. The Evaluation strived to conduct interviews with a sample of all stakeholders as well as representatives from companies who were rejected by the program.

As indicated later in this report, the common views expressed by informants suggest consensus around what the major issues were and their experiences.