Evaluation of the Policy on Title to Intellectual Property Arising Under Crown Procurement Contracts

3.0 Findings (Continued)

3.2 Performance (Effectiveness)

Effectiveness is the extent to which a program (or policy implementation) is achieving expected outcomes. In examining the Policy's effectiveness, three evaluation questions were considered:

• Is the Policy being implemented as intended (according to the Underlying Principles section of the Implementation Guide)?
• How have the departments selected for this evaluation implemented the Policy and what are the opportunities for improvement?
• What are the outcomes of the Policy?

3.2.1: Is the Policy being implemented as intended?

The expectations for implementation of the Policy were compared with the day-to-day implementation in departments. Expectations concerning the monitoring of the Policy by IC and TBS were also reviewed.

3.2.1.1 Implementation by Departments

Intended Implementation

The intention of the Policy was that all departments would apply the Policy in their procurement activities and modify their reporting systems. The following specific requirements were found in the Policy:

• Departments were to include a reference to IP ownership in RFPs and sole source procurements;
• Contracting authorities were responsible for ensuring that their standard clauses reflected the requirements of the Policy;
• Deputy Heads were expected to ensure sufficient resources for training and the orientation of personnel and to ensure that reporting requirements were implemented; and
• Departments were expected to modify their reporting systems to capture IP information.

Observed Implementation

The following areas were examined when assessing whether the Policy was being implemented as intended in departments:

• General application of the Policy including reporting, the consideration of IP in the contracting process, and departmental guidance;
• Appropriateness of the use of exceptions; and
• Awareness of the Policy and training.

General Application

From the GWPRS IP Reports, it can be seen that all departments do report on their procurement contracts over $25,000. Their ability to report also indicates that departments have modified their reporting systems to capture IP information which they now submit to PWGSC.

The evidence suggests that IP was considered in all departments that were studied in detail in this evaluation. Two departments had specific procedures in place to provide a specialist review of IP ownership within the contract or RFP preparation activities. One of these procedures was used throughout the entire department, whereas the other was used locally in one organization within the department. In all other situations, the IP determination was prescribed as part of the routine discussions that occur between the project authority and the contracting authority. However, there is no guarantee that this happened in all cases or that these discussions were informed by the spirit and content of the Policy.

Departmental guidance mostly consisted of copying portions of the Implementation Guide or extracting content and developing checklists or components of departmental training materials. There was no indication that any departments suffered from any delays in the development of their departmental guidance although the Case Studies and interviews did not focus on the early years of the new Policy. The Implementation Guide was complete in that it addressed all clauses of the Policy, and included sample contract clauses and frequently asked questions.

Appropriateness of Use of Exceptions

All contractor and academia interviewees felt the exceptions in the Policy were comprehensive and appropriate. However, as the interviewees deal with multiple departments, they noted that not all departments applied the Policy in a consistent manner.

In addition, one contractor expressed the strong belief that procurement officials are not implementing the Policy in its intended spirit. He believed that because the assignment of IP rights to the contractor might present some risks, contracting officials and project authorities have often retained Crown ownership as the safest course of action. The contractor also believed that the rigor in challenging the use of exceptions has been insufficient. This view was supported by one high tech industry association and two interviewees in academia.

The survey of the procurement community confirmed that differences of opinion between contractors and departments have occurred. About 12% of the survey respondents (49 of 412) had experienced a situation where the contractor disagreed with the decision to use an exception and retain ownership of the IP.

Overall, procurement officials believed that the list of exceptions in the Policy was comprehensive. In interviews, departmental officials could not think of any instances where there had not been a suitable exception that would meet their needs.

The Use of Exceptions at the Aggregate Level

According to the data in the GWPRS IP Report, the majority of contracts across the Government of Canada that were expected to generate IP did not actually vest the ownership of the IP with the contractor. In 2008, the Crown retained IP ownership in 53% of these kinds of contracts. In 2009, the figure rose to 59%.

There are two reasons for considering the GWPRS data to be indicative of levels of compliance with the Policy rather than an authoritative source for assessing compliance.

1. Data Entry Errors. There is some evidence that the data being entered into GWPRS may not reflect the actual decisions made by procurement officers.

   In 2010, IC conducted a review of the 17 contracts over $25K in 2008 in which it had claimed a Crown ownership exception. In 18% of these contracts (3/17), the review found data entry errors. None of these errors posed a risk to the Crown.

   The Case Studies found that the figures reported in GWPRS did not match the information in the actual contracts in 22% of the cases. While the number of cases of over-counting of Crown ownership was about equal to the number of cases of under-counting, it is possible that GWPRS data could have overstated the number of Crown ownership exceptions in some departments.

2. Appropriate Use of Exceptions May Generate High IP Retention Rates. While the intent of the Policy is to have IP ownership vest in a majority of cases with the contractor, there is the possibility that a proper application of exceptions to protect Crown interests could, in theory, result in the Crown retaining the IP in a majority of cases.

   Unfortunately, the appropriateness of the use of exceptions in individual contracts could not be evaluated because detailed documentation justifying IP ownership decisions was absent in all contract files that were reviewed for the departmental Case Studies. The lack of supporting documentation meant that it was impossible to review or validate the reasoning behind IP ownership decisions.

The Use of Exceptions at the Departmental Level

A more detailed analysis of the use of exceptions revealed a wide range of IP retention rates across the federal government. Appendix J contains calculations for the use of exceptions among departments with more than ten IP-generating contracts in 2008 and 2009.

According to the 2009 GWPRS data, there were six departments that had retention rates higher than 90%. These findings are a concern in terms of adherence to the Policy. If these six departments are excluded from aggregate totals, then the aggregate IP retention rate for the Government of Canada drops from 59% to 45%. In other words, IP developed in the course of a Crown procurement contract would, in most cases, vest with the contractor in accordance with the intent of the Policy.

Exhibit 7: Departments with Higher Use of IP Exceptions in 2009 — 2009 GWPRS Data

Federal Organization	Total Number of IP Generating Contracts	Total Crown Ownership Exceptions	Percent of IP Contracts with Exceptions (i.e. IP retained by The Crown)
Canada School of Public Service	45	45	100%
Western Economic Diversification Canada	10	10	100%
Office of the Chief Electoral Officer	83	82	99%
Office of the Superintendent of Financial Institutions	61	60	98%
Indian and Northern Affairs Canada	416	404	97%
Correctional Service of Canada	400	362	91%

top of page

A similar pattern holds for the 2008 GWPRS data. There were three departments that invoked Crown ownership exceptions in more than 90% of their IP related contracts. If these three departments of concern are excluded from Government of Canada totals, then the aggregate IP retention rate for the federal government drops from 51% to 41%.

Exhibit 8: Departments with Higher Use of IP Exceptions in 2008 — 2008 GWPRS Data

Federal Organization	Total Number of IP Generating Contracts	Total Crown Ownership Exceptions	Percent of IP Contracts with Exceptions (i.e. IP retained by The Crown)
Correctional Service of Canada	546	528	97%
Canada School of Public Service	136	128	94%
Western Economic Diversification Canada	15	14	93%

It is important to note that, in some cases, the use of an exception may have been appropriate. For example, about half of the exceptions reported by the Canada School of Public Service concerned copyright. Given the large volume of training materials that are developed, these exceptions may be justified. However, the high rates of Crown ownership are a source for concern. (Appendix J includes the detailed breakdown of which exceptions were used by the six departments in 2009.)

Awareness and Training

Awareness was high within the procurement community as demonstrated by the fact that 79% of survey respondents reported that they were aware of the Policy.

About 57% of respondents reported having taken training on procurement that included a section on the Policy. Their training was mostly from either the Canada School of Public Service or their department.

Despite the high self-reported levels of awareness, the Case Studies found data errors and misapplications of the exceptions. This suggested that the Policy was not always well understood. In particular, there were contracts which included information subject to privacy restrictions (and hence exempt from the Policy), cases where department exceptions were misapplied, and contracts where national security was invoked when it was difficult to believe that national security was an issue. Despite the fact that the Policy explicitly requires written confirmation from the contractor for the use of one particular exception, no such documentation was found in any files studied where this exception had been claimed. These errors and omissions suggest that although training has been available and has been given, issues remain.

The results of the survey support these findings. About 42% of the respondents reported that they had problems at some point in implementing the Policy. About 17% also said they had experienced problems understanding if the exception had been applied properly.

Finally, the interviews also suggested additional training and clarification may be required. Two procurement officers expressed a concern that defaulting ownership of IP to contractors could place departments in a position where they might be required to contract follow-on work to the original contractor because that contractor would own the associated IP. However, these concerns seem to indicate an incomplete understanding of the licenses that the Crown retains when granting IP to the contractor. Policy authorities noted that Exception 6.4.3 in the Policy protects the Crown from this occurring. In addition, they noted that the Standard Clauses provide usage rights to the Crown so that a department would not be restricted to the original contractor for future work.

3.2.1.2 Monitoring of the Policy by IC and TBS

Intended Implementation

IC and TBS were responsible for monitoring the application of the Policy and were expected to focus on cases where exceptions were invoked (Section 11.1). It can be inferred that this monitoring role should have examined why the exceptions were being invoked in order to ensure that the spirit of the Policy was respected.

Observed Implementation

Both IC and TBS have individuals whose responsibilities include the Policy. Both IC and TBS have received the GWPRS IP Report annually. However, from 2000 until recently, there was no evidence of proactive follow-up on these reports. There was no evidence of discussions with the Heads of Procurement regarding the reported use of Treasury Board exemptions or discussions with departments with high IP retention rates to determine whether the reported numbers were the result of erroneous data, the nature of the work, or a failure to comply with the Policy.

On the positive side, both IC and TBS have, at times, played an active role in managing the Policy. The Baseline Assessment identified data issues that occurred during the first four years of implementation and data collection processes were improved.¹⁰ In 2010, IC conducted a review of its own contracting files where a Crown IP ownership exception was used, and encouraged other departments to do the same. Similarly, IC shared the results of the 2009 GWPRS IP Report with the Assistant Deputy Minister Committee on Science and Technology, and TBS shared the findings with the Treasury Board Advisory Committee on Contracts in order to build awareness and encourage compliance.

Overall, the question of roles and responsibilities is poorly defined. The Policy made both IC and TBS responsible for monitoring and evaluation without identifying specific roles. In interviews, it was observed that both departments felt that the other needed to take a stronger role in monitoring the Policy.

Despite these challenges, the Implementation Guide and Contract Policy Notices were developed as required and e-mail records indicate that the effort has been collaborative with IC assuming the lead in content for training materials. As issues have emerged (e.g. the OAG Report), IC and TBS have worked together.

top of page

3.2.2: How have the departments selected for this evaluation implemented the Policy and what are the opportunities for improvement?

To identify opportunities for improvement, the evaluation team examined how the Policy has been implemented in selected departments. The evaluation team examined departmental policies and procedures, education and training materials, awareness activities, and contract files.

General Observations on the Implementation of the Policy in Departments

The Heads of Procurement in the four departments were well aware of the Policy. However, only one Head of Procurement identified the Policy as a high priority or concern. In one department, the Procurement Review Committee considered the IP assignment decision in its deliberations and there was an IP Office located within its contracting organization in order to ensure the consistent application of the Policy for all contracts over $25K. While one department had conducted an internal management review of its use of Crown IP exceptions, none of the departments had conducted formal audits or evaluations of its implementation of the Policy.

In terms of processes, departments ensured consistency through the use of standard IP clauses in contracts. Most departments used standard clauses that had been developed by PWGSC and departmental contracting templates that reflected the Policy. All four departments used checklists that included IP questions to be addressed with Project Authorities. These checklists were then used by the contracting authority to select the appropriate clause in the final contract. Of the four departments, only one department had a review of IP ownership decisions in its processes.

The Policy was mentioned in departmental procurement manuals and some documentation was available to support contracting authorities. In two departments, IP and the Policy occupied only two pages in a 160 page overview document on procurement. The Intranets for two of the departments studied had IP information available to assist project authorities. One department developed a full binder of material exclusively on the Policy and gave the workshop in 2001. This material was later used to prepare and periodically deliver a shorter training session on the Policy. In general, departments seemed to find the Implementation Guide and the Policy document self-explanatory so that a mention of its existence and links to the TBS webpage were deemed sufficient.

In general, the level of understanding of the Policy varied by department. From the interviews, it was determined that departments felt that additional clarification was required on certain types of contracts such as software IP and training materials. Differing opinions also existed on the limitations of Crown use when the contractor retained ownership. Some interviewees expressed the concern that exceptions were used too frequently and when not needed because departments did not fully understand the rights that licensing provides to the Crown.

In terms of future training, all departments will eventually have access to an online course on the Policy that is being developed by the Canada School of Public Service. Those that have seen the prototype of the course believe that it will be a useful tool once certain edits are made to the content. Two departments are trying to get additional training for staff.

Some interviewees also felt that their departments would benefit from a central IP resource for the Government of Canada to answer in-depth questions. This comment echoed comments received from survey participants when asked what could be done to improve implementation.

In the course of conducting the Case Studies, the evaluation team noted several specific aspects of the Policy that could lead to misunderstandings or inconsistent applications. They include the treatment of foreign contractors, personal information that falls under the Privacy Act, the definition of national security, suggested documentation for cases where a contractor is not interested in owning the IP, licensing and copyright, the treatment of IP that will be transferred in the future, copyright, and Treasury Board exemptions. These specific points and suggestions for clarification for the Implementation Guide can be found in Appendix K.

Challenges Concerning Discrepancies between Centrally Reported Data and Contract Files

The examination of how departments have implemented the Policy revealed noticeable differences between the data reported centrally in the GWPRS IP Report and the information in the contract files. The following table outlines the errors observed in a review of 65 contracts across three departments.¹¹

Exhibit 9: Contract Information and GWPRS Data

Dept	Total Number of Contracts Reviewed	Number of Cases where GWPRS Data Did Not Match Contracts	Error Rate
A	23	8	35%
C	20	1	5%
D	22	5	23%
Total	65	14	22%

Overall, there were differences between the contract files and the information in GWPRS in about 22% of all contracts. In some cases, the exception that appeared in GWPRS differed from the exception in the contract file. In other cases, the difference was in the code that identified whether an exception had been invoked. Overall, the number of cases of over-counting of Crown ownership in GWPRS was about equal to the number of cases of under-counting.

It is important to recognize that there is not always a negative impact on either the department or the contractor if a contract does not match what is coded in GWPRS. This is particularly true when the wrong exception code is used. In these instances, the real impact is only in the accuracy of the GWPRS IP Reports. To be clear, it is the actual text of the contract that matters in terms of whether the Crown or a contractor owns the IP, not the coding in GWPRS.

Crown IP ownership exceptions were typically recorded before any contract tendering took place. In most departments, the decision was recorded by entering a code into a contracting information system (such as the Integrated Financial and Materiel System at IC). The contracting authority then took note of this in order to insert a Crown IP Ownership clause into the contract. This meant that the system for entering the IP ownership information that eventually flowed into GWPRS IP Report was separate from the actual contracting documentation.

This duplicate entry process created the potential for a clerk to enter the code for an exception when one was not intended (i.e., by mistake) or a different exception than intended (e.g. 6.3 rather than 6.2), or even fail to enter the exception at all. At IC, the failure to enter an exception code would generate a default contractor owned IP condition in the contract.

Challenges Concerning Potentially Inappropriate Applications of the Policy

A review of 95 contracts found that 13% of the contracts exhibited questionable uses of Crown IP ownership exceptions. An inappropriate retention of Crown ownership would constitute a barrier to commercialization. The following table outlines the potential errors observed across the four departments.

Exhibit 10: Analysis of Case Study Contracts

Department	Contracts Reviewed	Total Contracts of Concern
A	23	4	17%
B	30	7	23%
C	20	0	0%
D	22	1	5%
Total	95	12	13%

The 12 contracts of concern included:

• Two contracts that claimed a national security exception when there was no evidence of any national security issue in the contract;
• Two contracts that claimed an exception for generating knowledge for public dissemination but the nature of the reports was clearly intended for internal management purposes;
• Three contracts that claimed that the contractor had declined ownership of the IP. While the Policy clearly stipulates the contractor must declare this in writing, there were no records documenting that this had occurred; and
• Five contracts that did not identify which exception was being claimed.

The Case Studies also examined seven contracts in greater detail and interviewed key participants. In general, the use of exceptions in these contracts was appropriate and reasonable. However, there were decisions that were subject to interpretation:

• There was one example where similar work within the same department resulted in two different decisions being taken. In both instances, a contract was let for the use of computer programming support under very similar situations. In one, it was identified that there was no IP. In the other, IP was identified and an exception was invoked because the IP formed part of a larger computer system where the Crown already owned portions of the IP.
• In another instance, a contract identified that there was no IP created when the contract called for delivery of architectural drawings, documents and designs.

What was striking was the absence of supporting documentation for IP ownership decisions in all of the contracts reviewed. Among the four departments, not a single contract template asked for the rationale for why an exception was chosen. While one department included a review of the IP ownership decision as part of its contracting process, the other three departments simply noted the exception had been used. These practices are in sharp contrast to the procedures required for sole source contracts where a project authority is required to write a justification.

Having a justification on file would allow an independent reviewer to validate the appropriate use of the exception and encourage adherence to the Policy. It would allow managers to review staff work and identify root causes for the excessive use of an exception. As a result, departmental guidance could be improved to prevent reoccurrences and staff could be trained on the use of specific exceptions.

---

¹⁰ Baseline Assessment Report, 2004. (Return to reference 10)

¹¹ Once the sample of contracts was created for the Case Studies, the evaluation team initiated the process of matching these contracts with GWPRS data. Unfortunately, GWPRS does not retain contract numbers so significant effort was involved to complete this task. Given time constraints, contracts could not be matched to GWPRS records for one department. (Return to reference 11)