In previous blogs, we discussed the importance of recall and precision for the Search Quality Index (SQI). We also explained how expected recall and precision values vary for different search use cases and are not directly related to each other. In our FTO series, we emphasized the importance of scoping and communication (FTO risk matrix).
In this blog post, we’ll analyze the most common errors in freedom to operate patent searches and why even highly experienced patent analysts often miss important documents in high-recall searches. In addition, we’ll examine how similar situations are managed in other zero-tolerance industries such as aviation, hospitals and others. Then – next time – we’ll discuss how the Evalueserve team uses the Mind+Machine concept to address this challenge.
Root Cause Analysis first! What is a bad search (e.g. FTO) and what kind of errors do we see?
Let’s look at some of the potential errors that can occur in the most complex patent search: a freedom to operate (or clearance) search. These searches require a particularly high recall (identifying all critical patents), and can be difficult to scope.
Ten potential errors in an FTO search:
- Searcher, R&D associate and/or patent attorney have a poor or incorrect understanding of the subject matter or drop important details in communication between them
- Selecting inappropriate retrieval systems (such as not choosing the ideal database for the use case)
- Basic mistakes in query building (wrong terminology, flawed usage of indices)
- Too aggressive (too narrow) query building
- Retrieval system errors
- Searcher draws the wrong conclusion during review (false negative) ignoring relevant patents
- Searcher is biased based on previous experience
- Missing specific types of documents due to “incomplete scoping” or ”blind spots”
- Complexity of forming the final legal opinion
As this list shows, searchers face a range of potential pitfalls from communication and handover issues to knowledge gaps, interaction issues with machines, human errors and more.
High-risk processes in zero tolerance industries
Aviation and hospitals are two industries where errors are unacceptable as human lives are at stake. One of the first lessons from these industries is that quality control does not rest on one individual pilot, mechanic, nurse or doctor. In these industries, the best practice is to set up fault-tolerant processes to ensure that system or process continues to operate properly in the event of the failure of some (one or more faults within) of its components.
For example, some key elements of these processes include:
- A properly-mapped process considering failure risks and a clear shared understanding of the overall process and sub-processes
- Systems redundancy – a back-up in case one system fails
- Checklists and documentation
- A key focus on the human-to-machine interface and handover challenges
- (Confidential) fault-reporting system in the interest of improving reliability
- Agile learning from errors
We all expect our surgeons and pilots to conduct proper pre-surgery or pre-flight briefings, going through formal checklists. We want flight computers to support the pilots and technology such as image recognition and medical-algorithm software during a diagnostics process to support doctors. We’d all feel uncomfortable having an operation done by a single surgeon or being flown by a single pilot. We certainly expect errors to be documented, and that airlines and hospitals learn from these mistakes.
Lesson summary and outlook
The high-recall patent search process is complex. Errors can happen at any stage. Some mistakes result from communication issues, or from the searcher’s lack of expertise, and some errors occur if patent searchers act under pressure, forcing them to take costly risks.
From these lessons, Evalueserve ensures high-quality patent searches by focusing on the following principles:
- Training and back-up: Experts must have rigorous training and experience, and also ensure there is a back-up process to catch mistakes, flag bias and support knowledge growth. Human errors must be factored in, a lesson we borrow from other high-risk processes.
- Detailed handover process: Issues that occur during the hand-over process from the attorney to searcher and back – at the beginning and end of the search – are often the source of many errors. The remedy is formal, documented processes.
- Learning from mistakes: Every error should be reported, analyzed and should be used to improve the process
- Mind+Machine collaboration: Deep thinking about how people and machines interface and collaborate will result in better processes and greater control. In some cases (and this is very important) the human experts control the machine, and in other cases heavy workload is taken away from the expert and the machine supports the process as required.
In our next blog post in the series, we’ll discuss how the Evalueserve team uses an advanced process to support the experienced patent search and Innovation Intelligence analysts in all process steps – the searchers "exoskeleton.”