Six months ago we launched the Information Adventurers Blog sharing thought-provoking and insightful pieces on the intersection between data science and the information needs of IP and R&D teams. Thank you for joining us on our quest for search quality!
So it seems like the perfect time to summarize some of the highpoints of our first summer season, and comment on what others are saying on the topic.
Is it acceptable that our industry doesn’t measure patent search and patent landscaping quality?
While great work is being done on evaluating the quality of patent databases or information retrieval systems (IRS), there is surprisingly little work published on the quality of patent search – the combination of the performance of the IRS with that of human experts. (Tip: In my opinion, the best source in this field is the recently updated ‘Current Challenges in Patent Information Retrieval‘ from Lupu et al.)
In some of our blog posts we discussed market strategies, from ‘parallel searching‘ to relying on an ‘experienced searcher‘. Patent search quality is critical for high patent quality delivered by patent offices, but here examples are more often than not either post-mortem benchmarking (e.g. UK patent office) or a focus on various process improvements and audits (e.g. at the EPO). These initiatives are great, but aren’t actually useful in evaluating individual-search quality.
What else have we seen over the summer about patent search quality?
The Patent Olympiad in Milan – similar to the long-standing Japanese event – was an interesting addition to the industry in Europe for awareness raising and training. However, it doesn’t provide any useful tools to end customers. A glance at the PIUG annual event program also quickly shows more of a focus on performance and optimal usage of IRS than on search outcomes. In our opinion, relying blindly on Trust isn’t an acceptable basis for assessing patent search or patent landscape.
Using the Search Quality Index to understand the economics of patent search and patent analytics.
We believe change is needed! So in July, we introduced our Search Quality Index (SQI) and its derivatives (IQI) Intelligence Quality Index and Alert Quality Index (AQI). We discussed how it links to economics – it is obvious that some searches have a market value of < 100 USD (these are typically done by tools), while other searches have market prices of 500 USD, 1500 USD up to several 10k of USD, so choice of approach is defined by the financial and legal risks involved.
The Search Quality Indices (SQIs) were developed based on some key metrics in general information science – in essence it is about recall (% of relevant documents found), precision (% of noise), and insights (a balanced scorecard measuring and targeting various factors).
Using SQIs to determine use cases for machines/AI or for the expert searcher.
2018 is the year of major investment into IP intelligence platforms: Patsnap getting additional funding and Lexis Nexis picking up Patent Sight are just the tip of the iceberg. Artificial Intelligence or AI-enabled tools have captured not only investor money but have also extended the range of potential use cases.
So, it’s vital for decision-makers to clearly determine which use cases are suitable for tools, and which require an expert searcher, and the SQIs are a helpful way to make this decision. We’re currently discussing this question for the various use cases around patent watch/IPR&D Alerts and will investigate not only ‘machine versus analysts’ but also optimal combinations for deriving value.
How to use SQIs to improve and monitor patent search or patent analytics quality.
The SQIs can be influenced and monitored at various steps of the process. This requires: the right communication between end user and searcher (briefing); using and combining the right sources and appropriate queries for retrieval (keeping economics in mind); taking correct decisions when screening through the initial hits; AI-enabled secondary and tertiary searches to retrieve hits outside the query universe; and appropriate, user-friendly and precise deliverables.
We’ve started to look into the details of this process and so far have analysed the elements of search query generation and search query strategies, considered tagging as the basis for patent analytics, and the power and limitations of machine learning in patent analytics. It’s probably a good moment to mention that a lot of our thinking has been heavily influenced by the work of the Evalueserve co-founder and group strategy officer, Marc Vollenweider, around the concept of mind+machine™. At Evalueserve, we have various research-focused Lines of Business and those of us in the IPR&D business have learned a lot from innovation in other areas such as equity research, competitive intelligence, market intelligence, and data analytics, and applied this know-how to patent search and patent analytics.
Winter season – more detailed analysis and discussion about sample deliverables.
Over the next few months, you can expect in to read more details on our own innovations and the processes that we follow to deliver high-quality results. We’ll use sample projects and deliverables to discuss search quality using practical examples. We’ll introduce you to more Evalueserve and external experts and their thinking. Add to this analysis of related discussions, such as the topical discussion among Uber executives on patent quality. We also plan to come up with a lighter approach to patent searching and IPR&D Intelligence for the Christmas season.
We hope you find our Information Adventurers Blog inspirational and we’re eager to hear from you! If you happen to be at IP Service World in Munich, please join my panel lecture on November 27 at 1.30 pm. Questions can also be sent upfront to #ipsw18 @EvalueserveNews.