Legal Search Science is an interdisciplinary field of study and practice concerning the search, review, and classification of large collections of electronic documents to identify targeted information for use as evidence in legal proceedings. It is a subset of the field of Information Science concerned with information retrieval and the unique problems faced by lawyers in the discovery of relevant evidence. It is also a subset of the legal field of electronic discovery and engineering field of Big Data search software. As such, it is an interdisciplinary field combining law, technology, and science.
Most specialists in legal search science use a variety of search methods when searching large datasets, referred to here as a multimodal approach, but primarily rely on supervised or semi-supervised machine learning (a type of artificial intelligence (AI)) using an active learning approach. I refer to this as AI-enhanced review or AI-enhanced search. In information science it may be referred to as active machine learning, and in legal circles as Predictive Coding.
For reliable information on Legal Search Science see the works of attorney, Maura Grossman, and her information scientist partner, Gordon Cormack, including:
- Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Richmond Journal of Law and Technology, Vol. XVII, Issue 3, Article 11 (2011);
- The Grossman-Cormack Glossary of Technology-Assisted Review, 2013 Fed. Cts. L. Rev. 7 (January 2013).
The Grossman-Cormack Glossary explains that in machine learning:
Supervised Learning Algorithms (e.g., Support Vector Machines, Logistic Regression, Nearest Neighbor, and Bayesian Classifiers) are used to infer Relevance or Non-Relevance of Documents based on the Coding of Documents in a Training Set. In Electronic Discovery generally, Unsupervised Learning Algorithms are used for Clustering, Near-Duplicate Detection, and Concept Search.
Multimodal search uses both machine learning algorithms and unsupervised learning search tools (clustering, near-duplicates and concept), as well as keyword search and even some limited use of traditional linear search. This is further explained below in the section, Hybrid Multimodal Bottom Line Driven Review.
The key definition of the Glossary is for Technology Assisted Review, their term for AI-enhanced review.
Technology-Assisted Review (TAR): A process for Prioritizing or Coding a Collection of Documents using a computerized system that harnesses human judgments of one or more Subject Matter Expert(s) on a smaller set of Documents and then extrapolates those judgments to the remaining Document Collection. Some TAR methods use Machine Learning Algorithms to distinguish Relevant from Non-Relevant Documents, based on Training Examples Coded as Relevant or Non-Relevant by the Subject Matter Experts(s), …. TAR processes generally incorporate Statistical Models and/or Sampling techniques to guide the process and to measure overall system effectiveness.
The Grossman-Cormack Glossary makes clear the importance of Subject Matter Experts (SMEs) by including their use as the document trainer into the very definition of TAR. Nevertheless, experts agree that good predictive coding software is able to tolerate some errors made in the training documents. For this reason experiments are being done on ways to minimize the central role of the SMEs, to see if lesser-qualified persons could also be used in document training, at least to some degree. See Webber & Pickens, Assessor Disagreement and Text Classifier Accuracy (SIGIR, 2013); John Tredennick, Subject Matter Experts: What Role Should They Play in TAR 2.0 Training? (2013). These experiments are of special concern to software developers and others who would like to increase the utilization of AI-enhanced software because, at the current time, very few SMEs in the law have the skills or time necessary to conduct AI-enhanced searches. This is one reason that predictive coding is still not widely used, even though it has been proven effective in multiple experiments and adopted by several courts.
For in-depth information on key experiments already performed in the field of Legal Search Science, see the TREC Legal Track reports whose home page is maintained by a leader in the field, information scientist, Doug Oard. Professor Oard is a co-founder of the TREC Legal track. Also see the research and reports of Herb Rotiblat and the Electronic Discovery Institute, and my papers on TREC (and otherwise as listed below): Analysis of the Official Report on the 2011 TREC Legal Track – Part One, Part Two and Part Three; and Secrets of Search: Parts One, Two, and Three.
For general legal background on the field of Legal Search Science see the works of the attorney co-founder of TREC Legal Track, Jason R. Baron, including:
- Baron & Freeman, Quick Peek at the Math Behind the Black Box of Predictive Coding (2013)
- DESI, Sedona and Barcelona (2013)
- Paul & Baron, Information Inflation: Can The Legal System Adapt? 13 Rich J.L. & Tech 10 (2007)
- Baron, Editor, The Sedona Conference® Best Practices Commentary on Search & Retrieval Methods (Aug. 2007)
As explained in Baron and Freeman’s Quick Peek at the Math, and my blog introduction thereto, the supervised learning algorithms behind predictive coding utilize a hyper-dimensional space. Each each document in the dataset, including its metadata, represent a different dimension mapped in trans-Cartesian space, called hyper-planes. Each document is placed according to a multi-dimensional dividing line of relevant and irrelevant. The important document ranking feature of predictive coding is performed by measure as to how far from the dividing line a particular document lies. Each time a training session is run the line moves and the ranking fluctuates in accordance with the new information provided. The below diagram attempts to portray this hyperplane division and document placement. The points shown in red designate irrelevant documents and the blue points relevant documents. The dividing line would run through multiple dimensions, not just the usual two of a Cartesian graph. This is depicted in this diagram by folding fields. For more read the entire Quick Peek article.
For a scientific and statistical view of Legal Search Science that is often at least somewhat intelligible to lawyers and other non-scientists, see the blog of information scientist and consultant, William Webber, Evaluating e-Discovery. For writings designed for the general reader on the subject of predictive coding, see the many articles by attorney Karl Scheineman, another pioneer in the field.
AI-Enhanced Search Methods
AI-enhanced search represents an entirely new method of legal search, which requires a completely new approach to large document reviews. Below is the diagram that I created to show the new workflow that I use in a typical predictive coding project.
For a more detailed description of the eight steps see the Electronic Discovery Best Practices page on predictive coding. For another somewhat similar workflow description see the diagram below of Kroll Ontrack, the vendor whose predictive coding software I now frequently use. Their seven-step model is described at pages 3-4 of Kroll Ontrack’s white paper, Technology Assisted Review: Driving Ediscovery Efficiencies in the Era of Big Data (2013).
I have found that proper AI-enhanced review requires the highest skill levels and is, for me at least, the most challenging activity in electronic discovery law. See: Electronic Discovery Best Practices for a description of the ten types of legal services involved in e-discovery. I am convinced that predictive coding is The big new tool that we have all been waiting for. When used properly, good AI-enhanced software allows attorneys to find the information they need in vast stores of ESI, and to do so in an effective and affordable manner.
In my experience the best software and training methods use AI type active learning process in steps four and five of my chart above and steps 2-5 of Kroll Ontrack’s chart. My preferred active learning process in the iterative machine learning steps is threefold:
- The computer selects documents for review where the software classifier is uncertain of the correct classification. This helps the classifier algorithms to learn by adding diversity to the documents presented for review. This in turn helps to locate outliers of a type your initial judgmental searches in step two and five have missed. This is machine selected sampling, and, according to a basic text in information retrieval engineering, a process is not a bona fide active learning search without this ability. Manning, Raghavan and Schutze, Introduction to Information Retrieval, (Cambridge, 2008) at pg. 309.
- Some reasonable percentage of the documents presented for human review in step five are selected at random. This again helps maximize recall and premature focus on the relevant documents initially retrieved.
- Other relevant documents that a skilled reviewer can find using a variety of search techniques. This is called judgmental sampling. See Baron, Jason, Co-Editor, The Sedona Conference® Commentary on Achieving Quality in the E-Discovery Process (2009). Judgmental sampling can use a variety of search tools, including both the mentioned Supervised and Unsupervised Learning Algorithms, and is more further described below.
The initial seed set generation, step two in my chart, should also use some random samples, plus judgmental multimodal searches. Steps three and six in my chart always use pure random samples and rely on statistical analysis. For background on the three types of sampling see my article, Three-Cylinder Multimodal Approach To Predictive Coding.
After the first round of training, aka the seed set, judgmental sampling continues along with random and machine selected sampling in steps four and five. In judgmental sampling the human reviewer often selects additional documents for review and training that are based on the machine selected or random selected documents presented for review. Sometimes, however, the SME human reviewer follows a new search idea unrelated to the new documents seen. When an experienced searcher sees new documents this often leads to new search ideas.
All kinds of searches can be used for judgmental sampling, which is why I call it a multimodal search. This may include some linear review of selected custodians or selected date ranges, parametric Boolean keyword searches, similarity searches of all kinds, clustering searches, concept searches, as well as several unique predictive coding probability searches. These document probability searches are based primarily on the unique document ranking capabilities of most AI-enhanced search software. I find the ranking based searches to be extremely helpful to maximize efficiency and effectiveness.
I also often use informal random sampling of returned search sets as part of the judgmental sampling review and evaluation process. This is a process where I browse through search results, both according to ranking and random views. I will also use various document sorting views to get a better understanding of the documents returned. I will also use different search methods and document views according to the type of data, the custodian, where it was originally stored, or even any sub-search-goal or issue I might be focused on at any one point in time. A good search follows systems and is repeatable, but it is also fluid. It is adaptable to new information uncovered from the documents searched, or from new information received elsewhere about the case, or from new documents added to the search in mid-course. Good search and review software is designed to allow for both flexibility and a systems approach.
All of these methods allow an experienced legal searcher to get a feel for the underlying data. This obviates the need for full linear review of the multimodal search results in the judgmental sampling process. Still, it is sometimes appropriate to read a few hundred documents in a linear fashion, for instance, all email a key witness received on a critical day.
AI featured multimodal search represents a move from traditional legal art to science. See: Seven Years of the e-Discovery Team Blog, Art to Science. But there is still room for art in the sense of the deeply engrained skills and intuition that lawyers can only gain from years of experience with legal search. Knowledgable lawyers have unique insights into the evidence and witnesses involved in a particular case or type of dispute. This background knowledge and experience allows a skilled SME to improvise. The searcher can change directions depending on the documents found, and depending on new documents added to the dataset, or even new issues and a changed scope of relevance. Indeed, the search methods used in multimodal judgmental sampling vary considerably, both on a project by project basis, and over time in the same project, as understanding of the data, targets, and search develops. This is where years of legal search experience can be extremely valuable. All well designed predictive coding software allows for such a flexible approach to empower the attorneys conducting the search.
My insistence on the use of multimodal judgmental sampling in steps two and five to locate relevant documents follows the consensus view of information scientists specializing in information retrieval, but is not followed by several prominent predictive coding vendors. They instead rely entirely on machine selected documents for training, or even worse, rely entirely on random selected documents to train the software. In my writings I call these processes the Borg approach, after the infamous villains in Star Trek, the Borg, a race half-human robots that assimilates people into machines. (I further differentiate between three types of Borg in Three-Cylinder Multimodal Approach To Predictive Coding.) Like the Borg, these approaches unnecessarily minimize the role of individuals, the SMEs. They exclude other types of search to supplement an active learning process. I advocate the use of all types of search, not just predictive coding.
Hybrid Human Computer Information Retrieval
Further, in contradistinction to Borg approaches, where the machine controls the learning process, I advocate a hybrid approach where Man and Machine work together. In my hybrid search and review projects the expert reviewer remains in control of the process, and their expertise is leveraged for greater accuracy and speed. The human intelligence of the SME is a key part of the search process. In the scholarly literature of information science this hybrid approach is known as Human–computer information retrieval (HCIR). (My thanks to information scientist Jeremy Pickens for pointing out this literature to me.)
The classic text in the area of HCIR, which I endorse, is Information Seeking in Electronic Environments (Cambridge 1995) by Gary Marchionini, Professor and Dean of the School of Information and Library Sciences of U.N.C. at Chapel Hill. Professor Marchionini speaks of three types of expertise needed for a successful information seeker:
- Domain Expertise. This is equivalent to what we now call SME, subject matter expertise. It refers to a domain of knowledge. In the context of law the domain would refer to the particular type of lawsuit or legal investigation, such as antitrust, patent, ERISA, discrimination, trade-secrets, breach of contract, Qui Tam, etc. The knowledge of the SME on the particular search goal is extrapolated by the software algorithms to guide the search. If the SME also has the next described System Expertise and Information Seeking Expertise, they can run the search project themselves. That is what I like to call the Army of One approach. Otherwise, they will need a chauffeur or surrogate with such expertise, one who is capable of learning enough from the SME to recognize the relevant documents.
- System Expertise. This refers to expertise in the technology system used for the search. A system expert in predictive coding would have a deep and detailed knowledge of the software they are using, including the ability to customize the software and use all of its features. In computer circles a person with such skills is often called a power-user. Ideally a power-user would have expertise in several different software systems. They would also be an expert in one or more particular method of search.
- Information Seeking Expertise. This is a skill that is often overlooked in legal search. It refers to a general cognitive skills related to information seeking. It is based on both experience and innate talents. For instance, “capabilities such as superior memory and visual scanning abilities interact to support broader and more purposive examination of text.” Professor Marchionini goes on to say that: “One goal of human-computer interaction research is to apply computing power to amplify and augment these human abilities.” Some lawyers seem to have a gift for search, which they refine with experience, broaden with knowledge of different tools, and enhance with technologies. Others do not.
Id. at pgs.66-69, with the quotes from pg. 69.
All three of these skills are required for an attorney to attain expertise in legal search today, which is one reason I find this new area of legal practice so challenging. It is difficult, but not impossible like this Penrose triangle.
It is not enough to be an SME, or a power-user, or have a special knack for search. You have to be able to do it all, and so does your software. However, studies have shown that of the three skill-sets, System Expertise, which in legal search primarily means mastery of the particular software used, is the least important. Id. at 67. The SMEs are more important, those who have mastered a domain of knowledge. In Professor Marchionini’s words:
Thus, experts in a domain have greater facility and experience related to information-seeking factors specific to the domain and are able to execute the subprocesses of information seeking with speed, confidence, and accuracy.
Id. That is one reason that the Grossman Cormack glossary quoted before builds in the role of SMEs as part of their base definition of technology assisted review. Glossary at pg. 21 defining TAR.
According to Marchionini, Information Seeking Expertise, much like Subject Matter Expertise, is also more important than specific software mastery. Id. This may seem counter-intuitive in the age of Google, where an illusion of simplicity is created by typing in words to find websites. But legal search of user-created data is a completely different type of search task than looking for information from popular websites. In the search for evidence in a litigation, or as part of a legal investigation, special expertise in information seeking is critical, including especially knowledge of multiple search techniques and methods. Again quoting Professor Marchionini:
Expert information seekers possess substantial knowledge related to the factors of information seeking, have developed distinct patterns of searching, and use a variety of strategies, tactics and moves.
Id. at 70.
In the field of law this kind of information seeking expertise includes the ability to understand and clarify what the information need is, in other words, to know what you are looking for, and articulate the need into specific search topics. This important step precedes the actual search, but is an integral part of the process. As one of the basic texts on information retrieval written by Gordon Cormack, et al, explains:
Before conducting a search, a user has an information need, which underlies and drives the search process. We sometimes refer to this information need as a topic …
Buttcher, Clarke & Cormack, Information Retrieval: Implementation and Evaluation of Search Engines (MIT Press, 2010) at pg. 5. The importance of pre-search refining of the information need is stressed in the first step of the above diagram of my methods, ESI Discovery Communications. It seems very basic, but is often under appreciated, or overlooked entirely in the litigation context where information needs are often vague and ill-defined, lost in overly long requests for production and adversarial hostility.
Hybrid Multimodal Bottom Line Driven Review
I have a long descriptive name for what Marchionini calls the variety of strategies, tactics and moves that I have developed for legal search: Hybrid Multimodal AI-Enhanced Review using a Bottom Line Driven Proportional Strategy. See eg. Bottom Line Driven Proportional Review (2013). I refer to it as a multimodal method because, although the predictive coding type of searches predominate (shown on the below diagram as AI-enhanced review - AI), I also use the other modes of search, including the mentioned Unsupervised Learning Algorithms (clustering and concept), keyword search, and even some traditional linear review (although usually very limited). As described, I do not rely entirely on random documents, or computer selected documents for the AI-enhanced searches, but use a three-cylinder approach that includes human judgment sampling and AI document ranking. The various types of legal search methods used in a multimodal process are shown in this search pyramid.
Most information scientists I have spoken to agree that it makes sense to use multiple methods in legal search and not just rely on any single method, even the best AI method. UCLA Professor Marcia J. Bates first advocated for using multiple search methods back in 1989, which she called it berrypicking. Bates, Marcia J. The Design of Browsing and Berrypicking Techniques for the Online Search Interface, Online Review 13 (October 1989): 407-424. As Professor Bates explained in 2011 in Quora:
An important thing we learned early on is that successful searching requires what I called “berrypicking.” … Berrypicking involves 1) searching many different places/sources, 2) using different search techniques in different places, and 3) changing your search goal as you go along and learn things along the way. This may seem fairly obvious when stated this way, but, in fact, many searchers erroneously think they will find everything they want in just one place, and second, many information systems have been designed to permit only one kind of searching, and inhibit the searcher from using the more effective berrypicking technique.
This berrypicking approach, combined with HCIR, is what I have found from practical experience works best with legal search. They are the Hybrid Multimodal aspects of my AI-Enhanced Review Bottom Line Driven Review method.
Why AI-Enhanced Search and Review Is Important
I focus on this sub-niche area of e-discovery becausee I am convinced that it is critical to advancement of the law in the 21st Century. The new search and review methods that I have developed from my studies and experiments in legal search science allow a skilled attorney using readily available predictive coding type software to review at remarkable rates of speed and cost. Review rates are more than 250-times faster than traditional linear review, and costs less than a tenth as much. See eg Predictive Coding Narrative: Searching for Relevance in the Ashes of Enron, and the report by the Rand Corporation, Where The Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery.
Thanks to the new software and methods, what was considered impossible, even absurd, just a few short years ago, namely one attorney accurately reviewing over a million documents by him or herself in 14-days, is attainable by many experts. I have done it. That is when I came up with the Army of One motto and realized that we were at a John Henry moment in Legal Search. Maura tells me that she once did a seven-million document review by herself. Maura and Gordon were correct to refer to TAR as a disruptive technology in the Preface to their Glossary. Technology that can empower one skilled lawyer to do the work of hundreds of unskilled attorneys is certainly a big deal, one for which we have Legal Search Science to thank.
More Information On legal Search Science
For further information on Legal Search Science see all of the articles cited above, along with my articles listed below. Most of my articles were written for the general reader, some are highly technical but still accessible with study. All have been peer-reviewed in my blog by most of the founders of this field who are regular readers and thousands of other readers. Also see the CAR procedures described on Electronic Discovery Best Practices.
I am especially proud of the legal search experiments I have done using AI-enhanced search software provided to me by Kroll Ontrack to review the 699,083 public Enron documents and my reports on these reviews. Comparative Efficacy of Two Predictive Coding Reviews of 699,082 Enron Documents. (Part Two); A Modest Contribution to the Science of Search: Report and Analysis of Inconsistent Classifications in Two Predictive Coding Reviews of 699,082 Enron Documents. (Part One). I have been told by scientists in the field that my over 100 hours of search, consisting of two fifty-hour search projects using different methods, is the largest search project by a single reviewer that has ever been undertaken, not only in Legal Search, but in any kind of search. I do not expect this record will last for long, as others begin to understand the importance of Information Science in general, and Legal Search Science in particular. But for now I will enjoy both the record and lessons learned from the hard work involved. I may also attempt a third search project soon to continue to make contributions to Legal Search Science. Stay tuned. I may extend my record to 150 hours.
Articles by Ralph Losey on Legal Search
- Less Is More: When it comes to predictive coding training, the “fewer reviewers the better” - Parts One, Two, and Three.
- My Basic Plan for Document Reviews: The “Bottom Line Driven” Approach, PDF version suitable for print, or HTML version that combines the blogs published in four parts.
- Relevancy Ranking is the Key Feature of Predictive Coding Software.
- Why a Receiving Party Would Want to Use Predictive Coding?
- Vendor CEOs: Stop Being Empty Suits & Embrace the Hacker Way
- Comparative Efficacy of Two Predictive Coding Reviews of 699,082 Enron Documents. (Part Two).
- A Modest Contribution to the Science of Search: Report and Analysis of Inconsistent Classifications in Two Predictive Coding Reviews of 699,082 Enron Documents. (Part One).
- Introduction to Guest Blog: Quick Peek at the Math Behind the Black Box of Predictive Coding that pertains to the higher-dimensional geometry that makes predictive coding support vector machines possible.
- Keywords and Search Methods Should Be Disclosed, But Not Irrelevant Documents.
- Reinventing the Wheel: My Discovery of Scientific Support for “Hybrid Multimodal” Search.
- There Can Be No Justice Without Truth, And No Truth Without Search (statement of my core values as a lawyer explaining why I think predictive coding is important).
- Three-Cylinder Multimodal Approach To Predictive Coding.
- Robots From The Not-Too-Distant Future Explain How They Use Random Sampling For Artificial Intelligence Based Evidence Search. Video Animation.
- Borg Challenge: Report of my experimental review of 699,082 Enron documents using a semi-automated monomodal methodology (a five-part written and video series comparing two different kinds of predictive coding search methods).
- Predictive Coding Narrative: Searching for Relevance in the Ashes of Enron in PDF form for easy distribution and the blog introducing this 82-page narrative, with second blog regarding an update.
- Journey into the Borg Hive: a Predictive Coding Narrative in science fiction form.
- The Many Types of Legal Search Software in the CAR Market Today.
- Georgetown Part One: Most Advanced Students of e-Discovery Want a New CAR for Christmas.
- Escape From Babel: The Grossman-Cormack Glossary.
- NEWS FLASH: Surprise Ruling by Delaware Judge Orders Both Sides To Use Predictive Coding.
- Does Your CAR (“Computer Assisted Review”) Have a Full Tank of Gas? (and you can also click here for the alternate PDF version for easy distribution).
- Analysis of the Official Report on the 2011 TREC Legal Track – Part One.
- Analysis of the Official Report on the 2011 TREC Legal Track – Part Two.
- Analysis of the Official Report on the 2011 TREC Legal Track – Part Three
- An Elusive Dialogue on Legal Search: Part One where the Search Quadrant is Explained.
- An Elusive Dialogue on Legal Search: Part Two – Hunger Games and Hybrid Multimodal Quality Controls.
- Random Sample Calculations And My Prediction That 300,000 Lawyers Will Be Using Random Sampling By 2022.
- Second Ever Order Entered Approving Predictive Coding.
- Predictive Coding Based Legal Methods for Search and Review.
- New Methods for Legal Search and Review.
- Perspective on Legal Search and Document Review.
- LegalTech Interview of Dean Gonsowski on Predictive Coding and My Mission to Make Predictive Coding Software More Affordable.
- My Impromptu Video Interview at NY LegalTech on Predictive Coding and Some Hopeful Thoughts for the Future.
- The Legal Implications of What Science Says About Recall.
- Reply to an Information Scientist’s Critique of My “Secrets of Search” Article.
- Secrets of Search – Part I.
- Secrets of Search – Part II.
- Secrets of Search – Part III. (All three parts consolidated into one PDF document.)
- Information Scientist William Webber Posts Good Comment on the Secrets of Search Blog.
- Judge Peck Calls Upon Lawyers to Use Artificial Intelligence and Jason Baron Warns of a Dark Future of Information Burn-Out If We Don’t.
- The Information Explosion and a Great Article by Grossman and Cormack on Legal Search.
Please contact me at Ralph.Losey@gmail.com if you have any questions.