Twelfth Class: Step Two – Multimodal ECA
Multimodal Early Case Assessment – ECA – summarizes the second step in our 8-step work flow. We used to call the second step “Multimodal Search Review.” It is still the same activity, but we tweaked the name to emphasize the ECA significance of this step. After we have an idea of what we are looking for from ESI Communications in step one, we start to use every tool at our disposal to try to find the relevant documents. Every tool that is, except for active machine learning. Our first look at the documents is our look, not the machine’s. That is not because we do not trust the AI’s input. We do. It is because there is no AI yet. The predictive coding only begins after you feed training documents into the machine. That happens in step four.
Our Multimodal ECA step-two does not take that long, so the delay in bringing in our AI is usually short. In our experiments at TREC in 2015 and 2016 under the auspicious of NIST, where we skipped steps three and seven to save time, and necessarily had little ESI Communications in step one, we would often complete simple document reviews of several hundred thousand documents in just a few hours. We cannot match these results in real-life legal document review projects because the issues in law suits are usually much more complicated than the issues presented by most topics at TREC. Also, we cannot take the risk of making mistakes in a real legal project that we did in an academic event like TREC.
Again, the terminology revision to say Multimodal ECA is more a change of style than substance. We have always worked in this manner. The name change is just to better convey the idea that we are looking for the low hanging fruit, the easy to find documents. We are getting an initial assessment of the data by using all of the tools of the search pyramid except for the top tier active machine learning. The AI comes into play soon enough in steps four and five, sometimes as early as the same day.
I have seen projects where key documents are found during the first ten minutes of looking around. Usually the secrets are not revealed so easily, but it does happen. Step two is the time to get to know the data, run some obvious searches, including any keyword requests for opposing counsel. You use the relevant and irrelevant documents you find in step two as the documents you select in step four to train the AI.
In the process of this initial document review you start to get a better understanding of the custodians, their data and relevance. This is what early case assessment is all about. You will find the rest of the still hidden relevant documents in the iterated rounds of machine training and other searches that follow. Here is my video description of step two.
______
_______
Although we speak of searching for relevant documents in step two, it is important to understand that many irrelevant documents are also incidentally found and coded in that process. Active machine learning does not work by training on relevant documents alone. It must also include examples of irrelevant documents. For that reason we sometimes actively search for certain kinds of irrelevant documents to use in training. One of our current research experiments with Kroll Ontrack is to determine the best ratios between relevant and irrelevant documents for effective document ranking. See TREC reports at Mr. EDR as updated from time to time. At this point we have that issue nailed.
The multimodal ECA review in step two is carried out under the supervision of the Subject Matter Experts on the case. They make final decisions where there is doubt concerning the relevance of a document or document type. The SME role is typically performed by a team, including the partner in charge of the case – the senior SME – and senior associates, and e-Discovery specialist attorney(s) assigned to the case. It is, or should be, a team effort, at least in most large projects. As previously described, the final arbitrator on scope is made by the senior SME, who in turn is acting as the predictor of the court’s views. The final, final authority is always the Judge. The chart below summarizes the analysis of the SME and judge on the discoverability of any document.
When I do a project, acting as the e-Discovery specialist attorney for the case, I listen carefully to the trial lawyer SME as he or she explains the case. By extensive Q&A the members of the team understand what is relevant. We learn from the SME. It is not exactly a Vulcan mind-meld, but it can work pretty well with a cohesive team. Most trial lawyers love to teach and opine on relevance and their theory of the case.
General Moltke
Although a good SME team communicates and plans well, they also understand, typically from years of experience, that the intended relevance scope is like a battle plan before the battle. As the famous German military strategist, General Moltke the Elder said: No battle plan ever survives contact with the enemy. So too no relevance scope plan ever survives contact with the corpus of data. The understanding of relevance will evolve as the documents are studied, the evidence is assessed, and understanding of what really happened matures. If not, someone is not paying attention. In litigation that is usually a recipe for defeat. See Concept Drift and Consistency: Two Keys To Document Review Quality – Parts One, Two and Three.
The SME team trains and supervises the document review specialists, aka, contract review attorneys, who usually then do a large part of the manual reviews (step-six), and few if any searches. Working with review attorneys is a constant iterative process where communication is critical. Although I sometimes use an army-of-one approach where I do everything myself (that is how I did the EDI Oracle competition and most of the TREC topics), my preference now is to use two or three reviewers to help with the document review. With good methods, including culling methods, and good software, it is rarely necessary to use more reviewers than that. With the help of strong AI, say that included in Mr. EDR, we can easily classify a million or so documents for relevance with that size team. More reviewers than that may well be needed for complex redaction projects and other production issues, but not for a well-designed first-pass relevance search.
One word of warning when using document reviewers, it is very important for all members of the SME team to have direct and substantial contact with the actual documents, not just the reviewers. For instance, everyone involved in the project should see all hot documents found in any step of the process. It is especially important for the SME trial lawyer at the top of the expert pyramid to see them, but that is rarely more than a few hundred documents, often just a few dozen. Otherwise, the top SME need only see the novel and grey area documents that are encountered, where it is unclear on which side of the relevance line they should fall in accord with the last instructions. Again, the burden on the senior, and often technologically challenged senior SME attorneys, is fairly light under these Version 4.0 procedures.
The SME team relies on a primary SME, who is typically the trial lawyer in charge of the whole case, including all communications on relevance to the judge and opposing counsel. Thereafter, the head SME is sometimes only consulted on an as-needed basis to answer questions and make specific decisions on the grey area documents. There are always a few uncertain documents that need elevation to confirm relevance, but as the review progresses, their number usually decreases, and so the time and attention of the senior SME decreases accordingly.
Or pause to do this suggested “homework” assignment for further study and analysis.
SUPPLEMENTAL READING: This is a very important step and class. To really understand how this second step works you need to see it in action several times. It is never the exact same process, but instead depends on the circumstances of the search. The basic idea is to use every search tool you have, except of course AI ranking, to locate the easy-to-find documents, the low hanging fruit. In some projects this step only takes a few hours, in other it may take days. In TREC 2016 where we were looking to set speed records, we would usually take less than an hour, but we were searching a known dataset, the Jeb Bush email. For background reading on step two we suggest you look at articles on e-discovery and early case assessment. This is one of those ambiguous terms in e-discovery that can mean many different things, many different activities, many of them outside of our step two. It is good to have a familiarity with all of them. Predictive coding experts must also become early case assessment experts.
If you have not already done so, read Concept Drift and Consistency: Two Keys To Document Review Quality – Parts One, Two and Three. In step two your first ideas of relevance meet the data you intend to classify. If this meeting does not somehow alter your ides of relevance, at least somewhat, then chances are you are not paying attention.
EXERCISES: Consider how much more effective 26(f) conferences would be if both sides had completed this second step before the meeting. Try doing that in your next case. Leave us a comment on how this changed the meeting and discovery plan process. Also try doing all of the first seven steps before the conferences with opposing counsel. Notice how much easier the process becomes. Do not have the authority to accelerate the document review process to try these exercises? Plead your case to whomever is in charge. Try describing this as an experiment to improve your methods and abilities. Perhaps offer this to the client at an introductory reduced price (or free) service to show them the benefits.
As a separate exercise, ask your preferred vendor to help you to better understand one of the search tools on their platform that you have not used very much before or are otherwise not that familiar with. Ask for some free training just on that feature from their best expert. If your e-discovery vendor does not offer such guidance and free training, you should consider a new vendor. In addition to training from experts, just try out a new search feature on your own. See how it works for yourself. Ask questions. Most good software today has a multitude of search features. If yours does not, you should consider using different software. The days on one-trick ponies are long gone. The best approach is variety.
Students are invited to leave a public comment below. Insights that might help other students are especially welcome. Let’s collaborate!
_
e-Discovery Team LLC COPYRIGHT 2017
ALL RIGHTS RESERVED
_
