This blog introduces the e-Discovery Team’s latest insights and methods of document review.
We call this Predictive Coding 4.0 because it substantially improves upon, and replaces the methods and insights we announced in our October 2015 publication – Predictive Coding 3.0. In that two-part blog we explained the history of predictive coding software and methods in legal review, including versions 1.0 and 2.0. Then we described our new version 3.0 in some detail. Since that publication we have developed more enhancements to our methods, including many new, innovative uses of the predictive coding features of Kroll Ontrack’s EDR software. We even developed some new features not related to predictive coding. (Try out the new Folder Similar search in EDR for example.) Most of our new insights, just like our prior 3.0 version methodologies, can also be used on other software platforms. To use all of the features, however, the software will have to have bona fide active machine learning capacities. Most do not. More on that later.
These improvements naturally evolved to a certain degree as part of the e-Discovery Team members normal work supervising hundreds, maybe even thousands of document review projects over the past year. But the new insights that require us to make a complete restatement, a new Version 4.0, arose just recently. Major advances were attained as part of an intensive three months of experiments, all conducted outside of our usual legal practice and document reviews. The e-Discovery Team doing this basic research consisted of myself and several of Kroll Ontrack’s top document review specialists, including especially Jim Sullivan and Tony Reichenberger. They have now fully mastered the e-discovery team search and review Hybrid Multimodal methodologies. As far as I can see, at this point in the race for the highest quality legal document review, no one else comes even close to their skill level. Yes, e-discovery is highly competitive, but they trained hard and are now looking back and smiling.
The insights we gained, and the skills we honed, including speed, did not come easily. It took full time work on client projects all year, plus three full months of research, often in lieu of real summer vacations (my wife is still waiting). This is hard work, but we love it. See: Why I Love Predictive Coding. This kind of dedication of time and resources by an e-discovery vendor or law firm is unprecedented. There is a cost to attain the research benefits realized, both hard out-of-pocket costs and lost time. So I hope you understand that we are only going to share some of our techniques. The rest we will keep as trade-secrets. (Retain us and watch. Then you can see them in action.)
Kroll Ontrack understands the importance of pure research and enthusiastically approved these expenditures. (My thanks again to CEO Mark Williams, a true visionary leader in this industry who approved and supported the research program.) I suggest you ask your vendor, or law firm, how much time they spent last year researching and experimenting with document review methods? As far as we know, the only other vendor with an active research program is Catalyst, whose work is also to be commended. (No one else showed up for TREC.) The only other law firm we know of is Maura Grossman’s new solo practice. Her time spent with research is also impressive.
The results we attained certainly make this investment worthwhile, even if many in the profession do not realize it, much less appreciate it. They will in time, so will the consumers. This is a long term investment. Pure research is necessary for any technology company, including all companies in the e-Discovery field. The same holds true, albeit to a lesser extent, to any law firm claiming to have technological superiority.
Experience from handling live projects alone is too slow an incubator for the kind of AI breakthrough technologies we are now using. It is also too inhibiting. You do not experiment on important client data or review projects. Any expert will improvise somewhat during such projects to match the circumstances, and sometimes do post hoc analysis. But such work on client projects alone is not enough. Pure research is needed to continue to advance in AI-enhanced review. That is why the e-Discovery Team spent a substantial part of our waking hours in June, July and August 2016 working on experiments with Jeb Bush email. The Jeb Bush email collection was our primary laboratory this year. As a result of the many new things we learned, and new methods practiced and perfected, we have now reached a point where a complete restatement of our method is in order. Thus we here release Predictive Coding 4.0.
Our latest breakthroughs this summer primarily came out of the e-Discovery Team’s participation in the annual Text Retrieval Conference, aka TREC, sponsored by the National Institute of Standards and Technology. This is the 25th year of the TREC event. We were honored to again participate, as we did last year, in the Total Recall Track of TREC. This is the closest Track that TREC now offers to a real legal review project. It is not a Legal Track, however, and so we necessarily did our own side-experiments, and had our own unique approach different from the Universities that participated. The TREC leadership of the Total Recall Track was once again in the capable hands of Maura Grossman, Gordon Cormack and other scientists.
This blog will not report on the specifics of the 2016 Total Recall Track. That will come at a later time after we finish analyzing the enormous amount of data we generated and submit our formal reports to TREC. In any event, the TREC related work we did this Summer went beyond the thirty-four research topics included in the TREC event. It went well beyond the 9,863,366 documents we reviewed with Mr. EDR’s help as part of the formal submittals. Countless more documents were reviewed for relevance if you include our side-experiments.
At the same time that we did the formal tests specified by the Total Recall Track we did multiple side-experiments of our own. Some of these tests are still ongoing. We did so to investigate our own questions that are unique to legal search and thus beyond the scope of the Total Recall Track. We also performed experiments to test unique attributes of Kroll Ontrack’s EDR software. It uses a proprietary type of logistic regression algorithm that was awarded a patent this year. Way to go KO and Mr. EDR!
Although this blog will not report on our TREC experiments per se, we will share the bottom line, the take-aways of this testing. Not everything will be revealed. We keep some of our methods and techniques trade-secret.
We will also not be discussing in this multi-part blog our future plans and spin-off projects. Let’s just say for now that we have several in mind. One in particular will, I think, be very exciting for all attorneys and paralegals who do document review. Maybe even fun for those of you who, like us, are really into and enjoy a good computer search. You know who you are! If my recommendations are accepted, we will open that one up to all of our fellow doc-review freaks. I will say no more at this point, but watch for announcements in the coming year from Kroll Ontrack and me. We are having too much fun here not to share some of the good times.
Even if we did adopt 100% transparency on our methods, it would take a book to write it all down, and it would still be incomplete. Many things can only be learned by doing, especially methods. Document review is, after all, a part of legal practice. As the scientists like to put it, legal search is essentially ad hoc. It changes and is customized to fit the particular evidence search assignments at hand. But we will try to share all of the basic insights. They have all been discussed here before. The new insights we gained are more like a deepening understanding and matter of emphasis. They are refinements, not radical departures, although some are surprising.
Nine Insights Concerning the Use of Predictive Coding in Legal Document Review
The diagram below summarizes the nine basic insights that have come out of our work this year. These are the key concepts that we now think are important to understand and implement. [Just like the 8-Step Workflow diagram above, this, and other diagrams in this blog may be freely used with attribution. But please do not change anything without my permission. I am also happy to provide you with higher resolution graphics if needed for presentation or publication purposes.]
The diagrams above and following will be explained in detail throughout the rest of this multipart blog, as will the restated 8-Step Workflow shown at the top of the page. These are not new concepts. I have discussed most of these here before. I am confident that all readers will be able to follow along as I set forth the new nuances we learned.
Although these concepts are all familiar, some of our deepened understanding of these concepts may surprise you. Some were surprising to us. These insights include several changes in thinking on our part. Some of the research results we saw were unexpected. But we follow the data. Our opinions are always held lightly. I have argued both sides of a legal issue too many times as a lawyer to fall into that trap. Our thinking follows the evidence, not our preconceptions. That is, after all, the whole point of research. Schedule permitting, we are also happy to provide in-person or online presentations that explain these concept-summary diagrams. If retained, you can also see it in action.
Although these insights and experiments were derived using Kroll Ontrack EDR software, they are essentially vendor neutral. The methods will work on any full-featured document review platform, but especially those that includes bona fide active machine learning abilities, aka Predictive Coding. As all experts in this field know, many of the most popular document review platforms do not have these features, even those stating they use Analytics. Active Machine Learning is very different, and far more advanced than Analytics, the early forms of which were called Concept Search. This type of machine learning is passive and clearly is not predictive coding. It has its place in any multimodal system such as ours, and can be a powerful feature to improve search and review. But such software is incomplete and cannot meet the standards and capability of software that includes active machine learning. Only full featured document review platforms with active machine learning abilities can use all of the Predictive Coding 4.0 methods described here.
Sorry dear start-up vendors, and others, but that’s the truth. Consumers, you get what you pay for. You know that. Not sure? Get the help of an independent expert advisor before you make substantial investments in e-discovery software or choose a vendor for a major project. Also, if you have tried predictive coding, or what you were told was advanced TAR, whatever the hell that is, and it did not work well, do not blame yourself. It could be the software. Or if not the software, then the antiquated version 1.0 or 2.0 methods used. There is a lot of bullshit out there. Excuse my French. There always has been when it comes to new technology. It does, however, seems especially prevalent in the legal technology field. Perhaps they think we lawyers are naive and technologically gullible. Do not be fooled. Again, look to an independent consultant if you get confused by all the vendor claims.
Contrary to what some vendors will tell you (typically the ones without bona fide predictive coding features), predictive coding 3.0, and now 4.0, methods are not rocket science. You do not have to be a TAR-whisperer or do nothing but search, like my A-team for TREC. With good software it is not really that hard at all. These methods do, however, require an attorney knowledgable in e-discovery and comfortable with software. This is not for novices. But every law firm should anyway have attorneys with special training and experience in technology and e-discovery. For instance, if you practice in the Northern District of California, an e-discovery liaison with such expertise is required in most cases. See Guidelines for the Discovery of Electronically Stored Information. Almost half of the Bar Associations in the U.S. require basic technology competence as an ethical imperative. See eg. ABA Model Rule 1.1, Comment [8] and Robert Ambrogi’s list of 23 states, and counting, that now require such competence. (My own law firm has had an e-discovery liaison program in place since 2010, which I lead and train. I am proud to say that after six years of work it is now a great success.) So no, you do not have to be a full-time specialist, like the members of my TREC e-Discovery team, to successfully use AI-enhanced review, which we call Hybrid Multimodal. This is especially true when you work with vendors like Kroll Ontrack, Catalyst and others that have teams of special consultants to guide you. You just have to pick your vendors wisely.
To be continued …
Discover more from e-Discovery Team
Subscribe to get the latest posts sent to your email.
Thank you, Ralph, for sharing your enthusiasm and for describing how diligent, detailed, and demanding work is required to capably conduct legal document review.
Your timing is perfect.
After all, last month the Gartner Group (http://www.gartner.com/newsroom/id/3412017) noted that machine learning technology is at the peak of “inflated expectations” which naturally leads to the “trough of disillusionment.”
Also last month Judge Peck rebutted TAR proponent’s claims that it is “Black Letter law” that courts “order” or “approve” or “require” TAR by ruling (in Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP) (S.D.N.Y. Aug. 1, 2016), that a producing party cannot be compelled to employ TAR.
Perhaps more importantly, you dispel the claim that using TAR is as simple and reliable as cooking a Thanksgiving Turkey and, therefore, courts should grant TAR projects a priori approval.
Your article demonstrates that TAR is not turnkey defensible discovery.
Instead, as you have said for years and is thoughtfully presented among your “Nine Insights,” TAR necessarily marshals tools and techniques and, yes, tenacity.
We look forward to learning more about what you and your team discovered because we all want to automate as much as possible.
But not more.
Good comment. Thanks. One or two small quibbles re language. I don’t think anyone is more of a Predictive Coding proponent than me, by which I mean a proponent of including active machine learning techniques in your search. This may well be different from your wording “TAR proponent.” But regardless, I certainly Do NOT support any side being able to order the other side to use TAR, or any other particular type of search, even my type of Hybrid Multimodal Predictive Coding 4.0. Nor do I think this is Black Letter law. I don’t think that’s the law at all, nor should it be. I was not even aware that anyone was pushing such an extreme position until I read that case.
Judge Peck made the correct ruling here and his comments were also all right on, at least IMO. The party trying to breach Principle Six was way out there in right field. I do not have any reason to believe that this is a common position for anyone, TAR proponents included. It has certainly never been my position.
I dont think any approval is needed by anyone to carry out whatever search they deem reasonable. (I may have one exception to that, but it is really begging the question, or cart before horse, since I do think that search by use of “Go Fish” keyword guessing without looking is usually going to produce unreasonably poor results, and thus that attorneys doing this type of search alone will usually not be in compliance with 26(g).) Now if they mess up in the search methods they use, and do not make a “reasonable effort,” then, and only then, the court and other side should be allowed to step in to approve or disapprove.
So what I’m saying, and most others that I have read, is there is no need to go to a court for approval to use TAR. That is not part of their job under the rules of discovery. Discovery is in the hands of counsel, not the court. That is why counsel is required to sign Rule 26(g) to certify that they made reasonable efforts under the circumstances presented. It is the court’s responsability to enforce that rule, including the automatic award of fees if it is broken (something they all too often fail to do). See the difference? I think that distinction is important. Do you agree?
We may well reach the point where the use of particular methods is presumptively an unreasonable effort, a red flag of sorts. But we are not there yet. If we get there, then it should NOT be for predictive coding, at least not IMO. It should be for search using blind Go Fish types of keyword search.
That should, in my opinion, trigger a presumption of unreasonable efforts and require heightened scrutiny of counsel’s efforts. This is still the most popular kind of search used nationwide, so that would force a major improvement in the quality of justice.
If Judge Peck took that on today in the right procedural context, as he already has in his landmark Gross Construction case, he could rule that Keywords search, if used alone, must be tested keywords. Mere guessing is not allowed. Otherwise, in most cases, search by merely guessed keywrods alone is inadequate. There are some exceptions, I concede, including very simple search projects. It does work sometimes, but not that often, and never as well as tested search and multimodal. But most of the times, used alone, which I think most attorneys still do, this is, IMO, a per se unreasonable effort. You should have to ask a judge for permission to do that.
I smell a strawman.
Can you cite one or more of the ‘TAR proponent’s claims that it is “Black Letter law” that courts “order” or “approve” or “require” TAR’ which you allege?
To be clear, the court said in Rio Tinto: “it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it.”
Your juxtaposition of the words “order,” “approve,” and “require” connotes that they are synonyms, which they are not. While one might argue that “permit” in the court’s opinion means “approve” and that the court’s opinion is an “order,” albeit a stipulated one, it would be tortured reading indeed to infer that the court had suggested that the use of TAR was or should be “required.”
If any proponents of TAR are advancing this tortured interpretation, please name them.
Whereas “GVC” asserts that I propose a straw man—an argument concocted to be dispelled easily—GVC’s criticism is more accurately this: what I wrote “presented false impressions.”
It’s been a while since I’ve been accused of such things.
Well, several weeks, anyway.
Back then the accuser was Dr. Gordon V. Cormack. Yes, the prolific author who holds an active learning related patent and claims a trademark over the phrase “Continuous Active Learning” (“CAL”).
Among Dr. Cormack’s various written complaints was that a webinar to which I contributed was based upon a “selective reading” and “presented false impressions.” More specifically, Dr. Cormack complained that the Webinar’s analysis of the 2012 Da Silva Moore court improperly focused only on definitions and articles to which the court had access in 2012 rather than definitions and articles Dr. Cormack wrote after 2012. Obviously, with respect to what the court could have known in 2012, whatever Dr. Cormack subsequently published is irrelevant. Consequently, our failing to discuss Dr. Cormack’s subsequent publications was not a manifestation of “selective reading” or “presenting false impressions.”
It was historically accurate.
As I mentioned in my response to Dr. Cormack’s criticism—all available here http://www.aceds.org/news/304124/Reconsidering-Dr.-Cormacks-Considered-Response.htm —I take Dr. Cormack’s criticisms seriously because they are indictments of my and the other panelists’ intentions, intellect and integrity.
Yet even though he received my response, Dr. Cormack failed to apologize or even to respond.
Whereas Dr. Gordon Cormack asserted that the webinar “presented false impressions” because it focused on topics he seems to prefer we didn’t, “GVC” asserts “I presented false impressions” in my comment regarding Ralph’s post.
Generally, my current comment on this e-discoveryteam site celebrated Ralph’s timing in offering the TAR industry some good news soon after both Gartner and Judge Peck published some sobering news, at least relative to what some TAR proponents have previously claimed.
The comment’s clause which drew “GVC’s” criticism did not, as GVC asserts, present “order” or “approve” or “require” as synonyms. Instead, those words appeared in various articles, blogs, press releases, etc., to describe the then current level of judicial acceptance of Technology Assisted Review (by its various names).
You, too, can find various examples of this by employing Google to search: “’[VERB]’ ‘[NOUN]’” where:
VERBs include separately “court orders,” “court approves” and “court requires“ and
NOUNs include separately “predictive coding,” “TAR,” “Technology Assisted Review,” and (the recently trademarked) “CAL”.
Then, if you need additional confirmation, please search for: “’[NOUN]’ ‘Black letter law’.”
I decline to list all the results here not only because there are so many such references but because my doing so may call out “GVC’s” competitors.
So, “CVG”—whoever you are—do you withdraw your criticism that I employed a misleading straw man argument?
Or is this just another fire-and-forget attack?
And, going forward, wouldn’t it be more productive to reach out and ask for clarification before you criticize using terms that prudent people employ only after positions are clearly delineated and, even then, only after pause?
It’s not like you fellows to put on blinders to the future. Any affordable technology that pulls away as clearly superior to its alternatives in performing a particular task is going to establish a de factor standard of care when examining the reasonableness of conduct. Do you think a doctor who orders only a conventional x-ray to assess a brain bleed after a serious head injury will be blameless when the patient expires from intracranial pressure prompted by subdural bleeding? Or that tugboats can choose not to have radios? Technology raises the bar on conduct, and it should.
I concede that the unwarranted presumtion of competence and good practice in Principle Six is the Get Out of Jail Free card for poor performers. That’s where we are today and that’s where certain interests will push the law so long as others sleep at their switches. But, to make blanket pronouncements that no one should ever be required to use a particular technology or methodology for search is to ignore the obligation to act with competence (which may return to the practice of law, possibly even in our lifetimes).
Until there is reason to anticipate competence in lawyers with respect to ESI and EDD, opponents must be able to learn how a search was conducted and, yes, even to ask the Court to require that alternatives be undertaken. It’s pointless to empower courts to determine the scope of discovery if they cannot also order the methods because, in a world of ESI, scope and methods are inextricably woven together. Searching a corpus without properly preparing it for search and using the appropriate means to conduct the search isn’t really searching the corpus.
You guys must live in a bubble where this is done well most of the time. Where I live, it’s rarely seen. Where most lawyers practice, it’s the Loch Ness Monster carrying Bigfoot on its back.
So, gentlemen, get real. OF COURSE, Courts will order specific search efforts when it becomes clear that one is vastly superior to another. To do otherwise would be to reward and excuse the incompetent. We already have enough prominent “standards” groups that do that.
Thanks for your, as usual, colorful comments. I happen to agree with you at core, as you know.
It is a matter of timing. I don’t think from my perspective that the profession is anywhere near requiring best practices before screw up. I will keep pushing, in my own way, as you will in yours. We are headed in the same direction .
Perhaps it’s because visions of Bigfoot carrying the Loch Ness Monster obscure a clear view, but while your comments, Craig, are interesting they seem unrelated my comment about Ralph’s original post.
Initially Ralph announced that his team’s hard work reaffirmed the importance of document review’s workflow marshaling people and technology.
My comment simply sought to highlight Ralph’s (“Nine Key Points”) (re)stating that competent document review centers not only software but also upon the techniques by which people employ software.
Are we to understand that you, Craig, think that this is a bubble-headed, unreal, blanket pronouncement?
Fair point, Bill. I threw you into the bath water unnecessarily. My point is that courts will eventually order better tools and/or techniques, once the superiority of alternatives is clear and it’s equally clear a producing party is wedded to a subpar methodology.
de facto, of course.
What Craig said.
I think that it is a misreading of Sedona principle 6 to say that it absolves the producing party of the responsibility to use a reasonable method, but I have heard it interpreted that way. The principal says that the producing party is “best positioned.”
I would argue that there is onus on the producing party to use a reasonable method, just and there is onus on a physician to prescribe a reasonable treatment. Of course the attending physician is “best positioned” to prescribe that treatment, but may need to engage a specialist, if his or her core competency is inadequate to the task.
I would further argue that outcome is one indication — but only one — of whether or not a reasonable treatment has been applied. Sometimes it is infeasible to measure the outcome for a particular patient (e.g. 5-year survival rate), and sometimes the outcome is not good, in spite of the treatment having been reasonable, or vice versa.
Finally, clinical studies stating that that “radiation is more effective than grapefruit for treating cancer” cannot reasonably be construed to advocate sticking the patient’s head in a microwave oven. Untoward results from the microwave treatment do not contradict the studies, and certainly do not indicate the efficacy of grapefruit.
We certainly need more studies and more research into what constitutes a reasonable discovery method, and how to prove it. In the meantime, I’d go with the radiation, not the grapefruit. And I’d chose Dr. Hibbert (goo.gl/ITxJWr) over Dr. Nick (goo.gl/SFz76E).
Great comment thanks. Stick you head in a microwave. Very colorful. Here are the doctor images you referred to. You Canadians are so funny.