e-Discovery Team

Predictive Coding Rejected in Las Vegas, But Only Because It Was Chosen Too Late.

There is a new federal court opinion in Las Vegas where predictive coding was rejected because the plaintiff suggested the protocol too late. The irony is Magistrate Judge Peggy A. Leen made clear in her decision that she would gladly have approved predictive coding, if only it had been suggested in a timely manner. Progressive Cas. Ins. Co. v. Delaney, No. 2:11-cv-00678-LRH-PAL, 2014 WL 2112927 (D. Nev. May 20, 2014). This is another two-part blog where I will analyze the many important lessons in this case. My goal is to help everyone to avoid the “if only” Vegas blues.

An Ill-Conceived Agreement to Use an Outdated Search Protocol

Plaintiff’s counsel in Progressive made an agreement with defense counsel as to how they would conduct search. They called it a Joint Proposed ESI Protocol. It resolved numerous disputes and Judge Leen entered an order adopting the Protocol. The agreement required the plaintiff to search through a designated collection of 1.8 million files for ESI requested by defendants. They agreed to conduct the search by using keyword search alone. A set of keywords agreed to by the parties would be run against the entire 1.8 million corpus. The documents containing the keywords would then either be:

1. All manually reviewed for relevance by teams of document reviewers who would screen for relevance, and privilege, and then only produce documents determined by the reviewers to be both relevant and  not privileged documents; or, at plaintiff’s election,
2. All be reviewed for probable privilege, withheld from production and reviewed and logged, will all non-privileged documents produced, without any review as to whether they were actually relevant or not.

In either case, all parties would be protected by clawback agreements and orders whereby privileged documents could be clawed back upon request of the producing party. Plaintiff’s counsel planned to do the 1.8 Million document search and review by, get this, using Summation software.

These kind of keyword search agreements are entered into every day by lawyers in thousands of cases across the world, especially in the U.S. where most of the world’s e-discovery takes place. In fact, defense counsel in the instant case later defended the reasonability of this agreement by pointing to five very similar law suits where a nearly identical protocol had been used. These agreements are common, but that does not make them right. This kind of simplistic keyword protocol can work in small cases, and even in many medium size cases, but in a large document case like this, not a chance. In a large document case they are an ineffective waste of time and money. They are unreasonable. Way too risky. If you do it just because all of the other lawyers did so, then you are just like an ill-fated lemming jumping off a cliff. You better have a parachute, but, as this case shows, sometimes it is too late to deploy a back-up plan.

Any informed, technologically savvy lawyer today knows that keyword search alone misses many documents, and includes many that are irrelevant. They also know that manual review is even worse. Plus, this kind of outdated approach to legal search in a large documents case is outrageously expensive. It is foolish to jump off that cliff.

All technology professionals in the electronic discovery industry know this, at least those who are taking the time to stay current. All of my readers know this very well.  They also know about the bubble. We all know by now that we technologically adept lawyers are still but a small enlightened minority. We practice law in a science based, multifaceted AI bubble. All around us is a monochrome legal world where lawyers still believe in magic keyword incantations.

Twentieth Century lawyers were not exactly caveman, but they did still believe in magic keyword incantations. I remember. I was there. I was a believer. That was the latest technology in e-discovery near the end of the last century. Similarity searches were just emerging. Concepts searches had not yet appeared on the scene. Predictive coding was completely unknown then to all but a few information scientists, none of whom had even considered applying it to legal search. Back then some of us (not me) also thought that low paid contract lawyers reading emails all day was a great new innovation, one that would somehow find relevant evidence because lawyers were eyeballing each document. What a deluded thought that proved to be. Paying high salaried associates to do document review all day was not any better.

That was then, the 1990s. We have learned a lot about legal search in the past twenty years, especially search using AI software against big data. The science is now in. The research shows that keywords and manual review are ineffective. It shows that humans are not the gold standard we once thought. In fact, when it comes to large-scale document review, we are terrible at it. We are not even consistent coders with the exact same documents. We are not wired for such tedious tasks, lawyers or not, and high paid or not. Only machines can work tirelessly and with perfect consistency. The well programmed machines, with good human trainers, are far superior to manual review and keywords in every way

The odd thing about the Progressive opinion is that Judge Leen understand all of this. She only denied the use of predictive coding because she did not like the timing or way it was presented to her. Despite the holding, Progressive is actually yet another pro-predictive coding opinion. Here are two take away quotes to prove it:

[T]he traditional ways lawyers have culled the universe of potentially responsive documents for production—manual human review, or keyword searches—are ineffective tools to cull responsive ESI in discovery. Predictive coding has emerged as a far more accurate means of producing responsive ESI in discovery. Studies show it is far more accurate than human review or keyword searches which have their own limitations. . . .

Had the parties worked with their e-discovery consultants and agreed at the onset of this case to a predictive coding based ESI protocol, the court would not hesitate to approve a transparent mutually agreed upon ESI protocol. However, this is not what happened.

What happened in Progressive is that the expert consultants were brought in too late. By the time they arrived, the bed, messy as it was, had already been made. The plaintiff was required to sleep in it. It was too little, too late.

Bring In the Experts

This case proves once again that most lawyers are still only comfortable with keyword search. That is the only kind of search they know how to do, so that is what they agree to do. That is, until they, or sometimes the client, realize that their magic keywords do not work, that the whole project is an outrageously expensive mess. Then, out of desperation, the better ones bring in a real expert, probably one of my readers, who diplomatically tells them they are going about it all wrong.

They explain that there is much more to search than negotiating keywords and leading vast armies of contract lawyers. They show them a better way guided by advanced software. Of course, the worst lawyers, the ones in a state of total delusion, never call an expert at all. Their ego, or possibly greed, gets in the way. Instead, they push on unassisted. They dig their clients into a deep hole of debt continuing on in a fools journey of keyword and manual review.

To the credit of plaintiff’s counsel in Progressive Cas. Ins. Co. v. Delaney, they fell into the former group, not the later. They tried to reform, it just did not work out very well because they acted a bit too late. Also, for reasons not well articulated in the opinion, the judge got the idea that plaintiff’s counsel was not cooperative or transparent enough in disclosing their proposed predictive coding protocol.

The moral of this case is that you have to come to the table early with a predictive coding protocol. If you have a big data case, bring him or her into the case right away. Do not wait to bring in a predictive coding expert. Never enter into a search protocol without their input. In fact, just let them do their job and delegate this task to them. Never try to muddle though on your own with the thought that you can always hire an expert later, if need be. That is penny wise and pound foolish as Progressive proves.

The facts of Progressive also show that most lawyers today, even top litigation attorneys in big law firms, do not understand the need for legal search experts. Most are not current on the latest search techniques. They may not even understand a basic search pyramid. They may think that the only way to search is by keywords. Bottom line, they are way behind the times. They are lawyers assigned to handle big data cases with small data skills. They think that they know it all, whereas in fact, everything they know is wrong.

I cannot tell you how many events I have been to where I have tried to spread the word on multimodal search guided by active machine learning, a/k/a Artificial Intelligence. I have also taken the time to write dozens of articles on the subject. See eg. LegalSearchScience.com and my Computer Assisted Review page. So too have several of my colleagues who specialize in legal search. Predictive coding AI enhanced multimodal search is Twenty First Century Search. It is the only effective way to handle search in big data cases like Progressive Cas. Ins. Co. v. Delaney.

This is not just my opinion, it is the consensus of the vast majority of legal search experts in the world, especially the information scientists who are not beholden to vendors. I am certainly not the only one out there trying to spread the word on this either. There are a couple of dozen lawyers out there with me writing and traveling around the CLE circuit to teach the merits of multimodal predictive coding. We are trying to help lawyers into the Twenty First Century of legal search. We are trying to help them to go beyond last century’s reliance on keyword search alone.

What makes the Progressive case all the more interesting is that a prominent predictive coding expert was brought into this case, but he was brought in too late. He was brought in after the attorneys in Progressive entered into their ill-conceived discovery agreement. I have been in this kind of position myself, where I was brought in late in the game. It is very difficult. It is far easier when, as in Da Silva Moore, predictive coding is proposed at the very beginning of the case.

To be continued ...

This entry was posted on Sunday, June 8th, 2014 at 5:27 pm and is filed under Lawyers Duties, Review, Search, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.