e-Discovery Team

This blog will report on a new article by Judge Andrew Peck endorsing the use of artificial intelligence and a recent speech by Jason R. Baron warning of a coming digital dark age of information overload. The two events, much like 7 of 9, make a compelling argument for law to embrace technology, not run from it.

Judge Peck’s Article on Predictive Coding

Judge Peck’s article brings good news to all those stressed by the costs and difficulties of e-discovery. The article shows that artificial intelligence can help alleviate this stress when it is used judiciously. Predictive Coding: Reading the Judicial Tea Leaves, (Law Tech. News, Oct. 17, 2011).

Judge Peck reviews the old days of linear paper review and is glad to see them go. He cites to the study by Herb Roitblatt, Ann Kershaw and Patrick Oot, that suggests computer assisted review is at least as accurate as manual review. Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, Journal of Am. Society for Information Science & Technology, 61(1):70-80 (2010). Then he cites the study by Maura Grossman and University of Waterloo professor Gordon Cormack, using data from the Text Retrieval Conference Legal Track, which concluded that “[T]he idea that exhaustive manual review is the most effective — and therefore the most defensible — approach to document review is strongly refuted. Technology-assisted review can (and does) yield more accurate results than exhaustive manual review, with much lower effort.” Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Richmond J. of Law & Tech., Vol. XVII, Issue 3, 1-48 (2011).

Next, Judge Peck addresses search based on keywords and shows how it is a rather primitive, last-century application of computer assistance, especially when applied without sampling, testing, and transparency.  He uses my Go Fish analogy to illustrate how even this simple technology is not being used correctly by many attorneys.

Because the volume of ESI has made full manual review virtually impossible, lawyers have turned to keywords to cull ESI (particularly e-mail) for further (manual) review. A basic problem is that absent cooperation, the way most lawyers engage in keyword searches is, as Ralph Losey suggests, the equivalent of “Go Fish.” The requesting party guesses which keywords might produce evidence to support its case without having much, if any, knowledge of the responding party’s “cards” (i.e., the terminology used by the responding party’s custodians). Indeed, the responding party’s counsel often does not know what is in its own client’s “cards.”

Judge Peck also reviews the case law in this area and his excellent “wake up call” contribution to this jurisprudence:

Judicial decisions have critiqued keyword searches. Important early decisions in this area came from magistrate judges John Facciola (District of Columbia) and Paul Grimm (Maryland). See United States v. O’Keefe, 37 F. Supp. 2d 14, 24 (D.D.C. 2008) (Facciola, M.J.); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331, 333 (D.D.C. 2008) (Facciola, M.J.); and Victor Stanley, Inc. v. Creative Pipe, Inc ., 250 F.R.D. 251, 260, 262 (D. Md. 2008) (Grimm, M.J.).

I followed their lead with William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134, 136 (S.D.N.Y. 2009) (Peck, M.J.).

“This Opinion should serve as a wake-up call to the Bar in this District about the need for careful thought, quality control, testing, and cooperation with opposing counsel in designing search terms or ‘keywords’ to be used to produce e-mails or other electronically stored information (‘ESI’),” I wrote.

After this good set-up, Judge Peck gets to the meat of his essay, predictive coding and why lawyers should embrace this new technology. First he makes a stab at explaining what this new technology is all about:

By computer-assisted coding, I mean tools (different vendors use different names) that use sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e., training by) a human reviewer.

Unlike manual review, where the review is done by the most junior staff, computer-assisted coding involves a senior partner (or team) who review and code a “seed set” of documents. The computer identifies properties of those documents that it uses to code other documents. As the senior reviewer continues to code more sample documents, the computer predicts the reviewer’s coding. (Or, the computer codes some documents and asks the senior reviewer for feedback.)

When the system’s predictions and the reviewer’s coding sufficiently coincide, the system has learned enough to make confident predictions for the remaining documents. Typically, the senior lawyer (or team) needs to review only a few thousand documents to train the computer.

Then Judge Peck addresses one reason that many lawyers are reluctant to use computer assisted analysis and review, that there is no judicial opinion that expressly approves of computer assisted review. He first asserts that no opinion expressly approves of keyword search either, in fact many criticize it, but attorneys use it anyway. I have to disagree with him slightly on that. Many cases by implication endorse keywords, at least if properly done, and one opinion by Judge Thomas F. Hogan enthusiastically endorses it.

… the glory of electronic information is not merely that it saves space but that it permits the computer to search for words or ‘strings’ of text in seconds.

In re Lorazepam & Clorazepate, 300 F.Supp.2d 43, 46 (D.D.C. 2004). To this I would add that the glory of electronic information is not merely that it permits keyword searches, but that it allows us to harness the artificial intelligence of computers to boost our own.

Judge Peck then address the Daubert straw-man that Judge Facciola unintentionally interjected into the discussion concerning the use of computer intelligence:

In addition to reluctance to be the guinea pig for a decision on predictive coding, lawyers perhaps are concerned that they will have to go through a Daubert hearing as to the “admissibility” of the results of predictive coding. Perhaps this fear comes from O’Keefe, where Judge Facciola said that opining on what keyword is better “is truly to go where angels fear to tread,” and is a topic “beyond the ken of a layman and requires that any such conclusion be based on evidence that, for example, meets the criteria of Rule 702 of the Federal Rules of Evidence,” dealing with expert opinions.

Lawyers’ fears in this regard seem largely misplaced. First, Facciola’s comments were directed at keywords, but everyone is using keywords, and I know of no decision after O’Keefe requiring expert testimony as to the use of keywords.

Second, with due respect to Facciola, I do not think Daubert applies — it applies when an expert will testify at trial in order to admit into evidence opinions or results (e.g., the result of DNA testing reveals a match).

Here, the hundreds of thousands of e-mails produced are not being offered into evidence at trial as the result of a scientific process. Rather, whether the handful of e-mails offered as trial exhibits is admissible is dependent on the document itself (e.g., whether it is a party admission or a business record), not how it was found during discovery.

Very good points Judge Peck. He then goes on to articulate standards for the judicious use of predictive coding, including sampling, cooperation, and transparency. You should read the article in full and study this part. He concludes his article with these reassuring words:

Until there is a judicial opinion approving (or even critiquing) the use of predictive coding, counsel will just have to rely on this article as a sign of judicial approval. In my opinion, computer-assisted coding should be used in those cases where it will help “secure the just, speedy, and inexpensive” (Fed. R. Civ. P. 1) determination of cases in our e-discovery world.

Artificial Intelligence and the Law

There you have it. Now at least one judge has endorsed the use of artificial intelligence to assist in the discovery and production of relevant evidence. I predict there will be many more. It is only common sense. We use computers for everything already. To paraphrase Judge Hogan, the glory of electronic information is that it allows us to harness the artificial intelligence of computers to boost our own.

Don’t get me wrong. I don’t think that AI is a panacea. Well trained lawyers are still indispensable. Ever heard the expression garbage in, garbage out? There is no substitute for skilled human judgments, but now our limited human intelligence and processing speed can be enhanced with predictive coding and other forms of computer assisted document review and analysis. As Judge Peck mentioned, new software is now entering the market that can help us to do that. This software is usually part of early case assessment tools, or built into review platforms, or both (my preference). The programs allow you to harness the power of the computer to assist in your searches, to amplify and expand upon your determinations of relevance, privilege and other issues. It is commonly called predictive coding, or technology assisted reviews, but I prefer to call it by the more general terminology of artificial intelligence, for that is what it is (“the study and design of intelligent agents”).

Two Ways to Use Predictive Coding

There are really two different ways to use AI in e-discovery today: 1. to search and find information, i.w. to cull-out; and, 2. to speed up review prior to production, i.w. quality control. Many people, even experts, get confused between the two and Judge Peck’s article does not address the distinction. It is important to understand the differences. The first is now well accepted as a replacement for manual culling and as a supplement for keyword culling. The second is also accepted by most experts as a good way to speed up manual review, but is not accepted as a total replacement for human review. The technology is not there yet, and neither is legal protection for inadvertent productions of privilege and confidential ESI. Sorry, but the New York Times article is wrong: Armies of Expensive Lawyers, Replaced by Cheaper Software. See: NY Times Discovers e-Discovery, But Gets the Jobs Report Wrong where I point out that the advancement of AI and the law will lead to more and better skilled jobs for lawyers, not less.

The first way to use predictive coding is to use the AI of the computer to help you select the ESI that is probably relevant from an unsorted larger mass of data.  This is ESI that is usually gathered from one or more custodians – their email and attachments, and their loose files. In this first, intake stage you use predictive coding to carve out the ESI in bulk, to cull out the probable irrelevant and cull in the probable relevant. This is using AI for search. So, for instance, as Judge Peck describes, an expert, a human expert, reviews and analyzes a few thousand documents and marks them as either relevant or irrelevant. If marked relevant, the expert also determines whether they are highly relevant, or just marginally so, whether they are privileged or not, or whether they pertain to certain issues, or confidentiality rankings. The computer then studies the documents marked, and using its programmed AI the computer applies what it has learned about relevance, issues, privilege and confidentiality to all of the remaining data.

The computer acts as an intelligent agent and uses what you taught it by your analysis to find the documents that it thinks are likley to be relevant, the issues they are relevant to, privileged, and confidential. It does this over billions of files and it does this very, very fast. The lawyer expert then studies the results and fine tunes the search. He or she teaches the computer some more. The process repeats according to budgetary constraints, and perhaps also precision and recall criteria, until the lawyer is satisfied that a good collection of probably-relevant-documents has been sorted from the larger whole. That is stage one, the culling search and intake stage, but it certainly does not end there. You do not simply produce all of the documents that you and the computer have found through sampling and AI to be probably relevant.

The state our of legal intelligence using this new software, although very high because lawyers now have an intelligent computer agents to assist, is still limited. Mistakes are still made. Guaranteed. If you have attained a precision ratio of 75% you are doing very good. But that means that 25% of the documents you have identified in stage one as likely relevant are not in fact relevant. That number may be as high as 50%. Regardless of the specific ratio, no system is anywhere near perfect at this early stage of AI development.

Your set of  probably-relevant ESI will contain many, many documents that are not. Further, it will contain privileged and secret documents that you are under no obligation to produce. These documents may cause irreparable damages if disclosed. That is where the next stage of human review before production comes in, a review that is again, or should be, computer assisted by predictive coding, including clustering of like documents. The second review stage is essentially a quality control function to weed out irrelevant ESI, identify highly relevant, and withhold privileged and confidential ESI.

Some requesting parties, especially those with little or no ESI of their own, think that the producing party should just skip this second stage and produce without review. Or that the second stage can be expedited by sampling and only some of the documents reviewed. Wrong. We are not there yet. The technology is not advanced enough yet for that. Moreover, despite some limited success with clawbacks and Rule 502, the law does not yet provide strong enough protections to the producing party for the inevitable errors in stage one.

The requesting party is only entitled to see the other side’s documents if they are relevant. That protects the requesting party from a document dump (something the government is famous for) and the producing party from a fishing expedition. What if the attorney for the requesting party has many different law suits for different clients against the requesting party? A document irrelevant for one suit might be highly relevant for another. It might even give the requesting attorney an idea for future law suits. Some large plaintiffs firms are known to go out and recruit plaintiffs for potentially lucrative class action litigation.

Further, privileged documents may be missed by the thousands in stage one culling, which is relevancy driven. Privileged communications and work product are missed in distressingly large quantities, even after a thorough stage two review. They would be missed in far greater quantities without it. That is the real game stopper. Once the bell has been rung, it can’t be unrung. Forced lobotomies are illegal is this country. The attorneys and requesting parties on the other side will know your secrets, both attorney client and work product, even of they do return your documents in a clawback.. That provides them with an unfair advantage, which, depending on the secrets, can be very significant. Then there is the whole question of confidential information, from key trade-secrets to very personal data. If not properly labeled or redacted, the secrets may be lost forever.

No. AI has not progressed to the point where we can dispense with stage two review of documents, all of the documents. Stage two cannot yet be performed with confidence based on sampling alone. But the computer with predictive coding and clustering can significantly speed up the manual review. We can go from fifty files per hour, to several hundred files per hour, at least in some cases where the ESI is fairly simple and not too dense. But the state of technology and law today still requires eyeballs on all ESI before it goes out the door and into the hands of the enemy. The negative consequences of disclosure of secrets, especially attorney-client privilege and work product privilege secrets, is simply too high. This will likely change in the future as judges start to put some real teeth into Rule 502 and technology improves, but we are not there yet.

The Dark Vision of Jason Baron

Jason Baron is a pioneer in this area, especially in the stage one use of AI for search, as all readers of this blog well know. But did you also know he was the 2011 Emmett Leahy Winner for excellence in records and information management? It is a big deal in the world of archivists and records managers. Take a look at the YouTube video below of Jason’s acceptance speech. Suggest you also watch the excellent introduction by Judge John Facciola. The several other intros are not that interesting for us non-librarian e-discovery types, so skip right to Judge Facciola at 1523 – 3313. John captures Jason’s playful creativity as an essential component of his intelligence. After that, I suggest you skip again to Jason’s speech at 4212-5845. It is good, even if it does end with a dark vision of a post-info-apocalyptic world. Apparently getting a big international award brings out the best in a person.

In his speech, after thanking everyone imaginable, Jason has three messages:

1. He calls for the end of the end-user being expected to act as the de facto records manager. He calls tongue-in-cheek for the workers of the world to unite in opposition to tedious records keeping enslavement. We must instead automate this work. We must turn these tedious records-filing tasks over to artificial intelligence. Amen to that!
2. He calls for more creative interdisciplinary action, which I might point out is also the theme of this blog with its emphasis on the team approach. Specifically, Jason calls upon lawyers to work as a team with experts in the fields of information retrieval and artificial intelligence. He calls upon the legal profession to partner with academia and industry and adopt standards for best-practices in legal search.
3. Jason then concludes with a disturbing vision that keeps him up at night. He warns of a coming dark age of digital flooding, of hopeless unwieldiness, where we all drown in too-much-information. He warns of information overload followed by inability to access anything of importance. He call this a vision of information dystopia. He asks us all to be smarter to avoid this catastrophe. Jason ends by promising to do his part with education on the importance of information management to prevent this from coming to pass.

Computers got us into this mess and they can get us out

Computers got us into the mess of too-much-information, too much complexity in forms of information, and too many places to store information. We are in danger of having so much information that no one can find anything. We are in danger of a world of justice where only the rich can afford to litigate. A world so filled with information that no one really knows anything. A world with too much information, but not enough  knowledge, much less wisdom. This is the nightmare scenario that keeps Jason and other deep thinkers awake at night.

Not me. I have other dreams. I have visions of smart computers augmenting our own intelligence. The know-nothing nightmare scenario can be avoided with still more and better technology, with intelligent agents. We must  fight fire with fire by harnessing the new AI capacities of computers. If we boost our own intelligence and abilities with algorithmic agents we will be able to find the evidence we need in the trillions of documents implicated by even average disputes. We have to use AI agents to boost our abilities to read, review, and analyze all of this ESI. Without AI agents we simply cannot find the right stuff for a reasonable sum. With them, we can and will do wonders. We can boost our analytic, reading, and classification speed a hundredfold. We can survive the flood, maybe even surf on it to ever greater records of achievement.

This is not a utopian dream. It will happen. As mentioned, we are not there yet. But it is only a matter of time as the algorithms, the AI, improves, and as our human skills improve. Yes, AI agents alone will not get us there. Advanced software is worthless without advanced attorneys and consultants to use them, and advanced judges to enforce the law. Again I come back to my training mantra, yet the marketplace, so far at least, seems much more interested in certification than education. Still, I remain hopeful. The new day is just beginning.

It is a New Dawn, Not a Sunset

Jason ends his speech by asking us to be smarter to avoid the information dystopia, the information dark ages. But he does not say how are we to get smarter. Here are my two main suggestions and I think Jason would agree. More education and more artificial intelligence. The only way we will ever get smart enough is by having our computers help us. Without AI agents, we will never be smart enough to deal with the deluge and Jason’s dark age will come true.

But not to worry too much, there will be no dark age. There will be smart computer agents to help us think and act smarter. Have you seen the new iPhone 4S with SIRI, the automated personal assistant? Just think what SIRI will be like in a few more years. Just think what the latest software tools in e-discovery will be like. Siri, find me the top ten relevant documents from John Smith’s email.

Sleep well. It is dark now, but we are certainly not approaching a dark age of information overload. The Technology Age is just beginning. AI and the law is just getting started. Thousands of coders around the world are working on this problem right now. A great light is coming. This is the dawn, not the sunset.

The future will not only bring more information, it will bring more processing. The new AI technologies will make us smart and the light will shine on all. Some day stage one search will reach near total recall and near perfect precision. Some day stage two AI assisted review will be so good that sampling will suffice, and production errors will approach zero. Some day the courts will protect producing parties when some errors still take place.

There is already a plethora of evidence out there, and, as Jason often points out, you ain’t seen nothing yet. The exponential curve in data multiplication is just starting. In the near future writings in the general sense of ESI will be even more voluminous and complex than we can now imagine. They will be everywhere. ESI on everything will be ubiquitous, but so will information processing, so will AI agents. Not to worry, the processing will keep up. The intelligence of our computers will increase even more dramatically than the information we generate with them. AI agents will be just as ubiquitous as raw information itself. Siri, pass me the salt.

Conclusion

We we will soon be able to preserve and mine the flood of facts that technology has made possible. We will be able to find the truth much more often and efficiently than ever before.  Witnesses will be assisted, neigh sometimes even forced to tell the truth by these past writings. The flood of truth will evoke a flood of justice. An age of legal enlightenment will follow the advances in information processing and analysis. See: We Are at the Dawn of a Golden Age of Justice. The truth is out there, we just need to learn how to find it. And with the help of Siri and Watson, and human sages like Peck and Baron, we will my friends, we will.

This entry was posted on Sunday, October 16th, 2011 at 8:50 pm and is filed under Evidence, Forensic Exam, Lawyers Duties, Metadata, New Rules, Related Legal Webs, Review, Search, Spoliation/Sanctions, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.