e-Discovery Team

This is the first in a series of narrative descriptions of a legal search project using predictive coding. Follow along while I search for evidence of involuntary employee terminations in a haystack of 699,082 Enron emails and attachments.

Joys and Risks of Being First

To the best of my knowledge, this writing project is another first. I do not think anyone has ever previously written a blow-by-blow, detailed description of a large legal search and review project of any kind, much less a predictive coding project. Experts on predictive coding speak only from a mile high perspective; never from the trenches (you can speculate why). That has been my practice here, until now, and also my practice when speaking about predictive coding on panels or in various types of conferences, workshops, and classes.

There are many good reasons for this, including the main one that lawyers cannot talk about their client’s business or information. That is why in order to do this I had to run an academic project and search and review the Enron data. Many people could do the same. In fact, each year the TREC Legal Track participants do similar search projects of Enron data. But still, no one has taken the time to describe the details of their search, not even the spacey TRECkies in TREC Legal Track.

A search project like this takes an enormous amount of time. In fact, only the 2011 Legal Track TRECkies recorded and reported the time that they put into the project, and even then it was just totals. In my narrative I will report the amount of time that I put into the project on a day-by-day basis, and also, sometimes, on a per task basis. I am a lawyer. I live by the clock and have done so for thirty-two years. Time is important to me, even non-money time like this. There is also a not-insignificant amount of time it takes to write it up a narrative like this. I did not attempt to record that.

There is one final reason this has never been attempted before, and it is not trivial: the risks involved. Any narrator who publicly describes their search efforts assumes the risk of criticism from monday morning quarterbacks about how the sausage was made. I get that. I think I can handle the inevitable criticism. A quote that Jason R. Baron turned me on to a couple of years ago helps, the famous line from Theodore Roosevelt in his Man in the Arena speech at the Sorbonne:

It is not the critic who counts: not the man who points out how the strong man stumbles or where the doer of deeds could have done better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly, who errs and comes up short again and again, because there is no effort without error or shortcoming, but who knows the great enthusiasms, the great devotions, who spends himself for a worthy cause; who, at the best, knows, in the end, the triumph of high achievement, and who, at the worst, if he fails, at least he fails while daring greatly, so that his place shall never be with those cold and timid souls who knew neither victory nor defeat.

I know this narrative is no high achievement, but we all do what we can, and this seems within my marginal capacities.

Desired Impact

I took the time to do this in the hope that such a narrative will encourage more attorneys and litigants  to use predictive coding technology. Everyone who tries this new technology agrees it is the best way yet to find evidence in an economical manner. It is the best way to counter all those who would use discovery as abuse, and not a tool for truth. See, eg.:

• Discovery As Abuse 
• E-Discovery Gamers: Join Me In Stopping Them
• Judge David Waxse on Cooperation and Lawyers Who Act Like Spoiled Children

Predictive coding can finally put an end to this abuse. We can use these methods to search large volumes of ESI in a fast, efficient, and economical manner. We have to do this. It is imperative because the volumes of ESI continue to grow, and, along with this flood, the costs of discovery continue to spiral out of control. Despite all of our efforts at cooperation and professionalism, there are still far too many attorneys out there who take advantage of this situation and use discovery as a weapon to try to force defendants to settle meritless cases. See, e.g.:

• Bondi v. Capital & Fin. Asset Mgmt. S.A., 535 F.3d 87, 97 (2d Cir. 2008)”This Court . . . has taken note of the pressures upon corporate defendants to settle securities fraud ‘strike suits’ when those settlements are driven, not by the merits of plaintiffs’ claims, but by defendants’ fears of potentially astronomical attorneys’ fees arising from lengthy discovery.”)
• Spielman v. Merrill Lynch, Pierce, Fenner & Smith, Inc., 332 F.3d 116, 122-23 (2d Cir. 2003) (“The PSLRA afforded district courts the opportunity in the early stages of litigation to make an initial assessment of the legal sufficiency of any claims before defendants were forced to incur considerable legal fees or, worse, settle claims regardless of their merit in order to avoid the risk of expensive, protracted securities litigation.”)
• Lander v. Hartford Life & Annuity Ins. Co., 251 F.3d 101, 107 (2d Cir. 2001) (“Because of the expense of defending such suits, issuers were often forced to settle, regardless of the merits of the action. PSLRA addressed these concerns by instituting . . . a mandatory stay of discovery so that district courts could first determine the legal sufficiency of the claims in all securities class actions.” (citations omitted))
• Kassover v. UBS A.G., 08 Civ. 2753, 2008 WL 5395942 at *3 (S.D.N.Y. Dec. 19, 2008) (“PSLRA’s discovery stay provision was promulgated to prevent conduct such as: (a) filing frivolous securities fraud claims, with an expectation that the high cost of responding to discovery demands will coerce defendants to settle; and (b) embarking on a ‘fishing expedition’ or ‘abusive strike suit’ litigation.”)

Follow me now while I search for relevance in the ashes of Enron.

699,082 Enron Documents: the Ashes of a Once-Great Empire

My search was of the 699,082 slice of the Enron database put together by EDRM. It is the V2 version that was processed by ZL Labs for EDRM. It was deduplicated at the custodian level, in other words, vertical deduplication, not horizontal. Specifically this EDRM Enron dataset includes:

• EDRM Enron PST Data Set: Enron e-mail messages and attachments organized in 32 zipped files, each less than 700 MB in size, containing 168 .pst files. The total size of the compressed files is approximately 19 GB. The total size of the uncompressed files is approximately 43 GB.
• EDRM File Formats Data Set: 381 files covering 200 file formats.
• EDRM Internationalization Data Set: A snapshot of selected Ubuntu localization mailing list archives covering 23 languages in 724 MB of email. (Note: in this review project I arbitrarily called any foreign language documents irrelevant, and did not consult with translators.)

I conducted the search using Kroll Ontrack (“KO”) Inview software. KO was kind enough to provide the EDRM Enron Data, software, and hosting without charge. Any failures or mistakes are to be attributed to me, not them, and certainly not their software. KO left me alone to do whatever, and only provided input on one occasion upon my request, which is later documented in Days Five and Six of a Predictive Coding Narrative: Deep into the weeds and a computer mind-meld moment.

This 699,082 slice of the Enron database looks like a somewhat random selection of emails and attachments, not unlike what you would find by review of the PST files from a number of key custodians. It went in time from the late 90s, when the company was doing great, and involuntary terminations were rare, to its eventual dissolution. In the early emails life for the 20,000 Enron employees was good and their email reflected that. Enron was growing and hiring. It was one of the hottest companies in America with revenues of over $100 Billion. But all of that changed rapidly near the end of the company, when it fell into bankruptcy in late 2001 and the emails slowly came to an end.

The date range in this Enron data collection, excluding a few outliers, ranged from January 1, 1997 to November 30, 2002. Here is a screen shot of one of the oldest and newest emails in the collection.

 .

My search for involuntary employee termination related documents led me to focus on the final sad months when the empire fell apart, and ultimately the company itself was dissolved. The search involved fine relevancy distinctions between voluntary and involuntary terminations, with only involuntary being relevant. Such distinctions are common to all search projects, and this was no exception.

You would have to be a cold human indeed not to feel some of the pain of the thousands of people who ended up losing their jobs, their incomes, their retirement savings, because of the dastardly behavior of a few bad apples at the top. I have read their Enron email, which included many personal notes to family, friends, and even lovers. I have seen family photos, read their jokes and inspirational messages. I have even seen their porn and their cursing in anger. It is all there in the email, their life.

I have intruded into their privacy, uninvited, and unwelcome. For that I almost feel a need to apologize, but this is now public data, and my purpose was an academic study, not commercial or personal exploitation. Still, out of respect for the hundreds of people whose privacy I have necessarily invaded by the search of 699,082 emails and attachments, I will not include any specific information in this narrative about these people and their lives. I owe them that much, and anyway, it seems like the decent thing to do. I do not think the omission in any way detracts from the value of the narrative.

Learn By Watching, Then Doing

The original point of the exercise was to provide training to a group of my firm’s e-discovery liaisons who attended a KO training session in Minnesota. We trained by a demonstration of the use of the Inview software to respond to a hypothetical RFP. During the exercise emphasis was placed on use and description of the predictive coding features.

The feedback from my liaisons was that this was a good way to learn. This is not surprising because  lawyers typically learn best by doing, and before we do something for the first time, in an ideal situation at least, we usually observe someone else who already knows how to do it. Any good law firm will, for instance, have a new associate watch an experienced partner take a few depositions before they let them take a deposition on their own. It is part of the legal apprenticeship program and one reason we call it the practice of law.

Overview of Efforts

I conducted this search over the course of eight workdays in May and June 2012. At the end of each day I sent out a description of what I had done. All of the lawyers in the training were invited to log onto the database and follow along. I have since edited these daily reports into a single narrative, all for general instruction purposes.

This Search is Just One Example Among Many

This narrative shows one example of the use of predictive coding in a typical legal setting. It is just one illustration, and many alternative approaches could have been followed. Indeed, if I were to do this over again, I would do many things different now that I have the benefit of 20/20 hindsight. Also, my knowledge of this particular software, Inview, has improved since this relatively early experiment, especially the ins and outs of how its predictive coding features work. It was, however, not my first such experiment with KO’s Inview, not to mention my work with other vendors’ software, each of which works slightly differently. As they say in Texas (and central Florida), this was not my first rodeo. Still, if I rode this particular bull again, I would do it differently. And, on any one day, I am sure that there are any number of people who could do it better, including many of my readers.

Best Practice

I do not contend that the particular search efforts here described were the best possible way the search of this data for this purpose could have been performed. Moreover, I readily admit that it was not even close to a perfect process. Perfection in legal search is never possible by anyone with any software. Perfection is never required by the law, in search and review, or anything else. I do contend, however, that the efforts here described constituted a reasonable search effort. It should, therefore, withstand any legal challenge as to its adequacy, since the law only requires reasonable search efforts.

Having said all that, to be honest, I think that the search here described was fairly well done. Otherwise I would not waste the reader’s time with the description, nor use this narrative for instruction. Since I specialize in this stuff, and am considered an expert in legal search, particularly predictive coding, I would go a bit further  and claim that my efforts were more than just adequate. (A quick footnote on my qualifications: I have over 30 years of experience searching for ESI on computers, a pending patent on one legal search method, and I am a published author and frequent speaker on the subject.) Right now predictive coding, and related legal doctrine and methods such as proportionality and bottom line driven review, are my primary interest in e-discovery. You could say I am obsessed and literally talk about it all of the time. See eg.:

Based on my background and experience, I think it is fair to contend that the search conducted was more than a mere legally adequate effort, more than just a reasonable effort. I would argue that it constitutes an example of a best practice of search and review. It qualifies as a best practice (as opposed to best possible) for two reasons: (1) advanced, predictive coding based software was used; and, (2) the search was conducted by a qualified expert. Still, the particular details and methods used in the search described in the narrative are just one example of a best practice; one among many possible approaches. Also, it is certainly not a standard to be followed in every search. It is important not to confuse those two things. Standards are more general. They are never single-case specific. They are never reviewer specific. So, after all of this long introduction, we finally come to the search narrative itself.

Come, Watson, come! The game is afoot.

The review began by judgmental sampling. I just looked around the 700-thousand documents to get a general idea of the types of documents in the dataset, the people involved, the date ranges, and the kinds of subjects their email addressed. This could have been done with reports generated by Inview, but I choose to do so by displaying all of them, and sorting the display in various ways, including one of my favorites, display by file types. That allowed very easy viewing of the underlying documents whenever I wanted.

Another good method I could have used was the Analytics view with graphics displays providing visual information about the data. This includes a pull down menu where you can review certain files types. Or you can review by custodian with a visual display of who is emailing who. Or you can view by graphical displays of date ranges. These kind of visual displays of ESI contours are now common in most advanced software.

I also ran a few obvious, and some not so obvious, keyword searches pertaining to employee termination and other things.

By just looking around as I did, and running a few easy keyword searches, I found some relevant documents, and many more irrelevant ones. Although I was just beginning to familiarize myself with the data, I went ahead and coded some documents where it was obvious they were either relevant or irrelevant; the lowest hanging fruit, if you will. I coded 412 documents in this manner.

I call this judgmental sampling because I was using my own judgment to select and review small samples of the overall data. Before I began the predictive coding search process, I would also include random sampling, as this is core to all predictive coding methods. But, as is usual for me, I started here with judgmental sampling.

By the way, although I will be very transparent here, I am not going to tell you exactly everything I did. I am going to save a few trade secrets, a little bit of secret sauce, such as exactly what keyword searches I ran at the very beginning. As Maura Grossman likes to call such disclosure, its translucent, not transparent. I hope you understand.

Category Coding

I designed only five coding choices in this exercise:

• Irrelevant,
• Undetermined (relevancy),
• Relevant,
• Highly relevant (a sub-category of Relevant),
• Privileged

My categorization screen shown here included these categories, plus a box to check to tell the computer to train on the document. This training box is always optional. This will be explained further along in this narrative as we dive deeper into the predictive coding aspects of the search. The training button should only be checked on a category chosen for the document.

The screen shot here shows an cross-categorization error; the wrong Training box has been checked. The computer will not allow you to proceed with the error. Here you would either have to uncheck the Training button on Relevant, and check it instead on Irrelevant, or not at all. Alternatively, you could change the category check box to Relevant, and leave the Training box checked. This kind of consistency safeguard is present in all software systems that I looked at. Ask your vendor to confirm that they have similar consistency safeguards in the coding.

Regarding the Privileged category, I only ran into a few privileged documents that were relevant. When this happened, I of course marked them as such. But they were so rare as to not be valuable to describe here. The narrative will instead focus entirely on my search for relevancy.

This initial orientation period lasted about three hours. (Whenever I report time herein, that is billable type time. I’m not including breaks or significant interruptions.)

First Predictive Coding Run

After this orientation I began the search and coding project in earnest by starting the predictive coding procedures of Inview. I began by generating the first random sample of the data. I used a 95% confidence level and a +/- 3% confidence interval. Based on these specifications the software randomly selected 1,507 documents. I’ll explain that number soon, as observant readers will note it seems too high.

This entry was posted on Sunday, July 1st, 2012 at 4:37 pm and is filed under Lawyers Duties, Review, Search, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.