e-Discovery Team ®

I have been working on the problem of out-of-control e-discovery costs since 2006. At that time I phased out my general trial practice, went full-time e-discovery, and started this blog. (By the way, did you notice the new ® in the blog title? It means the U.S. Patent and Trademark Office granted me the trademark to e-Discovery Team.) I focused on the expense side because it was obvious that crazy high e-discovery cost was a core problem of civil litigation. It still is. Indeed, the high price of e-discovery, and the uncertainty of  these costs, are the main reasons most attorneys still avoid e-discovery like the plague. For more reasons see Tell Me Why?

The primary expense of e-discovery comes from the document search and review process; most estimate that it constitutes from 60% to 80% of the total. The core expense of the review process comes from the final manual quality control checks of each document to be produced to verify relevancy and to protect confidentiality by redaction and privilege logging. Confidentiality protection is an enormous problem in litigation. See Anonymous, An Open Letter to the Judiciary – Can We Talk? Parts One and Two.

Further, you cannot just dispense with final manual review. As I explained in my series Secrets of Search, Parts One, Two and Three, we are not going to turn that over to the Borg anytime soon. I’ve asked around and no law firms do that now. No experts advocate that approach either, even the most extreme advocates for automation (of which I’m one). The only exception I have heard of is in non-litigation circumstances, such as second reviews with production to the government. Automated review is nowhere near good enough to go it alone. You use predictive coding to speed up the final manual review to be sure, but only a fool (or con artist trying to get at a producing parties secrets) trusts coding software today without human verification.

My thinking and experiments since 2006 have focused on how to control the final review costs. By early 2008 I came up with one possible method that looked promising. I have been testing and refining this invention ever since with several e-discovery teams. I have also talked about it with many other attorneys, friend and foe, and used this new method in many law suits, big and small. I am now ready to write publicly about my proposed fix for the first time. I call it Bottom Line Driven Proportional Review and Production. A more technical description for it, the one I used in a legal methods patent application, is: System and Method for Establishing, Managing, and Controlling the Time, Cost, and Quality of  Information Retrieval and Production in Electronic Discovery. But I usually just call it Bottom Line Driven Review, and who knows, if it catches on – and I think it should because it really works – I may trademark that phrase too.

In the meantime, try it out. The more attorneys that use this method, the more accepted it will be by judges. Right now they are hearing it from my teams for the first time, and, like anything new, it takes some explaining and getting used to. But, once understood, it appears obvious, and I expect all thinking clients will demand that their attorneys use this approach. It saves money.

Bottom Line Driven Review

The bottom line in e-discovery production is what it costs. Believe me, clients care about that …. a lot! In Bottom Line Driven Proportional Review and Production everything starts with the bottom line. What is the production going to cost? Despite what some lawyers and vendors may tell you, that is not an impossible question to answer. It takes an experienced lawyer’s skill to answer, but after a while, you can get quite good at such estimation. It is basically a matter of man-hours estimation. With my method it becomes a reliable art that you can count on.

Price estimation is second nature to me, and an obvious thing to do before you begin work on any big project. That is primarily because I worked as a construction estimator out of college to save up money for law school back in the seventies. Believe me, estimating review costs is basically the same thing, projecting materials and labor costs. In construction you come up with prices per square foot. In e-discovery you estimate prices per file, as I will explain in detail later.

My new strategy and methodology is based on the bottom line. It is based on projected review costs, defensible culling, and best-practices of review. Under this method the producing party determines the number of documents to be subjected to costly final review by calculating backwards from the bottom line of what they are willing, or required, to pay for the production.

The process begins by the producing party calculating the maximum amount of money appropriate to spend on ESI production. A budget. This requires not only an understanding of the ESI production requests, but also a careful evaluation of the merits of the case. The amount selected for the budget should be proportional to the monies and issues in the case. Any more than that is unduly burdensome and prohibited under Rule 26(b)(2)(C), Federal Rules of Civil Procedure and other rules that underlie what is now known generally known as the Proportionality Principle. See Rule 1, Rule 26(b)(2)(C), Rule 26(b)(2)(B), and Rule 26(g) Federal Rules of Civil Procedure; Commentary on Proportionality in Electronic Discovery, 11 SEDONA CONF. J. 289 (2010); Oot, Kershaw & Roitblat, Mandating Reasonableness in a Reasonable Inquiry, Denver University Law Review, 87:2, 522-559 (2010); Also see Rule 403 of the Federal Evidence Code (inadmissibility of cumulative evidence).

The budget becomes the bottom line that drives the review and keeps the costs proportional. The producing party seeks to keep the total costs within that budget. The budget should either be by agreement of the parties, or at least without objection, or by court order. The failure to estimate and project future costs, and to decide in advance to conduct the review so as to stay within the budget, is the primary reason that e-discovery costs are so high.

After analysis of the case merits and determination of the maximum expense for production proportional to a case, the responding party makes a good faith estimate of the likely maximum number of documents that can be reviewed within that budget. The document count represents the number of documents that you estimate can be reviewed for final decisions of relevance, confidentiality, privilege and other issues, and still remain within your budget. The review costs you estimate must be based on best practices and be accurate (no puffing).

The producing party then uses smart search techniques and quality controls to find the documents most likely to be responsive within the number of documents that the budget allows. This is usually based on relevancy ranking, and thus the need for hybrid multimodal best practices in the search and review. Predictive coding is inherently rank based and so it makes bottom line driven review especially easy to do. That is one reason I am especially pleased to see the price of predictive coding software finally coming down. It can be done without predictive coding ranking to be sure, but it is harder to be accurate, especially with recall. Using best methods allows you to get the most bang for your buck, the core truth, and thus persuades the requesting party or court to go along with your budgetary limits. More on the new gold standards in a minute.

Example

An example may help clarify how it works. If you set a proportional cost for a case of $100,000, and estimate that it will cost you $5.00 per file for the final manual review before production of the ESI at issue, then you can  review no more that 20,000 documents and stay within budget. It is basically that simple. No higher math is required.

The only difficult part is the legal analysis to determine a budget proportional to the real merits of the case. But that is nothing new. What is the golden mean in litigation expense?  How to balance just, with speedy and inexpensive? The essence of the ideal proportionality question has preoccupied lawyers for decades. It has also preoccupied scientists, mathematicians, and artists for centuries. They claim to have found an answer that they call the golden mean or golden ratio:

In law this is the perennial Goldilocks question. How much is too much? Too little? Just right? How much is an appropriate spend to produce documents? The issue is old. I have been dealing with this problem for over thirty years. What’s new is applying that legal analysis to a modern-day high-volume-ESI search and review plan. Unfortunately, unlike art and math, there is no accepted golden ratio in the law, so it has to be recalculated and reargued for each case. (Side Note: If the golden ratio were accepted in law as an ideal proportionality, the number is 1.61803399, aka Phi. That would mean 38% is the perfect proportion. I have argued that when applied to litigation that means the total cost of litigation should never exceed 38% of the amount at issue. In turn, the total cost of discovery should not exceed 38% the total litigation cost, and the cost of document production should not exceed 38% of the total costs of discovery.  (It’s like Russian dolls that get proportionally smaller.) Thus for a $1 Million case you should not spend more than $54,872 for document productions (1,000,000 – 380,000 – 144,400 – 54,872). See Losey, R., Beware of the ESI-discovery-tail wagging the poor old merits-of-the-dispute dog. But I digress too far.)

Estimation for bottom line driven review is essentially a method for marshaling evidence to support an undue burden argument under Rule 26(b)(2)(C). It is basically the same thing we have been doing to support motions for protective orders in the paper production world for over sixty years. The only difference is that now the facts are technological, the numbers and variety of documents are enormous, sometimes astronomical, and the methods of review are very complex and not yet standardized.

The calculation of projected cost per file to review can be quite complicated, and is frequently misunderstood, or is not based on best practices. Still, in essence this cost projection is also fairly simple. You basically project how long it will take to do the review and the total cost of the time. Thus, for example, and this is a gross over simplification, in a review project of 20,000 documents (after computer assisted culling – it probably started as 100,000 or 200,000), if the average review and coding rate is 50 files per hour, it will take 400 hours to complete. If the projected total cost for the reviewer time, including supervision and other costs, is $250 per hour, the projected total cost for the review is $100,000 ($5.00 per file).

This may seem high when you consider the cost of contract lawyers is $50 or less for their time, but you have to also include expensive partner and senior associate management time, direct supervision, quality control reviews, and privilege logging, etc. Do not be fooled by promises of $1.00 per file charges by contract review companies (or even less than that). That does not include the law firm of record time and expenses to supervise, etc., and often is based on a pre-culling rate for file count. In this business one of the hardest aspects for good estimation is getting true apples-to-apples comparisons from vendors.

Also, quality control is important and best practices can be expensive, even though with bottom line driven review the total cost is still dramatically less than old-school. As I said before, when you talk about capping the number of documents you review, you also have to talk about finding the most likely relevant documents for this capped review. You have to provide the most bang for your buck, the most truth. That, along with transparency to earn trust, is a key to the success of this method, a key to persuading the other side or court to accept this new approach to reasonable discovery.

Estimate of Projected Costs

Another key to persuade the requesting party or court is to be sure your estimate is realistic. You cannot just dream up estimates, or puff the likely expense. The estimate must be based on knowledge of the types of documents that you will be reviewing in your particular case. It must be based on the times that you find it takes for manual review of the documents. Some document collections are faster and easier to review than others. The speed is measured in files per hour. (I like the sound of that, plus pages per hour is just a relic of the paper world. Computer files don’t have pages, only paper print-outs do.)

The typical speeds we see today in final manual review are anywhere from 25 files per hour, for collections with a lot of dense long documents and crappy review software, to 100 or even 200 files per hour for collections with easier to skim documents and the best software. Putting aside the question of the wide divergence in the quality of review software, we tend to see faster files per hour rates in email-heavy cases with few attachments, than we do in cases with a high percentage of complex documents and spreadsheets. You have to know your case, know your ESI, to make a proper estimate.

The projected costs must also be based on best practices for economical review and not be inflated. You can’t justify $500 per hour partners to do all of the review (although they may be needed as subject matter experts to do the seed set review in predictive coding). Of course, old-fashioned full manual review is out of the question. It has to be hybrid with computers doing most of the first-pass document culling. Even if you use technology assisted review, you still must also use best practices for methods. You cannot justify old-fashioned stupid review methods, such as batch out to reviewers solely based on first in, chronological, or just random. It has to be a best practice based multimodal type of review, where, for instance, you batch out documents for manual review based on issues, clusters, language, or other smart review methods. Best practice also means quality control and a random button as discussed at length in the Secrets of Search series. If you do not use the nine best practices to get the most bang for your buck, the core truth, the requesting party or court may not agree to limit the number of documents to be reviewed.

The processes behind the estimate should also be transparent. This means you should be willing to disclose it to the requesting party. That is how you can convince them that the estimate is reasonable and that you are not still stuck in the old paradigm of hide-the-ball discovery games. I cannot overstate how important it is to develop trust between counsel on discovery and often the only way to do that is through transparency. You do not have to disclose all of your trade secrets, but you have to keep the requesting party pretty well informed and involved in the process. That is what cooperation looks like.

In general, I have found that in 2011, $5.00 per document was a good place to start in projecting costs for review of a typical email collection (an email is one file, and each attachment is another). This price includes the expensive redaction and privilege logging processes. Review with a simple relevant or irrelevant coding is the easiest and cheapest to do, and is also fairly rare. There are usually multiple additional factors to consider.

The five dollars per file is a starting point of estimation, a rule of thumb that is often correct, but sometimes way off. It is comparable to the rule of thumb in construction estimation where you start with the typical costs to build on a square footage basis. But in some cases, especially ones involving cross-border issues, the costs could go much higher, as high as $15 per file. In others, where the review is simple, it could go as low as $2.00 per file. It is just like construction where various buildings in different locations have different costs.

The $5.00 per file price is based on my recent experiences in 2011. In 2009 the average cost was more like $6.50 per file, and I expect average costs will keep going down a little in 2012 and then level off.

It is important to note that you can justify starting with a much higher number based on legal precedent alone. For instance, the Department of Justice spent $9.09 per document (or file, same thing) for review in the Fannie Mae case, even though it used contract lawyers for the review work. In re Fannie Mae Securities Litig., 552 F.3d 814, 817 (D.C. Cir. 2009) ($6,000,000/660,000 emails). There were no comments by the court that this price was excessive when the government later came back and sought cost shifting. My current $5.00 per file general rule is also lower than the $6.09 per document that Verizon paid for a massive second review project that enjoyed large economies of scale and, again, utilized contract review lawyers.  Roitblat, Kershaw, and Oot, Document categorization in legal electronic discovery: computer classification vs. manual review. Journal of the American Society for Information Science and Technology, 61(1):70–80, 2010 ($14,000,000 to review 2.3 million documents in four months).

So, if your experience suggests a starting review rate higher than $5.00 per file, there is legal justification to use a higher number. Just be prepared to go to the next steps and back it up.

The price per file is just a starting point, a way to get a quick picture, a quick estimate, without doing all of the detail work. A more accurate picture starts to emerge with sample reviews and more detailed analysis of the tasks required in the review and the actual data to be reviewed. You have to, as I like to say, get your hands dirty in the digital mud. You have to know your ESI collection. Even in just one type of ESI, the one most common in e-discovery today, email and attachments, the variances in email collections can be tremendous.

Once you get your hands on the data you need to start to breakdown and analyze the time involved in the various tasks required in the review project. Here, as in construction estimation, the spreadsheet is your friend. This move to actual examination of the ESI at issue, and study of the specific review tasks that need to be performed in your case, is equivalent to the move in construction estimation from rough estimates based on average per square foot prices, to a careful study of the buildings plans and specifications, and a site visit with inspection and measurements of all relevant conditions. No builder would bid on a project without first doing the detailed real world estimation work.

Even in the same organization, and just dealing with email, the variances between custodians can be tremendous. Some for instance may have large amounts of privileged communications. This kind of email takes the most time to review, and if relevant, to log. High percentages of confidential documents, especially partially confidential, can also significantly drive up the costs of review. All of the many unique characteristics of ESI collections can effect the speed of review and total costs of review. That is why you have to look at your data and test sample the emails in your collection to make accurate predictions. Estimation in the blind is never adequate. It would be like bidding on a building without first studying the plans and specs.

Even when you have dealt with a particular client’s email collection before, a repeat customer so to speak, the estimates can still vary widely depending on the type of law suit, the issues, and on the amount of money in controversy or general importance of the case.

Although this may seem counter-intuitive, the truth is, the complex, big-ticket cases are the easiest to do e-discovery, especially if your goal is to do so in a proportional manner. If there is a billion dollars at issue, a reasonable budget for ESI review is pretty big. On the other hand, proportional e-discovery in small cases is a real challenge, no matter how simple they supposedly are. Many cases that are small in monetary value are still very complex. And complex or not, all cases today have a lot of ESI.

The medium size to small cases are where my bottom line driven proportional review has the highest application for cost control and the greatest promise to bring e-discovery to the masses.

The Quest for Gold

In Secrets of Search Parts One, Two and Three, I outlined the five key characteristics of search today, using the rubric of secrets. To support my outline I used the latest scientific research on legal search, and focused on the work of William Webber. Re-examining the Effectiveness of Manual Review. In Part Three I summarized my ideas on search and review using the symbol of the Pythagoreans, the five-sided polygon, or pentagon:

With this blog on Bottom Line Driven Proportional Review I add a sixth idea, where the process gets real and takes money into consideration. Here I have shared my method to use estimation, projections, budget, cooperation, transparency, and the legal doctrine of proportionality to control the costs of search and review. With this final piece my proposal for a new gold standard of search and review is complete.

Bottom Line Driven Review is a method to try to control the key problem in electronic discovery law today, the run away costs of review. The number of documents we have to review seems to double every two to three years, so this new legal method is imperative. New and better software, especially predictive coding type, is also important. As shown, the ranking of relevancy and other categories built into the latest algorithms is, under my bottom line driven analysis, an especially helpful new capability.  You rank the documents within your budget limit that the computer predicts, based on your training, will be the most relevant to your case. But new technology alone is not enough. We must also have new legal methods. Technology and law have to work together, grounded in science, to create a new gold standard.

In Secrets of Search Part II, I proposed a new gold standard, one that would replace the now disgraced old-gold brute-force manual review unassisted by technology. I drew upon the findings in the latest scientific research, legal literature, and my over thirty years of experience with discovery to create a first draft list of the nine criteria of the new gold. The first criteria listed was Bottom Line Driven Proportional Review, which I promised to explain later and have now done so. Here is how I put it in Part II:

The old gold standard of average human reviewers, working in dungeons <smile>, unassisted by smart technology, and not properly managed, has been exposed as a fraud. What else do you call a 28% overlap rate? We must now develop a new gold standard, a new best practice for big data review. And we must do so with the help and guidance of science and testing. The exact contours of the new gold are now under development in dozens of law firms, private companies, and universities around the world. Although we do not know all of the details, we know it will involve:

1. Bottom Line Driven Proportional Review where the projected costs of review are estimated at the beginning of a project (more on this in a future blog);
2. High quality tech assisted review, with predictive coding type software, and multiple expert review of key seed-set training documents using both subject matter experts (attorneys) and AI experts (technologists);
3. Direct supervision and feedback by the responsible lawyer(s) (merits counsel) signing under 26(g);
4. Extensive quality control methods, including training and more training, sampling, positive feedback loops, clever batching, and sometimes, quick reassignment or firing of reviewers who are not working well on the project;
5. Experienced, well motivated human reviewers who know and like the AI agents (software tools) they work with;
6. New tools and psychological techniques (e.g. game theory, story telling) to facilitate prolonged concentration (beyond just coffee, $, and fear) to keep attorney reviewers engaged and motivated to perform the complex legal judgment tasks required to correctly review thousands of usually boring documents for days on end (voyeurism will only take you so far);
7. Highly skilled project managers who know and understand their team, both human and computer, and the new tools and techniques under development to help coach the team;
8. Strategic cooperation between opposing counsel with adequate disclosures to build trust and mutually acceptable relevancy standards; and,
9. Final, last-chance review of a production set before going out the door by spot checking, judgmental sampling (i.e. search for those attorney domains one more time), and random sampling.

I have probably missed a few key factors. This is a group effort and I cannot talk to everyone, nor read all of the literature. If you think I have missed something key here, please let me know. Of course we also need understanding clients who demand competence, and judges willing to get involved when needed to rein in intransigent non-cooperators and to enforce fair proportionality. Also, you should always go for confidentiality and clawback agreements and orders.

I repeated this nine-point list of the new gold in Part III of Secrets of Search, and again repeated my invitation for input with a comment on standards that bears repetition:

I have probably missed a few key factors. This is a group effort and I cannot talk to everyone, nor read all of the literature. If you think I have missed something key here, please let me know. I will be at Legal Tech New York for three days with four presentations. Seek me out and let’s talk. You can reach me at ralph.losey@gmail.com.

You may note that I am herewith joining the call of other leaders in the field to develop best practice standards, notably including Jason Baron, and have overcome my initial reluctance to go there for a variety of reasons. See Jason R. Baron, Law in the Age of Exabytes: Some Further Thoughts on ‘Information Inflation’ and Current Issues in E-Discovery Search, XVII RICH. J.L. & TECH. 9, at 29-33. My concerns on arbitrary standards and unfounded malpractice claims remain, but I think we have no choice but to develop some basic industry standards. The nine characteristics of good document review outlined above constitute a first modest step in that direction.

I will be at LegalTech NY on January 30th, 31st and February 1st. My invitation for dialogue and input from readers continues. Seek me out and let’s talk, but spare me the sales pitches, please. (I am, however, open to writing pitches.) My main focus right now is the quest for a new gold standard of search and review. I know that many of you share this quest, so let’s use the power of groups, or team work, to make it happen. What do you think of my nine-point first draft list? Any suggestions to add new criteria, consolidate, or add to what any of these nine mean?

Two of my readers have already responded to my outreach for input, Bill Hamilton and Larry Chapin. They provided some more concrete details to the criteria number six in the list, new tools and psychological techniques. They submitted an essay on one such new technique, which is actually of ancient origin, and well-known to the best trial lawyers, namely the use of story and storytelling to improve legal review. Storytelling: The Shared Quest For Excellence in Document Review. If you have ideas for an article, please send me an email with outline or first draft and I will consider it for possible publication.  As coaches everywhere love to say, there is no “i” in team, even if there is in e-Discovery Team ® <grin>.

Conclusion

I have a dream, like all humans do. It is one of the key attributes that separates us from machines. My dream is not as noble or stirring as the public dreams of Martin Luther King or John Lennon. Those were grand dreams indeed. But my dream is important to me, and you can probably relate to it. I dream of a day where man and computer work together to bring truth and justice for all, not just the elite few who can afford it now. I dream of a day where e-discovery is affordable and used in all size cases. This dream of truth and justice for all is deeply rooted in my psyche. I suspect it is in yours too. We all grew up understanding the importance of Superman’s never-ending battle for truth, justice, and the American way. Join me in this battle. Join the e-discovery team fight for truth, justice, and the American way. (For my many readers outside of the U.S., my American way reference is not meant to be nationalistic or exclusionary, but rather to refer to the highest ideals of a great country.) All professionals in the field are invited. So too are computers, especially the latest generation of super smart ones. Yeah, their programmers too. All behind the scenes coders, techs, and scientists are an important part of the e-Discovery Team.

High tech lawyers working with computers and their handlers are key to my version of the archetypical American dream of truth and justice. Techs and computers helped bring about the nightmare we must now overcome – the explosion of ESI that hides the truth and makes justice too expensive. They helped get us into this mess, they can help get us out. We cannot turn back.

Jason Baron’s depressing prophesy of information dystopia, where we all drown in a flood of information, is no prophetic dream. It is a realistic assessment of the current state of the law and the discovery of electronic evidence. The reality today is that the vast majority of lawyers avoid the discovery of information in computers, even though that is where the truth lies. They have a prejudice against it. They believe in the inherent superiority of paper. We all know that most of the truth left paper filing cabinets over a decade ago (with the sole exception, perhaps, of the federal government), yet most lawyers still look there, and only there, for justice. Jason’s dream of extreme information overload is a projection of the current reality getting worse. He speaks the truth, but only if we don’t do something about it, if we don’t continue in the never-ending battle. Truth and justice can triumph. They must.

The motivation is clear. So is the solution. Imagine a world where the fool’s errand, the paper chase, comes to an end. Imagine a world where these old ways, based as they are on ignorance and delusion, are replaced by an affordable and effective process of e-discovery. Imagine a world where all the people, the litigants in all size cases, can all afford to do e-discovery. Imagine all the people living life in justice. It isn’t hard to do. As John Lennon said: You may say I’m a dreamer, but I’m not the only one. I hope some day you’ll join us. And the world will be as one.

I have a dream of a new method of technology assisted discovery, where Man and Machine work together to find the core truth. This day will come, in fact it is already here. As William Gibson said: “The future is already here – it’s just not evenly distributed yet.” The key facts you need to try a case and to do justice can be found in any size case, big and small, at an affordable price. But you have to open your mind. You have to embrace change and adopt new legal and technical methodologies. The Bottom Line Driven Review method is, I suggest, an important part of that answer. It is working for me today, it can work for you too. Our dreams can come true. The nightmare scenarios of justice for only the super-rich can be avoided. The battle for truth and justice must continue.

I  see a way out, where we can overcome, where truth and justice can be attained for all the people. I see a day where the truth in our computers can be found and brought to the court room for justice to prevail.

Although I also have a dream of a new generation of tech-smart lawyers, who understand and apply the new methods, the new gold, to keep e-discovery available for all. We do not need to wait for this slow gradual change. We can win the battle now, even without the young geek Supermen. The time for change is now – in this generation, not the next. As King said in his I Have Dream speech, this is not time to take the tranquilizing drug of gradualism.

Join me in the dream of e-truth and justice today. As a team we can get there, we can and shall overcome, we shall be free of the paper-prejudices of the pre-computer world. And when that day comes, let the bells of truth and justice ring throughout the world. To quote the end of King’s famous speech:

And when this happens, when we allow freedom to ring … [we] will be able to join hands and sing in the words of the old Negro spiritual, “Free at last! free at last! thank God Almighty, we are free at last!”