e-Discovery Team

The Emperor's New Clothes

A report of the American College of Trial Lawyers concluded that electronic discovery is the main problem with the U.S. legal system today. Runaway e-discovery costs are making it too expensive and uncertain to try cases. Litigants, especially large companies, are driven to ADR or forced to settle at inflated prices just to avoid e-discovery. Trial lawyers, and some in-house counsel too, blame the new rules and our judges. They see the problem, but, as I have said before, they misdiagnose the cause. They are hoodwinking themselves like the emperor with his phony new clothes.

The true cause of the e-discovery morass is our profession’s failure to keep pace with the dizzying advances of new technologies. In less than a generation, writing and evidence has been completely transformed as George Paul has shown in Foundations of Digital Evidence.  To put it bluntly, most of us trial lawyers are not fully competent to practice law in today’s digital age of terabytes of potential evidence. Most of us do not know how to do e-discovery in an efficient and cost effective manner in that kind of an environment, much less employ effective quality control procedures. Vendors and others spin a web of pseudo-competence and feed our trial lawyer egos. Most litigants, even large corporations that should know better, go along with it. They do so either out of blind-cronyism, a similar lack of knowledge, or a false assumption of security from a big firm name. The clients take the huge bills in stride and accept that there is no way around it. Alternatively, they refuse to do e-discovery altogether and risk sanctions. One wonders how long it will take corporate America to realize that the emperor has no clothes? 

Ken Withers, one of the deepest thinkers around in e-discovery, is trying to coin a new word – protodigital – to help us all understand this crises of competence.  We litigators over forty were among the first to venture into the digital age, but we grew up in a paper world; we learned our lawyering skills with dead-trees, not ESI. Our thinking is molded by the paper world of the past in a million subliminal ways. It is hard to break out of that paper prison, but we must do it in order to stop mismanaging e-discovery. Here is how Ken explained it in the briliant keynote speech he recently gave at the Georgetown University 5th Annual Advanced E-Discovery Institute Program:

What do we mean by that? “Protodigital?”

This senior generation of litigators (and I’m at the tail end of it myself) is fully cognizant that we live in a digital world and are themselves likely to use computers to some extent – for word processing, for email, to read a court decision online, perhaps even to generate a spreadsheet.

But they are still thinking of the digital information system as a set of tools for producing information (the document, the email communication, the legal opinion or spreadsheet) that they will manage as though that information were paper-based. They think that it is somehow appropriate to manage digital information, and discovery, by analogy to the paper world. This failure of many litigation decision-makers to think beyond the protodigital is having catastrophic consequences for the ability of our civil justice system to deliver the just, speedy, and inexpensive determination of ANY action. (bold font added) . . . 

The massive mismanagement of e-discovery in the past few years by the protodigital generation has been grudgingly underwritten, to a large extent, by clients who had the resources to pay the bills and were never presented with alternatives.  (emphasis added) Those days are over, and the Net-Geners will soon be paying the tab, figuratively and literally, and calling the shots.

The inability of American lawyers to do e-discovery at a reasonable cost is becoming a world-wide embarrassment. Even the intellectual press in the U.K. is starting to pick up on it. For instance, an article in the Economist entitled The Big Data Dump, opined that because of expensive e-discovery, the U.S. “civil-justice system as a whole threatens to get bogged down.”  The article then quotes Supreme Court Justice Stephen Breyer, expressing concern that, with ordinary cases costing millions just in e-discovery work, “you’re going to drive out of the litigation system a lot of people who ought to be there” so that “justice is determined by wealth, not by the merits of the case.” The article also quotes my golfing friend, Pat Oot:

Patrick Oot, a lawyer for Verizon, an American telecoms giant that gets sued a lot, says that at the beginning of this decade e-discovery presented “a one-big-case, once-a-year problem”. In most cases information was still on paper, and its volume thus limited. In the rare event that electronic evidence was requested, 100 gigabytes (GB) was considered a large amount. Today, says Mr Oot, almost every case involves e-discovery and spits out “terabytes” of information—the equivalent of millions of pages. In an ordinary case, 200 lawyers can easily review electronic documents for four months, at a cost of millions of dollars, he says.

The solution proposed by the Economist echoes our own trial lawyers, change the rules again to stop e-discovery and blame our judges. But the British publication goes a step further and suggests that we yanks should move away from our current system of justice altogether. We should instead adopt the inquisitorial legal system used in common law countries in Europe, a process that the U.K. itself is moving towards. As the Economist puts it:

This is overwhelmingly an American problem. In countries such as France and Germany that have an inquisitorial legal tradition, e-discovery tends to be proportionate to the case, because judges largely determine what information is relevant.

I for one say no! We need to step up our game and protect the American system. As Ken Withers, Patrick Oot, and a most of my readers know fully well, there is another way, a way that preserves our legal traditions and still keeps e-discovery affordable. Electronic discovery can be brought under control by competent, informed management of the process. This requires experts who are fully competent in all aspects of e-discovery. They are professionals involved full time in this arcane pursuit, typically outside counsel, consultants, or a few in-house attorneys like Patrick Oot.

Unfortunately, the vast majority of lawyers and paralegals involved with e-discovery today, typically those involved only occasionally or on a part-time basis, do not have the skills and deep knowledge required for economical e-discovery. They may suspect their protodigital incompetence, just like the emperor may suspect his new suit, but they are typically in deep denial. Even giant vendor bills, sanctions, and huge losses do not wake them up. When things go wrong, it is always easier to blame forces beyond your control, such as the new rules, judges, and every-body’s personal favorite, selfish plaintiff’s attorneys.

As a reader of this blog you are probably among the small but growing minority who already understand what to do to control costs. You have already worked hard to make your own clothes, your own skills, or are in the process of doing so. You have escaped the weaver’s spell of flattery and false competence, based as it was on partial knowledge, protodigitality, and vendor dependence.

Some of you may have the requisite skills and knowledge, and may see the emperor for who he really is, but are afraid to say anything out of fear of retribution. After all, no emperor likes to be exposed, and usually does not take kindly to those who point out the obvious. Do you dare cry out that an e-discovery bill of a million dollars in a case worth two million, on its best day, is absurd and shows that somebody made a huge mistake? (Hint – it was not the judge.)

I know from your email that some of you know better, but are anyway forced to go along with the old game. You are stuck in a system that still follows the old protodigital paradigms of over-collection and over-review so as to keep the profit-machine running. This is a short-sided mistake, as you well know.

To turn around the profit-emperors requires a persistent, persuasive approach. Although I may be preaching to the choir here, I offer this sermon as a way to help you to get the word out about economical e-discovery. The economy is in severe decline. That means the time is now ripe for these sermons. 

Sermon of Protodigital Transcendence

The sermon of competent e-discovery as the path to salvation of costs and risks is always preached in a slightly different manner depending on who you ask and when you ask it. But the essential message of protodigital transcendence is the same. If e-discovery is done right, it can be done at a reasonable price. There is no need to abandon our system of justice, or angrily shake your fist at our judges and new rules. Instead, we avoid bankruptcy by full emersion into the digital clouds.

So here is my best answer today, but if you ask again in six months, it is likely to change again. This is a fast moving field and the only consistency is change. Reasonable, affordable e-discovery requires three steps (1) establish and operate an interdisciplinary tech/law e-discovery team, a step which many in-house counsel have already begun, (2) strategic cooperation with opposing counsel in technical areas of discovery to avoid unnecessary disputes, instead focusing adversarial arguments on case merits and 26(b)(2)(C) evaluation, and (3) case metrics.

The last point is critical and has been largely ignored by most law firms and vendors in representing their corporate clients. As a client, you have a right to know in advance what an e-discovery production will cost and how long it will take, or at least a range. You also have a right under all state and federal rules to limit the discovery costs proportionally to the value of the case. For too long, in-house counsel have been asked to get on board an e-discovery train without knowing the price of the ticket or how long the trip will take. That must stop in order to get e-discovery under control.

Most attorneys today agree to search terms for the production of their client’s information without first knowing what the likely cost will be to review and produce this information. Further, keywords are agreed to and used without first testing them on smaller data samples to evaluate their efficacy. What percentage of the documents they produce will actually be relevant? Since even small clients now have incredibly large amounts of data in their computer systems, the net result of this haphazard approach is that far too much data is collected and far too much time and fees are wasted on review of irrelevant information.

This over review and production is the cause of run away e-discovery costs, not the new rules or bad judges. This can be stopped by metrics and more scientifically based cost estimation. You should have a good idea of what an e-discovery project will cost, and how long it will take – and have your attorneys be ready to prove that in court, if need be – BEFORE you agree to it. If opposing counsel insists on keywords that your analysis shows will produce too many documents and too much irrelevant data, then take it to court immediately and argue that excessive burden. Exact predictions are not required, only good faith estimates based on expert analysis, preferably your own, and reasonable assumptions.

To do this you need to test various proposed keywords on smaller sample sets of data before any agreements are reached or productions are started. Blind faith guesses of what might work always results in over-review and over-production, or worse, in very expensive “do-overs.” This misguided kind of guess-searching is the root cause of the skyrocketing costs of e-discovery. It must be replaced by judgmental and statistical sampling techniques, iterative processes, quality control, and staged productions.

As David Baker put it, who is the only judge I know who has been playing with computers longer than I have:

Common sense dictates that sampling and other quality assurance techniques must be employed to meet requirements of completeness. If AZ took such steps, it has not identified or validated them.

In re: Seroquel Products Liability Litigation, 244 F.R.D. 650 (M.D. Fla. Aug. 21, 2007).

The new metric techniques allow for an informed evaluation of proposed key words wherein you can limit the number of electronic files to be reviewed and produced. This intelligent management process builds a strong case against obstructive opposing counsel trying to force excessive discovery costs for strategic purposes. The courts will enforce the limited proportionality of discovery costs if protective orders are properly sought with valid expert substantiation.


Under the system I have developed, and no doubt others in the industry have developed similar approaches, we test proposed key words and other culling factors (time range, number of players, etc.) before any agreement to use specific keywords with opposing counsel, or absent an agreement process, before the full work commences. The process is technical and complex, but essentially we use an iterative system of early review of sample data sets. We then project the cost to review the sample set to calculate an estimate for the likely total cost of review of all of the data. Thus, if we find that it takes 100 hours to review a 10% sample of the total ESI collected by the keywords under consideration, we can reasonably project a total review time of 1,000 hours.

The same sampling process is used to test the efficacy of key words and ultimately to pare down the number of keywords used so as to reduce the total amount of ESI to review. We do not undertake to use keywords until we are satisfied that the likely cost to review the amount of ESI those keywords will generate is within a range acceptable to the client and warranted by principles of proportionality (FRCP 26(b)(2)(C)) and the value of the case.

As part of this evaluation process, we also analyze the efficacy of the search parameters. We do that by various judgmental sample reviews. For example, a reviewer may look at a sample consisting of the information generated by the top ten keywords, the ones that generated the most hits. The reviewer then ranks the relevance of the ESI so uncovered. What percentage of these documents are irrelevant, that is “false-positives”? Are the relevant documents found of only marginal relevance, or are they strongly relevant? We also look at the flip side, where a reviewer will examine a sample set consisting of a random selection of the misses. What percentage of the files excluded by the keywords were actually relevant, that is, “false-negatives,” and if so, rank the relevance.

With this kind of quality control process and information we can make informed adjustments to the keywords, the Boolean logic of the keywords (should it be within 10 words instead of 5?), and other culling factors such as custodians, date ranges, computer systems, deduplication schemes, etc. Ultimately, we reach a formula that produces a number of electronic documents that can be reviewed at a reasonable cost in view of the amount at issue in the case and other factors. This is typically presented as a first disclosure step, and additional, much more focused discovery requests may follow.

This use of metrics and quality control is a best practice that should be of interest to all clients, especially in the current economy. This metric quality management process not only reigns in costs to a reasonable range, but also builds a strong case against obstructive opposing counsel. We all know the ones who did not get the Sedona “Cooperative Discovery Memo” and try to force excessive discovery costs for strategic purposes. There is every indication that courts will enforce the limited proportionality of discovery costs if protective orders are properly and timely sought with valid expert substantiation.

Kipperman v. Onex Corp.

A recent case proves a good example of what can happen using a protodigital approach without cooperation or timely, valid metrics. Kipperman v. Onex Corp., 2008 WL 4372005 (N.D. Ga., Sept. 19, 2008). The opinion concerns ongoing discovery of e-mail from Defendants’ backup tapes. There is no explanation in the opinion by Senior District Court Judge  J. Owen Forrester as to why the tapes had to be searched, but obviously there has been earlier problems in this case before the defendants’ latest motion for protective order. Here is Judge Forrester’s explanation of the case background on this issue:

During a hearing in January of 2008, the court ordered Defendants to produce e-mail from two electronic backup tapes selected by Plaintiff. Plaintiff selected two tapes and provided search terms. Defendants searched the mailboxes of seven people whose depositions Plaintiff had sought. Plaintiff filed a motion to compel arguing that Defendants should have searched all mailboxes on the two tapes.

First, one wonders why the two sides did not talk before the search about the keywords to be used and the custodians to be searched. The opinion makes it look like the defendants just blindly accepted the plaintiff’s keywords, and unilaterally decided to search only seven custodian PSTs (mailboxes). Naturally this approach did not work, leading instead to another hearing and another ruling preceding the instant order. Yes, the discovery litigation expenses continued to mount. Here is Judge Forrester’s explanation:

The court addressed this motion during the April 2008 hearing and ordered the Defendants to search all the mailboxes on the original two tapes as well as on an additional tape selected by Plaintiff. The court determined that despite the Defendants’ previous objections and representations, the backup tapes were producing meaningful discoverable information. The court did suggest, however, that Plaintiff be more artful with its search terms and that Plaintiff utilize a list of the people, provided by Defendants, to review whether all mailboxes needed to be searched. The court also granted Defendants the opportunity to narrow the search terms.

Now surely the defendants would talk and start to use sampling and metrics to narrow down the search terms and so reduce the cost of the project. The judge is practically begging them to do so. But no! Instead, defendants just go on using the plaintiff’s same old over-broad list of in-artful key words. Apparently no attempt is made to cull down the number of players either. Although I do not know the truth of what really happened here, from Judge Forrester’s report in this order, this appears to be a classic example of refusing to cooperate, or, better put, of cutting off your nose to spite your face. 

The Defendants did not provide this list of people; they did not narrow the terms. As such, they agreed to search and restore all the mailboxes with the search terms provided by Plaintiff.

When defendants ran the keywords on all of the boxes, apparently again with no advance sampling or other testing of any kind, they finally discovered that an incredibly large number of emails are produced by this search, and a high number of them are irrelevant. (No actual numbers were used in the opinion, again suggesting a complete lack of metrics.)

Defendants then moved the court for a protective order, not before. They sought relief from having to review and produce all of the results from the search; a search that they did not previously object to, or even talk about modifying. The judge’s reaction to that kind of scenario is not too hard to predict. He refused most of the protection requested, basically saying this was too little too late, you previously made your bed and it is too late to get out of sleeping in it now.

It is obvious from the order, however, that Judge Forrester was not unsympathetic to their plight of costly over-production and over-review from stupid search terms and custodian lists. Cooperation and a proper and timely use of metrics would have led to a far different result. Here is part of Judge Forrester’s explanation for denying the protective order:

Defendants contend that Plaintiff’s broad search terms resulted in thousands and thousands of irrelevant hits. For example, Plaintiff’s search terms included the word “republic.” The republic term sought to elicit e-mails regarding Republic Builders Products, one of the Magnatrax/ABCO acquisitions involved in this matter. Defendants claim that the search captured thousands of irrelevant pages due to one occurrence of the word “republic” often related to Onex business interests having nothing to do with Magnatrax in the “Republic of France,” “Republic of Ireland,” and “Czech Republic.”

Defendants’ actions have delayed the production of electronic discovery throughout this litigation. Defendants have consistently tried to minimize the likely value of this discovery. The court’s minimal forays into the electronic discovery that has been produced has shown just the opposite. Despite all of this, the court is not unsympathetic to the massive amount of discovery involved in this matter, the considerable burden of working with it, and the overproduction that often comes with e-mail production. Therefore, the court gave Defendants numerous tools by which to reduce the burden of e-mail discovery, including an opportunity to limit Plaintiff’s search terms and an opportunity to provide a list by which the number of peoples and the number of boxes being searched could be reduced. Defendants did not take advantage of these opportunities. Defendants must now lie in the bed that they have made. Thus, Defendants’ objections on the basis of relevancy and volume are DENIED. The court could deny Defendants’ requests with respect to Armtec and ONCAP for these reasons as well. However, the court finds it highly unlikely that these search terms and mailboxes will garner relevant, useful information and finds the parties’ and the court’s burden in working with this information to outweigh its relative usefulness. On these grounds, the court will GRANT Defendants’ request for relief with respect to these search terms and these mailboxes.

Conclusion

I am sure that the e-discovery costs in Kipperman have been high and are having a major impact on the case. You could blame Senior Judge Forrester, but would that be fair? Would that accurately reflect the conduct of the parties and their attorneys in this case? No. Like Ken Withers, I blame instead the protodigital mindset. Like the small child in Hans Christian Andersen’s tale, I suggest the obvious, that the emperor has no clothes. Contrary to popular belief among some litigators, e-discovery is not so simple that even a protohuman caveman lawyer can do it.

How do we fix this? How do we empower the legal emperors of America with legitimate garments? Do we surround them with flattering consultants, associates, and vendors eager to continue the profit-train? Do we exhort them to attend CLEs and hope that “brain-surgery made easy” really works? I think not. That has been tried, and if anything, we are finding that a little knowledge by dilettantes is a dangerous thing.

The long term answer lies with education in law schools and the slow take-over of the next generations of fully developed digital lawyers. But that will take years. In the meantime, as Ken Withers so aptly notes, gross mismanagement of e-discovery will have “catastrophic consequences for the ability of our civil justice system to deliver the just, speedy, and inexpensive determination of ANY action.” Therefore, in addition to the long term solution of law school education, we need a stop-gap solution, one that will cover us for the next five to ten years. 

Fortunately, that solution is already emerging, the development of e-discovery specialists; professionals who devote their full time and total efforts to legal/digital pursuits. This will come from old protodigital lawyers like me who force themselves into digital maturity through total immersion in the practice, and from younger lawyers like Patrick Oot, who devotes himself full time to managing Verizon’s e-Discovery team. The legal profession can save itself by the advent of specialists. This requires lawyers who do nothing but electronic discovery work, all nine steps. It also requires IT engineers who do nothing but e-discovery work to work with these lawyers.

This is already starting to happen in progressive law firms around the country. The specialists work with and support the trial lawyers. Once the litigators get used to the new arrangement, they come to accept it. Some even come to like it, even though it necessarily results in some diminishment of their control and authority. The delegation frees them to practice law in their usual protodigital manner, pretending the ESI the specialists find is just like paper. They are glad to leave the computer stuff to the geek-lawyers so that they can really practice law. These are at least healthier delusions.

The danger comes from the litigators so heavily entrenched in the old paper world mentality, that they view all discovery work as simplistic, worthy only of a young associate, except only key depositions. They think they know how to do e-discovery already. After all, anyone can do discovery; that has never been a specialty before. They know the rules, have read a few e-discovery cases, attended CLEs, and are adept at computers. They can supervise complex e-discovery work one day and complex commercial litigation the next. It is not really that hard – just hire a good vendor and quarrel over their bills. The sycophants surrounding them perpetuate and may even believe the delusion. The emperor’s new clothes are indeed marvelous to behold! That is, until the judge bursts the bubble or the client gets tired of paying the bills. That day is near my friends, which means an opportunity for the rest of us.

This entry was posted on Sunday, December 7th, 2008 at 7:46 pm and is filed under Lawyers Duties, Review, Search. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.