proportionality | Search Results

Proportionality Analysis Defeats Motion for Forensic Examination

May 28, 2018

It is rare of a judge to change their mind after making a decision. It is rarer still for a judge to celebrate doing so in a written opinion for the world to see. But that is exactly what Magistrate Judge Jeffrey Cole has done in his opinion dated May 17, 2018 in Motorola Sols., Inc v. Hytera Communications Corp., No. 17 C 1973 (N.D. Ill.).

This celebration is one reason that Judge Cole’s Order denying Motorola’s motion for forensic inspection is so remarkable. Another is that it begins with a quote, a rare occurrence in judicial orders, one that I always like. The quote celebrates the better late than never philosophy of changing your mind to follow a new understanding, a personal wisdom. The quote is by the late, great Supreme Court Justice, Felix Frankfurter. Felix served as a judge on the Supreme Court from 1939 to 1962. Before that he was, among other things, a Harvard Law Professor and co-founder of the American Civil Liberties Union. Here is the quote with which Cole begins his order:

“Wisdom too often never comes, and so one ought not to reject it merely because it comes late.”

Henslee v. Union Planters Nat. Bank & Trust Co., 335 U.S. 595, 600, 69 S. Ct. 290, 93 L. Ed. 259, 1949-1 C.B. 223 (1949)

(Frankfurter, J., dissenting)

Another unusual thing about Judge Cole’s order, and the real reason I am writing about it, is that the wisdom that came to him was from the doctrine of proportionality and Rule 26(b)(1). This was the basis for Judge Cole to deny plaintiff’s motion for a forensic inspection of defendant’s computers, in China no less.

District Court Judge Ronald Norgle had previously allowed the parties until October 6, 2017, to conduct discovery on the statute of limitations defense only and stayed all other discovery. The parties had one month in which to take discovery on a very limited topic of fraudulent concealment, which is a type of tolling within the doctrine of equitable estoppel of the limitations defense. Nothing else. After all, Motorola has waited almost ten years before filing a trade-secret theft suit against a Chinese corporation for allegedly stealing its radio wave technology. As Judge Cole colorfully described the situation (citations to record removed in all quotes) with a reference to Hannibal:

While the inquiry should have been uncomplicated, it has become a long, drawn out, pitched battle — one, in a rhetorical sense, to rival the Punic Wars — albeit without the elephants and the Alps and the sheer drama.

After that, the parties exchanged motions to compel repeatedly. Deadlines were extended, from one month to several. Thousands of pages of memoranda and exhibits were filed. ¹ And, again, this was all over the supposedly limited discovery on a limited topic that ought to have taken little time and effort. The very nature of what occurred tends to sustain the all too prevalent observation that discovery has become more important than the actual case. Bell Atl. Corp. v. Twombly, 550 U.S. 544, 595, n.13, 127 S. Ct. 1955, 167 L. Ed. 2d 929 (2007); A.H. Robins Co. v. Piccinin, 788 F.2d 994, 1013 (4th Cir. 1986).

¹ The filings and orders in this case which, again, is only in the preliminary stage of determining whether Motorola’s filing is timely, already cover more than 7,500 pages. See also n. 5, infra.

Five months into this “limited” discovery process Motorola asked to conduct a forensic examination of the computers of “key Hytera witnesses who have been involved in the use of Motorola’s confidential information and any relevant Hytera servers … on which Hytera has stored Motorola documents,” all of which are located in China. Motorola said it wanted to “begin with forensic inspection of the computers” of seven Hytera employees. Judge Cole said this request reminded him of Winston Churchill’s famous quip: “Now this is not the end. It is not even the beginning of the end.”

Flip Flopping Towards Wisdom

After several hearings Judge Cole was persuaded by the siren songs of plaintiff’s counsel from the well-known firm of Kirkland & Ellis LLP. They must have been very good orators and put on a compelling argument to support their motion. They convinced Judge Cole to allow them to begin a forensic examination process in China under elaborate Hague Convention procedures. Only after the hearings and oral decision to compel the inspection did Judge Cole realize the error of that decision. Judge Cole to his credit does not blame Kirkland and Ellis litigators for leading him astray. Following standard judicial protocol Judge Cole assumed full responsibility for the initial error:

Over the course of two lengthy hearings on March 21 and April 4, 2018, I tentatively concluded that forensic examination of Hytera’s computers would be appropriate, but only if the parties could arrive at a suitable protocol that would not, among other things, run afoul of Chinese law. As we discuss, infra at 5, that was a mistake. But the law frowns on relying on a blunder to gain an opportunistic advantage. Cf. Architectural Metal Systems, Inc. v. Consolidated Systems, Inc., 58 F.3d 1227, 1231 (7th Cir.1995); Market Street Associates; Packer Trading Co. v. CFTC, 972 F.2d 144, 150 (7th Cir 1992); Centex Construction v. James, 374 F.2d 921, 923 (8th Cir.1967). ²

² We should not be understood as ascribing fault to plaintiff’s counsel. After all, in our adversary system, lawyers properly play a partisan role. Masias v. Secretary of Health and Human Svcs, 2009 U.S. Claims LEXIS 281, at *27 (Fed. Cl. 2009); Philips Medical Systems Intern. B.V. v. Bruetman, 8 F.3d 600, 606 (7th Cir.1993) (Posner, J.). See also Smith v. Robbins, 528 U.S. 259, 293, 120 S. Ct. 746, 145 L. Ed. 2d 756 (2000) (Souter, J., dissenting) (“a partisan scrutiny of the record and assessment of potential issues, goes to the irreducible core of the lawyer’s obligation to a litigant in an adversary system … .”). Mistakes are ultimately (and in most cases) the responsibility of the court.

Judge Cole went on to celebrate a jurists right to change their mind in order to get things right.

The scope of discovery that I was initially inclined to allow was, in the context of the present inquiry that had been narrowed by the district court to the limitations issue, overbroad. What is being sought goes beyond the issue of equitable tolling. In the end, Motorola’s counsel and I were talking about relevance to allegations in Motorola’s complaint. And so, well beyond the statute of limitations, by the end of the April 4 hearing, discovery was encompassing documents related to Motorola’s entire case.

As we have said, “all judges make mistakes,” Fujisawa Pharm. Co., 115 F.3d at 1339, and, when possible, it is best that judges put them right.

Proportionality Applied to Restrain Discovery

In the May 17, 2018 order Judge Cole found the wisdom to say no and forbid the forensic examination of the computers in China. He did so because he found that this discovery was “out of proportion with the needs of this case, as presently limited by the district court” and cited Rule 26 (b)(1), Federal Rules of Civil Procedure. Although I am sure that he heard extensive argument and evidence concerning the estimated costs and burdens imposed by the forensic exams, his decision did not focus on costs. Instead it focused on one of the other very important factors in 26(b)(1), “the importance of the discovery in resolving the issues.” Judge Cole realized that the computers in China could not possibly have information in them of any real relevance to equitable tolling of the statute of limitations defense.

At a minimum, even if relevant to the present limited issue, discovery of computers in China is not proportional to the importance of discovery in resolving the issues and the burden and expense of the proposed discovery manifestly outweighs its likely benefit to the very limited question of equitable tolling. Although the federal discovery rules are permissive, they are not, as Judge Moran wisely put it, “a ticket to an unlimited … exploration of every conceivable matter that captures an attorney’s interest.” Sapia v. Bd. of Educ. of the City of Chi., 2017 U.S. Dist. LEXIS 73153, 2017 WL 2060344, at *2 (N.D. Ill. 2017); see also Leibovitch v. Islamic Republic of Iran, 2018 U.S. Dist. LEXIS 31713, 2018 WL 1072567, at *11 (N.D. Ill. 2018). “[J]udges should not hesitate to exercise appropriate control over the discovery process.” Herbert v. Lando, 441 U.S. 153, 177, 99 S. Ct. 1635, 60 L. Ed. 2d 115 (1979). Failure to exercise that control results in needless and enormous costs to the litigants and to the due administration of justice.

Judge Cole also understood that a forensic inspection is a drastic remedy that requires good cause not shown by plaintiff here:

The original idea here was for a month or so of discovery focused on the very limited issue of the statute of limitations. While it is rare for parties to complete discovery even by dates chosen by their counsel, there can be no dispute that things have already gone far beyond what was intended and what was necessary in the statute of limitations portion of this case, in terms of time and scope. Now, Motorola wants things to go very much further. Forensic examination is generally regarded as a drastic step even in general discovery. See, e.g.,John B. v. Goetz, 531 F.3d 448, 460 (6th Cir. 2008) (“mere skepticism that an opposing party has not produced all relevant information is not sufficient to warrant drastic electronic discovery measures.”). As the court said in In re Ford Motor Company, 345 F.3d 1315 (11th Cir. 2003):

““In the absence of a strong showing that the responding party has somehow defaulted in this obligation, the court should not resort to extreme, expensive, or extraordinary means to guarantee compliance. Forensic inspection of computer hard drives is an expensive process, and adds to the burden of litigation for both parties, as an examination of a hard drive by an expert automatically triggers the retention of an expert by the responding party for the same purpose. Furthermore, examination of a hard drive inevitably results in the production of massive amounts of irrelevant, and perhaps privileged, information … . This court is therefore loathe to sanction intrusive examination of an opponent’s computer as a matter of course, or on the mere suspicion that the opponent may be withholding discoverable information.”

The Advisory Committee Notes to Rule 34 recognize that courts must use caution in evaluating requests to inspect an opposing party’s electronic devices or systems for ESI, in order to avoid unduly impinging on a party’s privacy interests:

Inspection or testing of certain types of electronically stored information or of a responding party’s electronic information system may raise issues of confidentiality or privacy. The addition of testing and sampling to Rule 34(a) with regard to documents and electronically stored information is not meant to create a routine right of direct access to a party’s electronic information system, although such access might be justified in some circumstances. Courts should guard against undue intrusiveness resulting from inspecting or testing such systems.

Fed. R. Civ. P. 34, Advisory Committee Notes—2006 Amendment (emphasis added). Likewise, the Sedona Principles urge general caution in this area:

Civil litigation should not be approached as if information systems were crime scenes that justify forensic investigation at every opportunity to identify and preserve every detail … . [M]aking forensic image backups of computers is only the first step of an expensive, complex, and difficult process of data analysis that can divert litigation into side issues and satellite disputes involving the interpretation of potentially ambiguous forensic evidence.

The Sedona Principles, supra, at 34, 47. ⁴

Conclusion

Judge Cole’s wrap up is wise and witty and something you may want to quote in many discovery disputes, especially the footnote:

Parties are entitled to a reasonable opportunity to investigate the relevant facts — and no more. Upjohn Company v. United States, 449 U.S. 383, 390, 101 S. Ct. 677, 66 L. Ed. 2d 584 (1981); Vakharia v. Swedish Covenant Hosp., 1994 U.S. Dist. LEXIS 2712, at *2 (N.D. Ill. 1994) (Moran, J.). Motorola has already had that reasonable opportunity and far more. What was intended to be a month-long process of discovery on a very limited issue has turned into a protracted affair in which Motorola has received 700,000 documents — nearly 3 million pages — over a period of eight months. ⁵

⁵ A ream of paper is 500 sheets, which is 2 inches tall. Three million pages is 6,000 reams, meaning that 3 million pages of discovery, which is about 1,000 feet high or 100 stories high. By any measure, that is extraordinary.

Yet, apparently for Motorola, it’s not enough. It now wants a forensic inspection of several computers in China — and it warns that that is only the “beginning.” What should have been limited discovery on a “straightforward [issue has] spiral[ed] out of control.” Montanez v. Simon, 755 F.3d 547, 552 (7th Cir. 2014). The time has come to say: “enough is enough.” Walker v. Sheahan, 526 F.3d 973, 981 (7th Cir. 2008). Eight months of “limited,” single-issue discovery are now at an end. Motorola’s motion for forensic inspection is denied.

Enough is enough. To go further would have been a disproportionate burden, especially considering the very narrow issue allowed in discovery. Judge Cole at first made a mistake, and then he changed his mind and made it right. He is a wise judge. I wish there were more like him. Except of course if you change your mind to rule against me! <‘_’>

Leave a Comment » | Forensic Exam, Lawyers Duties, New Rules, Technology, wisdom | Tagged: discovery, PROPORTIONALITY | Permalink
Posted by Ralph Losey

Guest Blog: “Follow the Money” and Metrics: A User’s Guide to Proportionality in the Age of e-Discovery

April 8, 2018

This is a guest blog by a friend and colleague, Philip Favro. Phil is a consultant for Driven, Inc. where he serves as a trusted advisor to organizations and their counsel on issues relating to the discovery process and information governance. Phil is also currently active in The Sedona Conference. He obtained permission from them to include a description of a recent event they sponsored in Nashville on Proportionality.

“Follow the Money” and Metrics: A User’s Guide to Proportionality in the Age of e-Discovery

Moviegoers and political junkies have flocked to theaters over the past few months to watch period-piece epics including Darkest Hour and The Post. While there is undoubted attraction (especially in today’s political climate) in watching the reenactment of genuine leadership and courageous deeds these movies portray, The Post should have particular interest for counsel and others involved in electronic discovery.

With its emphasis on investigation and fact-gathering; culling relevant information from marginally useful materials; and decision-making on how such information should be disclosed and presented to adversaries, The Post features key traits associated with sophisticated discovery counsel.

Not coincidentally, those same attributes were on display in another drama from the 1970s of which The Post viewers were reminded: All The President’s Men. That investigative journalism classic depicts Washington Post correspondents Bob Woodward and Carl Bernstein as dogged reporters determined to identify the culprits responsible for the Watergate Hotel break-in in June 1972.

A critical aspect of their work involved Woodward’s furtive meetings with Mark Felt, who served at that time as the deputy director of the FBI. Known only as “Deep Throat” (until Felt revealed himself in 2005), Felt provided cryptic yet key direction that aided the reporters’ investigation. One of Felt’s most significant tips (as portrayed in the movie) was his suggestion that Woodward investigate the cash contributions made to help reelect then President Richard Nixon in 1972. Played by iconic actor Hal Holbrook in All The President’s Men, Felt’s soft-spoken but serious demeanor underscored the importance of his repeated direction to Woodward to “just follow the money.” By following the money, the Washington Post reporters helped discover many of the nefarious tactics that eventually brought down the Nixon presidency.

Proportionality

The directive to “follow the money” applies with equal force to counsel responsible for handling discovery. This is particularly the case in 2018 since courts now expect counsel to address discovery consistent with proportionality standards. Those standards – as codified in Federal Rule of Civil Procedure 26(b)(1) – require counsel, clients, and the courts to consider various factors bearing on the discovery analysis. They include:

(1) the importance of the issues at stake in this action; (2) the amount in controversy; (3) the parties’ relative access to relevant information; (4) the parties’ resources; (5) the importance of the discovery in resolving the issues; and (6) whether the burden or expense of the proposed discovery outweighs its likely benefit.

While all of the factors may be significant, monetary considerations – elements of which are found in both the “amount in controversy” and “burden or expense” factors – frequently predominate a proportionality analysis. As Ralph Losey (the owner, host, and principal author of this blog) has emphasized many times, “[t]he bottom line in e-discovery production is what it costs.” By following the money or, perhaps more appropriate for discovery, focusing on the money, counsel can drive an effective discovery process and obtain better results for the client.

As lawyers do so, they will find an increasingly sophisticated judiciary who expect counsel to approach discovery through the lens of proportionality. This certainly was the case in Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, which has been prominently spotlighted in this blog. In Oxbow, the court applied the Rule 26(b)(1) proportionality factors to a disputed document request, holding that it was not unduly burdensome and that it properly targeted relevant information. While the court examined all of the Rule 26(b)(1) proportionality standards, money was the clearly determinative factor. The amount in controversy, coupled with the comparative costs of discovery – discovery completed and still to be undertaken, tipped the scales in favor of ordering plaintiffs to respond defendants’ document requests.

The Critical Role of Metrics

Essential to Oxbow’s holding were the metrics the parties shared with the court. Metrics – typically defined as a standard of measurement or (as used in business world) a method for evaluating performance – offer counsel ways to assess the “performance” of a particular document production. Metrics can measure the extent to which a production contains relevant materials, undisclosed privileged information, and even nonresponsive documents. Metrics can also estimate – as was the case in Oxbow – the resources (including time, manpower, and costs) a party may be forced to incur to comply with a discovery request.

Metrics enable a court to follow the money and properly balance the burdens of discovery against its benefits. Without metrics, a responding party could hardly expect to establish that a request is disproportionate and thereby prevail in motion practice. As Ralph observed in his post entitled Judge Facciola’s Successor, Judge Michael Harvey, Provides Excellent Proportionality Analysis in an Order to Compel:

Successful arguments on motions to compel require hard evidence. To meet your burden of proof you must present credible estimates of the costs of document review. This requires . . . reliable metrics and statistics concerning the ESI that the requesting party wants the responding party to review.

As discussed later on, other courts have also emphasized the critical role of metrics in evaluating the proportionality of a particular discovery request.

The Sedona Conference, Proportionality, and Metrics

For counsel who wish to better understand the role of metrics in discovery, the directive to “follow the money” will bring them to The Sedona Conference (“Sedona”). Sedona is the preeminent legal institution dedicated to advancing thoughtful reforms on important legal issues. While Sedona addresses matters ranging from patent litigation and trade secret misappropriation to data privacy and cross-border data protection, the organization is best known for its work on electronic discovery.

Renowned for its volunteer model and for attracting many of the best minds in the legal industry, Sedona prepares authoritative resources that are regularly relied on by judges, lawyers, and scholars. This is particularly the case with proportionality standards and how they should drive the determination of discovery issues.

Sedona published its first Commentary on Proportionality in Electronic Discovery (“Commentary” or “Proportionality Commentary”) in 2010 and a second version in 2013. Last spring, Sedona released a third iteration of the Commentary. Collaboratively prepared by a group of renowned judges and practitioners, the third version of the Commentary provides common sense direction on how metrics can help achieve proportional results in discovery:

Burden and expense should be supported by hard information and not by unsupported assertions. For example, if a party claims that a search would result in too many documents, the party should run the search and be prepared to provide the opposing party with the number of hits and any other applicable qualitative metrics. If the party claims that the search results in too many irrelevant hits, the party may consider providing a description or examples of irrelevant documents captured by the search.

Quantitative metrics in support of a burden and expense argument may include the projected volume of potentially responsive documents. It may also encompass the costs associated with processing, performing data analytics, and review, taking into consideration the anticipated rate of review and reviewer costs, based upon reasonable fees and expenses.

As the Commentary makes clear, metrics can provide insights regarding the effectiveness of search methodologies or the nature and extent of a party’s burden in responding to a particular discovery request. By sharing these metrics with litigation adversaries, counsel can informally address legitimate discovery questions or crystallize the issues for resolution by a court. Either way represents a more cost effective approach to discovery than the opacity of traditional meet and confers or motion practice.

Framing the Issues through Sedona’s TSCI Event

These issues were on display last month at Sedona’s TSCI conference in Nashville, Tennessee. The TSCI event typically provides attendees with an annual opportunity to stay current on developing trends in e-Discovery. The 2018 TSCI event remained consistent with that objective, spotlighting practice developments for counsel “from ‘eDiscovery 1.0’ to New and Evolving Legal Challenges.” Expertly chaired by Jerone “Jerry” English and Maura Grossman, TSCI featured sessions covering discovery and other issues relating to artificial intelligence (AI), the Internet of Things (IoT), mobile applications, data breaches, cross-border discovery, and the always engaging case law panel and judicial round-table.

One of the more practical sessions focused on the importance of using metrics, analytics, and sampling to achieve proportionality in discovery. Entitled Using Data Analytics and Metrics to Achieve Proportionality, the purpose of this session was to help attendees understand how counsel should present analytics, metrics, and sampled data to a court. The session featured a fantastic line-up of speakers – Gareth Evans, Maura Grossman, U.S. Magistrate Judge Anthony Porcelli, U.S. Magistrate Judge Leda Dunn Wettre – who were well situated to provide views on these topics. Audience members additionally offered insightful comments on the issues.[1]

The most important guidance the speakers and audience emphasized was the need for more complete disclosure of supporting metrics. Unless specific metrics are disclosed, neither adversaries nor the court can address issues as varied as the performance of particular search terms, the reasonableness of a production made using TAR or other search methodologies, or the burdens of a particular discovery request.

On the latter issue of substantiating arguments of undue burden, one particularly insightful comment offered during the session concisely summarized the interplay between metrics, proportionality, and cost: “Follow the money.” This admonition dovetailed nicely with the discussion of two recent cases during that session – Duffy v. Lawrence Memorial Hospital, No. 2:14-cv-2256-SAC-TJJ, 2017 WL 1277808 (D. Kan. Mar. 31, 2017) and Solo v. United Parcel Service Co., No. 14-cv-12719, 2017 WL 85832 (E.D. Mich. Jan. 10, 2017). Both of these cases spotlight how reliable metrics enable a court to follow the money and resolve discovery disputes consistent with proportionality standards.

Duffy v. Lawrence Memorial Hospital

In Duffy, the court modified a discovery order issued less than two months beforehand that granted plaintiff’s requests for various categories of emergency room patient records. In that first round of motion practice, defendant had argued that plaintiff’s requests were disproportionate and unduly burdensome. The court overruled those objections, explaining that defendant failed to provide any substantive metrics to support those objections:

Defendant objects to every document request as being unduly burdensome, but provides no facts to support the objection. Neither does Defendant provide evidence of the costs it would incur in responding to the requests.

In summary, defendant’s failure to share any meaningful metrics regarding the time, manpower, or costs it would incur to comply with plaintiff’s requests ultimately left its arguments bereft of any evidentiary support.

In the second round of motion practice, defendant adopted a different approach that yielded a more proportional result. Confronted by the staggering reality of the court’s production order and having learned how to properly use supporting metrics in motion practice, defendant moved for a protective order.

In contrast to its prior briefing, defendant shared specific metrics associated with the burdens of production. Those burdens involved the deployment of staff to individually review 15,574 electronic patient files so as to identify particular patient visit information. Such a process would be labor intensive and cost well over $230,000:

Defendant estimates it would take 7,787 worker hours to locate and produce responsive information for 15,574 patient records. If Defendant had ten employees working on the task, they would spend more than ninety-seven days working eight hours a day, at an estimated cost to Defendant of $196,933.23.

After aggregating the information, Defendant asserts it would need to redact patients’ personal confidential information . . . redaction would take ten reviewers fourteen days at a cost of $37,259.50. The process would include a quality control attorney reviewer who would spend two hours a day, and reviewers who would review 15 documents per hour for eight hours a day.

In sum . . . producing the information relevant to RFP Nos. 40, 41, 43, and 58 would take 8,982 hours of work and cost in excess of $230,000 if done by contract staff.

Simply put, defendant urged the court to follow the money. By substantiating its proportionality arguments with appropriate metrics, the court recognized its initial production order placed an undue burden on defendant.

As a result, the court adopted a modified order that instead allowed defendant to produce a random sample of 257 patient records. While advancing a number of justifications for its modified order, the court ultimately relied on the tripartite mandate from Federal Rule of Civil Procedure 1. The order would provide the parties to the litigation with a substantively better, more efficient, and less expensive method for producing relevant information.

Solo v. United Parcel Service

Solo v. United Parcel Service reached a result analogous to the Duffy holding, ordering that defendant produce only a sample of the information sought by plaintiff. In Solo, plaintiffs served an interrogatory that sought identification of shipment information relating to its putative class action claims (plaintiffs claimed that defendant overcharged certain customers for “shipments that had a declared value of over $300”). The interrogatory sought shipping record information that spanned a period of six years.

Defendant argued in response that the interrogatory was unduly burdensome and would impose a disproportionate production obligation on the company. Because most of the requested information was archived on backup tapes, defendant shared specific metrics regarding the “overwhelming” burdens associated with responding to the interrogatory:

UPS estimates that it would take at least six months just to restore the archived tapes as described above, at a cost of $120,000 in labor . . . that estimate does not include the time and expense of analyzing the data once extracted in order to answer Interrogatory No. 1, which would require extensive additional analysis of each account number and the manual review of contract language for individual shipper. Such a process would also require a substantial amount of time and resources on the part of UPS.

Based on the metrics defendant disclosed and given that plaintiffs’ claims had yet to be certified as a class action, the court found the interrogatory to be disproportionate. Following the money and drawing on the linked concepts of cooperation and proportionality from Rule 1, the court instead ordered that defendant produce a sample of the requested information from a six-month period. The court also directed the parties to meet and confer on developing an agreeable sampling methodology.

Conclusion

Duffy and Solo reinforce the critical interplay between metrics, proportionality, and money. Just like Oxbow, the responding parties from Duffy and Solo could hardly expect to substantiate arguments regarding undue burden and disproportionality without metrics. Indeed, the court in Duffy initially rejected such arguments when defendant failed to support them with actual information. However, by disclosing metrics with reasonable estimates of time, manpower, and costs, Duffy and Solo resulted in production orders more consistent with proportionality limitations.

All of which translated into substantial cost savings for the responding parties. Defendant in Duffy was facing a discovery bill of over $230,000 to review 15,574 patient files. Dividing the projected cost of the entire review process into the number of patient records – $230,000 ÷ 15,574 – reveals that defendant would pay approximately $15 to review an individual patient record. Under the modified production order, the new projected cost – $15 multiplied by 257 patient records – equals $3,855. Follow the money: the tactical use of metrics apparently saved the client over $225,000!

Duffy and Solo are consistent with and confirmed by Oxbow, the Proportionality Commentary, and the Federal Rules of Civil Procedure. These authoritative resources collectively teach that counsel who use metrics and focus on cost can drive an effective discovery process. Lawyers that do so will ultimately obtain better results for the client in discovery.

_______

[1] To encourage candid and robust debate during its events, Sedona has promulgated a nondisclosure rule. Known as “The Sedona Rule,” it proscribes attendees from identifying the speakers or audience members by name who share particular insights. It also forbids divulging the contents of particular brainstorming or drafting projects that have yet to be released for publication. The Sedona Rule otherwise allows for the anonymous disclosure of session content from its events.

1 Comment | Lawyers Duties, New Rules, Review, Technology | Tagged: best practices, document review, ethics, PROPORTIONALITY | Permalink
Posted by Ralph Losey

Document Review and Proportionality – Part Two

March 28, 2018

This is a continuation of a blog that I started last week. Suggest you read Part One before this.

Simplified Six Step Review Plan for Small and Medium Sized Cases or Otherwise Where Predictive Coding is Not Used

Here is the workflow for the simplified six-step plan. The first three steps repeat until you have a viable plan where the costs estimate is proportional under Rule 26(b)(1).

Step One: Multimodal Search

The document review begins with Multimodal Search of the ESI. Multimodal means that all modes of search are used to try to find relevant documents. Multimodal search uses a variety of techniques in an evolving, iterated process. It is never limited to a single search technique, such as keyword. All methods are used as deemed appropriate based upon the data to be reviewed and the software tools available. The basic types of search are shown in the search pyramid.

In Step One we use a multimodal approach, but we typically begin with keyword and concept searches. Also, in most projects we will run similarity searches of all kinds to make the review more complete and broaden the reach of the keyword and concept searches. Sometimes we may even use a linear search, expert manual review at the base of the search pyramid. For instance, it might be helpful to see all communications that a key witness had on a certain day. The two-word stand-alone call me email when seen in context can sometimes be invaluable to proving your case.

I do not want to go into too much detail of the types of searches we do in this first step because each vendor’s document review software has different types of searches built it. Still, the basic types of search shown in the pyramid can be found in most software, although AI, active machine learning on top, is still only found in the best.

History of Multimodal Search

Professor Marcia Bates

Multimodal search, wherein a variety of techniques are used in an evolving, iterated process, is new to the legal profession, but not to Information Science. That is the field of scientific study which is, among many other things, concerned with computer search of large volumes of data. Although the e-Discovery Team’s promotion of multimodal search techniques to find evidence only goes back about ten years, Multimodal is a well-established search technique in Information Science. The pioneer professor who first popularized this search method was Marcia J. Bates, and her article, The Design of Browsing and Berrypicking Techniques for the Online Search Interface, 13 Online Info. Rev. 407, 409–11, 414, 418, 421–22 (1989). Professor Bates of UCLA did not use the term multimodal, that is my own small innovation, instead she coined the word “berrypicking” to describe the use of all types of search to find relevant texts. I prefer the term “multimodal” to “berrypicking,” but they are basically the same techniques.

In 2011 Marcia Bates explained in Quora her classic 1989 article and work on berrypicking:

An important thing we learned early on is that successful searching requires what I called “berrypicking.” . . .

Berrypicking involves 1) searching many different places/sources, 2) using different search techniques in different places, and 3) changing your search goal as you go along and learn things along the way. . . .

This may seem fairly obvious when stated this way, but, in fact, many searchers erroneously think they will find everything they want in just one place, and second, many information systems have been designed to permit only one kind of searching, and inhibit the searcher from using the more effective berrypicking technique.

Marcia J. Bates, Online Search and Berrypicking, Quora (Dec. 21, 2011). Professor Bates also introduced the related concept of an evolving search. In 1989 this was a radical idea in information science because it departed from the established orthodox assumption that an information need (relevance) remains the same, unchanged, throughout a search, no matter what the user might learn from the documents in the preliminary retrieved set. The Design of Browsing and Berrypicking Techniques for the Online Search Interface. Professor Bates dismissed this assumption and wrote in her 1989 article:

In real-life searches in manual sources, end users may begin with just one feature of a broader topic, or just one relevant reference, and move through a variety of sources. Each new piece of information they encounter gives them new ideas and directions to follow and, consequently, a new conception of the query. At each stage they are not just modifying the search terms used in order to get a better match for a single query. Rather the query itself (as well as the search terms used) is continually shifting, in part or whole. This type of search is here called an evolving search.

Furthermore, at each stage, with each different conception of the query, the user may identify useful information and references. In other words, the query is satisfied not by a single final retrieved set, but by a series of selections of individual references and bits of information at each stage of the ever-modifying search. A bit-at-a-time retrieval of this sort is here called berrypicking. This term is used by analogy to picking huckleberries or blueberries in the forest. The berries are scattered on the bushes; they do not come in bunches. One must pick them one at a time. One could do berrypicking of information without the search need itself changing (evolving), but in this article the attention is given to searches that combine both of these features.

I independently noticed evolving search as a routine phenomena in legal search and only recently found Professor Bates’ prior descriptions. I have written about this often in the field of legal search (although never previously crediting Professor Bates) under the names “concept drift” or “evolving relevance.” See Eg. Concept Drift and Consistency: Two Keys To Document Review Quality – Part Two (e-Discovery Team, 1/24/16). Also see Voorhees, Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 36 Info. Processing & Mgmt 697 (2000) at page 714.

SIDE NOTE: The somewhat related term query drift in information science refers to a different phenomena in machine learning. In query drift the concept of document relevance unintentionally changes from the use of indiscriminate pseudorelevance feedback. Cormack, Buttcher & Clarke, Information Retrieval Implementation and Evaluation of Search Engines (MIT Press 2010) at pg. 277. This can lead to severe negative relevance feedback loops where the AI is trained incorrectly. Not good. If that happens a lot of other bad things can and usually do happen. It must be avoided.

Yes. That means that skilled humans must still play a key role in all aspects of the delivery and production of goods and services, lawyers too.

UCLA Berkeley Professor Bates first wrote about concept shift when using early computer assisted search in the late 1980s. She found that users might execute a query, skim some of the resulting documents, and then learn things which slightly changes their information need. They then refine their query, not only in order to better express their information need, but also because the information need itself has now changed. This was a new concept at the time because under the Classical Model Of Information Retrieval an information need is single and unchanging. Professor Bates illustrated the old Classical Model with the following diagram.

The Classical Model was misguided. All search projects, including the legal search for evidence, are an evolving process where the understanding of the information need progresses, improves, as the information is reviewed. See diagram below for the multimodal berrypicking type approach. Note the importance of human thinking to this approach.

See Cognitive models of information retrieval (Wikipedia). As this Wikipedia article explains:

Bates argues that searches are evolving and occur bit by bit. That is to say, a person constantly changes his or her search terms in response to the results returned from the information retrieval system. Thus, a simple linear model does not capture the nature of information retrieval because the very act of searching causes feedback which causes the user to modify his or her cognitive model of the information being searched for.

Multimodal search assumes that the information need evolves over the course of a document review. It is never just run one search and then review all of the documents found in the search. That linear approach was used in version 1.0 of predictive coding, and is still used by most lawyers today. The dominant model in law today is linear, wherein a negotiated list of keyword is used to run one search. I called this failed method “Go Fish” and a few judges, like Judge Peck, picked up on that name. Losey, R., Adventures in Electronic Discovery (West 2011); Child’s Game of ‘Go Fish’ is a Poor Model for e-Discovery Search; Moore v. Publicis Groupe & MSL Group, 287 F.R.D. 182, 190-91, 2012 WL 607412, at *10 (S.D.N.Y. Feb. 24, 2012) (J. Peck).

The popular, but ineffective Go Fish approach is like the Classical Information Retrieval Model in that only a single list of keywords is used as the query. The keywords are not refined over time as the documents are reviewed. This is a mono-modal process. It is contradicted by our evolving multimodal process, Step One in our Six-Step plan. In the first step we run many, many searches and review some of the results of each search, some of the documents, and then change the searches accordingly.

Step Two: Tests, Sample

Each search run is sampled by quick reviews and its effectiveness evaluated, tested. For instance, did a search of what you expected would be an unusual word turn up far more hits than anticipated? Did the keyword show up in all kinds of documents that had nothing to do with the case? For example, a couple of minutes of review might show that what you thought would be a carefully and rarely used word, Privileged, was in fact part of the standard signature line of one custodian. All his emails had the keyword Privileged on them. The keyword in these circumstances may be a surprise failure, at least as to that one custodian. These kind of unexpected language usages and surprise failures are commonplace, especially with neophyte lawyers.

Sampling here does not mean random sampling, but rather judgmental sampling, just picking a few representative hit documents and reviewing them. Were a fair number of berries found in that new search bush, or not? In our example, assume that your sample review of the documents with “Privileged” showed that the word was only part of one person’s standard signature on every one of their emails. When a new search is run wherein this custodian is excluded, the search results may now test favorably. You may devise other searches that exclude or limit the keyword “Privileged” whenever it is found in a signature.

There are many computer search tools used in a multimodal search method, but the most important tool of all is not algorithmic, but human. The most important search tool is the human ability to think the whole time you are looking for tasty berries. (The all important “T” in Professor Bates’ diagram above.) This means the ability to improvise, to spontaneously respond and react to unexpected circumstances. This mean ad hoc searches that change with time and experience. It is not a linear, set it and forget it, keyword cull-in and read all documents approach. This was true in the early days of automated search with Professor Bates berrypicking work in the late 1980s, and is still true today. Indeed, since the complexity of ESI has expanded a million times since then, our thinking, improvisation and teamwork are now more important than ever.

The goal in Step Two is to identify effective searches. Typically, that means where most of the results are relevant, greater than 50%. Ideally we would like to see roughly 80% relevancy. Alternatively, search hits that are very few in number, and thus inexpensive to review them all, may be accepted. For instance, you may try a search that only has ten documents, which you could review in just a minute. You may just find one relevant, but it could be important. The acceptable range of number of documents to review in Bottom Line Driven Review will always take cost into consideration. That is where Step-Three comes in, Estimation. What will it costs to review the documents found?

Step Three: Estimates

It is not enough to come up with effective searches, which is the goal of Steps One and Two, the costs involved to review all of the documents returned with these searches must also be considered. It may still cost way too much to review the documents when considering the proportionality factors under 26(b)(1) as discussed in Part One of this article. The plan of review must always take the cost of review into consideration.

In Part One we described an estimation method that I like to use to calculate the cost of an ESI review. When the projected cost, the estimate, is proportional in your judgment (and, where appropriate, in the judge’s judgment), then you conclude your iterative process of refining searches. You can then move onto the next Step-Four of preparing your discovery plan and making disclosures of that plan.

Step Four: Plan, Disclosures

Once you have created effective searches that produce an affordable number of documents to review for production, you articulate the Plan and make some disclosures about your plan. The extent of transparency in this step can vary considerably, depending on the circumstances and people involved. Long talkers like me can go on about legal search for many hours, far past the boredom tolerance level of most non-specialists. You might be fascinated by the various searches I ran to come up with the say 12,472 documents for final review, but most opposing counsel do not care beyond making sure that certain pet keywords they may like were used and tested. You should be prepared to reveal that kind of work-product for purposes of dispute avoidance and to build good will. Typically they want you to review more documents, no matter what you say. They usually save their arguments for the bottom line, the costs. They usually argue for greater expense based on the first five criteria of Rule 26(b)(1):

the importance of the issues at stake in this action;
the amount in controversy;
the parties’ relative access to relevant information;
the parties’ resources;
the importance of the discovery in resolving the issues; and
whether the burden or expense of the proposed discovery outweighs its likely benefit.

Still, although early agreement on scope of review is often impossible, as the requesting party always wants you to spend more, you can usually move past this initial disagreement by agreeing to phased discovery. The requesting party can reserve its objections to your plan, but still agree it is adequate for phase one. Usually we find that after that phase one production is completed the requesting party’s demands for more are either eliminated or considerably tempered. It may well now to possible to reach a reasonable final agreement.

Step Five: Final Review

Here is where you start to carry out your discovery plan. In this stage you finish looking at the documents and coding them for Responsiveness (relevant), Irrelevant (not responsive), Privileged (relevant but privileged, and so logged and withheld) and Confidential (all levels, from just notations and legends, to redactions, to withhold and log. A fifth temporary document code is used for communication purposes throughout a project: Undetermined. Issue tagging is usually a waste of time and should be avoided. Instead, you should rely on search to find documents to support various points. There are typically only a dozen or so documents of importance at trial anyway, no matter what the original corpus size.

I highly recommend use of professional document review attorneys to assist you in this step. The so-called “contract lawyers” specialize in electronic document review and do so at a very low cost, typically in the neighborhood of $50 per hour. The best of them, who may often command slightly higher rates, are speed readers with high comprehension. They also know what to look for in different kinds of cases. Some have impressive backgrounds. Of course, good management of these resources is required. They should have their own management and team leaders. Outside attorneys signing Rule 26(g) will also need to supervise them carefully, especially as to relevance intricacies. The day will come when a court will find it unreasonable not to employ these attorneys in a document review. The savings is dramatic and this in turn increases the persuasiveness of your cost burden argument.

Step Six: Production

The last step is transfer of the appropriate information to the requesting party and designated members of your team. Production is typically followed by later delivery of a Log of all documents withheld, even though responsive or relevant. The withheld logged documents are typically: Attorney-Client Communications protected from disclosure under the client’s privilege; or, Attorney Work-Product documents protected from disclosure under the attorney’s privilege. Two different privileges. The attorney’s work-product privilege is frequently waived in some part, although often very small. The client’s communications with its attorneys is, however, an inviolate privilege that is never waived.

Typically you should produce in stages and not wait until project completion. The only exception might be where the requesting party would rather wait and receive one big production instead of a series of small productions. That is very rare. So plan on multiple productions. We suggest the first production be small and serve as a test of the receiving party’s abilities and otherwise get the bugs out of the system.

Conclusion

In this essay I have shown the method I use in document reviews to control costs by use of estimation and multimodal search. I call this a Bottom Line Driven approach. The six step process is designed to help uncover the costs of review as part of the review itself. This kind of experienced based estimate is an ideal way to meet the evidentiary burdens of a proportionality objection under revised Rules 26(b)(1) and 32(b)(2). It provides the hard facts needed to be specific as to what you will review and what you will not and the likely costs involved.

The six-step approach described here uses the costs incurred at the front end of the project to predict the total expense. The costs are controlled by use of best practices, such as contract review lawyers, but primarily by limiting the number of documents reviewed. Although it is somewhat easier to follow this approach using predictive coding and document ranking, it can still be done without that search feature. You can try this approach using any review software. It works well in small or medium sized projects with fairly simple issues. For large complex projects we still recommend using the eight-step predictive coding approach as taught in the TarCourse.com.

Leave a Comment » | Evidence, Lawyers Duties, New Rules, Review, Search, Spoliation/Sanctions, Technology, VENDORS | Tagged: best practices, computer assisted review, costs, discovery, document review, legal search, PROPORTIONALITY, search | Permalink
Posted by Ralph Losey

Document Review and Proportionality – Part One

March 18, 2018

In 2013 I wrote a law review article on how the costs of document review could be controlled using predictive coding and cost estimation. Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data, 26 Regent U. Law Review 1 (2013-2014). Today I write on how it can be controlled in document review, even without predictive coding. Here is the opening paragraph of my earlier article:

The search of electronic data to try to find evidence for use at trial has always been difficult and expensive. Over the past few years, the advent of Big Data, where both individuals and organizations retain vast amounts of complex electronic information, has significantly compounded these problems. The legal doctrine of proportionality responds to these problems by attempting to constrain the costs and burdens of discovery to what are reasonable. A balance is sought between the projected burdens and likely benefits of proposed discovery, considering the issues and value of the case. Several software programs on the market today have responded to the challenges of Big Data by implementing a form of artificial intelligence (“AI”) known as active machine learning to help lawyers review electronic documents. This Article discusses these issues and shows that AI-enhanced document review directly supports the doctrine of proportionality. When used together, proportionality and predictive coding provide a viable, long-term solution to the problems and opportunities of the legal search of Big Data.

The 2013 article was based on version 1.0 Predictive Coding. Under this first method you train and rank documents and then review only the higher ranking documents. Here is a more detailed description from pages 23, 24 of the article:

This kind of AI-enhanced legal review is typically described today in legal literature by the term

predictive coding. This is because the computer predicts how an entire body of documents should be coded (classified) based on how the lawyer has coded the smaller training sets. The prediction places a probability ranking on each document, typically ranging from 0% to 100% probability. Thus, in a

relevancy classification, each and every document in the entire dataset (the corpus) is ranked with a percentage of likely relevance and irrelevance. …

As will be shown, this ranking feature is key to the use of the legal doctrine of proportionality. The ability to rank all documents in a corpus on probable relevance is a new feature that no other legal search software has previously provided.71

It was a two phase procedure: train then review. Yes, some review would take place in the first training phase, but this would be a relatively small number, say 10-20% of the total documents reviewed. Most of the human review of documents would take place in phase two. The workflow of version 1.0 is shown in the diagram below and is described in detail in the article, starting at page 31.

Predictive Coding and the Proportionality Doctrine argued that attorneys should scale the number of documents for the second phase of document review based on estimated costs constrained to a proportional amount. No more spending $100,000 for document review in a $200,000 case. The number of documents you selected for review would be limited to proportional costs. Predictive coding and its ranking features allowed you to select the documents for review that were most likely to be relevant. If you could only afford to spend $20,000 on a document review project, then you would limit the number of documents reviewed to those within that scope that were the highest ranked as probable relevant. Here is the article’s description at pages 54-55 of the process and link between the doctrine of proportionality and predictive coding.

What makes this a marriage truly made in heaven is the document-ranking capabilities of predictive coding. This allows parties to limit the documents considered for final production to those that the computer determines have the highest probative value. This key ranking feature of AI-enhanced document review allows the producing party to provide the requesting party with the most bang for the buck. This not only saves the producing party money, and thus keeps its costs proportional, but it saves time and expenses for the requesting party. It makes the production much more precise, and thus faster and easier to review. It avoids what can be a costly exercise to a requesting party to wade through a document dump 192, a production that contains a high number of irrelevant or marginally relevant documents. Most importantly, it gives the requesting party what it really wants—the documents that are the most important to the case.

In the article, pages 58-60, I called this method Bottom-Line-Driven Proportional Review and describe the process in greater detail.

The bottom line in e-discovery production is what it costs. Despite what some lawyers and vendors may say, total cost is not an impossible question to answer. It takes an experienced lawyer’s skill to answer, but,

after a while, you can get quite good at such estimation. It is basically a matter of estimating attorney billable hours plus vendor costs. With practice, cost estimation can become a reliable art, a projection that you can count on for budgeting purposes, and, as we will see, for proportionality arguments. …

The new strategy and methodology is based on a bottom line approach where you estimate what review costs will be, make a proportionality analysis as to what should be spent, and then engage in defensible culling to bring the review costs within the proportional budget. The producing party determines the number of documents to be subjected to final review by calculating backwards from the bottom line of what they are willing, or required, to pay for the production. …

Under the Bottom-Line-Driven Proportional approach, after analyzing the case merits and determining the maximum proportional expense, the responding party makes a good faith estimate of the likely

maximum number of documents that can be reviewed within that budget. The document count represents the number of documents that you estimate can be reviewed for final decisions of relevance, confidentiality, privilege, and other issues and still remain within budget. The review costs you estimate must be based on best practices, which in all large review projects today means predictive coding, and the estimates must be accurate (i.e., no puffing or mere guesswork).

Note this last quote (emphasis added) from Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data shows an important limitation to the article’s budgetary proposal, it was limited to large review projects where predictive coding was used. Without this marriage to predictive coding, the promise of proportionality by cost estimations was lost. My article today fills this gap.

Here I will explain how document review can be structured to provide estimates and review constraints, even when predictive coding and its ranking are not used. This is, in effect, the single lawyers guide, one where there has not been a marriage with predictive coding. It is a guide for small and medium sized document review projects, which are, after all, the vast majority of projects faced by the legal profession.

To be honest, back when I first wrote the law review article I did not think it would be necessary to develop such a proportionality method, one that does not use AI document ranking. I assumed predictive coding would take off and by now would be used in almost all projects, no matter what the size. I assumed that since active machine learning and document ranking was such good new technology, that even our conservative profession would embrace it within the next few years. Boy was I wrong about that. The closing lines of Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data have been proven naive.

The key facts needed to try a case and to do justice can be found in any size case, big and small, at an affordable price, but you have to embrace change and adopt new legal and technical methodologies. The Bottom-Line-Driven Proportional Review method is part of that answer, and so too is advanced-review software at affordable prices. When the two are used together, it is a marriage made in heaven.

I blame both lawyers and e-discovery vendors for this failure, as well as myself for misjudging my peers. Law schools and corporate counsel have not helped much either. Only the judiciary seems to have caught on and kept up.

Proportionality as a legal doctrine took off as expected after 2013, but not the marriage with predictive coding. Lawyers have proven to be much more stubborn than anticipated. They will barely even go out with predictive coding, no matter how attractive she is, much less marry her. The profession as a whole remains remarkably slow to adopt new technology. The judges are tempted to use their shotgun to force a wedding, but so far have refrained from ordering a party to use predictive coding. Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP), 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016) (J. Peck: “There may come a time when TAR is so widely used that it might be unreasonable for a party to decline to use TAR. We are not there yet.”)

Changes Since 2013

A lot has happened since 2013 when Predictive Coding and the Proportionality Doctrine was written. In December 2015 Rule 26(b) on relevance was revised to strengthen proportionality and we have made substantial improvements to Predictive Coding methods. In the ensuing years most experts have abandoned this early two-step method of train then review in favor of a method where training continues throughout the review process. In other words, today we keep training until the end. See Eg. the e-Discovery Team’s Predictive Coding, version 4.0, with its Intelligently Spaced Training. (This is similar to a method popularized by Maura Grossman and Gordon Cormack, which they called Continuous Active Learning or CAL for short, a term they later trademarked.)

The 2015 revision to Rule 26(b) on relevance has spurred case law and clarified that undue burden, the sixth factor of proportionality under Rule 26(b)(1), must be argued in detail with facts proven.

the importance of the issues at stake in this action;
the amount in controversy;
the parties’ relative access to relevant information;
the parties’ resources;
the importance of the discovery in resolving the issues; and
whether the burden or expense of the proposed discovery outweighs its likely benefit.”

Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, No. 11-cv-1049 (PLF/GMH), 2017 WL 4011136, (D.D.C. Sept. 11, 2017). Although all factors are important and should be addressed, the last factor is usually the most important one in a discovery dispute. It is also the factor that can be addressed generally for all cases and is the core of proportionality.

Proportional “Bottom Line Driven” Method of Document Review that Does Not Require Use of Predictive Coding

I have shared how I use predictive coding with continuous training in my TARcourse.com online instruction program. The eight-step workflow is shown below.

I have not previously shared any information on the document review workflow that I follow in small and medium seized cases where predictive coding is not used. The rest of this article will do so now.

Please note that I have a well-developed and articulated list of steps and procedures for attorneys in my law firm to follow in such small cases. I keep this as a trade-secret and will not reveal them here. Although they are widely known in my firm, and slightly revised and updated each year, they are not public. Still, any expert in document review should be able to create their own particular rules and implementation methods. Warning, if you are not such an expert, be careful in relying on these high-level explanations alone. The devil is in the details and you should retain an expert to assist.

Here is a chart summarizing the SIX-Step Workflow and six basic concepts that must be understood for the process to work at maximum efficiency.

The first three steps iterate with searches to cull out the irrelevant documents, and then culminate with Disclosures of the plan developed for Steps Five and Six, Final Review and Production. The sixth production step is always in phases according to proportional planning.

A key skill that must be learned is project cost estimation, including fees and expenses. The attorneys involved must also learn how to communicate with themselves, the vendors, opposing counsel and the court. Rigid enforcement of work-product confidentiality is counter-productive to the goal of cost efficient projects. Agree on the small stuff and save your arguments for the cost-saving questions that are worth the effort.

The Proportionality Doctrine

he doctrine of proportionality as a legal initiative was launched by The Sedona Conference in 2010 as a reaction to the exploding costs of e-discovery. The Sedona Conference, The Sedona Conference Commentary on Proportionality in Electronic Discovery, 11 SEDONA CONF. J. 289, 292–94 (2010). See also John L. Carroll, Proportionality in Discovery: A Cautionary Tale, 32 CAMPBELL L. REV. 455, 460 (2010) (“If courts and litigants approach discovery with the mindset of proportionality, there is the potential for real savings in both dollars and time to resolution.”); Maura Grossman & Gordon Cormack, Some Thoughts on Incentives, Rules, and Ethics Concerning the Use of Search Technology in E-Discovery, 12 SEDONA CONF. J. 89, 94–95, 101–02 (2011).

The doctrine received a big boost with the adoption of the 2015 Amendment to Rule 26. The rule was changed to provide discovery must be both relevant and “proportional to the needs of the case.” Fed. R. Civ. P. 26(b)(1). To determine whether a discovery request is proportional, you are required weigh the following six factors: “(1) the importance of the issues at stake in this action; (2) the amount in controversy; (3) the parties’ relative access to relevant information; (4) the parties’ resources; (5) the importance of the discovery in resolving the issues; and (6) whether the burden or expense of the proposed discovery outweighs its likely benefit.” Williams v. BASF Catalysts, LLC, Civ. Action No. 11-1754, 2017 WL 3317295, at *4 (D.N.J. Aug. 3, 2017) (citing Fed. R. Civ. P. 26(b)(1)); Arrow Enter. Computing Solutions, Inc. v. BlueAlly, LLC, No. 5:15-CV-37-FL, 2017 WL 876266, at *4 (E.D.N.C. Mar. 3, 2017); FTC v. Staples, Inc., Civ. Action No. 15-2115 (EGS), 2016 WL 4194045, at *2 (D.D.C. Feb. 26, 2016).

“[N]o single factor is designed to outweigh the other factors in determining whether the discovery sought is proportional,” and all proportionality determinations must be made on a case-by-case basis. Williams, 2017 WL 3317295, at *4 (internal citations omitted); see also Bell v. Reading Hosp., Civ. Action No. 13-5927, 2016 WL 162991, at *2 (E.D. Pa. Jan. 14, 2016). To be sure, however, “the amendments to Rule 26(b) do not alter the basic allocation of the burden on the party resisting discovery to—in order to successfully resist a motion to compel—specifically object and show that . . . a discovery request would impose an undue burden or expense or is otherwise objectionable.” Mir v. L-3 Commc’ns Integrated Sys., L.P., 319 F.R.D. 220, 226 (N.D. Tex. 2016), as quoted by Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, No. 11-cv-1049 (PLF/GMH), 2017 WL 4011136, (D.D.C. Sept. 11, 2017).

The Oxbow case is discussed at length in my recent blog Judge Facciola’s Successor, Judge Michael Harvey, Provides Excellent Proportionality Analysis in an Order to Compel (e-Discovery Team,3/1/18). Judge Harvey carefully examined the costs and burdens claimed by plaintiffs and rejected the overly burdensome argument.

Plaintiffs’ counsel explained at the second hearing in this matter that Oxbow has spent $1.391 million to date on reviewing and producing approximately 584,000 documents from its nineteen other custodians and Oxbow’s email archive. See 8/24/17 TR. at 44:22-45:10. And again, Oxbow seeks tens of millions of dollars from Defendants. Through that lens, the estimated cost of reviewing and producing Koch’s responsive documents—even considering the total approximate cost of $142,000 for that effort, which includes the expense of the sampling effort—while certainly high, is not so unreasonably high as to warrant rejecting Defendants’ request out of hand. See Zubulake v. UBS Warburg, LLC, 217 F.R.D. 309, 321 (S.D.N.Y. 2003) (explaining, in the context of a cost-shifting request, that “[a] response to a discovery request costing $100,000 sounds (and is) costly, but in a case potentially worth millions of dollars, the cost of responding may not be unduly burdensome”); Xpedior Creditor Trust v. Credit Suisse First Boston (USA), Inc., 309 F. Supp. 2d 459, 466 (S.D.N.Y. 2003) (finding no “undue burden or expense” to justify cost-shifting where the requested discovery cost approximately $400,000 but the litigation involved at least $68.7 million in damages). …

In light of the above analysis—including the undersigned’s assessment of each of the Rule 26 proportionality factors, all of which weigh in favor of granting Defendants’ motion—the Court is unwilling to find that the burden of reviewing the remaining 65,000 responsive documents for a fraction of the cost of discovery to date should preclude Defendants’ proposed request. See BlueAlly, 2017 WL 876266, at *5 (“This [last Rule 26] factor may combine all the previous factors into a final analysis of burdens versus benefits.” (citing Fed. R. Civ. P. 26 advisory committee’s notes)).

For more analysis and case law on proportionality see Proportionality Φ and Making It Easy To Play “e-Discovery: Small, Medium or Large?” in Your Own Group or Class, (e-Discovery Team, 11/26/17). Also see The Sedona Conference Commentary on Proportionality, May 2017.

Learning How to Estimate Document Review Costs

The best way to determine a total cost of a project is by projection from experience and analysis on a cost per file basis. General experience of review costs can be very helpful, but the gold standard comes from measurement of costs actually incurred in the same project, usually after several hours of work, or days, depending on the size of the project. You calculate costs incurred to date and then project forward on a cost per file basis. The is the core idea of the Six Step document review protocol that this article begins to explain.

The actual project costs are the best possible metrics for estimation. Apparently that was never done in Oxbow because the plaintiffs counsel’s projected document review cost estimates varied so much. A per file cost analysis of the information in the Oxbow opinion shows that the parties missed a key metric. The costs projected ranged from an actual cost of $2.38 per file for the first 584,000 files, to an 1.17 per file estimate to review 214,000 additional files, to an estimate of $1.73 per file to review 82,000 more files, to an actual cost of $4.74 per file to review 12,074 files, to final estimate of $1.22 per file to review the remaining 69,926 files. The actual costs are way higher than the estimated costs meaning the moving party cheated themselves by failing to do the math.

Here is how I explained the estimation process in Predictive Coding and the Proportionality Doctrine at pages 60-61:

Under the Bottom-Line-Driven Proportional approach, after analyzing the case merits and determining the maximum proportional expense, the responding party makes a good faith estimate of the likely maximum number of documents that can be reviewed within that budget. The document count represents the number of documents that you estimate can be reviewed for final decisions of relevance, confidentiality, privilege, and other issues and still remain within budget.

A few examples may help clarify how this method works. Assume a case where you determine a proportional cost of production to be $50,000, and estimate, based on sampling and other hard facts, that it will cost you $1.25 per file for both the automated and manual review before production of the ESI at issue … Then you can review no more than 40,000 documents and stay within budget. It is that simple. No higher math is required.

Estimation for bottom-line-driven review is essentially a method for marshaling evidence to support an undue burden argument under Rule 26(b)(1). Let’s run through it again with greater detail and make a simple formula to illustrate the process.

First, estimate the total number of documents remaining to be reviewed after culling by your tested keywords and other searches (hereinafter “T”). This is the most difficult step but is something most attorney experts and vendors are well qualified to help you with. Essentially “T” represents is the number of documents left unreviewed for Step Five, Final Review. These are the documents found in Steps One and Two, ECA Multimodal Search and Testing. These steps, along with the estimate calculation, usually repeat several times to cull-in the documents that are most likely relevant to the claims and defenses. The T – Total Documents Left for Review – are the documents in the final revised keyword search folders and concept, similarity search folders. The goal is to obtain a relevance precision in these folders greater than 50%, preferably at least 75%.

To begin an example hypothetical, assume that the total document count in the folders set-up for final review is 5,000 documents. T=5,000. Next count how many relevant and highly relevant files have already been located (hereinafter “R”). Assume for our example that 1,000 relevant and highly relevant documents have been found. R=1,000.

Next, look up the total attorney fees already incurred in the matter to date for the actual document search and review work by attorneys and paralegals (hereinafter collectively “F”). Include the vendor charges in this total related to the review, but excluding forensics and collection fees. To do this more easily, make sure that the time descriptions that your legal team inputs are clear on what fees are for review. Always remember that you may be required to provide an affidavit or testimony someday to support this cost estimate in a motion for protective order. For our example assume that a total in $1,500 in costs and fees have already been incurred for document search and review work only. F=$1,500. The F divided by R creates the cost per file. Here it is $1.50 per file (F/R).

Finally, multiply the cost per file (F/R) by the number of documents still remaining to be reviewed, T. In other words T * (F/R). Here that is 5,000 (T) times the $1.50 cost per file (F/R), which equals $7,500. You can then disclose this calculation to opposing counsel to help establish the reasonableness (proportionality) of your plan. Step Four – Work Product Disclosure. Note you are estimating a total spend here for this review project of $9,000; $1,500 already spent, plus an estimated additional $7,500 to complete the project.

There are many ways to calculate probable fees to complete document review project. This simple formula method has the advantage of being based on actual experience and costs incurred. It is also simple and easy to understand compared to most other methods. The method could be criticized for inflating expected costs by observing that the work initially performed to find relevant documents is usually slower and more expensive than concluding work to review the tested search folders. This is generally true, but is countered by the fact that: (1) many of the initial relevant documents found in ECA (Step-One) were “low hanging fruit” and easier to locate than what remains; (2) the precision rate of the documents remaining to be reviewed after culling – T – will be much higher than the document folders previously reviewed (the higher the precision rate, the slower the rate of review, because it takes longer to code a relevant document than an irrelevant document); and, (3) additional time is necessarily incurred in the remaining review for redaction, privilege analysis, and quality control efforts not performed in the review to date.

To be concluded … In the conclusion of this article I will review the Six Steps and complete discussion of the related concepts.