This is a guest blog by a friend and colleague, Philip Favro. Phil is a consultant for Driven, Inc. where he serves as a trusted advisor to organizations and their counsel on issues relating to the discovery process and information governance. Phil is also currently active in The Sedona Conference. He obtained permission from them to include a description of a recent event they sponsored in Nashville on Proportionality.
“Follow the Money” and Metrics: A User’s Guide to Proportionality in the Age of e-Discovery
Moviegoers and political junkies have flocked to theaters over the past few months to watch period-piece epics including Darkest Hour and The Post. While there is undoubted attraction (especially in today’s political climate) in watching the reenactment of genuine leadership and courageous deeds these movies portray, The Post should have particular interest for counsel and others involved in electronic discovery.
With its emphasis on investigation and fact-gathering; culling relevant information from marginally useful materials; and decision-making on how such information should be disclosed and presented to adversaries, The Post features key traits associated with sophisticated discovery counsel.
Not coincidentally, those same attributes were on display in another drama from the 1970s of which The Post viewers were reminded: All The President’s Men. That investigative journalism classic depicts Washington Post correspondents Bob Woodward and Carl Bernstein as dogged reporters determined to identify the culprits responsible for the Watergate Hotel break-in in June 1972.
A critical aspect of their work involved Woodward’s furtive meetings with Mark Felt, who served at that time as the deputy director of the FBI. Known only as “Deep Throat” (until Felt revealed himself in 2005), Felt provided cryptic yet key direction that aided the reporters’ investigation. One of Felt’s most significant tips (as portrayed in the movie) was his suggestion that Woodward investigate the cash contributions made to help reelect then President Richard Nixon in 1972. Played by iconic actor Hal Holbrook in All The President’s Men, Felt’s soft-spoken but serious demeanor underscored the importance of his repeated direction to Woodward to “just follow the money.” By following the money, the Washington Post reporters helped discover many of the nefarious tactics that eventually brought down the Nixon presidency.
Proportionality
The directive to “follow the money” applies with equal force to counsel responsible for handling discovery. This is particularly the case in 2018 since courts now expect counsel to address discovery consistent with proportionality standards. Those standards – as codified in Federal Rule of Civil Procedure 26(b)(1) – require counsel, clients, and the courts to consider various factors bearing on the discovery analysis. They include:
(1) the importance of the issues at stake in this action; (2) the amount in controversy; (3) the parties’ relative access to relevant information; (4) the parties’ resources; (5) the importance of the discovery in resolving the issues; and (6) whether the burden or expense of the proposed discovery outweighs its likely benefit.
While all of the factors may be significant, monetary considerations – elements of which are found in both the “amount in controversy” and “burden or expense” factors – frequently predominate a proportionality analysis. As Ralph Losey (the owner, host, and principal author of this blog) has emphasized many times, “[t]he bottom line in e-discovery production is what it costs.” By following the money or, perhaps more appropriate for discovery, focusing on the money, counsel can drive an effective discovery process and obtain better results for the client.
As lawyers do so, they will find an increasingly sophisticated judiciary who expect counsel to approach discovery through the lens of proportionality. This certainly was the case in Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, which has been prominently spotlighted in this blog. In Oxbow, the court applied the Rule 26(b)(1) proportionality factors to a disputed document request, holding that it was not unduly burdensome and that it properly targeted relevant information. While the court examined all of the Rule 26(b)(1) proportionality standards, money was the clearly determinative factor. The amount in controversy, coupled with the comparative costs of discovery – discovery completed and still to be undertaken, tipped the scales in favor of ordering plaintiffs to respond defendants’ document requests.
The Critical Role of Metrics
Essential to Oxbow’s holding were the metrics the parties shared with the court. Metrics – typically defined as a standard of measurement or (as used in business world) a method for evaluating performance – offer counsel ways to assess the “performance” of a particular document production. Metrics can measure the extent to which a production contains relevant materials, undisclosed privileged information, and even nonresponsive documents. Metrics can also estimate – as was the case in Oxbow – the resources (including time, manpower, and costs) a party may be forced to incur to comply with a discovery request.
Metrics enable a court to follow the money and properly balance the burdens of discovery against its benefits. Without metrics, a responding party could hardly expect to establish that a request is disproportionate and thereby prevail in motion practice. As Ralph observed in his post entitled Judge Facciola’s Successor, Judge Michael Harvey, Provides Excellent Proportionality Analysis in an Order to Compel:
Successful arguments on motions to compel require hard evidence. To meet your burden of proof you must present credible estimates of the costs of document review. This requires . . . reliable metrics and statistics concerning the ESI that the requesting party wants the responding party to review.
As discussed later on, other courts have also emphasized the critical role of metrics in evaluating the proportionality of a particular discovery request.
The Sedona Conference, Proportionality, and Metrics
For counsel who wish to better understand the role of metrics in discovery, the directive to “follow the money” will bring them to The Sedona Conference (“Sedona”). Sedona is the preeminent legal institution dedicated to advancing thoughtful reforms on important legal issues. While Sedona addresses matters ranging from patent litigation and trade secret misappropriation to data privacy and cross-border data protection, the organization is best known for its work on electronic discovery.
Renowned for its volunteer model and for attracting many of the best minds in the legal industry, Sedona prepares authoritative resources that are regularly relied on by judges, lawyers, and scholars. This is particularly the case with proportionality standards and how they should drive the determination of discovery issues.
Sedona published its first Commentary on Proportionality in Electronic Discovery (“Commentary” or “Proportionality Commentary”) in 2010 and a second version in 2013. Last spring, Sedona released a third iteration of the Commentary. Collaboratively prepared by a group of renowned judges and practitioners, the third version of the Commentary provides common sense direction on how metrics can help achieve proportional results in discovery:
Burden and expense should be supported by hard information and not by unsupported assertions. For example, if a party claims that a search would result in too many documents, the party should run the search and be prepared to provide the opposing party with the number of hits and any other applicable qualitative metrics. If the party claims that the search results in too many irrelevant hits, the party may consider providing a description or examples of irrelevant documents captured by the search.
Quantitative metrics in support of a burden and expense argument may include the projected volume of potentially responsive documents. It may also encompass the costs associated with processing, performing data analytics, and review, taking into consideration the anticipated rate of review and reviewer costs, based upon reasonable fees and expenses.
As the Commentary makes clear, metrics can provide insights regarding the effectiveness of search methodologies or the nature and extent of a party’s burden in responding to a particular discovery request. By sharing these metrics with litigation adversaries, counsel can informally address legitimate discovery questions or crystallize the issues for resolution by a court. Either way represents a more cost effective approach to discovery than the opacity of traditional meet and confers or motion practice.
Framing the Issues through Sedona’s TSCI Event
These issues were on display last month at Sedona’s TSCI conference in Nashville, Tennessee. The TSCI event typically provides attendees with an annual opportunity to stay current on developing trends in e-Discovery. The 2018 TSCI event remained consistent with that objective, spotlighting practice developments for counsel “from ‘eDiscovery 1.0’ to New and Evolving Legal Challenges.” Expertly chaired by Jerone “Jerry” English and Maura Grossman, TSCI featured sessions covering discovery and other issues relating to artificial intelligence (AI), the Internet of Things (IoT), mobile applications, data breaches, cross-border discovery, and the always engaging case law panel and judicial round-table.
One of the more practical sessions focused on the importance of using metrics, analytics, and sampling to achieve proportionality in discovery. Entitled Using Data Analytics and Metrics to Achieve Proportionality, the purpose of this session was to help attendees understand how counsel should present analytics, metrics, and sampled data to a court. The session featured a fantastic line-up of speakers – Gareth Evans, Maura Grossman, U.S. Magistrate Judge Anthony Porcelli, U.S. Magistrate Judge Leda Dunn Wettre – who were well situated to provide views on these topics. Audience members additionally offered insightful comments on the issues.[1]
The most important guidance the speakers and audience emphasized was the need for more complete disclosure of supporting metrics. Unless specific metrics are disclosed, neither adversaries nor the court can address issues as varied as the performance of particular search terms, the reasonableness of a production made using TAR or other search methodologies, or the burdens of a particular discovery request.
On the latter issue of substantiating arguments of undue burden, one particularly insightful comment offered during the session concisely summarized the interplay between metrics, proportionality, and cost: “Follow the money.” This admonition dovetailed nicely with the discussion of two recent cases during that session – Duffy v. Lawrence Memorial Hospital, No. 2:14-cv-2256-SAC-TJJ, 2017 WL 1277808 (D. Kan. Mar. 31, 2017) and Solo v. United Parcel Service Co., No. 14-cv-12719, 2017 WL 85832 (E.D. Mich. Jan. 10, 2017). Both of these cases spotlight how reliable metrics enable a court to follow the money and resolve discovery disputes consistent with proportionality standards.
Duffy v. Lawrence Memorial Hospital
In Duffy, the court modified a discovery order issued less than two months beforehand that granted plaintiff’s requests for various categories of emergency room patient records. In that first round of motion practice, defendant had argued that plaintiff’s requests were disproportionate and unduly burdensome. The court overruled those objections, explaining that defendant failed to provide any substantive metrics to support those objections:
Defendant objects to every document request as being unduly burdensome, but provides no facts to support the objection. Neither does Defendant provide evidence of the costs it would incur in responding to the requests.
In summary, defendant’s failure to share any meaningful metrics regarding the time, manpower, or costs it would incur to comply with plaintiff’s requests ultimately left its arguments bereft of any evidentiary support.
In the second round of motion practice, defendant adopted a different approach that yielded a more proportional result. Confronted by the staggering reality of the court’s production order and having learned how to properly use supporting metrics in motion practice, defendant moved for a protective order.
In contrast to its prior briefing, defendant shared specific metrics associated with the burdens of production. Those burdens involved the deployment of staff to individually review 15,574 electronic patient files so as to identify particular patient visit information. Such a process would be labor intensive and cost well over $230,000:
Defendant estimates it would take 7,787 worker hours to locate and produce responsive information for 15,574 patient records. If Defendant had ten employees working on the task, they would spend more than ninety-seven days working eight hours a day, at an estimated cost to Defendant of $196,933.23.
After aggregating the information, Defendant asserts it would need to redact patients’ personal confidential information . . . redaction would take ten reviewers fourteen days at a cost of $37,259.50. The process would include a quality control attorney reviewer who would spend two hours a day, and reviewers who would review 15 documents per hour for eight hours a day.
In sum . . . producing the information relevant to RFP Nos. 40, 41, 43, and 58 would take 8,982 hours of work and cost in excess of $230,000 if done by contract staff.
Simply put, defendant urged the court to follow the money. By substantiating its proportionality arguments with appropriate metrics, the court recognized its initial production order placed an undue burden on defendant.
As a result, the court adopted a modified order that instead allowed defendant to produce a random sample of 257 patient records. While advancing a number of justifications for its modified order, the court ultimately relied on the tripartite mandate from Federal Rule of Civil Procedure 1. The order would provide the parties to the litigation with a substantively better, more efficient, and less expensive method for producing relevant information.
Solo v. United Parcel Service
Solo v. United Parcel Service reached a result analogous to the Duffy holding, ordering that defendant produce only a sample of the information sought by plaintiff. In Solo, plaintiffs served an interrogatory that sought identification of shipment information relating to its putative class action claims (plaintiffs claimed that defendant overcharged certain customers for “shipments that had a declared value of over $300”). The interrogatory sought shipping record information that spanned a period of six years.
Defendant argued in response that the interrogatory was unduly burdensome and would impose a disproportionate production obligation on the company. Because most of the requested information was archived on backup tapes, defendant shared specific metrics regarding the “overwhelming” burdens associated with responding to the interrogatory:
UPS estimates that it would take at least six months just to restore the archived tapes as described above, at a cost of $120,000 in labor . . . that estimate does not include the time and expense of analyzing the data once extracted in order to answer Interrogatory No. 1, which would require extensive additional analysis of each account number and the manual review of contract language for individual shipper. Such a process would also require a substantial amount of time and resources on the part of UPS.
Based on the metrics defendant disclosed and given that plaintiffs’ claims had yet to be certified as a class action, the court found the interrogatory to be disproportionate. Following the money and drawing on the linked concepts of cooperation and proportionality from Rule 1, the court instead ordered that defendant produce a sample of the requested information from a six-month period. The court also directed the parties to meet and confer on developing an agreeable sampling methodology.
Conclusion
Duffy and Solo reinforce the critical interplay between metrics, proportionality, and money. Just like Oxbow, the responding parties from Duffy and Solo could hardly expect to substantiate arguments regarding undue burden and disproportionality without metrics. Indeed, the court in Duffy initially rejected such arguments when defendant failed to support them with actual information. However, by disclosing metrics with reasonable estimates of time, manpower, and costs, Duffy and Solo resulted in production orders more consistent with proportionality limitations.
All of which translated into substantial cost savings for the responding parties. Defendant in Duffy was facing a discovery bill of over $230,000 to review 15,574 patient files. Dividing the projected cost of the entire review process into the number of patient records – $230,000 ÷ 15,574 – reveals that defendant would pay approximately $15 to review an individual patient record. Under the modified production order, the new projected cost – $15 multiplied by 257 patient records – equals $3,855. Follow the money: the tactical use of metrics apparently saved the client over $225,000!
Duffy and Solo are consistent with and confirmed by Oxbow, the Proportionality Commentary, and the Federal Rules of Civil Procedure. These authoritative resources collectively teach that counsel who use metrics and focus on cost can drive an effective discovery process. Lawyers that do so will ultimately obtain better results for the client in discovery.
_______
[1] To encourage candid and robust debate during its events, Sedona has promulgated a nondisclosure rule. Known as “The Sedona Rule,” it proscribes attendees from identifying the speakers or audience members by name who share particular insights. It also forbids divulging the contents of particular brainstorming or drafting projects that have yet to be released for publication. The Sedona Rule otherwise allows for the anonymous disclosure of session content from its events.
Reblogged this on Legal Tech Talent Network.