Document Review and Proportionality – Part One

March 18, 2018

In 2013 I wrote a law review article on how the costs of document review could be controlled using predictive coding and cost estimation. Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data, 26 Regent U. Law Review 1 (2013-2014). Today I write on how it can be controlled in document review, even without predictive coding. Here is the opening paragraph of my earlier article:

The search of electronic data to try to find evidence for use at trial has always been difficult and expensive. Over the past few years, the advent of Big Data, where both individuals and organizations retain vast amounts of complex electronic information, has significantly compounded these problems. The legal doctrine of proportionality responds to these problems by attempting to constrain the costs and burdens of discovery to what are reasonable. A balance is sought between the projected burdens and likely benefits of proposed discovery, considering the issues and value of the case. Several software programs on the market today have responded to the challenges of Big Data by implementing a form of artificial intelligence (“AI”) known as active machine learning to help lawyers review electronic documents. This Article discusses these issues and shows that AI-enhanced document review directly supports the doctrine of proportionality. When used together, proportionality and predictive coding provide a viable, long-term solution to the problems and opportunities of the legal search of Big Data.

The 2013 article was based on version 1.0 Predictive Coding. Under this first method you train and rank documents and then review only the higher ranking documents. Here is a more detailed description from pages 23, 24 of the article:

This kind of AI-enhanced legal review is typically described today in legal literature by the term
predictive coding. This is because the computer predicts how an entire body of documents should be coded (classified) based on how the lawyer has coded the smaller training sets. The prediction places a probability ranking on each document, typically ranging from 0% to 100% probability. Thus, in a
relevancy classification, each and every document in the entire dataset (the corpus) is ranked with a percentage of likely relevance and irrelevance. …
As will be shown, this ranking feature is key to the use of the legal doctrine of proportionality. The ability to rank all documents in a corpus on probable relevance is a new feature that no other legal search software has previously provided.71

It was a two phase procedure: train then review. Yes, some review would take place in the first training phase, but this would be a relatively small number, say 10-20% of the total documents reviewed. Most of the human review of documents would take place in phase two. The workflow of version 1.0 is shown in the diagram below and is described in detail in the article, starting at page 31.

Predictive Coding and the Proportionality Doctrine argued that attorneys should scale the number of documents for the second phase of document review based on estimated costs constrained to a proportional amount. No more spending $100,000 for document review in a $200,000 case. The number of documents you selected for review would be limited to proportional costs. Predictive coding and its ranking features allowed you to select the documents for review that were most likely to be relevant. If you could only afford to spend $20,000 on a document review project, then you would limit the number of documents reviewed to those within that scope that were the highest ranked as probable relevant. Here is the article’s description at pages 54-55 of the process and link between the doctrine of proportionality and predictive coding.

What makes this a marriage truly made in heaven is the document-ranking capabilities of predictive coding. This allows parties to limit the documents considered for final production to those that the computer determines have the highest probative value. This key ranking feature of AI-enhanced document review allows the producing party to provide the requesting party with the most bang for the buck. This not only saves the producing party money, and thus keeps its costs proportional, but it saves time and expenses for the requesting party. It makes the production much more precise, and thus faster and easier to review. It avoids what can be a costly exercise to a requesting party to wade through a document dump 192, a production that contains a high number of irrelevant or marginally relevant documents. Most importantly, it gives the requesting party what it really wants—the documents that are the most important to the case.

In the article, pages 58-60, I called this method Bottom-Line-Driven Proportional Review and describe the process in greater detail.

The bottom line in e-discovery production is what it costs. Despite what some lawyers and vendors may say, total cost is not an impossible question to answer. It takes an experienced lawyer’s skill to answer, but,
after a while, you can get quite good at such estimation. It is basically a matter of estimating attorney billable hours plus vendor costs. With practice, cost estimation can become a reliable art, a projection that you can count on for budgeting purposes, and, as we will see, for proportionality arguments.  …
The new strategy and methodology is based on a bottom line approach where you estimate what review costs will be, make a proportionality analysis as to what should be spent, and then engage in defensible culling to bring the review costs within the proportional budget. The producing party determines the number of documents to be subjected to final review by calculating backwards from the bottom line of what they are willing, or required, to pay for the production.  …
Under the Bottom-Line-Driven Proportional approach, after analyzing the case merits and determining the maximum proportional expense, the responding party makes a good faith estimate of the likely
maximum number of documents that can be reviewed within that budget. The document count represents the number of documents that you estimate can be reviewed for final decisions of relevance, confidentiality, privilege, and other issues and still remain within budget. The review costs you estimate must be based on best practices, which in all large review projects today means predictive coding, and the estimates must be accurate (i.e., no puffing or mere guesswork).
Note this last quote (emphasis added) from Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data shows an important limitation to the article’s budgetary proposal, it was limited to large review projects where predictive coding was used. Without this marriage to predictive coding, the promise of proportionality by cost estimations was lost. My article today fills this gap.
Here I will explain how document review can be structured to provide estimates and review constraints, even when predictive coding and its ranking are not used. This is, in effect, the single lawyers guide, one where there has not been a marriage with predictive coding. It is a guide for small and medium sized document review projects, which are, after all, the vast majority of projects faced by the legal profession.

To be honest, back when I first wrote the law review article I did not think it would be necessary to develop such a proportionality method, one that does not use AI document ranking. I assumed predictive coding would take off and by now would be used in almost all projects, no matter what the size. I assumed that since active machine learning and document ranking was such good new technology, that even our conservative profession would embrace it within the next few years. Boy was I wrong about that. The closing lines of Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data have been proven naive.

The key facts needed to try a case and to do justice can be found in any size case, big and small, at an affordable price, but you have to embrace change and adopt new legal and technical methodologies. The Bottom-Line-Driven Proportional Review method is part of that answer, and so too is advanced-review software at affordable prices. When the two are used together, it is a marriage made in heaven.

I blame both lawyers and e-discovery vendors for this failure, as well as myself for misjudging my peers. Law schools and corporate counsel have not helped much either. Only the judiciary seems to have caught on and kept up.

Proportionality as a legal doctrine took off as expected after 2013, but not the marriage with predictive coding. Lawyers have proven to be much more stubborn than anticipated. They will barely even go out with predictive coding, no matter how attractive she is, much less marry her. The profession as a whole remains remarkably slow to adopt new technology. The judges are tempted to use their shotgun to force a wedding, but so far have refrained from ordering a party to use predictive coding. Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP), 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016) (J. Peck: “There may come a time when TAR is so widely used that it might be unreasonable for a party to decline to use TAR. We are not there yet.”)

Changes Since 2013

A lot has happened since 2013 when Predictive Coding and the Proportionality Doctrine was written. In December 2015 Rule 26(b) on relevance was revised to strengthen proportionality and we have made substantial improvements to Predictive Coding methods. In the ensuing years most experts have abandoned this early two-step method of train then review in favor of a method where training continues throughout the review process. In other words, today we keep training until the end. See Eg. the e-Discovery Team’s Predictive Coding, version 4.0, with its Intelligently Spaced Training. (This is similar to a method popularized by Maura Grossman and Gordon Cormack, which they called Continuous Active Learning or CAL for short, a term they later trademarked.)

The 2015 revision to Rule 26(b) on relevance has spurred case law and clarified that undue burden, the sixth factor of proportionality under Rule 26(b)(1), must be argued in detail with facts proven.

  1. the importance of the issues at stake in this action;
  2. the amount in controversy;
  3. the parties’ relative access to relevant information;
  4. the parties’ resources;
  5. the importance of the discovery in resolving the issues; and
  6. whether the burden or expense of the proposed discovery outweighs its likely benefit.”

Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, No. 11-cv-1049 (PLF/GMH), 2017 WL 4011136, (D.D.C. Sept. 11, 2017). Although all factors are important and should be addressed, the last factor is usually the most important one in a discovery dispute. It is also the factor that can be addressed generally for all cases and is the core of proportionality.

Proportional “Bottom Line Driven” Method of Document Review that Does Not Require Use of Predictive Coding

I have shared how I use predictive coding with continuous training in my online instruction program. The eight-step workflow is shown below.

I have not previously shared any information on the document review workflow that I follow in small and medium seized cases where predictive coding is not used. The rest of this article will do so now.

Please note that I have a well-developed and articulated list of steps and procedures for attorneys in my law firm to follow in such small cases. I keep this as a trade-secret and will not reveal them here. Although they are widely known in my firm, and slightly revised and updated each year, they are not public. Still, any expert in document review should be able to create their own particular rules and implementation methods. Warning, if you are not such an expert, be careful in relying on these high-level explanations alone. The devil is in the details and you should retain an expert to assist.

Here is a chart summarizing the SIX-Step Workflow and six basic concepts that must be understood for the process to work at maximum efficiency.

The first three steps iterate with searches to cull out the irrelevant documents, and then culminate with Disclosures of the plan developed for Steps Five and Six, Final Review and Production.  The sixth production step is always in phases according to proportional planning.

A key skill that must be learned is project cost estimation, including fees and expenses. The attorneys involved must also learn how to communicate with themselves, the vendors, opposing counsel and the court. Rigid enforcement of work-product confidentiality is counter-productive to the goal of cost efficient projects. Agree on the small stuff and save your arguments for the cost-saving questions that are worth the effort.


The Proportionality Doctrine

The doctrine of proportionality as a legal initiative was launched by The Sedona Conference in 2010 as a reaction to the exploding costs of e-discovery. The Sedona Conference, The Sedona Conference Commentary on Proportionality in Electronic Discovery, 11 SEDONA CONF. J. 289, 292–94 (2010). See also John L. Carroll, Proportionality in Discovery: A Cautionary Tale, 32 CAMPBELL L. REV. 455, 460 (2010) (“If courts and litigants approach discovery with the mindset of proportionality, there is the potential for real savings in both dollars and time to resolution.”); Maura Grossman & Gordon Cormack, Some Thoughts on Incentives, Rules, and Ethics Concerning the Use of Search Technology in E-Discovery, 12 SEDONA CONF. J. 89, 94–95, 101–02 (2011).

The doctrine received a big boost with the adoption of the 2015 Amendment to Rule 26. The rule was changed to provide discovery must be both relevant and “proportional to the needs of the case.” Fed. R. Civ. P. 26(b)(1). To determine whether a discovery request is proportional, you are required weigh the following six factors: “(1) the importance of the issues at stake in this action; (2) the amount in controversy; (3) the parties’ relative access to relevant information; (4) the parties’ resources; (5) the importance of the discovery in resolving the issues; and (6) whether the burden or expense of the proposed discovery outweighs its likely benefit.” Williams v. BASF Catalysts, LLC, Civ. Action No. 11-1754, 2017 WL 3317295, at *4 (D.N.J. Aug. 3, 2017) (citing Fed. R. Civ. P. 26(b)(1)); Arrow Enter. Computing Solutions, Inc. v. BlueAlly, LLC, No. 5:15-CV-37-FL, 2017 WL 876266, at *4 (E.D.N.C. Mar. 3, 2017); FTC v. Staples, Inc., Civ. Action No. 15-2115 (EGS), 2016 WL 4194045, at *2 (D.D.C. Feb. 26, 2016).

“[N]o single factor is designed to outweigh the other factors in determining whether the discovery sought is proportional,” and all proportionality determinations must be made on a case-by-case basis. Williams, 2017 WL 3317295, at *4 (internal citations omitted); see also Bell v. Reading Hosp., Civ. Action No. 13-5927, 2016 WL 162991, at *2 (E.D. Pa. Jan. 14, 2016). To be sure, however, “the amendments to Rule 26(b) do not alter the basic allocation of the burden on the party resisting discovery to—in order to successfully resist a motion to compel—specifically object and show that . . . a discovery request would impose an undue burden or expense or is otherwise objectionable.” Mir v. L-3 Commc’ns Integrated Sys., L.P., 319 F.R.D. 220, 226 (N.D. Tex. 2016), as quoted by Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, No. 11-cv-1049 (PLF/GMH), 2017 WL 4011136, (D.D.C. Sept. 11, 2017).

The Oxbow case is discussed at length in my recent blog Judge Facciola’s Successor, Judge Michael Harvey, Provides Excellent Proportionality Analysis in an Order to Compel (e-Discovery Team,3/1/18). Judge Harvey carefully examined the costs and burdens claimed by plaintiffs and rejected the overly burdensome argument.

Plaintiffs’ counsel explained at the second hearing in this matter that Oxbow has spent $1.391 million to date on reviewing and producing approximately 584,000 documents from its nineteen other custodians and Oxbow’s email archive. See 8/24/17 TR. at 44:22-45:10. And again, Oxbow seeks tens of millions of dollars from Defendants. Through that lens, the estimated cost of reviewing and producing Koch’s responsive documents—even considering the total approximate cost of $142,000 for that effort, which includes the expense of the sampling effort—while certainly high, is not so unreasonably high as to warrant rejecting Defendants’ request out of hand. See Zubulake v. UBS Warburg, LLC, 217 F.R.D. 309, 321 (S.D.N.Y. 2003) (explaining, in the context of a cost-shifting request, that “[a] response to a discovery request costing $100,000 sounds (and is) costly, but in a case potentially worth millions of dollars, the cost of responding may not be unduly burdensome”); Xpedior Creditor Trust v. Credit Suisse First Boston (USA), Inc., 309 F. Supp. 2d 459, 466 (S.D.N.Y. 2003) (finding no “undue burden or expense” to justify cost-shifting where the requested discovery cost approximately $400,000 but the litigation involved at least $68.7 million in damages). …

In light of the above analysis—including the undersigned’s assessment of each of the Rule 26 proportionality factors, all of which weigh in favor of granting Defendants’ motion—the Court is unwilling to find that the burden of reviewing the remaining 65,000 responsive documents for a fraction of the cost of discovery to date should preclude Defendants’ proposed request. See BlueAlly, 2017 WL 876266, at *5 (“This [last Rule 26] factor may combine all the previous factors into a final analysis of burdens versus benefits.” (citing Fed. R. Civ. P. 26 advisory committee’s notes)).

For more analysis and case law on proportionality see Proportionality Φ and Making It Easy To Play “e-Discovery: Small, Medium or Large?” in Your Own Group or Class, (e-Discovery Team, 11/26/17). Also see The Sedona Conference Commentary on Proportionality, May 2017.

Learning How to Estimate Document Review Costs

The best way to determine a total cost of a project is by projection from experience and analysis on a cost per file basis. General experience of review costs can be very helpful, but the gold standard comes from measurement of costs actually incurred in the same project, usually after several hours of work, or days, depending on the size of the project. You calculate costs incurred to date and then project forward on a cost per file basis.  The is the core idea of the Six Step document review protocol that this article begins to explain.

The actual project costs are the best possible metrics for estimation. Apparently that was never done in Oxbow because the plaintiffs counsel’s projected document review cost estimates varied so much. A per file cost analysis of the information in the Oxbow opinion shows that the parties missed a key metric. The costs projected ranged from an actual cost of $2.38 per file for the first 584,000 files, to an 1.17 per file estimate to review 214,000 additional files, to an estimate of $1.73 per file to review 82,000 more files, to an actual cost of $4.74 per file to review 12,074 files, to final estimate of $1.22 per file to review the remaining 69,926 files. The actual costs are way higher than the estimated costs meaning the moving party cheated themselves by failing to do the math.

Here is how I explained the estimation process in Predictive Coding and the Proportionality Doctrine at pages 60-61:

Under the Bottom-Line-Driven Proportional approach, after analyzing the case merits and determining the maximum proportional expense, the responding party makes a good faith estimate of the likely maximum number of documents that can be reviewed within that budget. The document count represents the number of documents that you estimate can be reviewed for final decisions of relevance, confidentiality, privilege, and other issues and still remain within budget.
A few examples may help clarify how this method works. Assume a case where you determine a proportional cost of production to be $50,000, and estimate, based on sampling and other hard facts, that it will cost you $1.25 per file for both the automated and manual review before production of the ESI at issue … Then you can review no more than 40,000 documents and stay within budget. It is that simple. No higher math is required.

Estimation for bottom-line-driven review is essentially a method for marshaling evidence to support an undue burden argument under Rule 26(b)(1). Let’s run through it again with greater detail and make a simple formula to illustrate the process.

First, estimate the total number of documents remaining to be reviewed after culling by your tested keywords and other searches (hereinafter “T”). This is the most difficult step but is something most attorney experts and vendors are well qualified to help you with. Essentially “T” represents is the number of documents left unreviewed for Step Five, Final Review.  These are the documents found in Steps One and Two, ECA Multimodal Search and Testing. These steps, along with the estimate calculation, usually repeat several times to cull-in the documents that are most likely relevant to the claims and defenses. The T – Total Documents Left for Review – are the documents in the final revised keyword search folders and concept, similarity search folders. The goal is to obtain a relevance precision in these folders greater than 50%, preferably at least 75%.

To begin an example hypothetical, assume that the total document count in the folders set-up for final review is 5,000 documents. T=5,000. Next count how many relevant and highly relevant files have already been located (hereinafter “R”).  Assume for our example that 1,000 relevant and highly relevant documents have been found. R=1,000.

Next, look up the total attorney fees already incurred in the matter to date for the actual document search and review work by attorneys and paralegals (hereinafter collectively “F”). Include the vendor charges in this total related to the review, but excluding forensics and collection fees. To do this more easily, make sure that the time descriptions that your legal team inputs are clear on what fees are for review. Always remember that you may be required to provide an affidavit or testimony someday to support this cost estimate in a motion for protective order. For our example assume that a total in $1,500 in costs and fees have already been incurred for document search and review work only. F=$1,500. The F divided by R creates the cost per file. Here it is $1.50 per file (F/R).

Finally, multiply the cost per file (F/R) by the number of documents still remaining to be reviewed, T. In other words T * (F/R).  Here that is 5,000 (T) times the $1.50 cost per file (F/R), which equals $7,500. You can then disclose this calculation to opposing counsel to help establish the reasonableness (proportionality) of your plan. Step Four – Work Product Disclosure. Note you are estimating a total spend here for this review project of $9,000; $1,500 already spent, plus an estimated additional $7,500 to complete the project.

There are many ways to calculate probable fees to complete document review project. This simple formula method has the advantage of being based on actual experience and costs incurred. It is also simple and easy to understand compared to most other methods. The method could be criticized for inflating expected costs by observing that the work initially performed to find relevant documents is usually slower and more expensive than concluding work to review the tested search folders. This is generally true, but is countered by the fact that: (1) many of the initial relevant documents found in ECA (Step-One) were “low hanging fruit” and easier to locate than what remains; (2) the precision rate of the documents remaining to be reviewed after culling – T – will be much higher than the document folders previously reviewed (the higher the precision rate, the slower the rate of review, because it takes longer to code a relevant document than an irrelevant document); and, (3) additional time is necessarily incurred in the remaining review for redaction, privilege analysis, and quality control efforts not performed in the review to date.

To be concluded …  In the conclusion of this article I will review the Six Steps and complete discussion of the related concepts.

TAR Course Expands Again: Standardized Best Practice for Technology Assisted Review

February 11, 2018

The TAR Course has a new class, the Seventeenth Class: Another “Player’s View” of the Workflow. Several other parts of the Course have been updated and edited. It now has Eighteen Classes (listed at end). The TAR Course is free and follows the Open Source tradition. We freely disclose the method for electronic document review that uses the latest technology tools for search and quality controls. These technologies and methods empower attorneys to find the evidence needed for all text-based investigations. The TAR Course shares the state of the art for using AI to enhance electronic document review.

The key is to know how to use the document review search tools that are now available to find the targeted information. We have been working on various methods of use since our case before Judge Andrew Peck in Da Silva Moore in 2012. After we helped get the first judicial approval of predictive coding in Da Silva, we began a series of several hundred document reviews, both in legal practice and scientific experiments. We have now refined our method many times to attain optimal efficiency and effectiveness. We call our latest method Hybrid Multimodal IST Predictive Coding 4.0.

The Hybrid Multimodal method taught by the combines law and technology. Successful completion of the TAR course requires knowledge of both fields. In the technology field active machine learning is the most important technology to understand, especially the intricacies of training selection, such as Intelligently Spaced Training (“IST”). In the legal field the proportionality doctrine is key to the  pragmatic application of the method taught at TAR Course. We give-away the information on the methods, we open-source it through this publication.

All we can transmit by online teaching is information, and a small bit of knowledge. Knowing the Information in the TAR Course is a necessary prerequisite for real knowledge of Hybrid Multimodal IST Predictive Coding 4.0. Knowledge, as opposed to Information, is taught the same way as advanced trial practice, by second chairing a number of trials. This kind of instruction is the one with real value, the one that completes a doc review project at the same time it completes training. We charge for document review and throw in the training. Information on the latest methods of document review is inherently free, but Knowledge of how to use these methods is a pay to learn process.

The Open Sourced Predictive Coding 4.0 method is applied for particular applications and search projects. There are always some customization and modifications to the default standards to meet the project requirements. All variations are documented and can be fully explained and justified. This is a process where the clients learn by doing and following along with Losey’s work.

What he has learned through a lifetime of teaching and studying Law and Technology is that real Knowledge can never be gained by reading or listening to presentations. Knowledge can only be gained by working with other people in real-time (or near-time), in this case, to carry out multiple electronic document reviews. The transmission of knowledge comes from the Q&A ESI Communications process. It comes from doing. When we lead a project, we help students to go from mere Information about the methods to real Knowledge of how it works. For instance, we do not just make the Stop decision, we also explain the decision. We share our work-product.

Knowledge comes from observing the application of the legal search methods in a variety of different review projects. Eventually some Wisdom may arise, especially as you recover from errors. For background on this triad, see Examining the 12 Predictions Made in 2015 in “Information → Knowledge → Wisdom” (2017). Once Wisdom arises some of the sayings in the TAR Course may start to make sense, such as our favorite “Relevant Is Irrelevant.” Until this koan is understood, the legal doctrine of Proportionality can be an overly complex weave.

The TAR Course is now composed of eighteen classes:

  1. First Class: Background and History of Predictive Coding
  2. Second Class: Introduction to the Course
  3. Third Class:  TREC Total Recall Track, 2015 and 2016
  4. Fourth Class: Introduction to the Nine Insights from TREC Research Concerning the Use of Predictive Coding in Legal Document Review
  5. Fifth Class: 1st of the Nine Insights – Active Machine Learning
  6. Sixth Class: 2nd Insight – Balanced Hybrid and Intelligently Spaced Training (IST)
  7. Seventh Class: 3rd and 4th Insights – Concept and Similarity Searches
  8. Eighth Class: 5th and 6th Insights – Keyword and Linear Review
  9. Ninth Class: 7th, 8th and 9th Insights – SME, Method, Software; the Three Pillars of Quality Control
  10. Tenth Class: Introduction to the Eight-Step Work Flow
  11. Eleventh Class: Step One – ESI Communications
  12. Twelfth Class: Step Two – Multimodal ECA
  13. Thirteenth Class: Step Three – Random Prevalence
  14. Fourteenth Class: Steps Four, Five and Six – Iterative Machine Training
  15. Fifteenth Class: Step Seven – ZEN Quality Assurance Tests (Zero Error Numerics)
  16. Sixteenth Class: Step Eight – Phased Production
  17. Seventeenth Class: Another “Player’s View” of the Workflow (class added 2018)
  18. Eighteenth Class: Conclusion

With a lot of hard work you can complete this online training program in a long weekend, but most people take a few weeks. After that, this course can serve as a solid reference to consult during complex document review projects. It can also serve as a launchpad for real Knowledge and eventually some Wisdom into electronic document review. is designed to provide you with the Information needed to start this path to AI enhanced evidence detection and production.


Proportionality Φ and Making It Easy To Play “e-Discovery: Small, Medium or Large?” in Your Own Group or Class

November 26, 2017

Every judge who has ever struggled with discovery issues wishes that the lawyers involved had a better understanding of proportionality, that they had spent more time really thinking about how it applies to the requisites of their case. So too does every lawyer who, like me, specializes in electronic discovery. As Chief Justice Roberts explained in his 2015 Year-End Report on the Federal Judiciary on the new rules on proportionality:

The amended rule states, as a fundamental principle, that lawyers must size and shape their discovery requests to the requisites of a case. Specifically, the pretrial process must provide parties with efficient access to what is needed to prove a claim or defense, but eliminate unnecessary or wasteful discovery. The key here is careful and realistic assessment of actual need.

Proportionality and reasonableness arise from conscious efforts to realistically assess actual need. What is the right balance in a particular situation? What are the actual benefits and burdens involved? How can you size and shape your discovery requests to the requisites of a case?

There is more to proportionality than knowing the rules and case law, although they are a good place to start. Proportionality is a deep subject and deserves more than black letter law treatment. 2015 e-Discovery Rule Amendments: Dawning of the “Goldilocks Era” (, 11/11/15) (wherein I discuss proportionality, the Golden Ratio or perfect proportionality, aka Φ, which is shown in this graphic and much more, including the spooky “coincidence” at a CLE with Judge Facciola and the audience vote). Also see: Giulio Tononi, Phi Φ, a Voyage from the Brain to the Soul (Pantheon Books, 2012) (book I’m rereading now on consciousness and integrated information theory, another take on Phi Φ).

We want everyone in the field to think about proportionality. To be conscious of it, not just have information about it. What does proportionality really mean? How does it apply to the e-discovery tasks that you carry out every day? How much is enough? Too much? Too burdensome? Too little? Not enough? Why?

What is a reasonable effort? How do you know? Is there perfect proportionality? One that expresses itself in varying ways according to the facts and circumstances? Does Law follow Art? Is Law an Art? Or is it a Science? Is there Beauty in Law? In Reason? There is more to proportionality than meets the eye. Or is there?

Getting people to think about proportionality is one of the reasons I created the Hive Mind game that I announced in my blog last week: “e-Discovery: Small, Medium of Large?”

This week’s blog continues that intention of getting lawyers to think about proportionality and the requisites of their case. It concludes with a word document designed to make it easier to play along with your own group, class or CLE event. What discovery activities required in a Big Case are not necessary in a Small Case, or even a Medium Sized case? That is what requires thought and is the basis of the game.

Rules of Federal Procedure

Proportionality is key to all discovery, to knowing the appropriate size and shape of discovery requests in order to fit the requisites of a case. Reading the rules that embody the doctrine of proportionality is a good start, but just a start.  The primary rule to understand is how proportionality effects the scope of relevance as set forth in Rule 26(b)(1), FRCP:

Parties may obtain discovery regarding any nonprivileged matter that is relevant to any party’s claim or defense and proportional to the needs of the case, considering the importance of the issues at stake in the action, the amount in controversy, the parties’ relative access to relevant information, the parties’ resources, the importance of the discovery in resolving the issues, and whether the burden or expense of the proposed discovery outweighs its likely benefit.

But you also need to understand how it impacts a lawyer’s overall duty to supervise a discovery request and response as set forth in Rule 26(g). See Rule 26(g)(1)(B)(iii), FRCP:

neither unreasonable nor unduly burdensome or expensive, considering the needs of the case, prior discovery in the case, the amount in controversy, and the importance of the issues at stake in the action.

Many other rules have concepts of proportionality either expressly or implicitly built in, including Rule 26(b)(2)(B) (not reasonably accessible); Rule 26(b)(2)(C)(i) (cumulative); Rule 1 (just, speedy and inexpensive), Rule 34, Rule 37(e), Rule 45.

Case Law

Reading the key cases is also a help, indispensable really, but reading what the judges say is not enough either. Still you need to keep up with the fast growing case law on proportionality. See for instance the fine collection by K&L Gates at: and the must-read, The Sedona Conference Commentary on Proportionality_May 2017. Here a few of my favorites cases:

  • In re Bard IVC Filters Prods. Liab. Litig., D. Ariz., No. MDL 15-02641-PHX DGC, 2016 U.S. Dist. LEXIS 126448 (D. Ariz. Sept. 16, 2016). In this must-read opinion District Judge David G. Campbell, who was the chair of the Rules Committee when the 2015 amendments were passed, takes both lawyers and judges to task for not following the new rules on proportionality. He then lays it all out in a definitive manner.
  • In re Takata Airbag Prods. Liab. Litig., No. 15-02599-CIV-Moreno, MDL No. 5-2599 (S.D. Fla. Mar. 1, 2016). Judge Moreno quotes Chief Justice Roberts’ comments in the 2015 Year-End Report that the newly amended Fed.R.Civ.Pro. 26 “crystalizes the concept of reasonable limits in discovery through increased reliance on the common-sense concept of proportionality.” 2015 Year-End Report on the Federal Judiciary.
  • Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP), 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016) (Judge Peck: “While Hyles may well be correct that production using keywords may not be as complete as it would be if TAR were used, the standard is not perfection, or using the “best” tool, but whether the search results are reasonable and proportional. Cf. Fed. R. Civ. P. 26(g)(1)(B)”)
  • Johnson v Serenity TransportationCase No. 15-cv-02004-JSC (N.D. Cal. October 28, 2016) (“… a defendant does not have discretion to decide to withhold relevant, responsive documents absent some showing that producing the document is not proportional to the needs of the case.”)
  • Apple Inc. v. Samsung Elecs. Co., No. 12-CV-0630-LHK (PSG), 2013 WL 4426512, 2013 U.S. Dist. LEXIS 116493 (N.D. Cal. Aug. 14, 2013) (“But there is an additional, more persuasive reason to limit Apple’s production — the court is required to limit discovery if “the burden or expense of the proposed discovery outweighs its likely benefit.” This is the essence of proportionality — an all-to-often ignored discovery principle. Because the parties have already submitted their expert damages reports, the financial documents would be of limited value to Samsung at this point. Although counsel was not able to shed light on exactly what was done, Samsung’s experts were clearly somehow able to apportion the worldwide, product line inclusive data to estimate U.S. and product-specific damages. It seems, well, senseless to require Apple to go to great lengths to produce data that Samsung is able to do without. This the court will not do.)
  • PTSI, Inc. v. Haley, 2013 WL 2285109 (Pa. Super. Ct. May 24, 2013) (“… it is unreasonable to expect parties to take every conceivable step to preserve all potentially relevant data.”)
  •  Kleen Products, LLC, et al. v. Packaging Corp. of Amer., et al.Case: 1:10-cv-05711, Document #412 (ND, Ill., Sept. 28, 2012).

Also see: The Top Twenty-Two Most Interesting e-Discovery Opinions of 2016 (, 1/2/17) (the following top ranked cases concerned proportionality: 20, 18, 17, 15, 14, 11, 6, 4, 3, 2, 1); and, Good, Better, Best: a Tale of Three Proportionality Cases – Part Two ( 4/8/12) (includes collection of earlier case law).

Sedona Commentary

The Sedona Conference Commentary on Proportionality_May 2017 is more than a collection of case law. It includes commentary hashed out between competing camps over many years. The latest 2017 version includes Six Principles that are worthy of study. They can certainly help you in your own analysis of proportionality. The cited case law in the Commentary is structured around these six principles.


Principle 1: The burdens and costs of preserving relevant electronically stored information should be weighed against the potential value and uniqueness of the information when determining the appropriate scope of preservation.

Principle 2: Discovery should focus on the needs of the case and generally be obtained from the most convenient, least burdensome, and least expensive sources.

Principle 3: Undue burden, expense, or delay resulting from a party’s action or inaction should be weighed against that party.

Principle 4: The application of proportionality should be based on information rather than speculation.

Principle 5: Nonmonetary factors should be considered in the proportionality analysis.

Principle 6: Technologies to reduce cost and burden should be considered in the proportionality analysis.


Proportionality is one of those deep subjects where you should think for yourself, but also be open and listen to others. It is possible to do both, although not easy. It is one of those human tricks that will make us hard to replace by smart machines. The game I have created will help you with that. Try out the Small, Medium or Large? proportionality game by filling out the online polls I created.

But, you can do more. You can lead discussions at your law firm, company, class or CLE on the subject. You can become an e-discovery proportionality Game-Master. You can find out the consensus opinion of any group. You can observe and create statistics of how the initial opinions change when the other game players hear each others opinions. That kind of group interaction can create the so-called hive-effect. People often change their mind until a consensus emerges.

What is the small, medium or large proportionality consensus of your group? Even if you just determine majority opinion, and do not go through an interactive exercise, you are learning something of interest. Plus, and here is the key thing, you are giving game players a chance to exercise their analytical skills.

To help you to play this game on your own, and lead groups to play it, I created a Word Document that you are welcome to use.




%d bloggers like this: