2015 e-Discovery Rule Amendments: Dawning of the “Goldilocks Era”

November 11, 2015

Ralph_Losey_2013_abaThis blog presents my summary, analysis, and personal editorial comments on the 2015 Amendments to the Federal Rules of Civil Procedure that pertain to electronic discovery. The new rules go into effect on December 1st, 2015.

Overall Impressions

Overall the new Rules will be helpful, especially to newbies, but hardly the godsend that many hope for. The amendments will have very little impact on my legal practice. But that is only because the doctrines of proportionality and cooperation that the Amendments incorporate are already well-established in my firm. Many experienced attorneys say the same thing. The rule changes may make it a little easier to explain our positions to opposing counsel, and the court, but that is all.

Still, these rule amendments were not designed for experts. They were written for the vast majority of U.S. lawyers who are still struggling to do discovery in the modern age. Our profession remains embarrassingly computer challenged, so these rules are a necessary and good thing.

My only regret is that new Rule 37(e) may make it a little more difficult to catch and punish the few bad guys out there who try to cheat by destroying evidence. Still, we will get them. Fraudsters are never as smart as they think they are. When judges get the drift of what is happening, they will work around vagaries in new Rule 37(e) and send them packing. I am not overly concerned about that. Experienced federal judges can sniff out fraud a mile away and they do not hesitate to sanction bad faith attempts to game the system. We have defeated plenty of spoliating plaintiffs under the old rules and I am confident we will continue to do so under the new. The protests of some commentators on this issue seem a bit over-stated to me, although I do agree that the wording of new Rule 37(e) leaves much to be desired.

An Overly-Hard-Fought Victory for Proportionality

beavis-and-butt-head-fightingThe 2015 FRCP Rules Amendments were the most politicized and hard-fought in history. E-Discovery was the focus of all the battles. (Other changes are not-controversial and, not really that important, and will not be addressed here.) Large corporate and Plaintiffs attorney groups lobbied the supposedly independent Rules Committee for years. The well-funded defense Bar largely won, but the plaintiffs’ Bar still retained some bite and won several small victories. It was classic lobbying at its worst by both sides.

The Rules Committee should never have let itself get sucked into that kind of politics. They meant well, I’m sure, but they ended up with way too many conferences and bickering everywhere you looked. I personally got sick of it and cut way down on my schedule. I even quit one well-known group that allowed this infection to spoil its true purpose. Sad, but life is short. It is full of choices on what to do and who to do it with. I decided not to waste my time with silly games, nor watch the fall of a once great dynasty. I am consoled by the words of Churchill: “History will be kind to me for I intend to write it.”

Bottom line, partisan politics for court rule making must end with these 2015 Amendments. The judiciary and Rules Committee should be above politics. Make all of your meetings closed-door if you have to, but stop the circus. Hang your heads and learn from what happened.

As a result of the contentiousness of the proceedings, the final wording of most of the rules represent sausage making at its worst. The language is just filled with compromises. Years of interpretation litigation are assured. Maybe that can never be avoided, but certainly a better job could have been done by scholars working above the fray.

Proportionality Doctrine and the Beginning of the Goldilocks Era

In spite of the sordid background two high-minded themes emerged, much like flowers growing out of manure. The primary theme of all of the Amendments is clearly Proportionality. The secondary theme is an attempt to further attorney Cooperation by communication. Two doctrines promoted by the late, great founder of The Sedona Conference, Richard Braman.

goldilocksThe victory of proportionality proponents, myself included, may well usher in a new Goldilocks era for the Bar. Everyone who bothers to read the rules will know that they must look for discovery that is not too big, and not too small, but is just right. The just right Goldilocks zone of permitted discovery will balance out well-worn considerations, including costs, which are outlined by the rules.

This is not really new, of course. Old Rule 26(b)(2)(C) had the same intent for decades to avoid undue burden, by balance with benefits. But at least now the proportionality concerns to avoid undue expenses for discovery are up front and center to all discovery disputes. This forces judges to be more cost-conscious, and not allow liberal discovery, regardless of costs, delays and other burdens.

I have been promoting the proportionality doctrine at the heart of these amendments since at least 2010. So too did Richard Braman and his Sedona Conference. I recommend to you their latest version of the Commentary on Proportionality in Electronic Discovery (2013) that can be downloaded without cost at https://thesedonaconference.org/download-pub/1778.

For some of my articles on or featuring proportionality, please see:

Some contend that the changes in the rules embodying proportionality will make a big difference. Many long-term observers say that there are no real changes at all. It is just window dressing. So nothing will change. I think a “just rightGoldilocks type analysis suggests that the truth is somewhere in the middle, but inclined towards the “little change” side. See: Losey, R., One Man’s Trash is Another Man’s Evidence: Why We Don’t Need New Rules, We Need Understanding, Diligence, and Enforcement of Existing Rules (e-Discovery Team, 9/6/11) (criticizing the drive to solve the problems of e-discovery by just adding more rules, and suggesting instead that education and enforcement of existing rules were a better response).

Still, although a small change, and a sausage-like one at that, it is an important change. It should help all fair-minded attorneys to better serve their clients by protecting them from undue burdens in discovery, and also from undue burdens in preservation.

Goldilocks_chasedWe will all be arguing about the Goldilocks Zone now, where the burden is just right, is proportional, considering the criteria stated in the rules and the facts of the case. One size fits all is a thing of the past, especially when the one size is save everything and produce everything. Papa Bear’s big chair is way too large for most cases. And, small chair or not, every litigant is entitled to a seat at the discovery table, even a trespasser like Goldilocks.

New Rule 26(b)(1) – Discovery Scope and Limits

Here is the new language of 26(b)(1) which serves as the key provision in the 2015 Amendments implementing proportionality. Note that I have added the bullet-points here for clarity of reference and the bold. The original rules are, as usual, just one long run-on sentence.

Parties may obtain discovery regarding any nonprivileged matter that is relevant to any party’s claim or defense and proportional to the needs of the case, considering the

  • the importance of the issues at stake in the action,
  • amount in controversy,
  • the parties’ relative access to relevant information,
  • the parties’ resources,
  • the importance of the discovery in resolving the issues, and
  • whether the burden or expense of the proposed discovery outweighs its likely benefit.

Information within this scope of  discovery need not be admissible in evidence to be discoverable.

SubJect MatterThe first big change to 26(b)(1) is not seen here because it is an omission. The scope is now limited to “any  party’s claim or defense.” Previously a court could expand relevance scope to “SUBJECT MATTER” of the case, not just specific claims. This expansion was supposed to require a good cause showing, but, in practice, this condition was given little weight by judges and poorly understood by the Bar. Full subject matter discovery was commonly allowed with little or no real showing of cause. Often responding parties would simply capitulate and not demand a good cause showing. This could, in my experience, often lead to greatly expanded discovery. Now relevance cannot be expanded beyond actual claims made. This is a big improvement.

I am proud to say that this is a revision that I suggested to the Committee for adoption. I accomplished this without lobbying. My one direct conservation with the big-name Committee chair at a Bar event was about two minutes long. I outlined the idea and suggested the Committee at least consider it. The elevator-speech proposal was instantly rebuffed by her. She smiled and said that had been considered many times before over the years and simply was not “politically doable.” Silly me, to resurrect such an old, stale idea.

Still, I had a beginners mind on rule changes. I was convinced we needed to tighten the spigot of relevance to help counter-act the information deluge. I updated my prior blog on the proposal, added some more legal citations and analysis to make it more scholarly, and put forth my best argument. Rethinking Relevancy: A Call to Change the Rules to Narrow the Scope of ESI Relevance (e-Discovery Team, 1/24/2011). That’s it. I wrote a 3,700 word article. Nothing more. I knew the Committee would at least know about the article, and maybe some would read it, as I knew that some of them were regular readers.

Since the proposal had merit, as far as I was concerned, that was all that was required. No politics. No lobbying, just one chat where the chair said no-way, and then submission of an  article making my case for elimination of “subject matter” discovery. In my case that was all that was necessary. It worked. That is how it should work. I was actually completely surprised to see the elimination of the old subject matter provisions when an early draft was published by the Rule Committee. All the Committee Note says about this change is as follows:

The amendment deletes the former provision authorizing the court, for good cause, to order discovery of any matter relevant to the subject matter involved in the action. Proportional discovery relevant to any party’s claim or defense suffices. Such discovery may support amendment of the pleadings to add a new claim or defense that affects the scope of discovery.

The attention and politics of the Committee was focused on the new wording added to Rule 26(b)(1), which outlined the six criteria to consider to determine proportionality:

  1. the importance of the issues at stake in the action,
  2. amount in controversy,
  3. the parties’ relative access to relevant information,
  4. the parties’ resources,
  5. the importance of the discovery in resolving the issues, and
  6. whether the burden or expense of the proposed discovery outweighs its likely benefit.

Not sure why this was such a big deal for them because in fact none of this language is new at all. All the Committee ended up doing was use the exact same language that appeared in Rule 26(b)(2)(C) and then add the parties’ relative access to relevant information.  This last addition was a last minute addition. All it does is add the accessibility provision already in Rule 26(b)(2)(B) that was added in the 2006 Amendments. You would think the Committee would improve upon the language more to give the Bench and Bar more guidance. Still, the importance of the proportionality requirement is intended to be elevated by this move to the defined scope of relevance section, the section where discoverability is limited to information that is proportional to the needs of the case.

Here is the Committee Note explaining their revision, one that many now seize upon as some bold godsend:

The considerations that bear on proportionality are moved from present Rule 26(b)(2)(C)(iii). Although the considerations are familiar, and have measured the court’s duty to limit the frequency or extent of discovery, the change incorporates them into the scope of discovery that must be observed by the parties without court order.

Dear Committee, they are more than just familiar as your Note says, they are exactly the same! Please. Many had hoped for more, myself included. Oh well, what do you expect from political sausage?

In any in-person presentation of these rules I would now go through how these four factors play into discovery in various types of cases. In my firm I discourse at length on how this plays out in employment cases.

  1. the importance of the issues at stake in the action,
  2. amount in controversy,
  3. the parties’ relative access to relevant information,
  4. the parties’ resources,
  5. the importance of the discovery in resolving the issues.

Goldilocks ZoneThere is really nothing new here except the third point about relative access, and, in my opinion, that last minute addition by the Committee adds nothing. It was already in 26(b)(2)(B). I have been applying these factors and analysis for over thirty-five years. I have been calling it proportionality for over five years.

In most cases, but certainly not all, the main factor that comes into play is expense. Does the burden or expense of the proposed discovery outweighs its likely benefit? And what is the real, non-inflated, amount in controversy? The main change the proportionality labeled rules now force is a shift in thinking, to try to get the Bench and Bar to look at discovery as the tradeoff that it has always been, to try to get everyone thinking proportionally.

Reasonably_CalculatedThe final relevant change to 26(b)(1) already seems to be widely misunderstood by the Bar, namely the rewording of provisions in the rule pertaining to discovery and admissibility. The old rule, which many lawyers disliked for good reason, said: “Relevant information need not be admissible at the trial if the discovery appears reasonably calculated to lead to the discovery of admissible evidence.” It is true that this sentence was deleted, but it is not true that discovery is limited to admissible evidence. I have already seen at least one CLE, sponsored by the ABA no less, that incorrectly states that the old standard is dead. It is not. Weakened perhaps, but not gone. Remember, we are dealing with politics again and compromise language. The Plaintiffs’ Bar managed to keep the idea alive, but the sentence was modified and its placement shuffled. Rule 26(b)(1) still says:

Information within this scope of discovery need not be admissible in evidence to be discoverable.

Here is how the Committee Note explains this revision:

The former provision for discovery of relevant but inadmissible information that appears reasonably calculated to lead to the discovery of admissible evidence is also amended. Discovery of nonprivileged information not admissible in evidence remains available so long as it is otherwise within the scope of discovery. Hearsay is a common illustration. The qualifying phrase — “if the discovery appears reasonably calculated to lead to the discovery of admissible evidence” — is omitted. Discovery of inadmissible information is limited to matter that is otherwise within the scope of discovery, namely that which is relevant to a party’s claim or defense and proportional to the needs of the case. The discovery of inadmissible evidence should not extend beyond the permissible scope of discovery simply because it is “reasonably calculated” to lead to the discovery of admissible evidence.

I predict that we will be litigating that oh so subtle distinction for years. It remains to be seen what the Magistrates who usually rule of such issues will make of this change. It also remains to be seen what the practical impact of this change will be. I think that the “claims made” versus “subject matter of the litigation” change will have a far greater impact.

What is a Proportional Spend on e-Discovery?

Assuming that monetary factors are the primary considerations in a case, how much should be spent on electronic discovery? Do not just brush the question aside by saying every case is different. They are many similarities too. The longer you practice the more aware you become of the recurring patterns. What I want the Bench and Bar to do is start thinking proportionally. To start thinking from an over-all budgetary perspective.

Consider this hypothetical, one where all other factors being equal, money was the primary criteria to evaluate proportionality.

  • Assume a case with a real-world true value of $1,000,000.
  • What would be a proportional total cost of defense?
    • Assume $381,966 (@38%)
  • What would be a proportional total cost of all discovery in such a case?
    • Assume $145,147 (@14.6%)
  • Now for the punchline, what would be a proportional cost of e-discovery in a case like that?
    • Assume $55,157 (@5.6%)

Where am I getting these numbers?

In part I am getting these dollar amounts from my 35 years of experience as an attorney in private practice handling a wide variety of commercial litigation vases, mainly large ones, but also many smaller cases too, and lately including many, many employment law cases.

These numbers may not hold true in single plaintiff discrimination cases, or other small matters. But there may be some general truth here. You can see that from the fact that most Bar associations allow 40% recovery for fees in contingency cases. That compares to the 38% proportional expense assumed here: $381,966 in total fees and costs for a million dollar case (remember, assume true settlement value here, after weighing and discounting risks, not b.s. positions or demands). What do you think? Is approximately 38% of the true case value a proportional total expense in complex litigation? Does 38% seem appropriate? Too high, too low? What do you think is a proportional percentage? Please leave your comments below or send me a private email.

What about my assumption of a total cost for all discovery of $145,147 in a case where total fees are $381,966. Is it reasonable, proportional, to assume a spend of $145,147 for discovery of all types, including e-discovery? That represents around 14.6% of the total amount in controversy. Is that number too low? Too high?

Is it proportional to assume a spend of around 5.6% of the total amount in controversy for all e-discovery related activities in a case? Under this million dollar scenario that would be $55,157. Again, what do you think? And a different but related question, what has your experience been?

Now consider the bigger question, does a general metric for Proportionality of expenditures to true case value make sense in the law? Be it 38% or whatever?

Assuming it makes sense to talk of a general ratio for proportionality: Is the 6% of the total value of a case a reasonable amount for a party to pay for ESI discovery alone? If not, higher or lower? What ranges are you seeing in practice? I am seeing a wide variety, but I think that is because we are still in an early stage of maturity, and that it will eventually settle down into a pattern.

Did you know that Intel has gone on record many times as reporting that its e-Discovery spend in large cases averages 80% of its total spend for cost of defense? Since Intel says an average of 80% of its total litigation cost go to e-Discovery, if they spent $400,000 to defend a $1 Million case, that would be a spend of $320,000 on e-discovery, which is 32% of total case value, not 6%. Does this seem fair? Appropriate? Proportional?

I can see patterns and costs ranges, but at the same time I see outliers in cost, especially on the high-end. In my experience these are usually due to external factors such as extreme belligerence by one side or the other, attorney personality disorders, or intense spoliation issues. Sometimes it just may have to do with information management issues. But if you discount the low and high outliers a pattern starts to emerge. Hopefully someday an objective, systematic study will be done. My observations so far are just ad hoc.

Might the Golden Ratio Help Us to Navigate the Goldilocks Zone?

Aside from my long experience with what lawsuits cost, and the experience of others, the primary source of my hypothetical numbers here is from a famous ratio in math and art known as the Golden Ratio or Golden Mean: 1.61803399 to 1.0.


I came up with the numbers in the hypothetical by use of this ratio:

$1,000,000 – $381,966 (38%) – $145,147 (14.6%) – $55,1567 (5.6%)

These numbers are progressively smaller by .3819661%, and in this manner follows the proportion of the Golden Ratio.

The Golden Ratio is mathematically defined as the division where a+b is to a, as a is to b, as shown below. In other words, “two quantities are in the golden ratio if the ratio of the sum of the quantities to the larger quantity is equal to the ratio of the larger quantity to the smaller one.”


In Math this is known as PHI – Φ.

golden ratio shown in a model's faceIt does not matter if the math confuses you, just know that this proportion has been considered the perfect ratio to evoke feelings of beauty and contentment for thousands of years. It is well-known in all of the Arts, including especially Painting, Music and Architecture. It is the division that feels right, seems beautiful, and creates a deep intuitive sense of perfect division. It still does. Just look at the designs of most Apple products. This ratio is also found everywhere in nature, including the human body and structure of galaxies. It seems to be hardwired into our brains and all of nature.

I put together the two videos below to illustrate what I mean. There is far more to this than a math trick. The Golden Ratio seems to embody a basic truth. Every educated person should at least be familiar with this interesting phenomenon.


Perhaps the idea of perfect proportionality in art, math and science may also apply to law? Maybe it is a basic component of human reasonability and fairness? What do you think?

After giving a presentation much like this at a CLE I asked the question of whether the Golden Ratio in art, math and science might also apply to the law? I wanted to know what everyone thought and to get some interaction going. It was a daylong conference sponsored by Capital One and dedicated solely to the topic of Proportionality. Maura Grossman and I co-chaired the event in the Fall of 2010, when the doctrine was still new. Everyone had a clicker and answered yes or no to the question.

There was an eerie silence in the large auditorium after the results were quickly tabulated and shown on screen. Do you know what the proportion of yes and no answers were? 38% said NO, and 62% said Yes. The Golden Ratio came though in the opinion of the 200 or so attorneys and judges in attendance. You cannot make this stuff up. At first I thought maybe the boys in the tech booth were messing with me, but no. It was automatic and they were not paying attention. It was real. It was beautiful.

Facciola has proportionalityAsk Judge Facciola about it sometime. He was there and right after my opening spoke of the Golden Ratio as used in music. I remember he played an example from Bach. Judge Peck, Judge Mass and Judge Hedges were also there. So too was Judge Grimm, but by video appearance. Jason Baron, Conor Crowley, and Patrick Oot were also in attendance. So, I have a lot of witnesses to confirm what happened. It was a landmark event in many ways. One I will never forget for a whole host of reasons.

golden ratio is based in the Parthenon

A natural ratio clearly exists for proportionality in nature, art and math. I am not saying that this 38/62 ratio, 1.61803399, should apply in the law too as a general guide for proportionality. But it might. It is at least something to think about. What do you think? Again, let me know.

russian dollsFYI, I have written about this before with several examples, but never before described the Capital One conference and spooky Golden Ratio vote. My Basic Plan for Document Reviews: The “Bottom Line Driven” Approach (see especially footnote 15 and supporting text where I compare this drill-down proportionality analysis to Russian nesting dolls).

Change to Rule 37(e)

Rule 37(e) was completely rewritten and was the focus of most of the politics. That explains why the wording is such a mess. The Sedona Conference recommendations on how to revise the rules were largely ignored.

Judge Shira ScheindlinA large part of the politics here, from what I could see, was to counter-act Judge Shira Scheindlin (and a few other powerful judges, mostly in SDNY) who interpreted 2nd Circuit law to assert their right to impose case dispositive sanctions on the basis of gross-negligence alone. See: Pension Committee of the University of Montreal Pension Plan v. Banc of America SecuritiesLLC, 685 F.Supp.2d 456, 465 (S.D.N.Y.2010). Many in the defense Bar argued that there was a dangerous conflict in the Circuits, but since any large company can get sued in New York City, the SDNY views took practical priority over all conflicting views. They complained that the SDNY outlier views forced large corporations to over-preserve is a disproportionate manner.

Naturally Judge Scheindlin opposed these revisions and vigorously articulated the need to protect the judicial system from fraudsters. She proposed alternative language. The plaintiffs Bar stood behind her, but they lost. Sedona tried to moderate and failed for reasons I would rather not go into.

Other Circuits outside of New York make clear that case dispositive sanctions should only be imposed if INTENTIONAL or BAD FAITH destruction of evidence was proven. Many defense Bar types thought that this distinction with GROSS NEGLIGENCE was a big deal. So they fought hard and now pat themselves on the back. I think their celebration is overblown.

I personally do not think the difference between Bad Faith and Gross Negligence is all that meaningful in practice. For that reason I do not think that this rule change will have a big impact. Still, it is likely to make it somewhat easier for parties accused of spoliation to defend themselves and avoid sanctions, especially strong sanctions. If you think this is a good thing, then celebrate away. I don’t. The reality is this revision may well harm parties on both sides of the v., defendants and plaintiffs alike. I know we now see many Plaintiffs destroying evidence, especially cloud emails and Facebook posts. I expect they will rely upon this rule change to try to get away with it.

We will be litigating these issues for years, but as mentioned, I have faith in our federal judiciary. No matter what the rules, if they sniff out fraud, they will take appropriate action. The exact wording of the rules will not matter much. What was once labeled gross negligence will now be called bad faith. These concepts are so flexible and the entire pursuit of fraud like this is very fact intensive.

I think the best thing to do at this point is all of Rule 37(e) in full, as it bears repetitive reading:

If electronically stored information that should have been preserved in the anticipation or conduct of litigation is lost because a party failed to take reasonable steps to preserve it, and it cannot be restored or replaced through additional discovery, the court:

(1) upon finding prejudice to another party from loss of the information, may order measures no greater than necessary to cure the prejudice; or

(2) only upon finding that the party acted with the intent to deprive another party of the information’s use in the litigation may:

(A) presume that the lost information was unfavorable to the party;

(B) instruct the jury that it may or must presume the information was unfavorable to the party; or

(C) dismiss the action or enter a default judgment.

Judge Lee RosenthalI could talk up each paragraph, but this article is already overly long. I only pause to note how the rule now makes proportionality expressly relevant to preservation for the first time. Before this change our primary authority was the Order of the former Rules Committee chair, Judge Lee Rosenthal. Rimkus v Cammarata, 688 F. Supp. 2d 598 (S.D. Tex. 2010):

Whether preservation or discovery conduct is acceptable in a case depends on what is reasonable, and that in turn depends on whether what was done – or not done – was proportional to that case and consistent with clearly established applicable standards.

I strongly recommend that you read the extensive Committee Note that tries to explain this rule. The Notes can be cited and are often found to be persuasive, although, of course, never technically binding authority. Still, until we have case law on Rule 37(e), the Notes will be very important.

Minor Changes to Rules 26 & 34

Under Modified Rule 26(d)(2) a Request to Produce can be served anytime AFTER 21 days from the service of process.  You do not have to wait for the 26(f) conference. Under Modified Rule 34(b)(2)(A) a response to an early RFP is not due until 30 days after the parties’ first Rule 26(f) conference. This early service change was designed to encourage meaningful ESI discussion at 26(f) meet and greets.

Rule 34(b)(2)(B) was modified to require specific objections to request categories and “state whether any responsive materials are being withheld on the basis of that objection.” No duh, right? Yet I have seen this time and again, an objection is stated where no documents exist to begin with. Why?

Rule 26(f) was modified to include discussion of preservation, but also to include discussion of Evidence Rule 502 – Clawback Orders.

Change to Rule 16(b)

claws dementorNew language was added to Rule 16(b) as follows:

Scheduling Order may …

(iv) “… include any “agreements reached under Federal Rule of Evidence 502.”

(v) “direct that before moving for an order relating to discovery, the movant must request a conference with the court.”

Everyone is encouraged to entered into clawback agreements and 502(d) orders.

Change to Rule 1: An Already Great, But Underused Rule, Is Now Even Better

Waxse_LoseyI saved the best rule change for last, the change to Rule 1.0. Judge Waxse, the great promoter of Rule 1.0, should be happy (he often is anyway). Rule 1.0 FRCP – the JUST, SPEEDY AND INEXPENSIVE rule – is one of the most important rules in the book and yet, at the same time, one of the most overlooked and under-cited. All of the e-discovery knowledgable judges, not just David Waxse, can and do wax on and on about this rule. 

The 2015 Amendments are designed to further strengthen this important rule.  Rule 1 has long required judges to “construe and administer” all of the other rules in such a way as to not only secure “justice,” as you would expect, but also to secure “speedy and inexpensive” determinations. Surprised?

This dictate has long been an important policy for rule construction. It has been helpful to those who used it to oppose expensive, burdensome e-discovery. Nothing drives up expense more than “discovery about discovery” or detours into whether efforts to preserve, search or produce ESI have been extensive enough. Courts may allow this kind of expensive discovery if justice requires it, but only after balancing the other two equally important considerations of speed and expense. Here we have another proportionality analysis, one that applies indirectly to every other rule in the book.

The 2015 Amendments enlarged and strengthened the “just, speedy and inexpensive” dictates by making it clear that this duty not only applies to the court, the judges, but also to the parties and their attorneys. Moreover, the revised rule not only requires judges and parties to construe and administer the rules to attain just, speedy and inexpensive determinations, it also requires them to employ all of the other rules in this manner. The revised rule 1.0 reads as follows (new language in bold):

They (all of the rules) should be construed, administered, and employed by the court and the parties to secure the just, speedy, and inexpensive determination of every action and proceeding.

These revisions squarely place the burden of efficient litigation upon counsel, as well as the court. It is now a clear rule violation for an attorney to demand justice, no matter what it costs or how long it takes. All three criteria must be considered and implemented.

The Rule 1.0 change perfectly frames all of the other 2015 Amendments on proportionality and cooperation in ESI discovery and preservation.  As the Rules Committee Commentary to the Rule 1.0 amendments explains:

Most lawyers and parties cooperate to achieve these ends. But discussions of ways to improve the administration of civil justice regularly include pleas to discourage overuse, misuse, and abuse of procedural tools that increase cost and result in delay. Effective advocacy is consistent with — and indeed depends upon — cooperative and proportional use of procedure.”

Rule 1.0 now stands as powerful shield to oppose any counsel’s improper use of discovery as a weapon. Cost and delay must always be taken into consideration. Every motion for protective order should begin with a recitation to Rule One.


Ralph_x-mas_2013Update all of your discovery briefs to incorporate the new rules. Think proportional and act proportional. Sherlock Holmes was famous for his 7% solution, try mixing up your own 5.6% solution. That would be beautiful wouldn’t it, to only spend $5,600 total on e-Discovery in a $100,000 case? Try to come up with an overall budget. Figure out what you think is proportional to the case. Do not wait to respond to excessive demands. Be proactive. How many custodians are proportional? What is an appropriate date range? What ESI is really important and necessary to the case? How many files need be reviewed under a realistic cost-benefit analysis? What are the benefits? The burdens?

Talk about true case value with opposing counsel. Never bad mouth your client, but be honest. Get beyond the b.s. and posturing that does nothing but cause delay and expense. That is the only way your proportionality discussions will get real. The only way the judge will ever see things your way.

Other side won’t cooperate? Dealing with inept phonies? Have these discussions with the judge. Ask for a 16(b) conference to work out disagreements that surface in the 26(f) conference. Most judges have a pretty good feel for what certain kinds of cases are usually worth. Have the wake-up call early and try to save your client money. Analyze and argue benefit/burden. Also, be real and do not exaggerate what your e-discovery expenses will be. Back up your estimates with realistic numbers. Get vendors or specialists to help you.

All of this means that you must front-end your e-discovery work. It should come right after the retainer clears. The new Rule 37(e) is not a free pass to let up on preservation efforts or data collection. Find out what your problems are, if any, and talk about them asap. Bring them to the attention of the judge. Show that you are in good faith. The law never demands perfection, but does demand honest, reasonable efforts.

Make your discovery plan early. What do you want the other side to produce? Be specific. Have concrete discussions at the 26(f). The judges are getting fed up with drive-by meet and greets. It is dangerous to put off these discussions. Arrive at a fair balance between risk mitigation and cost control and move things along counsel. Speed counts, right up there with expense and justice. Your clients will appreciate that too.

Use honesty and diligence to navigate your way to the Goldilocks zone. Steer with realistic analysis. Be driven not only by the desire for justice, but also for quickness and sparse expense. Learn the new analytics, the new language and concepts of proportion. Master these new rules, as well as the e-discovery rules that remain from the 2006 Amendments, and you will prosper in the new Goldilocks Era.

Predictive Coding 3.0 – The method is here described as an eight-part work flow

October 18, 2015

This is the second part of a two-part article. Part One of Predictive Coding 3.0 described the errors in Predictive Coding 1.0 and 2.0, errors that are corrected by 3.0. The primary error addressed was the fallacy of the secret control set. The control set is very much accepted dogma among most e-discovery vendors and their hired experts. Still, after Part One came out, a few well-known experts spoke publicly in support of my anti-vendor-establishment critique. Many others have written to me privately to say they agree that control sets are b.s., that they have never used them, but few want to wade into the controversy. Never stopped me, especially when the attainment of just legal processes is concerned. Still, criticisms are easy. The articulation of positive replacements is the real challenge, and that is what this Part Two addresses.


Ralph_SimpsonThis concluding segment describes the Predictive Coding 3.0 methodology in terms of an eight-step work flow. Steps four, five and six iterate until the active machine training reaches satisfactory levels, and thereafter final quality control and productions are done. Although presented as sequential steps for pedantic purposes, Predictive Coding 3.0 is highly adaptive to circumstances and does not necessarily follow a rigid linear order. For instance, some of the quality control procedures are used throughout the search and review, and rolling productions can begin at any time. Also, the truth is, the work flow is far easier to do, then it is to put in words. I have only rarely been the smartest guy in the room (and they were usually small rooms in rural Florida where I live and went to school) and so, if I can do all of this, then you can too. It is easier than it looks. It just takes some practice and experience. A good guide is also very helpful at first.


Eight-Step Work Flow of Predictive Coding 3.0


The eight-step chart provides a model of the Predictive Coding 3.0 methodology. The circular flows depict the iterative steps specific to the predictive coding features. (You may download and freely distribute this chart without further permission, so long as you do not change it.) For background on how to plan for a complex predictive coding document review project, see Form Plan of a Predictive Coding Project. The plan consists of detailed Outline for the project. To understand the 3.0 method, you also need to understand how it is fits into an overall Dual-Filter Culling process. See License to Cull The Two-Filter Document Culling Method (2015).

The overall process is not nearly as complicated as version 1.0 and 2.0, as Grossman and Cormack criticize in their patent claim. See end of Part One of Predictive Coding 3.0 where this is discussed. I have found that it can be taught to any experienced lawyer in a second-chair type of hands-on training. Mere intellectual descriptions, as I am doing here, and have done before in the over fifty or so articles on the subject, can serve as a good preparation for effective apprenticeship training. The following is a full description of the work flow. It should look very familiar to prior readers of my articles on predictive coding. It is consistent with these prior articles, but has several important refinements and improvements that have emerged from my ongoing research and legal practice experience.

Step One: ESI Discovery Communications 

vulcan-mind-meldThe process starts with ESI Discovery Communications, not only with opposing counsel or other requesting parties, but also with the client and within the e-discovery team assigned to the case. Analysis of the scope of the discovery, and clear communications on relevance and other review procedures, are critical to all successful project management. The ESI Discovery Communications should be facilitated by the lead e-Discovery specialist attorney assigned to the case. But they must include the active participation by the whole team, including all trial lawyers not otherwise very involved in the ESI review. These communications are facilitated by a master plan, the details of which are refined in these initial communications. See eg. Form Plan of a Predictive Coding Project. Since nobody seems to have Spock’s Vulcan mind-meld abilities, this first step can sometimes be difficult, especially if there are many new members to the group. Still, a common understanding of relevance, the target searched, is critical to the successful outcome of the search. This includes the shared wisdom that this understanding will evolve and grow as discussed in Part One of this essay.

You begin with analysis and discussions with your client, your internal team, and then with opposing counsel, as to what it is you are looking for and the requesting party is looking for. The point is to clarify the information sought, the target. You cannot just stumble around and hope you will know it when you find it (and yet this happens all too often in legal search). You must first know what you are looking for. The target of most searches is the information relevant to disputed issues of fact in a case or investigation. But what exactly does that mean? If you encounter unresolvable disputes with opposing counsel on the scope of relevance, which can happen during any stage of the review despite your best efforts up-front, you may have to include the Judge in these discussions and seek a ruling.

Richard BramanThis dialogue approach is based on a Cooperative approach to discovery that was popularized by the late, great Richard Braman of the Sedona Conference. Cooperation is not only a best practice, but is, to a certain extent at least, a minimum standard required by rules of professional ethics and civil procedure. The primary goal of these dialogues for Predictive Coding purposes is to obtain a common understanding of the e-discovery requests, and reach agreement on the scope of relevancy and production. Additional conferences on other e-discovery issues are also key to attaining the now strongly rule endorsed doctrine of proportionality.

The dialogues in this first step may, in some cases, require disclosure of the actual search techniques used, which is traditionally protected by work product. The disclosures may also sometimes include limited disclosure of some of the training documents used, both relevant and irrelevant. Nothing in the rules requires disclosure of irrelevant ESI, but if adequate privacy protections are provided, it may be in the best interests of all parties to do so. Such discretionary disclosures may be advantageous as risk mitigation and efficiency tactics. If an agreement on search protocol is reached by the parties, or imposed by the court, the parties are better protected from the risk of expensive motion practice and repetitions of discovery search and production. Agreement on search protocols can also be used to implement bottom line driven proportional review practices. See Eg. the first case approving predictive coding search protocols by Judge Andrew Peck: Da Silva Moore v. Publicis Groupe, 2012 WL 607412 (S.D.N.Y. Feb. 24, 2012) (approved and adopted in Da Silva Moore v. Publicis Groupe, 2012 WL 1446534, at *2 (S.D.N.Y. Apr. 26, 2012)) and the many thereafter that followed Da Silva.

Andrew J. PeckAlso see Judge Andrew Peck’s more recent ruling on predictive coding, especially concerning disclosures: Rio Tinto v. Vale, 2015 WL 872294 (March 2, 2015, SDNY). Here Judge Peck wisely modifies somewhat his original views stated in Da Silva on the issue of disclosure. He no longer thinks that parties should necessarily disclose training documents, and may instead:

… insure that training and review was done appropriately by other means, such as statistical estimation of recall at the conclusion of the review as well as by whether there are gaps in the production, and quality control review of samples from the documents categorized as non-responsive. See generally Grossman & Cormack, Comments, supra, 7 Fed. Cts. L.Rev. at 301-12.

The Court, however, need not rule on the need for seed set transparency in this case, because the parties agreed to a protocol that discloses all non-privileged documents in the control sets. (Attached Protocol, ¶¶ 4(b)-(c).) One point must be stressed — it is inappropriate to hold TAR to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review.

Id. at *3. Also see Rio Tinto v. Vale, Stipulation and Order Re: Revised Validation and Audit Protocols for the use of Predictive Coding in Discovery, 14 Civ. 3042 (RMB) (AJP), (order dated 9/2/15 by Maura Grossman, Special Master, and adopted and ordered by Judge Peck on 9/8/15).

Judge Peck here follows the current prevailing view on disclosure that I also endorse, a view entirely in accord with Predictive Coding 3.0. Note that the review of quality control samples is specified in Step Seven, ZEN Quality Assurance Tests, of the 3.0 methodology. The clear trend today is away from full disclosure, especially for irrelevant documents. Counsel is advised to offer translucency, not transparency, and to run quality control tests of the efficacy of their work. The cooperative approach to discovery may sometimes require partial disclosure of relevant documents used for training, but only partial or otherwise limited disclosure of irrelevant documents used in training. Still, the polestar remains cooperation, a goal totally consistent with the protection of client rights and interests. Mancia v. Mayflower Begins a Pilgrimage to the New World of Cooperation, 10 Sedona Conf. J. 377 (2009 Supp.).

Step Two: Multimodal Search Review

Multimodal Search PyramidIn this step all types of search methods are used to try to find as many relevant documents as possible for the training rounds. In version 3.0 the samples found by the multimodal search methods in Step Two are selected by human judgment, not by random samples. The selections are made with the help of various software search features, including parametric Boolean keyword searches, similarity searches, and concept searches, and even strategic linear reviews of select custodians and date ranges. Documents outside of the dataset such a subpoenas or complaints may be included for training purposes too, even synthetic documents may be used as ideal exemplars.

predictive_coding_3.0All type of searches are used in Step Two except for Predictive coding based searches. They only reason they are not used here in Step Two is because you have not yet started predictive coding training. Step two is the search for the initial training set. It can be a long process, or a very short one. The same multimodal search process is carried out in Step-6, Hybrid Active Training, but now predictive coding is also used. So in that sense Step Six is where full multimodal search comes into play, including interaction with the AI you are training (that is the Hybrid part).

Although we speak of searching for relevant documents in Steps Two and Six, it is important to understand that many irrelevant documents are also incidentally found and coded in that process. Active machine learning does not work by training on relevant documents alone. It must also include examples of irrelevant documents. For that reason we sometimes actively search in Steps Two and Six for certain kinds of irrelevant documents to use in training. One of my current research experiments with Kroll Ontrack is to determine the best ratios between relevant and irrelevant documents for effective document ranking. See TREC reports at Mr. EDR as updated from time to time. This is one area where experience, art and skill now come into play, but we are working on standardizing that.

The multimodal search review in Steps Two and Six is carried out under the very general, second level supervision of the Subject Matter Experts on the case. They make final decisions where there is doubt concerning the relevance of a document or document type. The SME role is typically performed by a team, including the partner in charge of the case – the senior SME – and senior associates, and e-Discovery specialist attorney(s) assigned to the case. It is, or should be, a team effort, at least in most large projects.

The old-fashioned Predictive Coding 1.0 and 2.0 notions that a senior partner must work alone as the sole SME, and that he or she has to toil for days reviewing thousands of documents, including random junk files in a supposed control set, is not part of Predictive Coding 3.0. With no control set there is no need for such an inefficient process. Under my system a well-managed project has no SME time-demand problem. When I do a project, acting as the e-Discovery specialist attorney for the case, I listen carefully to the trial lawyer SME as he or she explains the case. By extensive Q&A the members of the team understand what is relevant. We learn from the SME. It is not exactly a Vulcan mind-meld, but it can work pretty well with a cohesive team.  Most trial lawyers love to teach and opine on relevance and their theory of the case.

Helmuth_Karl_Bernhard_von_MoltkeAlthough a good SME team communicates and plans well, they also understand, typically from years of experience, that the intended relevance scope is like a battle plan before the battle: No battle plan ever survives contact with the enemy. So too no relevance scope plan never survives contact with the corpus of data. The understanding of relevance will evolve as the documents are studied, the evidence is assessed, and understanding of what really happened matures. If not, someone is not paying attention. In litigation that is usually a recipe for defeat.

The SME team trains and supervises the document review specialists, aka, contract review attorneys, who usually then do a large part of the manual reviews (Step-Five), and few if any searches. Working with review attorneys is a constant iterative process where communication is critical. Although contract reviewers can be used for efficiency and money-saving purposes, instead of an army-of-one approach that I have also used, I typically use only a few reviewers, say from one to three. With good methods, including culling methods, and good software, it is rarely necessary to have more than that. With the help of strong AI, say that included in Mr. EDR, no more attorneys than that are needed to classify a million or so documents for relevance. More reviewers than that may well be needed for complex redaction projects and other production issues, but not for a well-designed relevance search.

When reviewers are used in relevance culling, it is very important for all members of the SME team to have direct and substantial contact with the actual documents, not just the reviewers. For instance, everyone involved in the project should see all hot documents found in any step of the process. It is especially important for the SME trial lawyer at the top of the expert pyramid to see them, but that is rarely more than a few hundred documents, often just a few dozen. Otherwise, the top SME need only see the novel and grey area documents that are encountered, where it is unclear on which side of the relevance line they should fall in accord with the last instructions. Again, the burden on the senior, and often technologically challenged senior SME attorneys, is fairly light under these Version 3.0 procedures.

The hands-on involvement of the entire SME team is especially needed in the second step, Multimodal search, and its echo Step Six, but is otherwise limited. The SME involvement up-front is needed to ensure that proper expertise is provided on relevance and the expected story to be told at trial. In some projects, at least one contract lawyer is brought in at Step Two to assist the SME team, and then later help in training of additional reviewers when they are included in Step Five. The e-Discovery specialist with expertise and experience with search, the Experienced Searcher, along with an expert on the software being used, the Power-User, should be involved in all stages of the project. Often these two roles (Power User and Experienced Searcher) are performed by one search expert, but rarely is that person also the sole SME of the legal issues. (I performed all three roles in the EDI Oracle experiment, but that was a rare exception.) In most real-world projects a team approach to the SME function is used. Still, the Experienced Searcher should always be a part if that SME team, if for no other reason than to ensure that the full communications outlined in Step One are maintained throughout the project.

The SME team relies on a primary SME, who is typically the trial lawyer in charge of the whole case, including all arguments of relevance to the judge and opposing counsel, at the start of the review. Thereafter, the head SME is only consulted on an as-needed basis to answer questions and make specific decisions on the grey area documents, again, typically in the echo Step Six, Hybrid Active Training, and Step Five, Document Review, as questions are raised by reviewers. There are always uncertain documents that need elevation to confirm relevance, but as the review progresses, their number usually decreases, and so the time and attention of the senior SME decreases accordingly.

The first round of machine training is also sometimes called the initial Seed Set Buildbut under 3.0 there is nothing special about it. The following training rounds are identified by number (assuming you even keep track of them at all), such as the second round of training, the third, etc. The only thing special about the first round of training is that it cannot include rank-based document searches because no predictive coding ranking has yet occurred. The ranking of documents according to probable relevance is only established by the machine training. So, of course, it cannot be used before the first training begins. It is instead used in Step Six Hybrid Active Training.

lexinton-inkedPersonally, I like to keep track of and control when the training happens, as opposed to having training running continuously in the background. That is where art and experience again come it. It is also where the man-machine hybrid aspects of my search methods come in. I like to see the impact on ranking of particular training documents. I like to see how it impacts the learning of Mr. EDR. If it is always on, you cannot really see it on a granular, document by document level. The conscious knowledge of training rounds is not a mandatory aspect of Predictive Coding 3.0, but does help me to maintain a close hybrid relationship with the AI in the software, the ghost in the machine. This is one of the things, for me at least, that makes predictive coding so much fun. Working with Mr. EDR can be a real blast. I hope to explain this a little better later in this essay, and in other essays that I plan to write to in the future on the joys of predictive coding.

Step Three: Random Baseline

dice_manyThe third step, which is not necessarily chronological, is essentially a computer function with statistical analysis. Here you create a random sample and analyze the results of expert review of the sample. Some review is thus involved in this step and you have to be very careful it is correctly done. This sample is taken for statistical purposes to establish a baseline for quality control purposes in Step Seven. Typically prevalence calculations are made at this point. Some software also uses this random sampling selection for purposes of a control set creation. As explained in Part One, Predictive Coding 3.0 does not use a control set, because it is so unreliable. In version 3.0 the sole purpose of the sample is to determine prevalence. Also see: In Legal Search Exact Recall Can Never Be Known. This can help guide your review and help you to decide when to stop training and move from the last iterative cycle of Step Six, into Step Seven – ZEN Quality Assurance Tests.

In Step Three an SME is only needed to verify the classifications of any grey area documents found in the random sample. The random sample review should be done by one reviewer, typically your best contract reviewer. They should be instructed to code as Uncertain any documents that are not obviously relevant or irrelevant based on their instructions and Step One. All relevance codings should be double checked, as well as Uncertain documents. The senior SME is only consulted on an as-needed basis.

Document review in Step Three is limited to the sample documents. Aside from that, this step is a computer function and mathematical analysis. Pretty simple after you do it a few times. In complex cases a consulting statistician or scientist might be needed for a short consult, especially if you want to go beyond simple random sampling and do stratification, or some other complex variation. Most of the time this is not necessary and any competent version 3.0 vendor expert should be able to help you through it.

Step Four: AI Predictive Ranking

Lexington-Web_basicThis is the Auto Coding Run where the software’s predictive coding calculations are performed. The software I use, at least most of the time, is Kroll Ontrack’s Mr. EDR. In the Fourth Step the software does all of the work. It applies all of the training provided by the lawyers to sort the data corpus according to their instructions. In Step Four the human trainers can take a coffee break while Mr. EDR ranks all of the documents for us according to probable relevance, or whatever other category we request. For instance, I usually like to train and rank on Highly Relevant and Privilege at the same time as plain Relevant – Irrelevant.

The first time the training runs used to be called the seed set training. Step Four repeats, with steps Five and Six, in an iterative process, which is also known as Continuous Active learning (CAL). The first repetition of the training is known as the second round of training, the next, the third round, etc. These iterations continue until the training is complete within the proportional constraints of the case. At that point the attorney in charge of the search may declare the search complete and ready for the next quality assurance test in Step Seven.

predictive_coding_3.0It is important to understand that this diagram is just a linear two-dimensional representation of Predictive Coding 3.0 for teaching purposes. These step descriptions are also a simplified explanation, at least to some extent. Step Four can take place just a soon as a single document has been coded. You could have continuous, ongoing machine training, all the time, if you wanted. That is what CAL means. Although it would be inefficient, you could in theory have as many rounds of training as there are documents reviewed and classified. In my TREC experiments with Mr. EDR, we would sometimes have over fifty rounds of training, and still complete the Topic review in just over a day.

As mentioned, I personally do not like the machine to train at certain arbitrarily set time intervals, which is the way most continuous training CAL 2.0 software does it (i.e. – every fifteen minutes). I like to be in control and to tell the machine exactly when and if to train. I do that to improve communication and understanding of the software ranking. It helps me to have a better intuitive understanding of the machine processes. It allows me to see for myself how a particular document, or usually a particular group of documents, impacts the overall ranking. This is an important part of the Hybrid aspects of the Predictive Coding 3.0 Hybrid Multimodal Method.

Lexie_robot_red_stickerStep Four in the eight-step workflow is a purely algorithmic function. The ranking of a million documents may take as long as an hour, or even more, depending on the complexity, the number of documents, and software. Or it might just take a few minutes. This depends on the circumstances and tasks presented. From the human trainer perspective Step Four is just slight break to relax and keep the mind clear, while the computer does all of the work.

The predictive coding software in this step is analyzing all of the document categorizations made in Step Three for the initial run, the seed set. Thereafter in all subsequent training rounds, when Step Four repeats, the Machine, for me Mr. EDR, not only uses the input from Steps Two and Three, but also the new documents reviewed in Step Five, and found and selected for training coded in Step Six. Note that skilled searchers rarely use all documents coded as training documents, and that is where the art and experience of search come in again. The concern is to avoid over-training on any one document type and thus lowering recall and missing a key black-swan document. There is also the question of the ideal relevance/irrelevance ratio for effective document ranking.

All documents selected for training are included in this Step Four computer processing. The software studies the documents marked for training, and then scans all of the data uploaded onto the review platform (aka, the corpus). It then ranks all of the documents according to probable relevance (and, as mentioned according to other categories too, such as Highly Relevant and Privilege, and does all of these categories at the same time, but for simplicity purposes here we will just consider the relevance rankings). It essentially assigns a probable value of from 0.01% to 99.9% probable relevance to each document in the corpus. (Note, some software uses different ranking values, but this is essentially what it is doing.) A value of 99.9% represents the highest probability that the document matches the category trained, such as relevant, or highly relevant, or privileged. A value of 0.01% means no likelihood of matching. A probability ranking of 50% represents equal likelihood. The machine is uncertain as to the document classification.

The first few times this AI-Ranking step is run, the software predictions as to a document’s categorization are often wrong, sometimes wildly so. It depends on the kind of search and data involved, and the number of documents already classified and included for training. That is why spot-checking and further training are always needed for predictive coding to work properly.

Predictive Ranking at this point in AI development is necessarily an iterative process where human feedback is provided throughout the process. Analytic software in the future may be far less dependent on human involvement in the iterative process, but for now it is critical. That is where the next two Steps Five and Six come in, Document Review and Hybrid Active Training.

Step Five: Document Review

This is the step where most of the actual document review is done, where the documents are seen and classified by human reviewers. In my experience, the human document review can take as little as one-second per document, assuming your software is good and fast, and it is an obvious document, to as long as a half-hour. The lengthy time to review a document is rare and only occurs where you have to fast-read a long document to be sure of its classification. Step five is the human time intensive part of Predictive Coding 3.0 and can take most of the time. Although, when I do a review, I usually spend more than half of the time in the other steps, sometimes considerable more. The TREC experiment was a good example of that, so was the Oracle EDI experiment.


Depending on the classification during Step Five Document Review, a document is either produced, if relevant and not-privileged, or not produced if irrelevant. If relevant and privileged, then it is logged, but not produced. If relevant, not privileged, but confidential for some reason, then it is either redacted and/or specially labeled before production. The special labeling performed is typically to prominently affix the word CONFIDENTIAL on the Tiff image production, or the phrase CONFIDENTIAL – ATTORNEYS EYES ONLY. The actual wording of the legends depends upon the parties confidentiality agreement or court order.

When redaction is required, the total time to review a document can sometimes go way up. The same goes for double and triple checking of privileged documents that sometime infect document collections in large numbers. In my TREC and Oracle experiments redactions and privilege double-checking were not required. The time-consuming redactions are often deferred to Step Eight – Productions. The equally as time-consuming privilege double-checking efforts can also be deferred to Step Seven – Quality Assurance, and again for a third-check in Step Eight.

When reviewing a document not already manually classified, the reviewer is usually presented with a document that the expert searcher running the project has determined is probably relevant. Typically this means it has higher than a 50% probable relevant ranking. The reviewer may, or may not, know the ranking. Whether you disclose that to a reviewer depends on a number of factors. Since I usually only use highly skilled reviewers, I trust them with disclosure. But sometimes you may not want to disclose the ranking.

During the review many documents predicted to be relevant, will not be. The reviewers will code them correctly, as they see them. If they are in doubt, they should consult the SME team. Furthermore, special quality controls in the form of second reviews on a random, or judgmental, selection process may be imposed on Man Machine disagreements. They often involve close questions and the ultimate results of the resolved conflicts are typically used in the next round of training. That is a decision made in Step Six. Prediction error corrections can be the focus of special searches in Step Six that look for such conflicts. Most quality version 3.0 software such as Mr. EDR have search functions built-in that are designed to locate all such conflicts. Reviewers then review and correct the computer errors by a variety of methods, or change their own prior decisions. This typically requires SME team involvement, but only very rarely senior level SMEs.

The predictive coding software learns from all of corrections to its predictive rankings. Steps 4 and 5 then repeat as shown in the diagram. This iterative process is considered a positive feedback loop that continues until the computer predictions are accurate enough to satisfy the proportional demands of the case.

Step Six: Hybrid Active Training

man_robotIn this step new documents are selected for review in the next iteration of Step Five. Moreover, in Step Six decisions are made as to what documents to include in training in the next round of Step Four, AI Predictive Ranking. Step Six is much like Step Two, Multimodal Search Review, except that now new types of document ranking search are possible. Since the documents are now all probability ranked in Step Four, you can use this ranking to select documents for the next round of document review (Step Five). For instance, the research of Cormack and Grossman, has shown that selection of the highest ranked documents can be a very effective method to continuously find and train relevant documents. Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic DiscoverySIGIR’14, July 6–11, 2014, at pg. 9. Also see Latest Grossman and Cormack Study Proves Folly of Using Random Search for Machine Training – Parts One,  TwoThree and Four. Another popular method, also tested and reported on by Grossman and Cormack, is to select mid-ranked documents, the ones the computer is uncertain about.

The preferred active learning process in the iterative machine learning steps of Predictive Coding 3.0 is now four-fold. How you mix and match the four methods is a matter of personal preference. Here are my team’s current preferences.

1. High Ranked Documents. My team will almost always look to see what the highest unreviewed ranked documents are after AI Predictive Ranking, Step Four. We may review them on a document by document basis, or only by spot-checking them. In the later, more common spot-checking scenario, a quick review of a certain probable relevant range, say all documents ranked between 95% to 99.9% (Mr. EDR has no 100%), may show that they all seem obvious relevant. We may then bulk code all documents in that range as relevant without actually reviewing them. This is a very powerful and effective method with Mr. EDR, and other software (so I’ve heard), so long as care is used not to over-extend the probability range. In other situations, we may only select the 99%+ probable relevant set for checking and bulk coding without review. The safe range typically changes as the review evolves and your latest conception of relevance is successfully imprinted on the computer.

EDR_Cape_found_itIn our cases the most enjoyable part of the review project comes when we see that Mr. EDR has understood our training and gone beyond us. He starts to see patterns that we cannot. He amazingly unearths documents that my team never thought to look for. The relevant documents he finds are sometimes dissimilar to any others found. They do not have the same key words, or even be the same known concepts. Still, Mr. EDR sees patterns in these documents that we do not. He finds the hidden gems of relevance, even outliers and black swans. That is when we think of Mr. EDR as going into superhero mode. At least that is the way my e-Discovery Team likes to talk about him.

By the end of most projects Mr. EDR attains a much higher intelligence and skill level than our own (at least on the task of finding the relevant evidence in the document collection). He is always lightening fast and inexhaustible, even untrained, but by the end of his education, he becomes a genius. Definitely smarter than any human as to this one task. Mr. EDR in that kind of superhero mode is what makes Predictive Coding 3.0 so much fun.

Watching AI with higher intelligence than your own, intelligence which you created by your training, is exciting. More than that, the AI you created empowers you to do things that would have been impossible before, absurd even. For instance, using Mr. EDR, my e-Discovery Team of three attorneys was able to do 30 review projects and classify 16,576,820 documents in 45 days. See TREC experiment summary at Mr. EDR. This is a very gratifying feeling of empowerment and augmentation of our own abilities. The high-AI experience comes though very clearly in the ranking of Mr. EDR near the end of the project, or really anytime before that, when he catches on to what you want and starts to find the hidden gems. I urge you all to give Predictive Coding 3.0 a try so you can have this same kind of advanced AI hybrid excitement.

Mr_EDR_Uncertain2. Mid-Ranked Uncertain Documents. We often choose to allow the machine, in our case Mr. EDR, to select the documents for review in the next iterated Step Five. We listen to what Mr. EDR tells us are the documents he wants to see. These are documents where the software classifier is uncertain of the correct classification. They are usually in the 40% to 60% probable relevant range. Human guidance on these documents as to their relevance helps the machine to learn by adding diversity to the documents presented for review. This in turn also helps to locate outliers of a type the initial judgmental searches in Step Two and Five may have missed.

3. Random. We may also select some documents at random, either by proper computer random sampling or, more often, by informal random selection, including spot-checking. This again helps maximize recall and premature focus on the relevant documents initially retrieved. Random samples taken in Steps Three and Seven are typically also all included for training, and, of course, are always very carefully reviewed. The use of random selection for training purposes alone is minimized in Predictive Coding 3.0.

4. Multimodal Human Search. Most of the time when not following the machine’s high ranked selection we are using whatever search method we can to try to find relevant documents in Step Six. It is a multimodal search process, except this time we can also use a variety of document ranking based searches. As mentioned, the ranked searches are not available in Step Two because the active machine learning had not already begun. The searches may include some linear review of selected custodians or dates, parametric Boolean keyword searches, similarity searches of all kinds, concept searches, as well as several unique predictive coding probability searches. We call that a multimodal approach. Again, you need not limit these searches to ESI in the original dataset, but can also use outside documents such a subpoenas or complaints; even synthetic documents may be used as ideal exemplars.

Step Seven: ZEN Quality Assurance Tests

ZEN here stands for Zero Error Numerics. Predictive Coding 3.0 requires quality control activities in all steps, but the efforts peak in this Step Seven. For more on the ZEN approach to quality control in document review see ZeroErrorNumerics.com.ZenBIn Step Seven a random sample is taken to try to evaluate the recall range attained in the project. The method currently favored is described in detail in Introducing “ei-Recall” – A New Gold Standard for Recall Calculations in Legal SearchPart One, Part Two and Part ThreeAlso see: In Legal Search Exact Recall Can Never Be Known.


ei-recallThe ei-Recall test is based on a random sample of all documents to be excluded from the Final Review for possible production. Unlike the ill-fated control set of Predictive Coding 1.0 methodologies, the sample here is taken at the end of the project. At that time the final relevance conceptions have evolved to their final form and therefore much more accurate projections of recall can be made from the sample. The documents sampled can be based on documents excluded by category prediction (i.e. probable irrelevant) and/or by probable ranking of documents with proportionate cut-offs. The focus is on a search for any false negatives (i.e., relevant documents incorrectly predicted to be irrelevant) that are Highly Relevant or otherwise of significance.

Total 100% recall of all relevant documents is said by the professors to be scientifically impossible (unless you produce all documents, 0% precision), a myth that I predict will soon be shattered. In any event, be it either impossible or very rare, total recall of all relevant document is legally unnecessary. The legal requirement is reasonable, proportional efforts to find the ESI that is important to resolve the key disputed issues of fact in the case. The goal is to avoid all false negatives of Highly Relevant documents. If this error is encountered, one or more additional iterations of Steps 4, 5 and 6 are required.

In Step Seven you also make and test the decision to stop the training (the repetition of Steps Four, Five and Six). This decision is evaluated by the random sample, but determined by a complex variety of factors that can be case specific. Typically it is determined by when the software has attained a highly stratified distribution of documents. See License to Kull: Two-Filter Document Culling and Visualizing Data in a Predictive Coding ProjectPart One, Part Two and Part Three, and Introducing a New Website, a New Legal Service, and a New Way of Life / Work; Plus a Postscript on Software Visualization.

When the stratification has stabilized you will see very few new documents found as predicted relevant that have not already been human reviewed and coded as relevant. You essentially run out of documents for Step Five review. Put another way, your Step Six no longer uncovers new relevant documents. This exhaustion marker may in many projects mean that the rate of newly found documents has slowed, but not stopped entirely. I have written about this quite a bit, primarily in Visualizing Data in a Predictive Coding ProjectPart One, Part Two and Part Three. The distribution ranking of documents in a mature project that has likely found all relevant documents of interest will typically look something like the diagram below. We call this the upside down champagne glass with red relevant documents on top and irrelevant on the bottom.data-visual_Round_5

Also see Postscript on Software Visualization where even more dramatic stratifications are encountered and shown.

Another key determinant of when to stop is the cost of further review. Is it worth it to continue on with more iterations of Steps Four, Five and Six? See Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data, 26 Regent U. Law Review 1 (2013-2014). Another criteria in the stop decision is whether you have found the information needed. If so, what is the purpose of continuing a search? Again, the law never requires finding all relevant, only reasonable efforts to find the relevant documents needed to decide the important fact issues in the case. This last point is often overlooked by inexperienced lawyers.

Step Eight: Phased Production

This last step is where the documents are actually produced. Technically, it has nothing to do with a predictive coding protocol, but for completeness sake, I wanted to include it in the work flow. This final step may also include document redaction, document labeling, and a host of privilege review issues, including double-checking, triple checking of privilege protocols. These are tedious functions where contract lawyers can be a big help. The actual identification of privileged documents from the relevant should have been part of the prior seven steps.

The production of electronic documents to the requesting party is done after a last quality control check of the media on which the production is made, typically CDs or DVDs. If you have to do FTP production to meet a tight deadline, I suggest also producing the same documents again the next day on a tangible media to keep a permanent record of the production. Always use a WORM medium for the production, meaning write once, and read many times. That means the data you produced cannot be altered. This might be helpful later for forensic purposes, along with hash, to confirm ESI authenticity and detect any changes.

The format of the production should be a non-issue. This is supposed to be discussed at the initial Rule 26(f) conference. Still, you might want to check again with the requesting party before you select the final production format and metadata fields. Remember, cooperation should be your benchmark and courtesy to opposing counsel on these small issues can go a long way. The existence of a clawback agreement and order, including a Rule 502(d) Order, should also be routinely verified before the production is made. Again, this should be a non-issue. The forms used should be worked out as part of the initial 26(f) meet and greet.

The final work included here is to prepare a privilege log. All good vendor review software should make this into a semi-automated process, and thus slightly less tedious. The logging is typically delayed until after production. Check with local rules on this and talk to the requesting party to let them know it is coming. Also, production is usually done in rolling stages as review is completed in order to buy more time and good will. As mentioned before, production of at least some documents can begin very early in the process and does not have to wait until the last step. Waiting to produce all of your documents at once is rarely a good idea, but is sometimes necessary.



After talking to many scientists in the information retrieval world I have found that they all agree it is a good idea to find relevant documents for training in any way you can. It makes no sense to limit yourself to any one search method. They agree that multimodal is the way to go, even if they do not use that language (after all, I did make up the term). They also all agree that effective text retrieval searches today should use some type of active machine learning (what we in the legal world calls predictive coding), and not just rely on the old search methods of keyword, similarity and concept. The multimodal use of all of the old methods to find training documents for the new method of active machine learning, is clearly the way to go. This hybrid approach exemplifies man and machine working together in an active partnership, a union where the machine augments human search abilities, not replaces them.

The Hybrid Multimodal Predictive Coding 3.0 approach described here is still not followed by most e-discovery vendors, including several prominent software vendors. They instead rely entirely on machine selected documents for training, or even worse, rely entirely on random selected documents to train the software. Others use all search methods except for predictive coding, primarily just keyword searches. They do so to try to keep it simple they say. It may be simple, but the power and speed given up for that simplicity is not worth it.

superman_animated3The users of the old software and old-fashioned methods will never know the genuine thrill that most search lawyers using AI experience when watching really good AI in action. The good times roll when you see that the AI you have been training has absorbed your lessons. When you see the advanced intelligence that you helped create kick-in to complete the project for you. When you see your work finished in record time and with record results. It is sometimes amazing to see the AI find documents that you know you would never have found on your own. Predictive coding AI in superhero mode can be exciting to watch.

My entire e-Discovery Team had a great time watching Mr. EDR do his thing in the thirty Recall Track TREC Topics in 2015. We would sometimes be lost, and not even understand what the search was for anymore. But Mr. EDR knew, he saw the patterns hidden to us mere mortals. In those cases we would just sit back and let him do the driving, occasionally cheering him on. That is when my Team decided to give Mr. EDR a cape and superhero status. He never let us down. It is a great feeling to see your own intelligence augmented and save you like that. It was truly a hybrid human-machine partnership at its best. I hope you get the opportunity soon to see this in action for yourself.


Get every new post delivered to your Inbox.

Join 4,339 other followers