Transparency in a Salt Lake TAR Pit?

November 11, 2018

A Salt Lake City Court braved the TAR pits to decide a “transparency” issue. Entrata, Inc. v. Yardi Systems, Inc., Case No. 2:15-cv-00102 (D.C. Utah, 10/29/18). The results were predictable for TAR, which is usually dark. The requesting party tried to compel the respondent to explain their TAR. Tried to force them to disclose the hidden metrics of Recall and Richness. The motion was too little, too late and was denied. The TAR pits of Entrata remain dark. Maybe TAR was done well, maybe not. For all we know the TAR was done by Sponge Bob Square Pants using Bikini Bottom software. We may never know.

Due to the Sponge Bobby type motion leading to an inevitable denial, the requesting party, Yardi Systems, Inc., remains in dark TAR. Yardi still does not know whether the respondent, Entrata,Inc., used active machine learning? Maybe they used a new kind of Bikini Bottom software nobody has ever heard of? Maybe they used KL’s latest software? Or Catalyst? Maybe they did keyword search and passive analytics and never used machine training at all? Maybe they threw darts for search and used Adobe for review? Maybe they ran a series of random and judgmental samples for quality control and assurance? Maybe the results were good? Maybe not?

The review by Entrata could have been a very well managed project. It could have had many built-in quality control activities. It could have been an exemplar of Hybrid Multimodal Predictive Coding 4.0. You know, the method we perfected at NIST’s TREC Total Recall Track? The one that uses the more advanced IST, instead of simple CAL? I am proud to talk about these methods all day and how it worked out on particular projects. The whole procedure is transparent, even though disclosure of all metrics and tests is not. These measurements are anyway secondary to method. Yardi’s motion to compel disclosure should not have been so focused on a recall and richness number. It should instead of focused on methods. The e-Discovery Team methods are spelled out in detail in the TAR Course. Maybe that is what Entrata followed? Probably not. Maybe, God forbid, Entrata used random driven CAL? Maybe the TAR was a classic Sponge Bob Square Pants production of shame and failure? Now Yardi will never know. Or will they?

Yardi’s Quest for True Facts is Not Over

About the only way the requesting party, Yardi, can possibly get TAR disclosure in this case now is by proving the review and production made by Entrata was negligent, or worse, done in bad faith. That is a difficult burden. The requesting party has to hope they find serious omissions in the production to try to justify disclosure of method and metrics. (At the time of this order production by Entrata had not been made.) If expected evidence is missing, then this may suggest a record cleansing, or it may prove that nothing like that ever happened. Careful investigation is often required to know the difference between a non-existent unicorn and a rare, hard to find albino.

Remember, the producing party here, the one deep in the secret TAR, was Entrata, Inc. They are Yardi Systems, Inc. rival software company and defendant in this case. This is a bitter case with history. It is hard for attorneys not to get involved in a grudge match like this. Looks like strong feelings on both sides with a plentiful supply of distrust. Yardi is, I suspect, highly motivated to try to find a hole in the ESI produced, one that suggests negligent search, or worse, intentional withholding by the responding party, Entrata, Inc. At this point, after the motion to compel TAR method was denied, that is about the only way that Yardi might get a second chance to discover the technical details needed to evaluate Entrata’s TAR. The key question driven by Rule 26(g) is whether reasonable efforts were made. Was Entrata’s TAR terrible or terrific? Yardi may never know.

What about Yardi’s discovery? Do they have clean hands? Did Yardi do as good a job at ESI search as Entrata? (Assuing that Yardi used TAR too.) How good was Yardi’s TAR? (Had to ask that!) Was Yardi’s TAR as tardy as its motion? What were the metrics of Yardi’s TAR? Was it dark too? The opinion does not say what Yardi did for its document productions. To me that matters a lot. Cooperation is a mutual process. It is not capitulation. The same goes for disclosure. Do not come to me demanding disclosure but refusing to reciprocate.

How to Evaluate a Responding Party’s TAR?

Back to the TAR at issue. Was Entrata’s TAR riddled with errors? Did they oppose Yardi’s motion because they did a bad job? Was this whole project a disaster? Did Entrata know they had driven into a TAR pit? Who was the vendor? What software was used? Did it have active machine learning features? How were they used? Who was in charge of the TAR? What were their qualifications? Who did the hands-on review? What problems did they run into? How were these problems addressed? Did the client assist? Did the SMEs?

Perhaps the TAR was sleek and speedy and produced the kind of great results that many of us expect from active machine learning. Did sampling suggest low recall? Or high recall? How was the precision? How did this change over the rounds of training. The machine training was continuous, correct? The “seed-set nonsense” was not used, was it? You did not rely on a control set to measure results, did you? You accounted for natural concept drift, didn’t you, where the understanding of relevance changes over the course of the review? Did you use ei-Recall statistical sampling at the end of the project to test your work? Was a “Zero Error” policy followed for the omission of Highly Relevant documents as I recommend?. Are corrective supplemental searches now necessary to try to find missing evidence that is important to the outcome of the case? Do we need to force them to use an expert? Require that they use the state of the art standard, the e-Discovery Team’s Predictive Coding 4.0 Hybrid Multimodal IST?

Yardi’s motion was weak and tardy so Entrata, Inc. could defend its process simply by keeping it secret. This is the work-product defense approach. This is NOT how I would have defended a TAR process. Or rather, not the only way. I would have objected to interference, but also made controlled, limited disclosures. I would have been happy, even proud to show what state of the art search looks like. I would introduce our review team, including our experts, and provide an overview of the methods, the work-flow.

I would also have demanded reciprocal disclosures. What method, what system did you use? TAR is an amazing technology, if used correctly. If used improperly, TAR can be a piece of junk. How did the Subject Matter Experts in this case control the review? Train the machine? Is that a scary ghost in the machine or just a bad SMI?

How did Entrata do it? How for that matter did the requesting party, Yardi, do it? Did it use TAR as part of its document search? Is Yardi proud of its TAR? Or is Yardi’s TAR as dark and hardy har har as Entrata’s TAR. Are all the attorneys and techs walking around with their heads down and minds spinning with secret doc review failures?

e-Discovery Team reviews routinely exceed minimal reasonable efforts; we set the standards of excellence.  I would have made reasonable reassurances by disclosure of method. That builds trust. I would have pointed them to the TAR Course and the 4.0 methods. I would have sent them the below eight-step work-flow diagram. I would have told then that we follow these eight steps or if any deviations were expected, explained why.

I would have invited opposing counsel to participate in the process with any suggested keywords, hot documents to use to train. I would even allow them to submit model fictitious training documents. Let them create fake documents to help us to find any real ones that might be like it, no matter what specific words are used. We are not trying to hide anything. We are trying to find all relevant documents. All relevant documents will be produced, good or bad. Repeat that often. Trust is everything. You can never attain real cooperation without it. Trust but verify. And clarify by talk. Talk to your team, your client, witnesses, opposing counsel and the judge. That is always the first step.

Of course, I would not spend unlimited time going over everything. I dislike meetings and talking to people who have little or no idea what I am saying. Get your own expert. Do the work. These big document review projects often go on for weeks and you could waste and spend a small fortune with too much interaction and disclosures. I don’t want people looking over my shoulder and I don’t want to reveal all of my tricks and work-product secrets, just the general stuff you could get by study of my books. I would have drawn some kind of line of secrecy in the sand, hopefully not quicksand, so that our disclosures were reasonable and not over-burdensome. In Entrata the TAR masters doing the search did not want reveal much of anything. They were very distrustful of Yardi and perhaps sensed a trap. More on that later. Or maybe Entrata did have something to hide? How do we get at the truth of this question without looking at all of the documents ourselves? That is very difficult, but one way to get at the truth is to look at the search methods used, the project metadata.

The dark TAR defense worked for Entrata, but do not count on it working in your case. The requesting party might not be tardy like Yardi. They might make a far better argument.

Order Affirming Denial of Motion to Compel Disclosure of TAR

The well-written opinion in Entrata, Inc. v. Yardi Systems, Inc., (D.C. Utah, 10/29/18) was  by United States District Judge Clark Waddoups. Many other judges have gone over this transparency issue before and Judge Waddoups has a good summary of the ones cited to him by the moving party. I remember tackling these transparency issues with Judge Andrew Peck in Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012), which is one of the cases that Judge Waddoups cites. At that time, 2012, there was no precedent even allowing Predictive Coding, much less discussing details of its use, including disclosure best-practices. We made strong efforts of cooperation on the transparency issues after Judge Peck approved predictive coding. Judge Peck was an expert in TAR and very involved in the process. That kind of cooperation that can be encouraged by a very active judge did not happen in Entrata. The cooperative process failed. That led to a late motion by the Plaintiff to force disclosure of the TAR.

The plaintiff, Yardi Systems, Inc, is the party who requested ESI from defendants in this software infringement case. It wanted to know how the defendant was using TAR to respond to their request. Plaintiff’s motion to compel focused on disclosure of the statistical analysis of the results, Recall and Prevalence (aka Richness). That was another mistake. Statistics alone can be meaningless and misleading, especially if range is not considered, including the binomial adjustment for low prevalence. This is explained and covered by my ei-Recall test. Introducing “ei-Recall” – A New Gold Standard for Recall Calculations in Legal SearchPart One, Part Two and Part Three (e-Discovery Team, 2015). Also see: In Legal Search Exact Recall Can Never Be Known.

Disclosure of the whole process, the big picture, is the best Defense Of Process evidence, not just a couple of random sample test results. Looks like the requesting party here might have just been seeking “gotcha material” by focusing so much on the recall numbers. That may be another unstated reason both the Magistrate and District Court Judges denied their late request for disclosure. That could why the attorneys for Entrata kept their TAR dark, even though they were not negligent or in bad faith. Maybe they were proud of their efforts, but were tired of bad faith hassling by the requesting party. Hard to know based on this opinion alone.

After Chief Magistrate Judge Warner denied Yardi’s motion to compel, Yardi appealed to the District Court Judge Waddoups and argued that the Magistrate’s order was “clearly erroneous and contrary to law.”  Yardi argued that “the Federal Rules of Civil Procedure and case law require Entrata, in the first instance, to provide transparent disclosures as a requirement attendant to its use of TAR in its document review.”

Please, that is not an accurate statement of the governing legal precedent. It was instead “wishful thinking” on the part of plaintiff’s counsel. Sounds like a Sponge Bob Square Pants move to me. Judge Waddoups considered taxing fees against the plaintiff under Rule 37(a)(5)(B) because of this near frivolous argument, but ultimately let them off by finding the position was not “wholly meritless.”

Judge Waddoups had no choice but to deny a motion like this filed under these procedures. Here is a key paragraph explaining his reasoning for denial.

The Federal Rules of Civil Procedure assume cooperation in discovery. Here, the parties never reached an agreement regarding search methodology. In the court’s view, the lack of any agreement regarding search methodology is a failure on the part of both parties. Nevertheless, Yardi knew, as early as May of 2017, that Entrata intended to use TAR. (See ECF No. 257-1 at 2.) The Magistrate Court’s September 20, 2017 Order stated, in part, that “[i]f the parties are unable to agree on . . . search methodology within 30 days of the entry of this Order, the parties will submit competing proposals . . . .” (ECF No. 124 at 2.) Yardi, as early as October 2, 2017, knew that “Entrata [was] refus[ing] to provide” “TAR statistics.” (See ECF No. 134 at 3.) In other words, Yardi knew that the parties had not reached an agreement regarding search methodology well before the thirty day window closed. Because Yardi knew that the parties had not reached an agreement on search methodology, it should have filed a proposal with the Magistrate Court. This would have almost certainly aided in resolving this dispute long before it escalated. But neither party filed any proposal with the Magistrate Court within 30 days of entry of its Order. Yardi has not pointed to any Federal Rule of Civil Procedure demonstrating that the Magistrate Court’s Order was contrary to law. This court rejects Yardi’s argument relating to the Federal Rules of Civil Procedure.

Conclusion

The requesting party in Entrata did not meet the high burden needed to reverse a magistrate,s discovery ruling as clearly erroneous and contrary to law. If you are ever going to win on a motion like this, it will likely be on a Magistrate level. Seeking to overturn a denial and meet this burden to reverse is extremely difficult, perhaps impossible in cases seeking to compel TAR disclosure. The whole point is that there is no clear law on the topic yet. We are asking judges to make new law, to establish new standards of transparency. You must be open and honest to attain this kind of new legal precedent. You must use great care to be accurate in any representations of Fact or Law made to a court. Tell them it is a case of first impression when the precedent is not on point as was the situation in Entrata, Inc. v. Yardi Systems, Inc., Case No. 2:15-cv-00102 (D.C. Utah, 10/29/18). Tell them the good and the bad. There was never a perfect case and there always has to be a first for anything. Legal precedent moves slowly, but it moves continuously. It is our job as practicing attorneys to try to guide that change.

The requesting party seeking disclosure of TAR methods in Entrata doomed their argument by case law  misstatements and in-actions. They might have succeeded by making full disclosures themselves, both of the law and their own TAR. The focus of their argument should be on the benefits of doing TAR right and the dangers of doing it wrong. They should have talked more about what TAR – Technology Assisted Review – really means. They should have stressed cooperation and reciprocity.

To make new precedent in this area you must first recognize and explain away a number of opposing principles,  including especially The Sedona Conference Principle Six. That says responding parties always know best and requesting parties should stay out of their document reviews. I have written about this Principle and why it should be updated. Losey, Protecting the Fourteen Crown Jewels of the Sedona Conference in the Third Revision of its Principles (e-Discovery Team, 2//2/17). The Sedona Principle Six argument  is just one of many successful defenses that can be used to protect against forced TAR disclosure. There are also good arguments based on the irrelevance of this search information to claims or defenses under Rule 26(b)(2) and under work-product confidentiality protection.

Any party who would like to force another to make TAR disclosure should make such voluntary disclosures themselves. Walk your talk to gain credibility. The disclosure argument will only succeed, at least for the first time (the all -important test case), in the context of proportional cooperation. An extended 26(f) conference is a good setting and time. Work-product confidentiality issues should be raised in the first days of discovery, not the last day. Timing is critical.

The 26(f) discovery conference dialogue should be directed towards creating a uniform plan for both sides. This means the TAR disclosures should be reciprocal. The ideal test case to make this law would be a situation where the issue is decided early at a Rule 16(b) hearing. It would involve a situation where one side is willing to disclose, but the other is not, or where the scope of disclosures is disputed. At the 16(b) hearing, which usually takes place in the first three months, the judge is supposed to consider the parties’ Rule 26(f) report and address any discovery issues raised, such as TAR method and disclosures.

The first time disclosure is forced by a judge it will almost certainly be a mutual obligation. Each side should will be required to assume the same disclosure obligations. This could include  a requirement for statistical sampling  and disclosure of certain basic metrics such as Recall range, Prevalence and Precision? Sampling tests like this can be run no matter what search method is used, even little old keyword search.

It is near impossible to come into court when both sides have extensive ESI and demand that your opponent do something that you yourself refuse to do. If you expect to be able to force someone to use TAR, or to disclose basic TAR methods and metrics, then you had better be willing to do that yourself. If you are going to try to force someone to disclose work-product protected information, such as an attorney’s quality control tests for Recall range in document review, then you had better make such a limited waiver yourself.

 


Do TAR the Right Way with “Hybrid Multimodal Predictive Coding 4.0”

October 8, 2018

The term “TAR” – Technology Assisted Review – as we use it means document review enhanced by active machine learning. Active machine learning is an important tool of specialized Artificial Intelligence. It is now widely used in many industries, including Law. The method of AI-enhanced document review we developed is called Hybrid Multimodal Predictive Coding 4.0. Interestingly, reading these words in the new Sans Forgetica font will help you to remember them.

We have developed an online instructional program to teach our TAR methods and AI infused concepts to all kinds of legal professionals. We use words, studies, case-law, science, diagrams, math, statistics, scientific studies, test results and appeals to reason to teach the methods. To balance that out, we also make extensive use of photos and videos. We use right brain tools of all kinds, even subliminals, special fonts, hypnotic images and loads of hyperlinks. We use emotion as another teaching tool. Logic and Emotion. Sorry Spock, but this multimodal, holistic approach is more effective with humans than an all-text, reason-only approach of Vulcan law schools.

We even try to use humor and promote student creativity with our homework assignments. Please remember, however, this is not an accredited law school class, so do not expect professorial interaction. Did we mention the TAR Course is free?

By the end of study of the TAR Course you will know and remember exactly what Hybrid Multimodal means. You will understand the importance of using all varieties of legal search, for instance: keywords, similarity searches, concept searches and AI driven probable relevance document ranking. That is the Multimodal part. We use all of the search tools that our KL Discovery document review software provides.

 

The Hybrid part refers to the partnership with technology, the reliance of the searcher on the advanced algorithmic tools. It is important than Man and Machine work together, but that Man remain in charge of justice. The predictive coding algorithms and software are used to enhance the lawyers, paralegals and law tech’s abilities, not replace them.

By the end of the TAR Course you will also know what IST means, literally Intelligently Spaced Training. It is our specialty technique of AI training where you keep training the Machine until first pass relevance review is completed. This is a type of Continuous Active Learning, or as Grossman and Cormack call it, CAL. By the end of the TAR Course you should also know what a Stop Decision is. It is a critical point of the document review process. When do you stop the active machine teaching process? When is enough review enough? This involves legal proportionality issues, to be sure, but it also involves technological processes, procedures and measurements. What is good enough Recall under the circumstances with the data at hand? When should you stop the machine training?

We can teach you the concepts, but this kind of deep knowledge of timing requires substantial experience. In fact, refining the Stop Decision was one of the main tasks we set for ourself for the  e-Discovery Team experiments in the Total Recall Track of the National Institute of Standards and Technology Text Retrieval Conference in 2015 and 2016. We learned a lot in our two years. I do not think anyone has spent more time studying this in both scientific and commercial projects than we have. Kudos again to KL Discovery for helping to sponsor this kind of important research  by the e-Discovery Team.

 

 

Working with AI like this for evidence gathering is a newly emerging art. Take the TAR Course and learn the latest methods. We divide the Predictive Coding work flow into eight-steps. Master these steps and related concepts to do TAR the right way.

 

Pop Quiz: What is one of the most important considerations on when to train again?

One Possible Correct Answer: The schedule of the humans involved. Logistics and project planning is always important for efficiency. Flexibility is easy to attain with the IST method. You can easily accommodate schedule changes and make it as easy as possible for humans and “robots” to work together. We do not literally mean robots, but rather refer to the advanced software and the AI that arises from the machine training as an imiginary robot.

 

 

 

 

 

 

 

 

 


The Importance of Witness Interviews: ‘What Happens in Vegas Shouldn’t Stay in Vegas’

September 16, 2018

A discovery order in Vegas shows the importance of witness interviews and what can happen when you take a cavalier attitude towards preservation. Small v. University Medical CenterCase No. 2:13-cv-0298-APG-PAL (D.C. Nev., 9/9/18) (FLSA class action seeking unpaid wages for skipped meal breaks). The lengthy order is entitled Report and Recommendation and Final Findings of Fact and Conclusions of Law and imposes severe sanctions on the defendant. The order proves, when it comes to e-discovery at least, what happens in Vegas doesn’t stay in Vegas. The truth does and should come out, including where’s the electronic evidence. Interviews are a good way to find out what really happened.

This is a long blog – 5,122 words – but it is still a lot shorter than the 123 page Short opinion, which is not short at all. I counted, it is 48,980 words. Not that I’m complaining, but it is one of the longest discovery orders I have ever read. It has many good instructional elements. Specialists should probably read and skim the whole opinion.

The Short opinion also has the distinction of having taken longer to prepare than any other discovery order I have ever read – FOUR YEARS! Can you imagine any decision taking that long? I am sure there were good reasons, but still. That is a full presidential term.

First Steps of e-Discovery: Prepare and Preserve

The FLSA suit arose from a DOL investigation that faulted the defendant employer hospital, UMC, for failing to keep “accurate records” of the time worked. UMC’s alleged records failures continued after it was sued. They failed to give timely preservation notices and failed to interview key custodians. That’s a failure of the first two legal tasks a lawyer is required to do in Electronic Discovery Best Practices (EDBP), steps two and three (step one is prepare). See EDBP.com (detail shown above right with all ten legal activities shown below); also see: Favro, Phillip, Vegas Court Spotlights the Importance of Custodian Interviews with New ESI Sources (LegalTech News 8/30/18) (further discussed below); John Patzakis, Three Key eDiscovery Preservation Lessons from Small v. University Medical Center (Next Generation eDiscovery Blog, 9/12/18).

Judge Peggy Leen’s Order

Magistrate Judge Peggy A. Leen is the learned judge who wrote the opinion in Small v. University Medical Center, Report and Recommendation and Final Findings of Fact and Conclusions of Law. The order affirms and implements most of the recommendations of the Special Master for e-Discovery appointed several years ago in this case, Daniel Garrie.

The Special Master’s Report was issued four years earlier on August 18, 2014, two years after the suit was filed in July 2012. The Report was notable for characterization of defendant’s discovery misconduct as so egregious as to “shock the conscience” and make “a mockery of the orderly administration of justice.” It was a long, complicated report.

When she completed her work she ruled in large part for the plaintiffs and  sanctioned the defendant:

VI. THE COURT’S FINDINGS AND CONCLUSIONS

The court has personally conducted a thorough review of the record prior to the special master’s appointment and the record of the proceedings conducted by the special master. The record before the court and the record developed by the special master amply supports his findings that UMC destroyed evidence by failing to identify, preserve, collect, process, and search multiple repositories of information relevant to the parties’ claims and defenses.

UMC failed to preserve several different types of ESI, including an estimated 26,000 text messages and 38,000 documents from a shared drive “containing human resources, corporate compliance, employee grievance, payroll, and DOL investigation data.” The documents lost include important policy and procedure manuals regarding meal breaks and compensation. Relevant ESI on laptops, desktops and local drives were not preserved until some 18 months into the litigation. UMC also failed to comply with multiple discovery orders, leading to the plaintiffs’ motions for sanctions.

Judge Leen did not follow the recommendation of the Special Master to impose a sanction of default judgment in favor of 613 class members on the Fair Labor Standards Act claims. Instead, she imposed a permissive adverse inference jury instruction, along with monetary sanctions. These jury instructions can have a profound impact on the jury, but not as  strong as a mandatory adverse inference instruction. The mandatory instruction almost always leads to a verdict against the spoliating party. The permissive kind of instruction imposed here still gives a defendant like UMC a chance. The sanctioned party can still prevail with a jury on the merits of the case, albeit a slim chance. Here is the specific language that Judge Leen suggested be used at trial with the jury:

2. UMC is sanctioned in the form of an instruction to the jury that the court has found UMC failed to comply with its legal duty to preserve discoverable information, failed to comply with its discovery obligations, and failed to comply with a number of the court’s orders. The instruction will provide that these failures resulted in the loss or destruction of some ESI relevant to the parties’ claims and defenses and responsive to plaintiffs’ discovery requests, and that the jury may consider these findings with all other evidence in the case for whatever value it deems appropriate.

Careful study of the long opinion shows a very practical, albeit unstated reason for Judge Leen to make this concession. It made her order much harder to appeal; some would say appeal-proof. (After you put four years into something you want it to last.) That is because near the end of the process at one of the hearings Judge Leen was able to get defendant’s own attorney to concede that an adverse inference jury instruction would be appropriate. You do not see that happen very often. But this attorney apparently saw the writing on the wall from the comments the judge was making and realized that accepting a permissive inference was the best they could hope for and certainly a lot better than default judgments for all 613 class members.

Here is Judge Leen’s explanation of how this admission came about.

During oral argument on its objections to the special master’s R & R, counsel for UMC stated “I’m not even going to tell you that I don’t think we shouldn’t be sanctioned.” (Hr’g Tr. 24:28-25:1, Oct. 21, 2014, ECF No. 229.) When asked what sanction he felt was appropriate based on the developed record, UMC’s counsel suggested that an adverse inference jury instruction would be appropriate. (Tr. 25:4-10.)

Here we see a wise and experienced judge in action. Too bad Peggy Leen retires in 2019.

Judge Leen had good reason under the law to hesitate to enter default judgments on 613 claims, effectively ending the cases except to determine the amount of damages, all without any hearing on the merits of the claims. Entry of the  lesser sanction of a permissive instruction was consistent with Judge Leen’s analysis of Rule 37(b) on sanctions for violation of court orders.

[T]he court cannot conclude that UMC’s multiple discovery failures and failure to comply with the court’s orders threatens to interfere with the rightful decision of this case on the merits.

The lesser sanction was also consistent with her analysis of 2015 revisions to Rule 37(e) on sanctions for ESI spoliation, Rule 1 on just-speedy-inexpensive, and Rule 26(b)(1) on proportionality. Here is Judge Leen’s well-accepted analysis of 37(e):

To summarize, the court may impose sanctions against UMC under the current version of Rule 37(e) only if it finds: (1) UMC failed to preserve ESI “that should have been preserved” in anticipation or conduct of litigation; (2) the information was lost because UMC failed to take reasonable steps to preserve it; (3) the ESI cannot be restored or replaced; and (4) the plaintiffs were prejudiced by the loss. If all of these prerequisites are met, the court may issue sanctions no greater than necessary to cure the prejudice caused by the loss. Only if the court finds UMC acted with intent to deprive may the court impose the most severe sanctions.

Judge Leen then applied the law to the facts.

The court has found that UMC failed to preserve ESI that should have been preserved in anticipation of litigation, and throughout the course of this litigation. The court has also found that the information was lost because UMC failed to take reasonable steps to preserve it. Thousands of text messages on UMC Blackberry devices were lost and cannot be restored. Tens of thousands of files from the Q-Drive were lost and cannot be restored prior to December 2013. . . .

However, the special master’s extraordinary expertise and persistence resulted in restoration, remediation, and production of a great deal of relevant and discoverable ESI. The special master was able to direct restoration of the time tracking systems UMC failed to disclose until near the end of special master proceedings. Fortunately, Jackie Panzeri, UMC’s payroll manager who described herself as a “pack rat” that “keeps documents forever” had a lot of documents on her personal drive and several archives full of emails she did not delete or modify. She was involved in the DOL investigation from the beginning and saved both documents collected and produced to the DOL and for this case. The court is also mindful that ESI is stored in multiple locations and that modified or lost data from the seven key custodians is likely to be found in other locations. . . .

Although the court finds plaintiffs have been prejudiced by the loss of data from key repositories and custodians, the loss has not threatened to interfere with the rightful decision of the case on its merits given the large volume of ESI the special master was able to ensure that UMC produced. For these reasons, the court finds that lesser sanctions are appropriate, proportional, and no greater than necessary to cure the prejudice caused by the loss of ESI uncovered by the special master.

As you can see, hope springs eternal. Judge Leen’s still thinks that the now lost ESI from the seven key custodians is likely to be found in other locations. 

I doubt the Special Master Garrie would share the same optimism. He has already called defendant’s conduct a mockery of the orderly administration of justice. In his Report the Special Master said he has “serious doubts that UMC can complete discovery in a defensible manner going forward without increased candor to the Court and their own counsel, and more competent technical assistance.’ Well, maybe they will change. If not, and Judge Leen is wrong and the missing ESI is not found, then Judge Lee or her successor might reconsider and upgrade the sanction to a mandatory adverse inference. Special Master Garrie may yet get his way.

Defendant’s Threshold Errors

The quotes below from Small summarize the key factual findings of defendants’ threshold errors, the ones that lead to most of the others (emphasis added), much like a domino effect. To me these are the most important errors made and you should study Judge Leen’s words here closely.

D. UMC Executives Failed to Accept Responsibility for Ensuring that ESI was Preserved and Failed to Notify Key Custodians and IT Staff to Preserve, and Prevent Loss, or Destruction of Relevant, Responsive ESI

The record amply supports the special master’s findings that UMC had no policy for issuing a litigation hold, and that no such hold was issued for the first eight months of this litigation until after Mr. Espinoza was deposed on April 8, 2013, and was asked about UMC’s response to plaintiff’s August 6, 2012 preservation letter. The special master accurately found that UMC executives were unaware of their preservation duties, ignored them altogether, or at best addressed them “in the hallway in passing.” . . .

The special master’s finding that UMC executives failed to accept responsibility for their legal duty to preserve is amply supported in the record. UMC executives and counsel failed to communicate with and provide adequate instructions to the department heads and IT personnel of repositories containing discoverable ESI to prevent the loss or destruction of potentially relevant ESI. . . .

There is no evidence in the record, and UMC does not suggest there is any, that current or former counsel gave instructions to UMC to suspend business as usual to prevent the destruction, deletion or modification of ESI responsive to plaintiffs’ discovery requests. . . .

It is also undisputed that UMC’s prior and current counsel failed to conduct timely custodian interviews. Custodian interviews were not conducted until well into the special master proceedings when it became apparent they had not been done. The special master required the interviews to be conducted a second time because the initial custodian interviews conducted by counsel were inadequate. . . .

There is ample support in the record that UMC executives displayed a cavalier attitude about their preservation obligations addressing them in passing, and that UMC executives repeatedly took the position in declarations and testimony that responsibility for preservation was someone else’s job. . . .

The special master correctly found that current and former counsel failed to conduct timely custodian interviews to identify individuals with discoverable information and key repositories of discoverable ESI.

The record in this matter is very complex and voluminous. That is why the Special Master Report and the Order by Judge Leen are so lengthy; 123 pages for the order alone. Suffice it to say, if witness interviews of key custodians been conducted when they should have, shortly after suit was filed, a great deal of relevant evidence that ultimately was lost could have been saved. The Special Master’s detailed findings make that obvious. The lost-files could have been identified and preserved unaltered. Lines of responsibility to comply with legal preservation obligations could have been clarified and enforced. Had these interviews been conducted, and the ESI found quickly, the relevant ESI could have been bulk-collected and the evidence saved from spoliation.

As it is, the actions and mistakes of defendant here have severely weakened their case. That’s what can easily happen when a company has a cavalier attitude to compliance with their legal obligation to preserve potentially relevant ESI.

Eight Failed Challenges to the Special Master’s Report

Judge Leen considered and rejected eight challenges to the Special Master’s report that were raised by the defendant employer, UMC:

  1. Competence and Impartiality of the Special Master, Daniel Garrie.
  2. UMC’s Failure to Comply with the Court’s Orders to Preserve and Produce ESI.
  3. UMC’s Failure Have a Preservation Policy or Litigation Hold Policy and Failure to Timely Implement One.
  4. UMC’s Executives Failure to Accept Responsibility for Ensuring that ESI was Preserved and Failure to Notify Key Custodians and IT Staff to Preserve, and Prevent Loss, or Destruction of Relevant, Responsive ESI.
  5. UMC’s Failure to Disclose the Existence of Relevant ESI Repositories, Including Multiple Timekeeping Systems and the Q-Drive Until Late in the Special Master Proceedings.
  6. UMC Modified, Lost, Deleted and/or Destroyed ESI Responsive to Plaintiffs’ Discovery Requests.
  7. UMC’s Failure to Comply with its Legal Duty to Preserve, Failure to Put in Place a Timely Litigation Hold, Failure to Comply with Multiple Court Orders to Preserve and Produce Responsive ESI, and Loss and Destruction of Responsive ESI (1) Necessitated the Appointment of a Special Master, (2) Caused Substantial Delay of these Proceedings, and (3) Caused Plaintiffs to Incur Needless Monetary Expenses.
  8. The Special Master Correctly Concluded UMC Repeatedly Misrepresented the Completeness of its Production of Documents Produced to DOL; However, UMC Was Not Ordered to Produce Kronos Payroll Data in Spreadsheet Format.

Defendants failed in their challenges to the Special Master’s findings, including the threshold challenge to Special Master Dan Garrie’s competence. Ouch! Garrie is a Senior Managing Partner of Law & Forensics. He has written numerous articles and books on law, technology and e-discovery. See eg. D. Garrie & Yoav Griver. Dispute Resolution and E-Discovery, Thomson Reuters (2nd ed. 2013). Garrie earned a Masters degree in computer science at Brandeis University before going on to law school. A challenge to his expertise was misplaced.

The challenge did not go over well with the supervising Judge who studied his work more closely than anyone. After emphatically rejecting the hospital arguments, Judge Peggy Leen stated:

The court has conducted a de novo review of all of the special master proceedings and finds that he was professional and courteous, if occasionally frustrated by testimony displaying a lack of appreciation of UMC’s legal duties to preserve and produce responsive ESI. He was repeatedly told by UMC executives and employees that they did not know about their duty to preserve, had not learned about their preservation obligations from counsel, did not know what a litigation hold was, and had not explored relevant repositories of information responsive to plaintiffs’ discovery requests.

Bench Slap of Defendant’s Attorneys

With a background like that it is not surprising that the Special Master uncovered so much evidence of incompetence and malfeasance in preserving evidence. Judge Leen held: (emphasis added)

UMC was on notice that its timekeeping, time systems, payroll policies, and procedures were relevant to this litigation. UMC also knew it was unable to document that employees were being compensated for actual time worked. Both UMC and its former and current counsel failed to comply with UMC’s legal duty to suspend routine document retention/destruction policies to ensure the preservation of relevant documents. UMC failed to communicate the need to preserve relevant documents and ESI to employees in possession or likely to be in possession of discoverable information, or for that matter to communicate this duty even to “key players.” UMC and its counsel failed to identify, locate, and maintain information relevant to specific, predictable, and identifiable claims involved in this litigation.

Note that Judge Leen goes out of her way to include the defendant and its lawyers in the blame, both its  prior attorneys and its present attorneys. All of these attorneys failed in the “legal duty to suspend routine document retention/destruction policies to ensure the preservation of relevant documents.” In situations of shared blame like this the attorneys involved are sometimes personally sanctioned along with the client, but this has not happen here. Judge Leen did make several sharp comments against the defendants lawyers, includi9ng this finding:

UMC’s current counsel blamed former counsel and their ESI consultants for the delay in producing responsive ESI. Counsel for UMC advised the court at the hearing on June 25, 2013, that the client did not have any real understanding of what MPP had done or what data had been collected. This representation turned out to be false. . . . Thus, the representation UMC’s current counsel made to the court that the client did not have any real idea of what prior counsel had done regarding ESI collection was patently false. In the light most favorable to current counsel, they did not ask the right questions of the individuals involved in the initial collection. The people involved in the process— MPP, its vendors and consultants, and the IT personnel at UMC who did the collection of ESI from 26 custodians—were simply not asked until after the special master was appointed and made the appropriate inquiries.

You do not see comments like that very often. Basically the judge is saying you lied to me and I cannot trust you. Again, more conscience shocking conduct by these attorneys, well outside the norm of accepted behavior.

Importance and Art of Custodian Interviews

The interviews that eventually were taken under the Special Master’s order and supervision show that critical evidence could have been saved from routine destruction, if the interviews been done at the time the suit was filed, not years later. The interviews would have ensured that preservation notices were properly given, understood and followed, and the right ESI was collected and effectively searched. See William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134, 136 (S.D.N.Y. 2009) (custodian interviews to assist also in keyword search formulation).

It is important to note that the custodian interviews in Small had to be done twice. The attorneys botched the first attempt at witness interviews. They were ordered to do it again. I am not surprised. Many people underestimate the complexity and sophistication of interviews in cases like this. They also underestimate the wiliness of custodians and tendency of some of them to evade questions.

It is very difficult for most attorneys to conduct an interview on the subject of information storage, IT systems, company document storage systems, email, texts, other personal messaging, social media, personal computers, phones, other devices and  software programs used. Questions on these subjects are very different from questions on the merits of a case. A good custodian interview requires special technical knowledge and skills, which, unfortunately, most lawyers still lack. Too bad, because witness interviews are so very important to big cases with complex, messy ESI systems.

Philip Favro, an expert consultant for Driven, Inc., makes this point well in his excellent article on Small:

Fulsome custodian interviews are essential for ensuring that relevant electronically stored information (ESI) is preserved. Such interviews are characterized by exhaustive questioning on any number of topics including traditional and newer sources of ESI.

Properly conducted, custodian interviews should provide counsel with a thorough understanding of the nature and types of relevant information at issue in the litigation, together with the sources where that information is located. If custodian interviews are neglected or deficient, parties are vulnerable to data loss and court sanctions. The Small v. University Medical Center case is instructive on these issues.

Vegas Court Spotlights the Importance of Custodian Interviews with New ESI Sources (LegalTech News 8/30/18).

Phil’s explanation of some of the facts behind the Special Master ordered redo of the interviews shows how difficult some custodian interviews can be, especially when they want to hide something from the lawyers:

Once conducted, the interviews were deemed insufficient by the special master and (later on) the court. In its order, the court spotlighted some of the evasive answers that UMC’s custodians provided. For example, UMC’s director of human resources disclosed the existence of only one relevant timekeeping application despite having approved the use of other timekeeping systems for certain employees. UMC argued that its HR director was only obligated to disclose the timekeeping application he actually used:

[The custodian] did not use those applications himself and therefore had no obligation to disclose these systems in custodian interviews ordered by the special master because a “custodian interview is aimed at uncovering the applications, systems, programs, data with which the actual custodian interfaces.” (emphasis added).

The court decried this limited notion of a custodian interview, observing that it failed to satisfy UMC’s “legal obligation to identify, locate, maintain, and produce relevant information.”

In Small they never did any custodian interviews until after the case blew up and a Special Master was appointed. Even when interviews were finally conducted by defense counsel, they did a poor job; they were not well-informed of the client IT systems and were not “tough enough” with the interviewees. They seemed to be easily deceived and accepted evasive, incomplete answers. You must cross-examine and be the devils advocate for effective interviews, especially when the custodian is evasive.

Favro recommends:

Interviews should go beyond cursory questioning and focus instead on identifying all sources of relevant information. Nor should they be limited to safe topics like “where can relevant messages be found in your email account” or “where are relevant documents stored on your laptop.” Interviews should now include questions regarding the existence of information exchanged through new communications media or stored in online locations . . .

There is an art to interviews like this. The witnesses have to be comfortable telling you the truth, the full truth, without fear of reprisals. Assurances of confidentiality and witness protection can be a good tongue loosener, but do not mislead them. Remind them who you represent, typically at the very beginning.

Trust, friendliness and rapport are important in interviews, but fear has its place too. I like to tell the witness up front how important it is for them to be fully truthful and candid. A short, but stern formal reminder can go a long way if delivered properly. Since interviews are usually not under oath this is especially important. Some formality is important as part of the tongue-loosening process. Moreover, interviews like this are typically done one-on-one with no court reporter and no written statement for the witness to read and sign at the end. An interview is just two people talking, one asking all of the questions, preferably face-to-face and preferably in the witnesses office with their computer equipment at the ready to show you something, if need be.

To encourage full honesty and to help get at the truth I also sometimes inform a witness that they will likely be deposed and subject to intense cross-exam by opposing counsel. (I might possibly exaggerate the adversaries capabilities from time to time.) I point out how it will all be under oath and penalty of perjury. Then I start my role of the devils advocate, saying these are the kind of questions you will be asked, and then tear into them and make sure the story is straight and the memory not too patchy. Hey, do not get mad at me for pressing on you; these are the kind of questions you can expect and we have to be prepared. That works. Fear can be a powerful motivator of truth. So can good cross-exam. The carrot and stick approach is usually effective.

Another important guardian of truth is for the questioning attorneys to be able to look the witness in the eye and follow exactly what they are saying; full technical understanding of the ESI questions. Do not speak the language? Too technical? Then bring a translator, an expert. Do not allow the witness to speak over your head. They may well be bs-ing you. Nodding your head at everything said, even when you do not understand, is a natural lawyer tendency that you must fight against. Do not be afraid to ask stupid questions. When it comes to technical interviews of any kind I interrupt and ask questions all of the time. Much of the language used in tech and e-discovery is vague and subject to multiple meanings. You need to ask questions. Only a fool is afraid to ask questions for fear of seeming foolish.

Good interviews are a best practice to start e-discovery off right and protect clients from wasted expense and unnecessary risks. See the fine article on point by Kelly Twigger, 5 Things A Great Custodian Interview Can Do For Your Case And Your Budget (Above The Law, 6/27/17).

Proper custodian interviews require skill and training. They require the attorney or paralegal doing the interview to have a basic understanding of technology, communications software and social media. It can be challenging in some situations and even advanced practitioners need a good detailed outline to do it right. Make sure your law firm or law department has a good ESI custodian interview outline. I suggest having both a short and long form. These help even experienced lawyers to make sure they do not forget to ask something.

Expert consultants like Kelly Twigger of ESI Attorneys can help you to prepare good outlines and other tools. They can also do the most challenging tasks for you, such as prepare custom Preservation Notices, conduct Custodian Interviews, supervise ESI Collection, attend the 26(f) conference and prepare an ESI discovery plan, and ultimately, document search, review and production. An e-discovery expert can make it far easier and less expensive to stay current with the many technical-legal issues in the field.

A custodian interview can provide a wealth of information to help lawyers to find and save important evidence, but only if done properly by skilled legal practitioners. Do not risk the judge ordering a redo. Make sure you do a proper interview of the key custodians as soon as possible

Conclusion

Small shows what can happen when you take a cavalier attitude towards ESI preservation and interviews. Small v. University Medical CenterCase No. 2:13-cv-0298-APG-PAL, Report and Recommendation and Final Findings of Fact and Conclusions of Law dated August 9, 2018. Preservation errors at the beginning of a case can easily cascade into serious negligence and ESI destruction. This often results in sanctions motions and discovery about discovery. That diverts everyone from the merits of the case. In Small the sanctions not only included a permissive inference jury instruction, but also monetary sanctions, amount yet to be determined. What happened to the defendant in federal court in Vegas in Small is something that you should fear and loathe ever happening to you.

Proper timely custodian interviews could have prevented the loss of data in Small, could have prevented any sanctions. We all know that what happens in Vegas does not stay in Vegas, at least not when discovery in a law suit is concerned. The truth will come out as it should. This is especially true in a case like Small with misconduct that shocks the conscience in a mockery of justice, as Special Master Dan Garrie put it back in 2014.

Early custodian Interviews are an important, well-accepted best-practice, especially in a large matter like Small v. UMC. Interviews are the third step in the ten-step best practices of Electronic Discovery shown below. Electronic Discovery Best Practices (EDBP.com). They are one of three important activities that attorneys must perform in every law suit to preserve potential electronic evidence (shown in blue in the diagram below): hold notices, interviews and ESI collections.

See EDBP on Preservation.

In a large firm like mine, which only does Labor and Employment law, you can use one of the specialists in e-discovery to assist in these tasks, at least until you become proficient on your own. Specialists in large firms are usually experienced attorneys that now limit their work to e-discovery. (I recommend against specializing too early, but some are able to do it effectively.) In my firm there is only one full-time specialist, me, but I have over fifty attorney liaisons to assist. They have special training in e-discovery and are the go-to e-discovery lawyers for their office (we have 50), but they spend most of their time in employment litigation and other services outside of e-discovery. Other large firms have more full-time e-discovery specialists, but fewer part-time specialists. I decided to try to spread out the knowledge.

One of the things a specialists do, full or part-time, is help to create and update good standard witness interview question outlines for use by other attorneys in the firm. For instance, I have both a long and short form that I recently updated. Your firm probably has something similar. If not, do it now. Better late than never.

If you are in a smaller firm and do not have a full-time specialist in your ranks, then you should consider retaining an outside specialist as co-counsel in larger e-discovery matters. They can help you to save on overall costs and, most importantly, prevent a disaster like Small v. University Medical Center from ever darkening your door.

 

 

 


Elusion Random Sample Test Ordered Under Rule 26(g) in a Keyword Search Based Discovery Plan

August 26, 2018

There is a new case out of Chicago that advances the jurisprudence of my sub-specialty, Legal Search. City of Rockford v. Mallinckrodt ARD Inc., 2018 WL 3766673, Case 3:17-cv-50107 (N.D. Ill., Aug. 7, 2018). This discovery order was written by U.S. Magistrate Judge Iain Johnston who entitled it: “Order Establishing Production Protocol for Electronically Stored Information.” The opinion is both advanced and humorous, destined to be an oft-cited favorite for many. Thank you Judge Johnston.

In City of Rockford an Elusion random sample quality assurance test was required as part of the parties discovery plan to meet the reasonable efforts requirements of Rule 26(g). The random sample procedure proposed was found to impose only a proportional, reasonable burden under Rule 26(b)(1). What makes this holding particularly interesting is that an Elusion test is commonly employed in predictive coding projects, but here the parties had agreed to a keyword search based discovery plan. Also see: Tara Emory, PMP, Court Holds that Math Matters for eDiscovery Keyword Search,  Urges Lawyers to Abandon their Fear of Technology (Driven, (August 16, 2018) (“party using keywords was required to test the search effectiveness by sampling the set of documents that did not contain the keywords.”)

The Known Unknowns and Unknown Unknowns

Judge Johnston begins his order in City of Rockford with a famous quote by Donald Rumseld, a two-time Secretary of Defense.

“[A]s we know there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. . .”
Donald Rumseld

For those not familiar with this famous Known Knowns quip, here is a video of the original:

Here the knowledge logic is spelled out in a chart, since I know we all love that sort of thing. Deconstructing Rumsfeld: Knowledge and Ignorance in the Age of Innovation (Inovo 5/114).

Anybody who does complex investigations is familiar with this problem. Indeed, you can argue this insight is fundamental to all of science and experimental method. Logan, David C. (March 1, 2009). “Known knowns, known unknowns, unknown unknowns and the propagation of scientific enquiry”, Journal of Experimental Botany 60 (3). pp. 712–4. [I have always wanted to quote a botany journal.]

How do you deal with the known unknowns and the unknown unknowns, the information that we don’t even know that we don’t know about? The deep, hidden information that is both obtuse and rare. Information that is hard to retrieve and harder still to prove does not exist at all. Are you chasing something that might not exist? Something unknown because nonexistent? Such as an overlooked Highly Relevant document? (The stuff of nightmares!) Are you searching for nothing? Zero? If you find it, what does that mean? What can be known and what can never be known? Scientists, investigators and the Secretary of Defense alike all have to ponder these questions and all want to use the best tools and best people possible to do so. See: Deconstructing Rumsfeld: Knowledge and Ignorance in the Age of Innovation (Inovo 5/114).

Seeking Knowledge of the Unknown Elusion Error Rate

These big questions, though interesting, are not why Judge Johnston started his opinion with the Rumseld quote. Instead, he used the quote to emphasize that new e-discovery methods, namely random sampling and statistical analysis, can empower lawyers to know what they never did before. A technical way to know the known unknowns. For instance, a way to know the number of relevant documents that will be missed and not produced: the documents that elude retrieval.

As the opinion and this blog will explain, you can do that, know that, by using an Elusion random sample of the null-set. The statistical analysis of the sample transforms the unknown quantity to a known (subject to statistical probabilities and range). It allows lawyers to know, at least within a range, the number of relevant documents that have not been found. This is a very useful quality assurance method that relies on objective measurements to demonstrate success of your project, which here is information retrieval. This and other random sampling methods allow for the calculation of Recall, meaning the percent of total relevant documents found. This is another math-based, quality assurance tool in the field of information retrieval.

One of the main points Judge Johnston makes in his order is that lawyers should embrace this kind of technical knowledge, not shy away from it. As Tara Emory said in her article, Court Holds that Math Matters for eDiscovery Keyword Search:

A producing party must determine that its search process was reasonable. In many cases, the best way to do this is with objective metrics. Producing parties often put significant effort into brainstorming keywords, interviewing witnesses to determine additional terms, negotiating terms with the other party, and testing the documents containing their keywords to eliminate false positives. However, these efforts often still fail to identify documents if important keywords were missed, and sampling the null set is a simple, reasonable way to test whether additional keywords are needed. …

It is important to overcome the fear of technology and its related jargon, which can help counsel demonstrate the reasonableness of search and production process. As Judge Johnston explains, sampling the null set is a process to determine “the known unknown,” which “is the number of the documents that will be missed and not produced.” Judge Johnson disagreed with the defendants’ argument “that searching the null set would be costly and burdensome.” The Order requires Defendants to sample their null set at a 95% +/-2% margin of error (which, even for a very large set of documents, would be about 2,400 documents to review).[4] By taking these measures—either with TAR or with search terms, counsel can more appropriately represent that they have undertaken a “reasonable inquiry” for relevant information within the meaning of FRCP 26(g)(1).

Small Discovery Dispute in an Ocean of Cooperation

Judge Johnston was not asked to solve the deep mysteries of knowing and not knowing in City of Rockford. The parties came to him instead with an interesting, esoteric discovery dispute. They had agreed on a great number of things, for which the court profusely congratulated them.

The attorneys are commended for this cooperation, and their clients should appreciate their efforts in this regard. The Court certainly does. The litigation so far is a solid example that zealous advocacy is not necessarily incompatible with cooperation. The current issue before the Court is an example of that advocacy and cooperation. The parties have worked to develop a protocol for the production of ESI in this case, but have now reached an impasse as to one aspect of the protocol.

The parties disagreed on whether to include a document review quality assurance test in the protocol. The Plaintiffs wanted one and the Defendants did not. Too burdensome they said.

To be specific, the Plaintiffs wanted a test where the efficacy of any parties production would be tested by use of an Elusion type of Random Sample of the documents not produced. The Defendants opposed any specific test. Instead, they wanted the discovery protocol to say that if the receiving party had concerns about the adequacy of the producing party’s efforts, then they would have a conference to address the concerns.

Judge Johnston ruled for the plaintiff in this dispute and ordered a  random elusion sample to be taken after the defendant stopped work and completed production. In this case it was a good decision, but should not be routinely required in all matters.

The Stop Decision and Elusion Sample

One of the fundamental problems in any investigation is to know when you should stop the investigation because it is no longer worth the effort to carry on. When has a reasonable effort been completed? Ideally this happens after all of the important documents have already been found. At that point you should stop the effort and move on to a new project. Alternatively, perhaps you should keep on going and look for more? Should you stop or not?

In Legal Search we all this the “Stop Decision.” Should you conclude the investigation or continue further AI training rounds and other search. As explained in the e-Discovery Team TAR Course:

The all important stop decision is a legal, statistical decision requiring a holistic approach, including metrics, sampling and over-all project assessment.You decide to stop the review after weighing a multitude of considerations. Then you test your decision with a random sample in Step Seven.

See: TAR Course: 15th Class – Step Seven – ZEN Quality Assurance Tests.

If you want to go deeper into this, then listen in on this TAR Course lecture on the Stop decision.

____________

Once a decision is made to Stop, then a well managed document review project will use different tools and metrics to verify that the Stop decision was correct. Judge Johnston in City of Rockford used one of my favorite tools, the Elusion random sample that I teach in the e-Discovery Team TAR Course. This type of random sample is called an Elusion sample.

Judge Johnston ordered an Elusion type random sample of the null set in City of Rockford. The sample would determine the range of relevant documents that likely eluded you. These are called False Negatives. Documents presumed Irrelevant and withheld that were in fact Relevant and should have been produced. The Elusion sample is designed to give you information on the total number of Relevant documents that were likely missed, unretrieved, unreviewed and not produced or logged. The fewer the number of False Negatives the better the Recall of True Positives. The goal is to find, to retrieve, all of the Relevant ESI in the collection.

Another way to say the same thing is to say that the goal is Zero False Negatives. You do not miss a single relevant file. Every file designated Irrelevant is in fact not relevant. They are all True Negatives. That would be Total Recall: “the Truth, the Whole Truth …” But that is very rare and some error, some False Negatives, are expected in every large information retrieval project. Some relevant documents will almost always be missed, so the goal is to make the False Negatives inconsequential and keep the Elusion rate low.

Here is how Judge Iain Johnston explained the random sample:

Plaintiffs propose a random sample of the null set. (The “null set” is the set of documents that are not returned as responsive by a search process, or that are identified as not relevant by a review process. See Maura R. Grossman & Gordon v. Cormack, The Grossman-Cormack Glossary of Technology-Assisted Review, 7 Fed. Cts. L. Rev. 1, 25 (2013). The null set can be used to determine “elusion,” which is the fraction of documents identified as non-relevant by a search or review effort that are, in fact, relevant. Elusion is estimated by taking a random sample of the null set and determining how many or what portion of documents are actually relevant. Id. at 15.) FN 2

Judge Johnston’s Footnote Two is interesting for two reasons. One, it attempts to calm lawyers who freak out when hearing anything having to do with math or statistics, much less information science and technology. Two, it does so with a reference to Fizbo the clown.

The Court pauses here for a moment to calm down litigators less familiar with ESI. (You know who you are.) In life, there are many things to be scared of, including, but not limited to, spiders, sharks, and clowns – definitely clowns , even Fizbo. ESI is not something to be scared of. The same is true for all the terms and jargon related to ESI. … So don’t freak out.

Accept on Zero Error for Hot Documents

Although this is not addressed in the court order, in my personal view, no False Negatives, iw – overlooked  documents – are acceptable when it comes to Highly Relevant documents. If even one document like that is found in the sample, one Highly Relevant Document, then the Elusion test has failed in my view. You must conclude that the Stop decision was wrong and training and document review must recommence. That is called an Accept on Zero Error test for any hot documents found. Of course my personal views on best practice here assume the use of AI ranking, and the parties in City of Rockford only used keyword search. Apparently they were not doing machine training at all.

The odds of finding False Negatives, assuming that only a few exist (very low prevalence) and the database is large, are very unlikely in a modest sized random sample. With very low prevalence of relevant ESI the test can be of limited effectiveness. That is an inherent problem with low prevalence and random sampling. That is why statistics have only limited effectiveness and should be considered part of a total quality control program. See Zero Error Numerics: ZEN. Math matters, but so too does good project management and communications.

The inherent problem with random sampling is that the only way to reduce the error interval is to increase the size of the sample. For instance, to decrease the margin of error to only 2% either way, a total error of 4%, a random sample size of around 2,400 documents is needed. Even though that narrows the error rate to 4%, there is still another error factor of the Confidence Level, here at 95%. Still, it is not worth the effort to review even more sample documents to reduce that to a 99% Level.

Random sampling has limitations in low prevalence datasets, which is typical in e-discovery, but still sampling can be very useful. Due to this rarity issue, and the care that producing parties always take to attain high Recall, any documents found in an Elusion random sample should be carefully studied to see if they are of any significance. We look very carefully at any new documents found that are of a kind not seen before. That is unusual. Typically  any relevant documents found by random sample of the elusion set are of a type that have been seen before, often many, many times before. These “same old, same old” type of documents are of no importance to the investigation at this point.

Most email related datasets are filled with duplicative, low value data. It is not exactly irrelevant noise, but it is not a helpful signal either. We do not care if we  get all of that kind of merely relevant data. What we really want are the Hot Docs, the high value Highly Relevant ESI, or at least Relevant and of a kind not seen before. That is why the Accept On Zero Error test is so important for Highly Relevant documents.

The Elusion Test in City of Rockford 

In City of Rockford Judge Johnston considered a discovery stipulation where the parties had agreed to use a typical keyword search protocol, but disagreed on a quality assurance protocol. Judge Johnston held:

With key word searching (as with any retrieval process), without doubt, relevant documents will be produced, and without doubt, some relevant documents will be missed and not produced. That is a known known. The known unknown is the number of the documents that will be missed and not produced.

Back to the False Negatives again, the known unknown. Judge Johnston continues his analysis:

But there is a process by which to determine that answer, thereby making the known unknown a known known. That process is to randomly sample the nullset. Karl Schieneman & Thomas C. Gricks III, The Implications of Rule26(g) on the Use of Technology-Assisted Review, 2013 Fed. Cts. L. Rev. 239, 273 (2013)(“[S]ampling the null set will establish the number of relevant documents that are not being produced.”). Consequently, the question becomes whether sampling the null set is a reasonable inquiry under Rule 26(g) and proportional to the needs of this case under Rule 26(b)(1).

Rule 26(g) Certification
Judge Johnston takes an expansive view of the duties placed on counsel of record by Rule 26(g), but concedes that perfection is not required:

Federal Rule of Civil Procedure 26(g) requires all discovery requests be signed by at least one attorney (or party, if proceeding pro se). Fed. R. Civ. P. 26(g)(1). By signing the response, the attorney is certifying that to the best of counsel’s knowledge, information, and belief formed after a reasonable inquiry, the disclosure is complete and correct at the time it was made. Fed. R. Civ. P. 26(g)(1)(A). But disclosure of documents need not be perfect. … If the Federal Rules of Civil Procedure were previously only translucent on this point, it should now be clear with the renewed emphasis on proportionality.

Judge Johnston concludes that Rule 26(g) on certification applies to require the Elusion sample in this case.

Just as it is used in TAR, a random sample of the null set provides validation and quality assurance of the document production when performing key word searches.  Magistrate Judge Andrew Peck made this point nearly a decade ago. See William A. Gross Constr. Assocs., 256 F.R.D. at 135-6 (citing Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 262 (D. Md. 2008)); In re Seroquel Products Liability Litig., 244 F.R.D. 650, 662 (M.D. Fla. 2007) (requiring quality assurance).

Accordingly, because a random sample of the null set will help validate the document production in this case, the process is reasonable under Rule 26(g).

Rule 26(b)(1) Proportionality

Judge Johnston considered as a separate issue whether it was proportionate under Rule 26(b)(1) to require the elusion test requested. Again, the court found that it was in this large case on the pricing of prescription medication and held that it was proportional:

The Court’s experience and understanding is that a random sample of the null set will not be unreasonably expensive or burdensome. Moreover and critically, Defendants have failed to provide any evidence to support their contention. Mckinney/Pearl Rest. Partners, L.P. v. Metro. Life Ins. Co., 322 F.R.D. 235, 242 (N.D.Tex. 2016) (party required to submit affidavits or offer evidence revealing the nature of the burden)
Once again we see a party seeking protection from having to do something because it is so burdensome then failing to present actual evidence of burden. We see this a lot lately. Responding Party’s Complaints of Financial Burden of Document Review Were Unsupported by the Evidence, Any Evidence (e-Discovery Team, 8/5/18);

Judge Johnston concludes his “Order Establishing Production Protocol for Electronically Stored Information” with the following:

The Court adopts the parties’ proposed order establishing the production protocol for ESI with the inclusion of Plaintiffs’ proposal that a random sample of the null set will occur after the production and that any responsive documents found as a result of that process will be produced. Moreover, following that production, the parties should discuss what additional actions, if any, should occur. If the parties cannot agree at that point, they can raise the issue with the Court.

Conclusion

City of Rockford is important because it is the first case to hold that a quality control procedure should be used to meet the reasonable efforts certification requirements of Rule 26(g). The procedure here required was a random sample Elusion test with related, limited data sharing. If this interpretation of Rule 26(g) is followed by other courts, then it could have a big impact on legal search jurisprudence. Tara Emory in her article, Court Holds that Math Matters for eDiscovery Keyword Search goes so far as to conclude that City of Rockford stands for the proposition that “the testing and sampling process associated with search terms is essential for establishing the reasonableness of a search under FRCP 26(g).”

The City of Rockford holding could persuade other judges and encourage courts to be more active and impose specific document review procedures on all parties, including requiring the use of sampling and artificial intelligence. The producing party cannot always have a  free pass under Sedona Principle Six. Testing and sampling may well be routinely ordered in all “large” document review cases in the future.

It will be very interesting to watch how other attorneys argue City of Rockford. It will continue a line of cases examining methodology and procedures in document review. See eg., William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134 (S.D.N.Y. 2009) (“wake-up call” for lawyers on keyword search); Winfield v. City of New York (SDNY, Nov. 27, 2017), where Judge Andrew Peck considers methodologies and quality controls of the active machine learning process. Also see Special Master Maura Grossman’s Order Regarding Search Methodology for ESI, a validation Protocol for the Broiler Chicken antitrust cases.

The validation procedure of an Elusion sample in City of Rockford is just one of many possible review protocols that a court could impose under Rule 26(g). There are dozens more, including whether predictive coding should be required. So far, courts have been reluctant to order that, as Judge Peck explained in Hyles:

There may come a time when TAR is so widely used that it might be unreasonable for a party to decline to use TAR. We are not there yet.

Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP), 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016):

Like a kid in the backseat of the car, I cannot help but ask, are we there yet? Hyles was published over two years ago now. Maybe some court, somewhere in the world, has already ordered a party to do predictive coding against their will, but not to our knowledge. That is a known unknown. Still, we are closer to “There” with the City of Rockford’s requirement of an Elusion test.

When we get “there,” and TAR is finally ordered in a case, it will probably arise in a situation like City of Rockford where a joint protocol applicable to all parties is involved. That is easier to sell than a one-sided protocol. The court is likely to justify the order by Rule 26(g), and hold that it requires all parties in the case to use predictive coding. Otherwise, they will not meet the  reasonable effort burdens of Rule 26(g). Other rules will be cited too, of course, including Rule 1, but Rule 26(g) is likley to be key.

____________

___

 

____

 

 


%d bloggers like this: