Cautionary Tale from Brooklyn: Search Terms ‘Designed To Fail’

October 20, 2019

Every lawyer who thinks e-discovery is not important, that you can just delegate it to a vendor, should read Abbott Laboratories, et al. v. Adelphia Supply USA, et al., No. 15 CV 5826 (CBA) (LB) (E.D.N.Y. May 2, 2019). This opinion in a trademark case in Brooklyn District Court (shown here) emphasizes, once again, that e-discovery can be outcome-determinative. If you mess it up, you can doom your case. If a lawyer wants to litigate today, they either have to spend the substantial time it takes to learn the many intricacies of e-discovery, or associate with a specialist who does. The Abbott Labs case shows how easily a law suit can be won or lost on e-discovery alone. Here the numbers did not add up, key custodians were omitted and guessed keywords were used, keywords so bad that opposing counsel called them designed to fail. The defendants reacted by firing their lawyers and blaming everything on them, but the court did not buy it. Instead, discovery fraud was found and judgment was entered for the plaintiff.

Magistrate Judge Lois Bloom (shown right) begins the Opinion by noting that the plaintiff’s motion for case ending sanctions “… presents a cautionary tale about how not to conduct discovery in federal court.” The issues started when defendant made its first electronic document production. The Electronically Stored Information was all produced in paper, as Judge Bloom explained “in hard copy, scanning them all together, and producing them as a single, 1941-page PDF file.” Opinion pg. 3. This is not what the plaintiff Abbott Labs wanted. After Abbott sought relief from the court the defendants on March 24, 2017 were ordered  to “produce an electronic copy of the 2014 emails (1,941 pages)” including metadata. Defendant then “electronically produced 4,074 pages of responsive documents on April 5, 2017.” Note how the page count went from 1,942 to 4,074. There was no explanation of this page count discrepancy, the first of many, but the evidence helped Abbott justify a new product counterfeiting action (Abbott II) where the court ordered a seizure of defendant’s email server. That’s were the fun started. As Judge Bloom put it:

Once plaintiffs had seized H&H’s email server, plaintiffs had the proverbial smoking gun and raised its concerns anew that defendants had failed to comply with the Court’s Order to produce responsive documents in the instant action (hereinafter “Abbott I”). On July 12, 2017, the Court ordered the H&H defendants to “re-run the document search outlined in the Court’s January 17 and January 21 Orders,” “produce the documents from the re-run search to Abbott,” and to produce “an affidavit of someone with personal knowledge” regarding alleged technical errors that affected the production.³ Pursuant to the Court’s July 12, 2017 Order to re-run the search, The H&H defendants produced 3,569 responsive documents.

Opinion pg. 4 (citations to record omitted).

Too Late For Vendor Help and a Search Strategy Designed to Fail

After the seizure order in Abbott II, and after Abbott Labs again raised issues regarding defendants’ original production, Judge Bloom ordered the defendants to re-run the original search. Defendants then retained the services of an outside vendor, Transperfect, to re-run the original search for them. In supposed compliance with that order, the defendants, aka H&H, then produced 3,569 documents. Id. at 8. Defendants also filed an affidavit by Joseph Pochron, Director in the Forensic Technology and Consulting Division at Transperfect (“Pochron Decl.”) to try to help their case. It did not work. According to Judge Bloom the Pochron Decl. states:

… that H&H utilized an email archiving system called Barracuda and that there are two types of Barracuda accounts, Administrator and Auditor. Pochron Decl. ¶ 13. Pochron’s declaration states that the H&H employee who ran the original search, Andrew Sweet, H&H’s general manager, used the Auditor account to run the original search (“Sweet search”). Id. at ¶ 19. When Mr. Pochron replicated the Sweet search using the Auditor account, he obtained 1,540 responsive emails. Id. at ¶ 22. When Mr. Pochron replicated the Sweet search using the Administrator account, he obtained 1,737 responsive emails. Id. Thus, Mr. Pochron attests that 197 messages were not viewable to Mr. Sweet when the original production was made. Id. Plaintiffs state that they have excluded those 197 messages, deemed technical errors, from their instant motion for sanctions. Plaintiffs’ Memorandum of Law at 9; Waters Decl. ¶ 8. However, even when those 197 messages are excluded, defendants’ numbers do not add up. In fact, H&H has repeatedly given plaintiffs and the Court different numbers that do not add up.

Moreover, plaintiffs argue that the H&H defendants purposely used search terms designed to fail, such as “International” and “FreeStyle,” whereas H&H’s internal systems used item numbers and other abbreviations such as “INT” and “INTE” for International and “FRL” and “FSL” for FreeStyle. Plaintiff’s Memorandum of Law at 10–11. Plaintiffs posit that defendants purposely designed and ran the “extremely limited search” which they knew would fail to capture responsive documents …

Opinion pgs. 8-9 (emphasis by bold added). “Search terms designed to fail.” This is the first time I have ever seen such a phrase in a judicial opinion. Is purposefully stupid keyword search yet another bad faith litigation tactic by unscrupulous attorneys and litigants? Or is this just another example of dangerous incompetence? Judge Bloom was not buying the ‘big oops” theory, especially considering the ever-changing numbers of relevant documents found. It looked to her, and me too, that this search strategy was intentionally design to fail, that it was all a shell-game.

This is the wake-up call for all litigators, especially those who do not specialize in e-discovery. Your search strategy had better make sense. Search terms must be designed (and tested) to succeed, not fail! This is not just incompetence.

The Thin Line Between Gross Negligence and Bad Faith

The e-discovery searches you run are important. The “mistakes” made here led to a default judgment. That is the way it is in federal court today. If you think otherwise, that e-discovery is not that important, that you can just hire a vendor and throw stupid keywords at it, then your head is dangerously stuck in the sand. Look around. There are many cases like Abbott Laboratories, et al. v. Adelphia Supply USA.

I say “mistakes” made here in quotes because it was obvious to Judge Bloom that these were not mistakes at all, this was fraud on the court.

E-Discovery is about evidence. About truth. You cannot play games. Either take it seriously and do it right, do it ethically, do it competently; or go home and get out. Retire already. Discovery gamesmanship and lawyer bumbling are no longer tolerated in federal court. The legal profession has no room for dinosaurs like that.

Abbott Labs responded the way they should, the way you should always expect in a situation like this:

Plaintiffs move for case ending sanctions under Federal Rule of Civil Procedure 37 and invoke the Court’s inherent power to hold defendants in default for perpetrating a fraud upon the Court. Plaintiffs move to strike the H&H defendants’ pleadings, to enter a default judgment against them, and for an order directing defendants to pay plaintiffs’ attorney’s fees and costs, for investigating and litigating defendants’ discovery fraud.

Id.

Rule 37(e) was revised in 2015 to make clear that gross negligence alone does not justify a case-ending sanction, that you must prove bad faith. This change should not provide the incompetent with much comfort. As this case shows, the difference between mistake and intent can be a very thin line. Do your numbers add up? Can you explain what you did and why you did it? Did you use good search terms? Did you search all of the key custodians? Or did you just take the ESI the client handed to you and say thank you very much? Did you look with a blind eye? Even if bad faith under Rule 37 is not proven, the court may still find the whole process stinks of fraud and use the court’s inherent powers to sanction misconduct.

As Judge Bloom went on to explain:

Under Rule 37, plaintiffs’ request for sanctions would be limited to my January 17, 2017 and January 27, 2017 Orders which directed defendants to produce documents as set forth therein. While sanctions under Rule 37 would be proper under these circumstances, defendants’ misconduct herein is more egregious and goes well beyond defendants’ failure to comply with the Court’s January 2017 discovery orders. . . .  Rather than viewing the H&H defendants’ failure to comply with the Court’s January 2017 Orders in isolation, plaintiffs’ motion is more properly considered in the context of the Court’s broader inherent power, because such power “extends to a full range of litigation abuses,” most importantly, to fraud upon the court.

Opinion pg. 5.

Judge Bloom went on the explain further the “fraud on the court” and defendant’s e-discovery conduct.

A fraud upon the court occurs where it is established by clear and convincing evidence “that a party has set in motion some unconscionable scheme calculated to interfere with the judicial system’s ability impartially to adjudicate a matter by . . . unfairly hampering the presentation of the opposing party’s claim or defense.” New York Credit & Fin. Mgmt. Grp. v. Parson Ctr. Pharmacy, Inc., 432 Fed. Appx. 25 (2d Cir. 2011) (summary order) (quoting Scholastic, Inc. v. Stouffer, 221 F. Supp. 2d 425, 439 (S.D.N.Y. 2002))

Opinion pgs. 5-6 (subsequent string cites omitted).

Kill All The Lawyers

The defendants here tried to defend by firing and blaming their lawyers. That kind of Shakespearean sentiment is what you should expect when you represent people like that. They will turn on you. They will use you for their nefarious ends, then lose you. Kill you if they could.

Judge Bloom, who was herself a lawyer before becoming a judge, explained the blame-game defendants tried to pull in her court.

Regarding plaintiffs’ assertion that defendants designed and used search terms to fail, defendants proffer that their former counsel, Mr. Yert, formulated and directed the use of the search terms. Id. at 15. The H&H defendants state that “any problems with the search terms was the result of H&H’s good faith reliance on counsel who . . . decided to use parameters that were less robust than those later used[.]” Id. at 18. The H&H defendants further state that the Sweet search results were limited because of Mr. Yert’s incompetence. Id.

Opinion pg. 9.

Specifically defendants alleged:

… the original search parameters were determined by Mr. Yert and that he “relied on Mr. Yert’s expertise as counsel to direct the parameters and methods for a proper search that would fulfill the Court’s Order.” Sweet Decl. ¶ 3–4.  As will be discussed below, the crux of defendants’ arguments throughout their opposition to the instant motion seeks to lay blame on Mr. Yert for their actions; however, defendants cannot absolve themselves of liability here by shifting blame to their former counsel.

Opinion pg. 11.

Here is how Judge Bloom responded to this “blame the lawyers” defense:

Defendants’ attempt to lay blame on former counsel regarding the design and use of search terms is equally unavailing. It is undisputed that numerous responsive documents were not produced by the H&H defendants that should have been produced. Defendants’ prior counsel conceded as much. See generally plaintiffs’ Ex. B, Tr. Of July 11, 2017 telephone conference.

Mr. Yert was asked at his deposition about the terms that H&H used to identify their products and he testified as follows:

Q. Tell me about the general discussions you had with the client in terms of what informed you what search terms you should be using.

A. Those were the terms consistently used by H&H to identify the particular product.

Q. So the client told you that FreeStyle and International are the terms they consistently used to refer to International FreeStyle test strips; is that correct?

A. That’s what I recall.

Q. Did the client tell you that they used the abbreviation FSL to refer to FreeStyle?

A. I don’t recall.

Q. If they had told you that, you would have included that as a search term, correct?

A. I don’t recall if it was or was not included as a search term, sir.

Opinion pgs. 10-11.

The next time you are asked to dream up keywords for searches to find your client’s relevant evidence, remember this case, remember this deposition. Do not simply use keywords that the client suggests, as the attorneys did here. Do not simply use keywords. As I have written here many, many times before, there is a lot more to electronic evidence search and review than keywords. This is the Twenty First Century. You should be using AI, specifically active machine learning, aka Predictive Coding.

You need an expert to help you and you need them at the start of a case, not after sanctions motions.

Judge Lois Bloom went on to explain that, even if defendant’s story of innocent reliance on it lawyers was true:

It has long been held that a client-principal is “bound by the acts of his lawyer agent.” Id. (quoting Link v. Wabash RR. Co., 370 U.S. 626, 634 (1962)). As the Second Circuit stated, “even innocent clients may not benefit from the fraud of their attorney.” Id. . . .

However, notwithstanding defendants’ assertion that the search terms “FreeStyle” and “International” were used in lieu of more comprehensive search terms at the behest of Mr. Yert, it is undisputed that Mr. Sweet, H&H’s general manager, knew that H&H used abbreviations for these terms. Mr. Sweet admitted this at his deposition. See Sweet Dep. 81:2-81:24, Mar. 13, 2018. . . . The Court need not speculate as to why defendants did not use these search terms to comply with defendants’ obligation to produce pursuant to the Court’s Order. Mr. Sweet, by his own admission, states that “on several occasions he contacted Mr. Yert with specific questions about whether to include certain emails in production.” Sweet Decl. ¶ 7. It is inconceivable that H&H’s General Manager, who worked closely with Mr. Yert to respond to the Court’s Order, never mentioned that spelling out the terms used, “International” and “FreeStyle”, would not capture the documents in H&H’s email system. Mr. Sweet knew that H&H was required to produce documents regarding International FreeStyle test strips, regardless of whether H&H’s documents spelled out or abbreviated the terms. Had plaintiffs not seized H&H’s email server in the counterfeiting action, plaintiffs would have never known that defendants failed to produce a trove of responsive documents. H&H would have gotten away with it.

Opinion pgs. 12-13.

Defendants also failed to produce any documents by three custodians Holland Trading, Howard Goldman, and Lori Goldman. Again, they tried to blame that omission on their attorney, who they claim directed the search. Oh yeah, for sure. To me he looks like a mere stooge, a tool of unscrupulous litigants. Judge Bloom did not accept that defense either, holding:

While defendants’ effort to shift blame to Mr. Yert is unconvincing at best, even if defendants’ effort could be credited, counsel’s actions, even if they were found to be negligent, would not shield the H&H defendants from responsibility for their bad faith conduct.

Opinion pgs. 19-20. Then Judge Bloom went on to cite the record at length, including the depositions and affidavits of the attorneys involved, to expose this blame game as a sham. The order then concludes on this point holding:

There is no credible explanation for why the Holland Trading, Howard Goldman, and Lori Goldman documents were not produced except that the documents were willfully withheld. Defendants’ explanation that there were no documents withheld, then that any documents that weren’t produced were due to technical glitches, then that the documents didn’t appear in Mr. Sweet’s original search, then that if documents were intentionally removed, they were removed per Mr. Yert’s instructions cannot all be true. The H&H defendants have always had one more excuse up their sleeve in this “series of episodes of nonfeasance,” which amounts to “deliberate tactical intransigence.” Cine, 602 F.2d at 1067. In light of the H&H defendants’ ever-changing explanations as to the withheld documents, Mr. Sweet’s inconsistent testimony, and assertions of former counsel, the Court finds that the H&H defendants have calculatedly attempted to manipulate the judicial process. See Penthouse, 663 F.2d 376–390 (affirming entry of default where plaintiffs disobeyed an “order to produce in full all of [their] financial statements,” engaged in “prolonged and vexatious obstruction of discovery with respect to closely related and highly relevant records,” and gave “false testimony and representations that [financial records] did not exist.”).

Opinion pgs. 22-23.

The plaintiff, Abbott Labs, went on to argue that “the withheld documents freed David Gulas to commit perjury at his deposition. The Court agrees.” Id. at 24. The Truth has a way of finding itself out, especially with competent counsel on the other side and a good judge.

With this evidence the Court concluded the only adequate sanction was a default judgment in plaintiff’s favor. Message to spoliating defendants, game over, you lose.

Based on the full record of the case, there is clear and convincing evidence that defendants have perpetrated a fraud upon the court. Defendants’ initial conduct of formulating search terms designed to fail in deliberate disregard of the lawful orders of the Court allowed H&H to purposely withhold responsive documents, including the Holland Trading, Howard Goldman, and Lori Goldman documents. Defendants proffered inconsistent positions with three successive counsel as to why the documents were withheld. Mr. Sweet’s testimony is clearly inconsistent if not perjured from his deposition to his declaration in opposition to the instant motion. Mr. Goldman’s deposition testimony is evasive and self-serving at best. Finally, Mr. Gulas’ deposition testimony is clearly perjured. Had plaintiffs never seized H&H’s server pursuant to the Court’s Order in the counterfeiting case, H&H would have gotten away with their fraud upon this Court. H&H only complied with the Court’s orders and their discovery obligations when their backs were against the wall. Their email server had been seized. There was no longer an escape from responsibility for their bad faith conduct. This is, again, similar to Cerruti, where the “defendants did not withdraw the [false] documents on their own. Rather, they waited until the falsity of the documents had been detected.” Cerruti.,169 F.R.D. at 583. But for being caught in a web of irrefutable evidence, H&H would have profited from their misconduct. . . .

The Court finds that the H&H defendants have committed a fraud upon the court, and that the harshest sanction is warranted. Therefore, plaintiffs’ motion for sanctions should be granted and a default judgment should be entered against H&H Wholesale Services, Inc., Howard Goldman, and Lori Goldman.

Conclusion

Attorneys of record sign responses under Rule 26(g) to requests for production, not the client. That is because the rules require them to control the discovery efforts of their clients. That means the attorney’s neck is on the line. Rule 26(g) does not allow you to just take a client’s word for it. Verify. Supervise. The numbers should add up. The search terms, if used, should be designed and tested to succeed, not fail. This is your response, not the client’s. You determine the search method, in consultation with the client for sure, but not by “just following orders.” You must see everything, not nothing. If you see no email from key custodians, dig deeper and ask why. Do this at the beginning of the case. Get vendor help before you start discovery, not after you fail. Apparently the original defense attorneys here did just what they were asked, they went along with the client. Look where it got them. Fired and deposed. Default judgment entered. Cautionary tale indeed.

 

 


Transparency in a Salt Lake TAR Pit?

November 11, 2018

A Salt Lake City Court braved the TAR pits to decide a “transparency” issue. Entrata, Inc. v. Yardi Systems, Inc., Case No. 2:15-cv-00102 (D.C. Utah, 10/29/18). The results were predictable for TAR, which is usually dark. The requesting party tried to compel the respondent to explain their TAR. Tried to force them to disclose the hidden metrics of Recall and Richness. The motion was too little, too late and was denied. The TAR pits of Entrata remain dark. Maybe TAR was done well, maybe not. For all we know the TAR was done by Sponge Bob Square Pants using Bikini Bottom software. We may never know.

Due to the Sponge Bobby type motion leading to an inevitable denial, the requesting party, Yardi Systems, Inc., remains in dark TAR. Yardi still does not know whether the respondent, Entrata,Inc., used active machine learning? Maybe they used a new kind of Bikini Bottom software nobody has ever heard of? Maybe they used KL’s latest software? Or Catalyst? Maybe they did keyword search and passive analytics and never used machine training at all? Maybe they threw darts for search and used Adobe for review? Maybe they ran a series of random and judgmental samples for quality control and assurance? Maybe the results were good? Maybe not?

The review by Entrata could have been a very well managed project. It could have had many built-in quality control activities. It could have been an exemplar of Hybrid Multimodal Predictive Coding 4.0. You know, the method we perfected at NIST’s TREC Total Recall Track? The one that uses the more advanced IST, instead of simple CAL? I am proud to talk about these methods all day and how it worked out on particular projects. The whole procedure is transparent, even though disclosure of all metrics and tests is not. These measurements are anyway secondary to method. Yardi’s motion to compel disclosure should not have been so focused on a recall and richness number. It should instead of focused on methods. The e-Discovery Team methods are spelled out in detail in the TAR Course. Maybe that is what Entrata followed? Probably not. Maybe, God forbid, Entrata used random driven CAL? Maybe the TAR was a classic Sponge Bob Square Pants production of shame and failure? Now Yardi will never know. Or will they?

Yardi’s Quest for True Facts is Not Over

About the only way the requesting party, Yardi, can possibly get TAR disclosure in this case now is by proving the review and production made by Entrata was negligent, or worse, done in bad faith. That is a difficult burden. The requesting party has to hope they find serious omissions in the production to try to justify disclosure of method and metrics. (At the time of this order production by Entrata had not been made.) If expected evidence is missing, then this may suggest a record cleansing, or it may prove that nothing like that ever happened. Careful investigation is often required to know the difference between a non-existent unicorn and a rare, hard to find albino.

Remember, the producing party here, the one deep in the secret TAR, was Entrata, Inc. They are Yardi Systems, Inc. rival software company and defendant in this case. This is a bitter case with history. It is hard for attorneys not to get involved in a grudge match like this. Looks like strong feelings on both sides with a plentiful supply of distrust. Yardi is, I suspect, highly motivated to try to find a hole in the ESI produced, one that suggests negligent search, or worse, intentional withholding by the responding party, Entrata, Inc. At this point, after the motion to compel TAR method was denied, that is about the only way that Yardi might get a second chance to discover the technical details needed to evaluate Entrata’s TAR. The key question driven by Rule 26(g) is whether reasonable efforts were made. Was Entrata’s TAR terrible or terrific? Yardi may never know.

What about Yardi’s discovery? Do they have clean hands? Did Yardi do as good a job at ESI search as Entrata? (Assuing that Yardi used TAR too.) How good was Yardi’s TAR? (Had to ask that!) Was Yardi’s TAR as tardy as its motion? What were the metrics of Yardi’s TAR? Was it dark too? The opinion does not say what Yardi did for its document productions. To me that matters a lot. Cooperation is a mutual process. It is not capitulation. The same goes for disclosure. Do not come to me demanding disclosure but refusing to reciprocate.

How to Evaluate a Responding Party’s TAR?

Back to the TAR at issue. Was Entrata’s TAR riddled with errors? Did they oppose Yardi’s motion because they did a bad job? Was this whole project a disaster? Did Entrata know they had driven into a TAR pit? Who was the vendor? What software was used? Did it have active machine learning features? How were they used? Who was in charge of the TAR? What were their qualifications? Who did the hands-on review? What problems did they run into? How were these problems addressed? Did the client assist? Did the SMEs?

Perhaps the TAR was sleek and speedy and produced the kind of great results that many of us expect from active machine learning. Did sampling suggest low recall? Or high recall? How was the precision? How did this change over the rounds of training. The machine training was continuous, correct? The “seed-set nonsense” was not used, was it? You did not rely on a control set to measure results, did you? You accounted for natural concept drift, didn’t you, where the understanding of relevance changes over the course of the review? Did you use ei-Recall statistical sampling at the end of the project to test your work? Was a “Zero Error” policy followed for the omission of Highly Relevant documents as I recommend?. Are corrective supplemental searches now necessary to try to find missing evidence that is important to the outcome of the case? Do we need to force them to use an expert? Require that they use the state of the art standard, the e-Discovery Team’s Predictive Coding 4.0 Hybrid Multimodal IST?

Yardi’s motion was weak and tardy so Entrata, Inc. could defend its process simply by keeping it secret. This is the work-product defense approach. This is NOT how I would have defended a TAR process. Or rather, not the only way. I would have objected to interference, but also made controlled, limited disclosures. I would have been happy, even proud to show what state of the art search looks like. I would introduce our review team, including our experts, and provide an overview of the methods, the work-flow.

I would also have demanded reciprocal disclosures. What method, what system did you use? TAR is an amazing technology, if used correctly. If used improperly, TAR can be a piece of junk. How did the Subject Matter Experts in this case control the review? Train the machine? Is that a scary ghost in the machine or just a bad SMI?

How did Entrata do it? How for that matter did the requesting party, Yardi, do it? Did it use TAR as part of its document search? Is Yardi proud of its TAR? Or is Yardi’s TAR as dark and hardy har har as Entrata’s TAR. Are all the attorneys and techs walking around with their heads down and minds spinning with secret doc review failures?

e-Discovery Team reviews routinely exceed minimal reasonable efforts; we set the standards of excellence.  I would have made reasonable reassurances by disclosure of method. That builds trust. I would have pointed them to the TAR Course and the 4.0 methods. I would have sent them the below eight-step work-flow diagram. I would have told then that we follow these eight steps or if any deviations were expected, explained why.

I would have invited opposing counsel to participate in the process with any suggested keywords, hot documents to use to train. I would even allow them to submit model fictitious training documents. Let them create fake documents to help us to find any real ones that might be like it, no matter what specific words are used. We are not trying to hide anything. We are trying to find all relevant documents. All relevant documents will be produced, good or bad. Repeat that often. Trust is everything. You can never attain real cooperation without it. Trust but verify. And clarify by talk. Talk to your team, your client, witnesses, opposing counsel and the judge. That is always the first step.

Of course, I would not spend unlimited time going over everything. I dislike meetings and talking to people who have little or no idea what I am saying. Get your own expert. Do the work. These big document review projects often go on for weeks and you could waste and spend a small fortune with too much interaction and disclosures. I don’t want people looking over my shoulder and I don’t want to reveal all of my tricks and work-product secrets, just the general stuff you could get by study of my books. I would have drawn some kind of line of secrecy in the sand, hopefully not quicksand, so that our disclosures were reasonable and not over-burdensome. In Entrata the TAR masters doing the search did not want reveal much of anything. They were very distrustful of Yardi and perhaps sensed a trap. More on that later. Or maybe Entrata did have something to hide? How do we get at the truth of this question without looking at all of the documents ourselves? That is very difficult, but one way to get at the truth is to look at the search methods used, the project metadata.

The dark TAR defense worked for Entrata, but do not count on it working in your case. The requesting party might not be tardy like Yardi. They might make a far better argument.

Order Affirming Denial of Motion to Compel Disclosure of TAR

The well-written opinion in Entrata, Inc. v. Yardi Systems, Inc., (D.C. Utah, 10/29/18) was  by United States District Judge Clark Waddoups. Many other judges have gone over this transparency issue before and Judge Waddoups has a good summary of the ones cited to him by the moving party. I remember tackling these transparency issues with Judge Andrew Peck in Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012), which is one of the cases that Judge Waddoups cites. At that time, 2012, there was no precedent even allowing Predictive Coding, much less discussing details of its use, including disclosure best-practices. We made strong efforts of cooperation on the transparency issues after Judge Peck approved predictive coding. Judge Peck was an expert in TAR and very involved in the process. That kind of cooperation that can be encouraged by a very active judge did not happen in Entrata. The cooperative process failed. That led to a late motion by the Plaintiff to force disclosure of the TAR.

The plaintiff, Yardi Systems, Inc, is the party who requested ESI from defendants in this software infringement case. It wanted to know how the defendant was using TAR to respond to their request. Plaintiff’s motion to compel focused on disclosure of the statistical analysis of the results, Recall and Prevalence (aka Richness). That was another mistake. Statistics alone can be meaningless and misleading, especially if range is not considered, including the binomial adjustment for low prevalence. This is explained and covered by my ei-Recall test. Introducing “ei-Recall” – A New Gold Standard for Recall Calculations in Legal SearchPart One, Part Two and Part Three (e-Discovery Team, 2015). Also see: In Legal Search Exact Recall Can Never Be Known.

Disclosure of the whole process, the big picture, is the best Defense Of Process evidence, not just a couple of random sample test results. Looks like the requesting party here might have just been seeking “gotcha material” by focusing so much on the recall numbers. That may be another unstated reason both the Magistrate and District Court Judges denied their late request for disclosure. That could why the attorneys for Entrata kept their TAR dark, even though they were not negligent or in bad faith. Maybe they were proud of their efforts, but were tired of bad faith hassling by the requesting party. Hard to know based on this opinion alone.

After Chief Magistrate Judge Warner denied Yardi’s motion to compel, Yardi appealed to the District Court Judge Waddoups and argued that the Magistrate’s order was “clearly erroneous and contrary to law.”  Yardi argued that “the Federal Rules of Civil Procedure and case law require Entrata, in the first instance, to provide transparent disclosures as a requirement attendant to its use of TAR in its document review.”

Please, that is not an accurate statement of the governing legal precedent. It was instead “wishful thinking” on the part of plaintiff’s counsel. Sounds like a Sponge Bob Square Pants move to me. Judge Waddoups considered taxing fees against the plaintiff under Rule 37(a)(5)(B) because of this near frivolous argument, but ultimately let them off by finding the position was not “wholly meritless.”

Judge Waddoups had no choice but to deny a motion like this filed under these procedures. Here is a key paragraph explaining his reasoning for denial.

The Federal Rules of Civil Procedure assume cooperation in discovery. Here, the parties never reached an agreement regarding search methodology. In the court’s view, the lack of any agreement regarding search methodology is a failure on the part of both parties. Nevertheless, Yardi knew, as early as May of 2017, that Entrata intended to use TAR. (See ECF No. 257-1 at 2.) The Magistrate Court’s September 20, 2017 Order stated, in part, that “[i]f the parties are unable to agree on . . . search methodology within 30 days of the entry of this Order, the parties will submit competing proposals . . . .” (ECF No. 124 at 2.) Yardi, as early as October 2, 2017, knew that “Entrata [was] refus[ing] to provide” “TAR statistics.” (See ECF No. 134 at 3.) In other words, Yardi knew that the parties had not reached an agreement regarding search methodology well before the thirty day window closed. Because Yardi knew that the parties had not reached an agreement on search methodology, it should have filed a proposal with the Magistrate Court. This would have almost certainly aided in resolving this dispute long before it escalated. But neither party filed any proposal with the Magistrate Court within 30 days of entry of its Order. Yardi has not pointed to any Federal Rule of Civil Procedure demonstrating that the Magistrate Court’s Order was contrary to law. This court rejects Yardi’s argument relating to the Federal Rules of Civil Procedure.

Conclusion

The requesting party in Entrata did not meet the high burden needed to reverse a magistrate,s discovery ruling as clearly erroneous and contrary to law. If you are ever going to win on a motion like this, it will likely be on a Magistrate level. Seeking to overturn a denial and meet this burden to reverse is extremely difficult, perhaps impossible in cases seeking to compel TAR disclosure. The whole point is that there is no clear law on the topic yet. We are asking judges to make new law, to establish new standards of transparency. You must be open and honest to attain this kind of new legal precedent. You must use great care to be accurate in any representations of Fact or Law made to a court. Tell them it is a case of first impression when the precedent is not on point as was the situation in Entrata, Inc. v. Yardi Systems, Inc., Case No. 2:15-cv-00102 (D.C. Utah, 10/29/18). Tell them the good and the bad. There was never a perfect case and there always has to be a first for anything. Legal precedent moves slowly, but it moves continuously. It is our job as practicing attorneys to try to guide that change.

The requesting party seeking disclosure of TAR methods in Entrata doomed their argument by case law  misstatements and in-actions. They might have succeeded by making full disclosures themselves, both of the law and their own TAR. The focus of their argument should be on the benefits of doing TAR right and the dangers of doing it wrong. They should have talked more about what TAR – Technology Assisted Review – really means. They should have stressed cooperation and reciprocity.

To make new precedent in this area you must first recognize and explain away a number of opposing principles,  including especially The Sedona Conference Principle Six. That says responding parties always know best and requesting parties should stay out of their document reviews. I have written about this Principle and why it should be updated. Losey, Protecting the Fourteen Crown Jewels of the Sedona Conference in the Third Revision of its Principles (e-Discovery Team, 2//2/17). The Sedona Principle Six argument  is just one of many successful defenses that can be used to protect against forced TAR disclosure. There are also good arguments based on the irrelevance of this search information to claims or defenses under Rule 26(b)(2) and under work-product confidentiality protection.

Any party who would like to force another to make TAR disclosure should make such voluntary disclosures themselves. Walk your talk to gain credibility. The disclosure argument will only succeed, at least for the first time (the all -important test case), in the context of proportional cooperation. An extended 26(f) conference is a good setting and time. Work-product confidentiality issues should be raised in the first days of discovery, not the last day. Timing is critical.

The 26(f) discovery conference dialogue should be directed towards creating a uniform plan for both sides. This means the TAR disclosures should be reciprocal. The ideal test case to make this law would be a situation where the issue is decided early at a Rule 16(b) hearing. It would involve a situation where one side is willing to disclose, but the other is not, or where the scope of disclosures is disputed. At the 16(b) hearing, which usually takes place in the first three months, the judge is supposed to consider the parties’ Rule 26(f) report and address any discovery issues raised, such as TAR method and disclosures.

The first time disclosure is forced by a judge it will almost certainly be a mutual obligation. Each side should will be required to assume the same disclosure obligations. This could include  a requirement for statistical sampling  and disclosure of certain basic metrics such as Recall range, Prevalence and Precision? Sampling tests like this can be run no matter what search method is used, even little old keyword search.

It is near impossible to come into court when both sides have extensive ESI and demand that your opponent do something that you yourself refuse to do. If you expect to be able to force someone to use TAR, or to disclose basic TAR methods and metrics, then you had better be willing to do that yourself. If you are going to try to force someone to disclose work-product protected information, such as an attorney’s quality control tests for Recall range in document review, then you had better make such a limited waiver yourself.

 


Do TAR the Right Way with “Hybrid Multimodal Predictive Coding 4.0”

October 8, 2018

The term “TAR” – Technology Assisted Review – as we use it means document review enhanced by active machine learning. Active machine learning is an important tool of specialized Artificial Intelligence. It is now widely used in many industries, including Law. The method of AI-enhanced document review we developed is called Hybrid Multimodal Predictive Coding 4.0. Interestingly, reading these words in the new Sans Forgetica font will help you to remember them.

We have developed an online instructional program to teach our TAR methods and AI infused concepts to all kinds of legal professionals. We use words, studies, case-law, science, diagrams, math, statistics, scientific studies, test results and appeals to reason to teach the methods. To balance that out, we also make extensive use of photos and videos. We use right brain tools of all kinds, even subliminals, special fonts, hypnotic images and loads of hyperlinks. We use emotion as another teaching tool. Logic and Emotion. Sorry Spock, but this multimodal, holistic approach is more effective with humans than an all-text, reason-only approach of Vulcan law schools.

We even try to use humor and promote student creativity with our homework assignments. Please remember, however, this is not an accredited law school class, so do not expect professorial interaction. Did we mention the TAR Course is free?

By the end of study of the TAR Course you will know and remember exactly what Hybrid Multimodal means. You will understand the importance of using all varieties of legal search, for instance: keywords, similarity searches, concept searches and AI driven probable relevance document ranking. That is the Multimodal part. We use all of the search tools that our KL Discovery document review software provides.

 

The Hybrid part refers to the partnership with technology, the reliance of the searcher on the advanced algorithmic tools. It is important than Man and Machine work together, but that Man remain in charge of justice. The predictive coding algorithms and software are used to enhance the lawyers, paralegals and law tech’s abilities, not replace them.

By the end of the TAR Course you will also know what IST means, literally Intelligently Spaced Training. It is our specialty technique of AI training where you keep training the Machine until first pass relevance review is completed. This is a type of Continuous Active Learning, or as Grossman and Cormack call it, CAL. By the end of the TAR Course you should also know what a Stop Decision is. It is a critical point of the document review process. When do you stop the active machine teaching process? When is enough review enough? This involves legal proportionality issues, to be sure, but it also involves technological processes, procedures and measurements. What is good enough Recall under the circumstances with the data at hand? When should you stop the machine training?

We can teach you the concepts, but this kind of deep knowledge of timing requires substantial experience. In fact, refining the Stop Decision was one of the main tasks we set for ourself for the  e-Discovery Team experiments in the Total Recall Track of the National Institute of Standards and Technology Text Retrieval Conference in 2015 and 2016. We learned a lot in our two years. I do not think anyone has spent more time studying this in both scientific and commercial projects than we have. Kudos again to KL Discovery for helping to sponsor this kind of important research  by the e-Discovery Team.

 

 

Working with AI like this for evidence gathering is a newly emerging art. Take the TAR Course and learn the latest methods. We divide the Predictive Coding work flow into eight-steps. Master these steps and related concepts to do TAR the right way.

 

Pop Quiz: What is one of the most important considerations on when to train again?

One Possible Correct Answer: The schedule of the humans involved. Logistics and project planning is always important for efficiency. Flexibility is easy to attain with the IST method. You can easily accommodate schedule changes and make it as easy as possible for humans and “robots” to work together. We do not literally mean robots, but rather refer to the advanced software and the AI that arises from the machine training as an imiginary robot.

 

 

 

 

 

 

 

 

 


Elusion Random Sample Test Ordered Under Rule 26(g) in a Keyword Search Based Discovery Plan

August 26, 2018

There is a new case out of Chicago that advances the jurisprudence of my sub-specialty, Legal Search. City of Rockford v. Mallinckrodt ARD Inc., 2018 WL 3766673, Case 3:17-cv-50107 (N.D. Ill., Aug. 7, 2018). This discovery order was written by U.S. Magistrate Judge Iain Johnston who entitled it: “Order Establishing Production Protocol for Electronically Stored Information.” The opinion is both advanced and humorous, destined to be an oft-cited favorite for many. Thank you Judge Johnston.

In City of Rockford an Elusion random sample quality assurance test was required as part of the parties discovery plan to meet the reasonable efforts requirements of Rule 26(g). The random sample procedure proposed was found to impose only a proportional, reasonable burden under Rule 26(b)(1). What makes this holding particularly interesting is that an Elusion test is commonly employed in predictive coding projects, but here the parties had agreed to a keyword search based discovery plan. Also see: Tara Emory, PMP, Court Holds that Math Matters for eDiscovery Keyword Search,  Urges Lawyers to Abandon their Fear of Technology (Driven, (August 16, 2018) (“party using keywords was required to test the search effectiveness by sampling the set of documents that did not contain the keywords.”)

The Known Unknowns and Unknown Unknowns

Judge Johnston begins his order in City of Rockford with a famous quote by Donald Rumseld, a two-time Secretary of Defense.

“[A]s we know there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. . .”
Donald Rumseld

For those not familiar with this famous Known Knowns quip, here is a video of the original:

Here the knowledge logic is spelled out in a chart, since I know we all love that sort of thing. Deconstructing Rumsfeld: Knowledge and Ignorance in the Age of Innovation (Inovo 5/114).

Anybody who does complex investigations is familiar with this problem. Indeed, you can argue this insight is fundamental to all of science and experimental method. Logan, David C. (March 1, 2009). “Known knowns, known unknowns, unknown unknowns and the propagation of scientific enquiry”, Journal of Experimental Botany 60 (3). pp. 712–4. [I have always wanted to quote a botany journal.]

How do you deal with the known unknowns and the unknown unknowns, the information that we don’t even know that we don’t know about? The deep, hidden information that is both obtuse and rare. Information that is hard to retrieve and harder still to prove does not exist at all. Are you chasing something that might not exist? Something unknown because nonexistent? Such as an overlooked Highly Relevant document? (The stuff of nightmares!) Are you searching for nothing? Zero? If you find it, what does that mean? What can be known and what can never be known? Scientists, investigators and the Secretary of Defense alike all have to ponder these questions and all want to use the best tools and best people possible to do so. See: Deconstructing Rumsfeld: Knowledge and Ignorance in the Age of Innovation (Inovo 5/114).

Seeking Knowledge of the Unknown Elusion Error Rate

These big questions, though interesting, are not why Judge Johnston started his opinion with the Rumseld quote. Instead, he used the quote to emphasize that new e-discovery methods, namely random sampling and statistical analysis, can empower lawyers to know what they never did before. A technical way to know the known unknowns. For instance, a way to know the number of relevant documents that will be missed and not produced: the documents that elude retrieval.

As the opinion and this blog will explain, you can do that, know that, by using an Elusion random sample of the null-set. The statistical analysis of the sample transforms the unknown quantity to a known (subject to statistical probabilities and range). It allows lawyers to know, at least within a range, the number of relevant documents that have not been found. This is a very useful quality assurance method that relies on objective measurements to demonstrate success of your project, which here is information retrieval. This and other random sampling methods allow for the calculation of Recall, meaning the percent of total relevant documents found. This is another math-based, quality assurance tool in the field of information retrieval.

One of the main points Judge Johnston makes in his order is that lawyers should embrace this kind of technical knowledge, not shy away from it. As Tara Emory said in her article, Court Holds that Math Matters for eDiscovery Keyword Search:

A producing party must determine that its search process was reasonable. In many cases, the best way to do this is with objective metrics. Producing parties often put significant effort into brainstorming keywords, interviewing witnesses to determine additional terms, negotiating terms with the other party, and testing the documents containing their keywords to eliminate false positives. However, these efforts often still fail to identify documents if important keywords were missed, and sampling the null set is a simple, reasonable way to test whether additional keywords are needed. …

It is important to overcome the fear of technology and its related jargon, which can help counsel demonstrate the reasonableness of search and production process. As Judge Johnston explains, sampling the null set is a process to determine “the known unknown,” which “is the number of the documents that will be missed and not produced.” Judge Johnson disagreed with the defendants’ argument “that searching the null set would be costly and burdensome.” The Order requires Defendants to sample their null set at a 95% +/-2% margin of error (which, even for a very large set of documents, would be about 2,400 documents to review).[4] By taking these measures—either with TAR or with search terms, counsel can more appropriately represent that they have undertaken a “reasonable inquiry” for relevant information within the meaning of FRCP 26(g)(1).

Small Discovery Dispute in an Ocean of Cooperation

Judge Johnston was not asked to solve the deep mysteries of knowing and not knowing in City of Rockford. The parties came to him instead with an interesting, esoteric discovery dispute. They had agreed on a great number of things, for which the court profusely congratulated them.

The attorneys are commended for this cooperation, and their clients should appreciate their efforts in this regard. The Court certainly does. The litigation so far is a solid example that zealous advocacy is not necessarily incompatible with cooperation. The current issue before the Court is an example of that advocacy and cooperation. The parties have worked to develop a protocol for the production of ESI in this case, but have now reached an impasse as to one aspect of the protocol.

The parties disagreed on whether to include a document review quality assurance test in the protocol. The Plaintiffs wanted one and the Defendants did not. Too burdensome they said.

To be specific, the Plaintiffs wanted a test where the efficacy of any parties production would be tested by use of an Elusion type of Random Sample of the documents not produced. The Defendants opposed any specific test. Instead, they wanted the discovery protocol to say that if the receiving party had concerns about the adequacy of the producing party’s efforts, then they would have a conference to address the concerns.

Judge Johnston ruled for the plaintiff in this dispute and ordered a  random elusion sample to be taken after the defendant stopped work and completed production. In this case it was a good decision, but should not be routinely required in all matters.

The Stop Decision and Elusion Sample

One of the fundamental problems in any investigation is to know when you should stop the investigation because it is no longer worth the effort to carry on. When has a reasonable effort been completed? Ideally this happens after all of the important documents have already been found. At that point you should stop the effort and move on to a new project. Alternatively, perhaps you should keep on going and look for more? Should you stop or not?

In Legal Search we all this the “Stop Decision.” Should you conclude the investigation or continue further AI training rounds and other search. As explained in the e-Discovery Team TAR Course:

The all important stop decision is a legal, statistical decision requiring a holistic approach, including metrics, sampling and over-all project assessment.You decide to stop the review after weighing a multitude of considerations. Then you test your decision with a random sample in Step Seven.

See: TAR Course: 15th Class – Step Seven – ZEN Quality Assurance Tests.

If you want to go deeper into this, then listen in on this TAR Course lecture on the Stop decision.

____________

Once a decision is made to Stop, then a well managed document review project will use different tools and metrics to verify that the Stop decision was correct. Judge Johnston in City of Rockford used one of my favorite tools, the Elusion random sample that I teach in the e-Discovery Team TAR Course. This type of random sample is called an Elusion sample.

Judge Johnston ordered an Elusion type random sample of the null set in City of Rockford. The sample would determine the range of relevant documents that likely eluded you. These are called False Negatives. Documents presumed Irrelevant and withheld that were in fact Relevant and should have been produced. The Elusion sample is designed to give you information on the total number of Relevant documents that were likely missed, unretrieved, unreviewed and not produced or logged. The fewer the number of False Negatives the better the Recall of True Positives. The goal is to find, to retrieve, all of the Relevant ESI in the collection.

Another way to say the same thing is to say that the goal is Zero False Negatives. You do not miss a single relevant file. Every file designated Irrelevant is in fact not relevant. They are all True Negatives. That would be Total Recall: “the Truth, the Whole Truth …” But that is very rare and some error, some False Negatives, are expected in every large information retrieval project. Some relevant documents will almost always be missed, so the goal is to make the False Negatives inconsequential and keep the Elusion rate low.

Here is how Judge Iain Johnston explained the random sample:

Plaintiffs propose a random sample of the null set. (The “null set” is the set of documents that are not returned as responsive by a search process, or that are identified as not relevant by a review process. See Maura R. Grossman & Gordon v. Cormack, The Grossman-Cormack Glossary of Technology-Assisted Review, 7 Fed. Cts. L. Rev. 1, 25 (2013). The null set can be used to determine “elusion,” which is the fraction of documents identified as non-relevant by a search or review effort that are, in fact, relevant. Elusion is estimated by taking a random sample of the null set and determining how many or what portion of documents are actually relevant. Id. at 15.) FN 2

Judge Johnston’s Footnote Two is interesting for two reasons. One, it attempts to calm lawyers who freak out when hearing anything having to do with math or statistics, much less information science and technology. Two, it does so with a reference to Fizbo the clown.

The Court pauses here for a moment to calm down litigators less familiar with ESI. (You know who you are.) In life, there are many things to be scared of, including, but not limited to, spiders, sharks, and clowns – definitely clowns , even Fizbo. ESI is not something to be scared of. The same is true for all the terms and jargon related to ESI. … So don’t freak out.

Accept on Zero Error for Hot Documents

Although this is not addressed in the court order, in my personal view, no False Negatives, iw – overlooked  documents – are acceptable when it comes to Highly Relevant documents. If even one document like that is found in the sample, one Highly Relevant Document, then the Elusion test has failed in my view. You must conclude that the Stop decision was wrong and training and document review must recommence. That is called an Accept on Zero Error test for any hot documents found. Of course my personal views on best practice here assume the use of AI ranking, and the parties in City of Rockford only used keyword search. Apparently they were not doing machine training at all.

The odds of finding False Negatives, assuming that only a few exist (very low prevalence) and the database is large, are very unlikely in a modest sized random sample. With very low prevalence of relevant ESI the test can be of limited effectiveness. That is an inherent problem with low prevalence and random sampling. That is why statistics have only limited effectiveness and should be considered part of a total quality control program. See Zero Error Numerics: ZEN. Math matters, but so too does good project management and communications.

The inherent problem with random sampling is that the only way to reduce the error interval is to increase the size of the sample. For instance, to decrease the margin of error to only 2% either way, a total error of 4%, a random sample size of around 2,400 documents is needed. Even though that narrows the error rate to 4%, there is still another error factor of the Confidence Level, here at 95%. Still, it is not worth the effort to review even more sample documents to reduce that to a 99% Level.

Random sampling has limitations in low prevalence datasets, which is typical in e-discovery, but still sampling can be very useful. Due to this rarity issue, and the care that producing parties always take to attain high Recall, any documents found in an Elusion random sample should be carefully studied to see if they are of any significance. We look very carefully at any new documents found that are of a kind not seen before. That is unusual. Typically  any relevant documents found by random sample of the elusion set are of a type that have been seen before, often many, many times before. These “same old, same old” type of documents are of no importance to the investigation at this point.

Most email related datasets are filled with duplicative, low value data. It is not exactly irrelevant noise, but it is not a helpful signal either. We do not care if we  get all of that kind of merely relevant data. What we really want are the Hot Docs, the high value Highly Relevant ESI, or at least Relevant and of a kind not seen before. That is why the Accept On Zero Error test is so important for Highly Relevant documents.

The Elusion Test in City of Rockford 

In City of Rockford Judge Johnston considered a discovery stipulation where the parties had agreed to use a typical keyword search protocol, but disagreed on a quality assurance protocol. Judge Johnston held:

With key word searching (as with any retrieval process), without doubt, relevant documents will be produced, and without doubt, some relevant documents will be missed and not produced. That is a known known. The known unknown is the number of the documents that will be missed and not produced.

Back to the False Negatives again, the known unknown. Judge Johnston continues his analysis:

But there is a process by which to determine that answer, thereby making the known unknown a known known. That process is to randomly sample the nullset. Karl Schieneman & Thomas C. Gricks III, The Implications of Rule26(g) on the Use of Technology-Assisted Review, 2013 Fed. Cts. L. Rev. 239, 273 (2013)(“[S]ampling the null set will establish the number of relevant documents that are not being produced.”). Consequently, the question becomes whether sampling the null set is a reasonable inquiry under Rule 26(g) and proportional to the needs of this case under Rule 26(b)(1).

Rule 26(g) Certification
Judge Johnston takes an expansive view of the duties placed on counsel of record by Rule 26(g), but concedes that perfection is not required:

Federal Rule of Civil Procedure 26(g) requires all discovery requests be signed by at least one attorney (or party, if proceeding pro se). Fed. R. Civ. P. 26(g)(1). By signing the response, the attorney is certifying that to the best of counsel’s knowledge, information, and belief formed after a reasonable inquiry, the disclosure is complete and correct at the time it was made. Fed. R. Civ. P. 26(g)(1)(A). But disclosure of documents need not be perfect. … If the Federal Rules of Civil Procedure were previously only translucent on this point, it should now be clear with the renewed emphasis on proportionality.

Judge Johnston concludes that Rule 26(g) on certification applies to require the Elusion sample in this case.

Just as it is used in TAR, a random sample of the null set provides validation and quality assurance of the document production when performing key word searches.  Magistrate Judge Andrew Peck made this point nearly a decade ago. See William A. Gross Constr. Assocs., 256 F.R.D. at 135-6 (citing Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 262 (D. Md. 2008)); In re Seroquel Products Liability Litig., 244 F.R.D. 650, 662 (M.D. Fla. 2007) (requiring quality assurance).

Accordingly, because a random sample of the null set will help validate the document production in this case, the process is reasonable under Rule 26(g).

Rule 26(b)(1) Proportionality

Judge Johnston considered as a separate issue whether it was proportionate under Rule 26(b)(1) to require the elusion test requested. Again, the court found that it was in this large case on the pricing of prescription medication and held that it was proportional:

The Court’s experience and understanding is that a random sample of the null set will not be unreasonably expensive or burdensome. Moreover and critically, Defendants have failed to provide any evidence to support their contention. Mckinney/Pearl Rest. Partners, L.P. v. Metro. Life Ins. Co., 322 F.R.D. 235, 242 (N.D.Tex. 2016) (party required to submit affidavits or offer evidence revealing the nature of the burden)
Once again we see a party seeking protection from having to do something because it is so burdensome then failing to present actual evidence of burden. We see this a lot lately. Responding Party’s Complaints of Financial Burden of Document Review Were Unsupported by the Evidence, Any Evidence (e-Discovery Team, 8/5/18);

Judge Johnston concludes his “Order Establishing Production Protocol for Electronically Stored Information” with the following:

The Court adopts the parties’ proposed order establishing the production protocol for ESI with the inclusion of Plaintiffs’ proposal that a random sample of the null set will occur after the production and that any responsive documents found as a result of that process will be produced. Moreover, following that production, the parties should discuss what additional actions, if any, should occur. If the parties cannot agree at that point, they can raise the issue with the Court.

Conclusion

City of Rockford is important because it is the first case to hold that a quality control procedure should be used to meet the reasonable efforts certification requirements of Rule 26(g). The procedure here required was a random sample Elusion test with related, limited data sharing. If this interpretation of Rule 26(g) is followed by other courts, then it could have a big impact on legal search jurisprudence. Tara Emory in her article, Court Holds that Math Matters for eDiscovery Keyword Search goes so far as to conclude that City of Rockford stands for the proposition that “the testing and sampling process associated with search terms is essential for establishing the reasonableness of a search under FRCP 26(g).”

The City of Rockford holding could persuade other judges and encourage courts to be more active and impose specific document review procedures on all parties, including requiring the use of sampling and artificial intelligence. The producing party cannot always have a  free pass under Sedona Principle Six. Testing and sampling may well be routinely ordered in all “large” document review cases in the future.

It will be very interesting to watch how other attorneys argue City of Rockford. It will continue a line of cases examining methodology and procedures in document review. See eg., William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134 (S.D.N.Y. 2009) (“wake-up call” for lawyers on keyword search); Winfield v. City of New York (SDNY, Nov. 27, 2017), where Judge Andrew Peck considers methodologies and quality controls of the active machine learning process. Also see Special Master Maura Grossman’s Order Regarding Search Methodology for ESI, a validation Protocol for the Broiler Chicken antitrust cases.

The validation procedure of an Elusion sample in City of Rockford is just one of many possible review protocols that a court could impose under Rule 26(g). There are dozens more, including whether predictive coding should be required. So far, courts have been reluctant to order that, as Judge Peck explained in Hyles:

There may come a time when TAR is so widely used that it might be unreasonable for a party to decline to use TAR. We are not there yet.

Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP), 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016):

Like a kid in the backseat of the car, I cannot help but ask, are we there yet? Hyles was published over two years ago now. Maybe some court, somewhere in the world, has already ordered a party to do predictive coding against their will, but not to our knowledge. That is a known unknown. Still, we are closer to “There” with the City of Rockford’s requirement of an Elusion test.

When we get “there,” and TAR is finally ordered in a case, it will probably arise in a situation like City of Rockford where a joint protocol applicable to all parties is involved. That is easier to sell than a one-sided protocol. The court is likely to justify the order by Rule 26(g), and hold that it requires all parties in the case to use predictive coding. Otherwise, they will not meet the  reasonable effort burdens of Rule 26(g). Other rules will be cited too, of course, including Rule 1, but Rule 26(g) is likley to be key.

____________

___

 

____

 

 


%d bloggers like this: