VENDORS | e-Discovery Team ®

Cautionary Tale from Brooklyn: Search Terms ‘Designed To Fail’

October 20, 2019

Every lawyer who thinks e-discovery is not important, that you can just delegate it to a vendor, should read Abbott Laboratories, et al. v. Adelphia Supply USA, et al., No. 15 CV 5826 (CBA) (LB) (E.D.N.Y. May 2, 2019). This opinion in a trademark case in Brooklyn District Court (shown here) emphasizes, once again, that e-discovery can be outcome-determinative. If you mess it up, you can doom your case. If a lawyer wants to litigate today, they either have to spend the substantial time it takes to learn the many intricacies of e-discovery, or associate with a specialist who does. The Abbott Labs case shows how easily a law suit can be won or lost on e-discovery alone. Here the numbers did not add up, key custodians were omitted and guessed keywords were used, keywords so bad that opposing counsel called them designed to fail. The defendants reacted by firing their lawyers and blaming everything on them, but the court did not buy it. Instead, discovery fraud was found and judgment was entered for the plaintiff.

Magistrate Judge Lois Bloom (shown right) begins the Opinion by noting that the plaintiff’s motion for case ending sanctions “… presents a cautionary tale about how not to conduct discovery in federal court.” The issues started when defendant made its first electronic document production. The Electronically Stored Information was all produced in paper, as Judge Bloom explained “in hard copy, scanning them all together, and producing them as a single, 1941-page PDF file.” Opinion pg. 3. This is not what the plaintiff Abbott Labs wanted. After Abbott sought relief from the court the defendants on March 24, 2017 were ordered to “produce an electronic copy of the 2014 emails (1,941 pages)” including metadata. Defendant then “electronically produced 4,074 pages of responsive documents on April 5, 2017.” Note how the page count went from 1,942 to 4,074. There was no explanation of this page count discrepancy, the first of many, but the evidence helped Abbott justify a new product counterfeiting action (Abbott II) where the court ordered a seizure of defendant’s email server. That’s were the fun started. As Judge Bloom put it:

Once plaintiffs had seized H&H’s email server, plaintiffs had the proverbial smoking gun and raised its concerns anew that defendants had failed to comply with the Court’s Order to produce responsive documents in the instant action (hereinafter “Abbott I”). On July 12, 2017, the Court ordered the H&H defendants to “re-run the document search outlined in the Court’s January 17 and January 21 Orders,” “produce the documents from the re-run search to Abbott,” and to produce “an affidavit of someone with personal knowledge” regarding alleged technical errors that affected the production.³ Pursuant to the Court’s July 12, 2017 Order to re-run the search, The H&H defendants produced 3,569 responsive documents.

Opinion pg. 4 (citations to record omitted).

Too Late For Vendor Help and a Search Strategy Designed to Fail

After the seizure order in Abbott II, and after Abbott Labs again raised issues regarding defendants’ original production, Judge Bloom ordered the defendants to re-run the original search. Defendants then retained the services of an outside vendor, Transperfect, to re-run the original search for them. In supposed compliance with that order, the defendants, aka H&H, then produced 3,569 documents. Id. at 8. Defendants also filed an affidavit by Joseph Pochron, Director in the Forensic Technology and Consulting Division at Transperfect (“Pochron Decl.”) to try to help their case. It did not work. According to Judge Bloom the Pochron Decl. states:

… that H&H utilized an email archiving system called Barracuda and that there are two types of Barracuda accounts, Administrator and Auditor. Pochron Decl. ¶ 13. Pochron’s declaration states that the H&H employee who ran the original search, Andrew Sweet, H&H’s general manager, used the Auditor account to run the original search (“Sweet search”). Id. at ¶ 19. When Mr. Pochron replicated the Sweet search using the Auditor account, he obtained 1,540 responsive emails. Id. at ¶ 22. When Mr. Pochron replicated the Sweet search using the Administrator account, he obtained 1,737 responsive emails. Id. Thus, Mr. Pochron attests that 197 messages were not viewable to Mr. Sweet when the original production was made. Id. Plaintiffs state that they have excluded those 197 messages, deemed technical errors, from their instant motion for sanctions. Plaintiffs’ Memorandum of Law at 9; Waters Decl. ¶ 8. However, even when those 197 messages are excluded, defendants’ numbers do not add up. In fact, H&H has repeatedly given plaintiffs and the Court different numbers that do not add up.

Moreover, plaintiffs argue that the H&H defendants purposely used search terms designed to fail, such as “International” and “FreeStyle,” whereas H&H’s internal systems used item numbers and other abbreviations such as “INT” and “INTE” for International and “FRL” and “FSL” for FreeStyle. Plaintiff’s Memorandum of Law at 10–11. Plaintiffs posit that defendants purposely designed and ran the “extremely limited search” which they knew would fail to capture responsive documents …

Opinion pgs. 8-9 (emphasis by bold added). “Search terms designed to fail.” This is the first time I have ever seen such a phrase in a judicial opinion. Is purposefully stupid keyword search yet another bad faith litigation tactic by unscrupulous attorneys and litigants? Or is this just another example of dangerous incompetence? Judge Bloom was not buying the ‘big oops” theory, especially considering the ever-changing numbers of relevant documents found. It looked to her, and me too, that this search strategy was intentionally design to fail, that it was all a shell-game.

This is the wake-up call for all litigators, especially those who do not specialize in e-discovery. Your search strategy had better make sense. Search terms must be designed (and tested) to succeed, not fail! This is not just incompetence.

The Thin Line Between Gross Negligence and Bad Faith

The e-discovery searches you run are important. The “mistakes” made here led to a default judgment. That is the way it is in federal court today. If you think otherwise, that e-discovery is not that important, that you can just hire a vendor and throw stupid keywords at it, then your head is dangerously stuck in the sand. Look around. There are many cases like Abbott Laboratories, et al. v. Adelphia Supply USA.

I say “mistakes” made here in quotes because it was obvious to Judge Bloom that these were not mistakes at all, this was fraud on the court.

E-Discovery is about evidence. About truth. You cannot play games. Either take it seriously and do it right, do it ethically, do it competently; or go home and get out. Retire already. Discovery gamesmanship and lawyer bumbling are no longer tolerated in federal court. The legal profession has no room for dinosaurs like that.

Abbott Labs responded the way they should, the way you should always expect in a situation like this:

Plaintiffs move for case ending sanctions under Federal Rule of Civil Procedure 37 and invoke the Court’s inherent power to hold defendants in default for perpetrating a fraud upon the Court. Plaintiffs move to strike the H&H defendants’ pleadings, to enter a default judgment against them, and for an order directing defendants to pay plaintiffs’ attorney’s fees and costs, for investigating and litigating defendants’ discovery fraud.

Id.

Rule 37(e) was revised in 2015 to make clear that gross negligence alone does not justify a case-ending sanction, that you must prove bad faith. This change should not provide the incompetent with much comfort. As this case shows, the difference between mistake and intent can be a very thin line. Do your numbers add up? Can you explain what you did and why you did it? Did you use good search terms? Did you search all of the key custodians? Or did you just take the ESI the client handed to you and say thank you very much? Did you look with a blind eye? Even if bad faith under Rule 37 is not proven, the court may still find the whole process stinks of fraud and use the court’s inherent powers to sanction misconduct.

As Judge Bloom went on to explain:

Under Rule 37, plaintiffs’ request for sanctions would be limited to my January 17, 2017 and January 27, 2017 Orders which directed defendants to produce documents as set forth therein. While sanctions under Rule 37 would be proper under these circumstances, defendants’ misconduct herein is more egregious and goes well beyond defendants’ failure to comply with the Court’s January 2017 discovery orders. . . . Rather than viewing the H&H defendants’ failure to comply with the Court’s January 2017 Orders in isolation, plaintiffs’ motion is more properly considered in the context of the Court’s broader inherent power, because such power “extends to a full range of litigation abuses,” most importantly, to fraud upon the court.

Opinion pg. 5.

Judge Bloom went on the explain further the “fraud on the court” and defendant’s e-discovery conduct.

A fraud upon the court occurs where it is established by clear and convincing evidence “that a party has set in motion some unconscionable scheme calculated to interfere with the judicial system’s ability impartially to adjudicate a matter by . . . unfairly hampering the presentation of the opposing party’s claim or defense.” New York Credit & Fin. Mgmt. Grp. v. Parson Ctr. Pharmacy, Inc., 432 Fed. Appx. 25 (2d Cir. 2011) (summary order) (quoting Scholastic, Inc. v. Stouffer, 221 F. Supp. 2d 425, 439 (S.D.N.Y. 2002))

Opinion pgs. 5-6 (subsequent string cites omitted).

Kill All The Lawyers

The defendants here tried to defend by firing and blaming their lawyers. That kind of Shakespearean sentiment is what you should expect when you represent people like that. They will turn on you. They will use you for their nefarious ends, then lose you. Kill you if they could.

Judge Bloom, who was herself a lawyer before becoming a judge, explained the blame-game defendants tried to pull in her court.

Regarding plaintiffs’ assertion that defendants designed and used search terms to fail, defendants proffer that their former counsel, Mr. Yert, formulated and directed the use of the search terms. Id. at 15. The H&H defendants state that “any problems with the search terms was the result of H&H’s good faith reliance on counsel who . . . decided to use parameters that were less robust than those later used[.]” Id. at 18. The H&H defendants further state that the Sweet search results were limited because of Mr. Yert’s incompetence. Id.

Opinion pg. 9.

Specifically defendants alleged:

… the original search parameters were determined by Mr. Yert and that he “relied on Mr. Yert’s expertise as counsel to direct the parameters and methods for a proper search that would fulfill the Court’s Order.” Sweet Decl. ¶ 3–4. As will be discussed below, the crux of defendants’ arguments throughout their opposition to the instant motion seeks to lay blame on Mr. Yert for their actions; however, defendants cannot absolve themselves of liability here by shifting blame to their former counsel.

Opinion pg. 11.

Here is how Judge Bloom responded to this “blame the lawyers” defense:

Defendants’ attempt to lay blame on former counsel regarding the design and use of search terms is equally unavailing. It is undisputed that numerous responsive documents were not produced by the H&H defendants that should have been produced. Defendants’ prior counsel conceded as much. See generally plaintiffs’ Ex. B, Tr. Of July 11, 2017 telephone conference.

Mr. Yert was asked at his deposition about the terms that H&H used to identify their products and he testified as follows:

Q. Tell me about the general discussions you had with the client in terms of what informed you what search terms you should be using.

A. Those were the terms consistently used by H&H to identify the particular product.

Q. So the client told you that FreeStyle and International are the terms they consistently used to refer to International FreeStyle test strips; is that correct?

A. That’s what I recall.

Q. Did the client tell you that they used the abbreviation FSL to refer to FreeStyle?

A. I don’t recall.

Q. If they had told you that, you would have included that as a search term, correct?

A. I don’t recall if it was or was not included as a search term, sir.

Opinion pgs. 10-11.

The next time you are asked to dream up keywords for searches to find your client’s relevant evidence, remember this case, remember this deposition. Do not simply use keywords that the client suggests, as the attorneys did here. Do not simply use keywords. As I have written here many, many times before, there is a lot more to electronic evidence search and review than keywords. This is the Twenty First Century. You should be using AI, specifically active machine learning, aka Predictive Coding.

You need an expert to help you and you need them at the start of a case, not after sanctions motions.

Judge Lois Bloom went on to explain that, even if defendant’s story of innocent reliance on it lawyers was true:

It has long been held that a client-principal is “bound by the acts of his lawyer agent.” Id. (quoting Link v. Wabash RR. Co., 370 U.S. 626, 634 (1962)). As the Second Circuit stated, “even innocent clients may not benefit from the fraud of their attorney.” Id. . . .

However, notwithstanding defendants’ assertion that the search terms “FreeStyle” and “International” were used in lieu of more comprehensive search terms at the behest of Mr. Yert, it is undisputed that Mr. Sweet, H&H’s general manager, knew that H&H used abbreviations for these terms. Mr. Sweet admitted this at his deposition. See Sweet Dep. 81:2-81:24, Mar. 13, 2018. . . . The Court need not speculate as to why defendants did not use these search terms to comply with defendants’ obligation to produce pursuant to the Court’s Order. Mr. Sweet, by his own admission, states that “on several occasions he contacted Mr. Yert with specific questions about whether to include certain emails in production.” Sweet Decl. ¶ 7. It is inconceivable that H&H’s General Manager, who worked closely with Mr. Yert to respond to the Court’s Order, never mentioned that spelling out the terms used, “International” and “FreeStyle”, would not capture the documents in H&H’s email system. Mr. Sweet knew that H&H was required to produce documents regarding International FreeStyle test strips, regardless of whether H&H’s documents spelled out or abbreviated the terms. Had plaintiffs not seized H&H’s email server in the counterfeiting action, plaintiffs would have never known that defendants failed to produce a trove of responsive documents. H&H would have gotten away with it.

Opinion pgs. 12-13.

Defendants also failed to produce any documents by three custodians Holland Trading, Howard Goldman, and Lori Goldman. Again, they tried to blame that omission on their attorney, who they claim directed the search. Oh yeah, for sure. To me he looks like a mere stooge, a tool of unscrupulous litigants. Judge Bloom did not accept that defense either, holding:

While defendants’ effort to shift blame to Mr. Yert is unconvincing at best, even if defendants’ effort could be credited, counsel’s actions, even if they were found to be negligent, would not shield the H&H defendants from responsibility for their bad faith conduct.

Opinion pgs. 19-20. Then Judge Bloom went on to cite the record at length, including the depositions and affidavits of the attorneys involved, to expose this blame game as a sham. The order then concludes on this point holding:

There is no credible explanation for why the Holland Trading, Howard Goldman, and Lori Goldman documents were not produced except that the documents were willfully withheld. Defendants’ explanation that there were no documents withheld, then that any documents that weren’t produced were due to technical glitches, then that the documents didn’t appear in Mr. Sweet’s original search, then that if documents were intentionally removed, they were removed per Mr. Yert’s instructions cannot all be true. The H&H defendants have always had one more excuse up their sleeve in this “series of episodes of nonfeasance,” which amounts to “deliberate tactical intransigence.” Cine, 602 F.2d at 1067. In light of the H&H defendants’ ever-changing explanations as to the withheld documents, Mr. Sweet’s inconsistent testimony, and assertions of former counsel, the Court finds that the H&H defendants have calculatedly attempted to manipulate the judicial process. See Penthouse, 663 F.2d 376–390 (affirming entry of default where plaintiffs disobeyed an “order to produce in full all of [their] financial statements,” engaged in “prolonged and vexatious obstruction of discovery with respect to closely related and highly relevant records,” and gave “false testimony and representations that [financial records] did not exist.”).

Opinion pgs. 22-23.

The plaintiff, Abbott Labs, went on to argue that “the withheld documents freed David Gulas to commit perjury at his deposition. The Court agrees.” Id. at 24. The Truth has a way of finding itself out, especially with competent counsel on the other side and a good judge.

With this evidence the Court concluded the only adequate sanction was a default judgment in plaintiff’s favor. Message to spoliating defendants, game over, you lose.

Based on the full record of the case, there is clear and convincing evidence that defendants have perpetrated a fraud upon the court. Defendants’ initial conduct of formulating search terms designed to fail in deliberate disregard of the lawful orders of the Court allowed H&H to purposely withhold responsive documents, including the Holland Trading, Howard Goldman, and Lori Goldman documents. Defendants proffered inconsistent positions with three successive counsel as to why the documents were withheld. Mr. Sweet’s testimony is clearly inconsistent if not perjured from his deposition to his declaration in opposition to the instant motion. Mr. Goldman’s deposition testimony is evasive and self-serving at best. Finally, Mr. Gulas’ deposition testimony is clearly perjured. Had plaintiffs never seized H&H’s server pursuant to the Court’s Order in the counterfeiting case, H&H would have gotten away with their fraud upon this Court. H&H only complied with the Court’s orders and their discovery obligations when their backs were against the wall. Their email server had been seized. There was no longer an escape from responsibility for their bad faith conduct. This is, again, similar to Cerruti, where the “defendants did not withdraw the [false] documents on their own. Rather, they waited until the falsity of the documents had been detected.” Cerruti.,169 F.R.D. at 583. But for being caught in a web of irrefutable evidence, H&H would have profited from their misconduct. . . .

The Court finds that the H&H defendants have committed a fraud upon the court, and that the harshest sanction is warranted. Therefore, plaintiffs’ motion for sanctions should be granted and a default judgment should be entered against H&H Wholesale Services, Inc., Howard Goldman, and Lori Goldman.

Conclusion

Attorneys of record sign responses under Rule 26(g) to requests for production, not the client. That is because the rules require them to control the discovery efforts of their clients. That means the attorney’s neck is on the line. Rule 26(g) does not allow you to just take a client’s word for it. Verify. Supervise. The numbers should add up. The search terms, if used, should be designed and tested to succeed, not fail. This is your response, not the client’s. You determine the search method, in consultation with the client for sure, but not by “just following orders.” You must see everything, not nothing. If you see no email from key custodians, dig deeper and ask why. Do this at the beginning of the case. Get vendor help before you start discovery, not after you fail. Apparently the original defense attorneys here did just what they were asked, they went along with the client. Look where it got them. Fired and deposed. Default judgment entered. Cautionary tale indeed.

Leave a Comment » | Lawyers Duties, Review, Search, Spoliation/Sanctions, VENDORS | Tagged: best practices, ethics, legal profession, legal search, sanctions | Permalink
Posted by Ralph Losey

It’s Mueller Time! The Emergence of a New Legal Service Based on Artificial Intelligence

January 29, 2019

It’s Mueller Time! I predict we will be hearing this call around the world for decades, including boardrooms. Organizations will decide to investigate themselves on sensitive issues before the government does, or before someone sues them and triggers formal discovery. Not always, but sometimes, they will do so by appointing their own independent counsel to check on concerns. The Boards of tomorrow will not look the other way. If Robert Muller himself later showed up at their door, they would be ready. They would thank their G.C. that they had already cleaned house.

Most companies who decide it is Mueller Time, will probably not investigate themselves in the traditional “full calorie” Robert Muller way, as good as that is. Instead, they will order a less expensive, AI based investigation, a Mueller Lite. The “full calorie” traditional legal investigation is very expensive, slow and leaky. It involves many people and linear document review. The AI Assisted alternative, the Mueller Lite, will be more attractive because of its lower cost. It will still be an independent investigation, but will rely primarily on internal data and artificial intelligence, not expensive attorneys.

I call this E-Vestigations, for electronic investigations. It is a new type of legal service made possible by a specialized type of AI called “Predictive Coding” and newly perfected Hybrid Multimodal methods of machine training.

Mueller Lite E-Vestigations Save Money

Robert Mueller investigations typically cost millions and involves large teams of expensive professionals. AI Assisted investigations are cheap by comparison. That is because they emphasize company data and AI search of the data, mostly the communications, and need so few people to carry out. This new kind of investigation allows a company to quietly look into and take care of its own problems. The cost savings from litigation avoidance, and bad publicity, can be compelling. Plus it is the right thing to do..

E-Vestigations will typically be a quarter the cost of a traditional Mueller style, paper investigations. It may even be far less than that. Project fees depend on the data itself (volume and “messiness”) and the “information need” of the client (simple or complex). The competitive pricing of the new service is one reason I predict it will explode in popularity. This kind of dramatic savings is possible because most of the time consuming relevance sorting and document ranking work is delegated to the AI.

The computer “reads” or reviews at nearly the speed of light and is 100% consistent. But it has no knowledge on its own. An idiot savant. The AI cannot do anything without its human handlers and trainers. It is basically a learning machine designed to sort large collections of texts into binary sets, typically relevant or irrelevant.

The human investigators read much slower and sometimes make mistakes (plus they like to get compensated), but they are absolutely indispensable. Someday the team of humans may get even smaller, but we are already down to around seven or fewer people per investigation. Compare that to the hundreds involved in a traditional Muller style document review.

Proactive “Peace of Mind” Investigations

This new legal service allows concerned management to proactively investigate upon the first indications of possible wrong-doing. It allows you to have greater assurance that you really know what is going on in your organization. Management or the Board then retains an independent team of legal experts to conduct the quick E-Vestigation. The team provides subject matter expertise on the suspected problem and uses active machine learning to quickly search and analyze the data. They search for preliminary indications of what happened, if anything. This kind of search is ideal for sensitive legal inquiries. It gives management the information needed without breaking the bank or publicizing the results.

This New Legal Service Is Built Around AI

E-Vestigations are a pre-litigation legal service that relies heavily on artificial intelligence, but not entirely. Investigations like this are very complex. They are nowhere near a fully automated process, and as mentioned the AI is really just a learning machine that knows nothing except how to learn document relevance. The service still needs legal experts, but a much smaller team

AI assisted investigations such as E-Vestigations have five compelling positive traits:

Cost
Speed
Stealthiness
Confidentiality
Accuracy.

This article introduces the new service, discusses these five positive traits and provides background for my prediction that many organizations will order AI assisted investigations in the coming years. In fact, due to the disappearing trial, I predict that E-Vestigations will someday take the lead from Litigation in many law firms. This prediction of the future, like most, requires a preliminary journey into the past, to see the longer causal line of events. That comes next, but feel to skip the next three sections until you come to the heading, What is an E-Vestigation?

King Litigation Is Dead

The glory days of litigation are over. All trial lawyers who, like me, have lived through the last forty years of legal practice, have seen it change dramatically. Litigation has moved from a trial and discovery practice, where we saw each other daily in court, to a discovery, motion and mediation practice where we communicate by email and occasional calls.

Although some “trial dogs” will not admit it, we all know that the role of trials has greatly diminished in modern practice. Everything settles. Ninety-nine percent (99%) of federal court civil cases settle without trial. Although my current firm is a large specialty practice, and so is an exception, in most law firms trials are very rare. A so-called “Trial Practice” of a major firm could go years without having an actual trial. I have seen it happen in many law firms. Good lawyers for sure, but they do not really “do trials,” they do trial preparation.

For example, when I started practicing law in 1980 “dispute resolution” was king in most law firms. It was called the “Litigation Department” and usually attracted the top legal talent. It brought in strong revenue and big clients. Every case in the top firms was either a “Bet the Farm” type, or a little case for kiddie lawyer training, we had no form-practice. Friedmann & Brown, “Bet the Farm” Versus “Law Factory”: Which One Works? (Geeks and Law, 2011).

The opposite, “Commodity Litigation,” was rare; typically just something for some divorce lawyers, PI lawyers, criminal lawyers and bankruptcy lawyers. These were not the desired specialties in the eighties, to put it mildly. Factory like practices like that did not pay that well (honest ones anyway) and were boring to most graduates of decent law schools. This has not changed much until recently, when AI has made certain Commodity practices far more interesting and desirable. See Joshua Kubicki, The Emerging Competitive Frontier in Biglaw is Practice Venturing (Medium, 1/24/19).

Aside from the less desirable Commodity practice law firms, most litigators in the eighties would routinely take a case to trial. Fish or Cut Bait was a popular saying. Back then Mediation was virtually unknown. Although a majority of cases did eventually settle, a large minority did not. That meant physically going to court, wearing suits and ties every day, and verbal sparing. Lots of arguments and talk about evidence. Sometimes it meant some bullying and physical pushing too, if truth be told. It was a rough and tumble legal world in the eighties, made up in many parts of the U.S. almost entirely of white men. Many were smokers, including the all-white bench.

Ah, the memories. Some of the Litigation attorneys were real jerks, to put it mildly. But only a few were suspected crooked and could not be trusted. Most were honest and could be. We policed our own and word got around about a bad apple pretty fast. Their careers in town were then soon over, one way or the other. Many would just move away or, if they had roots, become businessmen. There were trials a plenty in both the criminal and civil sides.The trials could be dramatic spectacles. The big cases were intense.

Emergence of Mediation

But the times were a changing. In the nineties and first decade of the 21st Century, trials quickly disappeared. Instead, Mediation started to take over. I know, I was in the first group of lawyers ever to be certified as a Mediator of High Technology disputes in 1989. All types of cases began to settle earlier and with less preparation. I have seen cases settle at Mediation where none of the attorneys knew the facts. They just knew what their clients told them. Even more often, only one side was prepared a knew the facts. The other was just “shooting from the hip.”

At trial the unprepared were quickly demolished by the facts, the evidence. At Mediation you can get away with it. The evidence is often just one side’s contention. Why bother to learn the record when you can just BS your way through a mediation? The truth is what I say it is, nothing more. There is no cross-exam. Mediation is a “liars heaven,” although a good mediator can plow through that.

What happened to all the Trial Lawyers you might ask? Many became Mediators, including several of my good friends. A few started specializing in Mediation advocacy, where psychodrama and math are king (typically division). Mediation has become the everyday “Commodity” practice and trials are now the “Bet the Farm” rarity.

With less than one-percent of federal cases going to trial, it is a complete misnomer to keep calling ourselves Trial Lawyers. I know I have stopped calling myself that. Like it or not, that is reality. Our current system is designed to settle. It has become a relativistic opinion fest. It is not designed to determine final, binding objective truth. It is not designed to make findings of fact. It is instead designed to mediate ever more ingenious ways to split the baby. We no longer focus on the evidence, on the objective truth of what happened. We have lost our way.

Justice without Truth is Destabilizing

Justice without Truth is a mockery of Justice, a Post-Modern mockery at that, one where everything is relative. This is called Subjectivism, where one person’s truth is as good as another’s. All is opinion.

This relativistic kind of thinking was, and still is in most Universities, the dominant belief among academics. Truth is supposed to be relative and subjective, not objective, unless it happens to be science. Hard science is supposed to have a monopoly on objectivity. Unfortunately, this relativistic way of thinking has had some unintended consequences. It has led to the kind of political instability that we see in the U.S. today. That is the basic insight of a new book by Pulitzer Prize winner, Michiko Kakutani. The Death of Truth: Notes on Falsehood in the Age of Trump (Penguin, 2018). Also see Hanlin, Postmodernism didn’t cause Trump. It explains him. (Washington Post, 9/2/18).

Truth is truth. It is not just what the company with the biggest wallet says it is. It is not an opinion. Objective truth, the facts based on hard evidence, is real. It is not just an opinion. This video ad below by CNN was cited by Kakutani in her Death of Truth. It makes the case for objectivity in a simple, common sense manner. The political overtones are obvious.

There is a place for the insights of Post-Modern Subjectivism, especially as it concerns religion. But for now the objective-subjective pendulum has swung too far into the subjective. The pause between directions is over and it is starting to swing back. Facts and truth are becoming important again. This point in legal history will, I predict, be marked by the Mueller investigation. Evidence is once again starting to sing in our justice system. It is singing the body electric. The era of E-Vestigations has begun!

What are E-Vestigations?

E-Vestigations are confidential, internal investigations that focus on search of client data and metadata. They uses Artificial Intelligence to search and retrieve information relative to the client’s requested investigation, their information need. We use an AI machine training method that we call Hybrid Multimodal Predictive Coding 4.0. The basic search method is explained in the open-sourced TAR Course, but the Course does not detail how the method can be used in this kind of investigation.

E-Vestigation is done outside of Litigation and court involvement, usually to try to anticipate and avoid Litigation. Are the rumors true, or are the allegations just a bogus attempt to extort a settlement? E-Vestigations are by nature private, confidential investigations, not responses to formal discovery. AI Assisted investigations rely primarily on what the data says, not the rumors and suspicions, or even what some people say. The analysis of vast volumes of ESI is possible, even with millions of files, because e-Vestigations use Artificial Intelligence, both passive and active machine learning. Otherwise, the search of large volumes of ESI takes too long and is too prone to inaccuracies. That is the main reason this approach is far less expensive than traditional “full calorie” Muller type investigations.

The goal of E-Vestigation is to find quick answers based on the record. Interviews may not be required in many investigations and when they are, they are quick and, to the interviewee, mysterious. The answers to the information needs of a client are sometimes easily found. Sometimes you may just find the record is silent as to the issue at hand, but that silence itself often speaks volumes.

The findings and report made at the end of the E-Vestigation may clear up suspicion, or it may trigger a deeper, more detailed investigation. Sometimes the communications and other information found may require an immediate, more drastic response. One way or another, knowing provides the client with legitimate peace of mind.

The electronic evidence is most cases will be so overwhelming (we know what you said, to whom and when) that testimony will be superfluous, a formality. (We have your communications, we know what you did, we just need you to clear up a few details and help us understand how it ties into guys further up the power chain. That help will earn you a lenient plea deal.) This is what is happening right now, January 2019, with the investigation of Robert Mueller.

Defendants in criminal cases will still plea out, but based on the facts, on truth, not threats. Defendants in civil cases will do the same. So will the plaintiff in civil cases who makes unsubstantiated allegations. Facts and truth protect the innocent. Most of that information will be uncovered in computer systems. In the right hands, E-Vestigations can reveal all. It is a proactive alternative to Litigation with expensive settlements. The AI data review features of E-Vestigations make it far less expensive than a Muller investigations. Is it Mueller Time for your organization?

Robert Mueller never need ask a question of a witness to which he does not already know the answer based on the what the record said. The only real question is whether the witness will further compound their problems by lying. They often do. I have seen that several times in depositions of parties in civil cases. It is sheer joy and satisfaction for the questioner to watch the ethically challenged party sink into the questioner’s hidden traps. The “exaggerating witness” will often smile, just slightly, thinking they have you fooled, just like their own attorney. You smile back knowing their lies are now of record and they have just pounded another nail into their coffin.

E-Vestigations may lead to confrontation, even arrest, if the investigation confirms suspicions. In civil matters it may lead to employee discharge or accusations against a competitor. It may lead to an immediate out-of-court settlement. In criminal matters it may lead to indictment and an informed plea and sentencing. It may also lead to Litigation in civil matters with formal, more comprehensive discovery, but at least the E-Vestigating party will have a big head start. They will know the facts. They will know what specific information to ask for from the opposing side.

Eventually, civil suits will not be filed that often, except to memorialize a party’s agreement, such as a consent to a judgment. It will, instead, be a world where information needs are met in a timely manner and Litigation is thereby avoided. A world where, if management needs to know something, such as whether so and so is a sexual predator, they can find out, fast. A world where AI in the hands of a skilled legal team can mine internal data-banks, such as very large collections of employee emails and texts, and find hidden patterns. It may find what was suspected or may lead to surprise discoveries.

The secret mining of data, otherwise known as “reading other people’s emails without their knowledge” may seem like an egregious breach of privacy, but it is not, at least not in the U.S. under the computer use policies of most groups. Employees typically consent to this search as a condition of employment or computer use. Usually the employer owns all of the equipment searched. The employee has no ownership, nor privacy rights in the business related communications of the employer.

The use of AI assistants in investigations limits the exposure of irrelevant information to humans. First, only a few people are involved in the investigation at all because the AI does the heavy lifting. Second, the human reviewers are outside of the organization. Third, the AI does almost all of the document review. Only the AI reads all of the communications, not the lawyers. The humans look at far less than one percent of the data searched in most projects. They spend most of their time in study of the documents the AI has already identified as likely relevant.

The approach of limited investigations, of going in and out of user data only to search in separate, discreet investigations, provides maximum confidentiality to the users. The alternative, which some organizations have already adopted, is constant surveillance by AI of all communications. You can predict future behavior that way, to a point and within statistical limitations of accuracy. The police in some cities are already using constant AI surveillance to predict crimes and allocate resources accordingly.

I find this kind of constant monitoring to be distasteful. For me, it is too Big Brother and oppressive to have AI looking at my every email. It stifles creativity and, I imagine, if this was in place, would make me overly cautious in my communications. Plus, I would be very concerned about software error. If some baby AI is always on, always looking for suspicious patterns, it could make mistakes. The programming of the software almost certainly contains a number of hidden biases of the programmers, typically young white dudes.

The one-by-one investigation approach advocated here provides for more privacy protection. With E-Vestigations the surveillance is focused and time limited. It is not general and ongoing.

Five Virtues of E-Vestigations

Although I am not going to go into the proprietary details here of our E-Vestigations service (contact me through my law firm if you want to know more), I do want to share what I think are the five most important traits of our AI (robotic) assisted reviews: economics, confidentiality, stealth, speed and accuracy.

Confidentiality:

Complete Secrecy.
Artificial Intelligence means fewer people are required.
Employee Privacy Rights Respected.
Data need never leave corporate premises using specialized tools from our vendor.
Attorney-Client Privilege & Work Product protected.

Stealthiness:

Under the Radar Investigation.
Only some in client IT need know.
Sensitive projects. Discreet.
Stealth forensic copy and review of employee data.
Attorneys review off-site, unseen, via encrypted online connection.
Private interviews; only where appropriate.

Speed:

Techniques designed for quick results, early assessments.
Informal, high-level investigations. Not Litigation Discovery.
High Speed Document Review with AI help.
Example: Study of Clinton’s email server (62,320 files, 30,490 disclosed – 55,000 pgs.) is, at most, a one-week project with a first report after one day.

Accuracy:

Objective Findings and Analysis.
Independent Position.
Specialized Expertise.
Answers provided with probability range limitations.
Known Unknowns (Rumsfeld).

Clients are impressed with the cost of E-Vestigations, as compared to traditional investigations. That is important, of course, but the speed of the work is what impresses many. We produce results, use a flat fee to get there, and do so very FAST.

Certainly we can move much faster than the FBI reviewing email using its traditional methods of expert linear review. The Clinton email investigations took forever by our standards. Yet, Clinton’s email server had only 62,320 files, of which 30,490 were disclosed (around 55,000 pages.) This is, at most, a one-week E-Vestigations project with a first report after one day. Our projects are much larger. They involve review of hundreds of thousands of emails, or hundreds of millions. It does not make a big difference in cost because the AI, who works for free, is doing the heavy lifting of actual studying of all this text.

Most federal agencies, including the FBI, do not have the software, the search knowledge, nor attorney skills for this new type of AI assisted investigation. They also do not have the budget to acquire good AI for assist. Take a look at this selection from the official FBI collection of Clinton email and note that the FBI and US Attorneys office in Alexandra Virginia were communicating by fax in September 2015!

State and federal government agencies are not properly funded and cannot compete with private industry compensation. The NSA may well have an A-Team for advanced search, but not the other agencies. As we know, the NSA has their hands full just trying to keep track of the Russians and other enemies interfering with our elections, not to mention the criminals and terrorists.

Unintended Consequence of Mediation Was to Insert Subjectivism into the Law

As discussed, the rise and commoditization of Mediation over the last twenty years has had unintended consequences. The move from the courtroom to the mediator’s office in turn caused the Law to move from objective to subjective opinion. Discussion of the consequences of mediation, and the subjectivist attitude it brings, complicates my analysis of the death of Litigation, but is necessary. Litigation did not turn into private investigation work. One did not flow into another. Litigation is not changing directly into private Investigations, AI assisted or not. Mediation, and its unexpected consequences, is the intervening stage.

1. Litigation → 2. Mediation → 3. AI Assisted Investigations

Mediation brought down Litigation, at least the all important Trial part of Litigation, not AI or private investigations. There is never a judge making rulings at a mediation. There are only attorneys and assertions of what. Somebody must be lying, but with Mediation you never know who. Lawyers found they could settle cases without all that. They did not need the judge at all. At mediation there are no findings of fact, no rulings of law, just droll agreements as to who will pay how much to whom.

The next stage I predict of AI Assisted Investigations is filling a gap caused by the unintended consequence of Mediation. Mediation was never intended to spawn AI Assisted Investigations, no such thing even existed. It was not possible. We did not have the technology to do something like this. The forces driving the advent of AI Assisted Investigations, which I call E-Vestigations, have little to do with Mediation directly, but are instead the result of rapid advances in technology.

Mediation was intended to encourage settlement and reduce expensive trials. It has been wildly successful at that; exceeded all expectations. But this surprise success has also led to unexpected negative consequences. It has led to a new subjectivistic attitude in Litigation. It has led to the decline of evidence and an over-relativistic attitude where Truth was dethroned.

Most of my Mediator friends strongly disagree, but I have never heard a compelling argument to the contrary. The death of the trial is a stunning development. But mediation has had another impact. One that I have not seen discussed previously. It has not only killed trials, it has killed the whole notion of objective truth. It has led to a mediation mind-set where the “merits” are just a matter of opinion. Where cost of defense and the time value of money are the main items of discussion.

That foreseeable defect has led to the unforeseeable development of an AI Assisted alternative to Litigation. It is led to E-Vestigations. AI can now be used to help lawyers investigate and quickly find out the true facts of a situation.

Many lawyers who litigate today do not care what “really happened.” Very post-modern of them, but come on? A few lawyers just blindly believe whatever damned fool thing their client tells them. Most just say we will never know the absolute truth anyway, so let us just try our best to resolve the dispute and do what’s fair without any test of the evidence. They try to do justice with just a passing nod to the evidence, to the truth of what happened. I am not a fan. It goes against all of my core teachings as a young commercial litigation attorney who prepared and tried cases. It goes against my core values and belief. My opinion is that it is not all just opinion, that there is truth.

I object to that mediation, relativistic approach. After a life in the Law chasing smoking guns and taking depositions, I know for a fact that witnesses lie, that their memories are unreliable, all too human. But I also know that the writings made by and to these same witness often expose the lies, or, more charitably put, expose the errors in human memory. Fraudsters are human and almost always make mistakes. It is an investigator’s job to check the record to find the slip-ups in the con. (I dread the day when I have to try to trace a AI fraudster!)

I have been chasing and exposing con-men most of my adult life. I defended a few too. In my experience the truth has a way of finding its way out.

This is not an idealistic dream in today’s world of information floods. There is so much information, the real difficulty is in finding the important bits, the smoking guns, the needles. The evidence is usually there, but not yet found. The real challenge today is not in gathering the evidence, it is in searching for the key documents, finding the signal in the noise.

Conclusion

Objective accounts of what happened in the past are not only possible, they are probable in today’s Big Data world. Your Alexa or Google speakers may have part of the record. So too may your iWatch or Fitbit. Soon your refrigerator will too. Data is everywhere. Privacy is often an illusion. (Sigh.) The opportunity of liars and other scoundrels to “get away with it” and fool people is growing smaller every day. Fortunately, if lawyers can just learn a few new evidence search skills, they can use AI to help them find the information they need.

Juries and judges, for the most part, believe in objective truth. They are quite capable of sorting through competing versions and getting at the truth. Good judges and lawyers (and jurors) can make sure that happens.

As mentioned, many academics and sophisticates believe otherwise, that there is no such a thing as objective truth. They believe instead in Relativism. They are wrong.

As Michiko Kakutani said in her book, The Death of Truth:

The postmodernist argument that all truths are partial (and a function of one’s perspective) led to the related argument that there are many legitimate ways to understand or represent an event. . . .

Without commonly agreed-upon facts — not Republican facts and Democratic facts; not the alternative facts of today’s silo-world — there can be no rational debate over policies, no substantive means of evaluating candidates for political office, and no way to hold elected officials accountable to the people. Without truth, democracy is hobbled. The founders recognized this, and those seeking democracy’s survival must recognize it today.

It is possible to find the truth, objective truth. All is not just opinion and allegations. Accurate forensic reconstruction is possible today in ways that we could never have imagined before. So is AI assisted search. The record of what is happening grows larger every day. That record written electronically at the time of the events in question is far more reliable than our memories. We can find the truth, but for that need to look primarily to the documents, not the testimony. That is not new. That is wisdom upon which almost all trial lawyers agree.

The truth is attainable, but requires dedication and skilled efforts by everyone on a legal team to find it. It requires knowledge of course, and a proven method, but also impartiality, discipline, intelligence and a sense of empathy. It requires experience with what the AI can do, and just as important, what it cannot do. It requires common sense. Lawyers have that. Jurors have that.

Surely only a weak-minded minority are fooled by today’s televised liars. Most competent trial lawyers could persuade a sequestered jury to convict them. And convict they will, but that still will not cause of rebirth of Litigation. Its’ glory days are over. So too is its killer, Mediation, although its death will take longer (Mediation may not even have peaked yet).

Evidence speaks louder than any skilled mediator. Let the truth be told. Let the chips fall where they may. King Litigation is dead. Long live the new King, confidential, internal AI assisted E-Vestigations.

3 Comments | Evidence, informaton, knowledge, Lawyers Duties, Metadata, New Rules, Related Legal Webs, Review, Search, Spoliation/Sanctions, Technology, VENDORS, wisdom | Tagged: ai, computer assisted review, discovery, document review, ethics, evidence, legal profession, machine learning, Mueller, predictive coding, search, tar, too much information | Permalink
Posted by Ralph Losey

Document Review and Proportionality – Part Two

March 28, 2018

This is a continuation of a blog that I started last week. Suggest you read Part One before this.

Simplified Six Step Review Plan for Small and Medium Sized Cases or Otherwise Where Predictive Coding is Not Used

Here is the workflow for the simplified six-step plan. The first three steps repeat until you have a viable plan where the costs estimate is proportional under Rule 26(b)(1).

Step One: Multimodal Search

The document review begins with Multimodal Search of the ESI. Multimodal means that all modes of search are used to try to find relevant documents. Multimodal search uses a variety of techniques in an evolving, iterated process. It is never limited to a single search technique, such as keyword. All methods are used as deemed appropriate based upon the data to be reviewed and the software tools available. The basic types of search are shown in the search pyramid.

In Step One we use a multimodal approach, but we typically begin with keyword and concept searches. Also, in most projects we will run similarity searches of all kinds to make the review more complete and broaden the reach of the keyword and concept searches. Sometimes we may even use a linear search, expert manual review at the base of the search pyramid. For instance, it might be helpful to see all communications that a key witness had on a certain day. The two-word stand-alone call me email when seen in context can sometimes be invaluable to proving your case.

I do not want to go into too much detail of the types of searches we do in this first step because each vendor’s document review software has different types of searches built it. Still, the basic types of search shown in the pyramid can be found in most software, although AI, active machine learning on top, is still only found in the best.

History of Multimodal Search

Professor Marcia Bates

Multimodal search, wherein a variety of techniques are used in an evolving, iterated process, is new to the legal profession, but not to Information Science. That is the field of scientific study which is, among many other things, concerned with computer search of large volumes of data. Although the e-Discovery Team’s promotion of multimodal search techniques to find evidence only goes back about ten years, Multimodal is a well-established search technique in Information Science. The pioneer professor who first popularized this search method was Marcia J. Bates, and her article, The Design of Browsing and Berrypicking Techniques for the Online Search Interface, 13 Online Info. Rev. 407, 409–11, 414, 418, 421–22 (1989). Professor Bates of UCLA did not use the term multimodal, that is my own small innovation, instead she coined the word “berrypicking” to describe the use of all types of search to find relevant texts. I prefer the term “multimodal” to “berrypicking,” but they are basically the same techniques.

In 2011 Marcia Bates explained in Quora her classic 1989 article and work on berrypicking:

An important thing we learned early on is that successful searching requires what I called “berrypicking.” . . .

Berrypicking involves 1) searching many different places/sources, 2) using different search techniques in different places, and 3) changing your search goal as you go along and learn things along the way. . . .

This may seem fairly obvious when stated this way, but, in fact, many searchers erroneously think they will find everything they want in just one place, and second, many information systems have been designed to permit only one kind of searching, and inhibit the searcher from using the more effective berrypicking technique.

Marcia J. Bates, Online Search and Berrypicking, Quora (Dec. 21, 2011). Professor Bates also introduced the related concept of an evolving search. In 1989 this was a radical idea in information science because it departed from the established orthodox assumption that an information need (relevance) remains the same, unchanged, throughout a search, no matter what the user might learn from the documents in the preliminary retrieved set. The Design of Browsing and Berrypicking Techniques for the Online Search Interface. Professor Bates dismissed this assumption and wrote in her 1989 article:

In real-life searches in manual sources, end users may begin with just one feature of a broader topic, or just one relevant reference, and move through a variety of sources. Each new piece of information they encounter gives them new ideas and directions to follow and, consequently, a new conception of the query. At each stage they are not just modifying the search terms used in order to get a better match for a single query. Rather the query itself (as well as the search terms used) is continually shifting, in part or whole. This type of search is here called an evolving search.

Furthermore, at each stage, with each different conception of the query, the user may identify useful information and references. In other words, the query is satisfied not by a single final retrieved set, but by a series of selections of individual references and bits of information at each stage of the ever-modifying search. A bit-at-a-time retrieval of this sort is here called berrypicking. This term is used by analogy to picking huckleberries or blueberries in the forest. The berries are scattered on the bushes; they do not come in bunches. One must pick them one at a time. One could do berrypicking of information without the search need itself changing (evolving), but in this article the attention is given to searches that combine both of these features.

I independently noticed evolving search as a routine phenomena in legal search and only recently found Professor Bates’ prior descriptions. I have written about this often in the field of legal search (although never previously crediting Professor Bates) under the names “concept drift” or “evolving relevance.” See Eg. Concept Drift and Consistency: Two Keys To Document Review Quality – Part Two (e-Discovery Team, 1/24/16). Also see Voorhees, Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 36 Info. Processing & Mgmt 697 (2000) at page 714.

SIDE NOTE: The somewhat related term query drift in information science refers to a different phenomena in machine learning. In query drift the concept of document relevance unintentionally changes from the use of indiscriminate pseudorelevance feedback. Cormack, Buttcher & Clarke, Information Retrieval Implementation and Evaluation of Search Engines (MIT Press 2010) at pg. 277. This can lead to severe negative relevance feedback loops where the AI is trained incorrectly. Not good. If that happens a lot of other bad things can and usually do happen. It must be avoided.

Yes. That means that skilled humans must still play a key role in all aspects of the delivery and production of goods and services, lawyers too.

UCLA Berkeley Professor Bates first wrote about concept shift when using early computer assisted search in the late 1980s. She found that users might execute a query, skim some of the resulting documents, and then learn things which slightly changes their information need. They then refine their query, not only in order to better express their information need, but also because the information need itself has now changed. This was a new concept at the time because under the Classical Model Of Information Retrieval an information need is single and unchanging. Professor Bates illustrated the old Classical Model with the following diagram.

The Classical Model was misguided. All search projects, including the legal search for evidence, are an evolving process where the understanding of the information need progresses, improves, as the information is reviewed. See diagram below for the multimodal berrypicking type approach. Note the importance of human thinking to this approach.

See Cognitive models of information retrieval (Wikipedia). As this Wikipedia article explains:

Bates argues that searches are evolving and occur bit by bit. That is to say, a person constantly changes his or her search terms in response to the results returned from the information retrieval system. Thus, a simple linear model does not capture the nature of information retrieval because the very act of searching causes feedback which causes the user to modify his or her cognitive model of the information being searched for.

Multimodal search assumes that the information need evolves over the course of a document review. It is never just run one search and then review all of the documents found in the search. That linear approach was used in version 1.0 of predictive coding, and is still used by most lawyers today. The dominant model in law today is linear, wherein a negotiated list of keyword is used to run one search. I called this failed method “Go Fish” and a few judges, like Judge Peck, picked up on that name. Losey, R., Adventures in Electronic Discovery (West 2011); Child’s Game of ‘Go Fish’ is a Poor Model for e-Discovery Search; Moore v. Publicis Groupe & MSL Group, 287 F.R.D. 182, 190-91, 2012 WL 607412, at *10 (S.D.N.Y. Feb. 24, 2012) (J. Peck).

The popular, but ineffective Go Fish approach is like the Classical Information Retrieval Model in that only a single list of keywords is used as the query. The keywords are not refined over time as the documents are reviewed. This is a mono-modal process. It is contradicted by our evolving multimodal process, Step One in our Six-Step plan. In the first step we run many, many searches and review some of the results of each search, some of the documents, and then change the searches accordingly.

Step Two: Tests, Sample

Each search run is sampled by quick reviews and its effectiveness evaluated, tested. For instance, did a search of what you expected would be an unusual word turn up far more hits than anticipated? Did the keyword show up in all kinds of documents that had nothing to do with the case? For example, a couple of minutes of review might show that what you thought would be a carefully and rarely used word, Privileged, was in fact part of the standard signature line of one custodian. All his emails had the keyword Privileged on them. The keyword in these circumstances may be a surprise failure, at least as to that one custodian. These kind of unexpected language usages and surprise failures are commonplace, especially with neophyte lawyers.

Sampling here does not mean random sampling, but rather judgmental sampling, just picking a few representative hit documents and reviewing them. Were a fair number of berries found in that new search bush, or not? In our example, assume that your sample review of the documents with “Privileged” showed that the word was only part of one person’s standard signature on every one of their emails. When a new search is run wherein this custodian is excluded, the search results may now test favorably. You may devise other searches that exclude or limit the keyword “Privileged” whenever it is found in a signature.

There are many computer search tools used in a multimodal search method, but the most important tool of all is not algorithmic, but human. The most important search tool is the human ability to think the whole time you are looking for tasty berries. (The all important “T” in Professor Bates’ diagram above.) This means the ability to improvise, to spontaneously respond and react to unexpected circumstances. This mean ad hoc searches that change with time and experience. It is not a linear, set it and forget it, keyword cull-in and read all documents approach. This was true in the early days of automated search with Professor Bates berrypicking work in the late 1980s, and is still true today. Indeed, since the complexity of ESI has expanded a million times since then, our thinking, improvisation and teamwork are now more important than ever.

The goal in Step Two is to identify effective searches. Typically, that means where most of the results are relevant, greater than 50%. Ideally we would like to see roughly 80% relevancy. Alternatively, search hits that are very few in number, and thus inexpensive to review them all, may be accepted. For instance, you may try a search that only has ten documents, which you could review in just a minute. You may just find one relevant, but it could be important. The acceptable range of number of documents to review in Bottom Line Driven Review will always take cost into consideration. That is where Step-Three comes in, Estimation. What will it costs to review the documents found?

Step Three: Estimates

It is not enough to come up with effective searches, which is the goal of Steps One and Two, the costs involved to review all of the documents returned with these searches must also be considered. It may still cost way too much to review the documents when considering the proportionality factors under 26(b)(1) as discussed in Part One of this article. The plan of review must always take the cost of review into consideration.

In Part One we described an estimation method that I like to use to calculate the cost of an ESI review. When the projected cost, the estimate, is proportional in your judgment (and, where appropriate, in the judge’s judgment), then you conclude your iterative process of refining searches. You can then move onto the next Step-Four of preparing your discovery plan and making disclosures of that plan.

Step Four: Plan, Disclosures

Once you have created effective searches that produce an affordable number of documents to review for production, you articulate the Plan and make some disclosures about your plan. The extent of transparency in this step can vary considerably, depending on the circumstances and people involved. Long talkers like me can go on about legal search for many hours, far past the boredom tolerance level of most non-specialists. You might be fascinated by the various searches I ran to come up with the say 12,472 documents for final review, but most opposing counsel do not care beyond making sure that certain pet keywords they may like were used and tested. You should be prepared to reveal that kind of work-product for purposes of dispute avoidance and to build good will. Typically they want you to review more documents, no matter what you say. They usually save their arguments for the bottom line, the costs. They usually argue for greater expense based on the first five criteria of Rule 26(b)(1):

the importance of the issues at stake in this action;
the amount in controversy;
the parties’ relative access to relevant information;
the parties’ resources;
the importance of the discovery in resolving the issues; and
whether the burden or expense of the proposed discovery outweighs its likely benefit.

Still, although early agreement on scope of review is often impossible, as the requesting party always wants you to spend more, you can usually move past this initial disagreement by agreeing to phased discovery. The requesting party can reserve its objections to your plan, but still agree it is adequate for phase one. Usually we find that after that phase one production is completed the requesting party’s demands for more are either eliminated or considerably tempered. It may well now to possible to reach a reasonable final agreement.

Step Five: Final Review

Here is where you start to carry out your discovery plan. In this stage you finish looking at the documents and coding them for Responsiveness (relevant), Irrelevant (not responsive), Privileged (relevant but privileged, and so logged and withheld) and Confidential (all levels, from just notations and legends, to redactions, to withhold and log. A fifth temporary document code is used for communication purposes throughout a project: Undetermined. Issue tagging is usually a waste of time and should be avoided. Instead, you should rely on search to find documents to support various points. There are typically only a dozen or so documents of importance at trial anyway, no matter what the original corpus size.

I highly recommend use of professional document review attorneys to assist you in this step. The so-called “contract lawyers” specialize in electronic document review and do so at a very low cost, typically in the neighborhood of $50 per hour. The best of them, who may often command slightly higher rates, are speed readers with high comprehension. They also know what to look for in different kinds of cases. Some have impressive backgrounds. Of course, good management of these resources is required. They should have their own management and team leaders. Outside attorneys signing Rule 26(g) will also need to supervise them carefully, especially as to relevance intricacies. The day will come when a court will find it unreasonable not to employ these attorneys in a document review. The savings is dramatic and this in turn increases the persuasiveness of your cost burden argument.

Step Six: Production

The last step is transfer of the appropriate information to the requesting party and designated members of your team. Production is typically followed by later delivery of a Log of all documents withheld, even though responsive or relevant. The withheld logged documents are typically: Attorney-Client Communications protected from disclosure under the client’s privilege; or, Attorney Work-Product documents protected from disclosure under the attorney’s privilege. Two different privileges. The attorney’s work-product privilege is frequently waived in some part, although often very small. The client’s communications with its attorneys is, however, an inviolate privilege that is never waived.

Typically you should produce in stages and not wait until project completion. The only exception might be where the requesting party would rather wait and receive one big production instead of a series of small productions. That is very rare. So plan on multiple productions. We suggest the first production be small and serve as a test of the receiving party’s abilities and otherwise get the bugs out of the system.

Conclusion

In this essay I have shown the method I use in document reviews to control costs by use of estimation and multimodal search. I call this a Bottom Line Driven approach. The six step process is designed to help uncover the costs of review as part of the review itself. This kind of experienced based estimate is an ideal way to meet the evidentiary burdens of a proportionality objection under revised Rules 26(b)(1) and 32(b)(2). It provides the hard facts needed to be specific as to what you will review and what you will not and the likely costs involved.

The six-step approach described here uses the costs incurred at the front end of the project to predict the total expense. The costs are controlled by use of best practices, such as contract review lawyers, but primarily by limiting the number of documents reviewed. Although it is somewhat easier to follow this approach using predictive coding and document ranking, it can still be done without that search feature. You can try this approach using any review software. It works well in small or medium sized projects with fairly simple issues. For large complex projects we still recommend using the eight-step predictive coding approach as taught in the TarCourse.com.

Comments Off on Document Review and Proportionality – Part Two | Evidence, Lawyers Duties, New Rules, Review, Search, Spoliation/Sanctions, Technology, VENDORS | Tagged: best practices, computer assisted review, costs, discovery, document review, legal search, PROPORTIONALITY, search | Permalink
Posted by Ralph Losey

Document Review and Proportionality – Part One

March 18, 2018

In 2013 I wrote a law review article on how the costs of document review could be controlled using predictive coding and cost estimation. Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data, 26 Regent U. Law Review 1 (2013-2014). Today I write on how it can be controlled in document review, even without predictive coding. Here is the opening paragraph of my earlier article:

The search of electronic data to try to find evidence for use at trial has always been difficult and expensive. Over the past few years, the advent of Big Data, where both individuals and organizations retain vast amounts of complex electronic information, has significantly compounded these problems. The legal doctrine of proportionality responds to these problems by attempting to constrain the costs and burdens of discovery to what are reasonable. A balance is sought between the projected burdens and likely benefits of proposed discovery, considering the issues and value of the case. Several software programs on the market today have responded to the challenges of Big Data by implementing a form of artificial intelligence (“AI”) known as active machine learning to help lawyers review electronic documents. This Article discusses these issues and shows that AI-enhanced document review directly supports the doctrine of proportionality. When used together, proportionality and predictive coding provide a viable, long-term solution to the problems and opportunities of the legal search of Big Data.

The 2013 article was based on version 1.0 Predictive Coding. Under this first method you train and rank documents and then review only the higher ranking documents. Here is a more detailed description from pages 23, 24 of the article:

This kind of AI-enhanced legal review is typically described today in legal literature by the term

predictive coding. This is because the computer predicts how an entire body of documents should be coded (classified) based on how the lawyer has coded the smaller training sets. The prediction places a probability ranking on each document, typically ranging from 0% to 100% probability. Thus, in a

relevancy classification, each and every document in the entire dataset (the corpus) is ranked with a percentage of likely relevance and irrelevance. …

As will be shown, this ranking feature is key to the use of the legal doctrine of proportionality. The ability to rank all documents in a corpus on probable relevance is a new feature that no other legal search software has previously provided.71

It was a two phase procedure: train then review. Yes, some review would take place in the first training phase, but this would be a relatively small number, say 10-20% of the total documents reviewed. Most of the human review of documents would take place in phase two. The workflow of version 1.0 is shown in the diagram below and is described in detail in the article, starting at page 31.

Predictive Coding and the Proportionality Doctrine argued that attorneys should scale the number of documents for the second phase of document review based on estimated costs constrained to a proportional amount. No more spending $100,000 for document review in a $200,000 case. The number of documents you selected for review would be limited to proportional costs. Predictive coding and its ranking features allowed you to select the documents for review that were most likely to be relevant. If you could only afford to spend $20,000 on a document review project, then you would limit the number of documents reviewed to those within that scope that were the highest ranked as probable relevant. Here is the article’s description at pages 54-55 of the process and link between the doctrine of proportionality and predictive coding.

What makes this a marriage truly made in heaven is the document-ranking capabilities of predictive coding. This allows parties to limit the documents considered for final production to those that the computer determines have the highest probative value. This key ranking feature of AI-enhanced document review allows the producing party to provide the requesting party with the most bang for the buck. This not only saves the producing party money, and thus keeps its costs proportional, but it saves time and expenses for the requesting party. It makes the production much more precise, and thus faster and easier to review. It avoids what can be a costly exercise to a requesting party to wade through a document dump 192, a production that contains a high number of irrelevant or marginally relevant documents. Most importantly, it gives the requesting party what it really wants—the documents that are the most important to the case.

In the article, pages 58-60, I called this method Bottom-Line-Driven Proportional Review and describe the process in greater detail.

The bottom line in e-discovery production is what it costs. Despite what some lawyers and vendors may say, total cost is not an impossible question to answer. It takes an experienced lawyer’s skill to answer, but,

after a while, you can get quite good at such estimation. It is basically a matter of estimating attorney billable hours plus vendor costs. With practice, cost estimation can become a reliable art, a projection that you can count on for budgeting purposes, and, as we will see, for proportionality arguments. …

The new strategy and methodology is based on a bottom line approach where you estimate what review costs will be, make a proportionality analysis as to what should be spent, and then engage in defensible culling to bring the review costs within the proportional budget. The producing party determines the number of documents to be subjected to final review by calculating backwards from the bottom line of what they are willing, or required, to pay for the production. …

Under the Bottom-Line-Driven Proportional approach, after analyzing the case merits and determining the maximum proportional expense, the responding party makes a good faith estimate of the likely

maximum number of documents that can be reviewed within that budget. The document count represents the number of documents that you estimate can be reviewed for final decisions of relevance, confidentiality, privilege, and other issues and still remain within budget. The review costs you estimate must be based on best practices, which in all large review projects today means predictive coding, and the estimates must be accurate (i.e., no puffing or mere guesswork).

Note this last quote (emphasis added) from Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data shows an important limitation to the article’s budgetary proposal, it was limited to large review projects where predictive coding was used. Without this marriage to predictive coding, the promise of proportionality by cost estimations was lost. My article today fills this gap.

Here I will explain how document review can be structured to provide estimates and review constraints, even when predictive coding and its ranking are not used. This is, in effect, the single lawyers guide, one where there has not been a marriage with predictive coding. It is a guide for small and medium sized document review projects, which are, after all, the vast majority of projects faced by the legal profession.

To be honest, back when I first wrote the law review article I did not think it would be necessary to develop such a proportionality method, one that does not use AI document ranking. I assumed predictive coding would take off and by now would be used in almost all projects, no matter what the size. I assumed that since active machine learning and document ranking was such good new technology, that even our conservative profession would embrace it within the next few years. Boy was I wrong about that. The closing lines of Predictive Coding and the Proportionality Doctrine: a Marriage Made in Big Data have been proven naive.

The key facts needed to try a case and to do justice can be found in any size case, big and small, at an affordable price, but you have to embrace change and adopt new legal and technical methodologies. The Bottom-Line-Driven Proportional Review method is part of that answer, and so too is advanced-review software at affordable prices. When the two are used together, it is a marriage made in heaven.

I blame both lawyers and e-discovery vendors for this failure, as well as myself for misjudging my peers. Law schools and corporate counsel have not helped much either. Only the judiciary seems to have caught on and kept up.

Proportionality as a legal doctrine took off as expected after 2013, but not the marriage with predictive coding. Lawyers have proven to be much more stubborn than anticipated. They will barely even go out with predictive coding, no matter how attractive she is, much less marry her. The profession as a whole remains remarkably slow to adopt new technology. The judges are tempted to use their shotgun to force a wedding, but so far have refrained from ordering a party to use predictive coding. Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP), 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016) (J. Peck: “There may come a time when TAR is so widely used that it might be unreasonable for a party to decline to use TAR. We are not there yet.”)

Changes Since 2013

A lot has happened since 2013 when Predictive Coding and the Proportionality Doctrine was written. In December 2015 Rule 26(b) on relevance was revised to strengthen proportionality and we have made substantial improvements to Predictive Coding methods. In the ensuing years most experts have abandoned this early two-step method of train then review in favor of a method where training continues throughout the review process. In other words, today we keep training until the end. See Eg. the e-Discovery Team’s Predictive Coding, version 4.0, with its Intelligently Spaced Training. (This is similar to a method popularized by Maura Grossman and Gordon Cormack, which they called Continuous Active Learning or CAL for short, a term they later trademarked.)

The 2015 revision to Rule 26(b) on relevance has spurred case law and clarified that undue burden, the sixth factor of proportionality under Rule 26(b)(1), must be argued in detail with facts proven.

the importance of the issues at stake in this action;
the amount in controversy;
the parties’ relative access to relevant information;
the parties’ resources;
the importance of the discovery in resolving the issues; and
whether the burden or expense of the proposed discovery outweighs its likely benefit.”

Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, No. 11-cv-1049 (PLF/GMH), 2017 WL 4011136, (D.D.C. Sept. 11, 2017). Although all factors are important and should be addressed, the last factor is usually the most important one in a discovery dispute. It is also the factor that can be addressed generally for all cases and is the core of proportionality.

Proportional “Bottom Line Driven” Method of Document Review that Does Not Require Use of Predictive Coding

I have shared how I use predictive coding with continuous training in my TARcourse.com online instruction program. The eight-step workflow is shown below.

I have not previously shared any information on the document review workflow that I follow in small and medium seized cases where predictive coding is not used. The rest of this article will do so now.

Please note that I have a well-developed and articulated list of steps and procedures for attorneys in my law firm to follow in such small cases. I keep this as a trade-secret and will not reveal them here. Although they are widely known in my firm, and slightly revised and updated each year, they are not public. Still, any expert in document review should be able to create their own particular rules and implementation methods. Warning, if you are not such an expert, be careful in relying on these high-level explanations alone. The devil is in the details and you should retain an expert to assist.

Here is a chart summarizing the SIX-Step Workflow and six basic concepts that must be understood for the process to work at maximum efficiency.

The first three steps iterate with searches to cull out the irrelevant documents, and then culminate with Disclosures of the plan developed for Steps Five and Six, Final Review and Production. The sixth production step is always in phases according to proportional planning.

A key skill that must be learned is project cost estimation, including fees and expenses. The attorneys involved must also learn how to communicate with themselves, the vendors, opposing counsel and the court. Rigid enforcement of work-product confidentiality is counter-productive to the goal of cost efficient projects. Agree on the small stuff and save your arguments for the cost-saving questions that are worth the effort.

The Proportionality Doctrine

he doctrine of proportionality as a legal initiative was launched by The Sedona Conference in 2010 as a reaction to the exploding costs of e-discovery. The Sedona Conference, The Sedona Conference Commentary on Proportionality in Electronic Discovery, 11 SEDONA CONF. J. 289, 292–94 (2010). See also John L. Carroll, Proportionality in Discovery: A Cautionary Tale, 32 CAMPBELL L. REV. 455, 460 (2010) (“If courts and litigants approach discovery with the mindset of proportionality, there is the potential for real savings in both dollars and time to resolution.”); Maura Grossman & Gordon Cormack, Some Thoughts on Incentives, Rules, and Ethics Concerning the Use of Search Technology in E-Discovery, 12 SEDONA CONF. J. 89, 94–95, 101–02 (2011).

The doctrine received a big boost with the adoption of the 2015 Amendment to Rule 26. The rule was changed to provide discovery must be both relevant and “proportional to the needs of the case.” Fed. R. Civ. P. 26(b)(1). To determine whether a discovery request is proportional, you are required weigh the following six factors: “(1) the importance of the issues at stake in this action; (2) the amount in controversy; (3) the parties’ relative access to relevant information; (4) the parties’ resources; (5) the importance of the discovery in resolving the issues; and (6) whether the burden or expense of the proposed discovery outweighs its likely benefit.” Williams v. BASF Catalysts, LLC, Civ. Action No. 11-1754, 2017 WL 3317295, at *4 (D.N.J. Aug. 3, 2017) (citing Fed. R. Civ. P. 26(b)(1)); Arrow Enter. Computing Solutions, Inc. v. BlueAlly, LLC, No. 5:15-CV-37-FL, 2017 WL 876266, at *4 (E.D.N.C. Mar. 3, 2017); FTC v. Staples, Inc., Civ. Action No. 15-2115 (EGS), 2016 WL 4194045, at *2 (D.D.C. Feb. 26, 2016).

“[N]o single factor is designed to outweigh the other factors in determining whether the discovery sought is proportional,” and all proportionality determinations must be made on a case-by-case basis. Williams, 2017 WL 3317295, at *4 (internal citations omitted); see also Bell v. Reading Hosp., Civ. Action No. 13-5927, 2016 WL 162991, at *2 (E.D. Pa. Jan. 14, 2016). To be sure, however, “the amendments to Rule 26(b) do not alter the basic allocation of the burden on the party resisting discovery to—in order to successfully resist a motion to compel—specifically object and show that . . . a discovery request would impose an undue burden or expense or is otherwise objectionable.” Mir v. L-3 Commc’ns Integrated Sys., L.P., 319 F.R.D. 220, 226 (N.D. Tex. 2016), as quoted by Oxbow Carbon & Minerals LLC v. Union Pacific Railroad Company, No. 11-cv-1049 (PLF/GMH), 2017 WL 4011136, (D.D.C. Sept. 11, 2017).

The Oxbow case is discussed at length in my recent blog Judge Facciola’s Successor, Judge Michael Harvey, Provides Excellent Proportionality Analysis in an Order to Compel (e-Discovery Team,3/1/18). Judge Harvey carefully examined the costs and burdens claimed by plaintiffs and rejected the overly burdensome argument.

Plaintiffs’ counsel explained at the second hearing in this matter that Oxbow has spent $1.391 million to date on reviewing and producing approximately 584,000 documents from its nineteen other custodians and Oxbow’s email archive. See 8/24/17 TR. at 44:22-45:10. And again, Oxbow seeks tens of millions of dollars from Defendants. Through that lens, the estimated cost of reviewing and producing Koch’s responsive documents—even considering the total approximate cost of $142,000 for that effort, which includes the expense of the sampling effort—while certainly high, is not so unreasonably high as to warrant rejecting Defendants’ request out of hand. See Zubulake v. UBS Warburg, LLC, 217 F.R.D. 309, 321 (S.D.N.Y. 2003) (explaining, in the context of a cost-shifting request, that “[a] response to a discovery request costing $100,000 sounds (and is) costly, but in a case potentially worth millions of dollars, the cost of responding may not be unduly burdensome”); Xpedior Creditor Trust v. Credit Suisse First Boston (USA), Inc., 309 F. Supp. 2d 459, 466 (S.D.N.Y. 2003) (finding no “undue burden or expense” to justify cost-shifting where the requested discovery cost approximately $400,000 but the litigation involved at least $68.7 million in damages). …

In light of the above analysis—including the undersigned’s assessment of each of the Rule 26 proportionality factors, all of which weigh in favor of granting Defendants’ motion—the Court is unwilling to find that the burden of reviewing the remaining 65,000 responsive documents for a fraction of the cost of discovery to date should preclude Defendants’ proposed request. See BlueAlly, 2017 WL 876266, at *5 (“This [last Rule 26] factor may combine all the previous factors into a final analysis of burdens versus benefits.” (citing Fed. R. Civ. P. 26 advisory committee’s notes)).

For more analysis and case law on proportionality see Proportionality Φ and Making It Easy To Play “e-Discovery: Small, Medium or Large?” in Your Own Group or Class, (e-Discovery Team, 11/26/17). Also see The Sedona Conference Commentary on Proportionality, May 2017.

Learning How to Estimate Document Review Costs

The best way to determine a total cost of a project is by projection from experience and analysis on a cost per file basis. General experience of review costs can be very helpful, but the gold standard comes from measurement of costs actually incurred in the same project, usually after several hours of work, or days, depending on the size of the project. You calculate costs incurred to date and then project forward on a cost per file basis. The is the core idea of the Six Step document review protocol that this article begins to explain.

The actual project costs are the best possible metrics for estimation. Apparently that was never done in Oxbow because the plaintiffs counsel’s projected document review cost estimates varied so much. A per file cost analysis of the information in the Oxbow opinion shows that the parties missed a key metric. The costs projected ranged from an actual cost of $2.38 per file for the first 584,000 files, to an 1.17 per file estimate to review 214,000 additional files, to an estimate of $1.73 per file to review 82,000 more files, to an actual cost of $4.74 per file to review 12,074 files, to final estimate of $1.22 per file to review the remaining 69,926 files. The actual costs are way higher than the estimated costs meaning the moving party cheated themselves by failing to do the math.

Here is how I explained the estimation process in Predictive Coding and the Proportionality Doctrine at pages 60-61:

Under the Bottom-Line-Driven Proportional approach, after analyzing the case merits and determining the maximum proportional expense, the responding party makes a good faith estimate of the likely maximum number of documents that can be reviewed within that budget. The document count represents the number of documents that you estimate can be reviewed for final decisions of relevance, confidentiality, privilege, and other issues and still remain within budget.

A few examples may help clarify how this method works. Assume a case where you determine a proportional cost of production to be $50,000, and estimate, based on sampling and other hard facts, that it will cost you $1.25 per file for both the automated and manual review before production of the ESI at issue … Then you can review no more than 40,000 documents and stay within budget. It is that simple. No higher math is required.

Estimation for bottom-line-driven review is essentially a method for marshaling evidence to support an undue burden argument under Rule 26(b)(1). Let’s run through it again with greater detail and make a simple formula to illustrate the process.

First, estimate the total number of documents remaining to be reviewed after culling by your tested keywords and other searches (hereinafter “T”). This is the most difficult step but is something most attorney experts and vendors are well qualified to help you with. Essentially “T” represents is the number of documents left unreviewed for Step Five, Final Review. These are the documents found in Steps One and Two, ECA Multimodal Search and Testing. These steps, along with the estimate calculation, usually repeat several times to cull-in the documents that are most likely relevant to the claims and defenses. The T – Total Documents Left for Review – are the documents in the final revised keyword search folders and concept, similarity search folders. The goal is to obtain a relevance precision in these folders greater than 50%, preferably at least 75%.

To begin an example hypothetical, assume that the total document count in the folders set-up for final review is 5,000 documents. T=5,000. Next count how many relevant and highly relevant files have already been located (hereinafter “R”). Assume for our example that 1,000 relevant and highly relevant documents have been found. R=1,000.

Next, look up the total attorney fees already incurred in the matter to date for the actual document search and review work by attorneys and paralegals (hereinafter collectively “F”). Include the vendor charges in this total related to the review, but excluding forensics and collection fees. To do this more easily, make sure that the time descriptions that your legal team inputs are clear on what fees are for review. Always remember that you may be required to provide an affidavit or testimony someday to support this cost estimate in a motion for protective order. For our example assume that a total in $1,500 in costs and fees have already been incurred for document search and review work only. F=$1,500. The F divided by R creates the cost per file. Here it is $1.50 per file (F/R).

Finally, multiply the cost per file (F/R) by the number of documents still remaining to be reviewed, T. In other words T * (F/R). Here that is 5,000 (T) times the $1.50 cost per file (F/R), which equals $7,500. You can then disclose this calculation to opposing counsel to help establish the reasonableness (proportionality) of your plan. Step Four – Work Product Disclosure. Note you are estimating a total spend here for this review project of $9,000; $1,500 already spent, plus an estimated additional $7,500 to complete the project.

There are many ways to calculate probable fees to complete document review project. This simple formula method has the advantage of being based on actual experience and costs incurred. It is also simple and easy to understand compared to most other methods. The method could be criticized for inflating expected costs by observing that the work initially performed to find relevant documents is usually slower and more expensive than concluding work to review the tested search folders. This is generally true, but is countered by the fact that: (1) many of the initial relevant documents found in ECA (Step-One) were “low hanging fruit” and easier to locate than what remains; (2) the precision rate of the documents remaining to be reviewed after culling – T – will be much higher than the document folders previously reviewed (the higher the precision rate, the slower the rate of review, because it takes longer to code a relevant document than an irrelevant document); and, (3) additional time is necessarily incurred in the remaining review for redaction, privilege analysis, and quality control efforts not performed in the review to date.

To be concluded … In the conclusion of this article I will review the Six Steps and complete discussion of the related concepts.

3 Comments | Lawyers Duties, New Rules, Review, Search, Technology, VENDORS | Tagged: best practices, discovery, document review, legal profession, legal search, PROPORTIONALITY, technology | Permalink
Posted by Ralph Losey

e-Discovery Team ®

Cautionary Tale from Brooklyn: Search Terms ‘Designed To Fail’

Document Review and Proportionality – Part Two

Document Review and Proportionality – Part One

Ralph’s Tweets

Ralph’s eDiscovery Books

About the Blogger

Related Legal Webs

Follow Blog by Email

Math Tools for QC

Blog Stats

Tour of Blog’s 9 Pgs

Blog Pages

Sedona Principles 3rd Ed

Good Old Days

BLOGGING MUSIC

e-Discovery Team ®

Cautionary Tale from Brooklyn: Search Terms ‘Designed To Fail’

Share this:

Like this:

It’s Mueller Time! The Emergence of a New Legal Service Based on Artificial Intelligence

Share this:

Like this:

Document Review and Proportionality – Part Two

Share this:

Like this:

Document Review and Proportionality – Part One

Share this:

Like this:

Ralph’s Tweets

Ralph’s eDiscovery Books

About the Blogger

Related Legal Webs

Follow Blog by Email

Math Tools for QC

Blog Stats

Tour of Blog’s 9 Pgs

Blog Pages

Sedona Principles 3rd Ed

Good Old Days

BLOGGING MUSIC