Lawyers’ Job Security in a Near Future World of AI, the Law’s “Reasonable Man Myth” and “Bagley Two” – Part Two

January 22, 2017

This is the second and concluding section to the two-part blog, Lawyers’ Job Security in a Near Future World of AI, the Law’s “Reasonable Man Myth” and “Bagley Two.” Click here to read Part One.

Robot_handshake

___________

Next consider Judge Haight’s closing words to the opinion dated December 22, 2016, Ruling On Plaintiff’s Motion To Compel; Bagely v. Yale, Civil Action No. 3:13-CV-1890 (CSH):

However, requiring this additional production, or a further deposition in case of need, is in keeping with a governing objective of the Federal Rules of Civil Procedure: “By requiring disclosure of all relevant information, the discovery rules allow ultimate resolution of disputed issues to be based on full and accurate understanding of true facts.” 6 Moore’s Federal Practice § 26.02 (Matthew Bender 3d ed.). 6

__________

6 While Yale may not welcome the measurement of its obligations in the case at bar by these principles, it is worth recalling that the treatise’s principal initial author, James Wm. Moore, was a towering figure on the faculty of Yale Law School. In his preface to the first edition (1938), Professor Moore referred to his effort “at all times to accord to the Rules the interpretation which is most likely to attain the general objective of the new practice: the settlement of litigation on the merits.” That is the interpretation this Ruling attempts to adopt.

william_moore_prof_yale

Prof. Moore (1905-1994)

Poor Yale. Moore’s Federal Practice is one of the most cited treatises in the law. James W. Moore was the author of the 34-volume Moore’s Federal Practice (2d ed., 1948) and the three-volume Moore’s Manual: Federal Practice & Procedure (1962). He was also the Sterling Professor Emeritus of Law at Yale University, where he taught for 37 years. Who else but Yale can have anything in Moore’s thirty-four volume treatise held against them personally? Seems kind of funny, but I am sure Yale’s attorneys were not laughing.

Getting back to the case and Judge Haight’s decision. Aside from showing the malleability and limits of reason, Bagley Two provides some important new precedent for e-discovery, namely his rulings on privilege and the discoverability of a party’s preservation efforts. Judge Haight starts by repeating what is now established law, that a party’s preservation efforts are not satisfied by mere issuance of a notice, that a whole process is involved and the process must be reasonable. He then goes on to provide a pretty good list of the facts and circumstances that should be considered to determine reasonability.

[A] party’s issuance of a litigation hold notice does not put an end to the party’s obligation to preserve evidence; it is, rather, the first in a series of related steps necessary to ensure that preservation. As Magistrate Judge Francis aptly observed in Mastr Adjustable Rate Mortgages Trust 2006 v. UBS Real Estate Securities Inc., 295 F.R.D. 77, 85 (S.D.N.Y. 2013): “A litigation hold is not, alone, sufficient; instead compliance must be monitored.”

In spoliation cases involving litigation hold notices, one can discern from Second Circuit and district court opinions a number of decisive questions:

1. When did a party’s duty to preserve evidence arise?
2. Did the party issue a litigation hold notice in order to preserve evidence?
3. When did the party issue a litigation hold notice, in relation to the date its duty to preserve the evidence arose?
4. What did the litigation hold notice say?
5. What did recipients of the litigation hold notice do or say, in response to or as result of, the notice?
6. After receiving recipients’ responses to the litigation hold notice, what further action, if any, did the party giving the notice take to preserve the evidence?

Questions 2 through 6 are entirely fact-specific to a given case. Question 1 is a mixed question of law and fact, whose legal element the Second Circuit defined in Fujitsu Ltd. v. Federal Express Corp., 247 F.3d 423, 436 (2d Cir. 2001): “The obligation to preserve evidence arises when the party has notice that the evidence is relevant to litigation or when a party should have known that the evidence may be relevant to future litigation.”

In the case at bar, I am unable to accept Yale’s argument that the litigation hold notices it issued about Bagley and the recipients’ responses to the notices are immune from discovery because (in the absence of proof that spoliation had in fact occurred) such documents “are subject to the attorney-client and to work product privileges,” Defendants’ Brief [Doc. 192], at 3. That contention is something of a stretch. … . Assuming that all of Clune’s litigation hold notices were sent to employees of Yale, Clune was in effect communicating with his client. However, the predominant purpose of that communication was to give recipients forceful instructions about what they must do, rather than advice about what they might do. 3

I like the list of six key facts to consider to weigh the reasonability of preservation efforts, especially the last one. But my primary point here is the malleability of reason in classifying the notice as unprotected. A letter from in-house counsel telling employees that the law requires them to preserve is not advice entitled to privilege protection? It’s predominant purpose was instead unprotected instructions? The language of the litigation hold notices was earlier quoted in the opinion. It’s language included the following:

[A]ll members of the Yale faculty and staff who have information in their possession or control relating or referring in any way to Professor Bagley, her employment and teaching at SOM, or the circumstances relating to the non-renewal of her faculty appointment (collectively “this Matter”) have a legal obligation to preserve that information. The law imposes this obligation to prevent the loss of potential evidence during litigation. You must preserve and retain, and not alter, delete, remove, discard or destroy, directly or indirectly, any information concerning this Matter. Failure to preserve information could seriously undermine Yale’s legal position and lead to legal sanctions.

The lawyer’s letter tells employees that they “have a legal obligation to preserve,” and the legal consequences if they do not. Yet this letter is not advice because the predominant purpose is just an unprotected instruction? That is the holding.

mental_impressionsJudge Haight gets rid of work product protection too.

As for the work product doctrine, it “is not actually a privilege, but rather a qualified immunity from discovery,” codified in Fed. R. Civ. P. Rule 26(b)(3), whose purpose “is to protect an attorney’s mental processes so that the attorney can analyze and prepare for the client’s case without interference from an opponent.” 6 Moore’s Federal Practice, § 26.70[1] (Matthew Bender 3d ed.). 4 That purpose is not implicated by the present exercise.

__________

4 Fed. R. Civ. P. 26 (b)(3) of Civil Procedure protects from disclosure those materials which reveal “the mental impressions, conclusions, opinions, or legal theories of a party’s attorney.” See also In re Steinhardt Partners, L.P., 9 F.3d 230, 234 (2d Cir. 1993) (“At its core, the work product doctrine shelters the mental processes of the attorney, providing a privileged area within which he can analyze and prepare his client’s case.”) (quoting United States v. Nobles, 422 U.S. 225, 238 (1975)) (emphasis added).

I do not agree with Judge Haight on this aspect of his ruling. I think both work product and attorney client apply to these particular notices and his “reasoning”on this issue is wrong. I do, however, agree with his final ruling requiring production. I think the protections had been waived by the circumstances and actions of defense counsel, which, by the way, they were correct in doing. I think the waiver on their part was necessary. Judge Haight also mentioned waiver, but as dicta alternative grounds in footnote three:

3 The Court also notes that to the extent that Yale’s litigation hold notices included the text of the exemplar provided to Plaintiff as “document preservation notices,” that text has already been revealed publicly in this case, so that secrecy or privilege relating to that language was destroyed or waived. See Doc. 191-1, Ex. F.

triggerJudge Haight then looks at the question of when Yale’s duty to preserve commenced. Recall Yale kept adding custodians in eight stages. The first were pre-litigation notices. They were made, I note, after Yale’s lawyer mental processes told him that litigation was reasonably likely. The last were made after suit was filed, again based on the lawyer’s mental processes causing him to believe that these additional witnesses might have relevant evidence. The mental processes of Plaintiff’s attorneys led them to believe that all of the notices, including the pre-litigation notices, were sent too late and thus spoliation was likely. Here is Judge Haight’s analysis of the trigger issue:

When, during the course of this melancholy chain of events, should Yale have known that evidence pertinent to Bagley’s reappointment might be relevant to future litigation? That is a crucial question in spoliation analysis. A state of reasonable anticipation clearly antedates the actual filing of a complaint; in Fujitsu, 247 F.3d at 436, the Second Circuit was careful to couple actual present and possible future litigation as catalysts of equal strength for the preservation of evidence.

Bagley has not yet formally moved for spoliation sanctions, and so the question is not yet before me for decision, but some preliminary, non-binding observations may be made. The record previously made in the case shows that Bagley’s personal distress and institutional disapproval and distrust grew throughout the winter and spring of 2012 (the last year of her five-year appointment), so that when on May 24, 2012, Dean Snyder told Bagley that she would not be reappointed, it would not be irrational to suppose that Bagley might soon transform herself from disheartened academic to vengeful litigant. In fact, Bagley filed an internal discrimination complaint against Yale during the following month of June 2012 (which had the effect of bringing Provost Salovey out of the wings and onto the stage).

Predictable_IrrationalNote the Judge’s use of the phrase not be irrational to suppose. What is the impact of hindsight bias on this supposedly objective, rational analysis? Bagley’s later actions made it obvious that she would sue. She did sue. The law suit has been very contentious. But was it really all that obvious back in 2012 that Yale would end up in the federal courthouse? I personally doubt it, but, admit it is a close judgment call. We lawyers say that a lot. All that phrase really means is that reason is not objective. It is in the eye of the beholder.

Judge Haight then wraps up his analysis in Bagley Two.

What happened in this case is that Yale identified 65 individuals who might have evidence relevant to Bagley’s denial of reappointment, and issued them litigation hold notices in eight separate batches, a process that took a considerable amount of time. The first nine notices were sent nine months after Snyder told Bagley she would not be reappointed. The last was sent eight months after Bagley filed this action. To characterize the pace of this notification process as culpable or even negligent would be premature on the present record, but it is fair to say that it was leisurely, to an extent making it impossible to dismiss as frivolous Bagley’s suggestion that she might move for a spoliation sanction. The six questions outlined supra arise in this case, and the factors pertinent to resolving them include an unreasonable delay in issuing the notices and a subsequent failure to implement and monitor the recipients’ responses. Judge Sweet said in Stimson that the Second Circuit has left open “the question of whether a sufficiently indefensible failure to issue a litigation hold could justify an adverse inference on its own,” and an additional factor would be “the failure to properly implement the litigation hold even after it was issued.” 2016 WL 54684, at *6. These are legitimate questions in the case at bar. Bagley is entitled to discovery with respect to them. 5 (footnote citations omitted)

I certainly agree with Judge Haight on all of those points and law. Those factual circumstances do justify the modest amount of discovery requested by the plaintiff in this motion.

gavelNow we get to the actual Order on the pending motion to compel:

Therefore I conclude that in the circumstances of this case, Bagley’s “Motion to Compel” [Doc. 190] is GRANTED. Bagley is entitled to examine the litigation hold notices issued by Yale, and the responsive survey forms that notice recipients returned to Yale. These documents bear directly upon the questions courts identify as dispositive in spoliation cases. Bagley is entitled to discovery in these areas, in order to discern the merit or lack of merit of a formal claim for spoliation claim. To the extent that Yale objects to production of these documents on the grounds of privilege or the work product doctrine, the objections are OVERRULED.

For the same reasons, Bagley is also entitled to an affidavit from a Yale officer or employee (not a notice recipient or recipients) which describes what non-ESI documents Yale received from notice recipients and what was done with them. On a spoliation claim, Bagley will ultimately bear the burden of showing that pertinent evidence was destroyed or rendered unavailable. This discovery may cast light on that disputed issue. Yale may prefer not to have to produce that information; Yale’s counsel miss no opportunity to remind the Court how much discovery effort the case has previously required.

Judge Haight then ended his opinion with the previously quoted zinger regarding Yale’s famous law Professor Moore. This zinger and comments about Yale’s leisurely efforts and Yale counsel’s missing no opportunities to remind the court tell a story of their own. It shows the emotional undertone. So too does his earlier noted comment about “spoliation” being a cardinal litigation vice, well known to practicing attorneys and judges, but “perhaps unfamiliar” to academics. I suspect this goes beyond humor.

Artificial Intelligence and the Future of Employment

robot_whispererI am sure legal reason will improve in the future and become less subjective, less subject to hidden irrationalities and prejudices. By using artificial intelligence our legal doctrines and decision making can be improved, but only if the human judges remain in charge. The same comment goes for all attorneys. In fact, it applies to all current employment.

The doom and gloom futurists disagree. They think AI will replace humans at their jobs, not empower them. They envision a future of cold automation, not man-machine augmentation. They predict wide-spread unemployment with a loss of half of our current employment. An University of Oxford study predicted that almost half of all U.S. jobs could be lost to automation in the next twenty years. Even the influential World Economic Forum predicts predicts that Five Million jobs could be lost by 2020. Five Million Jobs by 2020: the Real Challenge of the Fourth Industrial Revolution. Also seeThe Future of Jobs: Employment, Skills and Workforce Strategy for the Fourth Industrial Revolution (World Economic Forum, Jan. 2016).

A contrary view “augmentation” oriented group predicts the opposite, that at least as many new jobs will be created as lost. This is a subject of hot debate. See EgArtificial intelligence will save jobs, not destroy them (World Economic Forum, 1/19/17). Readers know I am in the half-full camp.

James Bessen: Law Prophet of the Future of Employment

james_bessonMany are like me and have an overall positive outlook, including James Bessen, an economist  and Lecturer in Law at the Boston University School of Law. Jim Bessen, who was a good hacker with an entrepreneurial background (he created the first WYSIWYG desktop publishing software), has researched the history of computer use and employment since 1980. Jim’s research has shown that for those who can keep up with technology, there will be new jobs to replace the ones lost. Bessen, How Computer Automation Affects Occupations: Technology, Jobs & Economics, Boston University School of Law Law & Economics Working Paper No. 15-49 (1/16/16). He also found that wages in occupations that use computers grow faster, not slower:

[B]ecause higher wage occupations use computers more, computer use tends to increase well-paid jobs and to decrease low-paid jobs. Generally, computer use is associated with a substantial reallocation of jobs, requiring workers to learn new skills to shift occupations.

Also see the article in The Atlantic magazine by Bessen, The Automation Paradox: When computers start doing the work of people, the need for people often increases, (The Atlantic, 1/19, 2016) where he said:

…workers will have greater employment opportunities if their occupation undergoes some degree of computer automation. As long as they can learn to use the new tools, automation will be their friend.

This is certainly consistent with what I have seen in the legal profession since I started practice in 1980.

james_bessenJames Bessen has also written a book on this, Learning by Doing: The Real Connection Between Innovation, Wages, and Wealth. (Yale U. Press 2015). In this book, Bessen, in his words:

… looks at both economic history and the current economy to understand how new technology affects ordinary workers and how society can best meet the challenges it poses.

He notes that major new technologies always require new human work skills and knowledge, and that today, as before, they are slow and difficult to develop. He also makes the observation, which is again consistent with my own experience as a tech-lawyer, that relevant technical knowledge “develops slowly because it is learned through experience, not in the classroom.” In his analysis that is because the new knowledge is not yet standardized. I agree. This is one reason my work has been focused on the standardization of the use of active machine learning in the search for electronic evidence; see for example Predictive Coding 4.0 and my experiments at the TREC conference on predictive coding methods sponsored by the National Institute of Standards and Technology. Also see: Electronic Discovery Best Practices. In spite of my efforts on standards and best practices for e-discovery, we are still in the early, rapidly changing, non-standardized stage of new technology. Bessen argues that employer policies and government policies should encourage such on-the-job learning and perfection of new methods.

Jim Bessen’s findings are starting to be discussed by many who are now concerned with the impact of AI on employment. See for instance, Andrea Willige’s article in the World Economic Forum concerning Davos for 2017Two reasons computers won’t destroy all the jobs (“jobs don’t disappear, they simply move up the skills and wage ladder. For workers to move up the ranks, they must acquire the necessary skillset.”).

Standardization v. On-the-Job Training

Moving on up requires new employment skills. It requires workers who can step-in, step-up, step-aside, step-narrowly, or step-forward. Only Humans Need Apply; Dean Gonsowski, A Clear View or a Short Distance? AI and the Legal Industry, and, Gonsowski, A Changing World: Ralph Losey on “Stepping In” for e-Discovery, (Relativity Blog) (Interview with references to the the 5-steps described in Only Humans Need Apply). Unless and until standardization emerges, and this is taught in a classroom, the new skills will be acquired by on-the-job learning only, sometimes with experienced trainers, but more often self-taught by trial and error.

Borg_Ralph_headI have been working on creating the perfect, standard method for electronic document review using predictive coding since Da Silva Moore. I have used trial and error and on-the-job learning, buttressed by spending a month a year over the last five years on scientific research and experiments with my own team (remember my Borg experiments and videos?) and with TREC, EDI and Kroll Ontrack. Borg Challenge: Report of my experimental review of 699,082 Enron documents using a semi-automated monomodal methodology (a five-part written and video series comparing two different kinds of predictive coding search methods); Predictive Coding Narrative: Searching for Relevance in the Ashes of EnronEDI-Oracle Study: Humans Are Still Essential in E-Discovery (LTN Nov., 2013); e-Discovery Team at TREC 2015 Total Recall Track, Final ReportTREC 2016 Total Recall Track NOTEBOOK.

predictive_coding_4-0_simpleAfter years we have finally perfected and standardized a highly effective method for document review using predictive coding. We call it Predictive Coding 4.0. This method is complete, well-tested, proven and standardized for my team, but not yet accepted by the industry. Unfortunately, industry acceptance of one lawyer’s method is very difficult (impossible?) in the highly competitive, still young and emerging field of electronic document review. I create a standard because I have to in my work, not because I unrealistically expect the industry to adopt it. The industry is still too young for that. I will continue with my on-the-job training, content with that, just as Bessen, Davenport and Kirby observe is the norm for all new technologies. Someday a standard will be generally accepted and taught in classrooms, but we are far from it.

Conclusion

There is more going on in Bagley Two than objective reason, even assuming such a thing exists. Experienced attorneys can easily read between the lines. Reasoned analysis is just the tip of the iceberg, or top of the pyramid, as I envisioned in the new model for Holistic Law outlined in my prior article, Scientific Proof.

There is far more to Senior District Judge Charles S. Haight, Jr., than his ability to be logical and apply reason to the facts. He is not just a “thinking machine.” He has wisdom from decades on the bench. He is perceptive, has feelings and emotions, good intuitions and, we can see, a sense of humor. The same holds true for most judges and lawyers, perhaps even law professors. We are all human and have many other capacities beyond what robots can be trained to do.

Jason_Ralph_RobotReason is just one of the things that we humans do, and, as the work of Professor Ariely has shown, it is typically full of holes and clouded by hidden bias. We need the help of computers to get reason done right, to augment our logic and reasoning skills. Do not try to compete with, nor exclude robots from tasks involving reason. You will ultimately lose that battle. Instead, work with the robots. Invite them in, but remain in control of the processes; use the AI’s abilities to enhance and enlarge your own.

I am sure legal reason will improve in the future and become less subjective. This will happen when more lawyers Step-In as discussed in Davenport and Kirby, Only Humans Need Apply and Dean Gonsowski, A Clear View or a Short Distance? AI and the Legal Industry, and A Changing World: Ralph Losey on “Stepping In” for e-Discovery

alex_hafezMany of us have stepped-in, to use Davenport and Kirby’s language, to manage the use of TAR and AI in document review, not just me. Consider, for instance attorney Alexander Hafez, currently a “Solutions Engineer” for FTI. He was the only other attorney featured in Only Humans Need Apply. Alex bootstrapped his way from minimum wage contract document reviewer, to his current large vendor consultant “step-in” job, by, in the book’s words, “educational bricolage” composed of on-the-job learning and “a specialized course of two and some autodidactic reading.” Id. pg. 144. There are thousands of lawyers in e-Discovery doing quite well in today’s economy. The use of AI and other advanced technologies is now starting to appear in other areas of the law too, including contract review, analysis and construction. See eg. Kira Systems, Inc.

Great-Depression_LitigatorsAs the other areas of the Law become as enhanced and augmented as e-discovery, we will see new jobs open up for the steppers. Old mechanistic law jobs will be replaced. That is for sure. There will be jobs lost in the legal economy. But if Davenport, Kirby and Bessen are correct, and I for one think they are, new, better paying jobs will be created to replace them. Still, for most luddite lawyers, young and old, who are unable to adapt and learn new technologies, the impact of AI on the Law could be devastating. 

Only the tech-savvy will be able to move up the skill and wage ladder by stepping-in to make the technology work right. I attained the necessary skill set to do this with legal technology by teaching myself, by “hacking around” with computers. Yes, it was difficult, but I enjoyed this kind of learning. My story of on the job self-learning is very common. Thus the name of Bessen’s book, Learning by DoingOthers might do better in a more structured learning environment, such as a school, but for the fact there currently is none for this sort of thing, at least in the Law. It falls between the cracks of law school and computer science. For now the self-motivated, self-learners will continue to lead the way.

brad_smith_microsoftNot only do we need to improve our thinking with machines, we need to contribute our other talents and efforts. We need to engage and expand upon the qualities of our job that are most satisfying to us, that meet our human nature. This uniquely human work requires what is sometimes called “soft skills.” This primarily includes the ability for good interpersonal communication, but also such things as the ability to work collaboratively, to adapt to a new set of demands, and to solve problems on the fly. Legal counseling is a prime example according to the general counsel of Microsoft, Brad Smith. Microsoft’s Top Lawyer Toasts Legal Secretaries (Bloomberg Law, 1/18/17). The top lawyer, once CEO of Microsoft, also opined:

Individuals need to learn new skills to keep pace, and this isn’t always easy.  Over the next decade this could become more daunting still, as technology continues to change rapidly.  There is a broadening need for new technical skills and stronger soft skills.  The ability – and opportunity – to continue learning has itself become more important.

Brad Smith, Constructing a Future that Enables all Americans to Succeed, (Dept. of Commerce guest blog, 11/30/16).

The Wikipedia article on “soft skills” lists ten basic skills as compiled by Heckman and Kautz, Hard Evidence on Soft Skills, Labour Econ. 2012 Aug 1; 19(4): 451–464.

  • Communication – oral, speaking capability, written, presenting, listening.
  • Courtesy – manners, etiquette, business etiquette, gracious, says please and thank you, respectful.
  • Flexibility – adaptability, willing to change, lifelong learner, accepts new things, adjusts, teachable.
  • Integrity – honest, ethical, high morals, has personal values, does what’s right.
  • Interpersonal skills – nice, personable, sense of humor, friendly, nurturing, empathetic, has self-control, patient, sociability, warmth, social skills.
  • Positive attitude – optimistic, enthusiastic, encouraging, happy, confident.
  • Professionalism – businesslike, well-dressed, appearance, poised.
  • Responsibility – accountable, reliable, gets the job done, resourceful, self-disciplined, wants to do well, conscientious, common sense.
  • Teamwork – cooperative, gets along with others, agreeable, supportive, helpful, collaborative.
  • Work ethic – hard working, willing to work, loyal, initiative, self-motivated, on time, good attendance.

soft-skills_cartoon

As Brad Smith correctly observed, the skills and tasks needed to keep pace with technology include these kinds of soft skills as well as new technological know-how, things like the best methods to implement new predictive coding software. The tasks, both soft and technical, are generally not overly repetitive and typically require some creativity, imagination, flexibility and inventiveness and, in my view, the initiative to exceed original parameters.

cute_robotA concerned lawyer with real empathy who counsels fellow humans is not likely to be replaced anytime soon by a robot, no matter how cute. There is no substitute for caring, human relationships, for comforting warmth, wit and wisdom. The calm, knowledgeable, confident presence of a lawyer who has been through a problem many times before, and assures you that they can help, is priceless. It brings peace of mind, relaxation and trust far beyond the abilities of any machine.

Stepping-in is one solution for those of us who like working with new technology, but for the rest of humanity, soft-skills are now even more important. Even us tech-types need to learn and improve upon our soft skills. The team approach to e-discovery, which is the basic premise of this e-Discovery Team blog, does not work well without them.

ralph_17_pallate_knife_2Brad Smith’s comment on the need for continued learning is key for everyone who wants to keep working in the future. It is the same thing that Bessen, Davenport and Kirby say. Continued learning is one reason I keep writing. It helps me to learn and may help others to learn too, as part of their “autodidactic reading” and “educational bricolage.” (How else would I learn those words?) According to Bessen’s, Davenport and Kirby’s research most of the key skills needed to keep pace can only be learned on-the-job and are usually self-taught. That is one reason online education is so important. It makes it easier than ever for otherwise isolated people to have access to specialized knowledge and trainers.


Lawyers’ Job Security in a Near Future World of AI, the Law’s “Reasonable Man Myth” and “Bagley Two” – Part One

January 15, 2017

bad-robotDoes the inevitable triumph of AI robots over human reason and logic mean that the legal profession is doomed? Will Watson be the next generation’s lawyer of choice? I do no think so and have written many articles on why, including two last year: Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead and the Holistic Lawyer is Born; and The Law’s “Reasonable Man,” Judge Haight, Love, Truth, Justice, “Go Fish” and Why the Legal Profession Is Not Doomed to be Replaced by Robots. In the Reasonable Man article I discussed how reasonability is the basis of the law, but that it is not objective. It depends on many subjective factors, on psychology. In the Scientific Proof article I continued the argument and argued:

The Law’s Reasonable Man is a fiction. He or she does not exist. Never has, never will. All humans, including us lawyers, are much more complex than that. We need to recognize this. We need to replace the Law’s reliance on reason alone with a more realistic multidimensional holistic approach.

Scientific Proof Article

brain_gears_NOTo help make my argument in the Scientific Proof article I relied on the analysis of Thomas H. Davenport and Julia Kirby in Only Humans Need Apply: Winners and Losers in the Age of Smart Machines (Harper 2016) and on the scientific work of Dan Ariely, a Professor of Psychology and Behavioral Economics at Duke University.

I cite to Only Humans Need Apply: Winners and Losers in the Age of Smart Machines to support my thesis:

Although most lawyers in the profession do not know it yet, the non-reasoning aspects of the Law are its most important parts. The reasoning aspects of legal work can be augmented. That is certain. So will other aspects, like reading comprehension. But the other aspects of our work, the aspects that require more than mere reason, are what makes the Law a human profession. These job functions will survive the surge of AI.

If you want to remain a winner in future Law, grow these aspects. Only losers will hold fast to reason. Letting go of the grip of the Reasonable Man, by which many lawyers are now strangled, will make you a better lawyer and, at the same time, improve your job security.

Also see Dean Gonsowski, A Clear View or a Short Distance? AI and the Legal Industry; and, Gonsowski, A Changing World: Ralph Losey on “Stepping In” for e-Discovery, (Relativity Blog).

Professor Ariely has found from many experiments that We’re All Predictably Irrational. In my article, Scientific ProofI point my readers to his many easily accessible video talks on the subject. I consider the implication of Professor Ariely’s research on the law:

Our legal house needs a new and better foundation than reason. We must follow the physicists of a century ago. We must transcend Newtonian causality and embrace the more complex, more profound truth that science has revealed. The Reasonable Man is a myth that has outlived its usefulness. We need to accept the evidence, and move on. We need to develop new theories and propositions of law that confirm to the new facts at hand. Reason is just one part of who we are. There is much more to us then that: emotion, empathy, creativity, aesthetics, intuition, love, strength, courage, imagination, determination – to name just a few of our many qualities. These things are what make us uniquely human; they are what separate us from AI. Logic and reason may end up being the least of our abilities, although they are still qualities that I personally cherish. …

Davinci_whole_manSince human reason is now known to be so unreliable, and is only a contributing factor to our decisions, on what should we base our legal jurisprudence? I believe that the Reasonable Man, now that he is known to be an impossible dream, should be replaced by the Whole Man. Our jurisprudence should be based on the reality that we are not robots, not mere thinking machines. We have many other faculties and capabilities beyond just logic and reason. We are more than math. We are living beings. Reason is just one of our many abilities.

So I propose a new, holistic model for the law. It would still include reason, but add our other faculties. It would incorporate our total self, all human attributes. We would include more than logic and reason to judge whether behavior is acceptable or not, to consider whether a resolution of a dispute is fair or not. Equity would regain equal importance.

A new schemata for a holistic jurisprudence would thus include not just human logic, but also human emotions, our feelings of fairness, our intuitions of what is right and just, and multiple environmental and perceptual factors. I suggest a new model start simple and use a four-fold structure like this, and please note I keep Reason on top, as I still strongly believe in its importance to the Law.

4-levels-Holistic_Law_pyramid

My Scientific Proof article included a call to action, the response to which has been positive:

The legal profession needs to take action now to reduce our over-reliance on the Myth of the Reasonable Man. We should put the foundations of our legal system on something else, something more solid, more real than that. We need to put our house in order before it starts collapsing around us. That is the reasonable thing to do, but for that very reason we will not start to do it until we have better motivation than that. You cannot get people to act on reason alone, even lawyers. So let us engage the other more powerful motivators, including the emotions of fear and greed. For if we do not evolve our work to focus on far more than reason, then we will surely be replaced.

cyborg-lawyer

AI can think better and faster, and ultimately at a far lower cost. But can AI reassure a client? Can it tell what a client really wants and needs. Can AI think out of the box to come up with new, creative solutions. Can AI sense what is fair? Beyond application of the rules, can it attain the wisdom of justice. Does it know when rules should be bent and how far? Does it know, like any experienced judge knows, when rules should be broken entirely to attain a just result? Doubtful.

I go on to make some specific suggestions, just to start the dialogue, and then closed with the following:

We must move away from over-reliance on reason alone. Our enlightened self-interest in continued employment in the rapidly advancing world of AI demands this. So too does our quest to improve our system of justice, to keep it current with the rapid changes in society.

Where we must still rely on reason, we should at the same time realize its limitations. We should look for new technology based methods to impose more checks and balances on reason than we already have. We should create new systems that will detect and correct the inevitable errors in reason that all humans make – lawyers, judges and witnesses alike. Bias and prejudice must be overcome in all areas of life, but especially in the justice system.

Computers, especially AI, should be able to help with this and also make the whole process more efficient. We need to start focusing on this, to make it a priority. It demands more than talk and thinking. It demands action. We cannot just think our way out of a prison of thought. We need to use all of our faculties, especially our imagination, creativity, intuition, empathy and good faith.

Reasonable Man Article

Reasonable_man_cloudTo help make my argument in the earlier blog, The Law’s “Reasonable Man,” Judge Haight, Love, Truth, Justice, “Go Fish” and Why the Legal Profession Is Not Doomed to be Replaced by Robots, I quoted extensively from an Order Denying Defendant’s Motion for Protective Order. The order arose out of a routine employment discrimination case. Bagely v. Yale, Civil Action No. 3:13-CV-1890 (CSH) (Doc. 108) (order dated April 27, 2015). The Order examined the “reasonability” of ESI accessibility under Rule 26(b)(2)(B) and the “reasonable” efforts requirements under Rule 26(b). I used language of that Bagley Order to help support my argument that there is far more to The Law than mere reason and logic. I also argued that this is a very good thing, for otherwise lawyers could easily be replaced by robots.

Another e-discovery order was entered in Bagley on December 22, 2016. Ruling On Plaintiff’s Motion To Compel. Bagely v. Yale, Civil Action No. 3:13-CV-1890 (CSH). Bagley Two again provokes me to write on this key topic. This second order, like the first, was written by Senior District Judge Charles S. Haight, Jr.. The eighty-six year old Judge Haight is becoming one of my favorite legal scholars because of his excellent analysis and his witty, fairly transparent writing style. This double Yale graduate has a way with words, especially when issuing rulings adverse to his alma mater. He is also one of the few judges that I have been unable to locate an online photo of, so use your imagination, which, by the way, is another powerful tool that separates us from AI juiced robots.

Lady JusticeI pointed out in the Reasonable Man article, and it bears repetition, that I am no enemy of reason and rationality. It is a powerful tool in legal practice, but it is hardly our only tool. It is one of many. The “Reasonable Man” is one of the most important ideas of Law, symbolized by the balance scales, but it is not the only idea. In fact, it is not even the most important idea for the Law. That honor goes to Justice. Lady Justice holding the scales of reason is the symbol of the Law, not the scales alone. She is usually depicted with a blindfold on, symbolizing the impartiality of justice, not dependent on the social status or position of the litigants.

My view is that lawyer reasoning should continue in all future law, but should augmented by artificial intelligence. With machines helping to rid us of hidden biases in all human reason, and making that part of our evaluation easier and more accurate, we are free to put more emphasis on our other lawyer skills, on the other factors that go into our evaluation of the case. These include our empathy, intuition, emotional intelligence, feelings, humor, perception (including lie detection), imagination, inventiveness and sense of fairness and justice. Reason is only one of many human capacities involved in legal decision making.

In Reasonable Man article I analyzed the first Bagley Order to help prove that point:

Bagley shows that the dividing line between what is reasonable and thus acceptable efforts, and what is not, can often be difficult to determine. It depends on a careful evaluation of the facts, to be sure, but this evaluation in turn depends on many subjective factors, including whether one side or another was trying to cooperate. These factors include all kinds of prevailing social norms, not just cooperativeness. It also includes personal values, prejudices, education, intelligence, and even how the mind itself works, the hidden psychological influences. They all influence a judge’s evaluation in any particular case as to which side of the acceptable behavior line a particular course of conduct falls.

In close questions the subjectivity inherent in determinations of reasonability is obvious. This is especially true for the attorneys involved, the ones paid to be independent analysts and objective advisors. People can, and often do, disagree on what is reasonable and what is not. They disagree on what is negligent and what is not. On what is acceptable and what is not.

All trial lawyers know that certain tricks of argument and appeals to emotion can have a profound effect on a judge’s resolution of these supposedly reason-based disagreements. They can have an even more profound affect on a jury’s decision. (That is the primary reason that there are so many rules on what can and cannot be said to a jury.)

lady_justice_not_blindIn spite of practical knowledge by the experienced, the myth continues in our profession that reasonability exists in some sort of objective, platonic plane of ideas, above all subjective influences. The just decision can be reached by deep, impartial reasoning. It is an article of faith in the legal profession, even though experienced trial lawyers and judges know that it is total nonsense, or nearly so. They know full well the importance of psychology and social norms. They know the impact of cognitive biases of all kinds, including, for example, hindsight biasSee Roitblat, The Schlemiel and the Schlimazel and the Psychology of Reasonableness (Jan. 10, 2014, LTN) (link is to republication by a vendor without attribution) (“tendency to see events that have already occurred as being more predictable than they were before they actually took place“); Also see Rimkus v Cammarata, 688 F. Supp. 2d 598 (S.D. Tex. 2010) (J. Rosenthal) (“It can be difficult to draw bright-line distinctions between acceptable and unacceptable conduct in preserving information and in conducting discovery, either prospectively or with the benefit (and distortion) of hindsight.” emphasis added); Pension Committee of the University of Montreal Pension Plan, et al. v. Banc of America Securities, LLC, et al., 685 F. Supp. 2d 456 (S.D.N.Y. Jan. 15, 2010 as amended May 28, 2010) at pgs. 463-464 (J. Scheindlin) (‘That is a judgment call that must be made by a court reviewing the conduct through the backward lens known as hindsight.” emphasis added).

In my conclusion to Reasonable Man article I summarized my thoughts and tried to kick off further discussion of this topic:

The myth of objectivity and the “Reasonable Man” in the law should be exposed. Many naive people still put all of their faith in legal rules and the operation of objective, unemotional logic. The system does not really work that way. Outsiders trying to automate the law are misguided. The Law is far more than logic and reason. It is more than the facts, the surrounding circumstances. It is more than evidence. It is about people and by people. It is about emotion and empathy too. It is about fairness and equity. It’s prime directive is justice, not reason.

That is the key reason why AI cannot automate law, nor legal decision making. Judge Charles (“Terry”) Haight could be augmented and enhanced by smart machines, by AI, but never replaced. The role of AI in the Law is to improve our reasoning, minimize our schlemiel biases. But the robots will never replace lawyers and judges. In spite of the myth of the Reasonable Man, there is far more to law then reason and facts. I for one am glad about that. If it were otherwise the legal profession would be doomed to be replaced by robots.

Bagley Two

Now let us see how Judge Haight once again helps prove the Reasonable Man points by his opinion in Bagley Two. Ruling On Plaintiff’s Motion To Compel (December 22, 2016), Bagely v. Yale, Civil Action No. 3:13-CV-1890 (CSH). In this opinion the reasonability of defendant Yale’s preservation efforts was considered in the context of a motion to compel discovery. His order again reveals the complexity and inherent subjectivity of all human reason. It shows that there are always multiple factors at work in any judge’s decision beyond just thought and reason, including an instinct born out of long experience for fairness and justice. Once again I will rely primarily on Judge Haight’s own words. I do so because I like the way he writes and because you need to read his original words to appreciate what I am talking about. But first, let me set the stage.

Reasonable_guageYale sent written preservation notices to sixty-five different people, which I know from thousands of matters is a very large number of custodians to put on hold in a single-plaintiff discrimination case. But Yale did so in stages, starting on March 1, 2013 and ending on August 7, 2014. Eight different times over this period they kept adding people to their hold list. The notices were sent by Jonathan Clune, a senior associate general counsel of Yale University. The plaintiff argued that they were too late in adding some of the custodians and otherwise attacked the reasonability of Yale’s efforts.

The plaintiff was not seeking sanctions yet for the suspected unreasonable efforts, they were seeking discovery from Yale as to details of these efforts. Specifically they sought production of: (1) the actual litigation hold notices; (2) the completed document preservation computer survey forms that were required to be returned to the Office of General Counsel by each Litigation Hold Recipient; and, (3) an affidavit detailing the retention and production for all non-ESI documents collected from each of the Litigation hold Recipients.

Yale opposed this discovery claiming any more information as to its preservation efforts was protected from discovery under the attorney-client privilege and attorney work product protection.  Yale also argued that even if the privileges did not apply here, the discovery should still be denied because to obtain such information a party must first provide convincing proof that spoliation in fact occurred. Yale asserted that the plaintiff failed to provide sufficient proof, or even any proof, that spoliation had in fact occurred.

Here is the start of Judge Haight’s evaluation of the respective positions:

Mr. Clune’s litigation hold notices stressed that a recipient’s failure to preserve pertinent documents could “lead to legal sanctions” against Yale. Clune was concerned about a possible sanction against Yale for spoliation of evidence. While Clune’s notices did not use the term, “spoliation” is a cardinal litigation vice, known by that name to trial lawyers and judges, perhaps unfamiliar to academics unable to claim either of those distinctions. Clune’s notices made manifest his concern that a trial court might sanction Yale for spoliation of evidence relevant to the University SOM’s decision not to reappoint Bagley to its faculty.

skull_bones_yaleNote the jab at academics. By the way, in my experience his observation is correct about the cluelessness of most law professors when it comes to e-discovery. But why does Judge Haight take the time here to point that out? This case did not involve the Law School. It involved the business school professors and staff (as you would expect). It is important to know that Judge Haight is a double Yale graduate, both undergraduate and law school. He graduated from Yale Law in 1955. He was even a member of Yale’s infamous of Skull and Bones society. (What does 322 really mean? Eulogia?) Perhaps there are some underlying emotions here? Judge Haight does seem to enjoy poking Yale, but he may do that in all his cases with Yale out of an eccentric kind of good humor, like a friendly shoulder punch. But I doubt it.

To be continued … 



Top Twenty-Two e-Discovery Opinions of 2016: Number Two

December 31, 2016

Here is the e-Discovery Team’s second most interesting e-discovery opinion of 2016: Dynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue, No. 2685-11, 8393-12, 2016 WL 4204067 (T.C. July 13, 2016). It was written by Ronald L. Buch, Judge of the United States Tax Court.

SECOND –  Dynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue

judge_ronald_-buchThe Dynamo Holdings case is of great interest to the e-Discovery Team because it is all about our favorite topic, the thing we do best, Predictive CodingDynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue, No. 2685-11, 8393-12, 2016 WL 4204067 (T.C. July 13, 2016). Judge Ronald Buch’s opinion contains an excellent discussion of the law and application of predictive coding.

Recall that this is the second opinion by Judge Buch on predictive coding in this case. In 2014 the Court approved the use of predictive coding. It was one of the first courts to do so after Judge Peck’s Da Silva Moore. Dynamo Holdings Limited P’ship v. Commissioner, 143 T.C. 183 (2014). In Dynamo One, Judge Buch first stated his views on predictive coding:

Predictive coding is an expedited and efficient form of computer-assisted review that allows parties in litigation to avoid the time and costs associated with the traditional, manual review of large volumes of documents.

Id. at 190. In Dynamo One Judge Buch granted the Commissioner’s motion and compelled the plaintiffs to produce the backup tapes, but also granted the plaintiffs’ request for permission to use predictive coding. Id. at 194. In Dynamo Two Judge Buch considered objections to the plaintiff’s predictive coding work and rejected the Commissioner’s motion for the plaintiff to redo the document review using keyword search. Dynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue, No. 2685-11, 8393-12, 2016 WL 4204067 (T.C. July 13, 2016).

With this background Dynamo Holdings Two had a good chance at the top ranked case of 2016 but for two things. One, the top of the opinion contains this statement: Pursuant to Tax Court Rule 50(f), orders shall not be treated as precedent, except as otherwise provided. Not the kind of statement you want to see on a great opinion like this, but there it is.

The second reason Dynamo Two has to settle for number two is the fact that the predictive coding methodology used by the parties in Dynamo was totally bonkers.  This is not the judge’s fault, of course. Indeed, the author of second most interesting e-Discovery opinion of 2016, did a good job of explaining the crazy, random based predictive coding protocol the attorneys in this case came up with. Also see Judge Buch’s Order Concerning ESI Discovery (Dec. 11, 2015) that sets forth more detail of their ill-informed, compromise protocol.

The e-Discovery Team’s interest in Dynamo Holdings is in spite of our misgivings concerning the way predictive coding was used in this case back in 2014. We cannot get past the pathetic random based methods for training document selection, not to mention the old-fashioned version 1.0 methods they used. Suffice it to say that predictive coding can be done far, far better than it was here, and that the Team, like the Defendant in Dynamo Holdings, has serious misgivings as to the predictive coding based document review done by the Plaintiffs. So to do many others. See: Tredennick & Gricks, Discussion About Dynamo Holdings: Is 43% Recall Enough?.

Our objection is not to the predictive coding software, or the idea of predictive coding, it is an objection as to the specific method of use. The Dynamo Holdings method had very little in common with today’s state of the art methodology. See PREDICTIVE CODING 4.0. (method shown in diagram below.) Still, putting this objection aside, it is the second most interesting e-discovery opinion of 2016.

predictive_coding_4-0_web

The Plaintiffs in this case, called in Tax Court lingo, the Petitioners, did the document production using a predictive coding method that the Defendant, the Commissioner of the Internal Revenue Service, obviously had a had in developing in a semi-cooperative fashion. (Reminds me of the explanation of the Camel, a Horse designed by a committee.) The attorneys for both sides were obviously responsible for the mess of a compromise method they came up with. If they had any actual predictive coding experts to advise them, and I assume they did, it is obvious that their advice was not followed. See eg: Predictive Coding 3.0 article, part one and part two (part one describes the history and part two describes the method. Superseded by Predictive Coding 4.0 article, but still good background); Latest Grossman and Cormack Study Proves Folly of Using Random Search For Machine Training – Part One,  Part Two,  Part Three, and Part Four.

When the Petitioners completed the document review with predictive coding, the Defendant basically objected to the work and said it was bad, unreasonably inadequate, and the result was “incomplete.” They argued that the Petitioner’s attorneys should keep on looking for relevant documents, but this time use keyword search, not predictive coding (as if they were mutually exclusive, which they are not). We agree with the defense that the Petitioner’s work was bad, due to poor choice of method. It probably was an incomplete response, as the Commissioner defendant put it. But we also agree with Judge Buch’s refusal to order the Petitioner to redo or supplement the project. The Commissioner’s attorney helped create this error, so we think some estoppel applies here. But the main reason we agree with Judge Buch’s denial of any relief is the Commissioner’s failure to provide proof of the inadequacy of the review, nor an alternative method that would be better.

judge_ronald_buchHere is how Judge Ronald Buch, obviously a very intelligent, well-informed judge, described the dispute resolved by this opinion:

The quality of that response (by Petitioner) is now before us. Using a process described in more detail below, petitioners responded to the discovery requests by using predictive coding. The Commissioner, believing the response to be incomplete, served petitioners with a new discovery request asking for all documents containing any of a series of search terms. (Those same search terms had been used in a Boolean search during the predictive coding process to identify how many documents in the electronic records had each term.) Petitioners objected to this new discovery request as duplicative of the previous discovery responses made through the use of predictive coding. On June 17, 2016, the Commissioner filed a motion under Tax Court Rules 72(b)(2)¹ to compel the production of documents responsive to the Boolean search that were not produced through the use of predictive coding. The petitioners object.

Now we get into the excellent language used by Judge Buch to describe the new technology, language that we fully endorse, but with a caveat that methods differ and some, as we see in Dynamo Holdings, are downright defective.

When responding to a document request, technology has rendered the traditional approach to document review impracticable. The traditional method is labor intensive, with people reviewing documents to discern what is (or is not) responsive, with the responsive documents then reviewed for privilege, and with the responsive and non-privileged documents being produced. When reviewing documents in the dozens, hundreds, or low thousands, this worked fine. But with the advent of electronic recordkeeping, documents no longer number in the mere thousands, and various electronic search methods have developed.

When electronic records are involved, perhaps the most common technique that is employed is to begin with keyword searches or Boolean searches to a defined universe of documents. Then, the responding party typically reviews the results of those searches to identify what, in fact, is responsive to the request.
Implicit in this approach is the fact that some of the documents that are responsive to the word or Boolean search are responsive, while others are not.

An emerging approach, and the approach authorized in this case in our Opinion at 143 T.C. 183, is to use predictive coding to identify those documents that are responsive. A few key points of that Opinion are worth highlighting.

First, the Court authorized the responding party (petitioners) to use predictive coding, but the Court did not, in either its Opinion or its subsequent Order of September 17, 2014, mandate how the parties proceed from that point. …

Second, the Court held open the issue of whether the resulting document production would be sufficient, expressly stating “If, after reviewing the results, respondent believes that the response to the discovery request is incomplete, he may file a motion to compel at that time.” hl. at 189, 194. To state the obvious, (1) it is the obligation of the responding party to respond to the discovery, and (2) if the requesting party can articulate a meaningful shortcoming in that response, then the requesting party can seek relief. We turn now to those two points.

The opinion then goes into the tedious task of describing the was predictive coding was used here. As mentioned, there is no educational value in reviewing that, except to say, do not use this so called method. It was all wrong and the discussion by both sides on recall and precision is incorrect and does not follow bona fide statistical analysis. See eg: Predictive Coding 3.0 article, part one and part two (part one describes the history and part two describes the method. Superseded by Predictive Coding 4.0 article, but still good background); Introducing “ei-Recall” – A New Gold Standard for Recall Calculations in Legal SearchPart One, Part Two and Part ThreeIn Legal Search Exact Recall Can Never Be KnownConcept Drift and Consistency: Two Keys To Document Review Quality, part one, part two and part three.

Judge Buch’s bottom line on the argument in Dynamo Two was correct, and in line with that of the e-Discovery Team, when he assumed that the predictive coding done in this case was “flawed,” but held that “the question remains whether any relief should be afforded.” It was a flawed method alright, but the Defendant failed to provide good cause to justify the relief sought of an expensive do-over using keyword search.

Judge Buch correctly surmised that the Defendant’s request for relief was predicate upon two false premises, which he colorfully calls “myths.”

Respondent’s motion is predicated on two myths.

The first is the myth of human review. As noted in The Sedona Conference Best Practices Commentary on the Use of Search & Information Retrieval Methods in E-Discovery: “It is not possible to discuss this issue without noting that there appears to be a myth that manual review by humans of large amounts of information is as accurate and complete as possible – perhaps even perfect – and constitutes the gold standard by which all searches should be measured.” 15 Sedona Conf. J. 214, 230 (2014). This myth of human review is exactly that: a myth. Research shows that human review is far from perfect. Several studies are summarized in Nicholas M. Pace & Laura Zakaras, RAND Corp., Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery (2012) at 55. To summarize even further, if two sets of human reviewers review the same set of documents to identify what is responsive, research shows that those reviewers will disagree with each on more than half of the responsiveness claims. As the RAND report concludes:

Taken together, this body of research shows that groups of human reviewers exhibit significant inconsistency when examining the same set of documents for responsiveness under conditions similar to those in large-scale reviews. Is the high level of disagreement among reviewers with similar backgrounds and training reported in all of these studies simply a function of the fact that determinations of responsiveness or relevance are so subjective that reasonable and informed people can be expected to disagree on a routine basis? Evidence suggests that this is not the case. Human error in applying the criteria for inclusion, not a lack of clarity in the document’s meaning or ambiguity in how the scope of the production demand should be interpreted, appears to be the primary culprit. In other words, people make mistakes, and, according to the evidence, they make them regularly when it comes to judging relevance and responsiveness.

Id. at 58. (Indeed, even keyword searches are flawed. One study summarized in Moore v. Publicis Groupe & MSL Grp., 287 F.R.D. 182, 191 (S.D.N.Y. 2012), found that the average recall rate based on a keyword review was only 20%.)

The second myth is the myth of a perfect response. The Commissioner is seeking a perfect response to his discovery request, but our Rules do not require a perfect response. Instead, the Tax Court Rules require that the responding party make a “reasonable inquiry” before submitting the response. Specifically, Rule 70(f) requires the attorney to certify, to the best of their knowledge formed after a “reasonable inquiry,” that the response is consistent with our Rules, not made for an improper purpose, and not unreasonable or unduly burdensome given the needs of the case. Rule 104(d) provides that “an evasive or incomplete * * * response is to be treated as a failure to * * * respond.” But when the responding party is signing the response to a discovery demand, he is not certifying that he turned over everything, he is certifying that he made a reasonable inquiry and to the best of his knowledge, his response is complete.

Likewise, “the Federal Rules of Civil Procedure do not require perfection.” Moore, 287 F.R.D. at 191. Like the Tax Court Rules, the Federal Rule of Civil – 9 – Procedure 26(g) only requires a party to make a “reasonable inquiry” when making discovery responses.

The fact that a responding party uses predictive coding to respond to a request for production does not change the standard for measuring the completeness of the response. Here, the words of Judge Peck, a leader in the area of e-discovery, are worth noting:

One point must be stressed – it is inappropriate to hold TAR [technology assisted review] to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using from using TAR for review.

Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 129 (S.D.N.Y. 2015).

We agree with this holding in Dynamo Two and the black letter rule that only reasonable efforts are required, not perfection. That applies to all human endeavors, including every duty imposed by the law. The real issue is whether the predictive coding method was so bad here that is use was per se unreasonable. The proof of unreasonable efforts in any document review project requires proof that actual relevant documents were overlooked and not produced. The defenses did not do that, they just said there were documents with keywords in them that were missed. They did not go on to say that these documents were highly relevant. Apparently the defense here did not grasp a basic fact, that the mere presence of a keyword in a document does not automatically make that document relevant.

The defense also failed to use experts to attack the dreadful method used here to incorporate active machine learning into the document search. We suspect that is because the IRS Commissioner attorneys were part of the problem. It was a practical estoppel.

The opinion by Judge Buch in Dynamo Two is important, and very interesting, because it shows that any method used, be it one that uses predictive coding or keywords, or a method like the e-Discovery Team uses where all methods – multimodal – are employed, including active machine learning (predictive coding), will be presumed reasonable until the challenging party proves otherwise. Again, the proof of unreasonability would, in our view, require proof of both relevant documents missed, namely highly relevant documents where the omission would matter, and proof by experts that the method used was flawed, including statistical evidence, with suggestions for a reasonable effort. None of this evidence and testimony was provided in Dynamo Holdings.

That argument is an outgrowth of what specialists living in the Sedona bubble sometimes call the “dancing ponies” argument. Relevant documents could be chosen by any method, even by a system where every document that dancing ponies happen to step on are considered relevant. (The related image is one where a large stack of paper documents, typically exams, are tossed in the air at a stairway and the papers that reach the bottom of the stair are relevant (or if an exam, graded with an “A”). Many students today suspect their professors of using this grading method.) Dancing ponies, or stair tossing, are truly ridiculous methods, irrational in the extreme, but if they works, so the argument goes, then the method does not matter. The argument of the dancing ponies than goes on to assume that whether a method works, or not, can be determined by statistical methods. Tredennick & Gricks, Discussion About Dynamo Holdings: Is 43% Recall Enough? (Gricks: “As long as validation evinces (sic – I think he meant evidences) a reasonable production, that should be the end of the inquiry.”). The later assumption is, however, predicated by the proper use of statistics for validation, which requires recall to be stated as a probability range, not a point, and is anyway suspect. In Legal Search Exact Recall Can Never Be Known (e-Discovery Team, Dec. 2014).

Even when the range and uncertainties inherent in random sampling are understood, the use of statistics alone to verify reasonability of efforts is questionable. There is just too much uncertainty, especially when low prevalence datasets are involved. Examination of the exact methods used must be included in any defense of process. Dancing pony methods must be attacked and shown for what they are, pseudoscience. At the same time, proof of missing key or highly relevant documents should be shown. That is true, or should be, for all methods, including keyword search and linear review, and also predictive coding. Judge Peck and Judge Buch are right, “it is inappropriate to hold TAR [technology assisted review] to a higher standard than keywords or manual review.” Dynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue, No. 2685-11, 8393-12, 2016 WL 4204067 (T.C. July 13, 2016).

Still, inherent in this equality statement is the underlying premise that all document review methods should be held to a standard of reasonable efforts. In an important case, where proportionality factors favor it, the standard of care should be high. The courts have, in general, been somewhat lax in this department so far, but we think that Dynamo Two is a herald of things to come. In the future we will see predictive coding cases, along with keyword and linear review cases, where the court holds the standard of proportional, reasonable efforts has not been met.

Document review is the essence of all e-discovery. It must be done right for the truth, the whole truth and nothing but the truth to be found. Just because you use some predictive coding in your document review does not mean that the review was done right. There are just as many ways to screw up predictive coding as there are to screw up keyword search or linear review. We conclude our review of Dynamo Holdings Two with the inverse statement to the one made by Judge Andrew Peck and approved by Judge Buch:

It is inappropriate to hold TAR [technology assisted review] to a lower standard than keywords or manual review. Doing so discourages parties from using TAR properly, with expert advice, active machine learning training methods that have been proven to be effective, and statistically correct sample verifications.

Predictive coding is not a magic pill. It is a powerful algorithm that brings active machine learning, a form of artificial intelligence, into legal document review. This requires a high degree of skill to be done properly. Some day this point will be made in a case like Dynamo Holdings where unreasonable, unskilled predictive coding methods are used. We foresee that the objecting party will not complain about the use of predictive coding, but rather HOW it was used. They will likely suggest an alternative predictive coding method, such as the e-Discovery Team’s Predictive Coding 4.0, including quality control and quality assurance tests, sampling and statistical analysis or Recall, Precision and F1. If they make this argument, and buttress it with a showing that important ESI was missed by the first amateur attempt, then we think the objection will be sustained and a redo required.

We suspect that such a ruling is still a few years down the road. In the meantime we should all exercise constant vigilance against bad science and even worse methods. When opposing counsel says they plan to use predictive coding, the appropriate response is great, what type of method will you use? When I am asked that, I hand them my 97 page description, Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow. Others may use that as well. You can call it the standard e-Discovery Team method. Sligh variations are expected to fit the particular assignment. In the future I expect many experts will come up with explanations and detailed description of the method they use. Right now I am the only one putting it out there. Still, I talk to lots these folks, and if they ever do get around to writing it up, it will not be very different from our Hybrid Multimodal method. Among the top experts in the field, including scientists and professors, there is wide spread agreement on basic methods.


%d bloggers like this: