Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead and the Holistic Lawyer is Born

July 17, 2016

brain_gears_NOLast month I wrote about the place of reason in the law. The Law’s “Reasonable Man,” Judge Haight, Love, Truth, Justice, “Go Fish” and Why the Legal Profession Is Not Doomed to be Replaced by Robots. That article discussed how reasonability is the basis of the law, but that it is not objective. It depends on many subjective factors, on psychology. This article elaborates on this key point. The Law’s Reasonable Man is a fiction. He or she does not exist. Never has, never will. All humans, including us lawyers, are much more complex than that. We need to recognize this. We need to replace the Law’s reliance on reason alone with a more realistic multidimensional holistic approach.

Although logic and reasoning are important, we have many other important capacities, including empathy, intuition and imagination. All of these capacities are required for the practice of law. That is why lawyers cannot be replaced by robots. The fact the Reasonable Man is a fiction is something we lawyers should celebrate, not sweep under the carpet.

Most human decisions are not even based on reason. Quaint notions to the contrary are derived from the 18th Century Age of Reason. They are completely out of touch with reality. They are contrary to what science today is telling us about how humans process information and reach decisions.

frozenbrainsScientific research shows that the cornerstone of the Law – Reasonability – is not solid granite many had thought. There are no hard gears in our head, just soft, gelatinous, pinkish-beige matter. (Our brain is only soft grey matter when dead.) The ratiocination abilities of the brain are just one small part of its many incredible capacities. (For example, MIT scientists have shown that we can identify images seen for as little as 13 milliseconds, 13/1,000ths of one second.) We are far more than just rational, and that is a good thing.

Going Beyond the Age of Enlightenment Into the Modern Era of Science

Only_Humans_Need_ApplyThis article will offer proof that the Law’s Reasonable Man is dead. This is a cause for optimism because, as noted, if we were just reason-based workers, then our functions would soon be automated. We would soon all be out of work. See eg. Thomas H. Davenport, Julia Kirby, Only Humans Need Apply: Winners and Losers in the Age of Smart Machines (Harper 2016). Although most lawyers in the profession do not know it yet, the non-reasoning aspects of the Law are its most important parts. The reasoning aspects of legal work can be augmented. That is certain. So will other aspects, like reading comprehension. But the other aspects of our work, the aspects that require more than mere reason, are what makes the Law a human profession. These job functions will survive the surge of AI.

If you want to remain a winner in future Law, grow these aspects. Only losers will hold fast to reason. Letting go of the grip of the Reasonable Man, by which many lawyers are now strangled, will make you a better lawyer and, at the same time, improve your job security.

In today’s AI driven economy we all need to change our work to include more of our human capacities than mere reason. This is transforming all work, not just legal. We are far more than a thinking machine. We must open our eyes and see the truth. That is the true meaning and ultimate conclusion of the Age of Enlightenment.

Schrodinger's CatScience based on reason and the experimental method has taken Man beyond the rational, has shown the limitations of reason. Just as the evidence from physics experiments forced scientists to go beyond Newtonian Causality, and required them to embrace the seemingly irrational truth of Relativity and Quantum Mechanics, so too must the Law now evolve its thinking and procedures. As proof for this proposition in this chapter I will proffer the testimony of one expert witness, a noted MIT and Duke University Psychologist and Behavioral Economist.

The Legal Profession Must Awaken from the Daydream of Rationality

Reasonable_guageMy last blog, The Law’s “Reasonable Man”, laid the foundation for the introduction of this evidence. I noted how the law is based on the assumption that people make reasoned decisions and are capable of acting in a reasonable manner. I offered preliminary evidence that this assumption is contrary to the findings of research psychologists. I referred to a recent article by one such psychologist, Herb Roitblat: The Schlemiel and the Schlimazel and the Psychology of Reasonableness (Jan. 10, 2014, LTN) (link is to republication by a vendor without attribution).

I will now offer further, more detailed proof that humans do not act solely out of reason. Some just delude themselves into thinking so. I will then argue that these findings require us to make fundamental reforms to our system of justice. These reforms will both improve the our justice system and ensure the survival of the legal profession. Lawyers will remain, but they will look and work much differently than they do today. They will be augmented by AI, but not automated and replaced by AI. Productivity and efficiency will go through the roof. Our system of justice will vastly improve. To get there the profession will first have to awaken from the daydream of rationality. This article is designed as a wake up call.

This evidence of logic’s limitations is abundant. With only a little search I am sure you will find much more proof than I will now proffer. Many of you already know this from long experience in the courtrooms and law offices of the world, but may now have heard of the scientific proof. The evidence proves that the old assumptions on human reasonability, assumptions built centuries ago when the Age of Reason first began, are false. The evidence shows that the Reasonable Man is a legal fiction.

As Exhibit “A” to the assumption busting proposition I rely on the work of Dan Ariely, a Professor of Psychology and Behavioral Economics at Duke University. As an introduction to his work I ask readers to stop and take a few minutes, right now, to watch the TED video by Professor Ariely, Are We In Control of Our Own Decisions? He refers to his many scientific experiments at MIT and Duke that show we are not in control of many of our own decisions, even seemingly simple ones. These experiments prove my point.

Predictable Irrationality and Swearing on Bibles

Need more proof? Then please consider additional testimony from Professor Ariely on predictable irrationality. It is on another video called We’re All Predictably Irrational. This discourse even mentions every e-discovery lawyer’s favorite company, Enron, and examines our basic moral code, our personal fudge factor. Dan has conducted many experiments on the all too human tendency to cheat and lie, if only just a little, and the moving grey line between acceptable and unacceptable behavior. This is the line that the Law is constantly asked to draw, and to evaluate. These psychological insights are important to all lawyers, especially discovery lawyers, of the “e” only type like me, or not. Again, please listen carefully and consider the implications of these findings on the Law.

One interesting finding from Professor Ariely’s scientific experiments on cheating, one that you can easily miss in the predictable irrationality video, is that asking people to swear on a Bible significantly reduces cheating. This even works for atheists! I kid you not. Perhaps we should bring back the old tradition of requiring all witnesses to swear on a bible before beginning their testimony?

Ralph_swearing_oath_bibleI have done this myself long ago when I was out taking depositions as a young lawyer. In the early eighties many court reporters in rural counties of Florida would still pull out a Bible before a deposition began (they all used to carry them around for that purpose, and yes, that was way before they started carrying around computers). The court reporter would then ask the deponent to raise their right hand and put their left hand on the Bible. All the witnesses I saw instantly complied, thinking erroneously that this was a legal requirement. They placed their hand on the Bible, some nervously, and some like they did that all the time. Then they were asked to solemnly swear on the Bible that they would tell the truth, the whole truth and nothing but the truth so help me God. They did as asked by the serious court reporter, and some seemed pretty impressed by the whole ceremony. I recall that overall the testimony from these witnesses was pretty good, meaning less lies than usual.

I only saw this done a few times, and, as a typical arrogant big city lawyer (yes, out in the rural areas where they were still doing this, they all thought of Orlando as a big city), I dismissed it as a quaint old custom. But now science shows that it works.

What are the implications of these findings about human behavior? Maybe we should bring back Bibles into the courtrooms? Or at least bring back a bunch of solemn oaths? If we do not require swearing on or to a Bible, due to Church and State, or whatever, then perhaps we should ask people giving testimony to swear on something else. Most anything seems to work, even if it does not really exist. Dan Ariely’s experiments found that it even worked to have MIT students swear on an honor code that didn’t exist. Maybe asking lawyers to swear on their ethics codes would work too? Maybe that is the next reform in the procedural rules we should push for. Maybe we should update Rule 603 of the Federal Rules of Evidence:

Before testifying, a witness must give an oath or affirmation to testify truthfully. It must be in a form designed to impress that duty on the witness’s conscience.

prisoner_ralph_chainsWe need to work on forms designed to impress today’s savvy witnesses. Maybe bringing back Bibles will work for some, or something custom-fit to the particular witnesses. Who knows, for a chemist, it might be the periodic table. For others it might be a picture of their mother. Maybe the oath should be administered by prisoners in chains and mention the penalties of imprisonment for perjury. I think that would be pretty effective. Have you ever seen prisoners in chains up close in the courtroom? A few judges I know used to handcuff and shackle fathers who were delinquent in child support payments before their hearings. I am told it had a very sobering effect. Some experiments with this should be conducted because our current systems are not working very well. We rarely impress witnesses enough to awaken their latent conscience, much less their lawyers.

Maybe we should also amend Rule 26(g) to add swearing and a reference to ethics codes? Maybe stronger, more impressive oaths by lawyers signing 26(g) discovery requests and responses would work. Perhaps that would magically make more all too human lawyers start taking the requirements of the rules more seriously.

Lord Phillips 2009Maybe we should follow the British and make our judges wear fancier robes and make our lawyers and judges wear wigs? (One of Ariely’s experiments found clothing had an impact on honesty.) Let us build even more impressive courtrooms while we are at it, and let’s not only say Your Honor, but how about Your Lordship too? Or Your Grace? Maybe all lawyers should start adding courtly formalities to their 26(f) conferences? I can just imagine defense attorneys beginning every one of their responsive statements with things like: “The right honorable attorney representing the plaintiffs in this proceeding has made a point with some validity, but …” Maybe that would motivate lawyer conduct that would in fact please the court?

Of course I jest, but Ariely’s work shows that irrational approaches have a better chance of success than appeals to abstract knowledge alone. Forget about using reason to appeal to lawyers to cooperate, we have all seen how far that gets us.

Doing the Right Things for the Wrong Reasons

Are you a die-hard rationalist and demand more proof that the Reasonable Man is a myth? More evidence? Then listen to Dan Ariely’s Doing The Right Things for The Wrong Reasons.

Professor Ariely talks about more of his experiments. They show how immediate, tangible, emotions and concrete facts are a much more powerful motivator than all abstract knowledge. This means that one sanctions case invoking fear will do much more to encourage cooperation than a thousand law review articles. In my experience judges that threaten harsh punishment, that are known not to tolerate discovery misconduct, tend to have fewer disputes. Now we know why. Fear is a more powerful motivator than reason. As he shows in the video, for some people a good glass of wine is a powerful motivator too.

Dan_Ariely_toastProfessor Ariely’s testimony in this video examines the big gap between everyone’s knowledge of what they should be doing, and what they actually are doing. The truth is, we often do not act reasonably. There are many other more powerful forces at work. One of the most important is environment, and thus my earlier comments on impressive courtrooms, wigs, courtly conduct, and the like. Brand names and price have the same kind of impact. Many clients are still impressed by the big-firm, fancy reception room syndrome. People tend to think about fine wine and lawyers in the same way.

In the second half of this testimony Dan Ariely started to share some of the solutions he has come up with to these problems, ways to trick yourself and others into doing the right thing. One such motivator is public recognition, pride. Remember his discussion regard Prius owners. So how about Cooperation awards of lawyers? Proportionality awards for judges, etc. Let’s award a whole lot of gold, silver and especially bronze medals. I am serious about this awards and recognition proposal. If you have any interest in funding such awards, or otherwise being involved, please let me know. This would be a good opportunity for vendors in the legal space, especially e-discovery vendors.

Mere intellectual appeals to change behavior are almost useless. You have to persuade the whole human, and that requires addressing emotions and many other subconscious factors. That requires far more than abstract, knowledge-based writings.

The Power of Emotions and the Myth of Reasoned Behavior

Judge_Paul_GrimmThe power of emotions, and immediate gratification, should never be underestimated. This includes the positive motivators, like praise and recognition. An active judiciary can do much more to impact reasonable, ethical conduct than all appeals to reason. Judges need to be in your face, with both criticism and praise, stick and carrot. The motivations need to be immediate and real, not abstract and future oriented. See eg. Victor Stanley and the impact of Judge Grimm’s threats of immediate imprisonment of Pappas, the ultimate hide-the-ball litigant. Only that last jail contempt order in that case slowed the games. Victor Stanley, Inc. v. Creative Pipe, Inc., 269 F.R.D. 497, 506 (D. Md. 2010) (the attempts to collect the many fines and judgment entered in that case are still ongoing).

This all reminds me of Judge Waxse’s well-known quip that lawyers are like elementary particles, they change when observed (by judges). He has found that lawyers are more inclined to cooperate simply by including a possibility that a judge might someday watch a video of their behavior. Maybe we should require that all lawyer-to-lawyer communications be taped? Maybe we should triple the number of judges and give them all sensitivity training? Who knows? But the research shows that all manner of alternatives like that would be more successful than mere appeals to reason alone.

Waxse_Losey

This all makes me wonder why I even bother to continue to write, but then again, you may have noticed that I try to include non-rational appeals in my writing, such as images, videos and the great irrational motivator of humor. Humor is an elusive emotion to reach, but well worth the effort. It is difficult to resist the ideas of anyone who makes you laugh. Personally, I refuse to emulate anyone who does not at least make me smile. If they make me laugh out loud, well, I will dig in deep to try to understand them and their ideas. (This is on reason I highly recommend the new self-help book by Scott Adams, the creator of Dilbert – How to Fail at Almost Everything and Still Win Big: Kind of the Story of My Life.)

Law is Like Economics: Both Are Still Based on an Irrational Reliance on Reason

Predictable_IrrationalAs you have seen from the videos, Dan Ariely is not only witty, but also a psychologist and an economist. He has one PhD in Psychology and another in Business Administration. He is also an author of a number of books that explain his works to the general reader, including the best seller: Predictably Irrational: The Hidden Forces That Shape Our Decisions.

Dan evaluates the implications of his irrationality findings in Psychology on the field of Economics. So too are many other pundits in the field. See eg. Post-Rational Economic Manand Exploring the Post-Rational 21st CenturyAriely and others have amassed a growing body of evidence that humans are not rational machines. Yet most economists, much like most lawyers, do not believe that. They still believe that people make rational decisions. For instance, that purchases are based on reason alone. See Rational Choice Theory. That is the basis of classic economic theory, and since that presumption is wrong, so is the theory. Economics is now struggling with the development of new theories based on the way people really act. Dan is a leader of that movement, which he calls Behavioral Economics.

Learning a little about Dan’s insights and proposals to reform economic theories, and make them more realistic, and empirically based, can provide insights into the Law and reforms we should make. Surely we can do better than propose more videotapes of lawyers, in your face judges, bibles and oaths, solemn court reporters, and British style ceremonial conduct. But these are a start.

Reasonable_guageMore fundamentally, we need to consider how we should speak of legal negligence in the future. We need to stop referring to whether an act is reasonable, and instead speak of acceptability, with reason just one of several factors to consider in evaluating acceptable behavior. That is what I call, for lack of a better term, Holistic Jurisprudence. More on that later. Perhaps some law professors and judges are already thinking and writing about this, and I am not aware of their writings. If not, then what are we waiting for? The evidence of innate irrationally based, yet acceptable, behavior, is strong. That is our everyday reality. So why do we use a measure of acceptable conduct that does not mirror reality? Legal theory needs to change as much as economic theory, and so too does legal practice.

Robots and Neuroscience?

Facc_RobotI know what some of you are thinking. Maybe the answer is simply to turn our justice system over to robots programmed to make rational decisions. They will not suffer from innate irrationality like our judges do. (Yes, even judges are human and thus even judges suffer from the same cognitive disorders, same irrational drivers, that other humans do). Rational machines could also be programmed to fairly consider the innate irrationality of humans. We could create super robojudges by using active machine learning. They could receive training in just-decision-making by our top judges. Imagine, for instance, the wisdom and wit of retired Judge Facciola programmed into an AI entity. The input from our top judges would thereby, in theory at least, live forever. The experience and intelligence of our best judges would then be available to all litigants, not just the lucky few who appear before them. This puts a while new positive spin onto the Ghost in the Machine image.

The AI enhanced robojudges would, of course, be far more than mere rational machines. They would be trained by our legal experts to render judgments based on the Whole Man, one that actually exists, and not the legal fiction of the Reasonable Man. They would be programmed in a post-rational manner following models of real human behavior of acceptable conduct. (Our best human judges and lawyers already do that anyway, even if the jurisprudence theory says otherwise.) The day may come when many litigants will prefer smart, well-trained robots to serve as judges to evaluate acceptable conduct, especially when there are good human appeals judges to oversee the process. That day is, however, still in the remote future.

ex_machinaOf course, if Ray Kurzweil is right about the Singularity coming soon, then all bets are off. But Kurzweil is probably wrong about how fast AI will advance, and so I do not see this anything like this happening this century. Moreover, the live forever proponents are, in my view, seriously deluded. (Fear of death tends to do that to people. Just look at the many wacky beliefs people have.) There is far more to the human soul than logic can ever replicate. We are more than a set of synapses that can be replicated with on off switches. Still, I could be wrong.

Automation by full human replication is not The answer (at least not in this century). The use of AI enhanced tools in the law, such as predictive coding for document review, is more realistic. It will continue and expand into many other legal activities. Very soon many more types of lawyers, in addition to contract review lawyers, will need to retool in order to stay employed. The simple logic and reason tasks of lawyers will be automated. All mere logic workers will have to change or be out of work. But, at the same time, new employment positions will open for those involved in the new technologies. The jobs that open up will require greater technology skills, intellect, empathy, leadership, creativity and imagination. They will require uniquely human attributes that are way beyond the programming of any robots, now and perhaps forever. Again see Only Humans Need Apply for a good review of this subject in the context of the economy.

I cannot imagine exactly how this will all play out, but, new advanced technologies will have to be part of all future legal reforms. Many of the technologies are probably still unknown and thus impossible to project. But some will be based on existing technologies, just significantly improved.

Facciola_computerPerhaps that will include active machine learning and AI based law clerks for judges. It is not hard to imagine a judge’s consideration of an AI enhanced suggested view of the case. After all, they already do this based on their clerk’s views. I suspect judicial clerks will be replaced way before the judges themselves. Judges need to be enhanced with better computers, not replaced by them. They need to be augmented, not automated.

To take a more mundane example than robots and AI, I suspect that lie detection technologies will soon advance enough to be of greater assistance to the Law. How about acceptably intrusive truth-compelling technologies? I can easily imagine neural nets with electronic brain monitors built into “truth hats.” Witnesses would be required to wear the truth-indicating hats and give the attorneys, judges and juries more and better insights into their testimony. Not only intentional lies could be revealed, but strength of recollection, areas of brain accessed, etc. This would not have to be dispositive, but suggestive. This could provide us with something more to evaluate credibility than raw instinct and intuition, as important as these faculties are.

Meet-the-Parents-lie-detector with Harry Potter twist

We should be looking for all kinds of ways to bring the recent incredible advances in Neuroscience into the justice system. This is not futuristic science fiction, nor my over-active imagination. It is already happening. Many neuroscientists are looking into lie detection and other possible neuroscience applications in the Law. See eg Harvard’s Center for Law, Brain and Behavior and its program on Lie Detection & the Neuroscience of Deception.

Final Word From Dan Ariely 

Dan-Ariely_WSJGetting back to Dan, in addition to teaching and running very clever experiments at MIT and Duke, Dan is the founder of an organization with a name that seems both funny and ironic, The Center for Advanced Hindsight. He is also a prolific writer and video maker, both activities I admire. See for instance his informative page at MIT, his blog at DanAriely.com, his several books, and his videos, and even though its slightly boring, see his web page at Duke.

As a final piece of evidence on over reliance on reason I offer more testimony by Professor Ariely’s via another video, one which is not at all boring, I swear. It is called The Honest Truth About Dishonesty.

The video concludes with a subject near and dear to all lawyers, conflicts of interest. The non-rational impact of such conflicts turns out to be very strong and the law is wise to guard against them. Perhaps we should even step up our efforts in this area? 

Cornerstone Made of Pudding

CornerstoneThe scientific experiments of Dan Ariely and others show that the cornerstone of the Law – reasonability – is not made of granite as we had thought, it is made of pudding. You can hide your head in the sand, if you wish, and continue to believe otherwise. We humans are quite good at self-delusion. But that will not change the truth. That will not change quicksand into granite.

Our legal house needs a new and better foundation than reason. We must follow the physicists of a century ago. We must transcend Newtonian causality and embrace the more complex, more profound truth that science has revealed. The Reasonable Man is a myth that has outlived its usefulness. We need to accept the evidence, and move on. We need to develop new theories and propositions of law that confirm to the new facts at hand. Reason is just one part of who we are. There is much more to us then that: emotion, empathy, creativity, aesthetics, intuition, love, strength, courage, imagination, determination – to name just a few of our many qualities. These things are what make us uniquely human; they are what separate us from AI. Logic and reason may end up being the least of our abilities, although they are still qualities that I personally cherish.

Science has shown that our current reason-only-based system of justice is on shaky grounds. It is now up to us to do something about it. No big brother government, or super think-tank guru is going to fix this for us. Certainly not scientists either, but they should be able to help, along with technologists, programmers and engineers. This fix will have to come from within the legal profession itself. No one else knows our system of justice well enough to do this for us, certainly not scientists nor engineers. I know this from hard personal experience.

homer-simpson-brain-scanWhat are the implications of the findings of unreliable mental processes on the Law and our ability to reach just decisions? We should ask these questions concerning the Law, just like Professor Ariely is asking concerning Economics. Our fundamental legal assumption that all people can act out of reason and logic alone is false. Decisions made with these faculties alone are the exception, not the rule. There are a number of other contributing factors, including emotions, intuition, and environment. What does this mean to negligence law? To sanctions law? Now that the Reasonable Man is dead, who shall replace him?

Just as classical economic theory has had it all wrong, so too has classical legal theory. People are not built like reasonable machines. That includes lawyers, judges, and everyone else in the justice system, especially the litigants themselves.

If Not Reason, Then What?

Davinci_whole_manSince human reason is now known to be so unreliable, and is only a contributing factor to our decisions, on what should we base our legal jurisprudence? I believe that the Reasonable Man, now that he is known to be an impossible dream, should be replaced by the Whole Man. Our jurisprudence should be based on the reality that we are not robots, not mere thinking machines. We have many other faculties and capabilities beyond just logic and reason. We are more than math. We are living beings. Reason is just one of our many abilities.

So I propose a new, holistic model for the law. It would still include reason, but add our other faculties. It would incorporate our total self, all human attributes. We would include more than logic and reason to judge whether behavior is acceptable or not, to consider whether a resolution of a dispute is fair or not. Equity would regain equal importance.

A new schemata for a holistic jurisprudence would thus include not just human logic, but also human emotions, our feelings of fairness, our intuitions of what is right and just, and multiple environmental and perceptual factors. I suggest a new model start simple and use a four-fold structure like this, and please note I keep Reason on top, as I still strongly believe in its importance to the Law.

4-levels-Holistic_Law_pyramid

Some readers may notice that this model is similar to that of Carl Jung’s four personality types and the popular Myers Briggs personality tests. I am not advocating adoption of any of their ideologies, or personality theories, but I have over the years found their reference models to be useful. The above model, which is proposed only as a starting point for further discussion, is an extrapolation of these psychological models.

Call For Action

The legal profession needs to take action now to reduce our over-reliance on the Myth of the Reasonable Man. We should put the foundations of our legal system on something else, something more solid, more real than that. We need to put our house in order before it starts collapsing around us. That is the reasonable thing to do, but for that very reason we will not start to do it until we have better motivation than that. You cannot get people to act on reason alone, even lawyers. So let us engage the other more powerful motivators, including the emotions of fear and greed. For if we do not evolve our work to focus on far more than reason, then we will surely be replaced.

cyborg-lawyer

AI can think better and faster, and ultimately at a far lower cost. But can AI reassure a client? Can it tell what a client really wants and needs. Can AI think out of the box to come up with new, creative solutions. Can AI sense what is fair? Beyond application of the rules, can it attain the wisdom of justice. Does it know when rules should be bent and how far? Does it know, like any experienced judge knows, when rules should be broken entirely to attain a just result? Doubtful.

To get specific on the reforms needed now, we should bring back equity, and down play law. This was common in the first half of the Twentieth Century. At that time it was common to have Courts of Law and separate Courts of Equity. By the middle of the last century, Courts of Law won out in most states except Delaware, Mississippi, New Jersey, South Carolina, and Tennessee. Separate Equity Courts were closed down in favor of Courts of Law. Maybe we got it backwards. Maybe we were all led astray by our false confidence in reason.

Perhaps most courts should be Courts of Equity and Courts of Law become the exception. How has it worked out for the states that kept equity courts? Have Chancellors truly been able to side-step strict rules of law when they felt it was equitable to do so? If so, how has that worked out? Has power been abused? Or has justice been attained more often? What can we learn from chancery courts that might help us build a more holistic court of the future? We should apply analytics to study these questions to help us to reshape the law in a more human, holistic manner. Law Professors need to study this and help guide the profession.

A Few More Specific Suggestions of Reform

The AI enhancements already moving the law will continue to expand. That much is certain. Predictive coding, my speciality, is currently the prime example, but there will soon be many others.  They will enhance and improve our abilities. They will help us be more efficient. They will also help us to stay fair and honest. They could help end, or at least mitigate, human bias, stereotypes and prejudice.

Maybe timely reminders of ethics codes and serious under penalties of perjury type threats will also help? Maybe new, improved, and customized oaths will help? Oaths have been shown to be effective by Ariely’s research, so we should modify the rules accordingly. Let’s consider an update Rule 603 of the Federal Rules of Evidence.

Electrodes_EEG_RalphMaybe new truth recognition technologies should be used? Could a truth hat with built-in neural net be that far off? How about Google Glasses type apps that provide reliable new feedback of all kinds on the people you watch testifying? That cannot be too far off.  (The lie detection apps already on the market for iPhones, etc., all look bogus to me, which is not unexpected based on the limited biofeedback the phone sensors can provide.) Even if the information is not admissible as evidence, it could still be quite valuable to lawyers. Perhaps some of the recent discoveries in neuroscience could begin to be used in the justice system in all types of unexpected ways?

trophy_LawMaybe public recognition and awards to lawyers and judges who get it right will help? Ariely’s research suggests it will. And awards to litigants who do the right thing too, even if they lose the case? How about a discretionary set-off for defendants like that? How about the converse? Shame can be a powerful motivator too. Some judges already do this in subliminal manner. Let us encourage a more open application of emotion and creativity in judicial activity.

Maybe we should change the conditions and environments of places where witnesses are questioned, where mediations and trials are conducted? Maybe we should provide special training to court reporters on oath giving? Maybe we should have trials again, and not just settlements?

We need to look for all kinds of motivators. Knowledge and reason alone are not a solid foundation for justice. We also need wisdom. See eg. Losey, R., Information → Knowledge → Wisdom: Progression of Society in the Age of Computers.

Conclusion

Ralph_mosiac_7-16All social structures today are experiencing disruptive change, including the Law. Technology is driving these transformations. We need courage to face the reality of rapid change, instead of fearful avoidance. We need to shape the changes in the Law, not be overridden  by them. We need to be proactive, creative. We need to be guided by truth, not tradition. That means paying attention to analytics and psychology, and not just digging deeper into the Law alone for answers. The insights we need will come from a multidisciplinary approach, but one that is led by legal professionals. Only we truly understand legal practice. But our efforts to shape the change in our profession must include knowledge from all fields, including science, engineering and art. It must also include input from all other participants in the legal system, especially clients, litigants, plaintiffs, defendants. Legal practitioners, judges and scholars alone cannot provide a holistic view.

We must move away from over-reliance on reason alone. Our enlightened self-interest in continued employment in the rapidly advancing world of AI demand this. So too does our quest to improve our system of justice, to keep it current with the rapid changes in society.

Where we must still rely on reason, we should at the same time realize its limitations. We should look for new technology based methods to impose more checks and balances on reason than we already have. We should create new systems that will detect and correct the inevitable errors in reason that all humans make – lawyers, judges and witnesses alike. Bias and prejudice must be overcome in all areas of life, but especially in the justice system.

Computers, especially AI, should be able to help with this and also make the whole process more efficient. We need to start focusing on this, to make it a priority. It demands more than talk and thinking. It demands action. We cannot just think our way out of a prison of thought. We need to use all of our faculties, especially our imagination, creativity, intuition, empathy and good faith.


e-Discovery Team’s Best Practices Education Program

May 8, 2016

EDBP_BANNER

EDBP                   Mr.EDR         Predictive Coding 3.0
59 TAR Articles
Doc Review  Videos

_______

TEAM_TRAINING_screen_shot

e-Discovery Team Training

Information → Knowledge → Wisdom

Ralph_4-25-16Education is the clearest path from Information to Knowledge in all fields of contemporary culture, including electronic discovery. The above links take you to the key components of the best-practices teaching program I have been working on since 2006. It is my hope that these education programs will help move the Law out of the dangerous information flood, where it is now drowning, to a safer refuge of knowledge. Information → Knowledge → Wisdom: Progression of Society in the Age of Computers; and How The 12 Predictions Are Doing That We Made In “Information → Knowledge → Wisdom.” For more of my thoughts on e-discovery education, see the e-Discovery Team School Page.

justice_guage_negligenceThe best practices and general educational curriculum that I have developed over the years focuses on the legal services provided by attorneys. The non-legal, engineering and project management practices of e-discovery vendors are only collaterally mentioned. They are important too, but students have the EDRM and other commercial organizations and certifications for that. Vendors are part of any e-Discovery Team, but the programs I have developed are intended for law firms and corporate law departments.

LIFE_magazine_Losey_acceleratesThe e-Discovery Team program, both general educational and legal best-practices, is online and available 24/7. It uses lots of imagination, creative mixes, symbols, photos, hyperlinks, interactive comments, polls, tweets, posts, news, charts, drawings, videos, video lectures, slide lectures, video skits, video slide shows, music, animations, cartoons, humor, stories, cultural themes and analogies, inside baseball references, rants, opinions, bad jokes, questions, homework assignments, word-clouds, links for further research, a touch of math, and every lawyer’s favorite tools: words (lots of them), logic, arguments, case law and precedent.

All of this to try to take the e-Discovery Team approach from just information to knowledge →. In spite of these efforts, most of the legal community still does not know e-discovery very well. What they do know is often misinformation. Scenes like the following in a law firm lit-support department are all too common.

supervising-tipsThe e-Discovery Team’s education program has an emphasis on document review. That is because the fees for lawyers reviewing documents is by far the most expensive part of e-discovery, even when contract lawyers are used. The lawyer review fees, and review supervision fees, including SME fees, have always been much more costly than all vendor costs and expenses put together. Still, the latest AI technologies, especially active machine learning using our Predictive Coding 3.0 methods, are now making it possible to significantly reduce review fees. We believe this is a critical application of best practices. The three steps we identify for this area in the EDBP chart are shown in green, to signify money. The reference to C.A. Review is to Computer Assisted Review or CAR, using our Hybrid Multimodal methods.

EDBP_detail_LARGE

____

Predictive Coding 3.0 Hybrid Multimodal Document Search and Review

Control-SetsOur new version 3.0 techniques for predictive coding makes it far easier than ever before to include AI in a document review project. The secret control set has been eliminated, so too has the seed set and SMEs wasting their time reviewing random samples of mostly irrelevant junk. It is a much simpler technique now, although we still call it Hybrid Multimodal.

robot-friendHybrid is a reference to the Man/Machine interactive nature of our methods. A skilled attorney uses a type of continuous active learning to train an AI to help them to find the documents they are looking for. This Hybrid method greatly augments the speed and accuracy of the human attorneys in charge. This leads to cost savings and improved recall. A lawyer with an AI helper at their side is far more effective than lawyers working on their own. This means that every e-discovery team today could use a robot like Kroll Ontrack’s Mr. EDR to help them to do document review.

Search_pyramidMultimodal is a reference to the use of a variety of search methods to find target documents, including, but not limited to, predictive coding type ranked searches. We encourage humans in the loop running a variety of searches of their own invention, especially at the beginning of a project. This always makes for a quick start in finding relevant and hot documents. Why the ‘Google Car’ Has No Place in Legal Search. The multimodal approach also makes for precise, efficient reviews with broad scope. The latest active machine learning software when fully integrated with a full suite of other search tools is attaining higher levels of recall than ever before. That is one reason Why I Love Predictive Coding.

Mr_EDRI have found that Kroll Ontrack’s EDR software is ideally suited for these Hybrid, Multimodal techniques. Try using it on your next large project and see for yourself. The Kroll Ontrack consultant specialists in predictive coding, Jim and Tony, have been trained in this method (and many others). They are well qualified to assist you in every step of the way and their rates are reasonable. With you calling the shots on relevancy, they can do most of the search work for you and still save your client’s money. If the matter is big and important enough, then, if I have a time opening, and it clears my firm’s conflicts, I can also be brought in for a full turn-key operation. Whether you want to include extra time for training your best experts is your option, but our preference.

Team_TREC_2

__________

Embrace e-Discovery Team Education to Escape Information Overload

____


Five Reasons You Should Read the ‘Practical Law’ Article by Maura Grossman and Gordon Cormack called “Continuous Active Learning for TAR”

April 11, 2016

Maura-and-Gordon_Aug2014There is a new article by Gordon Cormack and Maura Grossman that stands out as one of their best and most accessible. It is called Continuous Active Learning for TAR (Practical Law, April/May 2016). The purpose of this blog is to get you to read the full article by enticing you with some of the information and knowledge it contains. But before we go into the five reasons, we will examine the purpose of the article, which aligns with our own, and touch on the differences between their trademarked TAR CAL method and our CAR Hybrid Multimodal method. Both of our methods use continuous, active learning, the acronym for which, CAL, they now claim as a Trademark. Since they clearly did invent the acronym, CAL, we for one will stop using it – CAL – as a generic term.

The Legal Profession’s Remarkable Slow Adoption of Predictive Coding

The article begins with the undeniable point of the remarkably slow adoption of TAR by the legal profession, in their words:

Adoption of TAR has been remarkably slow, considering the amount of attention these offerings have received since the publication of the first federal opinion approving TAR use (see Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012)).

Winners in Federal CourtI remember getting that landmark ruling in our Da Silva Moore case, a ruling that pissed off plaintiffs’ counsel, because, despite what you may have heard to the contrary, they were strenuously opposed to predictive coding. Like most other lawyers at the time who were advocating for advanced legal search technologies, I thought Da Silva would open the flood gates, that it would encourage attorneys to begin using the then new technology in droves. In fact, all it did was encourage the Bench, but not the Bar. Judge Peck’s more recent ruling on the topic contains a good summary of the law. Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125 (S.D.N.Y. 2015). There were a flood  of judicial rulings approving predictive coding all around the country, and lately, around the world. See Eg. Pyrrho Investments v MWB PropertyEWHC 256 (Ch) (2/26/16).

The rulings were followed in private arbitration too. For instance, I used the Da Silva More ruling a few weeks after it was published to obtain what was apparently the first ruling by an arbitrator in AAA approving use of predictive coding. The opposition to our use of cost-saving technology in that arbitration case was again fierce, and again included personal attacks, but the arguments for use in arbitration are very compelling. Discovery in arbitration is, after all, supposed to be constrained and expedited.

IT_GovernanceAfter the Da Silva Moore opinion, Maura Grossman and I upped our speaking schedule (she far more than me), and so did several tech-minded judges, including Judge Peck (although never at the same events as me, until the cloud of false allegations created by a bitter plaintiff’s counsel in Da Silva Moore could be dispelled). At Legal Tech for the next few years Predictive Coding is all anybody wanted to talk about. Then IG, Information Governance, took over as the popular tech-child of the day. In 2015 we had only a few predictive coding panels at Legal Tech, but they were well attended.

The Grossman Cormack speculates that the cause of the remarkably slow adoption is:

The complex vocabulary and rituals that have come to be associated with TAR, including statistical control sets, stabilization, F1 measure, overturns, and elusion, have dissuaded many practitioners from embracing TAR. However, none of these terms, or the processes with which they are associated, are essential to TAR.

Control-SetsWe agree. The vendors killed what could have been their golden goose with all this control set nonsense and their engineers love of complexity and misunderstanding of legal search. I have ranted about this before. See Predictive Coding 3.0. I will not go into that again here, except to say the statistical control set nonsense that had large sampling requirements was particularly toxic. It was not only hard and expensive to do, it led to mistaken evaluations of the success or failure of projects because it ignored the reality of the evolving understand of relevance, so called concept drift. Another wrong turn involved the nonsense of using only random selection to find training documents, a practice that Grossman and I opposed vigorously. See Latest Grossman and Cormack Study Proves Folly of Using Random Search For Machine Training – Part One,  Part Two,  Part Three, and Part Four. Grossman and Cormack correctly criticize these old vendor driven approaches in Continuous Active Learning for TAR. They call them SAL and SPL protocols (a couple of acronyms that no one wants to trademark!).

Bottom line, the tide is changing. Over the last several years the few private attorneys who specialize in legal search, but are not employed by a vendor, have developed simpler methods. Maura and I are just the main ones writing and speaking about it, but there are many others who agree. Many have found that it is counter-productive to use control sets, random input, non-continuous training with its illogical focus on the seed set, and misleading recall point projections.

grossman_cormack_filteredWe do so in defiance of the vendor establishment and other self-proclaimed pundits in this area who benefitted by such over-complexity. Maura and Gordon, of course, have their own software (Gordon’s creation), and so never needed any vendors to begin with. Not having a world renowned information scientist like Professor Cormack as my life partner, I had no choice but to rely on vendors for their software. (Not that I complaining, mind you. I’m married to a mental health counselor, and it does not get any better than that!)

MrEdr_CapedAfter a few years I ultimately settled on one vendor, Kroll Ontrack, but I continue to try hard to influence all vendors. It is a slow process. Even Kroll Ontrack’s software, which I call Mr. EDR, still has control set functions built in. Thanks to my persistence, it is easy to turn off these settings and do things my way, with no secret control sets and false recall calculations. Hopefully soon that will be the default setting. Their eyes have been opened. Hopefully all of the other major vendors will soon follow suit.

All of the Kroll Ontrack experts in predictive coding are now, literally, a part of my Team. They are now fully trained and believers in the simplified methods, methods very similar to those of Grossman and Cormack, albeit, as I will next explain, slightly more complicated. We proved how well these methods worked at TREC 2015 when the Kroll Ontrack experts and I did 30 review projects together in 45 days. See e-Discovery Team at TREC 2015 Total Recall Track, Final Report (116 pg. PDF), and  (web page with short summary). Also see – Mr. EDR with background information on the Team’s participation in the TREC 2015 Total Recall Track.

We Agree to Disagree with Grossman and Cormack on One Issue, Yet We Still Like Their Article

Team_TRECWe are fans of Maura Grossman and Gordon Cormack’s work, but not sycophants. We are close, but not the same; colleagues, but not followers. For those reasons we think our recommendation for you to read this article means more than a typical endorsement. We can be critical of their writings, but, truth is, we liked their new article, although we continue to dislike the name TAR (not important, but we prefer CAR). Also, and this is of some importance, my whole team continues to disagree with what we consider the somewhat over-simplified approach they take to finding training documents, namely reliance on the highest ranking documents alone.

LogisticRegressionWindowLogisticFitChart6Despite what some may think, the high-ranking approach does eventually find a full diversity of relevant documents. All good predictive coding software today pretty much uses some type of logistic regression based algorithms that are capable of building out probable relevance in that way. That is one of the things we learned by rubbing shoulders with text retrieval scientists from around the world at TREC when participating in the 2015 Total Recall Track that Grossman and Cormack helped administer. This regression type of classification system works well to avoid the danger of over-training on a particular relevancy type. Grossman and Cormack have proven that before to our satisfaction (so have our own experiments), and they again make a convincing case for this approach in this article.

4_Cylinder_engineStill, we disagree with their approach of only using high-ranking documents for training, but we do so on the grounds of efficiency and speed, not effectiveness. The e-Discovery Team continues to advocate a Hybrid Multimodal approach to active machine learning. We use what I like to call a four-cylinder type of CAR search engine, instead of one-cylinder, like they do.

  1. High-ranking documents;
  2. Mid-level, uncertain documents;
  3. A touch, a small touch, of random documents; and,
  4. Human ingenuity found documents, using all type of search techniques (multimodal) that seem appropriate to the search expert in charge, including keyword, linear, similarity (including chains and families), concept (including passive machine learning, clustering type search).

Predictive Coding 3.0 – The method is here described as an eight-part work flow (Step 6 – Hybrid Active Training).

The latest Grossman and Cormack’s versions of CAL (their trademark) only uses the highest-ranking documents for active training. Still, in spite of this difference, we liked their article and recommend you read it.

The truth is, we also emphasize the high-probable relevant documents for training. The difference between us is that we use the three other methods as well. On that point we agree to disagree. To be clear, we are not talking about continuous training or not, we agree on that. We are not talking about active training, or not (passive), we agree on that. We are not talking about using what they call using SAL or SPL protocols (read their article for details), we agree with them that these protocols are ineffective relics invented by misguided vendors. We are only talking about a difference in methods to find documents to use to train the classifier. Even that is not a major disagreement, as we agree with Grossman and Cormack that high-ranking documents usually make the best trainers, just not in the first seed set. There are also points in a search, depending on the project, where the other methods can help you get to the relevant documents in a fast, efficient manner. The primary difference between us is that we do not limit ourselves to that one retrieval method like Grossman and Cormack do in their trademarked CAL methodology.

Cormack and Grossman emphasize simplicity, ease of use, and reliance on the software algorithms as another way to try to overcome the Bar’s continued resistance to TAR. The e-Discovery Team has the same goal, but we do not think it is necessary to go quite that far for simplicity sake. The other methods we use, the other three cylinders, are not that difficult and have many advantages. e-Discovery Team at TREC 2015 Total Recall Track, Final Report (116 pg. PDF and web page with short  summary). Put another way, we like the ability of fully automatic driving from time to time, but we want to keep an attorney’s learned hand at or near the wheel at all times. See Why the ‘Google Car’ Has No Place in Legal Search.

Accessibility with Integrity: The First Reason We Recommend the Article

Professor Gordon Cormack

Here’s the first reason we like Grossman & Cormack’s article, Continuous Active Learning for TAR: you do not have to be one of Professor Cormac’s PhD students to understand it. Yes. It is accessible, not overly technical, and yet still has scientific integrity, still has new information, accurate information, and still has useful knowledge.

It is not easy to do both. I know because I try to make all of my technical writings that way, including the 57 articles I have written on TAR, which I prefer to call Predictive Coding, or CAR. I have not always succeeded in getting the right balance, to be sure. Some of my articles may be too technical, and perhaps some suffer from breezy information over-load and knowledge deficiency. Hopefully none are plain wrong, but my views have changed over the years. So have my methods. If you compare my latest work-flow (below) with earlier ones, you will see some of the evolution, including the new emphasis over the past few years with continuous training.

predictive_coding_revised_small_size

The Cormacks and I are both trying hard to get the word out to the Bar as to the benefits of using active machine learning in legal document review.  (We all agree on that term, active machine learning, and all agree that passive machine learning is not an acceptable substitute.) It is not easy to write on this subject in an accurate, yet still accessible and interesting manner. There is a constant danger that making a subject more accessible and simple will lead to inaccuracies and misunderstandings. Maura and Gordon’s latest article meets this challenge.

Search ImageTake for example the first description in the article of their continuous active training search method using highest ranking documents:

At the outset, CAL resembles a web search engine, presenting first the documents that are most likely to be of interest, followed by those that are somewhat less likely to be of interest. Unlike a typical search engine, however, CAL repeatedly refines its understanding about which of the remaining documents are most likely to be of interest, based on the user’s feedback regarding the documents already presented. CAL continues to present documents, learning from user feedback, until none of the documents presented are of interest.

That is a good way to start an article. The comparison with a Google search having continued refinement based on user feedback is well thought out; simple, yet accurate. It represents a description honed by literally hundreds of presentations on the topic my Maura Grossman. No one has talked more on this topic than her, and I for one intend to start using this analogy.

Rare Description of Algorithm Types – Our Second Reason to Recommend the Article

Another reason our Team liked Continuous Active Learning for TAR is the rare description of search algorithm types that it includes. Here we see the masterful touch of one of the world’s leading academics on text retrieval, Gordon Cormack. First, the article makes clear the distinction between effective analytic algorithms that truly rank documents using active machine learning, and a few other popular programs now out there that use passive learning techniques and call it advanced analytics.

The supervised machine-learning algorithms used for TAR should not be confused with unsupervised machine-learning algorithms used for clustering, near-duplicate detection, and latent semantic indexing, which receive no input from the user and do not rank or classify documents.

Old_CAR_stuck_mudThese other older, unsupervised search methods are what I call concept search. It is not predictive coding. It is not advanced analytics, no matter what some vendors may tell you. It is yesterday’s technology – helpful, but far from state-of-the-art. We still use concept search as part of multimodal, just like any other search tool, but our primary reliance to properly rank documents is placed on active machine learning.

hyperplanes3d_2The Cormack-Grossman article goes farther than pointing out this important distinction, it also explains the various types of bona fide active machine learning algorithms. Again, some are better than others. First Professor Cormack explains the types that have been found to be effective by extensive research over the past ten years or so.

Supervised machine-learning algorithms that have been shown to be effective for TAR include:

–  Support vector machines. This algorithm uses geometry to represent each document as a point in space, and deduces a boundary that best separates relevant from not relevant documents.

– Logistic regression. This algorithm estimates the probability of a document’s relevance based on the content and other attributes of the document.

Conversely Cormack explains:

Popular, but generally less effective, supervised machine-learning algorithms include:

– Nearest neighbor. This algorithm classifies a new document by finding the most similar training document and assuming that the correct coding for the new document is the same as its nearest neighbor.

– Naïve Bayes (Bayesian classifier). This algorithm estimates the probability of a document’s relevance based on the relative frequency of the words or other features it contains.

Ask your vendor which algorithms its software includes. Prepare yourself for double-talk.

Hot-or-Not

If you try out your vendors software and the Grossman-Cormack CAL method does not work for you, and even the e-Discovery Team’s slightly more diverse Hybrid Multimodal method does not work, then your software may be to blame. As Grossman-Cormack put it, where the phrase “TAR tool” means software:

[I]t will yield the best possible results only if the TAR tool incorporates a state-of-the-art learning algorithm.

That means software that uses a type of support vector machine and/or logistic regression.

Teaching by Example – Our Third Reason to Recommend the Article

The article uses a long example involving search of Jeb Bust email to show you how their CAL method works. This is an effective way to teach. We think they did a good job with this. Rather than spoil the read with quotes and further explanation, we urge you to check out the article to see for yourself. Yes, it is an oversimplification, after all this is a short article, but it is a good one, and is still accurate.

 Quality Control Suggestions – Our Fourth Reason to Recommend the Article

quality_diceAnother reason we like the article are the quality control suggestions it includes. They essentially speak of using other search methods, which is exactly what we do in Hybrid Multimodal. Here are their words:

To increase counsel’s confidence in the quality of the review, they might:

Review an additional 100, 1,000, or even more documents.

Experiment with additional search terms, such as “Steve Jobs,” “iBook,” or “Mac,” and examine the most-likely relevant documents containing those terms.

Invite the requesting party to suggest other keywords for counsel to apply.

Review a sample of randomly selected documents to see if any other documents of interest are identified.

We like this because it shows that the differences are small between the e-Discovery Team’s Hybrid Multimodal method (hey, maybe I should claim Trademark rights to Hybrid Multimodal, but then again, no vendors are using my phrase to sell their products) using continuous active training, and the Grossman-Cormack trademarked CAL method. We also note that their section on Measures of Success essentially mirrors our own thoughts on metric analysis and ei-Recall. Introducing “ei-Recall” – A New Gold Standard for Recall Calculations in Legal SearchPart One, Part Two and Part Three.

Article Comes With an Online “Do it Yourself” CAL Trial Kit – Our Fifth Reason to Recommend the Article

We are big believers in learning by doing. That is especially true in legal tasks that seem complicated in the abstract. I can write articles and give presentations that provide explanations of AI-Enhanced Review. You may get an intellectual understanding of predictive coding from these, but you still will not know how to do it. On the other hand, if we have a chance to show someone an entire project, have them shadow us, then they will really learn how it is done. It is like teaching a young lawyer how to try a case. For a price, we will be happy to do so (assuming conflicts clear).

Jeb_BushMaura and Gordon seem to agree with us on that learn by doing point and have created an online tool that anyone can use to try out their method. In allows for a search of the Jeb Bush email, the same set of 290,099 emails that we used in ten of the thirty topics in 2015 TREC. In their words:

There is no better way to learn CAL than to use it. Counsel may use the online model CAL system to see how quickly and easily CAL can learn what is of interest to them in the Jeb Bush email dataset. As an alternative to throwing up their hands over seed sets, control sets, F1 measures, stabilization, and overturns, counsel should consider using their preferred TAR tool in CAL mode on their next matter.

You can try out their method with their online tool, or in a real project using your vendor’s tool. By the way, we did that as part of our TREC 2015 experiments, and the Kroll Ontrack software worked about the same as theirs, even when we used their one-cylinder, high ranking only, CAL (their trademark) method.

Here is where you can find their CAL testing tool: cormack.uwaterloo.ca/cal. Those of you who are still skeptical can see for yourself how it works. You can follow the example given in the article about searching for documents relevant to Apple products, to verify their description of how that works. For even more fun, you can dream up your own searches.

030114-O-0000D-001 President George W. Bush. Photo by Eric Draper, White House.

Perhaps, if you try hard enough, you can find some example searches where their high-end only method, which is built into the test software, does not work well. For example, try finding all emails that pertain to, or in any way mention, the then President, George Bush. Try entering George Bush in the demo test and see for yourself what happens.

It becomes a search for George + Bush in the same document, and then goes from there based on your coding the highest ranked documents presented as either relevant or non-relevant. You will see that you quickly end up in a TAR pit. The word Bush is in every email (I think), so you are served up with every email where George is mentioned, and believe me, there are many Georges, even if there is only one President George Bush. Here is the screen shot of the first document presented after entering George Bush. I called it relevant.

Screen Shot 2016-04-10 at 4.13.24 PM

These kind of problem searches do not discredit TAR, or even the Grossman Cormack one-cylinder search method. If this happened to you in a real search project, you could always use our Hybrid Multimodal™ method for the seed set (1st training), or start over with a different keyword or keywords to start the process. You could, for instance, search for President Bush, or President within five of George, or “George Bush.” There are many ways, some faster and more effective than others.

Even using the single method approach, if you decided to use the keywords “President + Bush”, then the search will go quicker than “George + Bush.” Even just using the term “President” works better than George + Bush, but still seems like a TAR pit, and not a speeding CAR. It will probably get you to the same destination, high recall, but the journey is slightly longer and, at first, more tedious. This high recall result was verified in TREC 2015 by our Team, and by a number of Universities who participated in the fully automatic half of the Total Recall Track, including Gordon’s own team. This was all done without any manual review by the fully automatic participants because there was instant feedback of relevant or irrelevant based on a prejudged gold standard. See e-Discovery Team at TREC 2015 Total Recall Track, Final Report (116 pg. PDF), and (web page with short  summary). With this instant feedback protocol, all of the teams attained high recall and good precision. Amazing but true.

You can criticized this TREC experiment protocol, which we did in our report, as unrealistic to legal practice because:

(1) there is no SME who works like that (and there never will not be, until legal knowledge itself is learned by an AI); and,

(2) the searches presented as tasks were unrealistically over-simplistic. Id.

But you cannot fairly say that CAL (their trademark) does not work. The glass is most certainly not half empty. Moreover, the elixir in this glass is delicious and fun, especially when you use our Hybrid Multimodal™ method. See Why I Love Predictive Coding: Making document review fun with Mr. EDR and Predictive Coding 3.0.

Conclusion

Ralph_head_2016Active machine learning (predictive coding) using support vector or logistic regression algorithms, and a method that employs continuous active training, using either one cylinder (their CAL), or four (our Hybrid Multimodal), really works, and is not that hard to use. Try it out and see for yourself. Also, read the Grossman Cormack article, it only takes about 30 minutes. Continuous Active Learning for TAR (Practical Law, April/May 2016). Feel free to leave any comments below. I dare say you can even ask questions of Grossman or Cormack here. They are avid readers and will likely respond quickly.


%d bloggers like this: