Beware of the TAR Pits! – Part Two

February 23, 2014

This is the conclusion of a two part blog. For this to make sense please read Part One first.

Quality of Subject Matter Experts

Poppy_headThe quality of Subject Matter Experts in a TAR project is another key factor in predictive coding. It is one that many would prefer to sweep under the rug. Vendors especially do not like to talk about this (and they sponsor most panel discussions) because it is beyond their control. SMEs come from law firms. Law firms hire vendors. What dog will bite the hand that feeds him? Yet, we all know full well that not all subject matter experts are alike. Some are better than others. Some are far more experienced and knowledgeable than others. Some know exactly what documents they need at trial to win a case. They know what they are looking for. Some do not. Some have done trials, lots of them. Some do not know where the court house is. Some have done many large search projects, first paper, now digital. Some are great lawyers; and some, well, you’d be better off with my dog.

The SMEs are the navigators. They tell the drivers where to go. They make the final decisions on what is relevant and what is not. They determine what is hot, and what is not. They determine what is marginally relevant, what is grey area, what is not. They determine what is just unimportant more of the same. They know full well that some relevant is irrelevant. They have heard and understand the frequent mantra at trials: Objection, Cumulative. Rule 403 of the Federal Evidence Code. Also see The Fourth Secret of Search: Relevant Is Irrelevant found in Secrets of Search – Part III.

Quality of SMEs is important because the quality of input in active machine learning is important. A fundamental law of predictive coding as we now know it is GIGO, garbage in, garbage out. Your active machine learning depends on correct instruction. Although good software can mitigate this somewhat, it can never be eliminated. See: Webber & Pickens, Assessor Disagreement and Text Classifier Accuracy, SIGIR 2013 (24% more ranking depth needed to reach equivalent recall when not using SMEs, even in a small data search of news articles with rather simple issues).

Jeremy_PickensInformation scientists like Jeremy Pickens are, however, working hard on ways to minimize the errors of SME document classifications on overall corpus rankings. Good thing too because even one good SME will not be consistent in ranking the same documents. That is the Jaccard Index scientists like to measure. Less Is More: When it comes to predictive coding training, the “fewer reviewers the better” – Part Two, and search of Jaccard in my blog.

Unique_Docs_VennIn my Enron experiments I was inconsistent in determining the relevance of the same document 23% of the time. That’s right, I contradicted myself on relevancy 23% of the time. (If you included irrelevancy coding the inconsistencies were only 2%.) Lest you think I’m a complete idiot (which, by the way, I sometimes am), the 23% rate is actually the best on record for an experiment. It is the best ever measured, by far. Other experimentally measured rates have inconsistencies of from 50% to 90% (with multiple reviewers). Pathetic huh? Now you know why AI is so promising and why it is so important to enhance our human intelligence with artificial intelligence. When it comes to consistency of document identifications in large scale data reviews, we are all idiots!

With these human  frailty facts in mind, not only variable quality in expertise of subject matter, but also human inconsistencies, it is obvious why scientists like Pickens and Webber are looking for techniques to minimize the impact of errors and, get this, even use these inevitable errors to improve search. Jeremy Pickens and I have been corresponding about this issue at length lately. Here is Jeremy’s later response to this blog. In TAR, Wrong Decisions Can Lead to the Right Documents (A Response to Ralph Losey). Jeremy does at least concede that coding quality is indeed important. He goes on to argue that his study shows that wrong decisions, typically on grey area documents, can indeed be useful.

Penrose_triangle_ExpertiseI do not doubt Dr. Pickens’ findings, but am skeptical of the search methods and conclusions derived therefrom. In other words, how the training was accomplished, the supervision of the learning. This is what I call here the driver’s role, shown on the triangle as the Power User and Experienced Searcher. In my experience as a driver/SME, much depends on where you are in the training cycle. As the training continues the algorithms eventually do become able to detect and respond to subtle documents distinctions. Yes, it take a while, and you have to know what and when to train on, which is the drivers skill (for instance you never train with giant documents), but it does eventually happen. Thus, while it may not matter if you code grey area documents wrong at first, it eventually will, that is unless you do not really care about the distinctions. (The TREC overturn documents Jeremy tested on, the ones he called wrong documents, were in fact grey area documents, that is, close questions. Attorneys disagreed on whether they were relevant, which is why they were overturned on appeal.) The lack of precision in training, which is inevitable anyway even when one SME is used, may not matter much in early stages of training, and may not matter at all when testing simplistic issues using easy databases, such as news articles. In fact, I have used semi-supervised training myself, as Jeremy describes from old experiments in Pseudo Relevance Feedback. I have seen it work myself, especially in early training.

Still, the fact some errors do not matter in early training does not mean you should not care about consistency and accuracy of training during the whole ride. In my experience, as training progresses and the machine gets smarter, it does matter. But let’s test that shall we? All I can do is report on what I see, i.w. – anecdotal.

Outside of TREC and science experiments, in the messy real world of legal search, the issues are typically maddeningly difficult. Moreover, the difference in cost of review of hundreds of thousands of irrelevant documents can be mean millions of dollars. The fine points of differentiation in matured training are needed for precision in results to reduce costs of final review. In other words, both precision and recall matter in legal search, and all are governed by the overarching legal principle of proportionality. That is not part of information science of course, but we lawyers must govern our search efforts by proportionality.

Also See William Webber’s response: Can you train a useful model with incorrect labels? I believe that William’s closing statement may be correct, either that or software differences:

It may also be, though this is speculation on my part, that a trainer who is not only a subject-matter expert, but an expert in training itself (an expert CAR driver, to adopt Ralph Losey’s terminology) may be better at selecting training examples; for instance, in recognizing when a document, though responsive (or non-responsive), is not a good training example.

alchemyI hope Pickens and Webber get there some day. In truth, I am a big supporter of their efforts and experiments. We need more scientific research. But for now, I still do not believe we can turn lead into gold. It is even worse if you have a bunch of SMEs arguing with each other about where they should be going, about what is relevant and what is not. That is a separate issue they do not address, which points to the downside of all trainers, both amateurs and SMEs alike. See: Less Is More: When it comes to predictive coding training, the “fewer reviewers the better” – Parts OneTwo, and Three.

For additional support on the importance of SMEs, see again Monica’s article, EDI-Oracle Studywhere she summarizes the conclusion of Patrick Oot from the study that:

Technology providers using similar underlying technology, but different human resources, performed in both the top-tier and bottom-tier of all categories. Conclusion: Software is only as good as its operators. Human contribution is the most significant element. (emphasis in original)

Also see the recent Xerox blog, Who Prevails in the E-Discovery War of Man vs. Machine? by Gabriela Baron.

Teams that participated in Oracle without a bona fide SME, much less a good driver, well, they were doomed. The software was secondary. How could you possibly replicate the work of the original SME trial lawyers that did the first search without having an SME yourself, one with at least a similar experience and knowledge level.

map_lost_navigator_SMEThis means that even with a good driver, and good software, if you do not also have a good SME, you can still end up driving in circles. It is even worse when you try to do a project with no SME at all. Remember, the SME in the automobile analogy is the navigation system, or to use the pre-digital reality, the passenger with the map. We have all seen what happens where the navigation system screws up, or the map is wrong, or more typically, out of date (like many old SMEs). You do not get to the right place. You can have a great driver, and go quite fast, but if you have a poor navigator, you will not like the results.

The Oracle study showed this, but it is hardly new or surprising. In fact, it would be shocking if the contrary were true. How can incorrect information ever create correct information? The best you can hope for is to have enough correct information to smooth out the errors. Put another way, without signal, noise is just noise. Still, Jeremy Pickens claims there is a way. I will be watching and hope he succeeds where the alchemists of old always failed.

Tabula Rasa

blank_slateThere is one way out of the SME frailty conundrum that I have high hopes for and can already understand. It has to do with teaching the machine about relevance for all projects, not just one. The way predictive coding works now the machine is a tabula rasa, a blank slate. The machine knows nothing to begin with. It only knows what you teach it as the search begins. No matter how good the AI software is at learning, it still does not know anything on its own. It is just good at learning.

That approach is obviously not too bright. Yet, it is all we can manage now in legal search at the beginning of the Second Machine Age. Someday soon it will change. The machine will not have its memory wiped after every project. It will remember. The training from one search project will carry over to the next one like it. The machine will remember the training of past SMEs.

That is the essential core of my PreSuit proposal: to retain the key components of the past SME training so that you do not have to start afresh on each search project. PreSuit: How Corporate Counsel Could Use “Smart Data” to Predict and Prevent Litigation. When that happens (I don’t say if, because this will start happening soon, some say it already has) the machine could start smart.

Scarlett_Johansson - Samantha in HERThat is what we all want. That is the holy grail of AI-enhanced search — a smart machine. (For the ultimate implications of this, see the movie Her, which is about an AI enhanced future that is still quite a few years down the road.) But do not kid yourself, that is not what we have now. Now we only have baby robots, ones that are eager and ready to learn, but do not know anything. It is kind of like 1-Ls in law school, except that when they finish a class they do not retain a thing!

When my PreSuit idea is implemented, the next SME will not have to start afresh. The machine will not be a tabula rasa. It will be able to see litigation brewing. It will help general counsel to stop law suits before they are filed. The SMEs will then build on the work of prior SMEs, or maybe build on their own previous work in another similar project. Then the GIGO principle will be much easier to mitigate. Then the computer will not be completely dumb, it will have some intelligence from the last guy. There will be some smart data, not just big dumb data. The software will know stuff, know the law and relevance, not just know how to learn stuff.

When that happens, then the SME in a particular project will not be as important, but for now, when working from scratch with dumb data, the SME is still critical. The smarter and more consistent the better. Less Is More: When it comes to predictive coding training, the “fewer reviewers the better” – Parts OneTwo, and Three.

Professor Marchionini, like all other search experts, recognizes the importance of SMEs to successful search. As he puts it:

Thus, experts in a domain have greater facility and experience related to information-seeking factors specific to the domain and are able to execute the subprocesses of information seeking with speed, confidence, and accuracy.

That is one reason that the Grossman Cormack glossary builds in the role of SMEs as part of their base definition of computer assisted review:

A process for Prioritizing or Coding a Collection of electronic Documents using a computerized system that harnesses human judgments of one or more Subject Matter Expert(s) on a smaller set of Documents and then extrapolates those judgments to the remaining Document Collection.

Glossary at pg. 21 defining TAR.

Most SMEs Today Hate CARs
(And They Don’t Much Like High-Tech Drivers Either)

simpsonoldmanThis is an inconvenient truth for vendors. Predictive coding is defined by SMEs. Yet vendors cannot make good SMEs step up to the plate and work with the trainers, the drivers, to teach the machine. All the vendors can do is supply the car and maybe help with the driver. The driver and navigator have to be supplied by the law firm or corporate clients. There is no shortage of good SMEs, but almost all of them have never even seen a CAR. They do not like them. They can barely even speak the language of the driver. They don’t much like most of the drivers either. They are damn straight not going to spend two weeks of their lives riding around in one of those new fangled horseless carriages.

ringo and old guy

That is the reality of where we are now. Also see: Does Technology Leap While Law Creeps? by Brian Dalton, Above the Law. Of course this will change with the generations. But for now, that is the way it is. So vendors work on error minimization. They try to minimize the role of SMEs. That is anyway a good idea, because, as mentioned, all human SMEs are inconsistent. I was lucky to only be inconsistent 23% of the time on relevance. But still, there is another obvious solution.

There is another way to deal today with the reluctant SME problem, a way that works right now with today’s predictive coding software. It is a kind of non-robotic surrogate system that I have developed, and I’m sure a several other professional drivers have as well. See my CAR page for more information on this. But, in reality it is one of those things I would just have to show you in a driver education school type setting. I do it frequently. It involves action in behalf of an SME, and dealing with the driver for them. It places them in their comfort zone, where they just make yes no decisions on the close question documents, although there is obviously more to it than that. It is not nearly as good as the surrogate system in the movie Her, and of course, I’m no movie star, but it works.

HER_Samantha_Surrogate

My own legal subject matter expertise is, like most lawyers, fairly limited. I know a lot about a few things, and am a stand alone SME in those fields. I know a fair amount about many more legal fields, enough to understand real experts, enough to serve as their surrogate or right hand. Those are the CAR trips I will take.

If I do not know enough about a field of law to understand what the experts are saying, then I cannot serve as a surrogate. I could still drive of course, but I would refuse to do that out of principle, unless I had a navigator, an SME, who knew what they were doing and where they wanted to go. I would need an SME willing to spend the time in the CAR needed to tell me where to go. I hate a TAR pit as much as the next guy. Plus at my age and experience I can drive anywhere I want, in pretty much any CAR I want. That brings us to the final corner of the triangle, the variance in the quality of predictive coding software.

Quality of the CAR Software

I am not going to spend a lot of time on this. No lawyer could be naive enough to think that all of the software is equally as good. That is never how it works. It takes time and money to make sophisticated software like this. Anybody can simply add on open source machine learning software code to their review platforms. That does not take much, but that is a Model-T.

Old_CAR_stuck_mud

To make active machine learning work really well, to take it to the next level, requires thousands of programming hours. It takes large teams of programmers. It takes years. It take money. It takes scientists. It takes engineers. It takes legal experts too. It takes many versions and continuous improvements of search and review software. That is how you tell the difference between ok, good, and great software. I am not going to name names, but I will say the Gartner’s so called Magic Quadrant evaluation of e-discovery software is not too bad. Still, be aware that evaluation of predictive coding is not really their thing, or even a primary factor for rating review software.

Gartner_Magic_Quadrant

It is kind of funny how pretty much everybody wins in the Gartner evaluation. Do you think that’s an accident? I am privately much more critical. Many well known programs are very late to the predictive coding party. They are way behind. Time will tell if they are ever able to catch up.

Still, these things do change from year to year, as new versions of software are continually released. For some companies you can see real improvements, real investments being made. For others, not so much, and what you do see is often just skin deep. Always be skeptical. And remember, the software CAR is only as good as your driver and navigator.

car_mind_meld

When it comes to software evaluation what counts is whether the algorithms can find the documents needed or not. Even the best driver navigator team in the world can only go so far in a clunker. But give them a great CAR, and they will fly. The software will more than pay for itself in saved reviewer time and added security of a job well done.

Deja Vu All Over Again. 

Predictive coding is a great leap forward in search technology. In the longterm predictive coding and other AI-based software will have a bigger impact on the legal profession than did the original introduction of computers into the law office. No large changes like this are without problems. When computers were first brought into law offices they too caused all sorts of problems and had their pitfalls and nay sayers. It was a rocky road at first.

Ralph in the late 1980s

I was there and remember it all very well. The Fonz was cool. Disco was still in. I can remember the secretaries yelling many times a day that they needed to reboot. Reboot! Better save. It became a joke, a maddening one. The network was especially problematic. The partner in charge threw up his hands in frustration. The other partners turned the whole project over to me, even though I was a young associate fresh out of law school. They had no choice. I was the only one who could make the damn systems work.

Ifloppy_8incht was a big investment for the firm at the time. Failure was not an option. So I worked late and led my firm’s transition from electric typewriters and carbon paper to personal computers, IBM System 36 minicomputers, word processing, printers, hardwired networks, and incredibly elaborate time and billing software. Remember Manac time and billing in Canada? Remember Displaywriter? How about the eight inch floppy? It was all new and exciting. Computers in a law office! We were written up in IBM’s small business magazine.

For years I knew what every DOS operating file was on every computer in the firm. The IBM repair man became a good friend. Yes, it was a lot simpler then. An attorney could practice law and run his firm’s IT department at the same time.

ralph_1990sHey, I was the firm’s IT department for the first decade. Computers, especially word processing and time and billing software, eventually made a huge difference in efficiency and productivity. But at first there were many pitfalls. It took us years to create new systems that worked smoothly in law offices. Business methods always lag way behind new technology. This is clearly shown by MIT’s Erik Brynjolfsson and Andrew McAfee in their bestseller, Second Machine Age. It typically takes a generation to adjust to major technology breakthroughs. Also see Ted Talk by Brynjolfsson with video.

I see parallels with the 1980s and now. The main difference is legal tech pioneers were very isolated then. The world is much more connected now. We can observe together how, like in the eighties, a whole new level of technology is starting to make its way into the law office. AI-enhanced software, starting with legal search and predictive coding, is something new and revolutionary. It is like the first computers and word processing software of the late 1970s and early 80s.

It will not stop there. Predictive coding will soon expand into information governance. This is the PreSuit project idea that I, and others, are starting to talk about. See Eg: Information Governance Initiative. Moreover, many think AI software will soon revolutionize legal practice in a number of other ways, including contract generation and other types of repetitive legal work and analysis. See Eg: Rohit Talwar, Rethinking Law Firm Strategies for an Era of Smart Technology (ABA  LPT, 2014). The potential impact of supervised learning and other cognitive analytics tools on all industries is vast. See Eg: Deloitte’s 2014 paper: Cognitive Analytics (“For the first time in computing history, it’s possible for machines to learn from experience and penetrate the complexity of data to identify associations.”); Also see: Digital Reasoning software, and Paragon Science software. Who knows where it will lead the world, much less the legal profession? Back in the 1980s I could never have imagined the online Internet based legal practice that most of us have now.

The only thing we know for sure is that it will not come easy. There will be problems, and the problems will be overcome. It will take creativity and hard work, but it will be done. Easy buttons have always been a myth, especially when dealing with the latest advancements of technology. The benefits are great. The improvements from predictive coding in document review quality and speed are truly astonishing. And it lowers cost too, especially if you avoid the pits. Of course there are issues. Of course there are TAR pits. But they can be avoided and the results are well worth the effort. The truth is we have no choice.

Conclusion

retire

If you want to remain relevant and continue to practice law in the coming decades, then you will have to learn how to use the new AI-enhanced technologies. There is really no choice, other than retirement. Keep up, learn the new ways, or move on. Many lawyers my age are retiring now for just this reason. They have no desire to learn e-discovery, much less predictive coding. That’s fine. That is the honest thing to do. The next generation will learn to do it, just like a few lawyers learned to use computers in the 1980s and 1990s. Stagnation and more of the same is not an option in today’s world. Constant change and education is the new normal. I think that is a good thing. Do you?

Leave a comment. Especially feel free to point out a TAR pit not mentioned here. There are many, I know, and you cannot avoid something you cannot see.




Part Three of Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead, Long Live the Whole Man

February 1, 2014

This is the final part of a three-part blog. You will need to read the first two segments for this conclusion to makes sense. See Part One and Part Two of Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead, Long Live the Whole Man.

Final Word From Dan Ariely 

dan-arielyGetting back to Dan, the psychologist economist, in addition to teaching and running very clever experiments at MIT and Duke, Dan is the founder of an organization with a name that seems both funny and ironic, The Center for Advanced Hindsight. He is also a prolific writer and video maker, both activities I admire. See for instance his informative page at MIT, his blog at DanAriely.com, his several books, and his videos, and even though its slightly boring, see his web page at Duke.

As a final piece of evidence on overreliance on reason I offer more testimony by Professor Ariely’s via another video, one which is not at all boring, I swear. It is called The Truth About Dishonesty. It concludes with a subject near and dear to all lawyers, conflicts of interest. The non-rational impact of such conflicts turns out to be very strong and the law is wise to guard against them. Perhaps we should even step up our efforts in this area? 

Cornerstone Made of Pudding

The scientific experiments of Dan Ariely and others show that the cornerstone of the Law – reasonability – is not made of granite as we had thought, it is made of pudding. You can hide your head in the sand, if you wish, and continue to believe otherwise. We humans are quite good at self-delusion. But that will not change the truth. That will not change quicksand into granite. Our legal house needs a new and better foundation than reason. We must follow the physicists of a century ago. We must transcend Newtonian causality and embrace the more complex, more profound truth that science has revealed. The Reasonable Man is a myth that has outlived its usefulness. We need to accept the evidence, and move on. We need to develop new theories and propositions of law that confirm to the new facts at hand.

Science has shown that our current reason-only-based system of justice is on shaky grounds. It is now up to us to do something about it. No big brother government, or super think-tank guru is going to fix this for us. Certainly not scientists either, but they should be able to help, along with technologists, programmers and engineers.

homer-simpson-brain-scan

What are the implications of the findings of unreliable mental processes on the Law and our ability to reach just decisions? We should ask these questions concerning the Law, just like Professor Ariely is asking concerning Economics. Our fundamental legal assumption that all people can act out of reason and logic alone is false. Decisions made with these faculties alone are the exception, not the rule. There are a number of other contributing factors, including emotions, intuition, and environment. What does this mean to negligence law? To sanctions law? Now that the Reasonable Man is dead, who shall replace him?

Just as classical economic theory has had it all wrong, so too has classical legal theory. People are not built like reasonable machines. That includes lawyers, judges, and everyone else in the justice system, especially the litigants themselves.

If Not Reason, Then What?

Ralph_moustacheSince human reason is now known to be so unreliable, and is in fact, only a contributing factor to our decisions, on what should we base our legal jurisprudence. My answer is in the title of this blog. I believe that the Reasonable Man, now that he is known to be an impossible dream, should be replaced by the Whole Man. Our jurisprudence should be based on the reality that we are not robots, not mere thinking machines. We have many other faculties and capabilities beyond just logic and reason. We are more than math.

So I propose a new, holistic model for the law. It would still include reason, but add our other faculties. It would incorporate our total self, all human attributes. We would include more than logic and reason to judge whether behavior is acceptable or not, to consider whether a resolution of a dispute is fair or not.

A new schemata for a holistic jurisprudence would thus include not just human logic, but also human emotions, our feelings of fairness, our intuitions of what is right and just, and multiple environmental and perceptual factors. I suggest a new model start simple and use a four-fold structure like this, and please note I keep Reason on top, as I still strongly believe in its importance to the Law.

4-levels-Holistic_Law_pyramid

Some readers may notice that this model is similar to that of Carl Jung’s four personality types and the popular Myers Briggs personality tests. I am not advocating adoption of any of their ideologies, or personality theories, but I have over the years found their reference models to be useful. The above model, which is proposed only as a starting point for further discussion, is an extrapolation of these psychological models.

Call For Action

No one knows yet knows the full implications of the new data from science about the limited impact of logic and reason on human decisions. No one knows how to modify our legal systems to account these insights. Certainly I do not. But I do know that we should do something to reduce our overreliance on the Myth of the Reasonable Man. We should put the foundations of our legal system on something else, something more solid, more real than that.

In short, we need to put our house in order before it starts collapsing around us. That is the reasonable thing to do, but for that very reason we will not start to do it until we have better motivation than that. You cannot get people to act on reason alone, even lawyers. So let us engage the other more powerful motivators. To start the ball rolling, I will give special recognition and publicity to the best suggestions received from my readers to this problem, the best comments to this blog.

Maybe reason alone should always be secondary to simple fairness? Maybe that feeling of fairness, is more reliable than reasoned processes. Run the experiments please scientists. How reliable are our feelings of fairness? More importantly, what is the impact of feelings on our judges who pay attention to that? Maybe feelings should be on top of the new Holistic model. I personally doubt that, but who knows for sure until experiments are done. What I do not doubt is that feelings need to be taken into consideration more than they are now as true motivators of human action.

Maybe this means we should bring back equity, and down play law, like the old days, where we used to have Courts of Law and separate Courts of Equity. By the middle of the last century, Courts of Law won out in most states except Delaware, Mississippi, New Jersey, South Carolina, and Tennessee. Separate Equity Courts were closed down in favor of Courts of Law. Maybe we got it backwards. Maybe we were all led astray by our false confidence in reason.

Maybe we should now close our courts of Law and reopen our courts of Equity? How has it worked out for the states that kept equity courts? Have Chancellors truly been able to side-step strict rules of law when they felt it was equitable to do so? If so, how has that worked out? Has power been abused? Or has justice been attained more often? What can we learn from chancery courts that might help us build a more holistic court of the future?

A Few More Specific Suggestions of Reform

As discussed already, the AI enhancements now moving the law will continue to expand as a tool for the lawyers willing to learn how to use them. They will enhance and help improve our limited reasoning abilities. They will help us be more efficient.  They could also help us to stay completely honest, if we allow them to. So too will more emotional, in your face type judges, whether we let them or not. We need more judges who do not mind getting down into the weeds, to really understand the facts, and then tell you what they really think, both good and bad please.

Maybe timely reminders of ethics codes and serious under penalties of perjury type threats will also help? Maybe new, improved, and customized oaths will help? Oaths have been shown to be effective by Ariely’s research, so we should modify the rules accordingly.

Electrodes_EEG_RalphMaybe new truth recognition technologies should be used? Could a truth hat with built-in neural net be that far off? How about Google Glasses apps that provide reliable new feedback of all kinds on the people you watch testifying? That cannot be too far off.  (The lie detection apps already on the market for iPhones, etc., all look bogus to me, which is not unexpected based on the limited biofeedback the phone sensors can provide.) Even if the information is not admissible as evidence, it could still be quite valuable to lawyers. (Write me if you know of anyone working on any commercial projects like this for lawyers.) Perhaps some of the recent discoveries in neuroscience could begin to be used in the justice system in all types of unexpected ways?

trophy_LawMaybe public recognition and awards to lawyers and judges who get it right will help? And awards to litigants who do the right thing, even if they lose the case? How about a discretionary set-off for defendants like that? How about the converse? Shame can be a powerful motivator too.

Maybe we should change the conditions and environments of places where witnesses are questioned, where mediations and trials are conducted? Maybe we should provide special training to court reporters on oath giving? Maybe we should have trials again, and not just settlements?

We need to look for all kinds of motivators. Knowledge and reason alone are not a solid foundation for justice.

Conclusion

Changes are inevitable anyway in all social structures, so we should try to shape the ongoing changes in the Law. We should study what science has found and be guided by truth, not tradition.

We should try to move away from overreliance on reason alone. Where we must still rely on reason, and of course we must, we should look for science and technology based methods to impose even more checks and balances on reason than we already have. We should create new systems that will detect and correct the inevitable errors in reason that all humans will make – lawyers, judges and witnesses alike. Perhaps computers can help with this? Perhaps it would help to have easier and less expensive appeals? Especially interlocutory appeals? Perhaps greater use of experts, panels and special masters? We really need to start focusing on this, and, by the way, we cannot just think our way out of a prison of thought. We need to use all of our faculties.

We also need help from the scientific community. We need someone like Professor Ariely to focus on Law the way he has focused on Economics. So far I have not found anyone like that.

Please feel free to share any ideas you may have in the Comments to this blog below, or by private email to me. Again, the best comment will be recognized and praised. I may even give you some shout-outs at LegalTech this week. By the way, if you see me there, please take a moment to stop me and introduce yourself. I always like to meet my readers. If you know of any research psychologists who might be interested in these issues, please share this blog with them. I have already reached out to Dan Ariely. He responded right away and promised to provide a more detailed reaction later. When he does I will share his input in a later blog.


Part Two of Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead, Long Live the Whole Man

January 26, 2014

This is the second part of a three-part blog. You will need to read the first part for this segment to make sense. See Part One of Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead, Long Live the Whole Man.

Doing the Right Things for the Wrong Reasons

Dan_Ariely_toastAre you a die-hard rationalist and demand more proof that the Reasonable Man is a myth? More evidence? Then listen to Dan Ariely’s Doing The Right Things for The Wrong Reasons. Professor Ariely talks about more of his experiments. They show how immediate, tangible, emotions and concrete facts are a much more powerful motivator than all abstract knowledge. This means that one sanctions case invoking fear will do much more to encourage cooperation than a thousand law review articles. In my experience judges that threaten harsh punishment, that are known not to tolerate discovery misconduct, tend to have fewer disputes. Now we know why. Fear is a more powerful motivator than reason. For some people a good glass of wine is a powerful motivator too.

Professor Ariely’s testimony in this video examines the big gap between everyone’s knowledge of what they should be doing, and what they actually are doing. The truth is, we often do not act reasonably. There are many other more powerful forces at work. One of the most important is environment, and thus my earlier comments on impressive court rooms, wigs, courtly conduct, and the like.

In the second half of his testimony Dan Ariely started to share some of the solutions he has come up with to these problems, ways to trick yourself and others into doing the right thing. One such motivator is public recognition, pride. Remember the Prius. So how about Cooperation awards of lawyers? Proportionality awards for judges, etc. Let’s award a whole lot of gold, silver and especially bronze medals. I am serious about this awards and recognition proposal. If you have any interest in funding such awards, or otherwise being involved, please let me know. This would be a good opportunity for vendors in the legal space, especially e-discovery vendors.

Mere intellectual appeals to change behavior are almost useless. Proclamations included. You have to persuade the whole human, and that requires addressing emotions and many other subconscious factors. That requires far more than abstract, knowledge-based writings.

The Power of Emotions and the Myth of Reasoned Behavior

The power of emotions, and immediate gratification, should never be underestimated. This includes the positive motivators, like praise and recognition. An active judiciary can do much more to impact reasonable, ethical conduct than all appeals to reason. Judges need to be in your face, with both criticism and praise, stick and carrot. The motivations need to be immediate and real, not abstract and future oriented. See eg. Victor Stanley and the impact of Judge Grimm’s threats of immediate imprisonment of Pappas, the ultimate hide-the-ball litigant. Only that last jail contempt order slowed the games.

This all reminds me of Judge Waxse’s quip that lawyers are like elementary particles, they change when observed (by judges). He has found that lawyers are more inclined to cooperate simply by including a possibility that a judge might someday watch a video of their behavior. Maybe we should require that all lawyer-to-lawyer communications be taped? Maybe we should triple the number of judges and give them all sensitivity training? Who knows? But the research shows that all manner of alternatives like that would be more successful than mere appeals to reason alone.

Waxse_Losey

This all makes me wonder why I even bother to continue to write, but then again, you may have noticed that I try to include non-rational appeals in my writing, such as images and the great irrational motivator of humor. Humor is an elusive emotion to reach, but well worth the effort. It is difficult to resist the ideas of anyone who makes you laugh. Personally, I refuse to emulate anyone who does not at least make me smile. If they make me laugh out loud, well, I will dig in deep to try to understand them and their ideas.

Why else do you think I quote Jason R. Baron so much and like to present with him? He is always amusing, on and off stage. Even his arcane intellectual references can be funny; depends on the delivery. That is amazing when you consider that he is usually presenting on records management, now lately called information governance (apparently an attempt to make the topic seem vaguely interesting). They say that a smart comic can make even the reading of a phone book seem funny.

Attend some of my events with Jason at LegalTech and see for yourself. We are bound to at least make you smile. I promise to talk about killer robots, while Jason explains how AI will soon change information governance. Who knows, in the future maybe even the federal government will stop printing and filing emails as paper. But I digress.

Law is Like Economics:
Both Are Still Based on an Irrational Reliance on Reason

As you have seen from the videos, Dan Ariely is not only witty, but also a psychologist and an economist.  He has one PhD in Psychology and another in Business Administration. He is also an author of a number of books that explain his works to the general reader, including the best seller: Predictably Irrational: The Hidden Forces That Shape Our Decisions.

Dan evaluates the implications of his irrationality findings in Psychology on the field of Economics. So too are many other pundits in the field. See eg. Post-Rational Economic Manand Exploring the Post-Rational 21st CenturyAriely and others have amassed a growing body of evidence that humans are not rational machines. Yet most economists, much like most lawyers, do not believe that. They still believe that people make rational decisions. For instance, that purchases are based on reason alone. See Rational Choice Theory. That is the basis of classic economic theory, and since that presumption is wrong, so is the theory. Economics is now struggling with the development of new theories based on the way people really act. Dan is a leader of that movement, which he calls Behavioral Economics.

Learning a little about Dan’s insights and proposals to reform economic theories, and make them more realistic, and empirically based, can provide insights into the Law and reforms we should make. Surely we can do better than propose more videotapes of lawyers, in your face judges, bibles and oaths, solemn court reporters, and British style ceremonial conduct. But these are a start.

More fundamentally, we need to consider how we should speak of legal negligence in the future. We need to stop referring to whether an act is reasonable, and instead speak of acceptability, with reason just one of several factors to consider in evaluating acceptable behavior. That is what call, for lack of a better term, Holistic Jurisprudence. More on that later. Perhaps some law professors and judges are already thinking and writing about this, and I am not aware of their writings. (If they are, or you are, please let me know.) If not, then what are we waiting for? The evidence of innate irrationally based,  yet acceptable, behavior, is strong. That is our everyday reality. So why do we use a measure of acceptable conduct that does not mirror reality? Legal theory needs to change as much as economic theory, and so too does legal practice.

Robots and Neuroscience?

Facc_RobotI know what you are thinking. Maybe the answer is simply to turn our justice system over to robots programmed to make rational decisions. They will not suffer from innate irrationality like our judges do. (Yes, even judges are human and thus even judges suffer from the same cognitive disorders, same irrational drivers, that other humans do). Rational machines could also be programmed to fairly consider the innate irrationality of humans. We could create super robojudges by using active machine learning. They could receive training in just-decision-making by our top judges. Imagine, for instance, the wisdom and wit of Judge Facciola programmed into an AI entity. The input from our top judges would thereby, in theory at least, live forever. The experience and intelligence of our best judges would then be available to all litigants, not just the lucky few who appear before them. This puts a while new positive spin onto the Ghost in the Machine image.

The AI enhanced robojudges would, of course, be far more than mere rational machines. They would be trained by our legal experts to render judgments based on the Whole Man, one that actually exists, and not the legal fiction of the Reasonable Man. They would be programmed in a post-rational manner following models of real human behavior of acceptable conduct. (Our best human judges and lawyers already do that anyway, even if the jurisprudence theory says otherwise.) The day will come when many litigants will prefer smart, well-trained robots to serve as judges to evaluate acceptable conduct, especially when there are good human appeals judges to oversee the process.

It is inevitable that we will use artificial intelligence and big data in some way to reform the judicial system, to make it more effective. It has already happened in document review. Its application to the bench and judicial decision-making is also inevitable.

AI and big data will change the way judicial decisions are made, but robojudges? No. That may be the endgame, but I do not think we are ready for that, yet. Judge Facciola’s job is safe, even if his law clerk’s job is not. Unless, of course, Ray Kurzweil is right about the Singularity coming soon, then all bets are off. But Kurzweil is probably wrong about how fast AI will advance, and so I do not see this anything like this happening in the first half of this century. Computers now all have human programmers and programming errors. Who do you think would be designing the robojudges’ technology? Do you really want to replace our judges with machines like that?

No, that is not The answer (at least not yet), but it may be part of the answer. The use of AI enhanced tools in the law, such as what we call predictive coding for document review, is just beginning. It will continue and expand into many other legal activities. Very soon many more types of lawyers in addition to contract review lawyers will need to retool in order to stay employed. Their tasks will be automated, and they will be out of work. At the same time new employment positions will open for those involved in the new technologies. The jobs that open up will require greater skills, intellect, empathy, leadership, creativity and imagination. They will require uniquely human attributes way beyond the programming of any robots, now and perhaps forever.

I do not know exactly how it will play out, but, even if our judges remain human, new advanced technologies will necessarily be part of all future legal reforms. Many of the technologies are probably still unknown and thus impossible to project. But some will be based on existing technologies, just significantly improved.

Facciola_computerPerhaps that will include active machine learning and AI based law clerks for judges. It is not hard to imagine a judge’s consideration of an AI enhanced suggested view of the case. After all, they already do this based on their clerk’s views. I suspect judicial clerks will be replaced way before the judges themselves. Judges need to be enhanced with better computers, not replaced by them.

To take a more mundane example than robots and AI, I suspect that lie detection technologies will soon advance enough to be of greater assistance to the Law. How about acceptably intrusive truth-compelling technologies? I can easily imagine neural nets with electronic brain monitors built into “truth hats.” Witnesses would be required to wear the truth-indicating hats and give the attorneys, judges and juries more and better insights into their testimony. Not only intentional lies could be revealed, but strength of recollection, areas of brain accessed, etc. This would not have to be dispositive, but suggestive. This could provide us with something more to evaluate credibility than raw instinct and intuition, as important as these faculties are.

Meet-the-Parents-lie-detector with Harry Potter twist

We should be looking for all kinds of ways to bring the recent incredible advances in Neuroscience into the justice system. This is not futuristic science fiction, nor Losey’s over-active imagination. It is already happening. Many neuroscientists are looking into lie detection and other possible neuroscience applications in the Law. See eg Harvard’s Center for Law, Brain and Behavior and its program on Lie Detection & the Neuroscience of Deception.

END OF PART TWO.
Part three will follow next Sunday as I pack for LegalTech.


Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead, Long Live the Whole Man

January 19, 2014

brain_gearsThe Reasonable Man on which the law is based is a fiction of our collective legal imagination. He does not exist. Never has, never will. We humans are much more complex than that. Although reasoning is important, it is only one of our many capacities, including imagination. Most of our decisions are not even based on reason. Quaint notions to the contrary from the 18th Century Age of Reason are out of touch with reality. They are contrary to what science today is telling us about how humans process information and reach decisions.

Scientific research shows that the cornerstone of the Law – Reasonability – is not solid granite as we had thought. There are no hard gears in our head, just soft, gelatinous, pinkish-beige matter. (Our brain is only soft grey matter when dead.) The ratiocination abilities of the brain are just one small part of its many incredible capacities. (For example, recent experiments at MIT have shown that we can identify images seen for as little as 13 milliseconds, 13/1,000ths of one second.) We are far more than just rational, and that is a good thing.

Going Beyond the Age of Enlightenment
Into the Modern Era of Science

This blog will offer proof that the Law’s Reasonable Man is dead. Then I will encourage the profession, starting with you dear readers, to transcend the mere rational. We all need to change our work to include more of our human capacities. This does not mean a return to the Dark Ages and the discovery of truth by torture and combat. It means following the inevitable dictates of the Age of Reason, that we be guided by the findings of science and objective repeatable, experiments, no matter how irrational these findings may at first seem. To refuse to accept the truth, no matter how different it is from your current beliefs, is itself an irrational carryover from the Dark Ages. We must boldly go where science and reason takes us. The world is not flat and we are not governed by reason alone. We are far more than a thinking machine. We must open our eyes and see the truth. That is the true meaning of the Age of Enlightenment.

quantum-physics_headScience, based on reason and the experimental method, has taken Man beyond the rational, has shown the limitations of reason. Just as the evidence from physics experiments forced scientists to go beyond Newtonian Causality, and required them to embrace the seemingly irrational truth of Relativity and Quantum Mechanics, so too must the Law now evolve its thinking and procedures. As proof for this proposition in this blog I will proffer the testimony of one expert witness, a noted MIT and Duke University Psychologist and Behavioral Economist.

The Legal Profession Must Awaken from the Daydream of Rationality

My last blog, The Psychology of Law and Discovery, laid the foundation for the introduction of this evidence. I noted how law is based on the assumption that people make reasoned decisions and are capable of acting in a reasonable manner. I offered preliminary evidence that this assumption is contrary to the findings of research psychologists. I referred to a recent article by one such psychologist, Herb Roitblat, who is also an expert in legal search: The Schlemiel and the Schlimazel and the Psychology of Reasonableness (Jan. 10, 2014, LTN). I will now offer further, more detailed proof that humans do not act out of reason. I will do so by use of videotaped expert testimony of sorts. I will then argue that these findings require us to make fundamental reforms to our system of justice.

The consequences to the Law of the new experimental findings are profound. They raise many questions for which I have only a few preliminary answers. Many more questions will arise I am sure. This is much bigger than any one lawyer, or one or two blogs. The entire profession will have to awaken from the daydream of rationality. This is just the start of the discussion. We need to work together to change our system of justice to conform to the evidence of irrational behavior that science has uncovered.

This evidence is abundant. With only a little search I am sure you will find much more proof than I will now proffer. This is solid scientific evidence based on verifiable experiments. The evidence proves that our assumptions made in the law as to human reasonability, assumptions built centuries ago when the Age of Reason first began, are false assumptions. The evidence shows that the Reasonable Man is a legal fiction.

As Exhibit “A” to the assumption busting proposition I rely on the work of Dan Ariely, a Professor of Psychology and Behavioral Economics at Duke University. As an introduction to his work I offer a TED video of Professor ArielyAre We In Control of Our Own Decisions? He refers to his many scientific experiments at MIT, then Duke, that show we are not in control of many of our own decisions, even seemingly simple ones. These experiments prove my point. Listen carefully.

Predictable Irrationality and Swearing on Bibles

Need more proof? Then please consider additional testimony from Professor Ariely on predictable irrationality. This discourse even mentions every e-discovery lawyer’s favorite company, Enron, and examines our basic moral code, our personal fudge factor. Dan has conducted many experiments on the all too human tendency to cheat and lie, if only just a little, and the moving grey line between acceptable and unacceptable behavior. This is the line that the Law is constantly asked to draw, and to evaluate. These psychological insights are important to all lawyers, especially discovery lawyers, of the “e” only type like me, or not. Again, please listen carefully and consider the implications of these findings on the Law.

One interesting finding from Professor Ariely’s scientific experiments on cheating, one that you can easily miss in the video (see around frame 8:15), even if you can see 77 frames per second, is that asking people to swear on a Bible significantly reduces cheating. This even works for atheists! I kid you not. Perhaps we should bring back the old tradition of requiring all witnesses to swear on a bible before beginning their testimony?

Ralph_swearing_oath_bibleI have done this myself long ago when I was out taking depositions as a young lawyer. In the early eighties many court reporters in rural counties of Florida would still pull out a Bible before a deposition began (they all used to carry them around for that purpose, and yes, that was way before they started carrying around computers). The court reporter would then ask the deponent to raise their right hand and put their left hand on the Bible. All the witnesses I saw instantly complied, thinking erroneously that this was a legal requirement. They placed their hand on the Bible, some nervously, and some like they did that all the time, and then were asked to solemnly swear on the Bible that they would tell the truth, the whole truth and nothing but the truth so help me God. They did as asked by the serious court reporter, and some seemed pretty impressed by the whole ceremony. I recall that overall the testimony from these witnesses was pretty good.

Flying Spaghetti Monster

I only saw this done a few times, and, as a typical arrogant big city lawyer (yes, out in the rural areas where they were still doing this, they all thought of Orlando as a big city), I dismissed it as a quaint old custom. But now science shows that it works. Science shows that this quaint custom works, even for members of the Church of the Flying Spaghetti Monster.

What are the implications of these findings about human behavior? Maybe we should bring back Bibles into the court rooms? Or at least bring back a bunch of solemn oaths? If we do not require swearing on or to a Bible, due to Church and State, or whatever, then perhaps we should ask people giving testimony to swear on something else. Most anything seems to work, even if it does not really exist. Dan Ariely’s experiments found that it even worked to have MIT students swear on an honor code that didn’t exist. Maybe asking lawyers to swear on their ethics codes would work too? Maybe that is the reform in the procedural rules we should be pushing for, instead of Rule 37(e)? Maybe we should update Rule 603 of the Federal Rules of Evidence:

Before testifying, a witness must give an oath or affirmation to testify truthfully. It must be in a form designed to impress that duty on the witness’s conscience.

prisoner_ralph_chainsWe need to work on forms designed to impress today’s savvy witnesses. Maybe bringing back Bibles will work for some, or something custom-fit to the particular witnesses. Who knows, for a chemist, it might be the periodic table. For others it might be a picture of their mother. Maybe the oath should be administered by prisoners in chains and mention the penalties of imprisonment for perjury. I think that would be pretty effective. Have you ever seen prisoners in chains up close in the court room? A few judges I know used to handcuff and shackle fathers who were delinquent in child support payments like that before their hearings. I am told it had a very sobering effect. Some experiments with this should be conducted because our current systems are not working very well. We rarely impress witnesses enough to awaken their latent conscience, much less our lawyers.

Maybe we should also amend Rule 26(g) to add swearing and a reference to ethics codes? Maybe stronger, more impressive oaths by lawyers signing 26(g) discovery requests and responses would work. Perhaps that would magically make more all too human lawyers start taking the requirements of the rules more seriously.

Lord Phillips 2009Maybe we should follow the British and make our judges wear fancier robes and make our lawyers and judges wear wigs? (One of Ariely’s experiments found clothing had an impact on honesty.) Let us build even more impressive court rooms while we are at it, and let’s not only say Your Honor, but how about Your Lordship too? Or Your Grace? Maybe all lawyers should start adding courtly formalities to their 26(f) conferences? I can just imagine defense attorneys beginning every one of their responsive statements with things like: “The right honorable attorney representing the plaintiffs in this proceeding has made a point with some validity, but …” Maybe that would motivate lawyer conduct that would in fact please the court?

Of course I jest, but Ariely’s work shows that irrational approaches have a better chance of success than appeals to abstract knowledge alone. Forget about using reason to appeal to lawyers to cooperate, we have all seen how far that gets us.

END OF PART ONE.
Part two will follow next Sunday. I swear.


The Psychology of Law and Discovery

January 12, 2014

head_psych_rationalReasonability is a core concept in the law, right up there with the idea of justice itself. It not only permeates negligence law, it underlies discovery law as well. For instance, a party in litigation, and the attorneys representing them, are required to make reasonable efforts to find relevant documents requested. They are required to make efforts that are good enough to be considered reasonable. But lawyers and litigants are not required to make efforts beyond that; not required to make super-human, stellar efforts, and certainly not perfect efforts.

Conversely, litigants and their lawyers are not permitted to make anything less than reasonable efforts to find the information requested. They are not permitted to make sub-standard, negligent efforts, and certainly not  grossly negligence efforts. Let us not even talk about intentionally obstructive or defiant efforts. The difference between best practice and malpractice is where the red line of unreasonable negligence is drawn.

justice_guage_negligence

What is Reasonable?

What does it mean to be reasonable and to make reasonable efforts? The answer to that question fills a million legal volumes. It is the subject of countless judicial opinions and statutes. What is reasonable and thus acceptable efforts, and what is not, and the dividing line between the two, can often be difficult to determine. It depends on a careful evaluation of the facts, to be sure, but this evaluation in turn depends on many subjective factors. These factors include prevailing social norms, personal values, prejudices, education, intelligence, and even how the mind itself works, the hidden psychological influences. They all influence a judge’s evaluation in any particular case as to which side of the acceptable behavior line a particular course of conduct falls.

In close questions the subjectivity inherent in determinations of reasonability is obvious. This is especially true for the attorneys involved, the ones paid to be independent analysts and objective advisors. People can, and often do, disagree on what is reasonable and what is not. They disagree on what is negligent and what is not. On what is acceptable and what is not.

All trial lawyers know that certain tricks of argument and appeals to emotion can have a profound affect on a judge’s resolution of these supposedly reason-based disagreements. They can have an even more profound affect on a jury’s decision. (That is the primary reason that there are so many rules on what can and cannot be said to a jury.)

Study of Legal Psychology

brain_gearsEvery good student of the law knows this, but how many attempt to study the psychological dynamics of persuasion? How many attempt to study perceptions of reasonability? Of cognitive bias? Not many, and there are good reasons for this.

First and foremost, few teachers exist that have this kind of knowledge. The only ones that I know of are experienced trial lawyers and experienced judges. They know quite a lot about this, but not from any formal or systematic study. They pick up information on the psychological underpinnings of justice by many long years of practice. They learn about the psychology of reasonability through thousands of test cases. They learn by involvement in thousands of disputes as to what is reasonable.

But that is not to say that is the quickest or best way to learn these insights. Perhaps there is an even quicker and more effective way? Perhaps we could turn to psychologists and see what they have to say about the psychological foundations of perception of reasonability. After all, this is, or should be, a part of their field.

Up until now, not very much has been said from psychologists on law and reasonability, at least not to my knowledge. There are a few books on the psychology of persuasion, which I made a point in prior years of studying to try to become a better trial lawyer. But in fact, the field is surprisingly thin. There is not much there. It turns out that the two fields of Law and Psychology have not overlapped much, at least not in that way.

Perhaps this is because so few psychologists have been involved with legal arguments on reasonability. When psychologists are in the legal system, they are usually focused on legal issues of sanity, not negligence, or in cases involving issues of medial diagnoses.

The blame for the overly wide gulf between the two fields should fall on both sides. Most psychologists, especially research psychologists, have not been interested in the law and legal process. Or when they have, it has involved criminal law, not civil. See eg Tunnel Vision in the Criminal Justice System (May 2010, Psychology Today). This disinterest has been reciprocal. Most lawyers and judges are not really interested in hearing what psychologists have to say about reasonability. They consider their work to be above such subjective vagaries.

Myth of Objectivity

Lawyers and judges consider reasonability of conduct to be an exclusively legal issue. Lawyers and judges like to pretend that reasonability exists in some sort of objective, platonic plane of ideas, above all subjective influences. The just answer can be discovered by deep, impartial reasoning. This is the myth of objectivity.

PlatoAristotle

This myth continues to this day in legal culture, even though all experienced trial lawyers and judges know it is near total nonsense. They know full well the importance of psychology and social norms. They know the impact of cognitive biases of all kinds.

Experienced lawyers take advantage of these biases to win cases for their clients. They know how to push the buttons of judge and jury. See Cory S. Clements, Perception and Persuasion in Legal Argumentation: Using Informal Fallacies and Cognitive Biases to Win the War of Words, 2013 BYU L. Rev. 319 (2013)Justice is sometimes denied as a result.

More often than not the operation of cognitive biases happen unconsciously without any puppet masters intentionally pulling the strings. There is more to this than just rhetoric and sophistry. Justice is hard.

Even assuming that the lawyers and judges in the know could articulate their knowledge, they have little incentive to do so. (The very few law professors with such knowledge do have an incentive, as we see in Professor Clements’ article cited above, but these articles are rare and too academic.) Moreover, most judges and lawyers are incapable of explaining these insights in a systematic manner. They lack the vocabulary of psychology to do so, and, since they learned by long, haphazard experience, that is their style of teaching as well.

Shattering the Myth

This is all background to explain why I was so pleased this week to see that one psychologist has finally taken the time to study these issues and share his insights. They are myth shattering to be sure, and thus will be unwelcome to some idealists. But for me this is a much-needed analysis. The psychologist who has dared to expose the myth, to lift the curtain, is not a lawyer too, but has worked with lawyers for over a decade on discovery issues. He has even co-authored a law review article on reasonability with two distinguished lawyers. Oot, Kershaw, Roitblat, Mandating Reasonableness in a Reasonable Inquiry, Denver University Law Review, 87:2, 522-559 (2010).

Herb RoitblatYes, I’m talking about Herbert L. Roitbalt, who has a PhD in psychology. Herb did research and taught psychology for many years at the University of Hawaii, where, coincidently, he was friends with my brother George Losey, a professor of marine biology at UH. Only after a distinguished career as a research psychologist and professor did Herb turn his attention to computer search in general, and then ultimately to law and legal search, where he co-founded OrcaTec.

Schlemiel and Schlimazel

Herb has just written a small gem of a paper on law and reasonability. It is a must read for everyone, especially those who do discovery. The Schlemiel and the Schlimazel and the Psychology of Reasonableness (Jan. 10, 2014, LTN). I will not spoil the article by telling you Herb’s explanation of the Yiddish terms, Schlemiel and Schlimazel, nor what they have to do with reasonability and the law, especially the law of spoliation and sanctions. Only a schmuck would do that. Let me just say that I agree completely with his insights. They are insights that took me over 35 years to learn the hard way as a trial attorney and then discovery attorney.

Herb Roitblat’s article should be required reading for all judges and lawyers, especially new ones. Even those intelligentsia deep into the Sedona bubble should take time to study this. Understanding the many inherent vagaries of reasonability could, for instance, lead to a much more civil discourse on the subject of sanctions. Who knows, it could even lead to real dialogue and cooperation, instead of the theatre and politics we now see everywhere, even including the Sedona bubble.

Hindsight Bias

hind-sightRoitblat’s article contains a two paragraph introduction to another important psychological factor at work in many evaluations of reasonability: Hindsight Bias. This has to do with the fact that most legal issues consider past decisions and actions that have gone bad. The law almost never considers good decisions, much less great decisions with terrific outcomes. Instead it focuses on situations gone bad, where it turns out that wrong decisions were made. But were they necessarily negligent decisions?

The mere fact that a decision led to an unexpected poor outcome does not mean that the decision was negligent. But when we examine the decision with the benefit of 20/20 hindsight, we are naturally inclined towards a finding of negligence. In the same way, if the results prove to be terrific, the hindsight bias is inclined to perceive most any crazy decision as reasonable.

Due to hindsight bias, we all have, in Rotiblat’s words:

[A] tendency to see events that have already occurred as being more predictable than they were before they actually took place. We over-estimate the predictability of the events that actually happened and under-estimate the predictability of events that did not happen.  A related phenomenon is “blame the victim,” where we often argue that the events that occurred should have been predicted, and therefore, reasonably avoided.

Hindsight bias is well-known among experienced lawyers and you will often see it argued, especially in negligence and sanctions cases. Every good lawyer defending such a charge will try to cloak all of the mistakes as seemingly reasonable at the time, and any counter-evaluation as merely the result of hindsight bias. They will argue, for instance, that while it may now seem obvious that wiping the hard drives would delete relevant evidence, that is only because of the benefit of hindsight, and that it was not at all obvious at the time.

Good judges will also sometimes mention the impact of 20/20 hindsight, either on their own initiative, or in response to defense argument. See for instance the following analysis by Judge Lee H. Rosenthal in Rimkus v Cammarata, 2010 WL 645253 (S.D. Tex. Feb. 19, 2010):

These general rules [of spoliation] are not controversial. But applying them to determine when a duty to preserve arises in a particular case and the extent of that duty requires careful analysis of the specific facts and circumstances. It can be difficult to draw bright-line distinctions between acceptable and unacceptable conduct in preserving information and in conducting discovery, either prospectively or with the benefit (and distortion) of hindsight. Whether preservation or discovery conduct is acceptable in a case depends on what is reasonable,and that in turn depends on whether what was done–or not done–was proportional to that case and consistent with clearly established applicable standards.  [FN8] (emphasis added)

Judge Shira A. Scheindlin also recognized the impact hindsight in The Pension Committee of the University of Montreal Pension Plan, et al. v. Banc of America Securities, et al.,2010 WL 184312 (S.D.N.Y. Jan. 15, 2010) at pg. 6:

While many treatises and cases routinely define negligence, gross negligence, and willfulness in the context of tortious conduct, I have found no clear definition of these terms in the context of discovery misconduct. It is apparent to me that these terms simply describe a continuum. FN9 Conduct is either acceptable or unacceptable. Once it is unacceptable the only question is how bad is the conduct. That is a judgment call that must be made by a court reviewing the conduct through the backward lens known as hindsight. It is also a call that cannot be measured with exactitude and might be called differently by a different judge. That said, it is well established that negligence involves unreasonable conduct in that it creates a risk of harm to others, but willfulness involves intentional or reckless conduct that is so unreasonable that harm is highly likely to occur. (emphasis added)

The relatively well-known backward lens known as hindsight can impact anyone’s evaluation of reasonability. But there are many other less obvious psychological factors that can alter a judge or jury’s perception. Herb Roitblat mentions as few more such as the overconfidence effect, where people tend to inflate their own knowledge and abilities, and framing, an example of cognitive bias where the outcome of questions is impacted by the way they are asked. The later is one reason that trial lawyers fight so hard on jury instructions and jury interrogatories.

Encore!

My only problem with Herb’s article on the psychology of reasonableness is that it is too short. It leaves me wanting more. I hope that this is just a teaser to a full exposition of the subject by Dr. Roitblat. What we really need is a full-blown scholarly article, one that will encourage more psychologists to become familiar with the law and legal process. We need a full, systematic, detailed exposition. Herb has the right background and qualifications to deliver. I urge OrcaTec to give their chief scientist the time to write a full paper on this. The profession will appreciate it, I am sure. Resistance is futile.

We also need independent research in this area by full-time psychologists. Their findings could help guide reforms in the law, reforms that could make our justice system more fair and consistent, and less subject to the whims of individual jurists. Herb, if not you, perhaps you could persuade some of your psychology colleagues to pick up the torch here?

science_tech_law_triangleThe legal profession needs this kind of interdisciplinary cooperation. We have already benefitted greatly by the input of information scientists. Now we could use help from scientists in psychology, especially research scientists and those who study how decisions are made and the hidden impact of psychological processes. We need to have a much better understanding of all of the cognitive disorders and biases that impact legal decision making.

Many lawyers are interested in this intersection and the benefits that might be gained by cross-pollination of knowledge. I have a life-long interest in psychology, and so do many others, some with advanced degrees. That includes e-discovery expert, Maura R. Grossman, who has a Ph.D. in Clinical/School Psychology. A good discovery team needs the insights of both information scientists and psychologists.

Conclusion

TJusticehe myth of objectivity must be exposed. Many naive people still put all of their faith in rules and the operation of objective, unemotional, Platonic ideals. The Schlemiel and the Schlimazel and the Psychology of Reasonableness helps shatter that myth. But myth shattering is not enough. It is just the beginning. We must study the reality of what is happening in our courts, especially in determinations of what is reasonable. Law is more than logic and reason. It is about people and by people. Lawyers and judges are human, all too human. We need to better understand the subjective influences and psychological processes at play in all judicial decisions. Only in that way can the cause of justice continue to advance.


Follow

Get every new post delivered to your Inbox.

Join 3,106 other followers