Animation Showing How Not To Cooperate In An E-Discovery Conference

February 20, 2012

To my dear friends in the e-Discovery team community who prefer my videos and graphics to my many worrrrds, this blog’s for you.

___________

Don’t be a pseudo-cooperator. Make reasonable disclosures.

______

All characters appearing in this work are fictitious.

Any resemblance to real persons, living or dead, is purely coincidental.


“The Hacker Way” – What e-Discovery Can Learn From Facebook’s Culture and Management

February 5, 2012

Facebook’s regulatory filing for its initial public stock offering includes a letter to potential investors by 27 year old billionaire Mark Zuckerberg. The letter describes the culture and approach to management that he follows as CEO of Facebook. Zuckerberg calls it the Hacker Way. Mark did not invent this culture. In a way, it invented him. It molded him and made him and Facebook what they are today. This letter reveals the secrets of Mark’s success and establishes him as the current child prodigy of the Hacker Way.

Hacker History

This tradition and way of thinking and doing have been around since at least the sixties. It has little or nothing to do with illegal computer intrusions. It is often called the hacker ethic and, surprisingly, arose out of the hobby of model railroad building and MIT computer labs. This philosophy is well-known and has influenced many in the tech world, including the great Steve Jobs (who never fully embraced its doctrines), and Steve’s hacker friend, Steve Wozniak, the laughing Yoda of the Hacker Way. The Hacker approach is primarily known to software coders, but can apply to all kinds of work. Even a few lawyers know about the hacker work ethic and have been influenced by it.

Who is Mark Zuckerberg?

We have all seen a movie version of Mark Zuckerberg in The Social Network, who, by the way, will still own 56.9% voting control of Facebook after the public offering later this year. But who is Mark Zuckerberg really? His Facebook page may reveal some of his personal life and ideas, but how did he create a Hundred Billion Dollar company so fast?

How did he change the world at such a young age? There are now 850 million people on Facebook with over 100 billion connections. On any one day there are over 500 million people using Facebook. These are astonishing numbers. How did this kind of creative innovation and success come about? What drove Mark and his hacker friends to labor so long, and so well? The letter to investors that Mark published  gives us a glimpse into the answer and a glimpse into the real Mark Zuckerberg. Do I have your full attention yet?

The Hacker Way philosophy described in the investor letter explains the methods used by Mark Zuckerberg’s and his team to change the world. Regardless of who Mark really is, greedy guy or saint (or like Steve Jobs, perhaps a strange combination of both), Mark’s stated philosophy is very interesting. It has applications to anyone who wants to change the world, including those of us trying to change the law and e-discovery.

Hacker Culture and Management

Mark’s letter to investors explains the unique culture and approach to management inherent in the Hacker Way that he and Facebook have adopted.

As part of building a strong company, we work hard at making Facebook the best place for great people to have a big impact on the world and learn from other great people. We have cultivated a unique culture and management approach that we call the Hacker Way.

The word `hacker’ has an unfairly negative connotation from being portrayed in the media as people who break into computers. In reality, hacking just means building something quickly or testing the boundaries of what can be done. Like most things, it can be used for good or bad, but the vast majority of hackers I’ve met tend to be idealistic people who want to have a positive impact on the world.

The Hacker Way is an approach to building that involves continuous improvement and iteration. Hackers believe that something can always be better, and that nothing is ever complete. They just have to go fix it — often in the face of people who say it’s impossible or are content with the status quo.

Hackers try to build the best services over the long term by quickly releasing and learning from smaller iterations rather than trying to get everything right all at once. To support this, we have built a testing framework that at any given time can try out thousands of versions of Facebook. We have the words `Done is better than perfect’ painted on our walls to remind ourselves to always keep shipping.

Hacking is also an inherently hands-on and active discipline. Instead of debating for days whether a new idea is possible or what the best way to build something is, hackers would rather just prototype something and see what works. There’s a hacker mantra that you’ll hear a lot around Facebook offices: `Code wins arguments.’

Hacker culture is also extremely open and meritocratic. Hackers believe that the best idea and implementation should always win — not the person who is best at lobbying for an idea or the person who manages the most people.

To encourage this approach, every few months we have a hackathon, where everyone builds prototypes for new ideas they have. At the end, the whole team gets together and looks at everything that has been built. Many of our most successful products came out of hackathons, including Timeline, chat, video, our mobile development framework and some of our most important infrastructure like the HipHop compiler.

To make sure all our engineers share this approach, we require all new engineers — even managers whose primary job will not be to write code — to go through a program called Bootcamp where they learn our codebase, our tools and our approach. There are a lot of folks in the industry who manage engineers and don’t want to code themselves, but the type of hands-on people we’re looking for are willing and able to go through Bootcamp.

So sayst Zuckerberg. Hands-on is the way.

Application of the Hacker Way to e-Discovery

E-discovery needs that same hands-on approach. E-discovery lawyers need to go through bootcamp too, even if they primarily just supervise others. Even senior partners should go, at least if they purport to manage and direct e-discovery work. Partners should, for example, know how to use the search and review software themselves, and from time to time, do it, not just direct junior partners, associates, and contact lawyers. You cannot manage others at a job unless you can actually do the job yourself. That is the hacker key to successful management.

Also, as I often say, to be a good e-discovery lawyer, you have to get your hands dirty in the digital mud. Look at the documents, don’t just theorize about them or what might be relevant. Bring it all down to earth. Test your keywords, don’t just negotiate them. Prove your search concept by the metrics of the search results. See what works. When it doesn’t, change the approach and try again.

Iteration is king of ESI search and production. Phased production is the only way to do e-discovery productions. There is no one final, perfect production of ESI. As Voltaire said, perfect is the enemy of  good. For e-discovery to work properly it must be hacked. It needs lawyer hackers. Are you up to the challenge?

Mark’s Explanation to Investors of the Hacker Way of Management

Mark goes on to explain in his letter to investors how the Hacker Way translates into the core values for Facebook management.

The examples above all relate to engineering, but we have distilled these principles into five core values for how we run Facebook:

Focus on Impact

If we want to have the biggest impact, the best way to do this is to make sure we always focus on solving the most important problems. It sounds simple, but we think most companies do this poorly and waste a lot of time. We expect everyone at Facebook to be good at finding the biggest problems to work on.

Move Fast

Moving fast enables us to build more things and learn faster. However, as most companies grow, they slow down too much because they’re more afraid of making mistakes than they are of losing opportunities by moving too slowly. We have a saying: “Move fast and break things.” The idea is that if you never break anything, you’re probably not moving fast enough.

Be Bold

Building great things means taking risks. This can be scary and prevents most companies from doing the bold things they should. However, in a world that’s changing so quickly, you’re guaranteed to fail if you don’t take any risks. We have another saying: “The riskiest thing is to take no risks.” We encourage everyone to make bold decisions, even if that means being wrong some of the time.

Be Open

We believe that a more open world is a better world because people with more information can make better decisions and have a greater impact. That goes for running our company as well. We work hard to make sure everyone at Facebook has access to as much information as possible about every part of the company so they can make the best decisions and have the greatest impact.

Build Social Value

Once again, Facebook exists to make the world more open and connected, and not just to build a company. We expect everyone at Facebook to focus every day on how to build real value for the world in everything they do.

________

Applying the Hacker Way of Management to e-Discovery

Focus on Impact

Law firms, corporate law departments, and vendors need to focus on solving the most important problems, the high costs of e-discovery and the lack of skills. The cost problem primarily arises from review expenses, so focus on that. The way to have the biggest impact here is to solve the needle in the haystack problem. Costs can be dramatically reduced by improving search. In that way we can focus and limit our review to the most important documents. This incorporates the search principles of Relevant Is Irrelevant and 7±2 that I addressed in Secrets of Search, Part III. My own work has been driven by this hacker focus on impact and led to my development of Bottom Line Driven Proportional Review. Other hacker oriented lawyers and technologists have developed their own methods to give clients the most bang for their buck.

The other big problem in e-discovery is that most lawyers do not know how to do it, and so they avoid it altogether. This in turn drives up the costs for everyone because it means the vendors cannot yet realize large economies of scale. Again, many lawyers and vendors understand that lack of education and skill sets is a key problem and are focusing on it.

Move Fast

This is an especially challenging dictate for lawyers and law firms because they are overly fearful of making mistakes, of breaking things as Facebook puts it. They are afraid of looking bad and malpractice suits. But the truth is, professional malpractice suits are very rare in litigation. Such suits happen much more often in other areas of the law, like estates and trusts, property, and tax. As far as looking bad goes, they should be more afraid of the bad publicity from not moving fast enough, which is a much more common problem, one that we see daily in sanctions cases. Society is changing fast, if you aren’t too, you’re falling behind.

The problem of slow adoptions also afflicts the bigger e-discovery vendors who often drown in bureaucracy and are afraid to make big decisions. That is why you see individuals like me starting an online education program, while the big boys keep on debating. I have already changed my e-Discovery Team Training program five times since it went public a little over a year ago. `Code wins arguments.’ Lawyers must be especially careful of the thinking Man’s disease, paralysis by analysis, if they want to remain competitive.

A few lawyers and e-discovery vendors understand this hacker maxim and do move fast. Many vendors appreciate the value of getting there first, but only a few law firms do. These rare law firms understand that the risks of lost opportunities are far more dangerous and certain than the risks of a making a few mistakes along the way. The slower, too conservative law firms are already starting to see their clients move business to the innovators, the few law firms who are moving fast. These firms have more than just puffed-up websites claiming e-discovery expertise, they have dedicated specialists and, in e-discovery at least, they are now far ahead of the rest of the crowd. Will the slow and timid ever catch up, or will they simply dissolve like Heller Ehrman, LLP?

Be Bold

This is all about taking risks and believing in your visions. It is directly related to moving fast and embracing change; not for its own sake, but to benefit your clients. Good lawyers are experts in risk analysis. There is no such thing as zero-risk, but there is certainly a point of diminishing returns for every litigation activity that is designed to control risks. Good lawyers know when enough is enough and constantly consult with their clients on cost benefit analysis. Should we take more depositions? Should we do another round of document checks for privilege? Often lawyers err on the side of caution, without consulting with their clients on the costs involved. They follow an overly cautious approach wherein the lawyers profit by more fees. Who are they really serving when they do that?

The adoption of predictive coding provides a perfect example of how some firms and vendors understand technology and are bold, and others do not and are timid. Many lawyers are afraid of predictive coding because there is no legal opinion to bless the new technology. But, as Judge Peck pointed out in his recent article, the law does not work that way, and you are waiting needlessly. Predictive Coding: Reading the Judicial Tea Leaves, (Law Tech. News, Oct. 17, 2011). No legal opinion blesses brute force manual review or keywords search. A few law firms and vendors understand the enormous value of predictive coding and the wisdom of Judge Peck’s advice. They understand the minimal risks involved, and consult with their clients about it in an informed and intelligent manner. These firms and vendors are embracing the new search methods and reaping the rewards. They are harnessing the ranking power of predictive coding for defensible culling. They manage their risks with cooperation, sampling, high skilled reviewers, and other best practices.

The legal profession is like any other industry, it rewards the bold, the innovators who create new legal methods and law for the benefit of their clients. What client wants a wimpy lawyer who is over-cautious and just runs up bills? They want a bold lawyer, who at the same time remains reasonable, and involves them in the key risk-reward decisions inherent in any e-discovery project.

Be Open

In the world of e-discovery this is all about transparency and strategic lowering of the wall of work product. Transparency is a proven method for building trust in discovery. Select disclosure is what cooperation looks like. It is what is supposed to happen at Rule 26(f) conferences, but seldom does. The attorneys that use openness as a tool are saving their clients needless expense and disputes. They are protecting them from dreaded redos, where a judge finds that you did a review wrong and requires you to do it again, usually under very short timelines. There are limits to openness of course, and lawyers have an inviolate duty to preserve their client’s secrets. But that still leaves room for disclosure of information on your own methods of search and review when doing so will serve your client’s interests.

Build Social Value 

The law is not a business. It is a profession. Lawyers and law firms exist to do justice. That is their social value. We should never lose sight of that in our day-to-day work. Vendors who serve the legal profession must also support these lofty goals in order to provide value. In e-discovery we should serve the prime directive, the dictates of Rule 1, for just, speedy, and inexpensive litigation. We should focus on legal services that provide that kind of social value. Profits to the firm should be secondary. As Zuckerberg said in the letter to potential investors:

Simply put: we don’t build services to make money; we make money to build better services.

This social value model is not naive, it works. It eventually creates huge financial rewards, as a number of e-discovery vendors and law firms are starting to realize. But that should never be the main point.

Conclusion

Facebook and Mark Zuckerberg should serve as an example to everyone, including e-discovery lawyers and vendors. I admit it is odd that we should have to turn to our youth for management guidance, but you cannot argue with success. We should study Zuckerberg’s 21st Century management style and Hacker Way philosophy. We can learn from its tremendous success. Zuckerberg and Facebook have proven that these management principles work in the digital age. It is true if it works. That is the pragmatic tradition of American philosophy. We live in fast changing times. Embrace change that works. As the face of Facebook says: “The riskiest thing is to take no risks.”

Postscript

Check out the DevBeat article Famous hackers discuss Zuckerberg’s “Hacker Way”. Apparently I’m not the only one who finds this story compelling. Some out-of-the-closet hackers do too, including the open-source crowd, who are critical of Facebook and Zuckerberg. This in spite of the fact that much of Facebook’s software is developed open sourced. They want more. They essentially criticize Mark as too entrepreneurial. They want non-profit purity. Personally I think they are full of it. Hello! This is America. This is a capitalistic culture. Yes, idealism is fine, but you also have to be real. The DevBeat editors ended on a positive note that I agree with:

We’re glad to see hackers portrayed by the young CEO as heroes rather than criminals.

Also, Facebook’s special strain of hacker culture is worth examining in its own right. The company is famous for prizing the iconoclastic work of small teams, for valuing the best solution regardless of provenance, for moving quickly, and for fearlessly building new systems and features from the ground up.

While Zuckerberg’s hacker ethic lacks the purity of its philosophical forebears, it marries some of the principles of hacking with the competitive advantage of a huge, global business. Whether the hacker ethic itself demands purity is another discussion for another day.


The Legal Implications of What Science Says About Recall

January 29, 2012

I hear a lot about how different software will find all relevant documents. That would be 100% recall. I also hear demands from requesting parties to find and produce all relevant documents. In the context of large disorganized banks of electronic data, such as email collections, these claims and demands are not only contra to the rules of law, embedded as they are in reasonability, but they are also unrealistic and contra to the latest scientific research. In my Bottom Line Driven Proportional Review article I showed how this kind of demand for all relevant ESI is not permitted under the rules and Doctrine of Proportionality in big data cases (and most cases these days are big data cases). I explained, as many have done before me, that the rules do not require production of all relevant documents, if the burden to do so is disproportional. I also shared my method for keeping the costs for review proportional to the value and importance of the case and the production request. But aside from the cost issue, how practical is it expect to find all relevant ESI? I examined this question at length in my Secrets of Search series, volumes one, two and three. Still, people find it hard to accept, especially in view of the unregulated clamor of the marketplace.

So as my last gift to readers before Legal Tech starts tomorrow, the ultimate event of marketplace claims and competing exaggerations,  I present you with a hard dose of reality, I present you with more findings on legal search from the world of science. This time I direct you to an important article, Evaluation of Information Retrieval for E-DiscoveryArtificial Intelligence and Law, 18(4)347-386 (2011). It was written by leaders of TREC Legal Track and established giants in the filed of legal search: Douglas W. Oard, Jason R. Baron, Bruce Hedin, David D. Lewis, and Stephen Tomlinson. They analyzed the now fully published test results of the experiments in 2008, and carefully examined the interactive task, topic 301, as the best test of competing legal search technologies. This task made use of a subject matter expert and an appeals process for quality control on relevance determinations. Four teams of experts participated in the test, two academic and two commercial. A well known e-discovery vendor won the test (scientists hate it when I put it that way). They won because they attained better precision and recall scores than the three other participants.

Now we come to the punch line, the winning vendor attained a recall rate of only 62%. That’s right, they missed 38% of the relevant documents. And they were the winner. Think about it. The other three participants in the scientific experiment attained recall rates of less than 20%! That’s right, they missed over 80% of the relevant documents. Now what do you think about a requesting party who demands that you produce all of the relevant email?

Find my summary of the experiments hard to believe, then read the report for yourself. Here is the excerpt on which I rely at page 24 of Evaluation of Information Retrieval for E-Discovery:

On the basis of the adjudicated sample assessments, we estimated that there are 786,862 documents (11.4% of the collection) relevant to Topic 103 in the test collection (as the topic was de fined by the TA). All four teams attained quite high precision; point estimates ranged from 0.71 to 0.81. One team (notably the one that made the most use of TA time) attained relatively high recall (0.62), while the other three (all making significantly less use of TA time) obtained recall values below 0.20.

The team of information scientists, and their lawyer guide, Jason R. Baron, next report on the 2009 TREC experiments, specifically the one they found most representative, the interactive tasks, again with subject matter consultations and appeals. This time there were eleven teams participating in the experiment, three academic and eight commercial. That’s right, eight e-discovery vendors were in the game this time. How did they do? They did a little better, but not much. Five of the teams, and just five only, got a little over 70% recall.

The post-adjudication results for the 2009 topics showed some encouraging signs. Of the 24 submitted runs (aggregating across all seven topics), 6 (distributed across 5 topics) attained an F1 score (point estimate) of 0.7 or greater. In terms of recall, of the 24 submitted runs, 5 (distributed across 4 topics) attained a recall score of 0.7 or greater; of these 5 runs, 4 (distributed across 3 topics) simultaneously attained a precision score of 0.7 or greater.

Id. at pgs. 24-25. If you follow the article’s direction and see the Overview of the TREC 2009 Legal Track, by B. Hedin, S. Tomlinson, J. Baron, and D. Oard, you can find more details of the 2009 test results. After you wade through the wonderfully dense language that information scientists love to use to convey information, you find section 2.3.5 Final Results. There you are pointed to a table of numbers: Table 6: Post-adjudication estimates of recall, precision, and F1.

What does this chart tell us? The best anyone did was an 86.5% recall on one of the seven tasks. Look at the third column from the left for the recall rates attained. The lowest was 9%. Digging deeper the analysts found that the teams with the highest scores appealed the most, and those with the lowest scores, not at all. Consultation with the topic authority also helped improve scores. But the bottom line for purposes of my point today, is that the average recall rate was only 41% (993/24), and even the best attained on one search, by one team of experts, was only 86%. Demands for recall in the 80s for every project are thus unrealistic.

Conclusion

The scientific research proves, once again, that it is unreasonable to ask for any better recall than 70%, in fact, it should be substantially less. Law demand reasonable efforts, not perfection. The best recall results attainable in scientific experiments, with the best software and top experts at the helm, is way too high a standard for reasonable efforts. Reasonability should be more like average results attained by average lawyers making good faith efforts, not results attained by information scientists and specialists using the best software money can buy. So that means it should be less than less than the 41% average of experts. Even standards like that should be used with caution and the efforts meter should always be tempered by costs. Proportionality of efforts should, if they are in good faith and reasonable, always trump any quality control efforts. See Bottom Line Driven Proportional Review.

In fairness to my vendor friends, the latest reports from TREC are dated. That was then, 2008 and 2009, this is now, 2012. The test scores showed substantial progress from 2008 to 2009. In my experience, the predictive coding type search software has significantly improved in the last year or so. I have also heard unsubstantiated reports of much higher recall rates attained in the 2011 TREC Legal Track tests, but I take all of these claims with a big grain of salt. Until Dr. Oard and his information scientist crew (that, by the way, includes two lawyers, Jason Baron and Maura Grossman) publish results, obtuse as their publications are, I will remain skeptical. Right now science shows that if you can find an estimated 41% of the relevant documents in a large collection of ESI, then you are doing just as good as the experts. That has got to be good enough to meet the reasonable efforts required under the law.

You should be skeptical of any claims or demands for better results than that. You should stop chasing, or being chased, by unreasonable demands for high recall rates. The only way to attain 70% or higher rates today is by document dumps, where precision plummets as you produce irrelevant documents, or, perhaps by budget busting, near-endless iterations of search and seed-set training. Even then, your expensive pursuit is quixotic from the point of view of science, where the fuzziness measurement issue remains unresolved. Furthermore, and most importantly, in today’s world of big data, where everyone has 100,000 emails, it is wasteful in the extreme to try to find all relevant documents. If your are still trying to find them all, and not just the few super-relevant smoking guns, you have not understood that in today’s age, relevant is irrelevant, nor that the ultimate goal of discovery is to prepare for trial, where the 7±2 rule of persuasion reigns supreme.


Reply to an Information Scientist’s Critique of My “Secrets of Search” Article

January 28, 2012

One of the leading information scientists in the field of legal search, was kind enough to write a detailed critique of my Secrets of Search series. I tried to post a response on his blog, Information Discovery where it appeared. But the website would not accept the comments for some technical reasons, so I am replying here, knowing that Herb will find them, and maybe some of his readers. I bring Dr. Roitblat’s comments to your attention, even though they are in the nature of a critique, because I want my readers to hear all sides of the story, not just mine. I am just a lawyer, and especially welcome peer review from the scientific community. That is part of a team approach to e-discovery, where Law, Science, and IT work together, and learn from each other, to do e-discovery right. The world is too complex, the electronic haystacks too vast, for lawyers to find relevant evidence without such an interdisciplinary team approach.

It is also interesting to see that a lot of Herb Roitblat’s stated disagreements appear to be based on misunderstandings of what I was trying to say. That is common in interdisciplinary team efforts. I accept that it was probably my fault, as I was writing about information science topics in Secrets of Search. But I don’t beat myself up too much about it because it is just so damned difficult to write intelligibly on the matrix between e-discovery law and information science. That is one reason that almost no one even tries. Still, this apparent miscommunication presents an opportunity. By addressing Herb’s issues we can attain greater clarity of the emerging consensus. His comments, and my responses, suggest that we both agree on far more than we disagree. From my perspective, at least, the points of disagreement are really minor and technical. They pale in comparison to our mutual agreement as to the superiority of technology assisted review over mere manual review.

Before I post the reply, I have to give you an idea of  Dr. Roitblat’s commentary, so you can understand what I’m replying to. But better still, take few minutes to read his entire article, On Some Selected Search Secrets. Only in this way will my response make full sense. I do not really know Herb, although I think we’ve met at some events over the years, but I certainly know of him. He is one of the few info scientists around that is focused on legal search and actually makes a living out of it. (Apparently he also likes dolphins and killer whales.) He owns a company, OrcaTec, that, in his words, provides professional services and software for information discovery and information management. My Secrets of Search article frequently cited one of his works with the e-Discovery Institute: Roitblat, Kershaw, and Oot, Document categorization in legal electronic discovery: computer classification vs. manual review; Journal of the American Society for Information Science and Technology, 61(1):70–80, 2010.

Summary of Herb Roitblat’s Critique

Here is Herb’s analysis, which begins in a very flattering manner that I don’t deserve:

Ralph Losey recently wrote an important series of blog posts (here, here, and here) describing five secrets of search. He pulled together a substantial array of facts and ideas that should have a powerful impact on eDiscovery and the use of technology in it. He raised so many good points, that it would take up all of my time just to enumerate them. He also highlighted the need for peer review. In that spirit I would like to address a few of his conclusions in the hope of furthering discussions among lawyers, judges, and information scientists about the best ways to pursue eDiscovery.

These are the problematic points I would like to consider:
1. Machines are not that good at categorizing documents. They are limited to about 65% precision and 65% recall.
2. Webber’s analysis shows that human review is better than machine review
3. Reviewer quality is paramount.
4. Human review is good for small volumes, but not large ones.
5. Random samples with 95% confidence levels +/- 2 are unrealistically high.

Ralph’s Response: Thanks for taking the time to provide input on my article. I appreciate your comments, and actually agree with most of the points you make. I think you may have misunderstood some of what I was saying and your disagreement is actually agreement, but I much appreciate your clarifications. I will respond based on your enumerated points above.

Dr. Roitblat then explains the five problems that he had with a few of the conclusions that I made in Secrets of Search. Again, I urge you to read all of his comments, but for ease of reference, I here quote what I think is the essence of each of his five issues, and then follow with my response.

Issue [1]: Machines are not that good at categorizing documents. They are limited to about 65% precision and 65% recall. Losey quotes extensively from a paper written by William Webber, which reanalyzes some results from the TREC Legal Track, 2009, and some other sources. Like Losey’s commentary, this paper also has a lot to recommend it. Some of the conclusions that Losey reaches are fairly attributable to Webber, but some go beyond what Webber would probably be comfortable with. The most significant fact, because important arguments are based on it, is a description of some work by Ellen Voorhees that concluded that 65% recall at 65% precision is the best performance one can expect. The problem is that this 65% factoid is taken out of context. In the context of the TREC studies and the way that documents are ultimately determined to be relevant or not, this is thought to be the best that can be achieved. The 65% is not a fact of nature. It says, actually, nothing about the accuracy of the predictive coding systems being studied. Losey notes that this limit is due to the inherent uncertainty in human judgments of relevance, but goes on to claim that this is a limit on machine-based or machine assisted categorization. It is not. …

Ralph’s Response [1]: I agree with you. I was not trying to say 65% precision or recall is all that is possible to attain, just that the fuzziness of our lenses makes it hard to prove anymore than that, unless special review controls are put in place for the measurements. These controls have been lacking in most legal tests to date. TREC is making progress with limited subject matter expert input, but even there, thanks to monetary constraints, we still have a ways to go to use a true gold standard that could improve our measurements. So I agree with you that 65% is no “fact of nature” as you put it, or inherent limitation in human relevancy determinations. (I am not ruling that possibility out entirely, but if such a mental limit like that does exist, my experience tells me that it is higher than 65%.) This fuzziness issue is more than a mere anomaly and deserves wide-spread discussion and recognition. In so far as large-scale human reviews are concerned, reviews unassisted by technologies, the kind of reviews that were common in the past, the 65% fuzzy focus may well be an inherent human limit. With predictive coding and other automated process, however, this barrier can be broken. Finally, I like your suggestion to improve TREC experiments by using both an authoritative training set and an authoritative judgment set.

Issue [2]: Webber’s analysis shows that human review is better than machine review. I have no doubt that human review could sometimes be better than machine-assisted review, but the data discussed by Webber do not say anything one way or the other about this claim. Webber did, in fact, find that some of the human reviewers showed higher precision and recall than did the best-performing computer system on some tasks. But, because of the methods used, we don’t know whether these differences were due merely to chance, to specific methods used to obtain the scores, or to genuine differences among reviewers. Moreover, the procedure prevents us from making a valid statistical comparison. …

Ralph’s Response [2]: Again, I agree with you. I get that Webber’s analysis suggests that humans are only sometimes better, not always. In fact, I would go much further and say that humans always lose over large-scale review (weeks on end of 8 hours a day reviewing hundreds of thousands of boring documents) when paired against today’s good software. Still, Webber pointed out what no one else had before about the TREC results, that the humans sometimes did win on the small-scale, even when substandard manual review methods were used. I think it is wrong to just sweep that under the rug as an anomaly or luck. This realization of human abilities is important for proper application of the predictive coding process, where, in my opinion, input by experts on the seed coding is key. These experts need a clear understanding of what is relevant, and what is not. Otherwise, no matter how good the software, the computer principle of garbage in, garbage out, will control.

This realization of the continued importance of Man in the technology equation is also important to defeat the sophistic arguments of some plaintiffs’ lawyers (or better put “requesting party lawyers). They are now arguing in multiple courts around the country that a defendant (responding party) should forego any manual review and just turn over documents based solely on automated review. They use that argument to oppose motions for protective orders based on excessive cost and burden to review. They are misusing distorted reports of scientific research to try to force quick peek disclosures. But the truth is, automated coding is not good enough yet to dispense with final manual quality control reviews to protect confidential information in a litigation context. Webber’s findings help prove that. The advantage to plaintiffs’ counsel of such disingenuous, forced quick peek strategy is obvious and substantial. Claw backs and Rule 502 are inadequate protections. Once the bell has been rung, the damage is done, regardless of whether the documents are returned. The main point I was trying to make by publicizing Webber’s finding is that humans still have a place at the table, not that they should sit there alone without reliance on the latest software for culling review. I suspect you agree with me on that.

Issue [3]: Reviewer quality is paramount. Webber found that some assessors performed better than others. Continuing the argument of the previous section, though, we cannot infer from this that some assessors were more talented, skilled, or better prepared than others. … The best reviewers on each topic could have been the best because they got lucky and got an easy bin, or they got a bin with a large number of responsive documents, or just by chance. Unless we disentangle these three possibilities, we cannot claim that some reviewers were better or that reviewer quality matters. In fact, these data provide no evidence one way or the other relative to these claims. … In some sense, the ideal would be for the senior attorney in the case to read every single document with no effect of fatigue, boredom, distraction, or error. Instead, the current practice is to farm out first pass review to either a team of anonymous, ad hoc, or inexpensive reviewers or to search by keyword. Even if Losey were right, the standard is to use the kind of reviewers that he says are not up to the task.

Ralph’s Response [3]: I think you misunderstood my point and again assumed incorrectly that I was advocating for large-scale manual review. I am not. I agree the reviewers are not up to the task, even the best. As explained above, I think humans cannot perform over long periods of time, and so I am not advocating against machine review, I am advocating for hybrid review, man and machine working together. Like you, I advocate for change. So really we agree.

But I do disagree with some of your statements here. To paraphrase Shakespeare: me thinks thou dost protest too much. I don’t think it is wrong to assume a correlation between accuracy and skill. That connection is based on experience and common sense. All large-scale review project metrics show that some reviewers are better than others, just like some trial lawyers are better than others, and some scientists, etc. It is inherent that we all perform to different levels at different tasks. I do not understand the need to try to explain all of the variances as just luck or chance. (As the great golfer Gary Player used to say: the more I practice, the luckier I get.)  Although I concede some chance or luck is possible, the same could be said of the software tested. Perhaps the “winning software” just got lucky. I would not seriously make that argument, so I am surprised to hear it made about the reviewers here. TREC only tried to measure comparisons, as you said, and lady luck knows no favorites.

Issue [4]: Human review is good for small volumes, but not large ones. This claim may also be true, but the present data do not provide any evidence for or against it. The evidence that Losey cites in support of this claim is the same evidence that, I argued, failed to show that human review is better than machine review. It requires the same circular reasoning. … Based on other evidence from psychology and other areas, it is likely that performance will decline somewhat with larger document sets, but there is no evidence here for that. If this were the only factor, we could arrange the situation so that reviewers only looked at 500 documents at a time before they took a break.

Ralph’s Response [4]: I agree this was not tested. I was again relying on my experience outside of these experiments and relying on my common sense built from over 30 years of doing document review, paper and electric, big and small. But I get your point of scientific discipline that it was not tested and so not here proven. Still, I’m not a scientist, nor do I care to become one. Also, I write primarily for lawyers, not scientists (although I am very happy a few of you are interested enough to read them too,  at least when it touches on your work). I am a lawyer interested in learning from science for purposes of improving law, not visa versa, although that may be a secondary benefit. That would depend on scientists like yourself. Also, as you know, there is more to establishing best practices in review processes than simply adding in periodic breaks.

Issue [5]: Random samples with 95% confidence levels +/- 2% confidence intervals are unrealistically high. It’s not entirely clear what this claim means. On the one hand, there is a common misperception of what it means to have a 95% confidence level. Some people mistakenly assume that the confidence level refers to the accuracy of the results. But the confidence level is not the same thing as the accuracy level. … I suspect that Losey means something different. I suspect that he is referring to the relatively weak levels of agreement found by the TREC studies and others. If our measurement is not very precise, then we can hardly expect that our estimates will be more precise.

Ralph’s Response [5]: You have correctly divined my intent here on sampling. I was again referring to the measurements fuzziness issue reported by your scientific colleagues, Voorhees, Webber and to some extent Oard. I understand that you are uncomfortable with their findings and conclusions on accuracy. I sincerely hope that you and other scientists will work this issue out.

I want accurate measurements too, especially when important points of justice are at stake. I want all of the scientific research out there for full public view, even the troubling preliminary conclusions of Voorhees, Webber and Oard.  If the measurements are disputed, I want full disclosure on that. If it takes more money, time and effort to get these measurements done properly in scientific testing, then lets raise the funds to do it right. I support the important scientific research now going on in legal search. On that point I suspect we once again agree.

Again, thanks for your comments on my article.

_________________

DEAR READERS: I’m off to LegalTech, where I will not only be presenting with Craig Ball and Judge Andrew Peck in the much hyped debate on Tuesday, January 31st at 4:00 at the Sutton Center on the 2nd floor (sponsored by BIA), but I will also be presenting three more times on predictive coding related subjects. I am thinking of preparing for all of that the way Pat Sajak prepared to host Wheel of Fortune.

On Monday the 30th, I present at 12:30 on The Promise and Challenge of Predictive Coding and Other Disruptive Technologies with Judge Andrew Peck, Maura Grossman and Dean Gonsowski (sponsored by Clearwell/Symantec).

On Wednesday, February 1st, I present at 10:30 on Technology Assisted Review: When to Use it and How to Defend It, with Maura Grossman, Judge Frank Maas, and Ann Marie Gibbs (sponsored by Daegis).

My last gig on Wednesday is at 1:45 in the Sutton South Parlor on E-discovery Circa 2015: Will Aggressive Preservation/Collection and Predictive Coding be Commonplace? My fellow panelists are David Kessler, Robert Trenchard, Julie Colgan, Stephanie Blair, and Craig Carpenter (sponsored by Recommind and ARMA).

If you see me around, please stop and say hello. I like to meet all of my readers whenever possible. Please forgive me if you catch me at a time during the day when I don’t have time to chat, but I always have time to shake hands and say hello.