Animation Showing How Not To Cooperate In An E-Discovery Conference

February 20, 2012

To my dear friends in the e-Discovery team community who prefer my videos and graphics to my many worrrrds, this blog’s for you.

___________

Don’t be a pseudo-cooperator. Make reasonable disclosures.

______

All characters appearing in this work are fictitious.

Any resemblance to real persons, living or dead, is purely coincidental.


Impactful, Fast, Bold, Open, Values: Guidance of the “Hacker Way”

February 12, 2012

Facebook’s success suggests that for organizations to prosper in a rapidly changing technological society, in other words, in today’s world, they must: focus on impact, be fast, be bold, be open, and build social value. To attain these goals they must adopt iterative processes for continuous improvements. They must promote a hands-on meritocratic culture where an idea is adopted if it works, no matter who suggested it. This is the core meaning of Facebook’s motto – Code wins arguments. It is a very practical and egalitarian approach to management.

Last week in my blog “The Hacker Way” – What e-Discovery Can Learn From Facebook’s Culture and Management I brought Mark Zuckerberg’s Letter to Investors to your attention. The letter disclosed Facebook’s core values, which Zuckerberg calls the Hacker Way. My article also began an exploration of how this new philosophy of work can be applied in our world of e-discovery. This week I’ll take the application theme a little further. I am convinced that all organizations in the e-discovery world – law firms, vendors, corporations, governments and others – can benefit from applying Facebook’s three-fold code wins methods, and five-fold Hacker Way goals.

It doesn’t take a Zuckerberg to realize that these methods and goals could be applied to any 21st Century field of endeavor. Are you involved in some way in management of your law firm or company? Do you want to stay competitive and remain relevant? If you stand still and keep the old management style and values, you will quickly fall far behind. The business and professional motivations to study Facebook’s success are obvious. I hope you will think about it, leave comments below, and help me figure out how we can all benefit from Zuckerberg’s insights.

Personal Apps?

There may even be applications of the Hacker Way to your personal life and code of conduct. Do you want your life to have an impact on the world? To have meaning? Do you want your life to embody ethical values that you are proud of? Do you share, or are you closed off, an iconoclastic island? Are you open to change, no matter how fast it may come? Can you take action, make decisions? Can you quickly implement your decisions? Are you efficient and productive? Do you dare to be bold, or just follow timid conventions of old?

I certainly will not try to answer these questions here, nor share the kind of personal fractal solutions that I try to fashion on a day-to-day basis; the iterations of action, observations, analysis, adjusted actions, repeat.  I just raise the questions to stimulate your personal thinking. Everyone has their own solution to the enigma that is life. There is no one right answer. We each have our own path on the Hacker Way. This essay will instead focus on the practical applications of Hacker Way to e-discovery management.

Impactful

In e-discovery, like anything else, you have to focus and prioritize. You cannot do everything at once, at least not if you are going for impact, or for quality for that matter. Scatter-brained is a recipe for failure. So is not knowing what to focus on first. You have to pick and choose your battles, your time, energy, and money. All tasks are not created equal. Some are more important than others. Generally you should focus on them first, and then move on to secondary problems. A wise project manager knows what to focus on and when.

The big picture, from the highest elevation, shows that the core problems in e-discovery today are the high costs of e-discovery and the low skills of practitioners. You could say high costs and high risks, but I really think that the risk management problem is secondary to the skills problem. Attorneys trained in e-discovery have the skills to avoid the colossal mistakes that we read about in the court opinions every day. See eg. Pippins v. KPMG LLP, 2012 WL 370321 (S.D.N.Y. Feb. 3, 2012).

To maximize impact e-discovery teams everywhere should focus on these two, costs and training. They should look for bold new ways to control costs and train attorneys. As discussed briefly in “The Hacker Way” – What e-Discovery Can Learn From Facebook’s Culture and Management, for me cost control means focusing on search, since review is the most expensive part of any production project. It also means building new aggressive culling methods, such as Bottom Line Driven Proportional Review.

This is also an area for vendors to be bold. Vendors must lead the way by making advanced predictive coding type software far less expensive than it is today. Making a thousand dollars each on ten-thousand users makes you ten times as much money as making ten-thousand dollars on a hundred users. Come on software companies, you know who you are, bring down your prices! Don’t make me name names.

The high price of advanced search software is a core problem. When it is fixed it will have a big impact on e-discovery. Who will be the new Facebook of e-discovery search? Fame and fortune await the bold company who can now move fast. The same old timid low-volume, high-profit approach must be abandoned. Try a new value approach in your next iteration. You need to do so fast before the first-to-market competitor eats your lunch.

In addition to solving the needle-in-the-haystack problem, another good way to cut costs and maximize the impact of an e-discovery team is to better prepare for e-discovery. You can have a high impact on both costs and skills by engaging in what I call an e-discovery readiness program. It typically involves what the old EDRM referred to as Information Management. But e-discovery readiness actually involves far more than that, including massive amounts of education and training, not only for lawyers, outside and in-house, but also for corporate IT and management stake-holders. It involves forming a functioning corporate e-discovery team, updating electronic retention policies, especially email. It also requires design of proportional litigation hold procedures, and efficient ESI storage architecture and software.

This kind of pre-litigation readiness program should be adopted by all corporations, governments, and other large organizations who must routinely litigate and respond to subpoenas as a cost of doing business. It can result in tremendous savings, just like a more efficient culling and search method.

Fast

The 2006 Amendments to the Federal Rules of Civil Procedure put a premium on speed. You have to work fast to get your act together before the initial 26(f) conference. Under the new rules you are required to discuss and disclose your e-discovery plan, including your preferred format of production, preservation efforts, and plan for search and review. Evolving case law on litigation holds also requires you to act fast, to send out written notices quickly, to collect key ESI quickly.

Obviously most search, review and production projects also require you to act fast to meet tight time deadlines. How many files an hour can your CAR go, your Computer Assisted Review? You have got to be fast to succeed in e-discovery. Yet, at the same time, you must act reasonably and minimize mistakes. This means that you must have quality control methods built into your CARs. You have to know when to double-check your efforts. There are straightaways in e-discovery where you can go fast, such as irrelevancy culling, and there are curves, such as privilege review. You have to know when and where to slow down so you don’t go flying off the curves.

Some analyses and conversations need to happen quickly, right at the beginning of a case. Determination of key players and key player interviews and preservation instructions come to mind. So too do conversations with IT to suspend automatic deletions and old hard-drive recirculation. So many things in e-discovery are front-loaded. You have to act fast at the beginning of a case. This is a big change in culture where for decades Rule 26(f) meet and greets have been mere drive-bys, where the initial discovery analysis could be postponed. Now it cannot. Now you must act fast and early.

Fast is also important in law firm management. Firms need to quickly get their e-discovery departments up to speed. They need to move fast to ramp up their attorneys’ skills. Gradualism is a recipe of failure. You cannot just wait until the next generation figures it out. Yes, it may be easier for the twenty-somethings to learn this new area of law, but do you really want them to do that by trial and error on your clients’ cases? Do your clients want that? Anyway, they won’t learn the right way to do things. They will stagger along without the benefit of senior advisers.

For a law firm to move fast, they really need to do what my law firm did, and others have done. They need to hire an outside attorney who is already skilled. Then the senior experts can train and help others get up to speed, especially the young rock-star e-disco associates who otherwise practice e-discovery without adult supervision.

Vendors need to move fast too. Indeed, the pressure on vendors is even more intense because the competition is hotter. Speed and early adoption are all important in the world of e-discovery vendors. They cannot slow-poke around with yesterday’s technologies, such as keyword-only based search, or expensive on-site licenses. They need racing CARs. They need to move, and move quickly, to affordable SaaS based delivery systems, and to  bring those prices down, if they are going to win the race. They need advanced search with artificial intelligence (“AI”) type agents that the user trains, i.w. predictive coding type software. SIRI is here to stay. Take a look at Nokia’s and Research in Motion’s stock values as compared to Apple. Which team to do you want to be on?

By the way, some have questioned my use of the term AI in connection with the new search algorithms. Please note the generally accepted definition of AI is “the study and design of intelligent agents”[1] where an intelligent agent is a system that perceives its environment and takes actions that maximize its chances of success. This is what I mean by AI. SIRI is example of this. Admittedly, we are still in the early design stages of AI agents for e-discovery (and phones). I know that our programs today will look pretty stupid in a few years. (So will everything else as the pace of change and improvements continue to accelerate.) But the first vendors out there with smart AI agents will gain a huge competitive advantage. Those stuck in hyped-up keyword searches, which is just 1960s technology with better interface and faster execution, will go the way of the buggy whip manufactures. You know who you are. Don’t say I didn’t warn you. You had better go out there and hire a team of information scientists now if you want to keep up.

Bold

Timid and lawyer are two words that were never meant to go together. Yet for most AmLaw 100 law firms today, they do, at least when it comes to e-discovery. For a law firm to be bold, they need to do what my law firm did, and others have done. They need to hire outside attorneys who are already skilled, and they need to make a full commitment to these attorneys and what they bring to the table. That means hiring them as a Partner, like I was, or my friend Paul Weiner was, not as an Of Counsel or Special Counsel, or even worse, as non-partner track Staff Attorney. But there is more to this than just titles. The newly hired e-discovery experts should be provided with authority to make a real e-discovery team, not just design a marketing ploy. National e-Discovery Counsel, like Paul and I, need to have the clout of Partner status, and top management back-up, to be taken seriously and implement necessary changes.

The attorneys who specialize in e-discovery should dedicate their whole practice to e-discovery, and e-discovery alone. Be bold and go all the way. Unless you are full time, like Paul and me, and Paul’s partners Cecil Lynn and Michael McGuire, to name just a few, you really will not have the time and experience to master this complex subject. Would you want your heart surgeon to be a part-timer who also does podiatry? Would you want your anti-trust lawyer to also be a family law practitioner? Law, like medicine, is complex and e-discovery is one of the most complex subjects in the law today. Law firms had better move quickly to hire or facilitate full time practitioners. It does not work to just have a few lawyers in a firm that dabble part time in this new area of the law. Mistakes will inevitably follow as the dangerous little-bit-of-knowledge syndrome sets in. As Alexander Pope said in 1709:

A little learning is a dangerous thing;
drink deep, or taste not the Pierian spring:
there shallow draughts intoxicate the brain,
and drinking largely sobers us again.

Halfway, band-aid measures won’t cut it in e-discovery. You have got to go either all-in or all-out. The days of a law firm setting up a marketing type e-discovery department by sending out a few of its attorneys to CLEs, and then posturing them as experts, are long gone. It takes bold all-out efforts. You need to hire full-timers and give them authority. Mere marketing with part-timers can be set up fast, but that is not bold. That is same-old, same-old. That is timid.

Longtime e-discovery commentator and consultant George Socha is also leery of poser type law firm practice groups. In the recent LTN article, True Grit: Four Models to Rein in E-Discovery Costs, that featured Paul Weiner on its cover (who is no poser), George is quoted as saying that most law firms:

[C]ontinue to be marketing groups more than anything else. I continue to see that most lawyers at firms with putative internal EDD practice groups either do not know those groups exist or do not use them. Firms ought to do a better job of taking control of EDD, at least for those clients who lack the wherewithal to take on EDD themselves. Firms seem unwilling, however, to make the initial and on-going investments needed for that to happen.

How many of the AmLaw 100 law firm’s have bona fide e-discovery practice groups with at least one partner who does electronic discovery full time? That is the true litmus test for bold management, a test which most firm’s fail. It bears repeating: timid and lawyer are two words that were never meant to go together. Be bold law firm managers. Be a mighty mouse, not timid rodent. Hire a full-time expert e-discovery lawyer as a partner in your firm today and give him or her the juice they need to get things done.

As to vendors, be bold and be willing to take a chance, to go with the changes sweeping the industry. Stop trying to milk your outdated products for all they are worth from uneducated consumers who don’t know any better. Get rid of your old products instead of adding a few minor enhancements each year. Shorten your new product cycles. Invest in research. Buy or be bought already. Look to a company like Clearwell for a Hacker Way to do things right.

Above all dear vendors, be bold and lower your prices. I know that vendors are not like lawyers and law firms who have a fiduciary duty to their clients. Vendors instead have a duty to their shareholders to maximize profits. I get that. But you can still do that and lower prices, because this will increase the number of sales. It is a virtuous cycle. The more affordable e-discovery becomes, the more lawyers will do e-discovery. Lower prices will increase sales, and thereby increase profits too. Be bold. Be the first to step up to the plate and make dramatic price reductions, not just a little bit here and there.

Open

Vendors need to be more open about their profits and their black boxes. Got some special mojo? Then patent it like the rest of the technology industry does, and protect yourself that way, instead of by using trade-secrets. Software vendors, open up your black box! Show us the secret sauce and you will go a long way to building trust.

Openness builds trust. That is why transparency is now a key component to the new paradigm of discovery cooperation. The way to walk your talk in cooperation is by disclosure. How did you go about preservation? How did you go about finding the documents requested? Use the tools of sampling. Make aggressive disclosures so that you can support your proportionality arguments. As Judge McMahon said recently in Pippins v. KPMG LLP, 2012 WL 370321 (S.D.N.Y. Feb. 3, 2012):

But I cannot conclude that the cost of preserving the hard drives outweighs its benefit, as KPMG urges, any more than Judge Cott could, because the record before me is devoid of information necessary to conduct such an analysis. See Order at *8. KPMG refused to allow Judge Cott, or Plaintiffs examine even a single hard drive to ascertain the “benefit” of preservation, so there is nothing in the record before me to inform any decision. Even assuming that KPMG’s preservation costs are both accurate and wholly attributable to this litigation which I cannot verify – I cannot possibly balance the costs and benefits of preservations when I’m missing one side of the scale (the benefits). Neither can I conclude that KPMG has kept hard copy data of everything on the drives that might prove relevant to this action. KPMG cannot simultaneously demand that the Court analyze how long every Audit Associate worked and what every Audit Associate did and also ask the Court to sanction the destruction of what is probably the single best source of that information.

In short, KPMG is hoist on its own petard.

The petard here is KPMG’s refusal, no doubt upon advice of counsel, to make disclosure, to be open.

As Judge McMahon went on to explain:

I certainly do not intend to reverse Judge Cott’s Order on the purported ground that he erred by concluding that KPMG failed to demonstrate that preserving the hard drives was unreasonable. Frankly, the only things that were unreasonable were: (1) KPMG’s refusal to turn over so much as a single hard drive so its contents could be examined; and (2) its refusal to do what was necessary in order to engage in good faith negotiations over the scope of preservation with Plaintiffs’ counsel, in purported reliance on an order of this Court that it interpreted unreasonably. It smacks of chutzpah (no definition required) to argue that the Magistrate failed to balance the costs and benefits of preservation when KPMG refused to cooperate with that analysis by providing the very item that would, if examined, demonstrate whether there was any benefit at all to preservation.

KPMG’s attorneys here were bold in their litigation conduct, but not open. As a result their actions were seen as chutzpah. Judge McMahon earlier detailed some of the failures to be open that caused him to affirm the Magistrate’s earlier controversial ruling:

KPMG, hiding behind the stay of discovery, insisted it could not produce even one hard drive for inspection by Plaintiffs. It also refused to respond to any question regarding the content of the hard drives, furnish Plaintiffs’ access to any hard drives, inform Plaintiffs whether the data on the hard drives might be derived from other sources, or discuss the costs of possible alternatives to preserving the data on the hard drives. Instead, KPMG made a series of “take it or leave it” offers, and sought to have Plaintiffs agree that KPMG only had to preserve a smaller sample of the hard drives without giving Plaintiffs the opportunity to review the contents of any hard drive(s) first. (Citations to the record omitted.)

Old school, hide-the-ball, “take it or leave it” approaches no longer work in e-discovery. When are the old-timey, non-specialists trial lawyers going to get it? When are they going to move over and just let the full-time experts do e-discovery without interference? There will still be plenty for the trial lawyers to do, plenty to argue about. Above all, when are clients finally going to catch on and insist upon an open, expert approach?

Values

Building social value is no longer an idealistic pie-in-the-sky dream. It is an effective business strategy. It is an especially effective strategy for professions such as medicine and law, occupations that by nature have a higher calling than just making money. Lawyers serve their clients and the law. It is a dual task and if there is conflict between the two, the law must always prevail. That is why we have extensive an code of professional conduct.

The ethical codes of the legal profession embody our values. They provide a solid guide to the conduct of our services, our noble quest for truth, justice and liberty. The ethical codes require all lawyers to be competent, and, if faced with a legal task wherein they are not competent, such as e-discovery, to bring in other attorneys who are. Model Rule 1.1 (“A lawyer shall provide competent representation to a client. Competent representation requires the legal knowledge, skill, thoroughness, and preparation reasonably necessary for the representation.”)

Our ethics also requires diligence, a task that is impossible unless you actually know what to do and when to do it. Model Rule 1.3 (“A lawyer shall act with reasonable diligence and promptness in representing a client.”) The Hacker Way value of fast action is thus built into our code. It is emphasized again by Model Rule 3.2 that requires lawyers to expedite litigation. (“A lawyer shall make reasonable efforts to expedite litigation consistent with the interests of the client.”)

Our values also require candor towards the tribunal, the judges. Candor means openness and complete honesty. It is a core value that may never be broken under any circumstances. Should it violate your duty of loyalty to your client, you are required to withdraw from representation, rather than ever be dishonest and closed to the presiding judge. Model Rule 3.3 (“(a) A lawyer shall not knowingly: (1) make a false statement of fact or law to a tribunal or fail to correct a false statement of material fact or law previously made to the tribunal by the lawyer; . . . (3) offer evidence that the lawyer knows to be false. . . .”)

Our values as lawyers also requires fairness to the opposing party in litigation and fairness to the opposing counsel. This means, among other things, that games of hide-the-ball are forbidden. This does not mean that you should provide evidence harmful to your client that was not requested, or not relevant, or that you are not legally required to produce such as privileged information. But if it was requested, is relevant, and you are legally required to produce it, it is unethical not to do so. If the client refuses to do so, you should withdraw. Model Rule 4-3.4 (“A lawyer shall not: (a) unlawfully obstruct another party’s access to evidence or otherwise unlawfully alter, destroy, or conceal a document or other material that the lawyer knows or reasonably should know is relevant to a pending or a reasonably foreseeable proceeding; nor counsel or assist another person to do any such act.”)

In the United States, like the rest of the civilized world, citizens have a fundamental right to justice, to due process. If it costs too much to discover the truth, they are deprived of that right. If only the rich can afford to find the truth needed to do justice, then the values we have worked so hard to develop in this country and the world over the last several centuries will be destroyed. We cannot let that happen.

Lawyers and judges must take the proportionality principle very seriously. We must innovate and constantly improve our methods to control the costs of e-discovery, to make it affordable to all. Vendors have a key role to play in this process of cost containment. They can and must work side-by-side with law firms to keep our litigation just, speedy and inexpensive. That is how we can all work together to build social value in e-discovery.

Conclusion

We can all learn from the Hacker Way that has been implemented so successfully by Facebook. It’s five goals and three methods provide good advice to all, even to senior lawyers wrapped up in e-discovery who may think they already know it all. There is always room for improvement, for growth. Those of us who are hands-on everyday in e-discovery know this to be true.

The legal profession does not usually take advice from 27 year-olds, but that is part of a meritocracy. If a young associate is right on a point of law, they are right. Their age does not matter. The law has always striven to be a meritocracy, so this is really nothing new. We should take advice from wherever it comes, if it is good advice, if it has been proven to be sound. The success of Facebook is proof enough for me. Code wins arguments, so too do stock prices. Let’s watch the Facebook IPO and see what happens, and beyond, as it continues to change every aspect of the lives of millions of people around the world. The Hacker Way is worthy of further study. I look forward to your comments below.


“The Hacker Way” – What e-Discovery Can Learn From Facebook’s Culture and Management

February 5, 2012

Facebook’s regulatory filing for its initial public stock offering includes a letter to potential investors by 27 year old billionaire Mark Zuckerberg. The letter describes the culture and approach to management that he follows as CEO of Facebook. Zuckerberg calls it the Hacker Way. Mark did not invent this culture. In a way, it invented him. It molded him and made him and Facebook what they are today. This letter reveals the secrets of Mark’s success and establishes him as the current child prodigy of the Hacker Way.

Hacker History

This tradition and way of thinking and doing have been around since at least the sixties. It has little or nothing to do with illegal computer intrusions. It is often called the hacker ethic and, surprisingly, arose out of the hobby of model railroad building and MIT computer labs. This philosophy is well-known and has influenced many in the tech world, including the great Steve Jobs (who never fully embraced its doctrines), and Steve’s hacker friend, Steve Wozniak, the laughing Yoda of the Hacker Way. The Hacker approach is primarily known to software coders, but can apply to all kinds of work. Even a few lawyers know about the hacker work ethic and have been influenced by it.

Who is Mark Zuckerberg?

We have all seen a movie version of Mark Zuckerberg in The Social Network, who, by the way, will still own 56.9% voting control of Facebook after the public offering later this year. But who is Mark Zuckerberg really? His Facebook page may reveal some of his personal life and ideas, but how did he create a Hundred Billion Dollar company so fast?

How did he change the world at such a young age? There are now 850 million people on Facebook with over 100 billion connections. On any one day there are over 500 million people using Facebook. These are astonishing numbers. How did this kind of creative innovation and success come about? What drove Mark and his hacker friends to labor so long, and so well? The letter to investors that Mark published  gives us a glimpse into the answer and a glimpse into the real Mark Zuckerberg. Do I have your full attention yet?

The Hacker Way philosophy described in the investor letter explains the methods used by Mark Zuckerberg’s and his team to change the world. Regardless of who Mark really is, greedy guy or saint (or like Steve Jobs, perhaps a strange combination of both), Mark’s stated philosophy is very interesting. It has applications to anyone who wants to change the world, including those of us trying to change the law and e-discovery.

Hacker Culture and Management

Mark’s letter to investors explains the unique culture and approach to management inherent in the Hacker Way that he and Facebook have adopted.

As part of building a strong company, we work hard at making Facebook the best place for great people to have a big impact on the world and learn from other great people. We have cultivated a unique culture and management approach that we call the Hacker Way.

The word `hacker’ has an unfairly negative connotation from being portrayed in the media as people who break into computers. In reality, hacking just means building something quickly or testing the boundaries of what can be done. Like most things, it can be used for good or bad, but the vast majority of hackers I’ve met tend to be idealistic people who want to have a positive impact on the world.

The Hacker Way is an approach to building that involves continuous improvement and iteration. Hackers believe that something can always be better, and that nothing is ever complete. They just have to go fix it — often in the face of people who say it’s impossible or are content with the status quo.

Hackers try to build the best services over the long term by quickly releasing and learning from smaller iterations rather than trying to get everything right all at once. To support this, we have built a testing framework that at any given time can try out thousands of versions of Facebook. We have the words `Done is better than perfect’ painted on our walls to remind ourselves to always keep shipping.

Hacking is also an inherently hands-on and active discipline. Instead of debating for days whether a new idea is possible or what the best way to build something is, hackers would rather just prototype something and see what works. There’s a hacker mantra that you’ll hear a lot around Facebook offices: `Code wins arguments.’

Hacker culture is also extremely open and meritocratic. Hackers believe that the best idea and implementation should always win — not the person who is best at lobbying for an idea or the person who manages the most people.

To encourage this approach, every few months we have a hackathon, where everyone builds prototypes for new ideas they have. At the end, the whole team gets together and looks at everything that has been built. Many of our most successful products came out of hackathons, including Timeline, chat, video, our mobile development framework and some of our most important infrastructure like the HipHop compiler.

To make sure all our engineers share this approach, we require all new engineers — even managers whose primary job will not be to write code — to go through a program called Bootcamp where they learn our codebase, our tools and our approach. There are a lot of folks in the industry who manage engineers and don’t want to code themselves, but the type of hands-on people we’re looking for are willing and able to go through Bootcamp.

So sayst Zuckerberg. Hands-on is the way.

Application of the Hacker Way to e-Discovery

E-discovery needs that same hands-on approach. E-discovery lawyers need to go through bootcamp too, even if they primarily just supervise others. Even senior partners should go, at least if they purport to manage and direct e-discovery work. Partners should, for example, know how to use the search and review software themselves, and from time to time, do it, not just direct junior partners, associates, and contact lawyers. You cannot manage others at a job unless you can actually do the job yourself. That is the hacker key to successful management.

Also, as I often say, to be a good e-discovery lawyer, you have to get your hands dirty in the digital mud. Look at the documents, don’t just theorize about them or what might be relevant. Bring it all down to earth. Test your keywords, don’t just negotiate them. Prove your search concept by the metrics of the search results. See what works. When it doesn’t, change the approach and try again.

Iteration is king of ESI search and production. Phased production is the only way to do e-discovery productions. There is no one final, perfect production of ESI. As Voltaire said, perfect is the enemy of  good. For e-discovery to work properly it must be hacked. It needs lawyer hackers. Are you up to the challenge?

Mark’s Explanation to Investors of the Hacker Way of Management

Mark goes on to explain in his letter to investors how the Hacker Way translates into the core values for Facebook management.

The examples above all relate to engineering, but we have distilled these principles into five core values for how we run Facebook:

Focus on Impact

If we want to have the biggest impact, the best way to do this is to make sure we always focus on solving the most important problems. It sounds simple, but we think most companies do this poorly and waste a lot of time. We expect everyone at Facebook to be good at finding the biggest problems to work on.

Move Fast

Moving fast enables us to build more things and learn faster. However, as most companies grow, they slow down too much because they’re more afraid of making mistakes than they are of losing opportunities by moving too slowly. We have a saying: “Move fast and break things.” The idea is that if you never break anything, you’re probably not moving fast enough.

Be Bold

Building great things means taking risks. This can be scary and prevents most companies from doing the bold things they should. However, in a world that’s changing so quickly, you’re guaranteed to fail if you don’t take any risks. We have another saying: “The riskiest thing is to take no risks.” We encourage everyone to make bold decisions, even if that means being wrong some of the time.

Be Open

We believe that a more open world is a better world because people with more information can make better decisions and have a greater impact. That goes for running our company as well. We work hard to make sure everyone at Facebook has access to as much information as possible about every part of the company so they can make the best decisions and have the greatest impact.

Build Social Value

Once again, Facebook exists to make the world more open and connected, and not just to build a company. We expect everyone at Facebook to focus every day on how to build real value for the world in everything they do.

________

Applying the Hacker Way of Management to e-Discovery

Focus on Impact

Law firms, corporate law departments, and vendors need to focus on solving the most important problems, the high costs of e-discovery and the lack of skills. The cost problem primarily arises from review expenses, so focus on that. The way to have the biggest impact here is to solve the needle in the haystack problem. Costs can be dramatically reduced by improving search. In that way we can focus and limit our review to the most important documents. This incorporates the search principles of Relevant Is Irrelevant and 7±2 that I addressed in Secrets of Search, Part III. My own work has been driven by this hacker focus on impact and led to my development of Bottom Line Driven Proportional Review. Other hacker oriented lawyers and technologists have developed their own methods to give clients the most bang for their buck.

The other big problem in e-discovery is that most lawyers do not know how to do it, and so they avoid it altogether. This in turn drives up the costs for everyone because it means the vendors cannot yet realize large economies of scale. Again, many lawyers and vendors understand that lack of education and skill sets is a key problem and are focusing on it.

Move Fast

This is an especially challenging dictate for lawyers and law firms because they are overly fearful of making mistakes, of breaking things as Facebook puts it. They are afraid of looking bad and malpractice suits. But the truth is, professional malpractice suits are very rare in litigation. Such suits happen much more often in other areas of the law, like estates and trusts, property, and tax. As far as looking bad goes, they should be more afraid of the bad publicity from not moving fast enough, which is a much more common problem, one that we see daily in sanctions cases. Society is changing fast, if you aren’t too, you’re falling behind.

The problem of slow adoptions also afflicts the bigger e-discovery vendors who often drown in bureaucracy and are afraid to make big decisions. That is why you see individuals like me starting an online education program, while the big boys keep on debating. I have already changed my e-Discovery Team Training program five times since it went public a little over a year ago. `Code wins arguments.’ Lawyers must be especially careful of the thinking Man’s disease, paralysis by analysis, if they want to remain competitive.

A few lawyers and e-discovery vendors understand this hacker maxim and do move fast. Many vendors appreciate the value of getting there first, but only a few law firms do. These rare law firms understand that the risks of lost opportunities are far more dangerous and certain than the risks of a making a few mistakes along the way. The slower, too conservative law firms are already starting to see their clients move business to the innovators, the few law firms who are moving fast. These firms have more than just puffed-up websites claiming e-discovery expertise, they have dedicated specialists and, in e-discovery at least, they are now far ahead of the rest of the crowd. Will the slow and timid ever catch up, or will they simply dissolve like Heller Ehrman, LLP?

Be Bold

This is all about taking risks and believing in your visions. It is directly related to moving fast and embracing change; not for its own sake, but to benefit your clients. Good lawyers are experts in risk analysis. There is no such thing as zero-risk, but there is certainly a point of diminishing returns for every litigation activity that is designed to control risks. Good lawyers know when enough is enough and constantly consult with their clients on cost benefit analysis. Should we take more depositions? Should we do another round of document checks for privilege? Often lawyers err on the side of caution, without consulting with their clients on the costs involved. They follow an overly cautious approach wherein the lawyers profit by more fees. Who are they really serving when they do that?

The adoption of predictive coding provides a perfect example of how some firms and vendors understand technology and are bold, and others do not and are timid. Many lawyers are afraid of predictive coding because there is no legal opinion to bless the new technology. But, as Judge Peck pointed out in his recent article, the law does not work that way, and you are waiting needlessly. Predictive Coding: Reading the Judicial Tea Leaves, (Law Tech. News, Oct. 17, 2011). No legal opinion blesses brute force manual review or keywords search. A few law firms and vendors understand the enormous value of predictive coding and the wisdom of Judge Peck’s advice. They understand the minimal risks involved, and consult with their clients about it in an informed and intelligent manner. These firms and vendors are embracing the new search methods and reaping the rewards. They are harnessing the ranking power of predictive coding for defensible culling. They manage their risks with cooperation, sampling, high skilled reviewers, and other best practices.

The legal profession is like any other industry, it rewards the bold, the innovators who create new legal methods and law for the benefit of their clients. What client wants a wimpy lawyer who is over-cautious and just runs up bills? They want a bold lawyer, who at the same time remains reasonable, and involves them in the key risk-reward decisions inherent in any e-discovery project.

Be Open

In the world of e-discovery this is all about transparency and strategic lowering of the wall of work product. Transparency is a proven method for building trust in discovery. Select disclosure is what cooperation looks like. It is what is supposed to happen at Rule 26(f) conferences, but seldom does. The attorneys that use openness as a tool are saving their clients needless expense and disputes. They are protecting them from dreaded redos, where a judge finds that you did a review wrong and requires you to do it again, usually under very short timelines. There are limits to openness of course, and lawyers have an inviolate duty to preserve their client’s secrets. But that still leaves room for disclosure of information on your own methods of search and review when doing so will serve your client’s interests.

Build Social Value 

The law is not a business. It is a profession. Lawyers and law firms exist to do justice. That is their social value. We should never lose sight of that in our day-to-day work. Vendors who serve the legal profession must also support these lofty goals in order to provide value. In e-discovery we should serve the prime directive, the dictates of Rule 1, for just, speedy, and inexpensive litigation. We should focus on legal services that provide that kind of social value. Profits to the firm should be secondary. As Zuckerberg said in the letter to potential investors:

Simply put: we don’t build services to make money; we make money to build better services.

This social value model is not naive, it works. It eventually creates huge financial rewards, as a number of e-discovery vendors and law firms are starting to realize. But that should never be the main point.

Conclusion

Facebook and Mark Zuckerberg should serve as an example to everyone, including e-discovery lawyers and vendors. I admit it is odd that we should have to turn to our youth for management guidance, but you cannot argue with success. We should study Zuckerberg’s 21st Century management style and Hacker Way philosophy. We can learn from its tremendous success. Zuckerberg and Facebook have proven that these management principles work in the digital age. It is true if it works. That is the pragmatic tradition of American philosophy. We live in fast changing times. Embrace change that works. As the face of Facebook says: “The riskiest thing is to take no risks.”

Postscript

Check out the DevBeat article Famous hackers discuss Zuckerberg’s “Hacker Way”. Apparently I’m not the only one who finds this story compelling. Some out-of-the-closet hackers do too, including the open-source crowd, who are critical of Facebook and Zuckerberg. This in spite of the fact that much of Facebook’s software is developed open sourced. They want more. They essentially criticize Mark as too entrepreneurial. They want non-profit purity. Personally I think they are full of it. Hello! This is America. This is a capitalistic culture. Yes, idealism is fine, but you also have to be real. The DevBeat editors ended on a positive note that I agree with:

We’re glad to see hackers portrayed by the young CEO as heroes rather than criminals.

Also, Facebook’s special strain of hacker culture is worth examining in its own right. The company is famous for prizing the iconoclastic work of small teams, for valuing the best solution regardless of provenance, for moving quickly, and for fearlessly building new systems and features from the ground up.

While Zuckerberg’s hacker ethic lacks the purity of its philosophical forebears, it marries some of the principles of hacking with the competitive advantage of a huge, global business. Whether the hacker ethic itself demands purity is another discussion for another day.


The Legal Implications of What Science Says About Recall

January 29, 2012

I hear a lot about how different software will find all relevant documents. That would be 100% recall. I also hear demands from requesting parties to find and produce all relevant documents. In the context of large disorganized banks of electronic data, such as email collections, these claims and demands are not only contra to the rules of law, embedded as they are in reasonability, but they are also unrealistic and contra to the latest scientific research. In my Bottom Line Driven Proportional Review article I showed how this kind of demand for all relevant ESI is not permitted under the rules and Doctrine of Proportionality in big data cases (and most cases these days are big data cases). I explained, as many have done before me, that the rules do not require production of all relevant documents, if the burden to do so is disproportional. I also shared my method for keeping the costs for review proportional to the value and importance of the case and the production request. But aside from the cost issue, how practical is it expect to find all relevant ESI? I examined this question at length in my Secrets of Search series, volumes one, two and three. Still, people find it hard to accept, especially in view of the unregulated clamor of the marketplace.

So as my last gift to readers before Legal Tech starts tomorrow, the ultimate event of marketplace claims and competing exaggerations,  I present you with a hard dose of reality, I present you with more findings on legal search from the world of science. This time I direct you to an important article, Evaluation of Information Retrieval for E-DiscoveryArtificial Intelligence and Law, 18(4)347-386 (2011). It was written by leaders of TREC Legal Track and established giants in the filed of legal search: Douglas W. Oard, Jason R. Baron, Bruce Hedin, David D. Lewis, and Stephen Tomlinson. They analyzed the now fully published test results of the experiments in 2008, and carefully examined the interactive task, topic 301, as the best test of competing legal search technologies. This task made use of a subject matter expert and an appeals process for quality control on relevance determinations. Four teams of experts participated in the test, two academic and two commercial. A well known e-discovery vendor won the test (scientists hate it when I put it that way). They won because they attained better precision and recall scores than the three other participants.

Now we come to the punch line, the winning vendor attained a recall rate of only 62%. That’s right, they missed 38% of the relevant documents. And they were the winner. Think about it. The other three participants in the scientific experiment attained recall rates of less than 20%! That’s right, they missed over 80% of the relevant documents. Now what do you think about a requesting party who demands that you produce all of the relevant email?

Find my summary of the experiments hard to believe, then read the report for yourself. Here is the excerpt on which I rely at page 24 of Evaluation of Information Retrieval for E-Discovery:

On the basis of the adjudicated sample assessments, we estimated that there are 786,862 documents (11.4% of the collection) relevant to Topic 103 in the test collection (as the topic was de fined by the TA). All four teams attained quite high precision; point estimates ranged from 0.71 to 0.81. One team (notably the one that made the most use of TA time) attained relatively high recall (0.62), while the other three (all making significantly less use of TA time) obtained recall values below 0.20.

The team of information scientists, and their lawyer guide, Jason R. Baron, next report on the 2009 TREC experiments, specifically the one they found most representative, the interactive tasks, again with subject matter consultations and appeals. This time there were eleven teams participating in the experiment, three academic and eight commercial. That’s right, eight e-discovery vendors were in the game this time. How did they do? They did a little better, but not much. Five of the teams, and just five only, got a little over 70% recall.

The post-adjudication results for the 2009 topics showed some encouraging signs. Of the 24 submitted runs (aggregating across all seven topics), 6 (distributed across 5 topics) attained an F1 score (point estimate) of 0.7 or greater. In terms of recall, of the 24 submitted runs, 5 (distributed across 4 topics) attained a recall score of 0.7 or greater; of these 5 runs, 4 (distributed across 3 topics) simultaneously attained a precision score of 0.7 or greater.

Id. at pgs. 24-25. If you follow the article’s direction and see the Overview of the TREC 2009 Legal Track, by B. Hedin, S. Tomlinson, J. Baron, and D. Oard, you can find more details of the 2009 test results. After you wade through the wonderfully dense language that information scientists love to use to convey information, you find section 2.3.5 Final Results. There you are pointed to a table of numbers: Table 6: Post-adjudication estimates of recall, precision, and F1.

What does this chart tell us? The best anyone did was an 86.5% recall on one of the seven tasks. Look at the third column from the left for the recall rates attained. The lowest was 9%. Digging deeper the analysts found that the teams with the highest scores appealed the most, and those with the lowest scores, not at all. Consultation with the topic authority also helped improve scores. But the bottom line for purposes of my point today, is that the average recall rate was only 41% (993/24), and even the best attained on one search, by one team of experts, was only 86%. Demands for recall in the 80s for every project are thus unrealistic.

Conclusion

The scientific research proves, once again, that it is unreasonable to ask for any better recall than 70%, in fact, it should be substantially less. Law demand reasonable efforts, not perfection. The best recall results attainable in scientific experiments, with the best software and top experts at the helm, is way too high a standard for reasonable efforts. Reasonability should be more like average results attained by average lawyers making good faith efforts, not results attained by information scientists and specialists using the best software money can buy. So that means it should be less than less than the 41% average of experts. Even standards like that should be used with caution and the efforts meter should always be tempered by costs. Proportionality of efforts should, if they are in good faith and reasonable, always trump any quality control efforts. See Bottom Line Driven Proportional Review.

In fairness to my vendor friends, the latest reports from TREC are dated. That was then, 2008 and 2009, this is now, 2012. The test scores showed substantial progress from 2008 to 2009. In my experience, the predictive coding type search software has significantly improved in the last year or so. I have also heard unsubstantiated reports of much higher recall rates attained in the 2011 TREC Legal Track tests, but I take all of these claims with a big grain of salt. Until Dr. Oard and his information scientist crew (that, by the way, includes two lawyers, Jason Baron and Maura Grossman) publish results, obtuse as their publications are, I will remain skeptical. Right now science shows that if you can find an estimated 41% of the relevant documents in a large collection of ESI, then you are doing just as good as the experts. That has got to be good enough to meet the reasonable efforts required under the law.

You should be skeptical of any claims or demands for better results than that. You should stop chasing, or being chased, by unreasonable demands for high recall rates. The only way to attain 70% or higher rates today is by document dumps, where precision plummets as you produce irrelevant documents, or, perhaps by budget busting, near-endless iterations of search and seed-set training. Even then, your expensive pursuit is quixotic from the point of view of science, where the fuzziness measurement issue remains unresolved. Furthermore, and most importantly, in today’s world of big data, where everyone has 100,000 emails, it is wasteful in the extreme to try to find all relevant documents. If your are still trying to find them all, and not just the few super-relevant smoking guns, you have not understood that in today’s age, relevant is irrelevant, nor that the ultimate goal of discovery is to prepare for trial, where the 7±2 rule of persuasion reigns supreme.


Reply to an Information Scientist’s Critique of My “Secrets of Search” Article

January 28, 2012

One of the leading information scientists in the field of legal search, was kind enough to write a detailed critique of my Secrets of Search series. I tried to post a response on his blog, Information Discovery where it appeared. But the website would not accept the comments for some technical reasons, so I am replying here, knowing that Herb will find them, and maybe some of his readers. I bring Dr. Roitblat’s comments to your attention, even though they are in the nature of a critique, because I want my readers to hear all sides of the story, not just mine. I am just a lawyer, and especially welcome peer review from the scientific community. That is part of a team approach to e-discovery, where Law, Science, and IT work together, and learn from each other, to do e-discovery right. The world is too complex, the electronic haystacks too vast, for lawyers to find relevant evidence without such an interdisciplinary team approach.

It is also interesting to see that a lot of Herb Roitblat’s stated disagreements appear to be based on misunderstandings of what I was trying to say. That is common in interdisciplinary team efforts. I accept that it was probably my fault, as I was writing about information science topics in Secrets of Search. But I don’t beat myself up too much about it because it is just so damned difficult to write intelligibly on the matrix between e-discovery law and information science. That is one reason that almost no one even tries. Still, this apparent miscommunication presents an opportunity. By addressing Herb’s issues we can attain greater clarity of the emerging consensus. His comments, and my responses, suggest that we both agree on far more than we disagree. From my perspective, at least, the points of disagreement are really minor and technical. They pale in comparison to our mutual agreement as to the superiority of technology assisted review over mere manual review.

Before I post the reply, I have to give you an idea of  Dr. Roitblat’s commentary, so you can understand what I’m replying to. But better still, take few minutes to read his entire article, On Some Selected Search Secrets. Only in this way will my response make full sense. I do not really know Herb, although I think we’ve met at some events over the years, but I certainly know of him. He is one of the few info scientists around that is focused on legal search and actually makes a living out of it. (Apparently he also likes dolphins and killer whales.) He owns a company, OrcaTec, that, in his words, provides professional services and software for information discovery and information management. My Secrets of Search article frequently cited one of his works with the e-Discovery Institute: Roitblat, Kershaw, and Oot, Document categorization in legal electronic discovery: computer classification vs. manual review; Journal of the American Society for Information Science and Technology, 61(1):70–80, 2010.

Summary of Herb Roitblat’s Critique

Here is Herb’s analysis, which begins in a very flattering manner that I don’t deserve:

Ralph Losey recently wrote an important series of blog posts (here, here, and here) describing five secrets of search. He pulled together a substantial array of facts and ideas that should have a powerful impact on eDiscovery and the use of technology in it. He raised so many good points, that it would take up all of my time just to enumerate them. He also highlighted the need for peer review. In that spirit I would like to address a few of his conclusions in the hope of furthering discussions among lawyers, judges, and information scientists about the best ways to pursue eDiscovery.

These are the problematic points I would like to consider:
1. Machines are not that good at categorizing documents. They are limited to about 65% precision and 65% recall.
2. Webber’s analysis shows that human review is better than machine review
3. Reviewer quality is paramount.
4. Human review is good for small volumes, but not large ones.
5. Random samples with 95% confidence levels +/- 2 are unrealistically high.

Ralph’s Response: Thanks for taking the time to provide input on my article. I appreciate your comments, and actually agree with most of the points you make. I think you may have misunderstood some of what I was saying and your disagreement is actually agreement, but I much appreciate your clarifications. I will respond based on your enumerated points above.

Dr. Roitblat then explains the five problems that he had with a few of the conclusions that I made in Secrets of Search. Again, I urge you to read all of his comments, but for ease of reference, I here quote what I think is the essence of each of his five issues, and then follow with my response.

Issue [1]: Machines are not that good at categorizing documents. They are limited to about 65% precision and 65% recall. Losey quotes extensively from a paper written by William Webber, which reanalyzes some results from the TREC Legal Track, 2009, and some other sources. Like Losey’s commentary, this paper also has a lot to recommend it. Some of the conclusions that Losey reaches are fairly attributable to Webber, but some go beyond what Webber would probably be comfortable with. The most significant fact, because important arguments are based on it, is a description of some work by Ellen Voorhees that concluded that 65% recall at 65% precision is the best performance one can expect. The problem is that this 65% factoid is taken out of context. In the context of the TREC studies and the way that documents are ultimately determined to be relevant or not, this is thought to be the best that can be achieved. The 65% is not a fact of nature. It says, actually, nothing about the accuracy of the predictive coding systems being studied. Losey notes that this limit is due to the inherent uncertainty in human judgments of relevance, but goes on to claim that this is a limit on machine-based or machine assisted categorization. It is not. …

Ralph’s Response [1]: I agree with you. I was not trying to say 65% precision or recall is all that is possible to attain, just that the fuzziness of our lenses makes it hard to prove anymore than that, unless special review controls are put in place for the measurements. These controls have been lacking in most legal tests to date. TREC is making progress with limited subject matter expert input, but even there, thanks to monetary constraints, we still have a ways to go to use a true gold standard that could improve our measurements. So I agree with you that 65% is no “fact of nature” as you put it, or inherent limitation in human relevancy determinations. (I am not ruling that possibility out entirely, but if such a mental limit like that does exist, my experience tells me that it is higher than 65%.) This fuzziness issue is more than a mere anomaly and deserves wide-spread discussion and recognition. In so far as large-scale human reviews are concerned, reviews unassisted by technologies, the kind of reviews that were common in the past, the 65% fuzzy focus may well be an inherent human limit. With predictive coding and other automated process, however, this barrier can be broken. Finally, I like your suggestion to improve TREC experiments by using both an authoritative training set and an authoritative judgment set.

Issue [2]: Webber’s analysis shows that human review is better than machine review. I have no doubt that human review could sometimes be better than machine-assisted review, but the data discussed by Webber do not say anything one way or the other about this claim. Webber did, in fact, find that some of the human reviewers showed higher precision and recall than did the best-performing computer system on some tasks. But, because of the methods used, we don’t know whether these differences were due merely to chance, to specific methods used to obtain the scores, or to genuine differences among reviewers. Moreover, the procedure prevents us from making a valid statistical comparison. …

Ralph’s Response [2]: Again, I agree with you. I get that Webber’s analysis suggests that humans are only sometimes better, not always. In fact, I would go much further and say that humans always lose over large-scale review (weeks on end of 8 hours a day reviewing hundreds of thousands of boring documents) when paired against today’s good software. Still, Webber pointed out what no one else had before about the TREC results, that the humans sometimes did win on the small-scale, even when substandard manual review methods were used. I think it is wrong to just sweep that under the rug as an anomaly or luck. This realization of human abilities is important for proper application of the predictive coding process, where, in my opinion, input by experts on the seed coding is key. These experts need a clear understanding of what is relevant, and what is not. Otherwise, no matter how good the software, the computer principle of garbage in, garbage out, will control.

This realization of the continued importance of Man in the technology equation is also important to defeat the sophistic arguments of some plaintiffs’ lawyers (or better put “requesting party lawyers). They are now arguing in multiple courts around the country that a defendant (responding party) should forego any manual review and just turn over documents based solely on automated review. They use that argument to oppose motions for protective orders based on excessive cost and burden to review. They are misusing distorted reports of scientific research to try to force quick peek disclosures. But the truth is, automated coding is not good enough yet to dispense with final manual quality control reviews to protect confidential information in a litigation context. Webber’s findings help prove that. The advantage to plaintiffs’ counsel of such disingenuous, forced quick peek strategy is obvious and substantial. Claw backs and Rule 502 are inadequate protections. Once the bell has been rung, the damage is done, regardless of whether the documents are returned. The main point I was trying to make by publicizing Webber’s finding is that humans still have a place at the table, not that they should sit there alone without reliance on the latest software for culling review. I suspect you agree with me on that.

Issue [3]: Reviewer quality is paramount. Webber found that some assessors performed better than others. Continuing the argument of the previous section, though, we cannot infer from this that some assessors were more talented, skilled, or better prepared than others. … The best reviewers on each topic could have been the best because they got lucky and got an easy bin, or they got a bin with a large number of responsive documents, or just by chance. Unless we disentangle these three possibilities, we cannot claim that some reviewers were better or that reviewer quality matters. In fact, these data provide no evidence one way or the other relative to these claims. … In some sense, the ideal would be for the senior attorney in the case to read every single document with no effect of fatigue, boredom, distraction, or error. Instead, the current practice is to farm out first pass review to either a team of anonymous, ad hoc, or inexpensive reviewers or to search by keyword. Even if Losey were right, the standard is to use the kind of reviewers that he says are not up to the task.

Ralph’s Response [3]: I think you misunderstood my point and again assumed incorrectly that I was advocating for large-scale manual review. I am not. I agree the reviewers are not up to the task, even the best. As explained above, I think humans cannot perform over long periods of time, and so I am not advocating against machine review, I am advocating for hybrid review, man and machine working together. Like you, I advocate for change. So really we agree.

But I do disagree with some of your statements here. To paraphrase Shakespeare: me thinks thou dost protest too much. I don’t think it is wrong to assume a correlation between accuracy and skill. That connection is based on experience and common sense. All large-scale review project metrics show that some reviewers are better than others, just like some trial lawyers are better than others, and some scientists, etc. It is inherent that we all perform to different levels at different tasks. I do not understand the need to try to explain all of the variances as just luck or chance. (As the great golfer Gary Player used to say: the more I practice, the luckier I get.)  Although I concede some chance or luck is possible, the same could be said of the software tested. Perhaps the “winning software” just got lucky. I would not seriously make that argument, so I am surprised to hear it made about the reviewers here. TREC only tried to measure comparisons, as you said, and lady luck knows no favorites.

Issue [4]: Human review is good for small volumes, but not large ones. This claim may also be true, but the present data do not provide any evidence for or against it. The evidence that Losey cites in support of this claim is the same evidence that, I argued, failed to show that human review is better than machine review. It requires the same circular reasoning. … Based on other evidence from psychology and other areas, it is likely that performance will decline somewhat with larger document sets, but there is no evidence here for that. If this were the only factor, we could arrange the situation so that reviewers only looked at 500 documents at a time before they took a break.

Ralph’s Response [4]: I agree this was not tested. I was again relying on my experience outside of these experiments and relying on my common sense built from over 30 years of doing document review, paper and electric, big and small. But I get your point of scientific discipline that it was not tested and so not here proven. Still, I’m not a scientist, nor do I care to become one. Also, I write primarily for lawyers, not scientists (although I am very happy a few of you are interested enough to read them too,  at least when it touches on your work). I am a lawyer interested in learning from science for purposes of improving law, not visa versa, although that may be a secondary benefit. That would depend on scientists like yourself. Also, as you know, there is more to establishing best practices in review processes than simply adding in periodic breaks.

Issue [5]: Random samples with 95% confidence levels +/- 2% confidence intervals are unrealistically high. It’s not entirely clear what this claim means. On the one hand, there is a common misperception of what it means to have a 95% confidence level. Some people mistakenly assume that the confidence level refers to the accuracy of the results. But the confidence level is not the same thing as the accuracy level. … I suspect that Losey means something different. I suspect that he is referring to the relatively weak levels of agreement found by the TREC studies and others. If our measurement is not very precise, then we can hardly expect that our estimates will be more precise.

Ralph’s Response [5]: You have correctly divined my intent here on sampling. I was again referring to the measurements fuzziness issue reported by your scientific colleagues, Voorhees, Webber and to some extent Oard. I understand that you are uncomfortable with their findings and conclusions on accuracy. I sincerely hope that you and other scientists will work this issue out.

I want accurate measurements too, especially when important points of justice are at stake. I want all of the scientific research out there for full public view, even the troubling preliminary conclusions of Voorhees, Webber and Oard.  If the measurements are disputed, I want full disclosure on that. If it takes more money, time and effort to get these measurements done properly in scientific testing, then lets raise the funds to do it right. I support the important scientific research now going on in legal search. On that point I suspect we once again agree.

Again, thanks for your comments on my article.

_________________

DEAR READERS: I’m off to LegalTech, where I will not only be presenting with Craig Ball and Judge Andrew Peck in the much hyped debate on Tuesday, January 31st at 4:00 at the Sutton Center on the 2nd floor (sponsored by BIA), but I will also be presenting three more times on predictive coding related subjects. I am thinking of preparing for all of that the way Pat Sajak prepared to host Wheel of Fortune.

On Monday the 30th, I present at 12:30 on The Promise and Challenge of Predictive Coding and Other Disruptive Technologies with Judge Andrew Peck, Maura Grossman and Dean Gonsowski (sponsored by Clearwell/Symantec).

On Wednesday, February 1st, I present at 10:30 on Technology Assisted Review: When to Use it and How to Defend It, with Maura Grossman, Judge Frank Maas, and Ann Marie Gibbs (sponsored by Daegis).

My last gig on Wednesday is at 1:45 in the Sutton South Parlor on E-discovery Circa 2015: Will Aggressive Preservation/Collection and Predictive Coding be Commonplace? My fellow panelists are David Kessler, Robert Trenchard, Julie Colgan, Stephanie Blair, and Craig Carpenter (sponsored by Recommind and ARMA).

If you see me around, please stop and say hello. I like to meet all of my readers whenever possible. Please forgive me if you catch me at a time during the day when I don’t have time to chat, but I always have time to shake hands and say hello.