“People make mistakes.” This simple three word sentence is how Chief Justice John Roberts begins his opinion in Conkright v. Frommert, No. 08-810 (Apr. 21, 2010). He goes on to add: “Even administrators of ERISA plans.” Then he explains how complicated those plans can be. As a former ERISA litigator, I know he’s right, as I have read far too many ERISA plans myself. But let me tell you, as an attorney who left ERISA to focus solely on e-Discovery in 2006, it’s nothing compared to ESI plans.
As complicated as the facts and law are in employee benefits disputes, the world of electronic discovery with its ever-changing technologies is far more complicated. So if the Supreme Court is inclined to give ERISA administrators a break, which they did in Conkright, then surely they will do the same in e-discovery too, if and when such a case ever darkens their door. The Justices of the Supreme Court may not have a good grasp of today’s technology, as I examined in The-times-they-are-a-changin’ is a feeble excuse for disregard of duty, but they do have a good grasp on human nature and the law. They know the proper standard for judicial review is reasonability, not perfection.
The lower courts should also recognize this simple truth, that people make mistakes, and so too do the computer systems they design and run. The law should never demand perfection. Judges need to better understand that this applies to the world of ESI spoliation and sanctions too. The law should only demand reasonable, good faith efforts to preserve, collect, review and produce. If these efforts are made, but mistakes still happen, for instance an email is not preserved, or a privileged document is produced when it should have been withheld, there should be no sanctions. The judges should recognize the limits of our humanity, and the enormous complexity of our task, as Chief Justice Roberts did in Conkright. They should temper their response accordingly.
Judge Scheindlin gets it. In her most recent landmark case, The Pension Committee of the University of Montreal Pension Plan, et al. v. Banc of America Securities, et al., 2010 WL 184312 (S.D.N.Y. Jan. 15, 2010), which she amended twice after catching mistakes in her opinion, Judge Scheindlin said:
In an era where vast amounts of electronic information is available for review, discovery in certain cases has become increasingly complex and expensive. Courts cannot and do not expect that any party can meet a standard of perfection.
Id. pg. 1. She then goes on to qualify that although perfection is not required, reasonable efforts are required, and the standards for acceptable practice have evolved quite a bit since she wrote Zubulake. See Eg. Raising the Bar – Judge Scheindlin Defines Gross Negligence in Spoliation. Also see: BREAKING NEWS: Judge Scheindlin Amends Pension Committee Again.
But sadly, not all judges get it. If any mistakes are pointed out to them, they rush to sanctions, sometimes even including waiver of the attorney-client privilege. In my opinion, that is what happened to the Venable law firm in Mt. Hawley Ins. Co. v. Felman Production, Inc., 2010 WL 1990555 (S.D. W. Va. May 18, 2010). They missed several attorney-client emails, and accidentally produced them, instead of withholding them. According to the court’s own findings of fact, the mistake was made primarily because of an error of some kind in the indexing of one of thirteen Concordance databases the firm was using to run this large project.
The Magistrate Judge who issued the ruling seemed to be motivated more by the contents of what she read in the emails, than by the law. She departed from the law’s demand of reasonability, not perfection. She glossed over the software failure (we all know that no software is perfect and all computers can and will fail), and instead focused on the law firm’s failure “to perform critical quality control sampling to determine whether their production was appropriate and neither over-inclusive nor under-inclusive, even though Venable was counsel in the Victor Stanley case.” For the full story on this case see: The Good, the Bad, and the Ugly: “Mt. Hawley Ins. Co. v. Felman Production, Inc.”
What does this mean? People make mistakes. In e-discovery, particularly in complex cases, as Feldman surely was, mistakes happen all of the time. So, unless you have a wise and forgiving technophobe like our Chief Justice hearing your case, or a technologically sophisticated jurist like Judge Scheindlin, you may be required to double-check your work with sampling before it goes out the door. This may be what is now necessary to protect yourself against the inevitable mistakes. If you don’t, and a mistake causes harm (or someone argues it does), you may be found to have acted unreasonably. You know that movants for sanctions will cite Felman and others to argue that it is negligent not “to perform critical quality control sampling.”
People may make mistakes, that’s ok. But if the person is a lawyer or technician in an e-discovery project, they may now be expected to do some sampling to try to catch their mistakes. I am not saying this is right or wrong, but it does seem to be a trend.
So must we all now practice defensively and sample everything? Not only that, must we carefully document our sampling and be prepared to share it someday with a skeptical court? Shall I start working on a new online course for law schools called Sampling 101?
Will our clients agree to pay for this added step in an already pricey process? Do they have any choice if they want to improve their odds of protecting their privileged communications?
What do you think? Should sampling now be a requirement for reasonability? Is it already a best practice? If so, for all cases or just a few? If just a few, which ones? Moreover, what does sampling cost? What does it entail? How many lawyers really know how to sample anyway? How many know the significance of the number 1537? Do you know what it means to have an estimate of +/- 5% with a confidence interval of 95%? Do you care? If so, see eg. the EDRM Search Guide Appendix on Sampling. Are all lawyers expected to know this stuff? Or just the ones that happened to be in Victor Stanley case?
ERISA lawyers have it easy. If you want a real challenge, become an e-discovery lawyer.
Could you explain the significance of the number 1537?
Yes, I do think sampling is necessary in large e-discovery cases. Information inflation results in a situation where it is no longer practicable for every document to be manually reviewed. The alternative to linear, manual review is dynamic, largely automated review that utilizes advanced analytics to mass tag and/or predictively code documents. As you observed, software is not always right. Neither are the humans who use it. Thus, the reasonableness of such an approach is largely contingent on sampling. As there will always be confidence level and error condition, the approach assumes that perfection is not the standard.
The question of reasonableness is one of process, not results. The actual results in Pension Committee – 311 missing documents – are not particularly egregious. But the process was so bad, that it was more enough to demonstrate negligence. Had a reasonable process been followed – e.g., a proper hold had been instituted or plaintiffs had gotten defendants’ sign-off as part of a transparent, iterative, and documented 26(f) process – I expect the decisions comes out different.
No, I do not think every lawyer needs to know this stuff. E-discovery is a specialty, like tax or IP. What every lawyer needs to know is when to bring in a specialist.
Peg, 1537 is the sample size needed to achieve a confidence level of 95% and an error condition of 5% where the data follows a Poisson distribution (which most ESI does). One of primary characteristics of a Poisson distribution is that sample size is independent of population size. So 1537 would be the sample size for a 100,000 document population and a 100,000,000 document population.
I think you hit the nail on the head. I have been reading and re-reading the Mt. Hawley case to draw my own conclusions and that is where I came out. I think the Judge was upset at Fellman and their counsel and wrote the decision around that fact. There are a long list of steps counsel took to try and catch privileged documents as the Magistrate lays out. Yet, documents got through so those steps seemed to be for naught.
With millions of pages produced, even sampling won’t find every mistake, and possibly few of them. After all, accountants sample for fraud but miss it all the time. They warn the client that they will miss fraud as part of their engagement. If we use hindsight as the test, there will always be steps someone could have and now should have done.
The say bad facts make bad law. I think that is what is going on there at least.
Great writing Ralph.
I couldn’t agree more with Casey that eDiscovery is a specialty like tax or IP. With that, I think (statistical) sampling MUST be a tool every eDiscovery attorney has at their disposal for the right situation. Reasonableness governs the use of this tool like it does the larger process, so statistical sampling should not be required in every case but should be used in appropriate circumstances.
Sampling is one half of the clapping hand. There must be a standardized, repeatable process for sampling to have any reliability. And you can’t just sample once, at the end.
I once toured a water heater factory. The only differences between the five-year warranted heater and the 10-year warranted heater were pressure testing (one-in-30 for the five year; each one for the 10-year) the label and the price. They all came off the same production line, using the same quality tools and materials with the same tolerences. They were all engineered for a 30 year rated life. The manufacturer had experience with how many would fail in various periods of time, and the cost of replacing them when they did.
Sampling (the pressure test) was important because you didn’t want too many 10 year heaters failing on installation. Some people will pay a premium and have higher expectations than objectively warranted. Otherwise it didn’t matter. It was all calculated to optimize profit – they made a lot more money on the 10-year heaters, but had to compete in the down-market for commodity 5 year heaters.
What did matter was consistent process in making the heaters.