A new case involving the movie The Secret provides excellent analysis of new Evidence Rule 502 while at the same time taking irony to new heights. Heriot v. Byrne, 2009 WL 742769 (N.D. Ill. March 20, 2009).
I do not normally write a blog in the middle of the week, but the sarcastic potential and ironies in Heriot v. Byrne are just too much to resist – sleep can wait. The Law of Attraction compels this exposé. Consider the facts. This is a law suit about a movie called The Secret, where one of the corporate defendants is named The Secret, LLC. All of the parties in this law suit were somehow involved in the conception and making of The Secret. The movie was a huge New Age success. The exalted spiritual masters who conceived and produced The Secret did so, or so they say, to share the secret of the ages as to how to attract success and happiness in your life.
Well, now the real secret is out, these special masters have indeed attracted what they deserve, they have attracted lawyers and fights over money. The plaintiffs claim to own part of The Secret. The defendants deny that and have filed counter-claims of their own. The plaintiffs’ primary goal seems to be their accounting claims where they are trying to find out how much money The Secret has made, which, so far, the defendants have kept secret.
But wait, there’s still more secrets. The defendants accidentally received some of the plaintiffs secret emails to their attorney. They received these secrets because the plaintiffs’ e-discovery vendor made a mistake. The named of the vendor is Open Door Solutions. You cannot make this stuff up. If this were a movie, you would not believe it, which, by the way, appears to have been the judge’s reaction to the movie, but we will get to that in a minute.
Back to the open-door disclosure of the plaintiffs’ secret attorney communications. When these secrets were accidentally disclosed to defense counsel, they denied that they were secrets at all. Alternatively, they said that even if they were once secrets, they were no longer secrets, because the secrets were revealed. They argued that the disclosure of the secrets by handing them over to opposing counsel waived any attorney client privilege that these secrets might have once enjoyed.
Plaintiffs tried to protect their secrets by relying on new Evidence Rule 502. It protects attorney-client secrets and requires clawback of inadvertently disclosed secrets if certain conditions are met. Defendants denied clawback and the applicability of Rule 502. Defendants claimed that the plaintiffs’ disclosure of secrets was not inadvertent. Even if the disclosure was inadvertent, defendants argued that Rule 502 should still not protect the secrets because it only protects secrets where reasonable steps to prevent disclosure were taken. The claimed that the way-too-open-door handling of plaintiffs’ alleged secrets was negligent and thus not protected by Rule 502. Finally, they argued that even if these secrets were protected by Rule 502, they should still be allowed to tell these secrets, and not give them back, because of the crime-fraud exception to the attorney-client privilege. That is the rule where a client cannot keep secret any communications he may have with his attorneys about a crime or fraud that he may commit in the future. (Confidential communications about past crimes and fraud committed in the past can, however, be kept secret.)
So to summarize, open door mistakes led to a battle between the masters of The Secret over the attorney-client secrets of a man who claimed to own part of The Secret and wanted its secret profits disclosed. The defendants, purported sole owners of The Secret, including one called The Secret, LLC., wanted to keep their profits secret, but disclose the plaintiffs’ secrets, claiming they were not secrets at all, or if they were, they were bad secrets not entitled to secrecy, or they were waived by a way-too-open door handling of the secrets. (Try saying that real fast.)
Magistrate Judge Martin C. Ashman attracted to himself the task of sorting out this tangled web. As mentioned, Judge Ashman was not too impressed by the movie The Secret, as he begins his opinion by noting:
This case involves a controversy over The Secret, a made-for-television documentary that purports to reveal the “most powerful law in the universe.” The Secret, http://www.thesecret.tv/ (last visited Mar. 17, 2009). As if that declaration was not grandiose enough, the website further claims that, “[b]y applying the knowledge of this law, you can change every aspect of your life.”
Although Judge Ashman did not think The Secret the parties had created was much of a secret, he did think the plaintiff’s secrets were bona fide and entitled to protection. He upheld the attorney-client privilege by applying Evidence Rule 502. He held that the open-door disclosure was a mistake, and happened in spite of the reasonable steps to prevent disclosure that had been instituted by plaintiff’s attorneys. For that reason, Judge Ashman enforced the clawback provisions of Rule 502 and required that the defendants, including The Secret, LLC., to keep the plaintiffs’ secrets secret.
Factual and Procedural Background
This saga of secrets began innocently enough with the defendants’ request for production, including “[a]ll documents relating to United States visa applications filed by or on behalf of Heriot.” The plaintiff, Drew Heriot, happens to be a citizen of Australia, as are most of the other parties to this action.
Since all documents these days are created electronically, this triggered e-discovery, which in turn triggered the attorneys for Heriot to seek the help of an e-discovery vendor, the mentioned Open Door Solutions. Open Door provided electronic scanning, optical character recognition, and other discovery related services. Here is the court’s description of the process which Open Door (here called the “Vendor”) and Heriot’s attorneys set up for this production:
First, the Vendor created a database of the documents provided to it by Plaintiffs (“Master Database”), which Plaintiffs then could review. (Id. at 3-4.) Second, during April and May of 2008, Plaintiffs “had paralegals and other non-lawyers conduct a preliminary review [of] the documents in the Master Database,” assigning documents “general, pretrial discovery codes.” (Id. at 4.) One general code was “immigration,” which Plaintiffs “used to flag documents (primarily e[-]mails) that had anything to do with Mr. Heriot’s immigration to the United States.”(Id.) (Editors note: the misspelling of “immigration” in the Westlaw transcription was corrected as an obvious error.)
Third, Plaintiffs searched for responsive documents in their Master Database. (Id.) Fourth, once identified, these responsive documents would be “coded for subsequent copying and inclusion in the Production Database.” (Id.) Additionally, Plaintiffs would mark responsive documents “as either Confidential or Highly Confidential under the Stipulated Protective Order.”
Id. at *2.
This is a pretty standard plan of action, except that I do not see privilege review and logging specifically mentioned. To comply with the discovery request regarding Heriot’s immigration visa, plaintiffs requested and received a paper copy of the visa file from Heriot’s immigration lawyer. Next, plaintiffs gave the Heriot’s visa application to Open Door with instructions to:
(1) scan and add it to the Master Database, (2) copy the scanned versions to the Production Database, … (3) electronically stamp the application as ‘Highly Confidential’ for production[,] … [ (4) ] Bates Stamp all documents in the Production Database, and … [ (5) ] include Mr. Heriot’s visa application and other sensitive material at the front of the production.
Id.
Now we come to the mistake of Open Door. (By the way, mistakes like this can and do happen to most everyone sooner or later, but seem to happen more often to some than to others.) Frankly, from the court’s limited description of the facts in the opinion, I cannot understand exactly what the vendor did wrong that supposedly blew the privilege review, but I gather it had to do with mixing the sensitive visa related materials in with the general database and not placing them in the front of the production for review as instructed. The vague description makes me more sympathetic towards Open Door and a little suspicious about the attorneys. All we know for sure is that mistakes were made here and secret emails were unintentionally produced.
On August 25, 2008, plaintiffs produced around 1,499 documents comprised of 6,952 pages. This is a pretty small production by today’s standards. Plaintiffs did not withhold any documents for privilege and so did not produce a privilege log. That does seem odd. Two months later, on October 22, 2008, plaintiffs counsel discovered the mistake in production while preparing for a deposition scheduled for October 24, 2008. The next day, October 23rd, they notified defense counsel of the inadvertent production and demanded the return and destruction of certain documents they claimed were privileged. The defense complied, but kept one copy for use to challenge the claimed privilege protection. The defendants then promptly filed a motion to compel production, including the challenged documents, which were filed under seal. The plaintiffs opposed that motion and filed their own unrelated motions to compel.
Legal Analysis on Waiver
Judge Ashman begins the legal analysis with the following overview of the law (citations omitted):
Ordinarily, disclosure of confidential information to an unprotected third party operates as a waiver. Under FRE 502, however, disclosure of privileged information will not operate as a waiver when “(1) the disclosure is inadvertent; (2) the holder of the privilege or protection took reasonable steps to prevent disclosure; and (3) the holder promptly took reasonable steps to rectify the error, including (if applicable) following Federal Rule of Civil Procedure 26(b)(5)(B).” FED.R.EVID. 502(b)(1)-(3). All three elements described in FRE 502 must be satisfied to prevent a waiver. Id.
Id. at *5.
The court then considered the issue of whether the new rule superseded prior federal common law concerning waiver of attorney-client privilege. The court basically said it did, unless the prior law (here Judson Atkinson Candies, Inc., v. Latini-Hohberger Dhimantec, 529 F.3d 371, 387-88 (7th Cir.2008)) was not conflicting. Here Judson was not conflicting (it was harmonious) and so the court used Judson to supplement its analysis and articulated the following test (footnote omitted):
This Court therefore adopts the following test. First, a court determines whether the disclosed material is privileged. If it is not, the inquiry ends. If the material is privileged, the court applies FRE 502(b). If the court concludes that disclosing party satisfied all of the elements in FRE 502(b), the privilege is not waived. If, however, the disclosing party fails to satisfy any of the FRE 502 elements, the privilege is waived. In applying FRE 502(b), the court is free to consider any or all of the five Judson factors, provided they are relevant, to evaluate whether each element of FRE 502(b) has been satisfied. This Court applies and explains the details of this test in the following sections.
Id. at *7. Judge Ashman then applied the test where he began by finding that most of the emails in question were privileged. The judge personally reviewed all of the emails and found they contained legal advice to Heriot from his immigration lawyer. A few emails included prior non-privileged communication with a third party. They were to be produced, but with redactions to protect the privileged part of the email chain. A few others had been sent to third parties and, as to them, no privilege was allowed. Id. at *8-*9.
Next the court addressed the Crime-Fraud Exception to the attorney-client privilege. Here is the court’s summary of this law:
Although communications subject to the attorney-client privilege are protected, that protection can be forfeited when, for example, “ ‘the attorney … assist[s] his client to commit a crime or a fraud.’ “ U.S. v. Al-Shahin, 474 F.3d 941, 946 (7th Cir.2007) (quoting Mattenson v. Baxter Healthcare Corp., 438 F.3d 763, 769 (7th Cir.2006)). In other words, “[t]he crime-fraud exception places communications made in furtherance of a crime or fraud outside the attorney-client privilege.” U.S. v. BDO Seidman, LLP, 492 F.3d 806, 818 (7th Cir.2007) (citing U.S. v. Zolin, 491 U.S. 554, 563, 109 S.Ct. 2619, 105 L.Ed.2d 469 (1989)).
Id. at *9. It turns out that the defendant’s arguments here were very weak and there was no basis for a claim of fraud in these emails between Heriot and his immigration attorney.
Three Part Test of Rule 502(b)
Judge Ashman then moved on to consider whether the three requirements of FRE 502(b) were met wherein disclosure of privileged information will not operate as a waiver when:
(1) the disclosure is inadvertent;
(2) the holder of the privilege or protection took reasonable steps to prevent disclosure; and
(3) the holder promptly took reasonable steps to rectify the error, including (if applicable) following Federal Rule of Civil Procedure 26(b)(5)(B).”
Rule 502(b)(1)-(3) Fed.R.Evid.
1. Inadvertence
The court put the burden of proof on the party claiming the privilege in accord with established case law. This is the way it should be. Judge Ashman then found that the first requirement had been met, that the disclosure had been inadvertent. This was the right result, but he labored way-too-hard to get there. In my opinion, it is hard to see how a disclosure like this could be anything but “inadvertent,” taking the normal common sense meaning of the word as “unintentional,” or “not attentive.” Defense counsel here argued in accord with case law that “inadvertent” meant something completely different, that inadvertent here means “not negligent.” The defendants main argument on this point was that the disclosure was “not-inadvertent” because, in their words “Plaintiffs’ counsel … [was] asleep at the switch.” Heriot v. Byrne, supra at *11. Judge Ashman went along with that argument, but then went to great lengths to show they were not quite as asleep as alleged and so the disclosure was inadvertent as required under the rule.
I think this misreads the rule by conflating all three criteria into the first one. This incorrectly resurrects somewhat contra case law that Rule 502 supersedes. Obviously plaintiffs’ counsel was “asleep at the switch” when the production was made and that is exactly what inadvertent means, not attentive or unintentional. For part one of Rule 502(b) three-part test, the party should only have to prove that the disclosure was not done on purpose, that it was a mistake.
Here Judge Ashman made a finding that the disclosure was inadvertent because a variety of factors were met indicating the mistake was understandable and prompt corrective action was taken. These facts are relevant to the second and third prongs of the rule, not the first prong of inadvertence. For that reason I think Judge Ashman made a harmless error in stating:
To determine whether a disclosure was inadvertent, “this Court has … look[ed] to factors such as the total number of documents reviewed, the procedures used to review the documents before they were produced, and the actions of producing party after discovering that the documents had been produced.” WunderlichMalech Sys., Inc. v. Eisenmann Corp. (Eisenmann II), No. 05-C-4343, 2007 WL 3086006, at *3 (N.D.Ill. Oct. 18, 2007) (citing Wunderlich-Malech Sys., Inc. v. Eisenmann Corp. (Eisenmann I), No. 05-C-4343, 2006 WL 3370700, at *3 (N.D.Ill. Nov. 17, 2006)). This Court can find no reason to discard these factors, which aptly address the issue of whether a party inadvertently disclosed confidential information.
Id. at *11. The plain language of the rule is good reason enough for me. The court should have disregarded these prior case law factors, at least in so far as the determination of the “inadvertence” test is concerned. All a party should have to prove to meet the first part of the test is that the disclosure was not intentional. If someone intentionally made the disclosure, and then just changed their mind for some reason, like perhaps new counsel was retained, then you would fail the first test. But if the disclosure was a mistake, an unintentional accident, then according to the normal commonsense meaning of the word, it was inadvertent, and you move on to the next two steps.
This is, by the way, the only mistake I see in this opinion, which undertakes the difficult task of interpreting and applying a new rule with very little prior case law. This is otherwise a well reasoned opinion demonstrating excellent scholarship. I just hope that the many future cases that are likely to cite this decision do not make the same mistake of conflating the three criteria in Rule 502(b).
2. Reasonable Steps to Prevent
The next thing to be proved under the Rule is that “the holder of the privilege or protection took reasonable steps to prevent disclosure.” Now we get into the facts and circumstances incorrectly considered by Judge Ashby in the determination of the first step. The defendants here argued that it was unreasonable for plaintiffs not to review the ESI after it was readied for production. They made this argument, even though plaintiffs’ counsel they had already reviewed the email and other documents before they were delivered to the vendor, here Open Door, for final processing and production. The defendants’ position would require either a double review, or review only after processing by a vendor. Either alternative is not acceptable in most circumstances, although I for one do not like a vendor to make direct productions, and where possible would prefer to make a final quick overview myself before production, just to be sure obvious errors were not made. Judge Ashby wisely rejected defendants contention and held:
… a pre-copy review procedure is not per se unreasonable. The procedure used here-where Plaintiffs reviewed the documents and then provided them to the Vendor for production-was reasonable; no disclosure would have occurred but for the Vendor’s error.
Id. at *14. Of course, had plaintiffs counsel made a quick inspection before turning over the CD, this error might well have been caught. But whether mistakes were made is not the issue. Of course mistakes were made, or the unintentional disclosure would not have occurred. The issue is whether there was a reasonable procedure in place. Here the judge said there was and I do not disagree.
3. Reasonable Steps to Rectify
The third and final test under Rule 502(b) is whether the party that made the unintentional disclosure “promptly took reasonable steps to rectify the error.” Here there was a two week gap between the time of the production and the time the disclosure was noticed. Plaintiffs notified defendants the very next day after the mistake was discovered, so the defendant had to argue that the two month delay was not prompt or reasonable. This means they had to argue for the necessity of a post-production review. I know that some attorneys and judges contend this is the intent of Rule 502: that you have to look again at documents already produced to verify that you caught and logged all privileged documents; that to do otherwise is not reasonable and constitutes a violation of Procedure Rule 26(b)(5), which requires that all withheld privileged documents be logged. This same people are at a loss to explain the Rules Commentary that expressly states:
The rule does not require the producing party to engage in a post-production review to determine whether any protected communication or information has been produced by mistake.
I agree with Judge Ashman who held:
Plaintiffs had no duty to re-review the documents after providing them to the Vendor. Id. That would be duplicative, wasteful, and against the spirit of FRE 502. Additionally, imposing on disclosing parties a duty to re-review would chill the use of e-vendors, which parties commonly employ to comply with onerous electronic discovery. Against this grain the Court cannot cut.
Heriot v. Byrne supra at *13.
I think it is clear that Rule 502 does not require another review after production, just to make double-sure that a mistake was not made and privileged documents inadvertently disclosed. If your preproduction steps to prevent disclosure were reasonable, then you are done and should not have to do anything more to search for privileged documents. Your only remaining duty should be to rectify, if and when you happen to discover a mistake while doing something else, like in this case, reviewing documents to prepare for a deposition, or you are advised of the mistake by opposing counsel. Many times when the later happens, opposing counsel will cooperate and just notify you of the mistake and send it back. This is especially true in circumstances where the vendor or law firm make a big mistake, like mislabeling a CD and turning over information to the wrong party. This kind of thing can easily happen despite all kinds of reasonable precautions and good intentions. When it does, most opposing counsel are very polite about, as they should be, and they tell but don’t look. After all, we all know that “Where discovery is extensive, mistakes are inevitable ….” In re Sulfuric Acid Antitrust Litig., 235 F.R.D. 407, 417 (N.D. Ill. 2006), quoted with approval in Heriot at *11.
Conclusion
This case, much like the movie, shows just how much time and money can be wasted over a few garden variety secrets. The parties would have been much better off practicing what their movie preaches. Of course, if that were true, there would have been no lawsuit and these secrets of 502 would not have been revealed. At least, not in such an ironically attractive manner.
Posted by Ralph Losey 
The Fourth Circuit Court of Appeals has strengthened federal anti-hacker email privacy law by allowing a civil suit for punitive damages alone, even when there are no actual damages. 
This AOL account, like any online email service, is considered ”a facility through which an electronic communication service is provided” covered by the SCA. Leonard had somehow discovered Van Alstyne’s AOL password. We are not told how he did that, but he could have lawfully discovered it by looking on company computers. Regardless, Van Alstyne had never authorized Leonard to access her AOL account. Leonard used her AOL user-name and password to read her email. We do not know all of the nefarious motives behind this invasion of privacy, but certainly one motive was to try and find something that he could use against Van Alstyne to defend the sexual harassment claims. He found a few emails he liked, downloaded them, and provided them to his lawyer. The lawyer in turn surprised Van Alstyne with these emails during her deposition.
There are important lessons here for e-discovery lawyers too. The unauthorized access of a person’s private email account to discover and retrieve their email is a crime. Just because you know a person’s user name and password, does not give you the right to use it. This kind of self-help e-discovery is not only unethical, it is criminal. You must employ a request for production or subpoena. You cannot hack into their private email accounts or home computers any more than you can break into their house and steal papers.
The lesson to be learned here by attorneys is to always ask and be sure you understand where and how your client obtained email that they turn over to you. Especially in a heated case like this with allegations of sexual harassment, you need to be sure the emails were obtained legally. This is now a common problem in divorce cases. If your client has hacked into someone’s private email to get the evidence, they may well have committed a crime, as this case shows. They should be counseled accordingly. You cannot simply act like Sergeant Schultz in 
I am not suggesting that is what happened here. I do not know; but the facts stated in the opinion raise some disturbing questions. Clearly we should all exercise caution in the discovery of an adversaries’ email and other personal computer information. When it comes to private email we should forget the common proverb; we should carefully 
Judge Shira Scheindlin and I did a one-hour audio podcast last week on e-Discovery and Education. The focus of our talk was on the need of students and lawyers to master e-discovery and the resources available to help them to do that, including Judge Scheindlin’s new text books co-written with Professor Daniel Capra and The Sedona Conference, entitled 
Judge Shira Scheindlin used her lunch break in the middle of a jury trial to give this interview. I was honored to be asked by her to participate. We discussed our nearly identical views on the urgent need for improved e-discovery education and the crises of competence now faced by the legal profession. If you have any doubt about the crises, you have only to look at the steady stream of new judicial opinions demonstrating attorney negligence and refusal to cooperate on simple technical matters, all to the detriment of clients. The latest poster child case for this sad state of affaris comes out of my home court in Orlando in a well written opinion by Magistrate Judge Karla Spaulding. 
Judge Scheindlin said a lot of interesting things, including a quote that should be read by every litigation attorney in America who still thinks e-discovery is just a passing fad and can be ignored:
RALPH: Well I’m glad to hear your concern about that. I share that concern, as I am primarily doing defense work. Of course, in e-discovery it’s not so much defense as it is requester or producer, and the producing party if it’s a larger corporation, David and Goliath, can be behind the eight ball. But I do think that a large company is entitled to protection and that there is no need to try and find every piece of relevant email, but you should instead scale your search. We should be thinking in terms, as I know you do, of rolling productions in a series. Let’s start off – let’s see what we can get in an economical manner and produce that first, and then come back with secondary, more focused discovery. Because, you know, I don’t think you should have to spend a million dollars to find electronic evidence if the case is two million. I think that’s where you go immediately for protection and do not allow yourself to be extorted by the cost of a lawsuit.
KARL: First of all, Judge Scheindlin, why did you become interested in electronic discovery?
KARL: I just did a podcast about a week and a half ago with Judge Facciola, and he’s deeply concerned that lawyers need to learn something about electronic discovery if they’re litigators, or else they’re really risking malpractice. In fact, he gave a couple of examples of what he thought were pretty darn close to malpractice committed in his courtroom. Do you see that same type of need from your perspective on the bench?
JUDGE: Well I’ll just throw in a couple words here. I’m a little bit concerned about the risk of 
RALPH: Well, I’m teaching sanctions tomorrow in the class and we’re going to be talking about Zubulake. It’s great, I mean, it’s much more fun, and Judge, if you have an opportunity, I urge you to try it out. Compared to teaching CLEs, where they may or may not be paying attention, and they may be doing their Blackberries while you talk, the classes are much more focused and you have the opportunity to ask questions. So I do follow the Socratic method. I ask questions and the beautiful thing is they are starting to ask questions back. Last week before Spring break we were doing metadata and, you know, I had to say, “Okay, we’re going to have to move on now,” there were so many questions. I mean, it is something that I think the students understand that finally they’ve got an edge from what they’ve learned. They know they know more than their parents, and they know they know more than the law firms they’re going to be working with, and I think they get it and they see this opportunity. So there’s a real excitement there in teaching in law schools and the kids are very bright. A lot of them have some terrific backgrounds. So I do have a lot of hope for the future coming out of that.
KARL: And they got nothing. Not a single email back.
Shocking, but true! The country’s leading trial lawyers now admit that e-discovery is an “extraordinarily important … fact of life that is here to stay.” They understand that written evidence is no longer flat. It has digital depth wherein metadata, searchable indices, and other electronic mysteries reside.
Yes, Grandpa Simpson has stopped waving his clenched fist at cloud computing. He has opened his hand, apparently ready to shake with, not at, us geeky e-discovery types. Don’t get me wrong, the trial lawyers (especially the ones my age or older) are still mad as hell. Who can blame them? The world has changed way too fast and discovery is a mess. But at least the leaders of the trial bar appear ready to start a real dialogue, to go back to school, and start to figure it all out. 
These proposals by the College of Trial Lawyers for an expanded and much more active judiciary are truly radical. They might work, but only if the American public is willing to devote substantially more resources to the judiciary. Indeed, for this to work, the number of judges and support staff would probably have to be doubled and the pay increased. I do not think there is the political will for this yet in the United States, especially in the current economy. Since many of the proposals here hinge on such a fundamental change, I do not take them too seriously. I am not saying they are a bad idea, just that America is not prepared to fund them.
Electronic discovery has been called the “
You think you’ve got email problems, Jason R. Baron just received 200 million emails from the outgoing Bush administration! Jason is the Director of Litigation for the National Archives and Record Administration (“NARA”), the government agency responsible for maintaining and searching all of these emails, and more, in response to never-ending information requests. NARA keeps the permanent records of the U.S. government, including the emails and other records of the White House and Presidential Libraries. As difficult as it is to search for one relevant email in a universe of 200,000,000, the situation is getting worse all of the time. Jason expects the Obama administration, if it goes for two terms, to generate over a billion emails by 2017. These kinds of 
I say Jason Baron had time to provide a full overview of his ideas because it would take a day or more to flush out all of the details of his work on this subject. Typically, e-discovery CLEs include search as part of a curriculum, and, at best, you only hear Jason Baron as part of a panel with limited time. I know because I’ve been on two search panels with him. That is better than nothing, but not really adequate for a full airing of his views or mine. See eg. a webinar on March 19, 2009 where Jason is a panelist: 
Jason began his presentation with a story of his experience assisting trial attorneys in the Department of Justice on the tobacco litigation team. In the early 2000s, the team at DOJ worked with various agency counsel (including Jason representing NARA) on the task of responding to discovery requests from the tobacco industry in 
Most litigation lawyers today do not understand just how hard it is to search large data-sets. They think that when they request production of “all” relevant documents (and now ESI), that “all or substantially all” will in fact be retrieved by existing manual or automated search methods. This is a myth. The corollary of this myth is that the use of “keywords” alone in automated searches will reliably produce all or substantially all documents from a large document collection. Again, most litigators think this is true, but it is not. That is not just Jason’s opinion, or my opinion, it is what scientific, peer-reviewed research has shown to be true. 
Thus, for example, if your search only uncovered five documents, and you were lucky enough that all five were relevant, then you would have 100% Precision. There would be no False Positives. But in that circumstance, you would likely have attained a very low Recall rate. You may have found five relevant files, but left behind another five hundred. Thus, in that example, your Recall would be 5/505, or slightly less than one percent (.99%). That is the basic stuff of search analysis. The next instructional step after that, in my opinion, requires venturing into the world of sampling and thus is one of those things that requires a full day seminar, and (horrors) more math.
