e-Discovery Team

A common question these days between most lawyers discussing e-discovery is: What Keywords Did You Use? This is often followed by I’ll show you mine if you show me yours. Often this latter statement is made out of a bona fide spirit of cooperation, typically in cases where:

1. both sides had too much ESI to search manually;
2. they culled using simple keyword technology either because that is the only search they knew how to do, or they did not deem more advanced predictive coding technology to be appropriate for that case; and,
3. the attorneys knew how to cooperate to get discovery done without spending too much money.

In these cases attorneys freely exchange the final keywords they used. There is no wasted breath or valuable client dollars spilled over the question.

Only rarely would attorneys in this symmetrical position not only want to know the keywords finally chosen, but also the keywords, parametrics, Boolean logic, etc., tested and rejected along the way. If they asked for a list of all the keywords ever tested, the proper response is no, or in my case, no such list exists, and I can’t recall, but there were quite a few. They might want to ask whether you tried this, that, or the other keyword. That’s fair, and an expert searcher would probably say, yes, I tried all of those early on, and they were all rejected because (fill in the blank). Alternatively, they might say to one or more of the suggestions: No, I didn’t think of that one, but I’ll check on it later today. It’ll just take a minute to try it out, and then I’ll get back to you on that.

A Keyword Search Hiding Kimono Made of Work Product is a Kimono Made of Whole Cloth

But what about another scenario, the asymmetrical one? You know, where one side has tons of ESI (actually ESI is weightless, but this sounds good), and the other side that has virtually none, aside these days from a pesky Facebook page or two. In these cases the kind of cooperation described for symmetrical cases is often lacking. One side, typically the plaintiff, has nothing to disclose. The conversation is more like, you show me yours, but I can’t show you mine because, well, I don’t have one. So sad. All too often this conservation serves as a prelude to a real waste of client money fighting over the question.

The well-endowed defense counsel is often too shy to show theirs, keywords of course. So they hide theirs in a kimono made of a fabric called work product. This legal doctrine is designed to protect an attorney’s mental impressions, conclusions, opinions, or legal theories. It also protects from discovery documents and tangible things that are prepared in anticipation of litigation. Hickman v. Taylor, 329 U.S. 495 (1947); Rule 26(b)(3), FRCP. Therefore, you cannot send an interrogatory asking for the other side’s strategy to win the case; or more correctly stated, you can ask, but the attorney does not have to answer. 

Many lawyers have long considered the particular methods they used to find documents responsive to a request for production to be work product. It was, after all, their own thought processes and legal techniques that created the keywords. They object to disclosing the  keywords they used. They argue such disclosure would unfairly require them to disclose their theory of the case, their mental impressions of how to find relevant information.

This seems like a stretch to many attorneys, and judges, some of whom have rejected this argument outright. They do not think that any significant attorney ideas are revealed by something as mechanical as keywords, especially the final keywords used to cull a large dataset. They think keywords relate to the underlying facts of what documents are responsive to a document request, not mental impressions. They do not see any work product in keywords.

Still, many lawyers cling to this very broad interpretation of work product, especially when in asymmetrical litigation. In those cases defense counsel may respond to the question of what keywords did you use by saying something like: You cannot see my keywords, they are mine, all mine, and mine alone. No one may see my magic words. They are secret. They are protected by privilege. I would rather die than let you peek under my kimono. Well, ok, maybe the last phrase is not uttered too often, but the others essentially are. Passion by lawyers to protect their secrets can run high. This is usually displaced passion. It should be directed to protecting client confidences instead. Often lawyers grappling with e-discovery forget and confuse an attorney-client privilege with a work product privilege.

Attorney-client and work product are two completely different kinds of privilege. The AC privilege is owned by the client, not the lawyer. The lawyer has a strong ethical duty to protect the client’s AC privilege. This is dramatically contrasted with a work product privilege that is owned by the attorney and is given far less protection under the law. There is no ethical duty whatsoever for an attorney to keep his work product secret, except for the duty of competent representation. Often competence requires an attorney not to reveal his or her mental impressions of a case to opposing counsel. They reasonably construe the scope of the privilege and determine that:

1. disclosure would not be in their client’s best interests, and
2. the rules do not otherwise require them to share this particular aspect of their mind-set about the case.

But having said all of that, it is important to understand that attorneys often deem it to be in their client’s best interests to share some of their mental impressions of a case. Moreover, like it or not, the rules often require an attorney to make some disclosure of their mind-set, theories, etc., or material prepared for the case. So they do it. They move the case along. They do not get bogged down with a question of open or closed kimono, which is often just an ego-trip where a lawyer is over-valuing their own mental impressions at the client’s expense. Yes, kimono-closing motion play can be a very expensive process.

All actual trial lawyers, and not mere paper-pushers as we used to say, or now, maybe better said, mere electron-pushers, know full well that good lawyers share mental impressions with opposing counsel all of the time. Indeed, is that not what legal briefs are all about?

Some disclosure of work product is required for any attorney to comply with the rules of civil procedure, including the almighty Rule One, and especially the discovery rules. Discovery is built on the premise of cooperation, and that in turn requires some rudimentary sharing of mental impressions, such as what do you think is relevant, what documents do you want us to try to find to respond to this or that category in a Request For Production?

How could you possibly comply with Rule 26(f), for instance, without some select waiver of work product? Remember in subsection (2) it mandates attorneys to discuss the “nature and basis of their claims and defenses and the possibilities for promptly settling or resolving the case” and to develop a joint discovery plan. In subsection (3) the rules require lawyers to talk to each other and “state the parties’ views and proposals” on a topics A-F. Subsection (C) of Rule 26(f) in turn requires discussion of the views and proposals concerning “any issues about disclosure or discovery of electronically stored information…” All of these mandated discussions require disclosure of an attorney’s mental impressions, conclusions, opinions, and legal theories.

Trial lawyers have always disclosed some work product to each other to prepare for trail (of which discovery is a part), conduct trials, and settle cases. A lawyer can share his mental impressions with the other side, if he wants, and can do so without fear of opening the door of a complete waiver. Again, this happens all of the time, especially in any settlement discussions, where both sides will try to persuade the other of the strength of their case. They will explain why and how they will win and the other side will lose. The same kind of discussion is inherent in any proportionality issue.

Lawyers usually love to argue about their opinions, so why this recalcitrance about keywords? Could it be because they know or suspect that their keywords suck? Do they fear ridicule and reversal because they just dreamed up keywords without testing? Or worse, did they use bad keywords on purpose to try to hide the truth?

Bottom line: to represent a client’s best interests and comply with the Rules, a lawyer has to share mental impressions to a certain extent. If lawyers refuse to talk to each other, refuse to cooperate, all on some misguided notion that they have a right to remain to remain silent because of the work product doctrine, discovery will never get done. The case will go off track and may never be resolved on the merits. The hide-your-keywords under a kimono doctrine that seems to be in fashion among many e-discovery lawyers these days is misguided at best, and at worst, may be illegal.

Kimono closing lawyers, get over yourself and how valuable your mental impressions are. Tell the other side what your keywords are. Or are you hiding them because your keywords are so poor? Are you embarrassed by what you have to show? Then get a keyword search expert to help you out. Unlike predictive coding experts, there are plenty of power users and professional keyword searchers around.

Cases Supporting Disclosure of Keywords

Like it or not more and more judges are growing tired of obstructionism and expensive discovery side-shows. They are requiring lawyers to show their keywords, at least the ones used in final culling. They are compelling lawyers to open their kimonos. Consider the ruling in a recent trade-secret theft case in California that cites to the law of several other jurisdictions.

To the extent Plaintiff argues that disclosure of search terms would reveal privileged information, the Court rejects that argument. Such information is not subject to any work product protection because it goes to the underlying facts of what documents are responsive to Defendants’ document request, rather than the thought processes of Plaintiff’s counsel. See Romero v. Allstate Ins. Co., 271 F.R.D. 96, 109-10 (E.D. Pa. 2010) (finding that document production information, including search terms, did not fall under work product protection because such information related to facts) (citing Upjohn Co. v. United States, 449 U.S. 383, 395–96 (1981) (“Protection of the privilege extends only to communications and not to facts. The fact is one thing and a communication concerning that fact is entirely different.”)); see also Doe v. District of Columbia, 230 F.R.D. 47, 55-56 (D.D.C. 2005) (holding that Rule 26(b)(1) of the Federal Rules of Civil Procedure can be read to allow for discovery of document production policies and procedures and such information is not protected under the work product doctrine or attorney-client privilege). Moreover, Defendants’ substantial need for this information is apparent. See In re Enforcement of Subpoena Issued by F.D.I.C., 2011 WL 2559546, at *1 (N.D. Cal. June 28, 2011) (LaPorte, J.) (“Fact work product consists of factual material and is subject to a qualified protection that a showing of substantial need can overcome.”). There is simply no way to determine whether Plaintiff did an adequate search without production of the search terms used.

Formfactor, Inc. v. Micro-Probe, Inc., Case No. C-10-03095 PJH (JCS), 2012 WL 1575093, at *7 n.4 (N.D. Cal. May 3, 2012).

The holding in Formfactor was recently followed in the well-known Apple v Samsung case involving a third-party subpoena of Google. Apple Inc. v. Samsung Electronics Co. Ltd. The court compelled Google to disclose the keywords it used to respond to the subpoena and also disclose the names of the custodians whose computer records were searched. Their arguments that a third-party under Rule 45 did not have to make such disclosure were rejected. The court instead noted that discovery cooperation, including transparency of search methods, was required of anyone in litigation, both parties and non-parties.

Keyword Search Alone is Good Enough for Most Cases

Keywords are here to stay. Sure, it is old technology, but it is still an effective means of search. It should not be abandoned entirely. Predictive coding, by which I mean search using near-infinite-dimensional vector space probability analysis of all documents searched, is far more advanced than mere keyword search. But this kind of advanced-math search is hard to do correctly, and anyway, is not needed for all search projects.

Forgive the primitive image, but you do not need to use an elephant gun to kill a mouse. Since most cases today, even in federal court, involve less than $100,000 at issue, predictive coding is not needed in most suits. Keywords search alone, without including advanced analytics, is proportionally sound for most of these small value cases. Indeed, it is proportionally sound today for any case that does not involve high volumes of ESI or otherwise have complex search challenges.

Moreover, even in the big cases involving complex search problems, you would never use predictive coding search alone. That is about as silly as relying on random chance alone to train your predictive coding robots. You would use all kinds of search, what I call the multimodal approach. That includes keyword search using modern-day parametric Boolean features.

Anytime keywords are used to screen out files for review you should be prepared to disclose those keywords. I personally do not like to use keywords as an independent filter in a predictive coding process. But sometimes it happens, such as to limit the initial documents collected and thereafter searched with predictive coding. If that happens, and if the question is asked – What keywords did you use? – you should be prepared to answer. You should not try to hide that under your kimono. More and more courts consider that work product argument to be made of whole cloth.

Disclosure in Predictive Coding Search

Assuming that a predictive coding process is done properly, and keywords are not used to select what documents get searched, then the question of what keywords you used as part of the multimodal search becomes moot. A predictive coding CAR is not driven by keywords. It is driven by infinite dimensional probability math. Keywords are not in the black box anymore, hyper-dimensions are.

The keywords used in a predictive coding project are just one of many types of search used to find the documents that fuel the predictive coding engine. They help an expert searcher find documents to train the machine in an active learning process. The training documents are what cull out by probability ranking, not particular keywords. A predictive coding CAR runs on whole-documents, usually thousands of documents, not a few keywords. Should these training documents be disclosed, or not, is the issue in predictive coding search projects. Despite what some may say, there is no one set-answer to that question that applies to all cases. Da Silva Moore does not purport to provide the only possible answer for all cases. None of the orders in the field do that. The judges involved know better.

For now it is still an open question as to how far the work product doctrine applies to predictive coding search processes. It may not apply at all. You may have to share your mental impressions, your basic search plan. You may have to disclose what you did and why. You may have to explain what predictive coding search methods you used, but not disclose your entire training set, not disclose your irrelevant documents. Even if courts hold to the contrary that your search methods are protected by work product, you may want to share the methods  anyway to save time and client money. It may be in your client’s best interests to explain what you did. You will not lose any competitive advantage by doing so.

Sell Proportionality by Explaining How Good Your Search Is

I for one enjoy sharing how I did a complex, advanced analytics, iterative, multimodal search project. I do not enjoy this kind of show and tell because I like the sound of my own voice (well, ok that may be part of it), but because it helps persuade the requesting party that they are getting the most bang possible for the buck. It supports my proportionality argument. The abilities of predictive coding are truly mind-blowing. When done right, with good software (and, the truth is, most of the software out there is not good, is not bona fide active machine learning, and is not used properly), it can accomplish miracles. At least from our three-dimensional, keyword conditioned perspective, the search results seem miraculous.

If the requesting party, or judge, are still not convinced, and insist on an explanation on how the software black box really works, then you can bring an information scientist familiar with the software you used to explain it all. They like to talk almost as much as lawyers. They will go on and on. The other side may be sorry they asked, and most judges will be sorry they allowed a Daubert hearing for discovery. To me it is fascinating to hear how near-infinite-dimensional vector space probability analysis really works. So fascinating, in fact, that I have lined up a guest blog by Jason R. Baron and Jesse B. Freeman, a wiz-kid math genius he found, that introduces multidimensional support vector machines to lawyers. It is coming soon.

Predictive coding, when done right, is the best thing that ever happened to a requesting party in a complex ESI case. So if you are proud of what you have got, open up your predictive coding kimono for the other side to see. They should be impressed by the advanced methods, by the not just reasonable, but stellar, multidimensional efforts. After all, did they use hyper-dimensional based probability algorithms in their search? Did they used an iterative multimodal approach with both statistical quality control and quality assurance methods. Your best-practices in search justify a low-cost proportional approach.

You Should Not Have to Share Irrelevant Documents Unless and Until a Showing is Made that Your Predictive Coding Search Efforts Were Unreasonable

Does this mean that you must share all of the actual documents used in the machine training? Absolutely not. I have been talking about sharing process, not documents. The attorney work product has nothing to do with sharing, or not sharing, the client’s documents. We are not talking about documents prepared in connection with litigation. We are talking about documents prepared in the ordinary course of business, of life. These documents have their own protection from disclosure, for instance, the privilege held  by the client, not the lawyer, the attorney-client privilege. A lawyer cannot waive that. Only a client can. There are many other protections that apply to ESI, such as trade-secrets, or personal privacy laws. But the most central protection is that built into the rules, where only relevant information must be disclosed. Irrelevant ESI is not discoverable.

That means that unless there is some dispute as to the adequacy of the search efforts, only the relevant documents in a training set used in predictive coding need be produced. There is no legal basis in the initial stage to require production of all documents, including irrelevant documents.

Still, the client may choose to do so in certain cases, with certain sets of documents, and with certain protections in place. But that has nothing to do with work product analysis. It typically has to do with building confidence and trust between litigation parties and avoiding expensive disputes. It has to do with concerns that a reasonable effort to find relevant ESI is being made. It has to do with mitigating risk by cooperation and participation. It has to do with avoiding motions for sanctions, motions predicated upon allegations of an unreasonable search effort.

If a requesting party is kept in the dark, and the producing party does not reveal their search kung fu, and if the requesting party later makes a good cause showing that the producing party’s  search effort was unreasonable, then you are facing possible sanctions and the dreaded redo. All a requesting party will have to do to show good cause is provide proof that the producing party missed certain key hot documents. Then in a sanctions motion the reasonability of the producing party’s search efforts becomes relevant. This would usually happen in the context of a post-production motion.

Then, and only then, would there be legal authority to require production of the irrelevant documents used in the training sets. That is because these formerly irrelevant documents would then become relevant. They would then be relevant to the issue of reasonable efforts. They would then be discoverable, assuming the producing party claims its efforts were reasonable.

I contend that this magical transformation from irrelevant to relevant requires a good cause showing. There must at least be a justiciable issue of fact that the search efforts were unreasonable before the irrelevant documents in a training set become discoverable. Based on my CLE efforts, and listening to judges from around the country, I am confident that the judges, when they hear these arguments, will agree with this logic. They will, in most cases, only require disclosure of process, and not also require disclosure of irrelevant documents.

A recent case out of New York confirms this belief. Hinterberger v. Catholic Health Systems, Inc., 1:08-cv-00380-WMS-LGF, U.S. Dist. Crt., W. District of N.Y., Order dated 5/21/13. In this complicated case with tons of ESI the defendant finally gave up on using keyword search alone to try to find which of its millions of emails were likely relevant and needed to be reviewed for possible production. After Magistrate Judge Leslie Foshio pointed out the Da Silva Moore case to the parties, the defendant decided to drive a more advanced CAR, one that included a predictive coding search engine. They hoped that would allow them to accomplish their search task within budget.

Before the predictive coding work began, however, plaintiff demanded that the exact same Da Silva Moore search protocol be followed, and they be allowed to participate in the seed set generation, including a quick peek of all unprivileged training documents. Defendant objected, as well it should. No good cause had been shown to force such disclosure of irrelevant documents. Defendant argued that plaintiff’s motion was premature, and plaintiff misread the intent of Judge Peck’s order in Da Silva Moore. Defendant asserted that plaintiffs had no right to access to Defendants’ seed-set documents at this time. Judge Foshio essentially agreed with defendant, and denied plaintiff’s motion to compel, but did so largely upon defense counsel’s representation that they would cooperate with plaintiff. Judge Foshio got it right, so did the defendant: cooperation is the key, not disclosure of irrelevant documents or particular protocols agreed to in other cases

Conclusion

As the predictive coding landscape matures, and as counsel learn to cooperate and make disclosure of methods used, there will be no need to build trust by disclosing irrelevant documents in training sets. Judges will not have to go there. Counsel will only need to disclose the multimodal search processes used, including details of the predictive coding methods. I do this all the time, although in much greater detail than required. See eg: summary of CAR and list of over thirty articles on predictive coding theory and methods; Predictive Coding Narrative: Searching for Relevance in the Ashes of Enron (detailed description of a multimodal search of 699,082 ENRON documents); Borg Challenge  (description of same search using a semi-automated monomodal method). Worst case scenario, counsel may have to explain the black boxes of the predictive coding software they used. All they need do for that is pull a science rabbit out of their hat who will explain hyper-dimensional probability vectors, regression analysis and the like. There are experts for that too. Every good software company has at least one. I know several.

In simple keyword search cases a similar logic will prevail. Counsel will have to disclose the keywords used in final culling, but not the documents deemed irrelevant by keywords or second-pass relevance attorney review teams. The instruction books prepared for these human review teams, if any, will also be kept secret, but not the general methods used, such as the quality controls. Case specific reviewer instruction manuals are documents prepared for litigation. That is classic work product. Moreover, they typically include far more information than keyword disclosure, or search method disclosure. They often explain an attorney’s strategies and theories of a case. Here a clear line still exists to protect a lawyer’s work product. Yes, the kimono still lives, so too does the concept of relevance.