Fears and Loathing (and Pain) in Seattle: a Case Lesson in How NOT to Implement a Litigation Hold and Search for Email – Part Two

April 20, 2014

Ralph_Fear_Loathing_VegasThis is part two of a two-part blog, Fears and Loathing (and Pain) in Seattle. Part one is found here. This is not really a Hunter S. Thompson worthy story, but it is Seattle after all. And the name of the law firm involved here just begs for the analogy.

Before you begin reading part two of this sanctions saga, take a look at the poll results from Part One. If you have not already done so, cast your vote. I promise you it is all anonymous. The last time I checked it was about evenly split on both questions, but not enough readers have voted. So, please join in now.

Seattle Court’s Finding of Bad Faith

Seattle-skylineJudge Robart in Knickerbocker v Corinthian Colleges found that there was clear and convincing evidence the defendant, and their counsel, the Seattle law firm of Payne & Fears, had refused to participate forthrightly in the discovery process and that this refusal constitutes or is tantamount to bad faith. He found that they had delayed resolution of Plaintiffs’ claims, expended both the court’s and Plaintiffs’ limited resources on matters ancillary to the merits, and threatened to interfere with the rightful decision of this case.

Judge Robart did not think too much of defendants argument against all sanctions because the email was eventually found and produced. Here is his well written response to this argument (citations removed and emphasis added):

Corinthian argues that, at least with respect to emails, no spoliation has occurred because Corinthian has since recovered and produced all responsive employee emails from the backup tapes. The court notes that this argument contravenes what appears to have been Corinthian’s previous position that the backup tapes were not reasonably accessible. Corinthian’s characterization of the backup tapes has shifted with the winds throughout this litigation, adopting whatever posture is most convenient in the immediate context. (Compare Ruiz Decl. ¶ 17 (“I explained that it was unreasonable and impractical to search them . . . .”) with 12/12/13 Trans. (“It would be perfect. It would be one day, $1,000.”) (Mr. Brown testifying).)

Corinthian cannot have it both ways. If the information on the backup tapes was unavailable within the meaning of Federal Rule of Civil Procedure 26(b)(2)(B) such that Corinthian was not required to recover it, then the Plaintiffs’ deleted emails were, in fact, spoliated evidence. If, as Corinthian’s counsel represented at oral argument, the information on the backup tapes was accessible, then Corinthian had little basis for refusing to search the backup tapes under the parties’ Stipulated Order, no basis for filing a verification with the court affirming that it had searched “all available electronic sources”, and appears to have assumed a misleading stance with Plaintiffs from the beginning.

Corinthian counters that it encountered substantial technical difficulties and costs in retrieving the emails from the backup tapes. But any obstacles Corinthian faced in recovering the emails were the direct result of Corinthian’s inadequate discovery search, deletion of evidence, and lack of candor with both Plaintiffs and with the court. Such obstacles do not transform bad faith into good.

The judge basically accuses the defendant’s law firm, and thus the defendant itself, of not being straight with the court about plaintiffs’ emails and the defendant’s backup tapes.

Throughout the course of the litigation, Corinthian did not once provide a straight-forward explanation of the process and cost of extracting information from the tapes.

Here is how Judge Robart wrapped it all up.

In sum, the court finds, by clear and convincing evidence, that Corinthian’s and Corinthian’s counsel’s lackluster search for documents, failure to implement a litigation hold, deletion of evidence, refusal to cooperate with Plaintiffs in the discovery process (particularly as evidenced by its withholding of information regarding both the backup tapes and its interpretation of the parties’ Stipulated Order), reliance on a recklessly false declaration, shifting litigation positions, and inaccurate representations to the court constitute bad faith or conduct tantamount to bad faith.

Bad Faith Does Not Necessarily Mean Dispositive Sanctions

ZeroEven though the court found bad faith, no dispositive sanctions were granted. The adverse inference instruction the plaintiffs had requested was also denied. These harsh sanctions were denied because plaintiffs provided, as the judge put it – zero evidence that any evidence of significance to the case was not produced. They only offered conjecture. As Judge Robart noted: produced documents cannot form the basis for a spoliation instruction. 

I am kind of surprised by plaintiffs’ failure to offer up some evidence that relevant evidence was not produced. You would think the plaintiffs would be able to come up with something concerning their own email.

Based on this record, the wise Judge Robart, although obviously upset with defense counsel, wanted the racial discrimination case to be tried on the merits. Besides, perhaps he knew that the emails that were produced were good enough for the plaintiffs to prove their case. Or maybe it was the opposite. The plaintiffs could have had a very weak case. We cannot tell from this opinion. We can only tell that the judge wanted the case tried on the merits, despite the bad faith e-discovery by defendant.

The judge got his message across on his intolerance of bad faith by imposition of the $10,000 fine against the Payne & Fears law firm, and the $25,000 fine against defendant. He also awarded the plaintiff’s reasonable attorney fees and costs incurred in connection with the sanctions motions and duplicative discovery related thereto. Justice was done.

Lessons learned from Knickerbocker

no-BS-signSeveral lessons can be learned from this case. For one thing, there is the trial lawyers lesson. Be careful how you answer questions posed to you by the judge. Be sure you remember these magic words: I don’t know. Restrain the urge to speculate or BS. Just keep to the facts you do know. Ask to get back to the judge on important questions with a supplemental brief or something. This case clearly shows why that is important.

The obvious primary e-discovery lesson is to always implement a litigation hold. The hold should be in writing and there should be follow up by conversations with the custodians on hold and with IT. Auto-deletions programs should be suspended, and, if the size of the case warrants it under proportionality analysis, preservation of ESI by bulk IT collection should be done. In smaller cases, collection may not be required and preservation-in-place may be adequate. There is no one-size fits all in e-discovery. Although there are plenty of plaintiff’s experts out there ready to tell a court every case should be treated like the Taj Mahal. They should not. Efforts should be scaled proportionally. See eg: My Basic Plan for Document Reviews: The “Bottom Line Driven” Approach – Part Two (e-Discovery Team, 10/9/13)

Golden_ratio_line

The final lesson here pertains to backup tape restoration and search. It is never as easy as you think. Indeed, the tape or tapes may have deteriorated to the point that restoration is impossible. You never know until you try. Once you restore, finding the relevant ESI can also be a challenge. Do not ever sat easy peasy when it comes to backup tapes.

This opinion does not really go into the defendant’s search efforts here, merely stating that about 3,000 relevant emails were found from a search of the emails of all employees at one location. That still seems like a low production. But I suspect the “search” consisted of running keyword terms agreed upon with plaintiff’s counsel, and then manual review of the emails that contained the terms. If they were relevant, they were part of the 3,000 produced. If not, then of course they were not produced. You do not produce irrelevant email just because they happen to have an agreed upon search term. I suspect this kind of procedure was followed here, and if so, the plaintiffs cannot complain about the search efforts made by defense counsel. They were following the parties agreed upon protocol.

We really do not know what that protocol was, but if, as I suspect, it was a keyword search protocol, then, questions of estoppel aside, the issue of whether it was a reasonable effort would depend on whether the common sense dictates for keyword search contained in Judge Peck’s Gross Construction opinion were followed. William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134 (S.D.N.Y. 2009). Were the witnesses interviewed as to the language used? Were various keywords tested? Was the underlying data studied? The key documents? Or was it all done in the blind, like a child’s game of GO FISHChild’s Game of “Go Fish” is a Poor Model for e-Discovery Search (e-Discovery Team blog, 10/4/09).

Tested Keyword Search is Adequate for Most Cases

fear-loathingKeyword search alone, when done according to the standards set forth in Gross Construction, is a fair and adequate effort in most employment discrimination cases like the one in Knickerbocker v Corinthian CollegesMost employment cases are not really that complicated. For that reason the key documents needed to try most of these cases are not that difficult to find. Keyword search can and does work in the average case to meet the requirements of both Rule 26(g) and Rule 1 (just, speedy and inexpensive). It apparently worked just fine in Knickerbocker too, that is, after defense counsel stopped their Hunter S. Thompson routines and started playing it straight

There are some exceptional employment cases where keywords are inadequate. It depends on the case and the type of ESI, and the importance of the ESI to the case, and volume of ESI. But for most employment law cases the tested keyword search method of Gross Construction is reasonable and proportional. More sophisticated search methods, such as my favorite, predictive coding, may be needed in larger, more complex cases in other fields of law, as well as in some class action employment cases. But tested keywords work just fine for the vast majority of small cases that now flood our court system.

Most of these small cases in federal court are employment law cases. It seems like everyone has a beef these days. You would not believe the kind of frivolous cases that we see every day in my firm. Plaintiff’s counsel are not being selective. Many seem unable to overcome the natural trial lawyer tendency to be overconfident, unable to objectively predict the likely outcome of a potential client’s case. See: Lawyers as Legal-Fortune Tellers, (e-discovery Team, 3/30/14); Goodman-Delahunty, Granhag, Hartwig, Loftus, Insightful or Wishful: Lawyers’ Ability to Predict Case Outcomes, (Psychology, Public Policy, and Law, 2010, Vol. 16, No. 2, 133–157).

This limit of predictive coding to larger, more difficult cases will probably change in the future. The ever growing volume and types of ESI may demand the use of predictive coding in more and more cases. That should be made easier as the software costs of using predictive coding comes down even further. (For instance, my firm just closed a deal with Kroll Ontrack that lowers the costs for our clients even further. Look for press releases on this soon.) In the future predictive coding will expand to many more types and sizes of cases, but for now, predictive coding remains the exception in e-discovery, not the rule.

If your life revolves around discovery in the big cases, the complex cases with tons of ESI (actually, its weightless you know), then you should be using predictive coding all of the time. But for the vast majority of lawyers, dealing with the vast majority of relatively simple cases, it is not needed yet. You might as well hunt mosquitos with an elephant gun. Keyword search, done right, still works fine for the mosquito cases. Do not misunderstand me, mosquito bites can still hurt, especially if you get hit by too many of these blood suckers. You have to defend your company, but bad faith attempts to avoid discovery are never the way to go. Knickerbocker shows that.

Conclusion

Be straight with your judges. Always tell the truth. Talk about proportionality. They get it. The judges will protect you from the disproportionate use of e-discovery as an extortion tactic. We all know it still goes on. Has been for a long time as my parting string cite below reminds us. Both responding and requesting parties have to conduct discovery in good faith. When they do not, there are plenty of good judges around like James L. Robart to stop the abuse.

____________

Discovery abuse as a weapon. See, e.g.:

  • Advisory Committee Note to the 1983 Amendment of the Federal Rules of Civil Procedure creating Rule 26(g) (“Thus the spirit of the rules is violated when advocates attempt to use discovery tools as tactical weapons rather than to expose the facts and illuminate the issues by overuse of discovery or unnecessary use of defensive weapons or evasive responses.”)
  •  Branhaven LLC v. Beeftek, Inc., _F.R.D._, 2013 WL 388429 (D. Md. Jan. 4, 2013) (Rule 26(g) enforced and counsel sanctioned for reckless disregard of their discovery duties.) The Increasing Importance of Rule 26(g) to Control e-Discovery Abuses (e-Discovery Team, 2/24/13).
  • Judge Refers Defendant’s e-Discovery Abuse to U.S. Attorney for Criminal Prosecution of the Company and Four of Its Top Officers (e-Discovery Team, 4/10/11); Philips Electronics N.A. Corp. v. BC Technical, 2011 WL 677462 at *2 (D.Utah, Feb. 16, 2011).
  • Discovery As Abuse, (e-Discovery Team, 1/18/11); Discovery As Abuse, 69 B.U. L. REV. 635 (1989).
  • Kipperman v. Onex Corp., 2009 WL 1473708 (N.D.Ga., 2009) (“The court regards the instant case as a textbook case of discovery abuse.”)
  • Qualcomm Inc. v. Broadcom Corp., No. 05-CV-1958-B(BLM) Doc. 593 (S.D. Cal. Aug. 6, 2007) (Clear and convincing evidence that Qualcomm['s] counsel participated in an organized program of litigation misconduct and concealment throughout discovery, trial, and post-trial)
  • Malautea v. Suzuki Motor Co., Ltd., 987 F.2d 1536, 1542 (11th Cir.1993) (Fed.R.Civ.P. 26(g) was “designed to curb discovery abuse by explicitly encouraging the imposition of sanctions.”)
  • Bondi v. Capital & Fin. Asset Mgmt. S.A., 535 F.3d 87, 97 (2d Cir. 2008) (”This Court . . . has taken note of the pressures upon corporate defendants to settle securities fraud ‘strike suits’ when those settlements are driven, not by the merits of plaintiffs’ claims, but by defendants’ fears of potentially astronomical attorneys’ fees arising from lengthy discovery.”)
  • Spielman v. Merrill Lynch, Pierce, Fenner & Smith, Inc., 332 F.3d 116, 122-23 (2d Cir. 2003) (“The PSLRA afforded district courts the opportunity in the early stages of litigation to make an initial assessment of the legal sufficiency of any claims before defendants were forced to incur considerable legal fees or, worse, settle claims regardless of their merit in order to avoid the risk of expensive, protracted securities litigation.”)
  • Lander v. Hartford Life & Annuity Ins. Co., 251 F.3d 101, 107 (2d Cir. 2001) (“Because of the expense of defending such suits, issuers were often forced to settle, regardless of the merits of the action. PSLRA addressed these concerns by instituting . . . a mandatory stay of discovery so that district courts could first determine the legal sufficiency of the claims in all securities class actions.” (citations omitted))
  • Kassover v. UBS A.G., 08 Civ. 2753, 2008 WL 5395942 at *3 (S.D.N.Y. Dec. 19, 2008) (“PSLRA’s discovery stay provision was promulgated to prevent conduct such as: (a) filing frivolous securities fraud claims, with an expectation that the high cost of responding to discovery demands will coerce defendants to settle; and (b) embarking on a ‘fishing expedition’ or ‘abusive strike suit’ litigation.”)

Fears and Loathing (and Pain) in Seattle: a Case Lesson in How NOT to Preserve and Produce Email – Part One

April 13, 2014

Fear_Loathing_SeattleA recent case in Seattle provides a text-book example of how not to do e-discovery. It concludes with a sanctions order against the defendant, and the defendant’s law firm, Payne & Fears LLP. The law firm was fined $10,000, payable to the court, due to the conduct of two of its attorneys. The defendant, Corinthian Colleges, was fined another $25,000. Knickerbocker v Corinthian Colleges, Case No. C12-1142JLR, (WDWA, April 7, 2014).

How does a sanctions disaster like this come to pass? And in such a laid back city like Seattle? The court awarded sanctions because of a failure to preserve and a subsequent delay in producing evidence. Note that I did not say sanctions for a loss of evidence, only delay. The ESI at issue in the sanctions motions was the email of three of defendant’s former employees. They were the plaintiffs who were now suing Corinthian for alleged racial discrimination, harassment, and retaliation. Corinthian’s attorneys eventually found and produced the email from back-up tapes, but it was an ordeal to get there. The court got the distinct impression that the attorneys involved for the employer were not playing straight, that they were attempting to hide the ball.

Fear_Loathing_VegasDigging a little deeper into this 27-page Order, which is replete with facts, as all good sanctions orders are, we see a series of bad decisions. The decisions were all made by the attorneys for the defense. It would take me another 27-pages to review them all in detail, so I will just examine the segments that seem to me to have the most instructional value. (It is still going to be a two-part blog, even without these details!)

After you hear the story, and hopefully also read the opinion itself, you be the judge as to whether these bad decisions were just incompetence on the part of these attorneys, or actual bad faith. I will include a poll at the end of each segment where you can anonymously vote. The judge clearly thought it was bad faith by Pain & Fears’ attorneys. But, who knows for sure (aside from the attorneys themselves). It is often hard to tell the difference between dishonesty and incompetence, especially if all you know about the case is what you read in one court order.

Judge’s Quiz of Defense Counsel Uncovers a Complete Failure to Impose a Litigation Hold

trial_sceneThe first bad mistake made in this case was the defendant’s failure to issue a litigation hold. The specific facts surrounding this failure are interesting, so too are they way they came out. Maybe they tell a story of stupidity, or maybe intent to hide? Again, you be the judge. Personally, on this first error at least, I am inclined to think it was just a lack of knowledge and understanding. But I readily admit I could be wrong. Maybe they did not issue a hold because they wanted incriminating email to be destroyed. Naturally, that is what the plaintiffs’ alleged.

The facts of the no-holds bar (not a typo, think about it) were clarified during an evidentiary hearing on the plaintiff’s first motion for sanctions. Most of the clarification was attained by the judge’s questioning of the Payne & Fears attorneys themselves, and not the witnesses they had brought with them.

When a district court judge decides to quiz legal counsel about something he is curious about, you had better respond fully and truthfully. This is the same judge who is going to decide whether to sanction your client for misconduct. The compulsion to speak is especially strong in a situation like this where the other side is urging a sanction against your client for hiding the truth. Do you want to dig your hole even deeper to be buried in? No. The Payne & Fear attorney at the hearing, Jeffrey Brown, had no choice but to answer the judge as best he could.

Judge-RobartThe District Court Judge, James L. Robart, a wise and sage judge if there ever was one, cut to the chase and asked Jeffrey about the litigation holds. No doubt Jeffrey was nervous when he heard the question directed to him. He was looking up at Judge Robart in black robe several feet above him on his bench. I am pretty sure the Judge was not smiling like we see him in this photo. Jeffrey knew that the Judge would not like the answer he was about to give. He was right about that. Here is Judge Robart’s account of this exchange from the opinion:

At the hearing, Jeffrey Brown, counsel for Corinthian, admitted that, although Corinthian had issued litigation holds in previous litigations, Corinthian did not issue a litigation hold with respect to this case. (12/12/13 Trans. at 4-5.) Mr. Brown represented that, instead of issuing a company-wide notice, Corinthian had hand-selected certain employees and requested that they retrieve and retain relevant documents. (Id. at 5-7, 12, 18.)

Note the use of the word “represented” in Judge Robart’s account of Jeffrey’s response. It is a term of art, a kind of word you use when describing fraud. This is still early in the opinion, but experienced case law readers knows this is a set up word for things to come.

As we will see later when discussing other e-discovery blunders in this case, Mr. Brown quickly became a favorite target of Judge Robart’s questioning. His opinion is filled with quotes from poor Jeffrey. No doubt he acquired a major headache in that courtroom.

Oscar_Pistorius-Trial_headache

Here is Judge Robart’s summary of the defendant’s preservation mishaps (emphasis added and numerous citations to the record omitted):

Corinthian has issued litigation holds in previous actions. Nonetheless, for this case, Corinthian only requested that a subset of employees, whom it deemed to be “key” witnesses, search for and save relevant documents. Testimony by some of these “key” witnesses, however, casts doubt on Corinthian’s claim that these employees in fact performed any—let alone a thorough—search for relevant documents. Specifically, Ms. Austin and Ms. Paulino testified that they did not search for documents relevant to the litigation, and Ms. Givens and Ms. Phillips testified that they did not recall searching for documents.5 Such self-selection of a limited pool of discovery materials, combined with doubt as to what searches, if any, were performed of this pool of materials, gives the court no confidence in the quality of Corinthian’s discovery production. Yet, due to the lack of a litigation hold, it is not clear that the current additional discovery period, instituted 18 months after Plaintiffs filed suit, can remedy this deficiency.

So much for the corporation’s litigation hold procedures in this case. Turns out that the deposition testimony contradicted the representations made by Mr. Brown to Judge Robart as to his hand-selection of certain employees and request that they retrieve and retain relevant documents. These same witnesses testified that no one told them to search for anything, and they had not made a search. This suggests that the representations made to Judge Robart were not accurate. In fact, they seem down right misleading.

Not good. Not good at all. Perhaps now you understand the fines against the attorneys. But wait, there is more. An even bigger mistake was allowing all of plaintiff’s emails to be deleted from the corporation’s Exchange server after the case started. No doubt the judge was beginning to believe the plaintiffs’ allegations that the employer allowed the plaintiffs’ emails to be destroyed on purpose because they knew the content would harm the defense. But before we move on. What do you think? Did the employer not institute a litigation hold on purpose so that they could get away with destroying incriminating evidence, or is this just another case of lawyer incompetence?

More Drama Concerning the Defendant’s Destruction of Email

"You can’t handle the truth." -- Jack Nicholson in “A Few Good Men” (1989)The facts on this alleged destruction were cloudy and contradictory at the time of the first hearing. So, once again, what did Judge Robart do? That’s right, he turned to defense counsel and asked him if it was true, had his company deleted all of the plaintiffs’ email as plaintiffs’ allege. Here is Judge Robart’s later findings on this issue, which, once again, relies extensively on the responses of Mr. Brown to his impromptu questions:

First Corinthian’s counsel, Mr. Brown, conceded that Plaintiffs’ email boxes were in fact deleted pursuant to this practice. (“With respect to the plaintiffs’ email boxes, no, your Honor. Those emails exist on the backup tapes, but those email boxes were deleted per the policy that Mr. Banash explained to you.”) Mr. Brown also conceded that the deletion of Plaintiffs’ emails occurred after Corinthian received Plaintiffs’ EEOC notices:

If you put an order in that says delete the mailbox in 30 days, should somebody have spotted the EEOC charges and made the connection and gotten around and suspended that? That’s something I can’t argue. That is something that we have looked at. Yeah, we have to find a way to fix that system, your Honor. I cannot sit here and tell you that is the best way to do things.  

Mr. Brown also explained that no other document destruction program was in place at Corinthian. Although Plaintiffs pointed to a Corinthian document purporting to establish a six-month automatic email deletion policy, the policy was apparently never implemented. Plaintiffs claim that Mr. Ruiz represented that the reason Corinthian had produced so few emails was due to the alleged six-month deletion policy; Corinthian disputes that Mr. Ruiz ever made such a representation.  

Another big mistake was conceded by Mr. Brown. The plaintiff’s email was deleted after Corinthian received Plaintiffs’ EEOC notices. That means after a duty to preserve was triggered. He has just admitted a breach of duty. He has admitted spoliation. But he has a fall back argument, only that argument puts the credibility of other Payne & Fear attorneys at issue.

Mr. Brown was now arguing that the destruction of the email after notice was a harmless breach of duty, because, after all, there is a complete backup copy of all of the plaintiffs’ email. Did Mr. Brown forget that other attorneys in his firm had previously said they could not produce plaintiffs’ emails because the employer did not have them? They claimed that the emails were all destroyed in the normal course, and so the destruction was protected from sanctions by Rule 37(e). After the judge put Mr. Brown on the spot, and grilled him on his poor job of preservation, Mr. Brown responded by saying how easy it will be to just produce the emails off of the backup tapes. That reminds me of one of my favorite deposition questions: “Were you lying then, or are you lying now?

chief_judge_james_robartJudge James Robart, who has been a lawyer since he graduated from Georgetown in 1973, and a District Court Judge since his nomination by President Bush in 2004, knows a fair amount about IT. He was a member of the Ninth Circuit IT Committee from  2007 to 2009, and chief judge in 2008. I suspect Judge Robart knew a lot more about technology than any of the attorneys in this case, although they apparently did not know that.

Judge Robart analyzed the different things that he was being told by the attorneys for the defendant and reached this considered opinion:

Corinthian’s attempt to influence (if not misdirect) the court with such unsubstantiated information falls below acceptable standards of professional conduct.

In case you did not know it, that’s polite judge-speak for much stronger thoughts and condemnations, including my favorite deposition question.

Back to defense counsel’s fall back argument, that no harm was done because of the backup tapes, Judge Robart deals with this position in his sanction’s Order by again relying on counsel’s own words:

Specifically, Mr. Brown averred that:

First of all, there are backup tapes that have every single email that has been referred to on them. Every single day that Corinthian has operated in Bremerton there are backup tapes with those e-mails. Everything we are talking about in this motion has been preserved and is available.

The problem with this position, as the judge knew full well from study of the record, is that defendant never searched these tapes for the email. They never searched, even though another Pain & Fears lawyer had previously filed a certification with the court stating that:

Corinthian, at its own expense, conducted a full and complete search for all documents responsive to Plaintiffs’ Requests for Production Nos. 1, 2, 3, 4, 5, and 27 (subject to any and all objections and limitations previously agreed to by the parties) on all available electronic sources and/or servers . . . .

Judge Robart later found that this verification was, in the judge’s words: incorrect in that Corinthian’s backup tapes were not searched, despite the fact that Plaintiffs had not agreed that the backup tapes were not “available.” 

Obviously the lawyers were getting a little too cute with the use of the word available to try to justify their hiding the fact that they had these emails on electronic sources, namely backup tapes, but did not search them.

cant_handle_truthThis kind of squirmy behavior never goes over well with a judge, any judge, but especially not a U.S. District Court Judge like Judge Robart. He had long experience with complex litigation in Seattle, coupled with recent experience with IT. Believe me, a judge like that can handle the truth, the whole truth, and nothing but the truth. So that is what you had better give him, and you had better give it to him straight.

Here are Judge Robart’s words, where he once again quotes the words of Mr. Brown. Note how he begins each paragraph by invoking his name. (Again I am deleting the many citations to the record.)

Mr. Brown took the position that the Stipulated Order’s requirement for a “full and complete search” that “shall include documents on backup servers” did not extend to documents kept on backup tapes, because “[t]he tape is an entirely different ballgame from the servers.” (testimony by Mr. Banash confirming that Corinthian has both backup tapes and backup servers).) Plaintiffs’ counsel, on the other hand, claimed that they were not aware that Corinthian intended to draw a “distinction between backup servers and backup tapes,” and that they understood the Stipulated Order to refer “generally to backup media.” Similarly, with respect to the Verification of Compliance, which affirmed that Corinthian had searched “all available electronic sources and/or servers,” Plaintiffs maintained that they had not agreed that the backup tapes were not “available.”

Mr. Brown emphasized multiple times that accessing the information on the backup tapes would solve the spoliation problem facing the court: (paras added for ease of reading)

MR. BROWN: And the answer is lying right under our noses. We brought it up in April. We can go get those tapes. If there is something supposedly in Michelle Paulino’s mailbox, we can look at it. . . . We can do that. A thousand dollars a day and it is here and we are done.

THE COURT: Is it your position, sir, that that is not an intentional deletion of information once you are on notice of litigation?

MR. BROWN: Your Honor, number one, that is my position, because the emails exist on the backup tapes. And we can get them.  . . .  I know those things are still there. I can tell you, it is about a thousand dollars per day of recovery time. It can be done. It is sitting there.

Mr. Brown represented to Judge Robart during the hearing that the cost of retrieval of plaintiffs’ email from backup tapes would not be as expensive as Judge Robart feared. I wonder how Mr. Brown knew that, or thought he knew that? I suspect he just got carried away in trying to defend his client. Anyway, here is more Q&A between Mr. Brown and Judge Robart on the topic. You be the judge.

THE COURT: You are sitting here telling me over and over and over again, we have the backup tapes, it solves the entire problem. I don’t know if we can go back to 2004, but that would be, what, 365 days a year times ten years at $1,000. . . . Corinthian is going to have a very large bill.

MR. BROWN: The one day is a snapshot, though. For example, if you asked how would the world be different if we had sent a litigation hold that stopped the deletion of one of the plaintiffs’ emails, all we need—you can look at the termination date and get the backup tape for that date, and now you have got their email box exactly how it existed as of the date of their termination. It would be perfect. It would be one day, $1,000. If you wanted that for ten employees, then you get to $10,000. You don’t have to do it as separate days, because it is cumulative.

THE COURT: I think what I want it for, sir, is every employee of Corinthian, because I have no confidence in your search. . . .

MR. BROWN: . . . With respect to every employee, if you are looking at every employee at the Bremerton campus where this occurred, we can do that, your Honor. It can be done. We are not going to have to pay individually for each employee. If the backup tape for one day—That is across the whole campus. We can capture everybody that way. . . .

Judge Robart decided to accept the representations of Mr. Brown that recovering data from the backup tapes “would be perfect. It would be one day, $1000.” For that reason Judge Robart deferred ruling on Plaintiffs’ first motion for sanctions. Instead, he issued an order compelling defendant to “retrieve from the backup tapes all employee email accounts as they existed on or near the date that the last Plaintiff’s employment was terminated” and to “search the retrieved information according to the terms articulated in the parties’ stipulated order.” Because the parties’ trial date of January 6, 2014, was looming in less than one month, the court set a deadline of December 20, 2013.

tape-backupAs it turned out, and, as I dare say, could not have been overly surprising to the IT sophisticated Judge Robart, the defendant’s backup tape recovery process proved, in Judge Robart’s words, considerably less straightforward than Mr. Brown represented at oral argument. I just love the mastery of understatement that most judges seem to have. Judge Robart’s footnote four spells out the details. The first vendor defendant hired was unable to retrieve any information from the backup tapes. Oops! The tapes were then returned to defendant, who somehow managed to recover and produce some emails itself. Defendant then hired a second vendor to complete recovery of the emails. After still more delay, this vendor was able to retrieve a subset of the remaining emails.  The tapes were once again returned to defendant, who eventually somehow supposedly recovered and produced the rest of the missing emails.  

Needless to say, the December 20, 2013, production deadline was not met. In fact, defendant was seven weeks late, and even then, only produced about 3,000 additional emails from the backup tapes. Still, that looked good compared to defendant’s prior ESI search efforts, where it had only produced 110 email strings and 1,270 pages of other documents. After the late production, the court reopened discovery and extended the trial date until November 3, 2014.

The next hearing the court has on e-discovery in this case is a second motion for sanctions filed by plaintiffs just before trial. It is not heard until March 21, 2014. It is another one of those last minute attempts to win a case on e-discovery failures instead of the merits.

Hunter_ThompsonBut before we go to the grand finale, which I warn you is not as grand as you might expect, here is your second chance to express your opinion. Do you think the defendant intentionally withheld production of the emails? Or do you think it was all just accident and confusion. Maybe the result of legal recreational use or something. This is Seattle after all.

So straighten up and vote.

To be continued in next week’s blog . . .


IT-Lex Discovers a Previously Unknown Predictive Coding Case: “FHFA v. JP Morgan, et al”

March 2, 2014

brain_gearsThe researchers at IT-Lex have uncovered a previously unknown predictive coding case out of the SDNY, Federal Housing Finance Authority v. JP Morgan Chase & Co., Inc. et al. The et al here includes just about every other major bank in the world, each represented by one of the top 25 mega law firms in the country. The interesting orders approving predictive coding were entered in 2012, yet, until now, no one has ever talked about FHFA v JP Morgan. That is amazing considering the many players involved.

The two main orders in the case pertaining to predictive coding, are here (order dated July 24, 2012), and here (order dated July 31, 2012). I have highlighted the main passages in these long transcripts. These are Ore Tenus orders, but orders none the less. The Pacer file is huge, so IT-Lex may have missed others, but we doubt it. The two key memorandums underlying the orders are by the defendant, JP Morgan’s attorneys, Sullivan & Cromwell, dated July 20, 2012, and by the plaintiff, FHFA’s lawyers, Quinn Emanuel Urquhart & Sullivan, dated July 23, 2012.

The fact these are ore tenus rulings on predictive coding explains how they have remained under the radar for so long. The orders show the mastery, finesse, and wisdom of the presiding District Court Judge Denise Cote. She was hearing her first predictive coding issue and handled it beautifully. Unfortunately, under the transcript the trial lawyers arguing pro and con did not hold up as well. Still, they appear to have been supported by good e-discovery lawyer experts behind the scenes. It all seems to have all turned out relatively well in the end as a recent Order dated February 14, 2014 suggests. Predictive coding was approved and court ordered cooperation resulted in a predictive coding project that appears to have gone pretty well. 

Defense Wanted To Use Predictive Coding

JP_MorganThe case starts with the defense, primarily JP Morgan, wanting to use predictive coding and the plaintiff, FHFA, objecting. The FHFA wanted the defendant banks to review everything. Good old tried and true linear review. The plaintiff also had fall back objections on the way the defense proposed to conduct the predictive coding.

The letter memorandum by Sullivan & Cromwell for JP Morgan is only three pages in length, but has 63 pages of exhibits attached. The letter relies heavily on the then new Da Silva Moore opinion by Judge Peck. The exhibits include the now famous 2011 Grossman and Cormack law review article on TAR, a letter from plaintiff’s counsel objecting to predictive coding, and a proposed stipulation and order. Here are key segments of Sullivan and Cromwell’s arguments:

According to Plaintiff, it will not agree to JPMC’s use of any Predictive Coding unless JPMC agrees to manually review each and every one of the millions of documents that JPMC anticipates collecting. As Plaintiff stated: “FHF A’s position is straightforward. In reviewing the documents identified by the agreed-upon search terms, the JPM Defendants should not deem a document nonresponsive unless that document has been reviewed by an attorney.”

Plaintiffs stated position, and its focus on “non-responsive” documents, necessitates this request for prompt judicial guidance. Predictive Coding has been recognized widely as a useful, efficient and reliable tool precisely because it can help determine whether there is some subset of documents that need not be manually reviewed, without sacrificing the benefit, if any, gained from manual review. Predictive Coding can also aid in the prioritization of documents that are most likely to be responsive. As a leading judicial opinion as well as commentators have warned, the assumption that manual review of every document is superior to Predictive Coding is “a myth” because “statistics clearly show that computerized searches are at least as accurate, if not more so, than manual review.” Da Silva Moore v. Publicis Groupe, 2012 U.S. Dist. LEXIS 23350, at *28 (S.D.N.Y. Feb. 24, 2012) (Peck, Mag. J.) …

JPMC respectfully submits that this is an ideal case for Predictive Coding or “machine learning” to be deployed in aid of a massive, expedited document production. Plaintiffs claims in this case against JPMC concern more than 100 distinct securitizations, issued over a several year period by three institutions that were entirely separate until the end of that period, in 2008 (i.e., JPMorgan Chase, Bear Stearns & Co., and Washington Mutual). JPMC conservatively predicts that it will have to review over 2.5 million documents collected from over 100 individual custodians. Plaintiffhas called upon JPMC to add large numbers of custodians, expand date ranges, and otherwise augment this population, which could only expand the time and expense required? Computer assisted review has been approved for use on comparable volumes of material. See, e.g., DaSilva Moore, 2012 U.S. Dist. LEXIS 23350, at *40 (noting that the manual review of3 million emails is “simply too expensive.”).

Plaintiff’s Objections

FHFA

The plaintiff federal government agency, FHFA, filed its own three page response letter with 11 pages of exhibits. The response objects to use of predictive coding and the plaintiff’s proposed methodology. Here is the core of their argument:

First, JPMC’s proposal is the worst of both worlds, in that the set of documents to which predictive coding is to be applied is already narrowed through the use of search terms designed to collect relevant documents, and predictive coding would further narrow that set of documents without attorney review,1 thereby eliminating potentially responsive documents. …

Finally, because training a predictive coding program takes a considerable amount of time,2 the truncated timeframe for production of documents actually renders these Actions far from “ideal” for the use of predictive coding.

Poppy_headThe first objection on keyword search screening is good, but the second, that training would take too long, shows that the FHFA needed better experts. The machine learning training time is usually far less than the document review time, especially in a case like this, and the overall times savings from using predictive coding are dramatic. So the second objection was a real dog.

Still, FHFA made one more objection to method that was well placed, namely that their had been virtually no disclosure as to how Sullivan and Cromwell intended to conduct the process. (My guess is, they had not really worked that all out yet. This was all new then, remember.)

[I]t has similarly failed to provide this Court with any details explaining (i) how it intends to use predictive coding, (ii) the methodology or computer program that will be used to determine responsiveness, or (iii) any safeguards that will ensure that responsive documents are not excluded by the computer model. Without such details, neither FHFA nor this Court can meaningfully assess JPMC’s proposal. See Da Silva Moore v. Publicis Groupe SA, 2012 U.S. Dist. LEXIS 23350, at *23 (S.D.N.Y. Feb. 24, 2012) (“[Defendant’s] transparency in its proposed ESI search protocol made it easier for the Court to approve the use of predictive coding.”).4 JPMC’s proposed order sets forth an amorphous proposal that lacks any details. In the absence of such information, this Court’s authorization of JPMC’s use of predictive coding would effectively give JPMC carte blanche to implement predictive coding as it sees fit.

Hearing of July 24, 2012

Judge_Denise_CoteJudge Denise Cote came into the hearing having read the briefs and Judge Peck’s then recent landmark ruling in Da Silva Moore. It was obvious from her initial comments that her mind was made up that predictive coding should be used. She understood that this mega-size case needed predictive coding to meet the time deadlines and not waste a fortune on e-document review. Here are Judge Cote’s words at pages 8-9 of the transcript:

It seems to me that predictive coding should be given careful consideration in a case like this, and I am absolutely happy to endorse the use of predictive coding and to require that it be used as part of the discovery tools available to the parties. But it seems to me that the reliability and utility of predictive coding depends upon the process that takes place in the initial phases in which there is a pool of materials identified to run tests against, and I think that some of the documents refer to this as the seed — S-E-E-D — set of documents, and then there are various rounds of further testing to make sure that the code becomes smart with respect to the issues in this case and is sufficiently focused on what needs to be defined as a responsive document. And for this entire process to work, I think it needs transparency and cooperation of counsel.

I think ultimately the use of predictive coding is a benefit to both the plaintiff and the defendants in this case. I think there’s every reason to believe that, if it’s done correctly, it may be more reliable — not just as reliable but more reliable than manual review, and certainly more cost effective — cost effective for the plaintiff and the defendants.

To plaintiff’s counsel credit she quickly shifted her arguments from whether to how. Defense counsel also falls all over herself about how cooperative she has been and will continue to be, all the while implying that the other side is a closet non-cooperator.

As it turns out, very little actual conservation had occurred between the two lead counsel before the hearing, as both had preferred snarly emails and paper letters. At the hearing Judge Cote ordered the attorneys to talk first, and then rather than shoot off more letters, and to call her if they could not agree.

I strongly suggest you read the whole transcript of the first order to see the effect a strong judge can have on trial lawyers. Page 24 is especially instructive as to just how active a bench can be. At the second hearing of July 24, 2012, I suggest you read the transcript at pages 110-111 to get an idea as to just how difficult those attorneys meetings proved to be.

As a person obsessed with predictive coding I find the transcripts of the two hearings to be kind of funny in a perverse sort of way. The best way for me to share my insights is by using the format of a lawyer joke.

Two Lawyers Walked Into A Bar

star_trek_barOne e-discovery lawyer walks into a Bar and nothing much happens. Two e-discovery lawyers walks into a Bar and an interesting discussion ensues about predictive coding. One trial lawyer walks into a Bar the volume of the whole place increases. Two trial lawyers walk into a Bar and an argument starts.

The 37 lawyers who filed appearances in the FHFA case walk into a Bar and all hell breaks loose. There are arguments everywhere. Memos are written, motions are filed, and the big bank clients are billed a million or more just talking about predictive coding.

Then United States District Court Judge Denise Cote walks into the Bar. All the trial lawyers immediately shut up, stand up, and start acting real agreeable, nice, and polite. Judge Cote says she has read all of the letters and they should all talk less, and listen more to the two e-discovery specialists still sitting in the bar bemused. Everything becomes a cooperative love-fest thereafter, at least, as far as predictive coding and Judge Conte are concerned. The trial lawyers move on to fight and bill about other issues more within their kin.

Substantive Disputes in FHFA v. JP Morgan

disclosureThe biggest substantive issues in the first hearing of July 24, 2012 had to do with disclosure and keyword filtering before machine training. Judge Cote was prepared on the disclosure issue from having read the Da Silva Moore protocol, and so were the lawyers. The judge easily pressured defense counsel to disclose both relevant and irrelevant training documents to plaintiff’s counsel, with the exception of privileged documents.

As to the second issue of keyword filtering, the defense lawyers had been told by the experts behind the scenes that JP Morgan should be allowed to keyword filter the custodians ESI before running predictive coding. Judge Peck had not addressed that issue in Da Silva Moore, since the defense had not asked for that, so Judge Cote was not prepared to rule on that then new and esoteric issue. The trial lawyers were not able to articulate much on the issue either.

Judge Cote asked trial counsel if they had previously discussed this issue, not just traded memos, and they admitted no. So she ordered them to talk about it. It is amazing how much easier it is to cooperate and reach agreement when you actually speak, and have experts with you guiding the process. So Judge Cote ordered them to discuss the issue, and, as it turns out from the second order of July 31, 2012, they reached agreement. There would be no keyword filtering.

Although we do not know all of the issues discussed by attorneys, we do know they managed to reach agreement, and we know from the first hearing what a few of the issues were. They were outlined by plaintiff’s counsel who complained that they had no idea as to how defense counsel was going to handle the following issues at page 19 of the first hearing transcript:

What is the methodology for creating the seed set? How will that seed set be pulled together? What will be the number of documents in the seed set? Who will conduct the review of the seed set documents? Will it be senior attorneys or will it be junior attorneys? Whether the relevant determination is a binary determination, a yes or no for relevance, or if there’s a relevance score or scale in terms of 1 to 100. And the number of rounds, as your Honor noted, in terms of determining whether the system is well trained and stable.

So it seems likely all these issues and more were later discussed and accommodations reached.  At the second hearing of July 31, 2012, we get a pretty good idea as to how difficult the attorneys meetings must have been. At pages 110-111 of the second hearing transcript we see how counsel for JP Morgan depicted these meetings and the quality of input received from plaintiff’s counsel and experts:

We meet every day with the plaintiff to have a status report, get input, and do the best we can to integrate that input. It isn’t always easy, not just to carry out those functions but to work with the plaintiff.

The suggestions we have had so far have been unworkable and by and large would have swamped the project from the outset and each day that a new suggestion gets made. But we do our best to explain that and keep moving forward.

Defense counsel then goes into what most lawyers would call “suck-up” mode to the judge and says:

We very much appreciate that your Honor has offered to make herself available, and we would not be surprised if we need to come to you with a dispute that hasn’t been resolved by moving forward or that seems sufficiently serious to put the project at risk. But that has not happened yet and we hope it will not.

After that plaintiff’s counsel complains the defense counsel has not agreed to allow depositions transcripts and witness statements to be used as training documents. That’s right. The plaintiff wanted to include congressional testimony, depositions and other witness statements that they found favorable to their position as part of the training documents to find relevant information store of custodian information.

Judge Cote was not about to be tricked into making a ruling on the spot, but instead wisely told them to go back and talk some more and get real expert input on the advisability of this approach. She is a very quick study as the following exchange at page 114 of the transcript with defense counsel after hearing the argument of plaintiff’s counsel illustrates:

THE COURT: Good. We will put those over for another day. I’m learning about predictive coding as we go. But a layperson’s expectation, which may be very wrong, would be that you should train your algorithm from the kinds of relevant documents that you might actually uncover in a search. Maybe that’s wrong and you will all educate me at some other time. I expect, Ms. Shane, if a deposition was just shot out of this e-discovery search, you would produce it. Am I right?

MS. SHANE: Absolutely, your Honor. But your instinct that what they are trying to train the system with are the kinds of documents that would be found within the custodian files as opposed to a batch of alien documents that will only confuse the computer is exactly right.

It is indeed a very interesting issue, but we cannot see a report in the case on Pacer that shows how the issue was resolved. I suspect the transcripts were all excluded, unless they were within a custodian’s account.

2014 Valentines Day Hearing

kiss_me_im_a_custodian_keychainThe only other order we found in the case mentioning predictive coding is here (dated February 14, 2014). Most of the Valentine’s Day transcript pertains to trial lawyers arguing about perjury, and complaining that some key documents were missed in the predictive coding production by JP Morgan. But the fault appears due to the failure to include a particular custodian in the search, an easy mistake to have happen. That has nothing to do with the success of the predictive coding or not.

Judge Cote handled that well, stating that no review is “perfect” and she was not about to have a redo at this late date. Her explanation at pages 5-6 of the February 14, 2014 transcript provides a good wrap up for FHFA v. JP Morgan:

Parties in litigation are required to be diligent and to act in good faith in producing documents in discovery. The production of documents in litigation such as this is a herculean undertaking, requiring an army of personnel and the production of an extraordinary volume of documents. Clients pay counsel vast sums of money in the course of this undertaking, both to produce documents and to review documents received from others. Despite the commitment of these resources, no one could or should expect perfection from this process. All that can be legitimately expected is a good faith, diligent commitment to produce all responsive documents uncovered when following the protocols to which the parties have agreed, or which a court has ordered.

Indeed, at the earliest stages of this discovery process, JP Morgan Chase was permitted, over the objection of FHFA, to produce its documents through the use of predictive coding. The literature that the Court reviewed at that time indicated that predictive coding had a better track record in the production of responsive documents than human review, but that both processes fell well short of identifying for production all of the documents the parties in litigation might wish to see.

Conclusion

transparencyThere are many unpublished decisions out there approving and discussing predictive coding. I know of several more. Many of them, especially the ones that first came out and pretty much blindly followed our work in Da Silva Moore, call for complete transparency, including disclosure of irrelevant documents used in training. That is what happened in FHFA v. JP Morgan and the world did not come to an end. Indeed, the process seemed to go pretty well, even with a plaintiff’s counsel who, in the words of Sullivan and Cromwell, made suggestions everyday that were unworkable and by and large would have swamped the project … but we do our best to explain that and keep moving forward. Pages 110-111 of the second hearing transcript. So it seems cooperation can happen, even when one side is clueless, and even if full disclosure has been ordered.

Since the days of 2011 and 2012, when our Da Silva Moore protocol was developed, we have had much more experience with predictive coding. We have more information on how the training actually functions with a variety of chaotic email datasets, including the new Oracle ESI collection, and even more testing with the Enron dataset.

Based on what we know now, I do not think it is necessary to make disclosure of all irrelevant documents used in training. The only documents that have a significant impact on machine learning are the borderline, grey area documents. These are the ones who relevancy is close, and often a matter of opinion, of how you view the case. Only these grey area irrelevant documents need to be disclosed to protect the integrity of the process.

grey_area_disclosure

The science and other data behind that has to do with Jaccard Index classification inconsistencies, as well as the importance of mid-range ranked documents to most predictive coding algorithmic analysis. See Eg: Less Is More: When it comes to predictive coding training, the “fewer reviewers the better” – Part Three at the subheadings Disclosure of Irrelevant Training Documents and Conclusions Regarding Inconsistent Reviews. When you limit disclosure to grey area training documents, and relevant documents, the process can become even more efficient without any compromise in quality or integrity. This of course assumes honest evaluations of grey area documents and forthright communications between counsel. But then so does all discovery in our system of justice. So this is really nothing new, nor out of the ordinary.

All discovery depends on the integrity and trustworthiness of the attorneys for the parties. Fortunately, almost all attorneys honorably fulfill these duties, except perhaps for the duty of technology competence. That is the greatest ethical challenge of the day for all litigators.


Beware of the TAR Pits! – Part Two

February 23, 2014

This is the conclusion of a two part blog. For this to make sense please read Part One first.

Quality of Subject Matter Experts

Poppy_headThe quality of Subject Matter Experts in a TAR project is another key factor in predictive coding. It is one that many would prefer to sweep under the rug. Vendors especially do not like to talk about this (and they sponsor most panel discussions) because it is beyond their control. SMEs come from law firms. Law firms hire vendors. What dog will bite the hand that feeds him? Yet, we all know full well that not all subject matter experts are alike. Some are better than others. Some are far more experienced and knowledgeable than others. Some know exactly what documents they need at trial to win a case. They know what they are looking for. Some do not. Some have done trials, lots of them. Some do not know where the court house is. Some have done many large search projects, first paper, now digital. Some are great lawyers; and some, well, you’d be better off with my dog.

The SMEs are the navigators. They tell the drivers where to go. They make the final decisions on what is relevant and what is not. They determine what is hot, and what is not. They determine what is marginally relevant, what is grey area, what is not. They determine what is just unimportant more of the same. They know full well that some relevant is irrelevant. They have heard and understand the frequent mantra at trials: Objection, Cumulative. Rule 403 of the Federal Evidence Code. Also see The Fourth Secret of Search: Relevant Is Irrelevant found in Secrets of Search – Part III.

Quality of SMEs is important because the quality of input in active machine learning is important. A fundamental law of predictive coding as we now know it is GIGO, garbage in, garbage out. Your active machine learning depends on correct instruction. Although good software can mitigate this somewhat, it can never be eliminated. See: Webber & Pickens, Assessor Disagreement and Text Classifier Accuracy, SIGIR 2013 (24% more ranking depth needed to reach equivalent recall when not using SMEs, even in a small data search of news articles with rather simple issues).

Jeremy_PickensInformation scientists like Jeremy Pickens are, however, working hard on ways to minimize the errors of SME document classifications on overall corpus rankings. Good thing too because even one good SME will not be consistent in ranking the same documents. That is the Jaccard Index scientists like to measure. Less Is More: When it comes to predictive coding training, the “fewer reviewers the better” – Part Two, and search of Jaccard in my blog.

Unique_Docs_VennIn my Enron experiments I was inconsistent in determining the relevance of the same document 23% of the time. That’s right, I contradicted myself on relevancy 23% of the time. (If you included irrelevancy coding the inconsistencies were only 2%.) Lest you think I’m a complete idiot (which, by the way, I sometimes am), the 23% rate is actually the best on record for an experiment. It is the best ever measured, by far. Other experimentally measured rates have inconsistencies of from 50% to 90% (with multiple reviewers). Pathetic huh? Now you know why AI is so promising and why it is so important to enhance our human intelligence with artificial intelligence. When it comes to consistency of document identifications in large scale data reviews, we are all idiots!

With these human  frailty facts in mind, not only variable quality in expertise of subject matter, but also human inconsistencies, it is obvious why scientists like Pickens and Webber are looking for techniques to minimize the impact of errors and, get this, even use these inevitable errors to improve search. Jeremy Pickens and I have been corresponding about this issue at length lately. Here is Jeremy’s later response to this blog. In TAR, Wrong Decisions Can Lead to the Right Documents (A Response to Ralph Losey). Jeremy does at least concede that coding quality is indeed important. He goes on to argue that his study shows that wrong decisions, typically on grey area documents, can indeed be useful.

Penrose_triangle_ExpertiseI do not doubt Dr. Pickens’ findings, but am skeptical of the search methods and conclusions derived therefrom. In other words, how the training was accomplished, the supervision of the learning. This is what I call here the driver’s role, shown on the triangle as the Power User and Experienced Searcher. In my experience as a driver/SME, much depends on where you are in the training cycle. As the training continues the algorithms eventually do become able to detect and respond to subtle documents distinctions. Yes, it take a while, and you have to know what and when to train on, which is the drivers skill (for instance you never train with giant documents), but it does eventually happen. Thus, while it may not matter if you code grey area documents wrong at first, it eventually will, that is unless you do not really care about the distinctions. (The TREC overturn documents Jeremy tested on, the ones he called wrong documents, were in fact grey area documents, that is, close questions. Attorneys disagreed on whether they were relevant, which is why they were overturned on appeal.) The lack of precision in training, which is inevitable anyway even when one SME is used, may not matter much in early stages of training, and may not matter at all when testing simplistic issues using easy databases, such as news articles. In fact, I have used semi-supervised training myself, as Jeremy describes from old experiments in Pseudo Relevance Feedback. I have seen it work myself, especially in early training.

Still, the fact some errors do not matter in early training does not mean you should not care about consistency and accuracy of training during the whole ride. In my experience, as training progresses and the machine gets smarter, it does matter. But let’s test that shall we? All I can do is report on what I see, i.w. – anecdotal.

Outside of TREC and science experiments, in the messy real world of legal search, the issues are typically maddeningly difficult. Moreover, the difference in cost of review of hundreds of thousands of irrelevant documents can be mean millions of dollars. The fine points of differentiation in matured training are needed for precision in results to reduce costs of final review. In other words, both precision and recall matter in legal search, and all are governed by the overarching legal principle of proportionality. That is not part of information science of course, but we lawyers must govern our search efforts by proportionality.

Also See William Webber’s response: Can you train a useful model with incorrect labels? I believe that William’s closing statement may be correct, either that or software differences:

It may also be, though this is speculation on my part, that a trainer who is not only a subject-matter expert, but an expert in training itself (an expert CAR driver, to adopt Ralph Losey’s terminology) may be better at selecting training examples; for instance, in recognizing when a document, though responsive (or non-responsive), is not a good training example.

alchemyI hope Pickens and Webber get there some day. In truth, I am a big supporter of their efforts and experiments. We need more scientific research. But for now, I still do not believe we can turn lead into gold. It is even worse if you have a bunch of SMEs arguing with each other about where they should be going, about what is relevant and what is not. That is a separate issue they do not address, which points to the downside of all trainers, both amateurs and SMEs alike. See: Less Is More: When it comes to predictive coding training, the “fewer reviewers the better” – Parts OneTwo, and Three.

For additional support on the importance of SMEs, see again Monica’s article, EDI-Oracle Studywhere she summarizes the conclusion of Patrick Oot from the study that:

Technology providers using similar underlying technology, but different human resources, performed in both the top-tier and bottom-tier of all categories. Conclusion: Software is only as good as its operators. Human contribution is the most significant element. (emphasis in original)

Also see the recent Xerox blog, Who Prevails in the E-Discovery War of Man vs. Machine? by Gabriela Baron.

Teams that participated in Oracle without a bona fide SME, much less a good driver, well, they were doomed. The software was secondary. How could you possibly replicate the work of the original SME trial lawyers that did the first search without having an SME yourself, one with at least a similar experience and knowledge level.

map_lost_navigator_SMEThis means that even with a good driver, and good software, if you do not also have a good SME, you can still end up driving in circles. It is even worse when you try to do a project with no SME at all. Remember, the SME in the automobile analogy is the navigation system, or to use the pre-digital reality, the passenger with the map. We have all seen what happens where the navigation system screws up, or the map is wrong, or more typically, out of date (like many old SMEs). You do not get to the right place. You can have a great driver, and go quite fast, but if you have a poor navigator, you will not like the results.

The Oracle study showed this, but it is hardly new or surprising. In fact, it would be shocking if the contrary were true. How can incorrect information ever create correct information? The best you can hope for is to have enough correct information to smooth out the errors. Put another way, without signal, noise is just noise. Still, Jeremy Pickens claims there is a way. I will be watching and hope he succeeds where the alchemists of old always failed.

Tabula Rasa

blank_slateThere is one way out of the SME frailty conundrum that I have high hopes for and can already understand. It has to do with teaching the machine about relevance for all projects, not just one. The way predictive coding works now the machine is a tabula rasa, a blank slate. The machine knows nothing to begin with. It only knows what you teach it as the search begins. No matter how good the AI software is at learning, it still does not know anything on its own. It is just good at learning.

That approach is obviously not too bright. Yet, it is all we can manage now in legal search at the beginning of the Second Machine Age. Someday soon it will change. The machine will not have its memory wiped after every project. It will remember. The training from one search project will carry over to the next one like it. The machine will remember the training of past SMEs.

That is the essential core of my PreSuit proposal: to retain the key components of the past SME training so that you do not have to start afresh on each search project. PreSuit: How Corporate Counsel Could Use “Smart Data” to Predict and Prevent Litigation. When that happens (I don’t say if, because this will start happening soon, some say it already has) the machine could start smart.

Scarlett_Johansson - Samantha in HERThat is what we all want. That is the holy grail of AI-enhanced search — a smart machine. (For the ultimate implications of this, see the movie Her, which is about an AI enhanced future that is still quite a few years down the road.) But do not kid yourself, that is not what we have now. Now we only have baby robots, ones that are eager and ready to learn, but do not know anything. It is kind of like 1-Ls in law school, except that when they finish a class they do not retain a thing!

When my PreSuit idea is implemented, the next SME will not have to start afresh. The machine will not be a tabula rasa. It will be able to see litigation brewing. It will help general counsel to stop law suits before they are filed. The SMEs will then build on the work of prior SMEs, or maybe build on their own previous work in another similar project. Then the GIGO principle will be much easier to mitigate. Then the computer will not be completely dumb, it will have some intelligence from the last guy. There will be some smart data, not just big dumb data. The software will know stuff, know the law and relevance, not just know how to learn stuff.

When that happens, then the SME in a particular project will not be as important, but for now, when working from scratch with dumb data, the SME is still critical. The smarter and more consistent the better. Less Is More: When it comes to predictive coding training, the “fewer reviewers the better” – Parts OneTwo, and Three.

Professor Marchionini, like all other search experts, recognizes the importance of SMEs to successful search. As he puts it:

Thus, experts in a domain have greater facility and experience related to information-seeking factors specific to the domain and are able to execute the subprocesses of information seeking with speed, confidence, and accuracy.

That is one reason that the Grossman Cormack glossary builds in the role of SMEs as part of their base definition of computer assisted review:

A process for Prioritizing or Coding a Collection of electronic Documents using a computerized system that harnesses human judgments of one or more Subject Matter Expert(s) on a smaller set of Documents and then extrapolates those judgments to the remaining Document Collection.

Glossary at pg. 21 defining TAR.

Most SMEs Today Hate CARs
(And They Don’t Much Like High-Tech Drivers Either)

simpsonoldmanThis is an inconvenient truth for vendors. Predictive coding is defined by SMEs. Yet vendors cannot make good SMEs step up to the plate and work with the trainers, the drivers, to teach the machine. All the vendors can do is supply the car and maybe help with the driver. The driver and navigator have to be supplied by the law firm or corporate clients. There is no shortage of good SMEs, but almost all of them have never even seen a CAR. They do not like them. They can barely even speak the language of the driver. They don’t much like most of the drivers either. They are damn straight not going to spend two weeks of their lives riding around in one of those new fangled horseless carriages.

ringo and old guy

That is the reality of where we are now. Also see: Does Technology Leap While Law Creeps? by Brian Dalton, Above the Law. Of course this will change with the generations. But for now, that is the way it is. So vendors work on error minimization. They try to minimize the role of SMEs. That is anyway a good idea, because, as mentioned, all human SMEs are inconsistent. I was lucky to only be inconsistent 23% of the time on relevance. But still, there is another obvious solution.

There is another way to deal today with the reluctant SME problem, a way that works right now with today’s predictive coding software. It is a kind of non-robotic surrogate system that I have developed, and I’m sure a several other professional drivers have as well. See my CAR page for more information on this. But, in reality it is one of those things I would just have to show you in a driver education school type setting. I do it frequently. It involves action in behalf of an SME, and dealing with the driver for them. It places them in their comfort zone, where they just make yes no decisions on the close question documents, although there is obviously more to it than that. It is not nearly as good as the surrogate system in the movie Her, and of course, I’m no movie star, but it works.

HER_Samantha_Surrogate

My own legal subject matter expertise is, like most lawyers, fairly limited. I know a lot about a few things, and am a stand alone SME in those fields. I know a fair amount about many more legal fields, enough to understand real experts, enough to serve as their surrogate or right hand. Those are the CAR trips I will take.

If I do not know enough about a field of law to understand what the experts are saying, then I cannot serve as a surrogate. I could still drive of course, but I would refuse to do that out of principle, unless I had a navigator, an SME, who knew what they were doing and where they wanted to go. I would need an SME willing to spend the time in the CAR needed to tell me where to go. I hate a TAR pit as much as the next guy. Plus at my age and experience I can drive anywhere I want, in pretty much any CAR I want. That brings us to the final corner of the triangle, the variance in the quality of predictive coding software.

Quality of the CAR Software

I am not going to spend a lot of time on this. No lawyer could be naive enough to think that all of the software is equally as good. That is never how it works. It takes time and money to make sophisticated software like this. Anybody can simply add on open source machine learning software code to their review platforms. That does not take much, but that is a Model-T.

Old_CAR_stuck_mud

To make active machine learning work really well, to take it to the next level, requires thousands of programming hours. It takes large teams of programmers. It takes years. It take money. It takes scientists. It takes engineers. It takes legal experts too. It takes many versions and continuous improvements of search and review software. That is how you tell the difference between ok, good, and great software. I am not going to name names, but I will say the Gartner’s so called Magic Quadrant evaluation of e-discovery software is not too bad. Still, be aware that evaluation of predictive coding is not really their thing, or even a primary factor for rating review software.

Gartner_Magic_Quadrant

It is kind of funny how pretty much everybody wins in the Gartner evaluation. Do you think that’s an accident? I am privately much more critical. Many well known programs are very late to the predictive coding party. They are way behind. Time will tell if they are ever able to catch up.

Still, these things do change from year to year, as new versions of software are continually released. For some companies you can see real improvements, real investments being made. For others, not so much, and what you do see is often just skin deep. Always be skeptical. And remember, the software CAR is only as good as your driver and navigator.

car_mind_meld

When it comes to software evaluation what counts is whether the algorithms can find the documents needed or not. Even the best driver navigator team in the world can only go so far in a clunker. But give them a great CAR, and they will fly. The software will more than pay for itself in saved reviewer time and added security of a job well done.

Deja Vu All Over Again. 

Predictive coding is a great leap forward in search technology. In the longterm predictive coding and other AI-based software will have a bigger impact on the legal profession than did the original introduction of computers into the law office. No large changes like this are without problems. When computers were first brought into law offices they too caused all sorts of problems and had their pitfalls and nay sayers. It was a rocky road at first.

Ralph in the late 1980s

I was there and remember it all very well. The Fonz was cool. Disco was still in. I can remember the secretaries yelling many times a day that they needed to reboot. Reboot! Better save. It became a joke, a maddening one. The network was especially problematic. The partner in charge threw up his hands in frustration. The other partners turned the whole project over to me, even though I was a young associate fresh out of law school. They had no choice. I was the only one who could make the damn systems work.

Ifloppy_8incht was a big investment for the firm at the time. Failure was not an option. So I worked late and led my firm’s transition from electric typewriters and carbon paper to personal computers, IBM System 36 minicomputers, word processing, printers, hardwired networks, and incredibly elaborate time and billing software. Remember Manac time and billing in Canada? Remember Displaywriter? How about the eight inch floppy? It was all new and exciting. Computers in a law office! We were written up in IBM’s small business magazine.

For years I knew what every DOS operating file was on every computer in the firm. The IBM repair man became a good friend. Yes, it was a lot simpler then. An attorney could practice law and run his firm’s IT department at the same time.

ralph_1990sHey, I was the firm’s IT department for the first decade. Computers, especially word processing and time and billing software, eventually made a huge difference in efficiency and productivity. But at first there were many pitfalls. It took us years to create new systems that worked smoothly in law offices. Business methods always lag way behind new technology. This is clearly shown by MIT’s Erik Brynjolfsson and Andrew McAfee in their bestseller, Second Machine Age. It typically takes a generation to adjust to major technology breakthroughs. Also see Ted Talk by Brynjolfsson with video.

I see parallels with the 1980s and now. The main difference is legal tech pioneers were very isolated then. The world is much more connected now. We can observe together how, like in the eighties, a whole new level of technology is starting to make its way into the law office. AI-enhanced software, starting with legal search and predictive coding, is something new and revolutionary. It is like the first computers and word processing software of the late 1970s and early 80s.

It will not stop there. Predictive coding will soon expand into information governance. This is the PreSuit project idea that I, and others, are starting to talk about. See Eg: Information Governance Initiative. Moreover, many think AI software will soon revolutionize legal practice in a number of other ways, including contract generation and other types of repetitive legal work and analysis. See Eg: Rohit Talwar, Rethinking Law Firm Strategies for an Era of Smart Technology (ABA  LPT, 2014). The potential impact of supervised learning and other cognitive analytics tools on all industries is vast. See Eg: Deloitte’s 2014 paper: Cognitive Analytics (“For the first time in computing history, it’s possible for machines to learn from experience and penetrate the complexity of data to identify associations.”); Also see: Digital Reasoning software, and Paragon Science software. Who knows where it will lead the world, much less the legal profession? Back in the 1980s I could never have imagined the online Internet based legal practice that most of us have now.

The only thing we know for sure is that it will not come easy. There will be problems, and the problems will be overcome. It will take creativity and hard work, but it will be done. Easy buttons have always been a myth, especially when dealing with the latest advancements of technology. The benefits are great. The improvements from predictive coding in document review quality and speed are truly astonishing. And it lowers cost too, especially if you avoid the pits. Of course there are issues. Of course there are TAR pits. But they can be avoided and the results are well worth the effort. The truth is we have no choice.

Conclusion

retire

If you want to remain relevant and continue to practice law in the coming decades, then you will have to learn how to use the new AI-enhanced technologies. There is really no choice, other than retirement. Keep up, learn the new ways, or move on. Many lawyers my age are retiring now for just this reason. They have no desire to learn e-discovery, much less predictive coding. That’s fine. That is the honest thing to do. The next generation will learn to do it, just like a few lawyers learned to use computers in the 1980s and 1990s. Stagnation and more of the same is not an option in today’s world. Constant change and education is the new normal. I think that is a good thing. Do you?

Leave a comment. Especially feel free to point out a TAR pit not mentioned here. There are many, I know, and you cannot avoid something you cannot see.


Part Three of Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead, Long Live the Whole Man

February 1, 2014

This is the final part of a three-part blog. You will need to read the first two segments for this conclusion to makes sense. See Part One and Part Two of Scientific Proof of Law’s Overreliance On Reason: The “Reasonable Man” is Dead, Long Live the Whole Man.

Final Word From Dan Ariely 

dan-arielyGetting back to Dan, the psychologist economist, in addition to teaching and running very clever experiments at MIT and Duke, Dan is the founder of an organization with a name that seems both funny and ironic, The Center for Advanced Hindsight. He is also a prolific writer and video maker, both activities I admire. See for instance his informative page at MIT, his blog at DanAriely.com, his several books, and his videos, and even though its slightly boring, see his web page at Duke.

As a final piece of evidence on overreliance on reason I offer more testimony by Professor Ariely’s via another video, one which is not at all boring, I swear. It is called The Truth About Dishonesty. It concludes with a subject near and dear to all lawyers, conflicts of interest. The non-rational impact of such conflicts turns out to be very strong and the law is wise to guard against them. Perhaps we should even step up our efforts in this area? 

Cornerstone Made of Pudding

The scientific experiments of Dan Ariely and others show that the cornerstone of the Law – reasonability – is not made of granite as we had thought, it is made of pudding. You can hide your head in the sand, if you wish, and continue to believe otherwise. We humans are quite good at self-delusion. But that will not change the truth. That will not change quicksand into granite. Our legal house needs a new and better foundation than reason. We must follow the physicists of a century ago. We must transcend Newtonian causality and embrace the more complex, more profound truth that science has revealed. The Reasonable Man is a myth that has outlived its usefulness. We need to accept the evidence, and move on. We need to develop new theories and propositions of law that confirm to the new facts at hand.

Science has shown that our current reason-only-based system of justice is on shaky grounds. It is now up to us to do something about it. No big brother government, or super think-tank guru is going to fix this for us. Certainly not scientists either, but they should be able to help, along with technologists, programmers and engineers.

homer-simpson-brain-scan

What are the implications of the findings of unreliable mental processes on the Law and our ability to reach just decisions? We should ask these questions concerning the Law, just like Professor Ariely is asking concerning Economics. Our fundamental legal assumption that all people can act out of reason and logic alone is false. Decisions made with these faculties alone are the exception, not the rule. There are a number of other contributing factors, including emotions, intuition, and environment. What does this mean to negligence law? To sanctions law? Now that the Reasonable Man is dead, who shall replace him?

Just as classical economic theory has had it all wrong, so too has classical legal theory. People are not built like reasonable machines. That includes lawyers, judges, and everyone else in the justice system, especially the litigants themselves.

If Not Reason, Then What?

Ralph_moustacheSince human reason is now known to be so unreliable, and is in fact, only a contributing factor to our decisions, on what should we base our legal jurisprudence. My answer is in the title of this blog. I believe that the Reasonable Man, now that he is known to be an impossible dream, should be replaced by the Whole Man. Our jurisprudence should be based on the reality that we are not robots, not mere thinking machines. We have many other faculties and capabilities beyond just logic and reason. We are more than math.

So I propose a new, holistic model for the law. It would still include reason, but add our other faculties. It would incorporate our total self, all human attributes. We would include more than logic and reason to judge whether behavior is acceptable or not, to consider whether a resolution of a dispute is fair or not.

A new schemata for a holistic jurisprudence would thus include not just human logic, but also human emotions, our feelings of fairness, our intuitions of what is right and just, and multiple environmental and perceptual factors. I suggest a new model start simple and use a four-fold structure like this, and please note I keep Reason on top, as I still strongly believe in its importance to the Law.

4-levels-Holistic_Law_pyramid

Some readers may notice that this model is similar to that of Carl Jung’s four personality types and the popular Myers Briggs personality tests. I am not advocating adoption of any of their ideologies, or personality theories, but I have over the years found their reference models to be useful. The above model, which is proposed only as a starting point for further discussion, is an extrapolation of these psychological models.

Call For Action

No one knows yet knows the full implications of the new data from science about the limited impact of logic and reason on human decisions. No one knows how to modify our legal systems to account these insights. Certainly I do not. But I do know that we should do something to reduce our overreliance on the Myth of the Reasonable Man. We should put the foundations of our legal system on something else, something more solid, more real than that.

In short, we need to put our house in order before it starts collapsing around us. That is the reasonable thing to do, but for that very reason we will not start to do it until we have better motivation than that. You cannot get people to act on reason alone, even lawyers. So let us engage the other more powerful motivators. To start the ball rolling, I will give special recognition and publicity to the best suggestions received from my readers to this problem, the best comments to this blog.

Maybe reason alone should always be secondary to simple fairness? Maybe that feeling of fairness, is more reliable than reasoned processes. Run the experiments please scientists. How reliable are our feelings of fairness? More importantly, what is the impact of feelings on our judges who pay attention to that? Maybe feelings should be on top of the new Holistic model. I personally doubt that, but who knows for sure until experiments are done. What I do not doubt is that feelings need to be taken into consideration more than they are now as true motivators of human action.

Maybe this means we should bring back equity, and down play law, like the old days, where we used to have Courts of Law and separate Courts of Equity. By the middle of the last century, Courts of Law won out in most states except Delaware, Mississippi, New Jersey, South Carolina, and Tennessee. Separate Equity Courts were closed down in favor of Courts of Law. Maybe we got it backwards. Maybe we were all led astray by our false confidence in reason.

Maybe we should now close our courts of Law and reopen our courts of Equity? How has it worked out for the states that kept equity courts? Have Chancellors truly been able to side-step strict rules of law when they felt it was equitable to do so? If so, how has that worked out? Has power been abused? Or has justice been attained more often? What can we learn from chancery courts that might help us build a more holistic court of the future?

A Few More Specific Suggestions of Reform

As discussed already, the AI enhancements now moving the law will continue to expand as a tool for the lawyers willing to learn how to use them. They will enhance and help improve our limited reasoning abilities. They will help us be more efficient.  They could also help us to stay completely honest, if we allow them to. So too will more emotional, in your face type judges, whether we let them or not. We need more judges who do not mind getting down into the weeds, to really understand the facts, and then tell you what they really think, both good and bad please.

Maybe timely reminders of ethics codes and serious under penalties of perjury type threats will also help? Maybe new, improved, and customized oaths will help? Oaths have been shown to be effective by Ariely’s research, so we should modify the rules accordingly.

Electrodes_EEG_RalphMaybe new truth recognition technologies should be used? Could a truth hat with built-in neural net be that far off? How about Google Glasses apps that provide reliable new feedback of all kinds on the people you watch testifying? That cannot be too far off.  (The lie detection apps already on the market for iPhones, etc., all look bogus to me, which is not unexpected based on the limited biofeedback the phone sensors can provide.) Even if the information is not admissible as evidence, it could still be quite valuable to lawyers. (Write me if you know of anyone working on any commercial projects like this for lawyers.) Perhaps some of the recent discoveries in neuroscience could begin to be used in the justice system in all types of unexpected ways?

trophy_LawMaybe public recognition and awards to lawyers and judges who get it right will help? And awards to litigants who do the right thing, even if they lose the case? How about a discretionary set-off for defendants like that? How about the converse? Shame can be a powerful motivator too.

Maybe we should change the conditions and environments of places where witnesses are questioned, where mediations and trials are conducted? Maybe we should provide special training to court reporters on oath giving? Maybe we should have trials again, and not just settlements?

We need to look for all kinds of motivators. Knowledge and reason alone are not a solid foundation for justice.

Conclusion

Changes are inevitable anyway in all social structures, so we should try to shape the ongoing changes in the Law. We should study what science has found and be guided by truth, not tradition.

We should try to move away from overreliance on reason alone. Where we must still rely on reason, and of course we must, we should look for science and technology based methods to impose even more checks and balances on reason than we already have. We should create new systems that will detect and correct the inevitable errors in reason that all humans will make – lawyers, judges and witnesses alike. Perhaps computers can help with this? Perhaps it would help to have easier and less expensive appeals? Especially interlocutory appeals? Perhaps greater use of experts, panels and special masters? We really need to start focusing on this, and, by the way, we cannot just think our way out of a prison of thought. We need to use all of our faculties.

We also need help from the scientific community. We need someone like Professor Ariely to focus on Law the way he has focused on Economics. So far I have not found anyone like that.

Please feel free to share any ideas you may have in the Comments to this blog below, or by private email to me. Again, the best comment will be recognized and praised. I may even give you some shout-outs at LegalTech this week. By the way, if you see me there, please take a moment to stop me and introduce yourself. I always like to meet my readers. If you know of any research psychologists who might be interested in these issues, please share this blog with them. I have already reached out to Dan Ariely. He responded right away and promised to provide a more detailed reaction later. When he does I will share his input in a later blog.


Follow

Get every new post delivered to your Inbox.

Join 3,122 other followers