The Jimi Hendrix Experience: Sanctions For Destruction

December 2, 2018

James Marshall “Jimi” Hendrix, whom Rolling Stone ranked the greatest guitarist of all time, died intestate in 1970 at twenty-seven. His heirs have been embroiled in litigation ever since. They have recently entered the fiery realm of e-discovery and sanctions. Experience Hendrix, LLC v. Pitsicalis, No. 17-cv-1927 (PAE) (S.D.N.Y., 11/27/18). The opinion by District Court Judge Paul A. Engelmayer is interesting in its own right, but when you add the Hendrix name and family feud, you have a truly memorable order. After all, we are talking about the artist who created “Purple Haze,” “Foxy Lady,” “The Star-Spangled Banner,” “Hey Joe” and my personal favorite, his rendition of Bob Dylan’s “All Along the Watchtower”.

Case Background: The Hendrix Family Feud

The latest suit involves the usual serial litigants. On one side is Jimi’s step-sister, Janie Hendrix (shown right). She is, as Jimi would have said, a “Foxy Lady”. Janie assumed control of the Estate from Jimi’s natural father, Al Hendrix, when he died in 2002. On the other side is Jimi’s brother, Leon Hendrix and Leon’s business partner, Andrew Pitsicalis. Kerzner, Hendrix Sues Serial Infringer Andrew Pitsicalis (American Blues Scene, 3/20/17). There can be big money in the Hendrix name, the top guitarist of all time. I for one still get choked up when I hear his rendition of “The Star-Spangled Banner”:

Jimi’s father, Al Hendrix, won bitterly fought estate litigation in the seventies, but the trademark litigation has never stopped. Multiple injunctions are in place under the corporate name, Experience Hendrix, LLC, but the infringements just keep coming. Companies keep popping up to sell Jimi Hendrix branded goods. Andrew Pitsicalis and Leon Hendrix are frequently involved. Their latest attempts to profit from the Jimi brand include marijuana related products (apparently “Purple Haze” has long been a well know strain of cannabis. See: Legal Battle Ensues Over Jimi Hendrix Usage Rights (High Times, 3/29/17); Jimi Hendrix’s heirs wage court battle over branded cannabis, other products (Cannifornan, 3/22/17).

I am reminded of the closing line of the Hendrix classic, Are You Experienced:

Ah! But Are You Experienced?
Have you ever been experienced?

Not necessarily stoned, but beautiful.

Jimi’s brother, Leon Hendrix (shown right), is an artist and musician himself with his own following. Some think he was treated unfairly by his Dad and Step-Sister. For a variety of reasons, especially I suspect the impact of Pitsicalis, the CEO of “Purple Haze Properties” and Leon’s business partner, there is still bad blood. Chris Fry, Jimi Hendrix’s Brother Fires Back Against Estate (Courthouse News, 3/28/17).

This kind of family feud mentality is not uncommon in litigation, especially in cases involving the intentional destruction of evidence. I am reminded of a Hendrix line from Voodoo Child:

Well, the night I was born. Lord I swear the moon turned a fire red. The night I was born I swear the moon turned a fire red. Well my poor mother cried out “lord, the gypsy was right!” And I seen her, fell down right dead. Have mercy.

Spoliation sanctions generally arise from a haze, just not a stoned purple haze, more like an angry moon turned a fire red haze. Even a seasoned District Court Judge in the SDNY, Paul Engelmayer, was “dismayed” by the conduct of Pitsicalis and Leon Hendrix. Well, what did you expect in matters involving the Estate of a Voodoo Child musical genius? The best guitarist that ever lived?

Judge Engelmayer’s Sanction Order

The scholarly and well-written opinion by District Court Judge Paul A. Engelmayer (shown right) begins by observing:

As the docket in this matter reflects, the Court has been called upon dismayingly often to act when presented with evidence of the PHP defendants’ persistent non-compliance with basic discovery obligations. Plaintiffs now move this Court to sanction these defendants for (1) spoliation of evidence and, more generally, (2) “consistent, pervasive[,] and relentless discovery abuses by [d]efendants and their counsel, Thomas Osinski.” Dkt. 245. Plaintiffs request, inter alia, a preliminary injunction, an order of attachment, an adverse inference instruction at trial, and terminating sanctions. See Dkts. 237, 244. For the reasons below, the Court grants the motion for an adverse inference instruction and directs the PHP defendants to pay the reasonable fees and costs incurred by plaintiffs in bringing this motion.

Experience Hendrix, LLC v. Pitsicalis, No. 17-cv-1927 (PAE) (S.D.N.Y., 11/27/18). Expressing “dismay” is about as emotional as Judge Engelmayer gets in writing an opinion, even one sanctioning a party for destroying evidence and disobeying court orders.

The PHP defendants mentioned are Leon Hendrix, Andrew Pitsicalis and their corporation, Purple Haze Properties (PHP). As you can see from the first quote, the attorney who represents them, Thomas Osinski, was also accused of discovery abuse. That often happens in joker and the thief type cases like this.

A good sanctions case will always have a “parade of horribles” consisting of a list of things the spoliating party supposedly did wrong. Hendrix is no exception. That is how the severe sanctions are justified. It would take too long to list all of the abuses justifying sanctions in Hendrix, but here are the four main ones:

  1. PHP Defendants’ Failure to Produce Forensic Images as Ordered. Apparent intentional disobedience of court orders to make forensic copies of and produce certain drives, even after daily fines are imposed for late production. One of the excuses PHP offered was especially humorous, especially considering the NYC venue, but they actually claimed “that they had had difficulty hiring an expert technician who could image the hard drives.” Yeah, it’s real hard. Need I say more about Osinski’s veracity? When they finally did produce some, but not all of the forensic images, they were not “forensic” images. They were just copies of all active files (a “ghost” copy) with no forensic copy of the slack space. That is what a forensic copy means. It allows for search and examination of deleted files, which was the whole point of the court order.
  2. PHP Defendants’ Use of Anti-Forensic Software. Software allowing for the complete wiping of files was found installed on several of the computer images that were produced. In some there was evidence the software was installed immediately after a court order was entered requiring production. In these the plaintiff’s forensic expert could also show that the software, CleanMyMac, was actually used to wipe files and when, although it was not possible to know what files were destroyed. The moving party (Janie Hendrix and her company Experience Hendrix, L.L.C.)  proved the use was knowing when their expert, John T. Myers, showed how the software was configured to have a pop-out and warn the user to confirm complete elimination of the file (it cannot be recovered after that). The defendants testified that they did not recall ever using it. Sure. Spoliate evidence and then cover-up, or try to.
  3. Andrew Pitsicalis Deleted “Jimi”-related Text Messages from his iPhone. Plaintiff’s forensic expert was able to prove that more than 500 text messages had been deleted from Pitsicalis cell phone after the duty to preserve had arisen (suit was filed). Moreover, they were able to recover nine text messages pertaining to Jimi. As Judge Engelmayer explained: “Fortuitously, Myers was able to recover the deleted text messages from the imaged phone because those communications had been stored not in the applications used to send and receive them (e.g., iMessage), but in databases where files exist until overwritten or otherwise purged.”
  4. Key Computer at First Hidden, then After Discovery in Photograph, Goes South to Florida and is Never Examined. This one sounds like a bad game of Where’s Waldo. A “Seventh Computer” was found, one never reported by PHP, by plaintiff’s study of photos on PHP’s Facebook page. Very clever. One picture on FB showed Andrew Pitsicalis, sitting in his office, in immediate proximity to a mystery desktop computer. When asked about it under oath PHP’s fine attorney, Osinski, swore that he thought it was just a dummy Apple monitor on the office desk, not a computer. He said he did not know that the monitor, and key board next to it, were a real, functional computer, an iMac. What? Did he think these were IKEA props in a display room? They were sitting on his client’s desk in a Facebook photo taken after suit was filed. But wait, there is more, Osinski went on to swear that the computer had moved to Florida. As Judge Engelmayer explained:

“Osinski testified that his present understanding is that the desktop computer belonged to an individual named Hector David, Jr. who has moved to Florida and who, Osinski assumes, took the computer with him. Osinski, however, did not have personal knowledge of this, or of the contents of the desktop in Andrew Pitsicalis’ office. … Andrew Pitsicalis, for his part, denied owning the computer and testified that David was not employed by PHP.”

Apparently no one has been able to locate this mysterious Hector David or know where he took the iMac computer sitting on Pitsicalis desk.

Legal Standards of Spoliation in Hendrix

Judge Engelmayer’s opinion in Hendrix examines two legal standards, Rule 37 and Spoliation. He begins the discussion with the duty to preserve, the threshold issue in spoliation:

The first issue is whether the PHP defendants had an obligation to preserve the categories of evidence at issue. A party has an obligation to preserve evidence when it “has notice that the evidence is relevant to litigation . . . [or] should have known that the evidence may be relevant to future litigation.” Kronisch v. United States, 150 F.3d 112 , 126 (2d Cir. 1998) (internal citations omitted).

That standard is easily met here.

He then goes on to discuss whether that duty as breached, another no-brainer based on the impressive parade of horribles in this case:

The Court further finds—and the evidence to this effect is overwhelming—that the PHP defendants repeatedly breached this duty. The breaches fall in three categories: (1) the use of cleaning software on covered computing devices, (2) the failure to disclose the existence of a seventh computing device containing potentially relevant documents, and (3) the deletion of relevant text messages.

Judge Engelmayer’ then discusses the key issue of intent, the mens rea to spoliate.

Much of the PHP defendants’ spoliation of evidence, the Court finds, was intentional. …

The Court finds that, by installing anti-cleaning software on his own computer and causing it to be installed on Schmitt’s in the face of an unambiguous and known duty to preserve potentially relevant evidence, Pitsicalis intentionally caused the destruction of such evidence.

The defenses proffered by the PHP defendants are unavailing. That Schmitt personally may not have acted with the intent to deleted responsive files is beside the point. The relevant mens rea here is that of Andrew Pitsicalis, who owned PHP, for which Schmitt worked as an independent contractor, and who, despite being a repeat litigant amply on notice of his duty to preserve potentially relevant evidence, urged Schmitt to run this software to delete files. Tr. 125, 133. Also unhelpful is Pitsicalis’ [*13] explanation that, at some unspecified point, he went on “Google to search for ‘top anti-forensic software’ and went through the first 10 pages of search” without finding anything for CleanMyMac. Andrew Pitsicalis Decl. at 2-3. Regardless what Pitsicalis’ internet research may have shown, the evidence adduced at the hearing clearly established both that the CleanMyMac software had the capacity to cause the deletion (and shredding) of files, and that Pitsicalis knew this, not least because the software’s causation of such deletion was made explicit to the user each time. Pitsicalis does not offer any reason for installing and using this software on his computer, let alone for having done so without first creating an image of the full contents of the computer that would have assured preservation of the computer’s contents.

Andrew Pitsicalis’ deletion of relevant text messages was also clearly intentional. By his admission, he personally and deliberately deleted, among other text messages, a series of texts concerning the marketing of “Jimi”-related products, the very subject of this lawsuit. He did so one day after the Court issued an order requiring the Purple Haze Properties defendants to: “produce to plaintiffs the forensic images of” every device, including phones, containing files that are relevant to this action. Pitsicalis did not offer any coherent defense to this misconduct. The Court finds it to have been a willful and blatant violation of the duty to preserve relevant evidence.

Finally, the Court finds that the removal of a computer from Andrew Pitsicalis’ office and its transfer to a Floridian, Hector David, Jr., was an act of intentional spoliation. To be sure, the question is a closer one, if only because the contents of that computer are unknown, and so the Court cannot rule out the possibility that these contents were wholly extraneous to this litigation. The location of the computer in Pitsicalis’ office, however, suggests otherwise. Had the Court been notified of the existence of this computer, it assuredly would have ordered that the computer’s contents be searched for responsive materials. It is also noteworthy that Andrew Pitsicalis did not inform his attorney of the existence of this computer. While conceivably these circumstances, in isolation, might have been consistent with the merely reckless disposal of evidence, when this episode is viewed in the light of Pitsicalis’ other acts of willful spoliation, the Court has little difficulty finding it, too, to bespeak intentional misconduct.

Sanctions Imposed

Judge Engelmayer begins his analysis of the appropriate, proportional sanctions by stating the black letter law:

The trial judge must determine the appropriate sanction for spoliation [*14] of evidence on a case-by-case basis. F, 247 F.3d at 436 . Such sanctions should be designed to:

(1) deter parties from engaging in spoliation; (2) place the risk of an erroneous judgment on the party who wrongfully created the risk; and (3) restore the prejudiced party to the same position [they] would have been in absent the wrongful destruction of evidence by the opposing party.

West, 167 F.3d at 779 . Case-dispositive sanctions, however, “should be imposed only in extreme circumstances, usually after consideration of alternative, less drastic sanctions.” Id.

Based on these objectives Judge Engelmayer sanctioned defendants as follows:

Considering these objectives, the Court imposes the following two sanctions, regarding (1) Andrew Pitsicalis’ computer, iPhone, and desktop computer; and (2) Schmitt’s computer, as to each of which the Court has found intentional spoliation. First, the Court will instruct the finder of fact that it may draw an adverse inference from the PHP parties’ failure adequately to preserve and produce these materials, to wit, that the devices in question contained evidence of conduct by the PHP defendants in breach of their legal duties to plaintiffs in connection with the sale and marketing of Jimi Hendrix-related materials.8

Second, given the resources plaintiffs again have had to expend in establishing the above-chronicled acts of non-compliance by the PHP defendants with the Court’s discovery orders, plaintiffs are entitled to an award reflecting the reasonable attorneys’ fees and costs incurred in connection with bringing and litigating the instant successful motion.

Judge Engelmayer went on to explain why a lesser sanction was inappropriate:

The Court has carefully considered whether lesser sanctions are adequate to cure the harm caused by the disposition of these materials. The Court’s firm conclusion is that no lesser sanction than the combination of an adverse inference instruction and an order directing the prompt recompense of plaintiffs for costs reasonably incurred litigating the meritorious motions for sanctions based on spoliation would adequately remedy plaintiffs’ injury. See, e.g., Moody v. CSX Transp., Inc., 271 F. Supp. 3d 410 , 432 (W.D.N.Y. 2017) (finding adverse inference appropriate where defendants intentionally lost material evidence); Ottoson v. SMBC Leasing and Finance, Inc., 268 F. Supp. 3d 570 , 584 (S.D.N.Y. 2017) (granting an adverse inference instruction where plaintiff “has acted willfully or in bad faith” in [*15] violation of her duty to preserve certain emails); First Fin. Sec., Inc. v. Freedom Equity Grp., LLC, No. 15-CV-1893-HRL, 2016 U.S. Dist. LEXIS 140087 , [2016 BL 337069], 2016 WL 5870218 (N.D. Cal. Oct. 7, 2016) (imposing adverse inference instruction for intentional deletion of text messages and awarding plaintiffs attorneys fees incurred in bringing sanctions motions). The Court has also carefully considered whether this is the rare case in which terminating sanctions are merited, as plaintiffs have urged. See Dkt. 237. At the present time, the Court’s judgment is that such extreme sanctions are not warranted, although further acts of spoliation and/or other discovery abuses could produce a different result.

The footnotes in these last paragraphs are interesting. Footnote 8 explains that “The Court defers decision on the precise formulation of the adverse inference instruction until closer to trial.” That means it could become a mandatory presumption, or merely permissive. Footnote 9 acknowledges that there may be more discovery misconduct in the works. The court noted it could still strike all defenses, if the conduct continues, and save everyone the cost of a trial.

Conclusion

Even with just a permissive presumption, the case at this point will almost certainly be won by Janie Hendrix’ company, Experience Hendrix, L.L.C.. Experience Hendrix, LLC v. Pitsicalis. Yet another loss for Jimi’s brother, Leon, in a long list of losses. Another injunction and businesses shut-down, but for how long? The Estate and L.L.C. have won so many times before. Yet they keep coming back. Is this yet another Pyrrhic Victory in a long line of pointless litigation? How long before the next suit? Some things are just beyond Law’s reach. Purple Haze.

Purple Haze

Purple haze all in my brain
Lately things just don’t seem the same
Actin’ funny, but I don’t know why
‘Scuse me while I kiss the sky

Purple haze all around
Don’t know if I’m comin’ up or down
Am I happy or in misery?
Whatever it is, that girl put a spell on me

Help me help me
Oh no no… No

Yeah
Purple haze all in my eyes
Don’t know if it’s day or night
You’ve got me blowin, blowin my mind
Is it tomorrow or just the end of time?

No, help me aw yeah! Oh no no oh help me…

When you are the best in the world at something, like Jimi Hendrix was at guitar playing, and when you are still famous and admired by millions fifty years after your death, there will be profiteers around. When you add sibling rivalry and family resentments to the mix, then the trouble goes from bad to worse.

The Hendrix family saga, and this lawsuit, are tragedies. So too is the destruction of evidence and this Sanctions Order. It is part of his guitar star legend. Jimi Hendrix’ boy genius was born out of a troubled childhood and family. Diamond in the rough. Bigger than life. Exploded with art, fame and fortune in just three years. Dead at age 27 of an overdose. The day he was born the moon turned a fire red, “Lord, the gypsy was right!”

The greatest guitarist of all time was a Phoenix – tragic, fiery, short-lived, but beautiful and spell-binding too. Where will musical genius appear like that again?



Second Circuit Uses “Klipsch” to Speak Loud and Clear on e-Discovery Misconduct

September 23, 2018

Klipsch makes some of the best speakers in the world, especially their high-end Klipschorn speaker series shown here. The Second Court of Appeals used a Klipsch in a dispute recently to sound an alarm about e-discovery abuse and how it will not be tolerated. Klipsch Group, Inc. v. ePRO E-Commerce Ltd., 880 F. 3d 620 (2d Cir, 2018). The unanimous opinion written by Circuit Judge Gerard E. Lynch upheld the lower court’s sanctions against Klipch’s adversary in the suit, ePro. The adverse inference sanction entered means that Klipsch will now almost certainly win the case.

The Second Circuit also affirmed the full amount of monetary sanctions, $2,680,000. Klipsch was compensated for almost* all of the additional discovery efforts occasioned by ePRO’s misconduct. ePro was also required to immediately secure payment of the full $5 Million amount of any future judgment against them, which includes damage trebling and fees. I know that sounded great to Klipsch and their attorneys. Especially since they had a four day evidentiary hearing on their spoliation motion to get there.

* Somewhat sadly for Klipsch, however, as the third footnote explained, under the lower court’s opinion all of Klipsch’s fees and costs incurred were not awarded. There was another $300,000 or so that was not included, only because Klipsch could not produce proper credentials for one of the billing attorneys. Ouch. That is extremely rare and odd. Details are so very important.

The Second Circuit Court’s Rare Ruling on e-Discovery and Sanctions

The Klipsch opinion has language that will carry great weight, not only because it makes sense, but also because it is from a United States Court of Appeals panel. There are not that many appeals court opinions on sanctions or e-discovery so this is a rare and important opinion. It will certainly encourage more courts to do as the trial judge did here and sanction offending parties. It is also controlling law for all federal courts in its jurisdiction, which is Vermont, Connecticut and the all important New York.

In the rest of this blog I will let the language of the Klipsch opinion speak for itself with a few of my favorites selections. Klipsch Group, Inc. v. ePRO E-Commerce Ltd., 880 F. 3d 620 (2d Cir, 2018). The first quote from Circuit Judge Gerard Lynch’s opinion lays out the background.

In the course of defending against claims that it sold counterfeit products, defendant-appellant ePRO E-Commerce Limited (“ePRO”) engaged in persistent discovery misconduct: it failed to timely disclose the majority of the responsive documents in its possession, restricted a discovery vendor’s access to its electronic data, and failed to impose an adequate litigation hold even after the court directed it to do so, which omission allowed custodians of relevant electronic data to delete thousands of documents and significant quantities of data, sometimes permanently. As a result, the United States District Court for the Southern District of New York (Vernon S. Broderick, J.) concluded that ePRO had willfully engaged in spoliation. It accordingly granted in substantial part plaintiff-appellee Klipsch Group, Inc.’s (“Klipsch”) motion for discovery sanctions, including a $2.7 million monetary sanction to compensate Klipsch for its corrective discovery efforts and a corresponding asset restraint in that amount, permissive and mandatory jury instructions, and an additional $2.3 million bond to preserve Klipsch’s ability to recover damages and fees at the end of the case. . . .

We find no error in the district court’s factual findings, and we conclude that the monetary sanctions it awarded properly compensated Klipsch for the corrective discovery efforts it undertook with court permission in response to ePRO’s misconduct. In particular, we emphasize that discovery sanctions should be commensurate with the costs unnecessarily created by the sanctionable behavior. A monetary sanction in the amount of the cost of discovery efforts that appeared to be reasonable to undertake ex ante does not become impermissibly punitive simply because those efforts did not ultimately uncover more significant spoliation and fraud, or increase the likely damages in the underlying case. The district court’s orders imposing sanctions *624 are accordingly AFFIRMED in all respects.

The next quote gives you a good glimpse into the degree of frustration that recalcitrant attorneys who excel in gamesmanship can engender, even in an appellate court panel located in New York City:

[T]he history of the case makes clear that the sanctions and fees awarded in this case were carefully limited to costs Klipsch incurred in direct response to ePRO’s misconduct. Klipsch obtained approval from the magistrate judge prior to each of its substantive efforts, and in each case, that approval was given only after ePRO had already squandered an opportunity to correct its own errors.

For example, ePRO’s failure to implement a litigation hold was first discovered in March 2013, during Klipsch’s first round of depositions with ePRO employees, but ePRO was not sanctioned at that time, nor was Klipsch given carte blanche to explore ePRO’s files. Instead, ePRO was permitted to hire its own discovery expert to correct the error, which resulted in the production of substantial additional discovery. Klipsch then spent approximately $550,000 on a second round of depositions occasioned by that late production.

Paul Klipsch

It is evident that the district court did not detect any abusive conduct on the part of Klipsch, such as the piling on of discovery demands and investigatory initiatives in order to burden its *633 adversary with wasteful expenses, motions practice, and sanctions. ePRO does not appear to contest the reasonableness of permitting Klipsch to take those remedial depositions, nor can it plausibly assert that Klipsch would have insisted on doing so even if ePRO’s initial production had been complete or timely. And only in March 2014, after ePRO had repeatedly shown itself to be an untrustworthy participant in the discovery process, did the magistrate judge determine that Klipsch was “fully justified” in seeking to undertake an independent forensic examination. Joint App. at 1187.

Because the costs for which Klipsch is being compensated were reasonably incurred in direct response to ePRO’s misconduct, we cannot conclude that the district court abused its discretion by requiring ePRO to pay monetary sanctions in that amount.[7]

The defendant ePro then tried the “proportionality card” arguing that the multi-million dollar punitive damages exceeded the amount at issue in the case. That did not work. The Second Circuit explained that is not how proportionality works in sanctions. It has no bearing to the value of the case itself, just the amount of additional reasonable costs incurred because of the unreasonable conduct to be sanctioned. One party should not be damaged by the unreasonable conduct of the other. The offended party, here Klipsch, should, in effect, be indemnified from all of the burden and expense incurred because of what the offending party did or did not do. They are the innocent party. Here is how the court put it.

In sum, we see nothing in ePRO’s proportionality arguments compelling us to conclude that the district court abused its discretion by awarding full compensation for efforts that were ex ante a reasonable response to ePRO’s own evasive conduct.[9] The proportionality that matters here is that the amount of the sanctions was plainly proportionate — indeed, it was exactly equivalent — to the costs ePRO inflicted on Klipsch in its reasonable efforts to remedy ePRO’s misconduct.

The Second Court was cognizant of the issues and problems concerning electronic discovery. They had been briefed on the potential for misuses of spoliation sanctions motions. Although this was not at all present in the Klipsch, Judge Lynch address this concern near the end of the opinion.

Nothing that we say in this opinion should be taken as condoning excessive and disproportionate discovery demands, countenancing the tactical use of discovery sanction motions to inflict gratuitous costs on adversaries, or derogating from the responsibility *636 of district courts to ensure that litigation proceeds in a responsible and cost-efficient manner. See Fed. R. Civ. P. 1 (directing that the Rules of Civil Procedure “should be construed, administered, and employed by the court and the parties to secure the just, speedy, and inexpensive determination of every action”) (emphasis added); Fed. R. Civ. P. 26(b)(1) (scope of discovery should be “proportional to the needs of the case, considering [inter alia] the amount in controversy”). If it turns out, as the district court has estimated, that the amount of actual damages in this case is modest in relation to the costs spent on the litigation, that would be a highly regrettable outcome.

But the question before the district court, and before us, is which party should be held responsible for those costs. ePRO does not ever contend that Klipsch’s initial discovery demands were unreasonable or disproportionate to the merits of the case. Nor does it seriously argue that the magistrate judge erred in allowing Klipsch to take the steps it took to remedy ePRO’s refusal to comply with those demands. The district court reasonably concluded, after a full and fair hearing, that it was ePRO’s noncompliance with its legal obligations that occasioned the excessive costs in this case, and we find no reason why ePRO should not therefore be required to pay them.

I leave you with a short video biography of Paul Klipch, a recognized audio engineer leader whom I have long admired.


Elusion Random Sample Test Ordered Under Rule 26(g) in a Keyword Search Based Discovery Plan

August 26, 2018

There is a new case out of Chicago that advances the jurisprudence of my sub-specialty, Legal Search. City of Rockford v. Mallinckrodt ARD Inc., 2018 WL 3766673, Case 3:17-cv-50107 (N.D. Ill., Aug. 7, 2018). This discovery order was written by U.S. Magistrate Judge Iain Johnston who entitled it: “Order Establishing Production Protocol for Electronically Stored Information.” The opinion is both advanced and humorous, destined to be an oft-cited favorite for many. Thank you Judge Johnston.

In City of Rockford an Elusion random sample quality assurance test was required as part of the parties discovery plan to meet the reasonable efforts requirements of Rule 26(g). The random sample procedure proposed was found to impose only a proportional, reasonable burden under Rule 26(b)(1). What makes this holding particularly interesting is that an Elusion test is commonly employed in predictive coding projects, but here the parties had agreed to a keyword search based discovery plan. Also see: Tara Emory, PMP, Court Holds that Math Matters for eDiscovery Keyword Search,  Urges Lawyers to Abandon their Fear of Technology (Driven, (August 16, 2018) (“party using keywords was required to test the search effectiveness by sampling the set of documents that did not contain the keywords.”)

The Known Unknowns and Unknown Unknowns

Judge Johnston begins his order in City of Rockford with a famous quote by Donald Rumseld, a two-time Secretary of Defense.

“[A]s we know there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. . .”
Donald Rumseld

For those not familiar with this famous Known Knowns quip, here is a video of the original:

Here the knowledge logic is spelled out in a chart, since I know we all love that sort of thing. Deconstructing Rumsfeld: Knowledge and Ignorance in the Age of Innovation (Inovo 5/114).

Anybody who does complex investigations is familiar with this problem. Indeed, you can argue this insight is fundamental to all of science and experimental method. Logan, David C. (March 1, 2009). “Known knowns, known unknowns, unknown unknowns and the propagation of scientific enquiry”, Journal of Experimental Botany 60 (3). pp. 712–4. [I have always wanted to quote a botany journal.]

How do you deal with the known unknowns and the unknown unknowns, the information that we don’t even know that we don’t know about? The deep, hidden information that is both obtuse and rare. Information that is hard to retrieve and harder still to prove does not exist at all. Are you chasing something that might not exist? Something unknown because nonexistent? Such as an overlooked Highly Relevant document? (The stuff of nightmares!) Are you searching for nothing? Zero? If you find it, what does that mean? What can be known and what can never be known? Scientists, investigators and the Secretary of Defense alike all have to ponder these questions and all want to use the best tools and best people possible to do so. See: Deconstructing Rumsfeld: Knowledge and Ignorance in the Age of Innovation (Inovo 5/114).

Seeking Knowledge of the Unknown Elusion Error Rate

These big questions, though interesting, are not why Judge Johnston started his opinion with the Rumseld quote. Instead, he used the quote to emphasize that new e-discovery methods, namely random sampling and statistical analysis, can empower lawyers to know what they never did before. A technical way to know the known unknowns. For instance, a way to know the number of relevant documents that will be missed and not produced: the documents that elude retrieval.

As the opinion and this blog will explain, you can do that, know that, by using an Elusion random sample of the null-set. The statistical analysis of the sample transforms the unknown quantity to a known (subject to statistical probabilities and range). It allows lawyers to know, at least within a range, the number of relevant documents that have not been found. This is a very useful quality assurance method that relies on objective measurements to demonstrate success of your project, which here is information retrieval. This and other random sampling methods allow for the calculation of Recall, meaning the percent of total relevant documents found. This is another math-based, quality assurance tool in the field of information retrieval.

One of the main points Judge Johnston makes in his order is that lawyers should embrace this kind of technical knowledge, not shy away from it. As Tara Emory said in her article, Court Holds that Math Matters for eDiscovery Keyword Search:

A producing party must determine that its search process was reasonable. In many cases, the best way to do this is with objective metrics. Producing parties often put significant effort into brainstorming keywords, interviewing witnesses to determine additional terms, negotiating terms with the other party, and testing the documents containing their keywords to eliminate false positives. However, these efforts often still fail to identify documents if important keywords were missed, and sampling the null set is a simple, reasonable way to test whether additional keywords are needed. …

It is important to overcome the fear of technology and its related jargon, which can help counsel demonstrate the reasonableness of search and production process. As Judge Johnston explains, sampling the null set is a process to determine “the known unknown,” which “is the number of the documents that will be missed and not produced.” Judge Johnson disagreed with the defendants’ argument “that searching the null set would be costly and burdensome.” The Order requires Defendants to sample their null set at a 95% +/-2% margin of error (which, even for a very large set of documents, would be about 2,400 documents to review).[4] By taking these measures—either with TAR or with search terms, counsel can more appropriately represent that they have undertaken a “reasonable inquiry” for relevant information within the meaning of FRCP 26(g)(1).

Small Discovery Dispute in an Ocean of Cooperation

Judge Johnston was not asked to solve the deep mysteries of knowing and not knowing in City of Rockford. The parties came to him instead with an interesting, esoteric discovery dispute. They had agreed on a great number of things, for which the court profusely congratulated them.

The attorneys are commended for this cooperation, and their clients should appreciate their efforts in this regard. The Court certainly does. The litigation so far is a solid example that zealous advocacy is not necessarily incompatible with cooperation. The current issue before the Court is an example of that advocacy and cooperation. The parties have worked to develop a protocol for the production of ESI in this case, but have now reached an impasse as to one aspect of the protocol.

The parties disagreed on whether to include a document review quality assurance test in the protocol. The Plaintiffs wanted one and the Defendants did not. Too burdensome they said.

To be specific, the Plaintiffs wanted a test where the efficacy of any parties production would be tested by use of an Elusion type of Random Sample of the documents not produced. The Defendants opposed any specific test. Instead, they wanted the discovery protocol to say that if the receiving party had concerns about the adequacy of the producing party’s efforts, then they would have a conference to address the concerns.

Judge Johnston ruled for the plaintiff in this dispute and ordered a  random elusion sample to be taken after the defendant stopped work and completed production. In this case it was a good decision, but should not be routinely required in all matters.

The Stop Decision and Elusion Sample

One of the fundamental problems in any investigation is to know when you should stop the investigation because it is no longer worth the effort to carry on. When has a reasonable effort been completed? Ideally this happens after all of the important documents have already been found. At that point you should stop the effort and move on to a new project. Alternatively, perhaps you should keep on going and look for more? Should you stop or not?

In Legal Search we all this the “Stop Decision.” Should you conclude the investigation or continue further AI training rounds and other search. As explained in the e-Discovery Team TAR Course:

The all important stop decision is a legal, statistical decision requiring a holistic approach, including metrics, sampling and over-all project assessment.You decide to stop the review after weighing a multitude of considerations. Then you test your decision with a random sample in Step Seven.

See: TAR Course: 15th Class – Step Seven – ZEN Quality Assurance Tests.

If you want to go deeper into this, then listen in on this TAR Course lecture on the Stop decision.

____________

Once a decision is made to Stop, then a well managed document review project will use different tools and metrics to verify that the Stop decision was correct. Judge Johnston in City of Rockford used one of my favorite tools, the Elusion random sample that I teach in the e-Discovery Team TAR Course. This type of random sample is called an Elusion sample.

Judge Johnston ordered an Elusion type random sample of the null set in City of Rockford. The sample would determine the range of relevant documents that likely eluded you. These are called False Negatives. Documents presumed Irrelevant and withheld that were in fact Relevant and should have been produced. The Elusion sample is designed to give you information on the total number of Relevant documents that were likely missed, unretrieved, unreviewed and not produced or logged. The fewer the number of False Negatives the better the Recall of True Positives. The goal is to find, to retrieve, all of the Relevant ESI in the collection.

Another way to say the same thing is to say that the goal is Zero False Negatives. You do not miss a single relevant file. Every file designated Irrelevant is in fact not relevant. They are all True Negatives. That would be Total Recall: “the Truth, the Whole Truth …” But that is very rare and some error, some False Negatives, are expected in every large information retrieval project. Some relevant documents will almost always be missed, so the goal is to make the False Negatives inconsequential and keep the Elusion rate low.

Here is how Judge Iain Johnston explained the random sample:

Plaintiffs propose a random sample of the null set. (The “null set” is the set of documents that are not returned as responsive by a search process, or that are identified as not relevant by a review process. See Maura R. Grossman & Gordon v. Cormack, The Grossman-Cormack Glossary of Technology-Assisted Review, 7 Fed. Cts. L. Rev. 1, 25 (2013). The null set can be used to determine “elusion,” which is the fraction of documents identified as non-relevant by a search or review effort that are, in fact, relevant. Elusion is estimated by taking a random sample of the null set and determining how many or what portion of documents are actually relevant. Id. at 15.) FN 2

Judge Johnston’s Footnote Two is interesting for two reasons. One, it attempts to calm lawyers who freak out when hearing anything having to do with math or statistics, much less information science and technology. Two, it does so with a reference to Fizbo the clown.

The Court pauses here for a moment to calm down litigators less familiar with ESI. (You know who you are.) In life, there are many things to be scared of, including, but not limited to, spiders, sharks, and clowns – definitely clowns , even Fizbo. ESI is not something to be scared of. The same is true for all the terms and jargon related to ESI. … So don’t freak out.

Accept on Zero Error for Hot Documents

Although this is not addressed in the court order, in my personal view, no False Negatives, iw – overlooked  documents – are acceptable when it comes to Highly Relevant documents. If even one document like that is found in the sample, one Highly Relevant Document, then the Elusion test has failed in my view. You must conclude that the Stop decision was wrong and training and document review must recommence. That is called an Accept on Zero Error test for any hot documents found. Of course my personal views on best practice here assume the use of AI ranking, and the parties in City of Rockford only used keyword search. Apparently they were not doing machine training at all.

The odds of finding False Negatives, assuming that only a few exist (very low prevalence) and the database is large, are very unlikely in a modest sized random sample. With very low prevalence of relevant ESI the test can be of limited effectiveness. That is an inherent problem with low prevalence and random sampling. That is why statistics have only limited effectiveness and should be considered part of a total quality control program. See Zero Error Numerics: ZEN. Math matters, but so too does good project management and communications.

The inherent problem with random sampling is that the only way to reduce the error interval is to increase the size of the sample. For instance, to decrease the margin of error to only 2% either way, a total error of 4%, a random sample size of around 2,400 documents is needed. Even though that narrows the error rate to 4%, there is still another error factor of the Confidence Level, here at 95%. Still, it is not worth the effort to review even more sample documents to reduce that to a 99% Level.

Random sampling has limitations in low prevalence datasets, which is typical in e-discovery, but still sampling can be very useful. Due to this rarity issue, and the care that producing parties always take to attain high Recall, any documents found in an Elusion random sample should be carefully studied to see if they are of any significance. We look very carefully at any new documents found that are of a kind not seen before. That is unusual. Typically  any relevant documents found by random sample of the elusion set are of a type that have been seen before, often many, many times before. These “same old, same old” type of documents are of no importance to the investigation at this point.

Most email related datasets are filled with duplicative, low value data. It is not exactly irrelevant noise, but it is not a helpful signal either. We do not care if we  get all of that kind of merely relevant data. What we really want are the Hot Docs, the high value Highly Relevant ESI, or at least Relevant and of a kind not seen before. That is why the Accept On Zero Error test is so important for Highly Relevant documents.

The Elusion Test in City of Rockford 

In City of Rockford Judge Johnston considered a discovery stipulation where the parties had agreed to use a typical keyword search protocol, but disagreed on a quality assurance protocol. Judge Johnston held:

With key word searching (as with any retrieval process), without doubt, relevant documents will be produced, and without doubt, some relevant documents will be missed and not produced. That is a known known. The known unknown is the number of the documents that will be missed and not produced.

Back to the False Negatives again, the known unknown. Judge Johnston continues his analysis:

But there is a process by which to determine that answer, thereby making the known unknown a known known. That process is to randomly sample the nullset. Karl Schieneman & Thomas C. Gricks III, The Implications of Rule26(g) on the Use of Technology-Assisted Review, 2013 Fed. Cts. L. Rev. 239, 273 (2013)(“[S]ampling the null set will establish the number of relevant documents that are not being produced.”). Consequently, the question becomes whether sampling the null set is a reasonable inquiry under Rule 26(g) and proportional to the needs of this case under Rule 26(b)(1).

Rule 26(g) Certification
Judge Johnston takes an expansive view of the duties placed on counsel of record by Rule 26(g), but concedes that perfection is not required:

Federal Rule of Civil Procedure 26(g) requires all discovery requests be signed by at least one attorney (or party, if proceeding pro se). Fed. R. Civ. P. 26(g)(1). By signing the response, the attorney is certifying that to the best of counsel’s knowledge, information, and belief formed after a reasonable inquiry, the disclosure is complete and correct at the time it was made. Fed. R. Civ. P. 26(g)(1)(A). But disclosure of documents need not be perfect. … If the Federal Rules of Civil Procedure were previously only translucent on this point, it should now be clear with the renewed emphasis on proportionality.

Judge Johnston concludes that Rule 26(g) on certification applies to require the Elusion sample in this case.

Just as it is used in TAR, a random sample of the null set provides validation and quality assurance of the document production when performing key word searches.  Magistrate Judge Andrew Peck made this point nearly a decade ago. See William A. Gross Constr. Assocs., 256 F.R.D. at 135-6 (citing Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 262 (D. Md. 2008)); In re Seroquel Products Liability Litig., 244 F.R.D. 650, 662 (M.D. Fla. 2007) (requiring quality assurance).

Accordingly, because a random sample of the null set will help validate the document production in this case, the process is reasonable under Rule 26(g).

Rule 26(b)(1) Proportionality

Judge Johnston considered as a separate issue whether it was proportionate under Rule 26(b)(1) to require the elusion test requested. Again, the court found that it was in this large case on the pricing of prescription medication and held that it was proportional:

The Court’s experience and understanding is that a random sample of the null set will not be unreasonably expensive or burdensome. Moreover and critically, Defendants have failed to provide any evidence to support their contention. Mckinney/Pearl Rest. Partners, L.P. v. Metro. Life Ins. Co., 322 F.R.D. 235, 242 (N.D.Tex. 2016) (party required to submit affidavits or offer evidence revealing the nature of the burden)
Once again we see a party seeking protection from having to do something because it is so burdensome then failing to present actual evidence of burden. We see this a lot lately. Responding Party’s Complaints of Financial Burden of Document Review Were Unsupported by the Evidence, Any Evidence (e-Discovery Team, 8/5/18);

Judge Johnston concludes his “Order Establishing Production Protocol for Electronically Stored Information” with the following:

The Court adopts the parties’ proposed order establishing the production protocol for ESI with the inclusion of Plaintiffs’ proposal that a random sample of the null set will occur after the production and that any responsive documents found as a result of that process will be produced. Moreover, following that production, the parties should discuss what additional actions, if any, should occur. If the parties cannot agree at that point, they can raise the issue with the Court.

Conclusion

City of Rockford is important because it is the first case to hold that a quality control procedure should be used to meet the reasonable efforts certification requirements of Rule 26(g). The procedure here required was a random sample Elusion test with related, limited data sharing. If this interpretation of Rule 26(g) is followed by other courts, then it could have a big impact on legal search jurisprudence. Tara Emory in her article, Court Holds that Math Matters for eDiscovery Keyword Search goes so far as to conclude that City of Rockford stands for the proposition that “the testing and sampling process associated with search terms is essential for establishing the reasonableness of a search under FRCP 26(g).”

The City of Rockford holding could persuade other judges and encourage courts to be more active and impose specific document review procedures on all parties, including requiring the use of sampling and artificial intelligence. The producing party cannot always have a  free pass under Sedona Principle Six. Testing and sampling may well be routinely ordered in all “large” document review cases in the future.

It will be very interesting to watch how other attorneys argue City of Rockford. It will continue a line of cases examining methodology and procedures in document review. See eg., William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134 (S.D.N.Y. 2009) (“wake-up call” for lawyers on keyword search); Winfield v. City of New York (SDNY, Nov. 27, 2017), where Judge Andrew Peck considers methodologies and quality controls of the active machine learning process. Also see Special Master Maura Grossman’s Order Regarding Search Methodology for ESI, a validation Protocol for the Broiler Chicken antitrust cases.

The validation procedure of an Elusion sample in City of Rockford is just one of many possible review protocols that a court could impose under Rule 26(g). There are dozens more, including whether predictive coding should be required. So far, courts have been reluctant to order that, as Judge Peck explained in Hyles:

There may come a time when TAR is so widely used that it might be unreasonable for a party to decline to use TAR. We are not there yet.

Hyles v. New York City, No. 10 Civ. 3119 (AT)(AJP), 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016):

Like a kid in the backseat of the car, I cannot help but ask, are we there yet? Hyles was published over two years ago now. Maybe some court, somewhere in the world, has already ordered a party to do predictive coding against their will, but not to our knowledge. That is a known unknown. Still, we are closer to “There” with the City of Rockford’s requirement of an Elusion test.

When we get “there,” and TAR is finally ordered in a case, it will probably arise in a situation like City of Rockford where a joint protocol applicable to all parties is involved. That is easier to sell than a one-sided protocol. The court is likely to justify the order by Rule 26(g), and hold that it requires all parties in the case to use predictive coding. Otherwise, they will not meet the  reasonable effort burdens of Rule 26(g). Other rules will be cited too, of course, including Rule 1, but Rule 26(g) is likley to be key.

____________

___

 

____

 

 


%d bloggers like this: