e-Discovery Team

This is a continuation of Journey into the Borg Hive: Part One and Part Two of a SciFi Saga. This series is about a legal search project set sometime in the not too distant future. For a look at fifty years from now check out A Day in the Life of a Discovery Lawyer in the Year 2062: a Science Fiction Tribute to Ray Bradbury.

16(b) Hearing of the Future

The 16(b) hearing started when a buzzer went off and we all stood. The magistrate judge entered the room in his black robe, sat down at the bench about three feet above us, and started looking at his computer screen while the case was called. Everyone sat down. We then took turns making our appearances of record. One of the mikes had the volume set too low and that took a second for the clerk to adjust. Every word we said was being recorded. The judge occasionally glanced down and smiled as the attorneys introduced themselves and the client representatives. The 16(b) hearing notice required that they attend, along with each side’s vendor expert. This was one of two 16(b) hearings for the case, the one devoted to discovery issues. After the initial formalities the judge went right to it.

The judge noted that our vendor was new to him, but he had looked them up, and expressed an interest in their approach. Siri had come to the hearing and he greeted her as if he had met her before. Although some judges had commented on the Borg approach, all negatively, he had not. He did not address the merits of their approach, but instead moved on to China Space’s vendor. He had met their lead expert before in other cases and generally knew how their software worked. It was a popular type of multimodal predictive coding software. The judge chatted with him for a minute about an upcoming event. This judge did a fair amount of speaking at e-discovery invites, and unlike some, he got his invitations on merit. That meant he knew e-discovery better than the trial lawyers, and also knew me and my e-discovery lawyer counterpart for China Space. The judge  was pleasant to us as usual and threw in a few comic remarks at our expense. But he also made some flattering statements about us for the benefit of our clients.

He noted that all technical issues had been agreed to in the stipulation we filed last week. We had agreed to what was beginning to be called the mutual QT approach. QT stood for quasi-transparency. We agreed to limit our search disclosure of irrelevant documents to the random sample sets taken before and after the review as part of the quality assurance tests, including disclosure of the null-set of excluded documents. We did not have to disclose quality control data, nor the initial seed set, nor the subsequent training sets. Only the metrics of those searches had to be disclosed, along with all relevant documents at periodic production points designed to better control work product protection.

That meant the disclosure of irrelevant documents by both sides was very limited. Both sides also had to make only limited work-product type disclosures of their search methods. I made sure of that for my own reasons. Too much mandatory disclosure would destroy my plan to save the client from the Borg approach. Predictive coding on the QT suited me just fine in this case.

Both sides also reserved rights to object to the reasonability of the other sides search efforts. This was standard and anticipated. The judge just noted it and said he hoped our approaches worked and he did not have to have later hearings on that. He looked directly at Siri when he said that. I took this as a signal he had doubts about her system, much like I did. But I was there with her, so he moved on to other issues quickly. Still, I knew if the review screwed up, as I was afraid, he would not hesitate to impose sanctions. This judge was well-known for that. They would no be dispositive, but would likely involve monetary sanctions. At the very least we might have to do a redo, which can in itself be very expensive.

The judge went out of his way to thank us for reaching agreements. He said just yesterday he spent an hour hearing quality assurance test arguments. The core of the argument in the other case was the confusion matrix. Four of us in the room knew what he was talking about. The discovery partners and associates in my firm would too. I was always telling them that you need to understand the confusion matrix to avoid getting lost in it.

To our surprise the Judge turned on his projector and a standard GC Version of the Matrix came up. He explained he had it from the hearing yesterday. He then used the diagram to help ask us a few questions regarding the mutual quality assurance tests we had agreed to. The Judge knew this was critical to avoiding reasonable search arguments later on. He said he wanted to be sure we had covered all of the bases.

CONFUSION MATRIX

Accuracy = 100% – Error = (TP + TN) / (TP + TN + FP + FN)
Error = 100% – Accuracy = (FP + FN) / (TP + TN + FP + FN)
Elusion = 100% – Negative Predictive Value = FN / (FN + TN)
Fallout = False Positive Rate = 100% – True Negative Rate = FP / (FP + TN)
Negative Predictive Value = 100% – Elusion = TN / (TN + FN)
Precision = Positive Predictive Value = TP / (TP + FP)
Prevalence = Yield = Richness = (TP + FN) / (TP + TN + FP + FN)
Recall = True Positive Rate = 100% – False Negative Rate = TP / (TP + FN)

Our stipulations on the Matrix tests were somewhat general and the Judge wanted to know exactly how we had resolved the details that so bedeviled his hearing yesterday. I don’t think he had ruled yet in the other case.

Making Up the Law of the Case

We were very careful in our answers, well aware that our explanations would become binding agreements enforceable by court order, including contempt. That’s why the microphones were on and everything was recorded. For all I know the video cameras were also on. I had forgotten to check. Still, since I talked about this damned Matrix all of the time, I enjoyed a chance to talk shop with a judge who really understood. That was still a rarity, although there were many more knowledgeable judges around now than in the old days.

A couple of times I joked with the e-discovery lawyer and her vendor on the other side when the judge asked us a few questions that we had not even discussed before. We were quick enough on our feet and able to agree and clarify a few points with the Judge’s help. After a while there is no pressure and you get used to making law, especially here, where it was just for this one case.

How To Pass An Elusion Exam

The Judge seemed primarily interested in the details of the Elusion Test we had agreed to. Both sides were, of course, going to use the same tests to determine when their search would be concluded, at least for the all important first round of discovery. Very few of my cases actually engaged in the second, much less third round provided for in this Judge’s standard order. The quality assurance tests were not dispositive, and both sides always still reserved rights to object, even if the search passed all tests. They just could not object right away, but would have to wait until they had the full production.

Basically the Judge wanted to know what results would be considered a passing grade in the Elusion Test. We said although there was no specific maximum number, any higher than which would be considered a test failure and automatically trigger another round, we both expected the percentage of False Negatives (FN) to be very low. Our pass-fail test metrics had more to do with the types of FN found. We had agreed to a qualitative version of Accept on Zero Error. We agreed that the Machine Learning would be considered a success, and further rounds of training would be unnecessary, if two conditions were met: (1) none of the FN were highly relevant documents; and, (2) any documents demonstrating a new type of relevance, a new type of relevant document not seen before, would have to be ranked as only of marginal relevance.

We did not care about any missed relevant documents that were just more of the same. So much of e-discovery was like that. The same or similar documents and communications that were not technically duplicates would keep showing up time and again. The redundancy in today’s collections of documents was ridiculous. I wished that the vendors would finally agree amongst themselves about a new ISO standard for extended deduplication. It was as if we were all drowning in a mountain of unoriginal books, most of which just plagiarized off each other.

The judge understood the relevancy ranking distinctions we were making. That our Elusion Test pass-fail criteria primarily had to do with the importance, if any, of the particular FN documents to the merits of the case. The only kind of document assigned to Accept on Zero Error were highly relevant documents. I joked that I hoped we would not have to return to argue about whether an FN document was highly relevant or not, but noted that this kind of dispute was not uncommon.

What Is Relevant?

The judge then looked at the trial lawyers, who he knew would argue the relevancy issues, and told them with a mock-stern look that he did not want to see them again. They all laughed. Then the judge said he was satisfied with the electronic portions of the discovery stipulation and moved on to some deposition and interrogatory issues. After that he asked if counsel was prepared to make arguments on the one stipulated issue remaining for resolution – the relevancy issues triggered by the first production of documents, the random sample of 2,401 documents that each side had already produced to the other.

Of course they were ready, and the judge told the plaintiff’s lawyer to go first. He made an argument as to the relevancy of three documents that our side considered to be irrelevant to any of the genuine issues of material fact in the case. He also made a counter-argument to our side’s position that two of the documents he considered irrelevant were actually relevant. He forcefully argued that both of these documents had no probative value to any possible issue in the case. This was a typical relevancy argument. It focused on the  five documents in the initial random sample that the parties could not agree upon. Per local rule we exchanged our sample sets, with privilege only withheld and logged. A dispute on only five out of four thousand, eight hundred, and two documents was a pretty good achievement. Both sides had worked hard to get this far.

While the trial boys talked I took control of the court’s media display panels for attorneys and displayed the five documents at the appropriate times, which got pretty challenging when the judge started asking questions. The displays included the highlighting that counsel had agreed to. By the way, that damned agreement took over four hours to reach, since both sides’ trial counsel wanted to sign off on those important details. They all knew that highlighting could have persuasion consequences. They wanted to make a good first impression.

Next my attorney spoke and argued our positions on relevancy. We mirrored the other side and explained why three of our documents were not really relevant, that they were True Negatives, not False Negatives. Thankfully he knew enough of the vocabulary to use the terms correctly.

The Judge then asked a few questions, mainly probing why one side considered documents to be relevant that the other side had coded as irrelevant. After arguments he took a few minutes to write something on his computer while we all waited in strict silence. I was holding my breath, hoping that he would rule right away so I could start my search. I hated it when some judges made you wait a few months on relevancy issues.

He did not disappoint  The judge ruled against the pro-relevancy arguments of both sides. He agreed with the decisions made by both sides to identify the documents as irrelevant. I was relieved to hear him take a narrow view of relevancy. Then he briefly explained his rationale on the record as to each of the five documents. The judge said he would write a full written order later and looked over to his law clerk and winked. He asked if we had any questions on his bench ruling, for otherwise he would expect us to begin compliance with the ruling immediately. He pointed out that under local rule we now had sixty days to complete the initial cross-production of ESI.

I saw Siri on her cell phone on the way out talking to her document manager, telling him that they could begin the review project. The initial training had already been done, so they could proceed in earnest to begin the full automated review. I told them they could not begin until after we were certain of the initial seed set. We were now certain. The seed set we selected had been fully approved by the court, all three objections had been overruled. Siri knew she could begin and did not even ask me first for confirmation.

To be continued …

This entry was posted on Wednesday, January 23rd, 2013 at 3:34 pm and is filed under Evidence, Lawyers Duties, Metadata, New Rules, Related Legal Webs, Review, Search, Spoliation/Sanctions, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.