This is a continuation of Journey into the Borg Hive: Parts One, Two, Three, Four, Five, Six, Seven and Eight of a SciFi Saga. This series is about a legal search project set sometime in the not too distant future. For a look at fifty years from now check out A Day in the Life of a Discovery Lawyer in the Year 2062: a Science Fiction Tribute to Ray Bradbury. For a bunch of videos on who and what the Borg are, see Never Heard of Star Trek’s Borg?
Distractions Along the Way
The rest of the week was a blur. Two emergencies came in from opposite coasts. One was simple to fix with just a few hours intervention. One was a real mess and clean up took days to plan and implement. Somewhere in there a new review project came in.
I needed to do some ECA on the new project as the local liaison was unavailable. I started the project by creating a custom legal Culling plan for this data set. I also did some of the easy and quick culls unique to this data. Then I pushed the auto-cull button the software developers had added at my request. It automatically identified and segregated all ESI that could not be searched by machine training. That basically means the elimination of most non-text documents. They are either searched separately, or ignored, depending on the facts in dispute.
I always found these sixth-step challenges to be particularly interesting. Or as some of my reviewers would call it, more crap with the top green square, the one on top of the Review Column. They had not yet memorized all of the numbers. But they did know I was not talking about the kind of tech-culling that vendors do, such as deduplication and de-Nisting. I was talking about legal judgment based culling.
I would usually do my culling by using both judgmental and random samples, plus some quick searches. I would also look at the data in various ways using my software’s visual analytics. Those rapidly improving graphic features are what I once heard Jason Baron at Legal Tech call the next big thing in computer assisted review. I was already looking forward to the next generation of software that I saw in a recent private showing. Hang on to your hats folks, the software is going to get insanely good fast.
As I coded a few documents during the ECA for the new project my software’s machine learning functions, which my vendor calls Intelligent Review Technology, automatically kicks in. I could start displaying the probability rankings immediately, if I wanted to. But usually I would wait until I felt, or we felt if it was a team effort, that some kind of critical mass had been reached. The probability rankings were not too helpful at first because there was only a small amount of coded documents. Plus, I did not want the ranking distractions. I did not think the rankings would subconsciously influence my evaluation, but I knew that several information scientists disagreed. There was some interesting research going on now on that topic. Ultimately I think it is a matter of learning, understanding, and experience, just like everything else in life.
Machine Training Workflow
While dealing with these new projects I continued to supervise both of my review teams in the Google project. The Federation team finished first, of course, by three days. Both teams had continued until they reached a point where the number of new relevant documents found was sparse. Another indicator was when the only new documents the machine would find were of the same relevancy type, just more of the same. At that point you know you are probably done. You never know for sure, of course, but that is what the final sample test was for. That is step number six in the standard predictive coding circle chart. Although it is often called the Random QC Test, to be technically correct it is a Quality Assurance test, not control, but that subtle distinction still escapes most people.
Step six will either confirm the reasonability of your review efforts, or not. If you passed, you are done with the iterated searches, and can move on to step-seven in the predictive coding flow chart, Proportional Final Review. In the EDBP, which breaks it down into greater detail, the next step after Computer Assisted Review is Protections. Step Eight.
This has been one of the hardest steps in e-discovery for years.
If you failed the final quality assurance test, which meant your scores were too low or a hot document was missed, or several new types of relevant documents were found of some significance, but not necessarily Hot, then you had to do further rounds of machine training, not to mention other types of search. You would have to return back to the iterated steps Four and Five in the predictive coding circle. My reviewers call that a Bad Robot moment. So far it has only happened to us once.
There are metrics involved in the final random test, and the whole process is much more objective then it ever was in the early 2010s when advanced search first started to catch on. There is no question about that. But there could sometimes still be a tremendous amount of subjectivity involved in the evaluation of the tests. We would often have to seek judicial guidance, even though the legal standard was fairly low, mere reasonable efforts, not best practices. Lawyer argument remains a constant, even as our topics of argument rapidly change. That is the essence of our adversarial system of justice.
The Final Search Reports
Both teams finished the final random sample testing at the same time. We were all afraid that the Borg might get lucky and find our super Hot documents, but they did not. Big sigh of relief. The Confusion Matrix scores had the Federation team ahead of the Borg by 20% to 30% in all eight categories. So did the other measurements we used. It was still all smiles on my team. My reviewers had worked hard and they were glad it was over. They had high hopes they’d get paid, and paid well.
CONFUSION MATRIX
| Truly Non-Relevant | Truly Relevant | |
| Coded Non-Relevant | True Negatives (“TN”) | False Negatives (“FN”) |
| Coded Relevant | False Positives (“FP”) | True Positives (“TP”) |
- Precision = Positive Predictive Value = TP / (TP + FP)
- Recall = True Positive Rate = 100% – False Negative Rate = TP / (TP + FN)
- Elusion = 100% – Negative Predictive Value = FN / (FN + TN)
- Accuracy = 100% – Error = (TP + TN) / (TP + TN + FP + FN)
- Error = 100% – Accuracy = (FP + FN) / (TP + TN + FP + FN)
- Fallout = False Positive Rate = 100% – True Negative Rate = FP / (FP + TN)
- Negative Predictive Value = 100% – Elusion = TN / (TN + FN)
- Prevalence = Yield = Richness = (TP + FN) / (TP + TN + FP + FN)
But what really counted was my weighted Elusion Test. The Borg had missed all of the Hot game-over type documents that we had found. But the Borg Queen, Siri, did not know that yet. She thought she had done ok. She still had no idea there was a shadow team that not only beat her in every category, but blew her away in what really matters, the battle of persuasion.
I couldn’t wait to see her face when she found out at the Mediation that she had missed all of the Hot docs. Normally vendors don’t attend a mediation, but of course Siri considered herself special. She had wiggled her way into attending. Apparently she had convinced the GC that her friendship with the mediator might help. How naive. Our trial lawyers didn’t care because they love an audience. This would be their time to shine. Another pretty face would just make the show even better. These trial lawyers were such hams.
The Mediation
After introductions the distinguished looking mediator gave his usual opening speech: I’m so neutral, I’m so great, with my near divine help you are going to settle your case and settle it today. He did it well, but I’ve heard it so many times before.
Then it was plaintiff’s counsel’s time to speak. She was good. Her allotted thirty minutes turned into forty. It was as polished as any closing statement at trial that you are ever going to hear. She even had me going there from time to time when she would make a particularly persuasive point. Then I would remember the goods we had on China Space and breathe a little sigh of relief. Without our Hot docs this would have been a hard case for Google to win.
China Space had a few surprise documents of its own. Their attorney brandished them in her hands with a paper flurry. Our lead trial man looked them over when she solicitously handed him a copy. He said quite loudly, intentionally interrupting, yeah, we’ve seen these all before. That was a lie of course, we had only seen one. The others were by third parties and could not have been in our collection. There was no way we could have found them. They were really not all that important, especially compared to what we were about to lay on them. The fact that these documents were all they had prompted another big sigh of relief from everyone on our side.
After forty minutes of smooth talking their version of the facts, China Spaces’ attorney finally stopped. At that point we all knew, except for Siri, that victory for Google was now certain. Our client was fairly clean, just as they had told us all along. We all sat back in our chairs, relaxed, smiled and waited for our man to begin. The other side could sense that something was up. Our body language was projecting a confidence level that went through the roof. The senior attorneys on the other side already knew they were screwed. Their young associates look confused. They had that deer in the headlights look.
My head trial lawyer did not disappoint. When he got to the document slap down he had all eyes on the screen. I had the pleasure of advancing the slide to show the key email. It was magnified five times its normal size. You could sense a chill sweep through the other side of the table as they saw it and the additional Hot documents that followed. Ah, victory is sweet. Sports is nothing compared to high stakes litigation.
Of course, we could show no real emotion during the joint session. Still, the older lawyers among us were well-versed in the subtle nods, the looks that sent the message to the other side that we knew we had them by the balls.
Our lead attorney then began to squeeze. The merits of Google’s twenty million dollar counter-claim were extolled. Then he went on about the new claims that would be filed tomorrow if the case did not settle today. He even had a copy of the amended counter-claim and gently set that down in front of opposing counsel. She did not even look down. Instead, she held a practiced, slightly defiant, Mona Lisa smile that gave nothing away. But, her magnificent acting did no good. The experienced lawyers among us on both sides had all been there before. We all knew that China Space was totally screwed. At this point it was only a matter now of how much China Space would have to pay. We had seen their quarterly statements and knew they were flush with cash.
After the speeches the parties adjourned into separate rooms. The mediator then began his shuttle diplomacy. He’d go back and forth from one room to another, and sometimes huddle with just some of the attorneys in another room. Everyone on our side was giddy, we did not have to keep a straight face anymore. Everyone, that is, except for Siri. I could tell that she knew she was in trouble, she just did not yet know how deep. No one was talking about where the exhibits came from and she did not ask.
Let the Celebrations Begin!
The mediation ended ten hours later. It was dark out and we all wanted to go home. But we had to stop by a restaurant first. We had to celebrate the Ten Million Dollar settlement Google had just signed off on. How’s that for a defense verdict?
Payment would be made by China Space to Google’s attorneys within seven days. At that time the mutual releases signed today would be released from escrow and all of the claims and counter-claims dismissed with prejudice. A confidentiality agreement sewed it all up tight.
It was a good day for Google, and they were not shy in heaping praise on my firm. The trial lawyers had done a great job, so had the mediator in hammering home the adverse publicity dangers. He made it seem like ten million was getting off real cheap. Little did he or China Space know that we would have taken seven. We had lied to the mediator all day and told him ten was our absolute bottom line. He was an experienced mediator. He probably knew we were lying, but that’s all part of the game. Too bad he didn’t join us at the party. It was not a swank place, but it sure had its share of interesting characters.
Yes, it was a good day. Good for everyone on Google’s side except for Siri. She had not joined us. On our way out I thought I saw her in the mediator’s parking lot talking to a China Space engineer. She was showing him her special glasses. She was probably taking credit for finding the key documents. That would be the natural assumption. I think she was scouting out her next big project. A natural-born saleswoman that Siri. Too bad her company’s software and methods were flawed.
The Borg Are Defeated
I called Linda the next day at Google. Asked what she wanted me to do about Siri and the refund. She said just deliver the last reports to her and she would take it from there. She was right. It was better for me to stay out of that entirely. I trusted Linda. I knew she wanted to get that money back as much as I did. For once I did not have to do anything.
A week later I got the call. Linda says it’s done. Siri’s attorney had been relatively easy to deal with. The vendor seemed glad to be able to keep 25%. They knew they had screwed up. Google got 75% of their money back. That was outstanding! It meant that my team on a contingency would not only get paid, but they would get a nice bonus too. Before she went on about the details I asked Linda if she’d mind telling my review team herself the good news. I knew that would mean a lot to them. She agreed and we set a time for that afternoon.
At the call Linda laid praise on us all and told the reviewers how it turned out. Then Linda told me something I didn’t know. She said she had checked the flat fee agreement with my law firm. She found a provision in the discovery clauses that provided for a special bonus payment at the discretion of the client, which meant her, Google, in the event that documents were discovered that led to an early and favorable resolution of the case. She had talked it over with the GC and they had decided to exercise their discretion and make the bonus award. I had forgotten that provision, and frankly, did not think it applied because we had not been selected for the review services, the Borg company had. It was a very generous interpretation of the contract.
Then Linda told us the amount of the bonus award to the reviewers and my firm. That put the total fee payment to each of the reviewers at just under $40,000 a piece. The reviewers went crazy and thanked Linda profusely. They had never received such a big payday, and for less than a month’s work. Few people had. My firm would also be pleased, but it was comparatively small potatoes compared to the rest of the fees. Ah well, at least my reviewers got some immediate gratification. Mine, if any, would have to wait until the end of the fiscal year.
Conclusion
Two days later I get a fed-x package in the morning. In it were two first-class tickets to Honolulu and a note. Turns out my Federation review team was going to have a big party this weekend in Hawaii. I was invited, all expenses paid, along with a guest. A nice gesture, and one they knew I could not refuse, especially since they had already bought the tickets. I changed all our plans and got ready for the trip. It’s not every week that you vanquish the Borg. I’ll let you guess who I took with me.
[…] into the Borg Hive: Part Nine – http://bit.ly/Xq7knC (Ralph […]