Remember the child’s card game Go Fish? You know the one, where each player keeps their own cards secret and tries to guess what cards the others have? You try to get all four of the same rank cards. One player asks another if they have a certain card, such as: do you have any Kings? If that player has any Kings, they have to give them to the requesting player. If not, they say: Go Fish, ha ha, you did not guess right, and then it is the next players turn. (By the way, this game is typically played by very young children.) Based upon the requests one player makes, the other players guess what cards they want. They then try to discard those cards in such a way that the player cannot get to them. The whole enjoyment of the game is derived from not knowing for sure what cards the other players have and from keeping your own card-seeking goals secret. It can take quite a while to guess right and get all four suits of the same rank card. The game goes on until one player wins by only holding four-of-a-kind cards. The game is usually won by the person who is the lucky guesser. There is, to put it mildly, not much skill involved, which is why young kids love the game and older kids don’t.
Most Lawyers Do Search as if it were a Game of Go Fish
I submit that the negotiated key word search model prevalent in e-discovery today uses the same guessing game model as Go Fish. The party requesting ESI guesses what key words might produce evidence to support their case. Do you have any emails that use the keyword “King.” It is necessarily a guess as to what keywords to use because the requesting party cannot see the responding party’s cards. Only the responding party sees all of their own cards, and that is as it should be.
The responding party has a right to privacy. They should not be required to give the requesting party the keys to the server room, the whole deck of cards. The requesting party is either suing the responding party, or being sued by the responding party. Either way, the requesting party should not be permitted to enter and search every nook and cranny of their adversary’s inner sanctum. They should not be granted unfettered access to run ever-more-sophisticated search tools to look for something, anything, that might be incriminating. That kind of fishing expedition has long been prohibited by most courts in the United States. See eg.: Omnicare, Inc. v. Mariner Health Care Management Co., 2009 WL 1515609 at *3 (Del.Ch. May 29, 2009); Hedenburg v. Aramark American Food Services, 2007 U.S. Dist. LEXIS 3443 (W.D. Wash. Jan. 17, 2007). The advances of technology should not be permitted to change that rule. No, the rule must remain, but the game itself should change.
The way the game is now often played, the requesting party also keeps their secrets. They do not want to reveal exactly what it is that they are looking for. But this is, I contend, not as it should be. The requesting player is misusing the paper-world work product doctrine to hide their true discovery intentions. They accomplish this by using broad, general requests. That keeps secret what they really want. They claim a work product right to do so to protect their mental impressions and case strategy. Then they go after what they really want by playing the key word guessing game, both before and after the production. Not all requesters play this way on purpose. This explanation assumes that they have done their home work. It assumes that they know what the issues are and what evidence they need to prove the issues. It assumes that they know what they want, which, for some practitioners, is not always true. Some practitioners have no idea what the real issues are and what they are looking for.
Regardless of the reason for the requesting party’s non-disclosure, this system of discovery by guesses on effective keywords is a model of inefficiency. It may be fun to the players involved, some of whom may reap huge fees in the process, such as the responding party’s lawyers and vendors. It is not, however, designed to get the right cards on the table in the quickest and cheapest way possible. Quite the contrary – it is designed to stretch out the process in an iterative series of negotiated key words and searches. This process involves as much chance as skill.
This kind of approach to the pursuit of truth to attain justice is unreliable and inefficient. The process not only takes too long, the many bad guesses on keywords create a vast quantity of false hits. In the world of information science, that is called poor precision. A ninety percent miss-ratio is not uncommon. That is, for every ten documents that contain the specified keywords, nine are irrelevant, and only one is relevant. This 10% precision rate necessarily results in a tremendous waste of reviewer time. The irrelevant documents retrieved by the search are called false positives. The responding party must then spend a small fortune to screen the many false positives for relevance and privilege.
The Go Fish approach also misses many relevant documents. In the world of information science this is called a poor recall rate. The relevant documents not found are called false negatives. Again, a recall rate of only twenty percent, where eighty percent of the relevant ESI is missed, is not uncommon. This raises questions of the fairness of the process. Can justice be served when only 20% of the relevant ESI is located?
We Need a New Game
We must redesign the game of e-discovery search as it is now commonly played. We should design a new game where the responding party picks the search methods, not the requesting party. In this new game the goal is speedy, just and inexpensive discovery. Get the right cards on the table in a quick, fair and efficient manner. With this goal in mind, it is obvious that the cards should be picked by the person holding them, the responding party. The responding party should design the search strategy, not the requesting party. It is, after all, their hand, and so they can see for themselves what search procedures and terms will work or not.
In order for this new game to work, the responding party needs to know what the requesting party is really looking for. What cards do they want? They might be able to find them, but not if they do not know what they are looking for. Thus, for example, if the goal is to find all deuces, then the requesting party should specifically say so, rather than request all cards that are less than 5. Thus, in this new game the requesting party must show their hand first, they must explain what they need and why. Both sides need to discuss and narrow the issues and be frank and open about discovery. Then the responding party can then look at their own cards and see which are responsive to the defined issues. This approach is consistent with the goal of this new game: both sides work together to find the cards that the requester wants and get them on the table as fast as possible.
This new game can only work if the search is controlled by the responding party. It is, after all, their data, their IT systems, their data custodians, their employees, their agents, their attorneys, their language, their retention policies, their retention practices, etc. The responding party is not blind like the requesting party. They are in a far better position to design the culling and search strategies, including key words. They are in a far better position to find the information that the requesting party wants. They will know how to find all of the number 2 cards, assuming they have any. The responding party may still find nothing and say Go Fish. But the process will be much faster and less expensive than iterative Blind Man’s Bluff keyword negotiations.
The game as played now forces the blind man to make hundreds of guesses at a time, hundreds of key words, hoping that a few might be right. This hurts the respondent who has to review all of the junk generated by the blind guesses. It hurts the requester too, who eventually has to review all of the relevant and marginal calls. It is a colossal waste of time. It is inefficient even if the requester is given several guesses, not just one, and does some refining and talking in between the turns. That only makes the process slightly less wasteful.
Bottom line – we need to stop fooling around with search in e-discovery. That means taking the blindfolds off, but more fundamentally, it means redefining the goals of the game of discovery itself. All too often the goal of discovery today is to try to take your adversaries secrets, but keep your own. Lawyers try to win a case by discovery. Perhaps because they have so few trials, they lose track of the fact that discovery is not supposed to be an end in itself. It is just supposed to be a preparation for trial.
Trial is the time and place for the adversarial process and arguments, not discovery. Many litigators today forget this. They focus instead on a game where they try to only show their good cards, the ones that support their positions. Conversely, they try to keep secret all of their bad cards, the ones that undercut their positions or support the opposition’s positions. For them the goal of discovery is to put only their good cards on the table and keep their bad cards face down in the discard pile.
Today’s paradigm of negotiated search terms perpetuates that adversarial discovery model. It encourages feigned cooperation where each side secretly hopes that the other side will guess wrong. For if that happens, and bad search terms are picked, they will not have to show their bad cards. They can manipulate and hide the truth.
The New Game of Discovery is Won when Completed with Enough Money Still Left for Trial
This perverse game of selective disclosure might have worked in the paper world (although that is debatable), but it no longer works now. We now have a Saganesque number of cards – billions and billions. No one can afford to play this game any more. It should be obvious by now that if you play this game, you will quickly run out of time and money for the real game – the true purpose of litigation – a trial on the merits. How else do you explain a 96% settlement rate in federal court? Yes, trials are expensive. But if discovery were to cost less than it does now, perhaps far less, then there would be adequate resources remaining after discovery to conduct a trial.
Under the new cooperative based, producer-search-driven discovery here proposed, the trials themselves would also become simpler and more streamlined. If lawyers did not play the old games of truth manipulation, and just let the chips fall where they may, many unnecessary side issues would fade away before trial. When bad facts come out early, pseudo-issues go away early too. This inevitably results in fewer issues remaining for trial. Thus if discovery was changed as here recommended, the cost of trials could also be reduced.
The new goal here proposed for discovery is to find and place all of the important cards on the table as quickly and efficiently as possible. This requires cooperation and transparency on both sides. It requires the requesting party to explain what cards they want and why. It also requires them to make precise and narrow requests directed to specific, important issues in the case.
This new game also requires cooperation and transparency by the responding party, moreover it requires their initiative and leadership. The responding party can no longer just sit back and watch poor guesses being made. They must take the lead in getting the truth out. This is a burden, but the responding party is more than compensated for this burden by the protection this provides from over-broad, expensive, inefficient search. It also protects the responding party from having to show their whole deck of cards, their entire ESI collection. The protection of privacy rights is an important factor to many.
The party responding to requests for production must be proactive. They must design the search. As discussed, this only makes sense because it is their data. They have unfettered access to it. They know the language. They know the people involved. For these reasons, the responding party is always in the best position to search the data and, if asked, to fully explain how and why the search met the needs of the requesting party. The process must be transparent. It must also be performed competently. This may sometimes require the employment of experts and search design specialists.
Once the cards responsive to the request are found, they all have to be disclosed, the bad as well as the good. The only exception is privileged documents, which are logged. Honesty and good faith are critical in all discovery processes.
The process may still sometimes be iterative. A careful study by the requesting party of the ESI received may lead to new goals, new issues, and new more focused requests. But still, two fast, focused searches beats one long, over-broad search any day.
This is a new discovery game where both sides win if they complete discovery on time and under budget. This is restrained discovery where the parties only search for the facts they really need. This is discovery where all of the facts are freely disclosed, not just the ones that help your position. This is discovery that typically ends when the budget is exhausted, not the attorneys. For we all know that attorneys are hard working and capable of billing a mind boggling number of hours. These long hours will end in this new game because attorneys will no longer have to try to shape the truth. They will instead cooperate to put the truth on the table. They will save their arguments for what the facts mean under the governing law. This is cost-conscious, proportional discovery where, once completed, sufficient resources still remain for a trial on the merits.
This new game of discovery that I propose, along with many others, is designed for trials, not perpetual preparation. It is a model for those who want to fish, not just “cut bait.” It is a model for all true trial lawyers.
The new rules proposed here apply equally to plaintiff and defendant. Discovery is and will continue to be a two-way street. Both parties will have to find the cards that the other side is looking for. Then, they must put their cards on the table; good, bad or indifferent. If the cards do not exist, and this may happen often, the producer will have to explain exactly what they did. They will also have to remain open to additional searches.
Conclusion
I propose that producing parties always take the lead in the search of their own information. This does not mean that the requesting parties should do nothing and just accept with a smile whatever is handed to them. They will have a seat at the table. They will be heard, but clairvoyance will not be required. I propose a collaborative, transparent process where unnecessary application of the work product doctrine is curbed in favor of efficiency.
Indeed, although the responding party can see-all and thus must lead the search, the requesting party should always still play a key role. First of all, they have to fully describe what their Waldo looks like. In discovery each issue has its own Waldo and its own ideal search methodology to find him. There are recurrent patterns, especially where the responding party has gone through the drill many times. Yet there is rarely a one-size-fits-all search strategy, any more than there is a one-size-fits-all legal strategy.
The requesting party can, if they wish, make more contributions beyond describing their Waldos. The producing party may seek their advice. The requesting party may sometimes have far greater search expertise. Even if they do not, they may still have some good ideas and be able to contribute to the search process.
A leader with vision does not mean a dictator, nor does it imply blind obedience. The responding party should lead, but should also explain everything they did, or plan to do. They should be willing to answer all questions and to ask questions. They should be willing to listen if the other side has something to say. They should be open to constructive suggestions.
If the requester is not cooperative, the responder should also be willing to assume risks. The responding party should be ready and willing to go on their own if need be. They should be ready to explain everything to the supervising judge. The judges can help make this new game work, especially in circumstances of an uncooperative requester. The courts can do this by affirming all reasonable search efforts, absent only a showing of bad faith.
This new game is not a competitive game where one side wins and another loses. Either they both find Waldo or they both lose. The win-lose part of the process comes next. It comes after discovery when the case is decided by summary judgment or tried before a judge and jury. That is the way it should be. Neither discovery nor mediation are adequate substitutes for adjudication.
The new game of discovery here proposed implements strategic cooperation. In this way we can regain our adversary system of justice. We can start doing trials again, instead of playing endless rounds of Go Fish.
________________
I look forward to your comments and help in flushing out the details of this proposal. Please leave a comment below. This is just the beginning. No doubt I have missed some issues and may have gotten a few wrong. What do you think?
I appreciate this article as I am currently in a situation where Defendants are producing hundreds of thousands of pages pursuant to “search terms” which Plaintiffs were asked to provide.
Plaintiffs are not in a position to provide the “magic words” used by Defendants; thus, the search terms result in extremely large production accompanied by extremely long hours in document review and analysis. While nuggets are often found, it is much like you said whereby one relevant document turns up in every ten documents.
I agree that in a perfect world, the manner in which you propose discovery would work. However, when you are dealing with a Defendant who has no interest in putting the truth on the table so to speak, it is extremely difficult to trust them to lead the way on discovery. Defense firms get paid big money to hide the ball from Plaintiffs.
What do you propose Plaintiffs or the courts do that will encourage Defendants to put the facts on the table? Also, can you think of a system by which Plaintiffs can be assured adequate searches are being performed?
Thank you.
Thanks for your comments.
Most defendants, especially serial defendants, are interested in getting the truth out and either quickly settling or trying the case. They complain about high legal fees all of the time. Also, their burden to review is greater than the recipient. Remember they have to review for privilege, whereas the requesting party does not. That is a tremendous psychological and financial burden
Regarding your cooperation comment, as I have often said before, although I am not sure it is a correct analogy: “It takes two to tango.” If one side does not cooperate, then document and take it to the judge. But try first to get them to cooperate. You might be pleasantly surprised.
It is not true that defense firms get paid big money to hide the ball from plaintiffs, any more than it is true that plaintiffs lawyers take meritless cases just to try to extort settlement payoffs. The “us versus them” and stereotyping is not helpful.
The path to cooperation is paved with tolerance and realization that both plaintiffs and defendants are producers and requesters. Related to that, please see my comments below to Craig Ball.
We need to devise a new type of discovery that avoids abuses by both sides. We need discovery to follow Rule #1 – just, speedy and inexpensive. This move from “Go Fish” to “Where’s Waldo” may help us to get there, but only if we stay positive and solution oriented. There are no doubt many other good ideas and solutions, and I look forward to hearing them.
I LOVED this line and your shift to use Where’s Waldo as the vehicle:
“First of all, they have to fully describe what their Waldo looks like.”
Interesting in seeing some insightful comments from folks…
Ralph:
As always, your heart is in the right place. I’m not so sure about your perspective this go-round. You’re absolutely right that the producing parties have the right and responsibility to craft the search process in respect of their superior knowledge and exclusive access to the ESI. You’re also on the money that they can’t meet this duty in a vacuum but must be forthcoming concerning what they have, what they’re searching, how they’re searching and why they chose the method employed over other alternatives.
They also have to accept responsibility for failure when the method employed fails to surface material, responsive information.
There’s the rub.
You write as though requesting parties were the proponents of the “give me your keywords” approach to search. They weren’t. That practice grew out of responding parties efforts to deflect criticism and accountability for poorly-conceived search. “Hey Requesting Party, you can’t complain about the keywords we ran. YOU gave them to us. We just did what you asked.”
Requesting parties would LOVE to be confident that the other side is doing a good job in collecting and producing responsive material with minimal oversight. But, the reality is something else altogether. The quality of preservation and collection is spotty. The precision and recall of search is often no better than a monkey might achieve throwing his feces around the file room. Then, when the really good stuff does turn up, requesting parties face the same hurdle as was faced with paper: more energy and creativity goes into finding reasons not to produce it than went into finding it in the first instance! As the old saying goes, “Just because you’re paranoid doesn’t mean they aren’t out to get you.”
Ralph, requesting parties don’t want to play Go Fish. They don’t want to play the Dictionary Game either. But, if they don’t dictate some objective means to compel production, how do they keep those holding all the cards from stacking the deck?
As to requiring requesting parties to know what Waldo looks like, that’s devoutly to be wished. But is it enough to say he’s white male wearing a striped shirt and a cap, or do I have to specify his DOB, SSN, inseam and sexual preference, too? That’s the essence of the problem: it’s hard to translate something we understand as a person–Waldo–into a series of discrete descriptives that only relate to Waldo and not to Waldene and Wilmer.
The requesting party doesn’t know his name is Waldo or that he’s wearing a stocking cap. They just know the bad thing stuff he did and that he’s hiding somewhere in your house!
“A monkey throwing feces.” Good grief! To paraphrase one of your prior comments to me – “Did a ‘responding party lawyer’ run over your dog this weekend?” Notice I did not say “defense lawyer.” This is an issue that cuts both ways, although admittedly in PI type lawsuits the defendants usually have a few more computers than the plaintiff. In my current world of commercial litigation the sides usually have equal burdens.
Seriously, no matter what the field of law, the solution is cooperation. It starts with trust building and continues with transparency.
Anyway, thanks for the colorful comment. I hope you will join in the design of a new discovery game. It would be fun to create a new “Where’s Waldo” type program with you someday.
Hi Ralph,
Your thoughts on the respondent “leading” the ediscovery is really utopian, especially when you admit on the fact that “only good cards are palyed”
Both parties know the strengths of their case in a litigation matter. Both the litigants know the odds of winning and losing a particular case, based on the facts of the case (some of which is sought to be ediscovered). The so called ediscovery is discovery only in as far as the party seeking it. The party holding the cards has a fair idea of what they are holding(immaterial whether they know by heart all the cards that they are holding). This party also knows, what the end result would be, by putting them on the table. If the litigation is inherently sound from their standpoint, they would show them up, otherwise not. Therefore the burden of leading the ediscovery on the respondent is a little unrealistic.
On the other hand, the requestig party should not be absolved of their responsibility of adhering to the “Where’s Waldo” approach. They should not be given the liberty of playing “Go fish” at the cost and efforts of the respondent. A duty must be cast on them to spell out what’s and whys.
Then, possibly each of the ediscovery projects should be lead managed by a “regulatory body” or a “professional body” appointed by the court.
The methodlogy, as explained below would put an end to the problems of “false postives” and “false negatives” in addition to the achievement of the primary goal of justice being done fairly, early and efficiently.
1.This body should be composed of right mix of people who have the legal expertise to identify the issues and the formulation of strategy to pull out relevant data and the data search experts who execute the discovery strategy.
2.The requeting party submit their search criteria the body.
3.The respondent submits the data archives to them.
4.They conduct search. They are free to determine on key words or other criteria based on the issues framed.
5.Discovered data is then sent to respondent for identifying previledged info.
6.Culled data produced in the court.
This mechanism would ensure ediscovery, unbiased. This would ensure a pointed search strategy being formed and executed. This would of course result in time and cost savings.
Such a methodology of appointing a regulatory body is possible under the Indian legal system. I am not sure if it is so under the US legal system also.
I have one question:
What is the legal position of wilful concealment of incriminating evidence in a civil trial? Is it not one of the objectives of ediscovery to get that peace of evidence on the table for a fair trial and disposal of the case?
Agree it cuts both ways, so please notice that I didn’t refer to plaintiffs or defendants, just to requesters and responders.
It occurs to me why the “Where’s Waldo” analogy doesn’t resonate. So many responders take the position that, “There’s no one here named Waldo. There’s never been anyone around here like that, and we resent your even suggesting that we ever knew anyone like Waldo.”
The assumption (in another comment) that the responding party intimately knows its own ESI is out-of-step with my experience–at least when applied to counsel. You?
Bingo – give the man a cigar (Craig.)
Three things:
In Go Fish, at least the way my kids played, you only needed a pair, not four of a kind.
As Craig suggests, the central flaw in the proposition is that the responding party knows all the cards it holds. Most times it doesn’t, except where it is a David in the David and Goliath scenario. One well placed rock… But if David had missed with his sling, we’d be singing the songs of Goliath.
The problem is more akin to the fishing analogy of long liners vs. gill netters. We have cut broad swaths of barren ocean with gill nets. Very expensive at all steps in the process (boats, nets, false positive catch, environmental damage and high priced sorting to get the high value fish.) Long liners can do damage too, but only if they get their lines, bait and fish into a school. There is a lot of luck in that, and sometimes one boat will cooperate with others, but the winner is the boat that gets on the fish first with the right bait. Even then, there aren’t enough lines for all the fish, so some survive to be fished another day.
This seems like an area that will only be successful if the judge carries Teddy Roosevelt’s “big stick.” Initially at least, it seems that parties, especially the offending party, would be hesitant to quickly produce what will sink their case. Thus, the “big stick” of sanctions would have to be used copiously to encourage your proposed program.
Interesting take on the issue. It seems like as defendant companies (especially highly litigious ones) grow in technological complexity, their infrastructure should take litigation holds into account. Correct me if I’m wrong, but I believe a company just needs to have some kind of document retention policy in place, but no more specifics of that policy are outlined. I wouldn’t be surprised to see more “litigation-search-ready” services built into existing infrastructure, such as DLP (data loss prevention) systems and internal threat management. Thanks for the article.
Here is a horrifying example of a false negative scenario in keyword searches:
BOSTON (AP) — A pharmacy college graduate conspired with two other men in a terror plot to kill two prominent U.S. politicians and carry out a holy war by attacking shoppers in U.S. malls and American troops in Iraq, prosecutors said Wednesday.
But their plans — in which the men used code words like “peanut butter and jelly” for fighting in Somalia and “culinary school” for terrorist camps — were thwarted in part when they could not find training and were unable to buy automatic weapons, authorities said.
[…] like playing the card game Go Fish, an issue he covered brilliantly in a recent post on his blog (click here). Unless opposing counsel asks specifically in their request for production for a responsive […]
Finally a discussion about integrity of the process and its importance to the aim of justice being served so that clients can afford to pursue enforcement of legitimate claims and the system can more efficiently dispose of illegitimate claims.
[…] They fight over search protocols because they really have no idea how to find the relevant needles in the vast collections of electronic haystacks that their clients foolishly maintain. Since they do not know what to do, they just propose keyword search terms. They pretend like e-discovery search is the same as a Google or Westlaw search because that is all they know. They then engage in foolish games of competing keyword lists that quickly lead nowhere. I have also written about this before in Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search. […]
[…] 128 page opinion will be issued soon after he has had an chance to reflect upon the merits of the Go Fish v. Where’s Waldo […]
[…] for all relevant ESI without regard to whether it is positive or negative. See my prior article Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search where I explain the inherent defects in keyword search as it is now conducted by most law firms and […]
[…] The Multi-Modal “Where’s Waldo?” Approach to Search and My Mock Debate with Jason Baron, and Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search. The actions of plaintiff’ and his counsel in this case help explain why my preferred […]
[…] (For an earlier take on how to keep e-discovery from becoming like a child’s game, see Ralph Losey’s 2009 post, Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search.) […]
[…] First of all, and let me put this in very plain vernacular so that it will sink in, keyword search sucks. It does not work, that is, unless you consider a method that misses 80% of relevant evidence to be a successful method. Keyword search alone only catches 20% of relevant evidence in a large, complex dataset, such as an email collection. Yes, it works on Google, it works on Lexis and Westlaw, but it sucks in the legal world of evidence gathering. It only provides reliable recall value when used as part of a multimodal process that uses other search methods and quality controls, such as iterative testing, sampling, and adjustments. It fails miserably when used in the Go Fish context of blind guessing, which is the negotiated method still used by most lawyers today. I have written about this many times before and will not repeat it here again. See eg. Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search. […]
[…] also explained that keyword search would not be so bad if it were not done blindly like a game of Go Fish, where it achieves really pathetic recall percentages in the 4% to 20% range (the TREC batch […]
“Can justice be served when only 20% of the relevant ESI is located?”
I just have to wonder where the 20% number comes from.
Sampling the data that is not pulled in for review for responsive documents will make the search terms better. Not perfect, but reasonably better. You do this a couple of times with the remaining balance (budget permitting) and your search is muchos better.
Second, more often than not, the responding party also doesn’t know what their cards are. Responding party is shooting in the dark, too. Once the parties agree on the terms, the deal is done.
Whether or not that’s “fair” or “just” would be, in my view, irrelevant. The party’s agreement supersedes “justice” and gives the parties a way to move forward, for better or worse.
[…] Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search […]
[…] is the equivalent of the child’s game of ‘Go Fish.’” (Judge Peck cites Ralph Losey’s blog for the analogy. We’ve used it here in a different context.) The requesting party guesses […]
[…] is that absent cooperation, the way most lawyers engage in keyword searches is, as Ralph Losey suggests, the equivalent of “Go Fish,””, one hopes he is a better judge than a prophet: […]
As a contract attorney, I often find myself drowning in the wake of poorly orchestrated games of go fish. The problem begins with the initial disconnect between client and outside counsel. Outside counsel may have insight into an industry, but often does not have expert command of the nuances of a particular competitve landscape. As a result the seach terms fought over often produce information of little or no value
After outside and opposing counsel reaching an agreement regarding the search terms to be used, I am called on to review the actual results. Most of the time the search terms result in a high rate of documents which are responsive but irrelevant with little or no probability of leading to relevant information. In an ideal world, I would be able to bring this to the attention of the litigation team, and run a search which would eliminate the useless term. This process of continued refinement of search terms would bring the data into sharper focus and eventually reach documents of actually relevance to the matter.
Unfortunately, we do not live in an ideal world. If the only danger was contract attorneys slipping into boredom induced comas, then no harm, no foul (at least for outside counsel and client). The reality is that with every piece of responsive, irrelevant information produced, a competitor gains increased insight into the client’s organization. Outside counsel needs to think about the impact a child’s game of go fish will have on its client. Or better yet, outside counsel could become a true innovator and begin to marshal the strategic resources hidden away in the data, and go beyond solving the current crisis to providing competitive advantage.
Totally agree. Good comments.
Not sure you’d discovery anything worthwhile if the search were left to respondents.
[…] Is Keyword Search in eDiscovery Dead? I’ll answer that question, talk about the old days of endless games of Go Fish, some of my new search methods, and the Kleen Products case, but not the other case. Presenting […]
Trust is a major issue. A Special Master, or some other agreed upon neutral third-party, could be used to verify the search methodologies used by each of the parties. The parties would give the neutral the ESI from a sample custodian or custodians, along with the internal methodologies for processing and identifying responsive, relevant information. The neutral would then apply that process to the sample ESI. Essentially an “audit” would occur.
This sound expensive, and may scary budget conscience clients, and counsel. If done correctly, the cost savings released through the elimination of responsive, irrelevant information as quickly as possible will offset the additional expense in real dollar terms, quality of information produced, and organizational insight gained.
[…] Key word searches to reduce data sets also miss a large percentage of relevant documents. The typical practice of opposing parties choosing keywords resembles a game of “Go Fish,” as Ralph Losey once pointed out. […]
[…] the keyword list dreamed up by defense counsel, who apparently engaged in a rousing game of “Go Fish,” included such zingers as: contract*, loss, profit*, credit, refund, revenue, CL, HS*, […]
[…] Dead?” The problem with keyword searches, says Losey, is that they are much like the card game “go fish.” Parties applying keyword searches typically make blind guesses about which keywords might […]
[…] proportionality reform. Still, I am troubled by the reliance on Go Fish keyword search methods. See Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search. The lack of precision and recall in blind keyword search makes this method both expensive and […]
[…] produce evidence, without knowing anything about the other party’s “cards.” See Go Fish. Not to mention, keyword searches tend to be over-inclusive, and produce many irrelevant […]
[…] requesting parties, as Ralph Losey has noted, have been engaged for decades in a game of “Go Fish” — as they often are blindly guessing through requests to produce and more recently through […]
[…] what a party conceals in its back e-pocket. Ralph Losey, in his popular e-Discovery Team blog, has even referred to keyword searching as similar to a child’s game of Go Fish, in which both players try to guess the other side’s cards while attempting to conceal their […]
[…] and a judge (arbitrator) was actively involved. In other words, it was not a completely blind Go Fish keyword guessing […]
[…] in question. Ralph Losey, lawyer and author of the e-Discovery Team blog, refers to this process as “Go Fish”7 information […]
[…] would also like to see traditional Go Fish keyword search negotiations wrecked. You should always test keywords before you use them, not just […]
[…] relevant–even preferable in certain situations–keyword searches have frequently been compared to a game of “Go Fish.” Stated perhaps less derisively “[s]earching for an answer on Google (or Westlaw or Lexis) is very […]
[…] what a party conceals in its back e-pocket. Ralph Losey, in his popular e-Discovery Team blog,has even referred to keyword searching as similar to a child’s game of Go Fish, in which both players try to guess the other side’s cards while attempting to conceal their […]
[…] [19] William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134, 136 (S.D.N.Y. 2009). Also see: Losey, R. Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search found at http://e-discoveryteam.com/2009/10/04/childs-game-of-go-fish-is-a-poor-model-for-e-discovery-search/. […]
[…] [19] William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134, 136 (S.D.N.Y. 2009). Also see: Losey, R. Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search found at http://e-discoveryteam.com/2009/10/04/childs-game-of-go-fish-is-a-poor-model-for-e-discovery-search/. […]
[…] also explained that keyword search would not be so bad if it were not done blindly like a game of Go Fish, where it achieves really pathetic recall percentages in the 4% to 20% range (the TREC batch […]
[…] First of all, and let me put this in very plain vernacular so that it will sink in, keyword search sucks. It does not work, that is, unless you consider a method that misses 80% of relevant evidence to be a successful method. Keyword search alone only catches 20% of relevant evidence in a large, complex data set, such as an email collection. Yes, it works on Google, it works on Lexis and Westlaw, but it sucks in the legal world of evidence gathering. It only provides reliable recall value when used as part of a multimodal process that uses other search methods and quality controls, such as iterative testing, sampling, and adjustments. It fails miserably when used in the Go Fish context of blind guessing, which is the negotiated method still used by most lawyers today. I have written about this many times before and will not repeat it here again. See eg.Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search. […]
[…] The key documents? Or was it all done in the blind, like a child’s game of GO FISH? Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search (e-Discovery Team blog, […]
[…] Key word searches to reduce data sets also miss a large percentage of relevant documents. The typical practice of opposing parties choosing keywords resembles a game of “Go Fish,” as Ralph Losey once pointed out. […]
[…] is the equivalent of the child’s game of ‘Go Fish.’” (Judge Peck cites Ralph Losey’s blog for the analogy. We’ve used it here in a different […]
[…] (For an earlier take on how to keep e-discovery from becoming like a child’s game, see Ralph Losey’s 2009 post, Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search.) […]
[…] first against the actual data. This poor practice of using unverified keywords is what I call the Go Fish approach to keyword search. Child’s Game of “Go Fish” is a Poor Model for e- […]