If you are a data hoarder, an information pack-rat that saves everything, you will eventually drown in your own data and die. Maybe not literally killed, mind you, but figuratively. Maybe not you personally, but your enterprise, your group, your project, your network. Too much information can render you and your enterprise intellectually paralyzed, cut off and seriously misinformed or uninformed. Saving it all is physically and logistically difficult, if not possible. Even if you could, keeping it all would impede your search, making it hard to find the information you need, when you need it. I address these issues this week in my review of a new commentary by The Sedona Conference Principles and Commentary on Defensible Disposition (August 2018).
I
nformation overload is better than physical death I know, but still very bad in today’s Google world. You end up not being able to find the information you need, when you need it. That makes it hard to determine what really happened. It allows lies and liars to fester and grow. We are now seeing firsthand in the U.S. where this can lead. It is not good. It has put the whole world into a precarious situation. We need the truth to thrive as a culture; not smoke and mirrors, not conman games. A culture built on lies is a cancer. It is a deadly disease, especially for the Law, which depends on truth, on evidence, on real facts, to attain the goal of Justice.
Saving Too Much
Over-retention is the enemy of effective, efficient search. The more ESI there is to search, the more difficult the search. There can be exceptions to this rule, but for the most part it is true. That makes a “save everything” ESI policy an enemy of search. It interferes with the ability to find the information needed, which in my case is electronic evidence in legal proceedings, when it is needed. It is important for these information needs be filled quickly and completely.
Search is powerful. That is my field. The more data the better, is often true, but not always. It depends on the data and its effective life, how long a particular type of data is of any use to anyone. Big data allows for detection of patterns that would otherwise not be seen. This analysis takes CPU power. The advances in this area have been fantastic. We have the processing power, as well as the cheap storage, but our search and retrieval software has not otherwise kept up with the data explosion in volume and complexity. Predictive coding software and other AI applications have come a long way, but are still sometimes confused by the volume, variety and complexity of useless data that plagues most company IT systems.
Retrieval of specific documents and metadata takes time and specialized human skills. The more worthless data in a collection, such as spam, the greater the number false positives in a search, no matter how powerful the algorithms or skilled the searcher. Vast volumes of data make searches longer to execute and less precise. The more noise in the data, the more difficult to hear the signal. That is a fundamental law of information.
With high data volumes you can often still find the signal, the relevant documents that you need in large chaotic data collections, but it takes time and special tools and skills. There are often too many false positives in searches of data collections containing too much spam-like, useless data. Although search is strong, search alone is inadequate to meet the needs of most organizations. They also need data destruction and retention policies that govern all information. That is one reason why the success of information governance depends on data disposition.
An organization should save as much as it needs, but not too much, and also not too little. It is a Goldilocks situation. If you do not save data, you can never find it. If you save too little, then what you later need might not be there to be found. But if you save too much, you may never be able to find what you need. The signal may be in the collection to be found, in plain view, but hidden in the vast numbers, the noise of spam and other irrelevancies.
Search v. Destroy
I have debated Information Governance leaders for years the importance of search versus file destruction. I was pretty much the only advocate for search over disposition. I favored retention over destruction in most close cases, but I had a cost and proportionality overlay. I am reminded, for instance, of my debate with Jason Baron on the subject at the IQPC 10th Anniversary of Information Governance and eDiscovery, where he managed to quote Churchill at the end and won the debate hands-down. e-Disco News, Knowledge and Humor: What’s Happening Today and Likely to Happen Tomorrow (e-Discovery Team, June 7, 2015); Information Governance v Search: The Battle Lines Are Redrawn (e-Discovery Team, Feb. 8, 2015).
I 
did not consider it a fair debate because of Jason’s very successful pandering to the jury during his closing argument with a quote by Churchill from his speech, We Shall Fight on the Beaches. That’s the one about never surrendering in the fight against “the odious apparatus of Nazi rule” (sadly, this exhortation still has legs today in the US).
The debate was “unfair” primarily because this was an IG conference. Everybody in IG is pro-destruction and values disposition over search. I think most IG leaders go too far, that they are trigger happy to kill data. I pointed out in my debates that once a file is deleted, it cannot be found, no matter how good your filing, no matter how good your search (forensic recovery issues aside).
I am pro-search and think that the importance of management of ESI by filing and disposition is somewhat overblown. I think search is king, not data deletion. Still, even in my most strident of debates and pro-search arguments, I never advocated for the retention of all data. I always assumed that some file disposition was required and accepted that as a given. I was not a save everything and search advocate. I advocated for both, search and destroy. I advocated for more retention than most, but have never argued to retain everything.
There is a common core of agreement that some ESI should be deleted, that all data should not be saved. The disagreement is on how much data to save. How does a person or company know what is the “just right” data destruction policy for that company? There is agreement among experts that there is no one-size-fits-all solution, so custom work is required. Different retention and destruction policies should apply depending on the company and the particularities of their data universe. Many IG specialists advise clients on the custom fit they need. It involves careful investigation of the company, its data and activities, including law suits and other investigations.
The Sedona Conference Principles and Commentary on Defensible Disposition
Kevin Brady
These IG specialists, and the companies they serve, now have an excellent new resource tool to analyze and custom-fit data destruction policies. The Sedona Conference Principles and Commentary on Defensible Disposition (August 2018 Public Comment Version) (Editors-in-Chief, Kevin F. Brady and Dean Kuckelman). I highly recommend this new and excellent work by The Sedona Conferences. My commendations to the Drafting Team: Lauren A. Allen, Jesse Murray, Ross Gotler, Ken Prine, Logan J. Herlinger, David C. Shonka, Mark Kindy; the Drafting Team Leaders: Tara Emory and Becca Rausch; the Staff Editor: Susan McClaim, and Editors-in-Chief, Kevin F. Brady and Dean Kuckelman. Please send to them any comments you may have.
The Commentary begins in usual Sedona fashion by articulation of basic principles and comments tied to principles. The cases and legal authorities cited in all Commentaries by The Sedona Conference are excellent. This commentary on data disposition is no exception. I commend it for your detailed study and reference. Free download here from The Sedona Conference.
The Principles are:
PRINCIPLE 1. Absent a legal retention or preservation obligation, organizations may dispose of their information.
Comment 1.a. An organization should, in the ordinary course of business, properly dispose of information that it does not need.
Comment 1.b. When designing and implementing an information disposition program, organizations should consider the obligation to preserve information that is relevant to the claims and defenses and proportional to the needs of any pending or anticipated litigation.
Comment 1.c. When designing and implementing an information disposition program, organizations should consider the obligation to preserve information that is relevant to the subject matter of government inquiries or investigations that are pending or threatened against the organization.
Comment 1.d. When designing and implementing an information disposition program, organizations should consider applicable statutory and regulatory obligations to retain information.
PRINCIPLE 2. When designing and implementing an information disposition program, organizations should identify and manage the risks of over-retention.
Comment 2.a. Information has a lifecycle, including a time when disposal is beneficial.
Comment 2.b. To determine the “right” time for disposal, risks and costs of retention and disposal should be evaluated.
PRINCIPLE 3. Disposition should be based on Information Governance policies that reflect and harmonize with an organization’s information, technological capabilities, and objectives.
Comment 3.a. To create effective information disposition policies, organizations should establish core components of an Information Governance program, which should reflect what information it has, when it can be disposed of, how it is stored, and who owns it.
Comment 3.b. An organization should understand its technological capabilities and define its information objectives in the context of those capabilities.
Document Disposition and Information Governance
The Sedona Conference Principles and Commentary on Defensible Disposition builds upon Sedona’s earlier work, the Sedona Conference Commentary on Information Governance (Oct. 2014). Principle 6 of the Commentary on Information Governance provides the following guidance to organizations:
The effective, timely, and consistent disposal of physical and electronic information that no longer needs to be retained should be a core component of any Information Governance program. The Sedona Conference, Commentary on Information Governance, 15 SEDONA CONF. J. 125, 146 (2014) (“Information Governance” is “an organization’s coordinated, interdisciplinary approach to satisfying information compliance requirements and managing information risks while optimizing information value.” Id. at 126).
The Comment to Principle 6 goes on to explain:
It is a sound strategic objective of a corporate organization to dispose of information no longer required for compliance, legal hold purposes, or in the ordinary course of business. If there is no legal retention obligation, information should be disposed as soon as the cost and risk of retaining the information is outweighed by the likely business value of retaining the information. . . . Typically, the business value decreases and the cost and risk increase as information ages. Id. at 147.
The Sedona Conference concluded in 2018 that this 2014 advice, and similar advice from other sources, has not been followed by most organizations. instead, they continue to struggle to make “effective disposition decisions.” The group in Principles and Commentary on Defensible Disposition concluded in its Introduction that this struggle was caused by many factors, but identified the three main problems:
[T]he incorrect belief that organizations will be forced to “defend” their disposition
actions if they later become involved in litigation. Indeed, the phrase “defensible disposition” suggests that organizations have a duty to defend their information disposition actions. While it is true that organizations must make “reasonable and good faith efforts to retain information that is relevant to claims or defenses,” that duty to preserve information is not triggered until there is a “reasonably anticipated or pending litigation” or other legal demands for records. The Sedona Principles, Third Edition: Best Practices, Recommendations & Principles for Addressing Electronic Document Production, 19 SEDONA CONF. J. 1, 51, Principle 5, 93 (2018).Another factor in the struggle toward effective disposition of information is the difficulty in appreciating how such disposition reduces costs and risks.
Lastly, many organizations struggle with how to design and implement effective disposition as part of their overall Information Governance program.
The Principles and Commentary on Defensible Disposition attempt to address these three factors and provide guidance to organizations, and the professionals who counsel organizations, on developing and implementing an effective disposition program.
Disposition Challenges
The Sedona Conference Principles and Commentary on Defensible Disposition (August, 2018) concludes by identifying the main challenges to data deletion.
- Unstructured Information.
- Mergers and Acquisitions.
- Departed, Separated, or Former Employees
- Shared File Sites
- Personally Identifiable Information (“PII”)
- Law Firms, eDiscovery Vendors, and Adversaries
- In-House Legal Departments
- Hoarders (my personal favorite)
- Regulations
- Cultural Change and Training
There are more, I am sure, but this is a good top ten list to start. I only wish they had included more discussion of these top ten.
Conclusion
Search is still more important for me than destroy. I prefer Where’s Waldo over Kill Waldo! I have not changed my position on that. But neither has mainstream Information Governance. They still disagree with my emphasis on Search. But everyone agrees that we should do both: Search and Destroy. Even I do not want companies to save all of their data. Some data should be destroyed.
I agree with mainstream IG that saving everything forever is not a viable information governance policy, no matter how many resources you also put into ESI search and retrieval. I have never said that you should rely solely on search, just that you should give Search more importance and, when in doubt, that you should save more documents than less. The Search and Destroy argument has always been one of a matter of degree and balance, not whether there should be no destruction at all. The difficult questions involve what should be saved and for how long, which are traditional information management problems.
Where to draw the line on destruction is the big question for everyone. The answer is always company specific, even project specific. It involves questions of varying retention times, files type and custodian analysis. When it comes down to specific decisions, and close questions, I generally favor retention. What may appear to be useless today, may prove to be relevant evidence tomorrow. I hate not being able to prove my case because all of the documents have already been deleted. Then it is just one person’s word against another. IG experts, who usually no longer litigate, or never litigated, do not like my complaints. They are eager to kill, to purge and destroy data. I am more inclined to save and search, but not save too much. It is a question of balance.
Data destruction – the killing of data – can, if done properly, make the search for relevant content much easier. Some disposition of obviously irrelevant, spam and otherwise useless information makes sense on every level. It helps all users of the IT system. It also helps with legal compliance. Too much destruction of data, too aggressive, and you may end up deleting information that you were required by law to keep. You could lose a law suit because of one mistake in a data disposition decision. Where do you draw the line between save and delete? What is the scope of a preservation duty? What files types should be retained? What retention times should apply? How much is too much? Not enough?
The questions go on and on and there is no one right answer. It all depends on the facts and circumstances of the organization and its data. The new Sedona Conference Principles and Commentary on Defensible Disposition is an important new guide to help IT lawyers and technologists to craft custom answers to these questions.
Posted by Ralph Losey 
One of the largest cases in the U.S. today is a consolidated group of price-fixing cases in District Court in Chicago. 
I assume that there were hearings and attorney conferences before the hearings. But I do not know that for sure. I have not seen a transcript of the hearings with Judge Gilbert. All we know is that defense counsel told the judge that under the keywords selected the document review would cost between $1,200,000 and $1,700,000, and that they had no explanation on how the cost estimate was prepared, nor any specifics as to what it covered. Although I was not there, after four decades of doing this sort of work, I have a pretty good idea of what was or might have been said at the hearing.
EUCPs contend that Agri Stats’s claims of burden and cost are vastly overstated. The Court tends to agree with EUCPs on this record. It is not clear what it would cost in either time or money to review and produce the custodial ESI now being sought by EUCPs for the entire discovery period set forth in the ESI Protocol or even for the pre-October 3, 2102 period. It seems that Agri Stats itself also does not know for sure what it would have to do and how much it would cost because the parties have not finished that discussion. Because EUCPs say they are continuing to work with Agri Stats to reduce what it must do to comply with their discovery requests, the incremental burden on what Agri Stats now is being asked to do is not clear.
In order to obtain a protective order you need to estimate the costs that will likely be involved in the discovery from which you seek protection. Simple. Moreover, it obviously has to be a reasonable estimate, a good faith estimate, supported by the facts. The Brolier Chicken defendant, Agri Stats, came up with an estimate. They got that part right. But then they stopped. You never do that. You do not just throw up a number and hope for the best. You have to explain how it was derived. Blushing at any price higher than that is not a reasonable explanation, but is often honest.
This is federal court, not a political rally. You do not make bald assertions and leave the court wondering. Facts matter. Back of the envelope type guesses are not sufficient, especially in a big case like Broiler Chicken. Neither are guesstimates by people who do not know what they are doing. Make disclosure and cooperate with the requesting party to reach agreement. Do not just rush to the courthouse hoping to dazzle with smoke and mirrors. Bring in the experts. They may not dazzle, but they can get you beyond the magic mirrors.
Judge Paul S. Grewal, who is now Deputy G.C. of Facebook, said quoting The Sedona Conference in Vasudevan: 
Judge Facciola in O’Keffe held that document review issues require expert input and that this input should be provided with all of the protections provided by Evidence Rule 702.
In the Boiler Chicken Antitrust Order of July 27, 2018, a motion for protective order was denied because of inadequate evidence of burden. All the responding party did was quote a price-range, a number presumably provided by an expert, but there was no explanation. More evidence was needed, both expert and fact. I agree that generally document review cost estimation requires opinions of experts. The experts need to be proficient in two fields. They need to know and understand the science of document search and retrieval and the likely costs for these services for a particular set of data.
In this blog we discuss yet another case where the parties are bickering over keywords and the judge was asked to intervene. 
Plaintiff’s counsel also filed an affidavit where he swore that he reviewed the first 100 consecutively numbered documents to evaluate the burden. Very impressive effort. Not! He looked at the first one-hundred documents that happened to be on top of a 614,000 pile. He also swore that none of these first one-hundred were relevant. (One wonders how many of them were empty pst container files. They are often the “documents” found first in consecutive numbering of an email collection. A better sample might have been to look at the 100 docs with the most hits.)
udge Whalen agreed with plaintiff and held that:
Judge Whalen concluded his short Order with the following ruling:
The defense in Webastro violated their own stipulation by the use of a party’s product names without further Boolean limiters, such as “product name AND another term.” Then defense counsel added insult to injury by coming across as uncooperative. I don’t know if they alone were uncooperative, or if it was a two way street, but appearances are everything. The emails between counsel were attached to the motions, and the judge scowled at the defense here, not plaintiff’s counsel. No judge likes attorneys who ignore orders, stipulated or otherwise, and are uncooperative to boot. “Uncooperative” is label that you should avoid being called by a judge, especially in the world of e-discovery. Better to be an angel for discovery and save the devilish details for motions and trial.
If I had been handling this for the plaintiff, I would have gotten my hands much dirtier in the digital mud, meaning I would have done far more than just look at the first one-hundred of 614,000 documents. That was a poor quality control test, but obviously, here at least, was better than nothing. I would have done a sample review of each keyword and evaluated the precision of each. Some might have been ok as is, although probably not. They usually require some refinement. Sometimes it only takes a few minutes of review to determine that. Bottom line, I would have checked out the requested keywords. There were only ten here. That would take maybe three hours or so with the right software. You do not need big judgmental sampling most of the time to see the effectiveness, or not, or keywords.
Regarding the vendor experts, remember that even though they may be lawyers, they can only go so far. They can only provide technical advice, not legal, such as proportionality analysis under Rule 26, etc. That requires a practicing lawyer who specializes in e-discovery, preferably as a full-time specialty and not just something they do every now and then. If you are in a big firm, like I am, find the expert in your firm who specializes in e-discovery, like me. They will help you. If your firm does not have such an expert, better get one, either that or get used to losing and having your clients complain.
Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used 
Many courts have followed O’Keffe, even though it is a criminal case, and declined to step in and order specific searches without expert input. See eg. the well-known patent case, 
More recently, e-discovery expert Judge James Francis addressed this issue in 
I am inclined to agree with Judge Francis. I know from daily experience that legal search, even keyword search, can be very tricky, depends on many factors, including the documents searched. I have spent over a decade working hard to develop expertise in this area. I know that the appropriate searches to be run depends on experience and scientific, technical knowledge on information retrieval and statistics. It also depends on tests of proposed keywords; it depends on sampling and document reviews; it depends on getting your hands dirty in the digital mud of the actual ESI. It cannot be done effectively in the blind, no matter what your level of expertise. It is an iterative process of trial and errors, false positives and negatives alike.
Recently appointed U.S. Magistrate Judge Laura Fashing in Albuquerque, New Mexico, heard a case involving a dispute over keywords. 
Of course I agree with Judge Fashing’s concluding reminder to the parties. Cooperation is key, but so is expertise. There is a good reason for the fear felt by Facciola’s angels. They wisely knew that they lacked the necessary technical, scientific knowledge for the proper selection and implementation of keyword searches. I only wish that Judge Fashing’s order had reminded the parties of this need for experts too. It would have made her job much easier and also helped the parties. Sometimes the wisest thing to do is nothing, at least not until you have more information.
