Editor’s Preface: The article attached here in PDF version, My Basic Plan for Document Reviews: The “Bottom Line Driven” Approach, is a complete rewrite and update to earlier blogs written to explain the review methods I have developed over the years. Here is the HTML rewrite version.
I hope you will in turn take the time to read this new version and tell your colleagues about it. Also, I look forward to your comments.
You are free to circulate and use this article at CLE events, etc., so long as you do not change anything or charge anything for its distribution.
Discover more from e-Discovery Team
Subscribe to get the latest posts sent to your email.
Ralph, this current post and your recent Search series have raised the bar higher for all of us out there on the edisco speaking circuit, as you have done a wonderful job of educating the legal community on the present gap between a wave of technology-assisted review overhype, and the reality that we, as lawyers, still have important roles to play in any e-discovery review process. Anyone who thinks technology-assisted review is a “solved problem” for the legal profession needs to go back to re-read your columns!
As a point of privilege, I’d like to just say that I am not quite as dystopian as you may have suggested to your faithful readers. It is not so much that we are drowning in information that concerns me (although I do dream of e-mail free Fridays to actually accomplish some work). Rather, the exponentially growing amount of information — especially in the public sector — amounts to a vast, dark, and de facto inaccessible universe, not practically reachable by means of e-discovery, FOIA, or otherwise. That is why I have been on a quest for the past decade or so to find more powerful search techniques than mere reliance on keywords, so as to be able to properly categorize and extract knowledge from the vast e-haystacks of records that are accumulating all around us, as efficiently as possible. I actually am hopeful that in an “arms race” between on the one hand, preserving the newly created exabytes of record information and beyond, and on the other, employing new forms of AI to extract meaning from big data sets, that the AI will in the end win.
I can’t think of anything more exciting for a young lawyer to contemplate than thinking about how to apply state-of-the-art technology in continuing to tackle the volume and complexity issues posed by e-discovery. IMHO, your recent columns should be required reading in every civil procedure course taught in US law schools. Doug Oard and I actually have your search columns already listed on our syllabus for the e-discovery course we’re teaching this Spring to PhD and Masters candidates at U. Maryland’s School of Information Studies.
Thanks Jason. As usual, you were right about what we need next in e-discovery, namely standards. I am once again following in your footsteps by working on a foundation for these standards. Also, thanks for tolerating my false emphasis (I would rather say poetic license) of your information dystopia comment to serve as my foil. Dear readers, Jason is not really a prophet of doom. In fact, he is one of the funniest people I know.
All of us in the eDiscovery field are seeking to tame the “crazy high” costs of document review as you stated. We share the same dream. For in-house counsels such as myself who are actually writing the “check” for those services, the quest for starndards, is as pressing as it will ever be. The cost of review, however, is just one piece of the puzzle. I would like to go one step further and try to come up with estimates for the total cost of eDiscovery or total cost of document production. The challenge in estimating the total cost of production lies in that we have to take into account all that will be collected, processed, reviewed and, ultimately, the small set that will be produced. It is the same challenge in estimating a billable attorney’s rate, when you have to take into account all the overhead in a practice. In order to estimate the total cost of production, from an in-house perspective, one has to account for:
1) collection: costs of the software (if collection is done in-house), or costs of collection (if done by a 3rd party)
2)processing: costs of culling the data and costs of hosting the dataset plus access to reviewers
3) review: costs of contract attorneys for 1st pass, costs of QC, costs of lawfirm for secondary review and privilege, confidentiality codding/logging/redacting
4) production: costs of tiffing, converting data to media for production, costs of hardware.
All those costs have to be factored in when you are trying to determine the total cost per document ultimately produced.
Therein lies the challenge. Sorry to add to your already herculean effort to create standards but, to the extent that we are seeking standardization, I believe we ought to aim for creating processes that will give clients complete cost predictability and visibility. Thanks.
There are yet additional costs — such as the outside vendor’s hosting charges for the data collected. Vendors typically charge for this on a gigabyte basis and those costs can be quite high in a large volume case — e.g., $40,000-$50,000 a month. To the extent that a case and the discovery phase goes on for an extended period of time, those high, monthly costs to keep the data “alive” and available for subsequent searches to use in deposition preparation or for motions purposes can really mount. And, let’s not forget that there will be hosting charges for the data that is produced from the other side — which the receiving party will want to have available for the same types of searches and factual document research as the case progresses. (Query: Why is it that none of us in the e-discovery world ever speaks about the parallel search and review cost (and hosting fees) associated with examining the documents produced by the opposing party or parties? . . . All the steps and costs associated with producing documents in response to discovery is only one half of the story. Granted with documents produced to us, we don’t have to bear the preservation costs, but basically all the rest remains. Just wondering . . . .)
In my experience, most litigation is asymetrical, in that one party (usually plaintiff or government agency) is the recipient of far greater amount of documents than the responding party. So the responding party only has to worry about hosting its own documents, and the requestor has to worry about hosting the docs produced to them.
Even if the litigation is symetrical, the production volume is about 10% of the collected volume by the other side, so the relative added burden to the recipient is not that great.
Ralph,
I just finished reading your blog entry about Bottom Line Driven Proportional Review, and my initial response is that I am not sure if I buy into it.
I certainly agree that the bottom line is important and that any review should be proportional to the amount at issue in the case, but I find something inherently discomforting about deciding how many documents to review based upon a predetermined budget. If there are truly 30,000 relevant documents in a matter, how can I say I will only look at, and potentially produce 10,000 of them? Is this not a partial truth? If so, how is that really justice?
Of course proportionality must have its place. My concern with your method is that it may place too much emphasis on being able to choose what you will review and produce based on budget alone (ignoring the reality of how many responsive documents there are), and provide lawyers and clients an easy out that allows them to artificially truncate document review. To use a simplified (and perhaps somewhat antiquated) analogy, if you had a case about a motorcycle accident, I would argue it would be difficult to not use the search term “motorcycle accident” simply because it returned more hits than you budgeted for.
I think you attempt to address my issue by cautioning that you have to get your hands dirty and understand the data before creating a budget, and ideally, this would help. Unfortunately, in practice, too often the budget is created without input from e-discovery experts, and/or at times, the people creating the budget are not allowed access to the data until after they have submitted a budget and then been engaged for the work.
As I said, I just read this, so I need to think about it more, but I am left with the feeling that I have to choose between proportionality and the “complete” truth, and furthermore, that Bottom Line Driven Proportional Review may become a tool for abuse and misuse. Perhaps I am being cynical and need to have more faith in my fellow colleagues. Regardless, I do appreciate that you are trying to solve a significant problem, and therefore, whether or not I ultimately agree with you, what you wrote certainly has value to me.
Thanks,
Brandon
“If there are truly 30,000 relevant documents in a matter, how can I say I will only look at, and potentially produce 10,000 of them? Is this not a partial truth?”
Because you can’t know you have 30,000 relevant documents in a matter going in. In fact, you can never know how many relevant documents there are even after you’re done with the review! This is because you don’t know how many were missed by culling or by the reviewers.
It’s a world where a multiplication of imperfections add up to reasonableness. You are going in blind, but you’re coming out only with somewhat better vision. How much better? You’ll never know, unless the universe of documents can fit in a stack of paper on your desk.
“Complete truth” is not absolute. It’s the complete truth within the universe that makes sense given what’s at stake in the case. That’s your proportionality.
Hallelujah. I believe in your dream and I think it is essential that it come true.
This article is required reading for the “Budgeting, Risk Management, and Cost Control in E-Discovery” class taught by Scott Cohen at Bryan University. Many of us in the e-discovery program there subscribe to your blog and are better educated for it. Thank you.