Editor’s Preface: The article attached here in PDF version, My Basic Plan for Document Reviews: The “Bottom Line Driven” Approach, is a complete rewrite and update to earlier blogs written to explain the review methods I have developed over the years. Here is the HTML rewrite version.
I hope you will in turn take the time to read this new version and tell your colleagues about it. Also, I look forward to your comments.
You are free to circulate and use this article at CLE events, etc., so long as you do not change anything or charge anything for its distribution.
Ralph, this current post and your recent Search series have raised the bar higher for all of us out there on the edisco speaking circuit, as you have done a wonderful job of educating the legal community on the present gap between a wave of technology-assisted review overhype, and the reality that we, as lawyers, still have important roles to play in any e-discovery review process. Anyone who thinks technology-assisted review is a “solved problem” for the legal profession needs to go back to re-read your columns!
As a point of privilege, I’d like to just say that I am not quite as dystopian as you may have suggested to your faithful readers. It is not so much that we are drowning in information that concerns me (although I do dream of e-mail free Fridays to actually accomplish some work). Rather, the exponentially growing amount of information — especially in the public sector — amounts to a vast, dark, and de facto inaccessible universe, not practically reachable by means of e-discovery, FOIA, or otherwise. That is why I have been on a quest for the past decade or so to find more powerful search techniques than mere reliance on keywords, so as to be able to properly categorize and extract knowledge from the vast e-haystacks of records that are accumulating all around us, as efficiently as possible. I actually am hopeful that in an “arms race” between on the one hand, preserving the newly created exabytes of record information and beyond, and on the other, employing new forms of AI to extract meaning from big data sets, that the AI will in the end win.
I can’t think of anything more exciting for a young lawyer to contemplate than thinking about how to apply state-of-the-art technology in continuing to tackle the volume and complexity issues posed by e-discovery. IMHO, your recent columns should be required reading in every civil procedure course taught in US law schools. Doug Oard and I actually have your search columns already listed on our syllabus for the e-discovery course we’re teaching this Spring to PhD and Masters candidates at U. Maryland’s School of Information Studies.
Thanks Jason. As usual, you were right about what we need next in e-discovery, namely standards. I am once again following in your footsteps by working on a foundation for these standards. Also, thanks for tolerating my false emphasis (I would rather say poetic license) of your information dystopia comment to serve as my foil. Dear readers, Jason is not really a prophet of doom. In fact, he is one of the funniest people I know.
All of us in the eDiscovery field are seeking to tame the “crazy high” costs of document review as you stated. We share the same dream. For in-house counsels such as myself who are actually writing the “check” for those services, the quest for starndards, is as pressing as it will ever be. The cost of review, however, is just one piece of the puzzle. I would like to go one step further and try to come up with estimates for the total cost of eDiscovery or total cost of document production. The challenge in estimating the total cost of production lies in that we have to take into account all that will be collected, processed, reviewed and, ultimately, the small set that will be produced. It is the same challenge in estimating a billable attorney’s rate, when you have to take into account all the overhead in a practice. In order to estimate the total cost of production, from an in-house perspective, one has to account for:
1) collection: costs of the software (if collection is done in-house), or costs of collection (if done by a 3rd party)
2)processing: costs of culling the data and costs of hosting the dataset plus access to reviewers
3) review: costs of contract attorneys for 1st pass, costs of QC, costs of lawfirm for secondary review and privilege, confidentiality codding/logging/redacting
4) production: costs of tiffing, converting data to media for production, costs of hardware.
All those costs have to be factored in when you are trying to determine the total cost per document ultimately produced.
Therein lies the challenge. Sorry to add to your already herculean effort to create standards but, to the extent that we are seeking standardization, I believe we ought to aim for creating processes that will give clients complete cost predictability and visibility. Thanks.
There are yet additional costs — such as the outside vendor’s hosting charges for the data collected. Vendors typically charge for this on a gigabyte basis and those costs can be quite high in a large volume case — e.g., $40,000-$50,000 a month. To the extent that a case and the discovery phase goes on for an extended period of time, those high, monthly costs to keep the data “alive” and available for subsequent searches to use in deposition preparation or for motions purposes can really mount. And, let’s not forget that there will be hosting charges for the data that is produced from the other side — which the receiving party will want to have available for the same types of searches and factual document research as the case progresses. (Query: Why is it that none of us in the e-discovery world ever speaks about the parallel search and review cost (and hosting fees) associated with examining the documents produced by the opposing party or parties? . . . All the steps and costs associated with producing documents in response to discovery is only one half of the story. Granted with documents produced to us, we don’t have to bear the preservation costs, but basically all the rest remains. Just wondering . . . .)
In my experience, most litigation is asymetrical, in that one party (usually plaintiff or government agency) is the recipient of far greater amount of documents than the responding party. So the responding party only has to worry about hosting its own documents, and the requestor has to worry about hosting the docs produced to them.
Even if the litigation is symetrical, the production volume is about 10% of the collected volume by the other side, so the relative added burden to the recipient is not that great.
[…] in the legal blogosphere lately about the cost of production in e-discovery, including here, here, and here. Most of the attention is focused on improving the review process in order to limit […]
[…] the triumvirate opine on proportionality and how to control costs. As you know from my last blog, Bottom Line Driven Proportional Review, cost control is my current mission, my way of helping Superman in the never-ending battle for […]
[…] Bottom Line Driven Proportional Review […]
[…] Bottom Line Driven Proportional Review […]
Ralph,
I just finished reading your blog entry about Bottom Line Driven Proportional Review, and my initial response is that I am not sure if I buy into it.
I certainly agree that the bottom line is important and that any review should be proportional to the amount at issue in the case, but I find something inherently discomforting about deciding how many documents to review based upon a predetermined budget. If there are truly 30,000 relevant documents in a matter, how can I say I will only look at, and potentially produce 10,000 of them? Is this not a partial truth? If so, how is that really justice?
Of course proportionality must have its place. My concern with your method is that it may place too much emphasis on being able to choose what you will review and produce based on budget alone (ignoring the reality of how many responsive documents there are), and provide lawyers and clients an easy out that allows them to artificially truncate document review. To use a simplified (and perhaps somewhat antiquated) analogy, if you had a case about a motorcycle accident, I would argue it would be difficult to not use the search term “motorcycle accident” simply because it returned more hits than you budgeted for.
I think you attempt to address my issue by cautioning that you have to get your hands dirty and understand the data before creating a budget, and ideally, this would help. Unfortunately, in practice, too often the budget is created without input from e-discovery experts, and/or at times, the people creating the budget are not allowed access to the data until after they have submitted a budget and then been engaged for the work.
As I said, I just read this, so I need to think about it more, but I am left with the feeling that I have to choose between proportionality and the “complete” truth, and furthermore, that Bottom Line Driven Proportional Review may become a tool for abuse and misuse. Perhaps I am being cynical and need to have more faith in my fellow colleagues. Regardless, I do appreciate that you are trying to solve a significant problem, and therefore, whether or not I ultimately agree with you, what you wrote certainly has value to me.
Thanks,
Brandon
“If there are truly 30,000 relevant documents in a matter, how can I say I will only look at, and potentially produce 10,000 of them? Is this not a partial truth?”
Because you can’t know you have 30,000 relevant documents in a matter going in. In fact, you can never know how many relevant documents there are even after you’re done with the review! This is because you don’t know how many were missed by culling or by the reviewers.
It’s a world where a multiplication of imperfections add up to reasonableness. You are going in blind, but you’re coming out only with somewhat better vision. How much better? You’ll never know, unless the universe of documents can fit in a stack of paper on your desk.
“Complete truth” is not absolute. It’s the complete truth within the universe that makes sense given what’s at stake in the case. That’s your proportionality.
[…] in reasonability, but they are also unrealistic and contra to the latest scientific research. In my Bottom Line Driven Proportional Review article I showed how this kind of demand for all relevant ESI is not permitted under the rules and […]
[…] Part III. My own work has been driven by this hacker focus on impact and led to my development of Bottom Line Driven Proportional Review. Other hacker oriented lawyers and technologists have developed their own methods to give clients […]
[…] To maximize impact e-discovery teams everywhere should focus on these two, costs and training. They should look for bold new ways to control costs and train attorneys. As discussed briefly in “The Hacker Way” – What e-Discovery Can Learn From Facebook’s Culture and Management, for me cost control means focusing on search, since review is the most expensive part of any production project. It also means building new aggressive culling methods, such as Bottom Line Driven Proportional Review. […]
[…] Bottom Line Driven Proportional Review […]
[…] predictive coding culling is affordable in all sizes of cases. It is the ideal tool to implement my Bottom Line Driven Proportional Review […]
[…] 12 Journal of Technology Law & Policy 1 (June 2007). For more on cost control see my blog Bottom Line Driven Proportional Review. For more on the new approaches in general and predictive coding in particular, see eg. Chris […]
[…] Also see Final Report on the Joint Project of the American College of Trial Lawyers and the Institute for the Advancement of the American Legal System (2009) (At page 7: “Proportionality should be the most important principle applied to all discovery.”) Also see my recent blog: Bottom Line Driven Proportional Review. […]
[…] my blog, Bottom Line Driven Proportional Review I added the sixth idea, where the process gets real and takes money into consideration. In that […]
[…] Bottom Line Driven Proportional Review; […]
[…] new steps to control e-discovery costs, to make them proportionate. That is why I came up with my Bottom Line Driven Proportional Review approach. But the Patent Committee approach has the advantage of far greater simplicity. Moreover, […]
[…] legal method I promote for CAR is called: Bottom Line Driven Proportional Review. It is based on the well established legal doctrine of proportionality. See eg.: Good, Better, […]
[…] on my blog and went public with something I’ve been doing internally at my law firms and that’s bottom-line-driven proportional review. This is something we try to do every chance we get in every case to make sure that our production […]
Hallelujah. I believe in your dream and I think it is essential that it come true.
This article is required reading for the “Budgeting, Risk Management, and Cost Control in E-Discovery” class taught by Scott Cohen at Bryan University. Many of us in the e-discovery program there subscribe to your blog and are better educated for it. Thank you.
[…] I think this means they used enhanced keyword searches with concept search type expansion of keywords for each topic. Seems similar to the other participants’ description, but they used different software to do it. As the Legal Track Results page shows, TCDI used the automatic (Borg) approach in all of its test runs, and not the TechAssist (Hybrid) approach. They relied upon mathematics, more than Man, including a couple of my favorites, the Golden Ratio and prime numbers. See Eg. Good, Better, Best: a Tale of Three Proportionality Cases – Part One; and Bottom Line Driven Proportional Review. […]
[…] Bottom Line Driven Proportional Review (2013 Updated Version) […]
[…] in reasonability, but they are also unrealistic and contra to the latest scientific research. In my Bottom Line Driven Proportional Review article I showed how this kind of demand for all relevant ESI is not permitted under the rules and […]
[…] Bottom Line Driven Proportional Review (2013 Updated Version) – http://bit.ly/Ydy7Et (@RalphLosey) […]
[…] Bottom Line Driven Proportional Review (2013 Updated Version) – http://bit.ly/Ydy7Et (@RalphLosey) […]
[…] Bottom Line Driven Proportional Review 2013 Updated Version | e-Discovery Team ®. […]
[…] Bottom Line Driven Proportional Review (2013 Updated Version) | e-Discovery Team ®. […]
[…] Bottom Line Driven Proportional Review (2013 Updated Version). […]
[…] Multimodal Computer Assisted Review Bottom Line Driven Proportional Strategy. See eg. Bottom Line Driven Proportional Review (2013). I refer to it as a multimodal method because, although the predictive coding type of […]
[…] of harnessing and expanding the power of a few highly skilled legal experts. The future is the Army of One review […]
[…] and focusing on the SMEs, not the contract reviewers. They understand that the future is the Army of One review team, not hordes of contract lawyers. They understand how AI can change the law, can bring […]
[…] is what keeps bottom line driven proportional review fair and equitable. See Bottom Line Driven Proportional Review (2013 Updated Version). That is why it is the key feature you should focus on when making a software licensing […]
[…] Multimodal Computer Assisted Review Bottom Line Driven Proportional Strategy. See eg. Bottom Line Driven Proportional Review (2013). I refer to it as a multimodal method because, although the predictive coding type of […]
[…] Bottom Line Driven Proportional Review (2013 Updated Version). […]
[…] and focusing on the SMEs, not the contract reviewers. They understand that the future is the Army of One review team, not hordes of contract lawyers. They understand how AI can change the law, can bring […]