Project Cost Estimation Is Key To Opposing ESI Discovery as Disproportionately Burdensome Under Rule 26(b)(1)

May 6, 2018

If you are opposing ESI discovery as over-burdensome under Rule 26(b)(1), then you MUST provide evidence of the economic burden of the requested review. You cannot just say it is over-burdensome. Even if it seems obvious, you must provide some metrics, some data, some hard evidence to back that up. That requires the ability to estimate the costs and burdens involved in a document review. In the old days, the nineties, almost every litigator could estimate the cost of a paper review. It was not a tough skill. But today, where large volumes of ESI are common, everything is much more complicated. Today you need an expert to accurately and reliably estimate the costs of various types of ESI reviews.

Requiring proof of burden is nothing new to the law, yet most lawyers today need outside help to do it, especially in large ESI projects. For example, consider the defense team of lawyers representing the City of Chicago and other defendants in a major civil rights case with lots of press, Mann v. City of Chicago, Nos. 15 CV 9197, 13 CV 4531, (N.D. Ill. Sept. 8, 2017); Chicago sued for ‘unconstitutional and torturous’ Homan Square police abuse (The Guardian, 10/19/15). They did not even attempt to estimate the costs of the review they opposed. They also failed or refused to hire an expert who could do that for them. Sine they had no evidence, not even an estimate, their argument under Rule 26(b)(1) failed miserably.

Mann v. City of Chicago: Case Background

The background of the case is interesting, but I won’t go into the fact details here; just enough to set up the discovery dispute. Plaintiffs in later consolidated cases sued the City of Chicago and the Chicago police alleging that they had been wrongfully arrested, detained and abused at “off the books” detention centers without access to an attorney. Aside from the salacious allegations, it does not look like the plaintiffs have a strong case. It looks like a fishing expedition to me, in more ways than one as I will explain. With this background, it seems to me that if defendants had made any real effort to prove burden here, they could have prevailed on this discovery dispute.

The parties agreed on the majority of custodians whose ESI would be searched, but, as usual, the plaintiffs’ wanted more custodians searched, including that of the mayor himself, Rahm Emanuel. The defendants did not want to include the mayor’s email in the review. They argued, without any real facts showing burden, that the Mayor’s email would be irrelevant (a dubious argument that seemed to be a throw-away) and too burdensome (their real argument).

Here is how Magistrate Judge Mary M. Rowland summarized the custodian dispute in her opinion:

Plaintiffs argue Mayor Emanuel and ten members of his senior staff, including current and former chiefs of staff and communications directors are relevant to Plaintiffs’ Monell claim. (Id. at 5).[2] The City responds that Plaintiffs’ request is burdensome, and that Plaintiffs have failed to provide any grounds to believe that the proposed custodians were involved with CPD’s policies and practices at Homan Square. (Dkt. 74 at 1, 6). The City proposes instead that it search the two members of the Mayor’s staff responsible for liasoning with the CPD and leave “the door open for additional custodians” depending on the results of that search. (Id. at 2, 4).[3]

Another Silly “Go Fish” Case

As further background, this is one of those negotiated keywords Go Fish cases where the attorneys involved all thought they had the magical powers to divine what words were used in relevant ESI. The list is not shared, but I bet it included wondrous words like “torture” and “off the books,” plus every plaintiff’s favorite “claim.”

The parties agreed that the defendants would only review for relevant evidence the ESI of the custodians that happened to have one or more of the keyword incantations they dreamed up. Under this still all to common practice the attorneys involved, none of whom appear to have any e-discovery search expertise, the majority of documents in the custody of the defense custodians would never be reviewed. They would not be reviewed because they did not happen to have a “magic word” in them. This kind of untested, keyword filtering agreement is irrational, archaic and not a best practice in any but small cases, but that is what the attorneys for both sides agreed to. They were convinced they could guess that words were used by police, city administrators and politicians in any relevant document. It is a common delusion facilitated by Google’s search of websites.

When will the legal profession grow up and stop playing Go Fish when it comes to a search for relevant legal evidence? I have been writing about this for years. Losey, R., Adventures in Electronic Discovery (West 2011); Child’s Game of ‘Go Fish’ is a Poor Model for e-Discovery Search. Guessing keywords does not work. It almost always fails in both precision and recall. The keyword hits docs are usually filled with junk and relevant docs often used unexpected language, not to mention abbreviations and spelling errors. If you do not at least test proposed keywords on a sample custodian, then your error rate will multiply. I saw a review recently where the precision rate on keywords was only six percent, and that is with superficial feedback, i.w. – unskilled testing. You never want to waste so much attorney time, even if you are reviewing at low rates. The ninety-four irrelevant docs to find six is an inefficient expensive approach. We try to improve precision without a significant loss of recall.

When I first wrote about Go Fish and keywords back in 2010 most everyone agreed with me, even if they disagreed on the significance, the meaning and what you should do about it. That started the proportionality debate in legal search. E-Discovery search expert Judges Peck and Scheindlin joined in the chorus of criticism of negotiated keywords. National Day Laborer Organizing Network v. US Immigration and Customs Enforcement Agency, 877 F.Supp.2d 87 (SDNY, 2012) (J. Scheindlin) (“As Judge Andrew Peck — one of this Court’s experts in e-discovery — recently put it: “In too many cases, however, the way lawyers choose keywords is the equivalent of the child’s game of `Go Fish’ … keyword searches usually are not very effective.” FN 113“); Losey, R., Poor Plaintiff’s Counsel, Can’t Even Find a CAR, Much Less Drive One (9/1/13). Don’t you love the quote within a quote. A rare gem in legal writing.

Judge Rowland’s Ruling

I have previously written about the author of the Mann v. City of Chicago opinion, Judge Mary Rowland. Spoliated Schmalz: New Sanctions Case in Chicago That Passes-Over a Mandatory Adverse Inference. She is a rising star in the e-discovery world. Judge Rowland found that the information sought from the additional custodians would be relevant. This disposed of the defendants first and weakest argument. Judge Rowland then held that Defendants did not meet the burden of proof “—failing to provide even an estimate—” and for that reason granted, in part, Plaintiffs’ motion to compel, including their request to add the Mayor. Judge Rowland reviewed all six of the proportionality factors under Rule 26(b)(1), including the importance of the issues at stake and the plaintiffs’ lack of access to the requested information.

On the relevance issue Judge Rowland held that, in addition to the agreed-upon staff liaisons, the Mayor and his “upper level staff” might also have relevant information in their email. As to the burden argument, Judge Rowland held that the City did not “offer any specifics or even a rough estimate about the burden.” Judge Rowland correctly rejected the City’s argument that they could not provide any such information because “it is impossible to determine how many emails there may be ‘unless the City actually runs the searches and collects the material.’” Instead, the court held that the defendants should have at least provided “an estimate of the burden.” Smart Judge. Here are her words:

The City argues that it will be “burdened with the time and expense of searching the email boxes of nine (9) additional custodians.” (Dkt. 74 at 5). The City does not offer any specifics or even a rough estimate about the burden. See Kleen Prods. LLC 2012 U.S. Dist. LEXIS 139632, at *48 (“[A] party must articulate and provide evidence of its burden. While a discovery request can be denied if the `burden or expense of the proposed discovery outweighs its likely benefit,’ Fed. R. Civ. P. 26(b)(2)(C)(iii), a party objecting to discovery must specifically demonstrate how the request is burdensome.”) (internal citations and quotations omitted).

As the Seventh Circuit stated in Heraeus Kulzer, GmbH, v. Biomet, Inc., 633 F.3d 591, 598 (7th Cir. 2011):

[The party] could have given the district court an estimate of the number of documents that it would be required to provide Heraeus in order to comply with the request, the number of hours of work by lawyers and paralegals required, and the expense. A specific showing of burden is commonly required by district judges faced with objections to the scope of discovery . . . Rough estimates would have sufficed; none, rough or polished, was offered.

The City argues in its sur-reply that it is impossible to determine how many emails there may be “unless the City actually runs the searches and collects the material.” (Dkt. 78-1 at 4). Still, the City should have provided an estimate of the burden. The Court is not convinced by the City’s argument about the burden.

Judge Rowland also held that the City should have addressed the “other Rule 26 factors—the importance of the issues and of the discovery in resolving the issues, and the parties’ relative access to information and their resources.” She noted that these other factors: “weigh[ed] in favor of allowing discovery of more than just the two custodians proposed by the City.”  However, the court declined to compel the search of four proposed custodians based on their “short tenure” or the “time during which the person held the position,” concluding the requested searches were “not proportional to the needs of the case.”

Judge Rowland’s opinion notes with seeming surprise the failure of the City of Chicago to provide any argument at all on the five non-economic factors in Rule 26(b)(1). I do not fault them for that. Their arguments on these points were necessarily weak in this type of case, but a conciliatory gesture, a polite acknowledgement showing awareness, might have helped sweeten the vinegar. As it is, they came across as oblivious to the full requirements of the Rule.

What Chicago Should Have Done

What additional information should the defendants have provided to oppose the search and review of the additional nine custodians, including the Mayor’s email? Let’s start with the obvious. They should have shared the total document count and GB size of the nine custodians, and they should have broken that information down on a per-custodian basis. Then they should have estimated the costs to review that many emails and attachments.

The file count information should have been easy to ascertain from the City’s IT department. They know the PST sizes and can also determine, or at least provide a good estimate of the total document count. The problem they had with this obvious approach is that they wanted a keyword filter. They did not want to search all documents of the custodians, only the ones with keyword hits. Still, that just made the process slightly more difficult, not impossible.

Yes, it is true, as defendant’s alleged, that to ascertain this supporting information, they would have to run the searches and collect the material. So what? Their vendor or Chicago IT department should have helped them with that. It is not that difficult or expensive to do. No expensive lawyer time is required. It is just a computer process. Any computer technician could do it. Certainly any e-discovery vendor. The City could easily have gone ahead and done the silly keyword filtering and provide an actual file count. This would have provided the City some hard facts to support their burden argument. It should not be that expensive to do. Almost certainly the expense would have been less than this motion practice.

Alternatively, the City could have at least estimated the file count and other burden metrics. They could have made reasonable estimated based on their document review experience in the case so far. They had already reviewed uncontested custodians under their Go Fish structure, so they could have made projections based on past results. Estimates made by projections like this would probably have been sufficient in this case and was certainly better than the track they chose, not providing any information at all.

Another alternative, the one that would have produced the most persuasive evidence, would be to load the filtered ESI of at least a sample of the nine custodians, including the Mayor. Then begin the review, say for a couple of days, and see what that costs. Then project those costs for the rest of the review and rest of the custodians. By this gold standard approach you would not only have the metrics from the data itself — the file counts, page counts, GB size — but also metrics of the document review, what it costs.

You would need to do this on the Mayor’s email separately and argue this burden separately. The Mayor’s email would likely be much more expensive to review than any of the other custodians. It would take attorneys longer to review his documents. There would be more privileged materials to find and log and there would be more redactions. It is like reviewing a CEO’s email. If the attorneys for the City had at least begun some review of Emanuel’s email, they would have been able to provide extensive evidence on the cost and time burden to complete the review.

I suspect the Mayor was the real target here and the other eight custodians were of much less importance. The defense should have gauged their response accordingly. Instead, they did little or nothing to support their burdensome argument, even with the Mayor’s sensitive government email account.

We have a chance to learn from Chicago’s mistake. Always, at the very least, provide some kind of an estimate of the burden. The estimate should include as much information as possible, including time and costs. These estimates can, with time and knowledge, be quite accurate and should be used to set budgets, along with general historical knowledge of costs and expenses. The biggest problem now is a shortage of experts on how to properly estimate document review projects, specifically large ESI-only projects. I suggest you consult with such an cost-expert anytime you are faced with a disproportionate ESI review demands. You should do so before you make final decisions or reply in writing.

 Conclusion

Mann v. City of Chicago is one of those cases where we can learn from the mistakes of others. At least provide an estimate of costs in every dispute under Rule 26(b)(1). Learn to estimate the costs of document reviews. Either that or hire an expert who can do that for you, one that can provide testimony. Start with file counts and go from there. Always have some metrics to back-up your argument. Learn about your data. Learn what it will likely cost to review that data. Learn how to estimate the costs of document reviews. It will probably be a range. The best way to do that is by sampling. With sampling you at least start the document review and estimate total costs by projection of what it has actually cost to date. There are fewer speculative factors that way.

If you agree to part of the review requested, for instance to three out of ten custodians requested, then do that review and measure its costs. That creates the gold standard for metrics of burden under Rule 26(b)(1) and is, after all, required in any objections under Rule 34(b)(2)(B)&(C). See: Judge Peck Orders All Lawyers in NY to Follow the Rules when Objecting to Requests for Production, or Else ….

For more on cost burden estimation listen to my upcoming Ed-Talk on the subject, Proportional Document Review under the New Rules and the Art of Cost Estimation.

 



WHY I LOVE PREDICTIVE CODING: Making Document Review Fun Again with Mr. EDR and Predictive Coding 4.0

December 3, 2017

Many lawyers and technologists like predictive coding and recommend it to their colleagues. They have good reasons. It has worked for them. It has allowed them to do e-discovery reviews in an effective, cost efficient manner, especially the big projects. That is true for me too, but that is not why I love predictive coding. My feelings come from the excitement, fun, and amazement that often arise from seeing it in action, first hand. I love watching the predictive coding features in my software find documents that I could never have found on my own. I love the way the AI in the software helps me to do the impossible. I really love how it makes me far smarter and skilled than I really am.

I have been getting those kinds of positive feelings consistently by using the latest Predictive Coding 4.0 methodology (shown right) and KrolLDiscovery’s latest eDiscovery.com Review software (“EDR”). So too have my e-Discovery Team members who helped me to participate in TREC 2015 and 2016 (the great science experiment for the latest text search techniques sponsored by the National Institute of Standards and Technology). During our grueling forty-five days of experiments in 2015, and again for sixty days in 2016, we came to admire the intelligence of the new EDR software so much that we decided to personalize the AI as a robot. We named him Mr. EDR out of respect. He even has his own website now, MrEDR.com, where he explains how he helped my e-Discovery Team in the 2015 and 2015 TREC Total Recall Track experiments.

Bottom line for us from this research was to prove and improve our methods. Our latest version 4.0 of Predictive Coding, Hybrid Multimodal IST Method is the result. We have even open-sourced this method, well most of it, and teach it in a free seventeen-class online program: TARcourse.com. Aside from testing and improving our methods, another, perhaps even more important result of TREC for us was our rediscovery that with good teamwork, and good software like Mr. EDR at your side, document review need never be boring again. The documents themselves may well be boring as hell, that’s another matter, but the search for them need not be.

How and Why Predictive Coding is Fun

Steps Four, Five and Six of the standard eight-step workflow for Predictive Coding 4.0 is where we work with the active machine-learning features of Mr. EDR. These are its predictive coding features, a type of artificial intelligence. We train the computer on our conception of relevance by showing it relevant and irrelevant documents that we have found. The software is designed to then go out and find all other relevant documents in the total dataset. One of the skills we learn is when we have taught enough and can stop the training and complete the document review. At TREC we call that the Stop decision. It is important to keep down the costs of document review.

We use a multimodal approach to find training documents, meaning we use all of the other search features of Mr. EDR to find relevant ESI, such as keyword searches, similarity and concept. We iterate the training by sample documents, both relevant and irrelevant, until the computer starts to understand the scope of relevance we have in mind. It is a training exercise to make our AI smart, to get it to understand the basic ideas of relevance for that case. It usually takes multiple rounds of training for Mr. EDR to understand what we have in mind. But he is a fast learner, and by using the latest hybrid multimodal IST (“intelligently spaced learning“) techniques, we can usually complete his training in a few days. At TREC, where we were moving fast after hours with the Ã-Team, we completed some of the training experiments in just a few hours.

After a while Mr. EDR starts to “get it,” he starts to really understand what we are after, what we think is relevant in the case. That is when a happy shock and awe type moment can happen. That is when Mr. EDR’s intelligence and search abilities start to exceed our own. Yes. It happens. The pupil then starts to evolve beyond his teachers. The smart algorithms start to see patterns and find evidence invisible to us. At that point we sometimes even let him train himself by automatically accepting his top-ranked predicted relevant documents without even looking at them. Our main role then is to determine a good range for the automatic acceptance and do some spot-checking. We are, in effect, allowing Mr. EDR to take over the review. Oh what a feeling to then watch what happens, to see him keep finding new relevant documents and keep getting smarter and smarter by his own self-programming. That is the special AI-high that makes it so much fun to work with Predictive Coding 4.0 and Mr. EDR.

It does not happen in every project, but with the new Predictive Coding 4.0 methods and the latest Mr. EDR, we are seeing this kind of transformation happen more and more often. It is a tipping point in the review when we see Mr. EDR go beyond us. He starts to unearth relevant documents that my team would never even have thought to look for. The relevant documents he finds are sometimes completely dissimilar to any others we found before. They do not have the same keywords, or even the same known concepts. Still, Mr. EDR sees patterns in these documents that we do not. He can find the hidden gems of relevance, even outliers and black swans, if they exist. When he starts to train himself, that is the point in the review when we think of Mr. EDR as going into superhero mode. At least, that is the way my young e-Discovery Team members likes to talk about him.

By the end of many projects the algorithmic functions of Mr. EDR have attained a higher intelligence and skill level than our own (at least on the task of finding the relevant evidence in the document collection). He is always lighting fast and inexhaustible, even untrained, but by the end of his training, he becomes a search genius. Watching Mr. EDR in that kind of superhero mode is what makes Predictive Coding 4.0 a pleasure.

The Empowerment of AI Augmented Search

It is hard to describe the combination of pride and excitement you feel when Mr. EDR, your student, takes your training and then goes beyond you. More than that, the super-AI you created then empowers you to do things that would have been impossible before, absurd even. That feels pretty good too. You may not be Iron Man, or look like Robert Downey, but you will be capable of remarkable feats of legal search strength.

For instance, using Mr. EDR as our Iron Man-like suits, my e-discovery Ã-Team of three attorneys was able to do thirty different review projects and classify 17,014,085 documents in 45 days. See 2015 TREC experiment summary at Mr. EDR. We did these projects mostly at nights, and on weekends, while holding down our regular jobs. What makes this crazy impossible, is that we were able to accomplish this by only personally reviewing 32,916 documents. That is less than 0.2% of the total collection. That means we relied on predictive coding to do 99.8% of our review work. Incredible, but true.

Using traditional linear review methods it would have taken us 45 years to review that many documents! Instead, we did it in 45 days. Plus our recall and precision rates were insanely good. We even scored 100% precision and 100% recall in one TREC project in 2015 and two more in 2016. You read that right. Perfection. Many of our other projects attained scores in the high and mid nineties. We are not saying you will get results like that. Every project is different, and some are much more difficult than others. But we are saying that this kind of AI-enhanced review is not only fast and efficient, it is effective.

Yes, it’s pretty cool when your little AI creation does all the work for you and makes you look good. Still, no robot could do this without your training and supervision. We are a team, which is why we call it hybrid multimodal, man and machine.

Having Fun with Scientific Research at TREC 2015 and 2016

During the 2015 TREC Total Recall Track experiments my team would sometimes get totally lost on a few of the really hard Topics. We were not given legal issues to search, as usual. They were arcane technical hacker issues, political issues, or local news stories. Not only were we in new fields, the scope of relevance of the thirty Topics was never really explained. (We were given one to three word explanations in 2015, in 2016 we got a whole sentence!) We had to figure out intended relevance during the project based on feedback from the automated TREC document adjudication system. We would have some limited understanding of relevance based on our suppositions of the initial keyword hints, and so we could begin to train Mr. EDR with that. But, in several Topics, we never had any real understanding of exactly what TREC thought was relevant.

This was a very frustrating situation at first, but, and here is the cool thing, even though we did not know, Mr. EDR knew. That’s right. He saw the TREC patterns of relevance hidden to us mere mortals. In many of the thirty Topics we would just sit back and let him do all of the driving, like a Google car. We would often just cheer him on (and each other) as the TREC systems kept saying Mr. EDR was right, the documents he selected were relevant. The truth is, during much of the 45 days of TREC we were like kids in a candy store having a great time. That is when we decided to give Mr. EDR a cape and superhero status. He never let us down. It is a great feeling to create an AI with greater intelligence than your own and then see it augment and improve your legal work. It is truly a hybrid human-machine partnership at its best.

I hope you get the opportunity to experience this for yourself someday. The TREC experiments in 2015 and 2016 on recall in predictive coding are over, but the search for truth and justice goes on in lawsuits across the country. Try it on your next document review project.

Do What You Love and Love What You Do

Mr. EDR, and other good predictive coding software like it, can augment our own abilities and make us incredibly productive. This is why I love predictive coding and would not trade it for any other legal activity I have ever done (although I have had similar highs from oral arguments that went great, or the rush that comes from winning a big case).

The excitement of predictive coding comes through clearly when Mr. EDR is fully trained and able to carry on without you. It is a kind of Kurzweilian mini-singularity event. It usually happens near the end of the project, but can happen earlier when your computer catches on to what you want and starts to find the hidden gems you missed. I suggest you give Predictive Coding 4.0 and Mr. EDR a try. To make it easier I open-sourced our latest method and created an online course. TARcourse.com. It will teach anyone our method, if they have the right software. Learn the method, get the software and then you too can have fun with evidence search. You too can love what you do. Document review need never be boring again.

Caution

One note of caution: most e-discovery vendors, including the largest, do not have active machine learning features built into their document review software. Even the few that have active machine learning do not necessarily follow the Hybrid Multimodal IST Predictive Coding 4.0 approach that we used to attain these results. They instead rely entirely on machine-selected documents for training, or even worse, rely entirely on random selected documents to train the software, or have elaborate unnecessary secret control sets.

The algorithms used by some vendors who say they have “predictive coding” or “artificial intelligence” are not very good. Scientists tell me that some are only dressed-up concept search or unsupervised document clustering. Only bona fide active machine learning algorithms create the kind of AI experience that I am talking about. Software for document review that does not have any active machine learning features may be cheap, and may be popular, but they lack the power that I love. Without active machine learning, which is fundamentally different from just “analytics,” it is not possible to boost your intelligence with AI. So beware of software that just says it has advanced analytics. Ask if it has “active machine learning“?

It is impossible to do the things described in this essay unless the software you are using has active machine learning features.  This is clearly the way of the future. It is what makes document review enjoyable and why I love to do big projects. It turns scary to fun.

So, if you tried “predictive coding” or “advanced analytics” before, and it did not work for you, it could well be the software’s fault, not yours. Or it could be the poor method you were following. The method that we developed in Da Silva Moore, where my firm represented the defense, was a version 1.0 method. Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182, 183 (S.D.N.Y. 2012). We have come a long way since then. We have eliminated unnecessary random control sets and gone to continuous training, instead of train then review. This is spelled out in the TARcourse.com that teaches our latest version 4.0 techniques.

The new 4.0 methods are not hard to follow. The TARcourse.com puts our methods online and even teaches the theory and practice. And the 4.0 methods certainly will work. We have proven that at TREC, but only if you have good software. With just a little training, and some help at first from consultants (most vendors with bona fide active machine learning features will have good ones to help), you can have the kind of success and excitement that I am talking about.

Do not give up if it does not work for you the first time, especially in a complex project. Try another vendor instead, one that may have better software and better consultants. Also, be sure that your consultants are Predictive Coding 4.0 experts, and that you follow their advice. Finally, remember that the cheapest software is almost never the best, and, in the long run will cost you a small fortune in wasted time and frustration.

Conclusion

Love what you do. It is a great feeling and sure fire way to job satisfaction and success. With these new predictive coding technologies it is easier than ever to love e-discovery. Try them out. Treat yourself to the AI high that comes from using smart machine learning software and fast computers. There is nothing else like it. If you switch to the 4.0 methods and software, you too can know that thrill. You can watch an advanced intelligence, which you helped create, exceed your own abilities, exceed anyone’s abilities. You can sit back and watch Mr. EDR complete your search for you. You can watch him do so in record time and with record results. It is amazing to see good software find documents that you know you would never have found on your own.

Predictive coding AI in superhero mode can be exciting to watch. Why deprive yourself of that? Who says document review has to be slow and boring? Start making the practice of law fun again.

Here is the PDF version of this article, which you may download and distribute, so long as you do not revise it or charge for it.

 

 


Six Sets of Draft Principles Are Now Listed at AI-Ethics.com

October 8, 2017

Arguably the most important information resource of AI-Ethics.com is the page with the collection of Draft Principles underway by other AI Ethics groups around the world. We added a new one that came to our attention this week from an ABA article, A ‘principled’ artificial intelligence could improve justice (ABA Legal Rebels, October 3, 2017). They listed six proposed principles from the talented Nicolas Economou, the CEO of electronic discovery search company, H5.

Although Nicolas Economou is an e-discovery search pioneer and past Sedona participant, I do not know him. I was, of course, familiar with H5’s work as one of the early TREC Legal Track pioneers, but I had no idea Economou was also involved with AI ethics. Interestingly, I recently learned that another legal search expert, Maura Grossman, whom I do know quite well, is also interested in AI ethics. She is even teaching a course on AI ethics at Waterloo. All three of us seem to have independently heard the Siren’s song.

With the addition of Economou’s draft Principles we now have six different sets of AI Ethics principles listed. Economou’s new list is added at the end of the page and reproduced below. It presents a decidedly e-discovery view with which all readers here are familiar.

Nicolas Economou, like many of us, is an alumni of The Sedona Conference. His sixth principle is based on what he calls thoughtful, inclusive dialogue with civil society. Sedona was the first legal group to try to incorporate the principles of dialogue into continuing legal education programs. That is what first attracted me to The Sedona Conference. AI-Ethics.com intends to incorporate dialogue principles in conferences that it will sponsor in the future. This is explained in the Mission Statement page of AI-Ethics.com.

The mission of AI-Ethics.com is threefold:

  1. Foster dialogue between the conflicting camps in the current AI ethics debate.
  2. Help articulate basic regulatory principles for government and industry groups.
  3. Inspire and educate everyone on the importance of artificial intelligence.

First Mission: Foster Dialogue Between Opposing Camps

The first, threshold mission of AI-Ethics.com is to go beyond argumentative debates, formal and informal, and move to dialogue between the competing camps. See eg. Bohm Dialogue, Martin Buber and The Sedona Conference. Then, once this conflict is resolved, we will all be in a much better position to attain the other two goals. We need experienced mediators, dialogue specialists and judges to help us with that first goal. Although we already have many lined up, we could always use more.

We hope to use skills in both dialogue and mediation to transcend the polarized bickering that now tends to dominate AI ethics discussions. See eg. AI Ethics Debate. We need to move from debate to dialogue, and we need to do so fast.

_____

Here is the new segment we added to the Draft Principles page.

6. Nicolas Economou

The latest attempt at articulating AI Ethics principles comes from Nicolas Economou, the CEO of electronic discovery search company, H5. Nicolas has a lot of experience with legal search using AI, as do several of us at AI-Ethics.com. In addition to his work with legal search and H5, Nicholas is involved in several AI ethics groups, including the AI Initiative of the Future Society at Harvard Kennedy School and the Law Committee of the IEEE’s Global Initiative for Ethical Considerations in AI.

Nicolas Economou has obviously been thinking about AI ethics for some time. He provides a solid scientific, legal perspective based on his many years of supporting lawyers and law firms with advanced legal search. Economou has developed six principles as reported in an ABA Legal Rebels article dated October 3, 2017, A ‘principled’ artificial intelligence could improve justice. (Some of the explanations have been edited out as indicated below. Readers are encouraged to consult the full article.) As you can see the explanations given here were written for consumption by lawyers and pertain to e-discovery. They show the application of the principles in legal search. See eg TARcourse.com. The principles have obvious applications in all aspects of society, not just the Law and predictive coding, so their value goes beyond the legal applications here mentioned.

Principle 1: AI should advance the well-being of humanity, its societies, and its natural environment. The pursuit of well-being may seem a self-evident aspiration, but it is a foundational principle of particular importance given the growing prevalence, power and risks of misuse of AI and hybrid intelligence systems. In rendering the central fact-finding mission of the legal process more effective and efficient, expertly designed and executed hybrid intelligence processes can reduce errors in the determination of guilt or innocence, accelerate the resolution of disputes, and provide access to justice to parties who would otherwise lack the financial wherewithal.

Principle 2: AI should be transparent. Transparency is the ability to trace cause and effect in the decision-making pathways of algorithms and, in hybrid intelligence systems, of their operators. In discovery, for example, this may extend to the choices made in the selection of data used to train predictive coding software, of the choice of experts retained to design and execute the automated review process, or of the quality-assurance protocols utilized to affirm accuracy. …

Principle 3: Manufacturers and operators of AI should be accountable. Accountability means the ability to assign responsibility for the effects caused by AI or its operators. Courts have the ability to take corrective action or to sanction parties that deliberately use AI in a way that defeats, or places at risk, the fact-finding mission it is supposed to serve.

Principle 4: AI’s effectiveness should be measurable in the real-world applications for which it is intended. Measurability means the ability for both expert users and the ordinary citizen to gauge concretely whether AI or hybrid intelligence systems are meeting their objectives. …

Principle 5: Operators of AI systems should have appropriate competencies. None of us will get hurt if Netflix’s algorithm recommends the wrong dramedy on a Saturday evening. But when our health, our rights, our lives or our liberty depend on hybrid intelligence, such systems should be designed, executed and measured by professionals with the requisite expertise. …

Principle 6: The norms of delegation of decisions to AI systems should be codified through thoughtful, inclusive dialogue with civil society. …  The societal dialogue relating to the use of AI in electronic discovery would benefit from being even more inclusive, with more forums seeking the active participation of political scientists, sociologists, philosophers and representative groups of ordinary citizens. Even so, the realm of electronic discovery sets a hopeful example of how an inclusive dialogue can lead to broad consensus in ensuring the beneficial use of AI systems in a vital societal function.

Nicolas Economou believes, as we do, that an interdisciplinary approach, which has been employed successfully in e-discovery, is also the way to go for AI ethics. Note his use of the word “dialogue” and mention in the article of The Sedona Conference, which pioneered the use of this technique in legal education. We also believe in the power of dialogue and have seen it in action in multiple fields. See eg. the work of physicist, David Bohm and philosopher, Martin Buber. That is one reason that we propose the use of dialogue in future conferences on AI ethics. See the AI-Ethics.com Mission Statement.

_____

__

 

 


%d bloggers like this: