Contract Coders: e-Discovery’s “Wasting Asset”?

GUEST BLOG by Lawrence C. Chapin. Larry is an attorney with forty years experience in the law. He has a J.D. from Hofstra University and an LLM in Labor Law from NYU. He has also had a successful career as an executive on Wall Street, including serving as a Sr. V.P. for Kidder, Peabody & Co., and the President of Metzler Securities Corp. Larry also has a Master of Divinity degree from Union Theological Seminary. For the last eighteen months Larry has worked as a contract review lawyer on projects in New York City and Richmond, Virginia. I am proud to say that Larry is also a recent graduate of the e-Discovery Team Training program. I strongly recommend your close study of this excellent article on contract lawyers, the coders behind all large scale litigation.

Editor’s Note: The reference in the title to a “wasting asset” is a somewhat esoteric investment term. A wasting asset is a financial instrument, such as an option, that loses its time value (not its intrinsic value) with the passing of time. See Investopedia. Larry’s title is not intended to suggest that contract attorneys lose value over time, or are losing value. Instead, his use of this investment term is designed to emphasize that contract reviewers become more valuable to a project the earlier they are wisely employed in a project. Of course, it is also designed to convey a central theme of the essay that most contract coders are never properly utilized. Their talents and potential contributions to the project are wasted.

In my opinion contract reviewers are among the most exploited and unappreciated lawyers in the profession today. Their value to the successful functioning of any e-discovery team is vastly under-estimated. I publish this article with the hope that it will serve as a wake-up call to the profession to respect and properly utilize contract lawyers. We need to change the way review is done, so as to liberate the full  potential of all coders, contract, associate, partner, and paralegals alike. This not only involves new and better technologies and artificial computer intelligence, which is what I usually write about here, but also involves new and better work methods and psychology. It involves natural human intelligence. Technology without people trained and motivated to use it is worthless. So are systems based on boring drudgery. This article drives home the importance of the human element in review work. If you work with reviewers or contract lawyers in any way, or if you are one, this is a must read.


Today a young person graduating from law school and joining a large firm in one of our major cities can look forward to perhaps three or four years of doing nothing but sitting in front of a computer screen reviewing email and other electronic documents for litigation.

Kenneth J. Withers,
E-Discovery and the Combative Legal Culture:
Finding A Way Out of Purgatory[i]


Contract Coders: e-Discovery’s “Wasting Asset”?

The Wailing Wall

If document reviewers were to have a song of their own – an Anthem for the Coder Nation, let’s say – it would likely be a lament. It would be Ken Withers’ published lament of several years ago, his call to the profession that we have cited above over the purgatory into which he saw law school graduates being cast. Coincident with the ascendancy of electronic discovery, he noted, there were reports of a decline in civility and self-esteem in the legal profession. Litigators especially were losing touch “with the goals of the litigation, the long-term interests of clients, and even with each other.” The legal profession was changing in ways that put lawyers at risk of becoming just another “fierce people.” [ii]

Reviewers don’t have a song…but they do have their Wailing Wall. It is The Posse List website.  Like the Wailing Wall in old Jerusalem, it’s a place of (electronic) pilgrimage. First conceived as a clearing house for information about job opportunities for the contract labor force,[iii] the Posse List has become a place for the pouring out of hearts and the shedding of tears. At least that’s what it was recently when the news flashed across the Internet of the first-of-its-kind discovery malpractice lawsuit brought against McDermott Will & Emery.[iv]

McDermott revolved around documents produced in response to federal and state subpoenas issued during a Qui Tam whistle-blower investigation that contained numerous attorney-client privileged documents. The government attorneys asked McDermott to conduct a privilege review and to resubmit them, which the firm did using contract attorneys to conduct the review.  Once submitted, this second set of some 250,000 documents was also found to contain several thousand privileged documents. The government turned them over to the whistleblower, who refused to give them back on grounds of waiver. The producing party, J-M Manufacturing, then filed suit in California State Court alleging professional malpractice and a breach of fiduciary duty. What triggered the outpouring of angst on was the fact that it was brought not only against the McDermott law firm, but against 100 “John Doe” defendants as well, a group intended to include both unknown lawyers within the firm and other contract reviewers hired to participate in the document review process.

The Posse List said they received nearly one thousand emails about the case.[v] The reviewers wrote that the case highlighted the many problems in high volume, assembly line review. They complained about mind-numbing, monotonous work; about a lack of intellectual stimulation and an absence of quality control; about supervision more like that found in a classroom than in a law office. Under such demoralizing conditions, they said, unprofessional work was common and mistakes inevitable.[vi]

However subjective their complaints were, contract reviewers do occupy a world with certain objective realities: relatively low wages, defining uncertainties. Their labor is marked by repetitiveness more akin to assembly line work than the traditional practice of law for which most thought they were paying and being trained. It’s a culture marked by transience; contract attorneys can work side by side for months, and then, upon project completion, never see each other again. The incentive for a day of such work is the opportunity to return for another lump of the same thing.

Recently, the mainstream media has joined the chorus of lament, with the Wall Street Journal and the New York Times finding common cause in reporting the sad state of document reviewer affairs. The Journal portrayed their work as a last resort,[vii] the only port for discouraged lawyers in the current economic storm. Even in the growing field of discovery, it was a place for lawyers with shrinking hopes. The Times chimed in that however dim the personal prospects were for lawyers doing the work, there was only going to be less of it as software increasingly took their place. [viii] [ix]

The allegations in McDermott may well be a stretch, as some have said.[x] The claims may not be long-lived, now that the case has been removed to federal district court.[xi] Still, the John Doe appellation seems somehow fitting for these nameless, faceless lawyers who the system needs, but few seem to value. The work they do is everything that Withers lamented, except without the large law firm payday, or prospects.

That’s not to say that all reviewers go unappreciated. One law firm has gone so far as to acknowledge their reviewers publicly. The e-disco team of a prominent Richmond law firm has posted on their blog that “the document review team is on the front line. After a week or two of reading…they will know the overall document set better than most of the client employees. A good document review team is able to communicate that understanding to the client and to merits counsel, cementing the link that makes this team effort a success.”[xii]

The irony, made that much deeper by the prospect of more suits like McDermott, is this: something as important as the quality of the information upon which legal outcomes may depend, is itself often dependent on men and women who are seldom encouraged to consider that their work matters much at all.

There is a way out of this contradiction. It involves capitalizing on things we already know: about the essential elements of information seeking collaboration; about how to encourage and support people in the workplace; about the importance of teamwork on the job. Our sources are recent scholarly studies, depression-era workplace experiments, and the narratives of 19th century immigrants to the streets of New York City.

The Document Review Experience: A Scientific Study

The annual Text Retrieval Conference (TREC) is best known for its studies of the impact of technology on the science of information retrieval. Less well known is its related research into the human aspect of information search. As part of TREC 2009, one of the teams participating in that year’s Legal Track conducted an investigation into the nature of the collaborative information behavior of people working on electronic discovery tasks.[xiii] The study was the most recent of a series of studies that served as its foundation. The previous year’s study by the same group (from the School of Information Sciences of the University of Pittsburgh) identified three major characteristics of such behavior: first, that frequent communication is essential; second, that division of labor is common; and third, that awareness of the work of others is important.[xiv] In 2009, the team sought to focus on other aspects of “the essence” of a successful information seeking process.[xv]

The study involved two users of an electronic search tool employed to access the data collection, and a third person called the topic authority. In order to achieve diversity of opinion, the sponsors chose searchers with different backgrounds, one from the law school and the other from the school of information science. They were allowed to conduct their search individually or collaboratively. They could communicate with each other and had a shared space where they could store and organize their results. They had access to the topic authority who did not have access to the search service but contributed to the overall project by defining information needs, and judging the relevancy of results. Their assigned task was to find all of the relevant documents within the collection. Following the conclusion of the study, the three participants and the observer formed a focus group to review the results and go over their experience.

The first two sessions of search involved reading background material, exploring the collection and consulting the topic authority to ensure proper understanding. It was during this time that the searchers developed their two different search strategies. In the third and fourth sessions each searcher focused on his chosen path. In the final session, the searchers worked together to combine their results.

According to the subjective reviews of the focus group, the tasks in the first two sessions were the easiest because the searchers were working with general characteristics. It was while the searchers pursued their own separate strategies that the material became difficult, as they made judgment calls on individual documents. Although there were no tasks during these later sessions that were formally collaborative, communication remained frequent as the searchers “constantly contributed to each other with valuable information.”[xvi] Early on, this communication consisted mostly of texting, with verbal communication very rare. Later on, verbal communication between the searchers almost replaced texting, as familiarity and trust developed between them. Throughout this time, their only access to the topic authority was by phone.

While the study report itself falls somewhat short of perfect clarity, the findings are important, especially as they relate to two kinds of satisfaction important to collaborative information behavior: satisfaction with results and satisfaction with experience. First, as the difficulty of search tasks increased, collaboration between the searchers became increasingly asynchronous (intermittent and informal) and trust based.[xvii] Second, the participants were more satisfied with both the results and the experience when conducting collaborative tasks than they were with tasks that were individual and non-collaborative in character.[xviii] With these more difficult, collaborative tasks, satisfaction with search experience and satisfaction with search results went hand in hand. In other words, notably, especially with more difficult tasks, collaboration produces not just better experience, it produces better results.[xix]

Workplace Experience: The Hawthorne Effect

In the mid 20th century, the Hawthorne Works was a huge telephone factory in Chicago, the largest plant of the company that was the production arm of AT&T. In the 1920s the company undertook what became a series of studies on the impact of working conditions on productivity.[xx] In the initial study, they increased lighting for a certain group of employees at the plant. The results of the study: the more light the better. As lighting increased, so did production.

What followed was unexpected. There was no fall off for the study group when lighting was returned to former levels. They performed another study of the effects of lighting, this time with a control group. The test group had three levels of lighting, the control group had one. The surprise was that production improved for both groups.

A third experiment involved decreased lighting. As lighting fell, output still rose. Only when lighting fell to the point that workers began to complain that they could barely see, only then did production tail off. Another series of experiments continued, this time involving a variety of work breaks of different lengths. The result was a rise in productivity regardless of the length of the break. Even when they returned to a schedule that involved no breaks, output reached record levels.

Finally in 1939, the authors of the published study that resulted commented on what they had learned over time:

The volunteers lost much of their shyness and fear, or what came to be called their ‘apprehension of authority.’ They talked more freely among themselves and about themselves to officers of the company and to each other. They developed greater zest for work. New personal relations between members of the group arose which developed into strong bonds of friendship. They visited one another’s homes and went to parties, dances and the theatre together. At work it was not uncommon to find one increasing her output so that her friend, who might not be feeling well, could rest.[xxi]

Since then the Hawthorne studies have come under criticism for their methodology and controls, but a recent book on working life recalls the value of their lessons.[xxii] First, the author says, people love to experiment. Second, they want to be part of a team. Third, change a bunch of little things, and you get big change. I believe that other fundamental factors are also at work. The mere observation of an object can change the object observed.[xxiii] Paying attention to a person’s working circumstances constitutes a meaningful change in those circumstances. At work, the intention behind change can be the real change.

The Experience of Meaning In Work

Technology is driving much of the drudgery in document review. Could technology also redeem it? One discovery attorney has suggested that it might be possible to make document review fun![xxiv] The discovery industry, he says, has been focused on advanced analytical tools to organize and reduce the volume of the data to be reviewed by human beings sitting in front of computers. Much less effort, both in terms of software and process, “has been devoted to actually engage the document reviewer to enhance both the performance and quality of the review.”[xxv] He says that the answer might be “gamification,” the term for the methodologies, the tools and techniques, used by game developers in the software industry.

Video games, he observes, succeed in tapping into the pleasure centers of the brain, giving the player “a jolt of dopamine when they complete tasks, achieve goals, acquire virtual ‘loot’ and so on.” While it may be “more than just a stretch to think of document review as a ‘game,’ the effective implementation of game-style reward systems presents an opportunity to change the document review perspective and “reduce the mental fatigue and increase the engagement level of the document review team.” [xxvi]

Time will tell whether gaming will be a useful tool in the discovery industry.[xxvii] If so, it will certainly be because it has resulted in an improvement of both the information product and the search experience. Meanwhile, the gaming proposal has the intrinsic merit of innovative perspectives. Sometimes those perspectives are new. Sometimes they are from older stories, and tell us about simple things, like the meaning of hard work for those who do it.

We are accustomed to thinking a certain way about the relationship between hard work and success, Malcolm Gladwell reminds us in his book Outliers: The Story of Success.[xxviii] The story we like to tell ourselves is of self-made men and women who rise from nothing by lifting themselves up by bootstraps of their own making. Gladwell, however, says that the people who experience extraordinary success are “invariably the beneficiaries of hidden advantages and extraordinary opportunities and legacies that allow them to learn and work hard and make sense of the world in ways others cannot.” “Our ability to succeed at what we do,” he says, “is powerfully bound up with where we’re from.” The Jewish kids growing up during the depression when it was tough to be Jewish and even tougher to break into the top ranks of the legal profession;[xxix] the Asian kids whose persistence at working hard is the legacy of a hillside people who overcame the constant challenges of rice paddy cultivation;[xxx] the Jamaican girls lucky enough to have grandmothers whose foresight and energy made them “a force” in the life of their families;[xxxi] these are all Gladwell’s cases in point.

Another one of Gladwell’s stories is of the garment industry coming to the shores of America and the streets of New York City.[xxxii] In 1889, he tells us, Louis and Regina Borgenicht came to this country from Eastern Europe. Finding a tiny apartment on New York’s Lower East Side, they took to the streets in hopes of finding work. Soon they realized what would be most natural for them. It was clothes. He had grown up in the piece-goods trade of Poland, and she had run a dress making business in Hungary from the age of sixteen.  He had a good eye for novelty. She had an old sewing machine.

So they started making clothes, with a small batch of children’s aprons made of gingham, and selling for fifteen cents. When these sold out, the proceeds allowed them to buy the material for ten dozen more, and then another round as well, and then to hire others to help, and then to open a storefront of their own, and to expand their line to include adult aprons. Then it was on to petticoats and then to women’s dresses.

Gladwell explains that “when Borgenicht came home at night to his children, he may have been tired and poor and overwhelmed, but he was alive. He was his own boss. He was responsible for his own decisions and direction. He work was complex: it engaged his mind and imagination. And in his work, there was a relationship between effort and reward: the longer he and Regina stayed up at night sewing aprons, the more money he made the next day on the streets….Those three things – autonomy, complexity, and a connection between effort and reward – are, most people agree, the three qualities that work has to have if it is to be satisfying….Work that fulfills those three criteria is meaningful.”[xxxiii]

If Gladwell is right in his analysis of what makes work matter, it’s hard not to wonder where that leaves contract document reviewers. They are subject to a kind of supervision to which they are not accustomed. There is occasional complexity in the analysis that they are called upon to do, but very little in the act of coding itself. Their reward increases only with an increase in hours, and unlike other hourly workers, they are rarely eligible for overtime differentials.

Perhaps, though, it is simply not possible to reduce meaningful work to an algorithm. Perhaps, instead we should distill from Gladwell the insight that the meaning of hard work comes to us by way of our social environment, whether of the Islands, or the Chinese mainland, or old Eastern Europe. In other words, meaning is culturally derived. One of the essential functions of culture is to bestow meaning, to instruct us as to what matters, and why.

Here, Withers’ lament comes to mind. Among the things that distressed him was the loss of the connections that are so vital to working people, the connection with purpose, the connection with each other. So we should ask: regarding those vital connections, are there features of the current discovery landscape that might provide them?

The Case For Teams

The place of teams in electronic discovery is now well-established. Ralph Losey says teams are one of the pillars of the successful practice of discovery.[xxxiv] They’ve been called discovery’s “holy grail.”[xxxv] Since first being employed internally by Cisco Systems to deal with a flood of litigation following the stock market collapse of 2000-2001, multidisciplinary teams have become the solution of choice to some of the most vexing discovery problems. Fully developed, they’ve become the stuff of which “dreams” are made.[xxxvi] You find them in every setting: in general corporate settings and in-house legal departments, in law firms and consulting firms.

Teams are typically made up of representatives from three different fields: law, information technology, and business. They ensure that all the constituencies necessary to successful litigation are able to contribute, and that the complete set of communication skills necessary for discovery will be available. By calling on all members to do what they do best, teams have demonstrated the potential for maximizing efficiencies and cutting down on expense. Meanwhile, the cooperation and collaboration upon which teams depend serve as compelling models of the kind of institutional behavior that the federal rules require of all persons and all parties.

But as Losey has recognized in his work, the essential challenge in forming effective multi-disciplinary discovery teams is cultural. As he has said in his teaching, the cultures of business, law, and information technology are very different. “Not all in-house lawyers are cut out to work with computer techs. He or she may be a good lawyer, but computer-phobic…The same applies to IT personnel. Many dislike lawyers…and would rather work on code for hours than attend a meeting…..Management types drafted onto this project may have similar prejudices…They hear it has something to do with records management, lawyers and IT, and their eyes glaze over.”[xxxvii]

So, team leadership involves thinking culturally, inter-personally. Teams are where different people from these disparate backgrounds “work together on e-discovery challenges, speak a common language and interact as a team with mutual respect, positive leadership and goodwill.”[xxxviii] Even once the right team has been chosen, the work remains a challenge. Personal barriers are inevitable, it is said. “Gaps in communication are a persistent problem”[xxxix] that simply must be overcome if the team is to meet with success. Lawyers often have to function as emissaries on a diplomatic mission. Those who can do that “are worth their weight in gold.”[xl]

I believe that these same skills ought to be applied to extend the team concept and reshape the document review experience. In the first place, document review groups are themselves likely to be cross-disciplinary in their own right, comprising people from a wide variety of backgrounds and experiences. They constitute a potential resource for the entire group, a resource that I have never seen strategically tapped. Second, document reviewers are an essential part of the larger mechanism by which the federal values of cooperation and transparency are realized. How can those values be realized at the level of production if they are not practiced at the level of review? How can we expect the federal values to be embraced by reviewers if the reviewers themselves are not embraced as part of the larger discovery team?

Moreover, the team approach addresses head-on the three impediments to optimal review that we have identified here. Teams are a perfect framework for the efficient delivery of topic authority and effective collaboration towards which the University of Pittsburgh study points. Teams are a natural setting for the kind of support and encouragement highlighted in the Hawthorne experiments. Finally, teams are inherently a good source of the connectedness that has deprived document review workers of the sense that their work has meaning.

Of course, team leadership and management draw on a particular set of skills, the “soft skills that deliver hard results” as one consultant says.[xli] But these are skills for which lawyers have not traditionally been prepared or trained. In failing to do so, law schools may be increasingly alone among the professions.

According to the New York Times, the medical profession has been slow to realize that doctors can save lives but can sometimes be insufferable know-it-alls who bully nurses and do not listen to patients. Nor have medical schools trained them to behave better.[xlii] That is changing, apparently, spurred by a “growing catalog of studies that pin the blame for an appalling share of preventable deaths on  poor communication among doctors, patients and nurses that often results because some doctors, while technically competent, are socially inept.” One medical school cited in the Times article, in admitting students, weighs their willingness and ability to collaborate effectively, but also requires students to take team work classes. “Our school intends to graduate physicians who can communicate with patients and work in a team,” the dean was quoted as saying.[xliii] [xliv]

Business schools, too, are “tapping into their soft side,” the Wall Street Journal reports,[xlv] with more and more courses being offered in areas such as mediation, social intelligence and personal leadership.” The decisions to do so have been the reflection of the corporate experience that “job candidates with proficiency in softer skills ended up leading better, no matter their functional background.” As compared with hard skills like finance and accounting, soft skills –such as receiving hard news with grace and delivering authority with respect – mattered more and more as they ascended the corporate ladder.

I have not studied the matter closely, and I hope that I am wrong, but I suspect that what Professor Julie Macfarlane has found in Canadian law schools can be widely found south of the border as well. In Canada, she says, law schools neglect to teach lawyers about the affective dimension of lawyering….the meaning and impact of what happens …between the lawyer and client… (when) lawyers must have the skills to address the anger, frustration, despair or even indifference that legal interactions evoke.”[xlvi] That may well be explained by what she finds at the point of hiring where there is “little interest….in skills and qualities such as empathy, wise counsel, creativity and conflict resolution.”[xlvii] But it is also called into question by what Macfarlane found in a 2002 study among Canadian judges. When asked what abilities and skills they considered most important for a judge, the most consistent emphasis was on communications skills, such as listening and managing people, and personal attributes, such as empathy and patience.[xlviii]

Scholars have noted how electronic discovery is challenging some of law’s most venerable traditions. It has meant both new things to learn and new ways of learning them.[xlix] In this way, too, that seems to be true. Both “hard” technological skills and “soft” people skills – team skills – will be increasingly necessary for those who see themselves litigating at the highest levels.[l] It is work for which they are not yet being prepared.

Personal Reflections: Looking Out – And Up – From The Trenches

It has been said that the focus in the discovery space has been almost exclusively on making review technology smarter. “Little or no attention has been paid to making document reviewers smarter.”[li] This oversight seems to be getting worse, not better. In what is now a rapidly consolidating industry, technology is where the action is; it is both the sizzle and the steak. The verdict is in, some of our most authoritative writings tell us – the questions for the future to answer concern which technologies will improve human review the most, not whether technologies can improve it.[lii]

Still, if Mark Herrman, Vice President and Chief Counsel – Litigation at Aon is correct, the document review race (for market share) is being won by those entrants for whom review is something other – something smarter – than “sitting 50 lawyers at computer terminals for months on end and having them plow through materials until their minds are numbs and their eyes are bleary.” They’re genuinely concerned that document review supervision has often been left to those who know “little about quality control or keeping people motivated” in such an environment.[liii]

The more common perspectives on project management correctly emphasize things like planning and communication and learning, but manage to leave document reviewers out of the picture.[liv] Projects have a variety of stakeholders, we are told: those with a direct interest who need to be identified and understood, whose expectations need to be managed, who need sponsors or champions to keep everyone on the same page.[lv] Projects do indeed have many stakeholders….but few seem to believe that document reviewers are among them.[lvi] [lvii]

When we read stakeholder, we should think meaning. We should consider what Withers said about connections and what Gladwell said about the meaning of work. We should remember, too, that meaning in our work is not a zero sum commodity. There is plenty of it to go around.

For now, document reviewers remain largely uncounted, except as an unavoidable cost. This remains true – despite what we have learned about the importance of cross-disciplinary teamwork to e-discovery success, and despite what our own intuition tells us about the importance of team awareness to team work. It remains true despite what we know about the importance of the quality of the information product as it moves up the pipeline to its end-users and despite what we know about the impact it can have when people know that others are considering them. It remains true despite what we read – and, if we are honest with ourselves, despite what we instinctively know – about how important it is that all work has meaning for those who do it. It remains true despite the warnings that we have from thought leaders in the field about the loss of connection and purpose in basic discovery tasks.

In a landscape that is still shifting, the world of document reviewers remains largely outside the ambit of change. Their work product is, in the final analysis, subject to the same scrutiny, but they have little stake in the result. The incentives that exist for the other players in the marketplace have yet to reach reviewers.

Still, there is meaning to be found in document review. It does have some intrinsic satisfaction for me. It is cognitive work, a form of group puzzle solving. It is forensic work, an analytical exercise with courtroom implications.[lviii] It is group work, a source of constructive social engagement. It involves learning and new windows on the world. For some, of course, there is the hope of advancing in one of law’s few growth fields. For others, it is a place holder; it keeps them in the stadium, if not in what we think of as the real game. For still others, it’s a means of support while they’re chasing other dreams.[lix]

Yes. There is the downside. There are the lyrics of the document reviewer lament. There is monotony to the work; it can involve vast numbers of documents and does involve repetitive tasks performed over long periods of time. As with other lawyer’s work – indeed, as with all work – there are crunch times when time itself is the most precious commodity in serving the client, and when volume must mean the most to all involved. There is no need for document reviewers to be coddled any more than other professionals. It is often true that reviewers are brought in when the kitchen is already hot. Then, if you don’t want to sweat, you ought to stay out. Ultimately what should matter to everyone engaged in a project is the quality of the product, the information to be consumed by those higher up in the pipeline. No amount of esprit de corps will justify a product that falls short.


We can do better at enfranchising document reviewers. As we’ve explored here, we know how to do better. The spirit of cooperation and transparency that animates the practice of discovery law at its highest levels needs to permeate down to all of its levels in order for the information product to be as good as it can be from the start. Towards that end, drawn from my own experience both of what has been done well and what I wish had been done better, I offer the following to all those persons trying to decide how to get the best, and not just the most, from the document reviewers with whom they are working.

First, do the right things early. Communicate, and then communicate more.[lx] If there is time, spend time with the case memorandum more than once. It will pay dividends. Upfront, display respect; promise it and keep your promise. Talk briefly with your reviewers about federal discovery requirements and standards. They’re something everyone should know and will remind them that you think of them as attorneys. It’s at the beginning of review that people establish habits and patterns, so give them concrete initial guidance. That is, once they have read your memorandum, together as a group, have your people review model documents that you think will be representative of the larger corpus of documents. Revisit this, if necessary. Document review can be highly contextual so, if you are using supervisors to provide constant access to topic authority, make sure that they themselves have experience working with the same documents your reviewers are seeing.

Second, emphasize learning, especially shared learning. Tap into the potential of your group for solving problems together. Where indicated, turn impromptu individual questions into learning for the whole group. Encourage questions, especially in the beginning. So-called “dumb questions” especially are often the best way to examine unstated assumptions. Formalize learning; that is, schedule regular group sessions in which questions are asked and review experiences are shared. If questions are slow to come, require each reviewer to be ready with one question. Give a prize for the best among them.

Third, establish and reinforce a team ethic. At the outset, think about sharing brief bios of your reviewers. After all, there are many different backgrounds, and electronic discover is cross-disciplinary art. Have meetings. Keep them brief, but have them. As appropriate, keep your reviewers informed about the progress in the overall litigation such as deadlines and communication between the parties. Provide feedback from above; let them know what is being said about what is being found.

Fourth, listen carefully or greedily as Tolstoy says of one of his characters in War and Peace.[lxi] Despite the importance of listening to their clients, lawyers seem especially prone to the common malady of humankind, inattentiveness. We have been taught how important it is to observe and think and speak well, but as to listening – not so much, perhaps. It is another one of the soft skills about which law schools need to care and do more. Listen carefully, both to what is being said and what is not being said by your people. Sometimes reviewers have more worthwhile things to say about what they are finding than their software allows them to code. It’s worth special, one-on-one visits to do so. Ask the right questions (as we have been trained to do.) Listen closely to the answers. You may be surprised by what you learn.

Fifth, measure performance and provide feedback. Good reviewers like to know how productive they are. This may be all that is necessary, or appropriate, in the way of gaming. But also, be prepared to eat a few small marginal costs. From time to time, as the project size and circumstances permit, treat your reviewers to bagels for breakfast, or pizza for lunch, or dinner if you have asked them to start putting in extra hours. Doing so judiciously is like buying someone a sit-down cup of coffee just when it counts. Little things can mean a lot.[lxii] The goodwill you generate is worth many times the dollars and cents that such an effort will cost

Sixth, seek to innovate. Remember the lesson of the Hawthorne studies that people like to be part of experiments, and the TREC lesson that quality of experience and quality of product go hand in hand. Therefore, look for innovative ways of enhancing both the product and the experience. In this, technology can be an aid without being a crutch. For example, consider ways to “personalize” the electronic search. For example, in searches where there is real potential for “hot” documents, you might want to create a category of documents that is, by plan, beyond those built into the software. A special way of tagging or commentating should be designated from the outset and identified to the reviewers as unique to the project and personal with you (whether or not it has been used before.) It could be a dedicated email link for very limited use – once a day, perhaps – as a means of communicating especially heightened hunches about the importance of the document to the search. Search leaders should promise that searchers using the tag will receive feedback within a day. Wherever such promises are made, they should be kept.

The benefits of this kind of innovation should be apparent by now to readers of this essay. By putting it in place early, you set the tone for the entire search, that it is not entirely pre-packaged. You create opportunities for learning that reinforce a team ethic. You formalize the message that you are paying attention to what it is they are producing, that you intend to listen carefully to what they have to say as lawyers. You create a setting for high-value input and feedback. In other words, at the same time that you are sharpening the search, you are creating sources of meaning for the searchers.

There are, no doubt, other ways of improving the human component of search. Some of them will emerge from new technologies. Others will emanate from what we have long known about what it simply – and profoundly – means to be human. As to both, however, a few things are clear. First, the best ideas will almost certainly have a bearing on the quality of both the search product and the search experience. Second, smart players in the discovery market will aspire to a grasp of both, because success – both legally and financially – will almost certainly go to those who can deliver on the best ideas.


For Ken Withers, they were like canaries in a coal mine: his purgatorial vision of small armies of young lawyers strapped for years to their computer screens, and the reports reaching him of declining civility and esteem among attorneys. These things were warnings that there was more at stake in the digital deluge than missed deadlines, broken budgets, and stunted careers. The well-being of the profession and the integrity of the legal system were at risk. This was at the very time that a new and different ethic was being impressed by the courts upon discovery’s players. Cooperation and transparency were to be the by-words.

Ironically, those young law firm associates of Withers’ vision proved to be among litigation’s luckier ones. There are others doing the same work now, but without the same paycheck or prospects. They are our contract document reviewers. Now, for the first time, the circumstances of their labor is becoming a matter of public awareness, just as the stakes involved in such work are being raised. For in a statutory scheme where reasonableness is the standard, some are saying that the seemingly unavoidable imperfections of large document review are so unreasonable as to be actionable under traditional tort principles.

Perhaps the deepest irony of all, however, lies elsewhere – with the fact that contract reviewers are often the best holders of the information upon which litigation will depend,[lxiii] even though the reviewers are not thought of as having any real stake in the result. They are often not even thought of as real attorneys. The information the reviewers process and possess matters a great deal, but they do not. If our system of discovery is to progress, if the goals underlying the federal scheme are to be realized, this tension needs to be resolved.

Fortunately, the approach that offers real potential for doing so is already a feature on the electronic discovery landscape. It is the idea of discovery teams. Our studies, our experiments and our history all tell us that such teams, well led, offer the structures best suited to trust-based collaboration. They offer the advantages of a supportive and encouraging environment. Finally, they offer the kind of connections with purpose and persons that bestow meaning upon our work.

Challenges remain, of course. Such leadership is not generally the kind of thing taught in law school. Still, there are concrete possibilities for leaders to consider. They include formal shared learning and practiced listening. They include measured performance, true respect, kept promises, and openness to innovation. Until such ideas as these are tried, and found true, contract document review counsel will remain the untapped resource they are today. They will remain a wasting asset that our clients should not be asked to afford.

[i] Kenneth J. Withers, E-Discovery and the Combative Legal Culture; Finding a Way Out of Purgatory,; See also Ariana J. Tadler and Kenneth J. Withers, Toward a Less Hostile Discovery Process, Trial, March 2010.

[ii] Id.

[iv] J-M Manufacturing Co., Inc. v. McDermott Will & Henry, Los Angeles Superior Court, Case No. BC462832; Removed to U.S. District Court (C.D. Cal), Case No. CV11-6666.

[v] See, Posted August 1, 2011

[vi] Still, today’s review counsel might care to count their electronic blessings. The worst project I know of was entirely non-electronic. It took place fifty miles from the city, in the dead of summer heat, in an abandoned, unlit, un-cooled, cavernous underground garage. There, wearing masks to guard against asbestos particulates descending from rotting, seeping ceilings; in the dim light from small, dying, battery powered Coleman lanterns, hunched-over reviewers fingered through moldy boxes full of water-soaked mortgage documents from a then-defunct originator. The project lasted a month. The memory, I am told, will last a lifetime. (Courtesy of Harold Quadrino, Esq.)

[vii]Lawyers Settle…..for Temp Jobs”, The Wall Street Journal, June 15, 2011, B1.

[viii]Smarter Than You Think: Armies of Expensive Lawyers, Replaced by Cheaper Software,” The New York Times, March 5, 2011. Ralph Losey takes issue with the Times, saying that the paper has fallen for the Luddite fallacy, the assumption that new technologies will necessarily reduce employment.

[ix] When pondering the likelihood that contract attorneys might be unhappy both with the nature of the work and the threat of unemployment, I was reminded of the story about two older women complaining about the local restaurant. “The food is terrible!” the first one said. “I know,” said the second, ”and the portions are so small!”

[x] See Ralph Losey’s e-discovery team blog at where Losey characterizes the complaint as a whole-cloth production, and predicts that the case will end soon.

[xi] Removed to U.S. District Court (C.D. Cal), Case No. CV11-6666

[xii] See Daryl Shetterly, There is no Substitute for Boots on the Ground at Shetterly is a member of the electronic discovery group at LeClairRyan.

[xiii] See Zhen Yue and Daqing He, A Model for Understanding Collaborative Information Behavior in E-discovery, University of Pittsburgh at

[xiv] Id. at 4

[xv] Id.

[xvi] Id. at 5

[xvii] Id. at 7

[xviii] Id. at 6

[xix] Id.

[xx] For this discussion of the Hawthorne studies, I am indebted to consultant Dale Dauten’s helpful little book on working life: The Max Strategy: How a Businessman Got Stuck at an Airport and Learned to Make His Career Take Off, William Morrow and Company, Inc., 1996.

[xxi] Id. at 4,5

[xxii] Dauten, Dale, The Max Strategy: How a Businessman Got Stuck at an Airport and Learned to Make His Career Take Off, William Morrow and Company, Inc., 1996.

[xxiii] For an explanation of the Observer Effect in the physical sciences, see; Also see

[xxiv] See Graham Rollins’ article, Keyboard and Mouse or Game Controller? – A Debate Usually Reserved For Internet Forums, originally published in LeClairRyan’s blog, Rollins now practices e-discovery law with Morgan Lewis and Bockius in Washington, D.C

[xxv] Id.

[xxvi] Id.

[xxvii] “Tech-industry research firm Gartner estimates that by 2014, some 70% of large companies will use the techniques for at least one business application. Market researcher M2 Research estimates that revenue from gamification software, consulting and marketing will reach $938 million by 2014 from less than $100 million this year.” Wall Street Journal, October 11, 2011.

[xxviii] Malcolm Gladwell, Outliers: The Story of Success;  Little, Brown and Company, 2008.

[xxix] Id., Chapter Five, The Three Lessons of Joe Flom, at 116.

[xxx] Id., Chapter Eight, Rice Paddies and Math Tests, at 224.

[xxxi] Id., Epilogue, A Jamaican Story, at 270.

[xxxii] Id., Chapter Five, Lesson Number Three: The Garment Industry and Meaningful Work, at 139.

[xxxiii] Id. at 149.

[xxxiv] Losey has given teams a “home” on his electronic discovery blog, For him, interdisciplinary teams are one of the four pillars of successful e-discovery. The others are cooperation, metrics, and education. Education, he says, is the cornerstone of the rest. Losey is an adjunct professor of University of Florida School of Law. He offers extensive on-line training in the law of electronic discovery at This writer has had the privilege of studying with Professor Losey in that course.

[xxxv] See  wherein  Losey cites Ben Hawksworth of Ernst & Young saying that internal corporate team approach is the new ”holy grail” of e-discovery.

[xxxvi] See Jake Frazier, Esq. and Carolyn Southerland, Esq., E-Discovery ‘Dream Teams, Digital Discovery & e-evidence, July 2005.

[xxxvii] From Professor Losey’s private on-line training course, Electronic Discovery Law Training, at Module 2F.

[xxxviii] Id.

[xxxix] See Manfred Gabriel, Collaboration: A Requirement for Successful E-Discovery, The Metropolitan Corporate Counsel, 2008.

[xl] See Peter Caradonna, Bridging the Communications Gap in E-Discovery, Law Technology News; July 30,2010.

[xli]  See Peter Caradonna, Bridging the Communications Gap in E-Discovery, Law Technology News; July 30,2010

[xlii] New York Times, July 20, 2011.

[xliii] The value placed on patient skills has reached new and surprising highs. The University of Chicago recently announced that it is the recipient of a $42 million donation given to create an institute devoted to improving medical students’ handling of the doctor patient relationship. It is the gift of a wealthy Chicago couple who came under the care of Dr. Mark Siegler of the University Medical Center. Dr. Siegler, who will lead the institute, said it is an effort to put compassion and empathy “on the same pedestal as science and technology.”  New York Times, Thursday, September 22, 2011.

[xliv] Upon learning of the purpose of the Chicago gift, a skeptical cardiologist friend called it “another effort to teach the un-teachable.”A lawyer colleague of mine was equally skeptical of the value of such communications training…..but I don’t think he was really listening.

[xlv] See The Wall Street Journal, May 5, 2011, On the Lesson Plan: Feelings.

[xlvi] See Julie MacFarlane, The New Lawyer: How Settlement is Transforming the Practice of Law, UBC Press, 2008, 152.

[xlvii] Id., 14.

[xlviii] Id., 234.

[xlix] See Ralph Losey, The Law Firm Apprenticeship Tradition and Why Most Lawyers Are Still Untrained in e-Discovery,

[l] See Withers, Purgatory, 6 where Richard Braman notes that law schools need to teach cooperative practices.” Students need to be told that there’s a time for adversarialness…and there’s a time when you need to cooperate and collaborate with your adversary…”

[li] See Manfred Gabriel, Collaboration: A Requirement for Successful E-Discovery, The Metropolitan Corporate Counsel, 2008.

[lii] Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Maura R. Grossman & Gordon V. Cormack, XVII Rich. J.L. & Tech. 11 (2011)

[liii] Mark Herrman, Inside Straight: Why Vendors Win the Document Review Work,, August 25, 2011.

[liv] One e-Discovery company using the Internet identifies the stakeholders in communications planning as follows: “the IT guy from the corporate client, the in-house counsel, the outside counsel, the partner in charge and associates, the litigation support manager, or the service provider’s project and operation’s managers – basically anyone who’s involved with the project.” That seems to leave the reviewers themselves out. See Combating eDiscovery Project Risk with Effective Communications;  Daegis, The eDiscovery Company San Francisco, CA.

[lv] Kearney, David, 6 Project Management Practices to Apply to E-Discovery Cases, Law Technology News, 8-18-2011.

[lvi] There appears to be hope for the idea catching on. Law Technology News recently quoted William Hamilton as saying that “We need to change the culture of law firms…and the command environment…Hierarchy culture disenfranchises everybody in the operation, and doesn’t create a collaborative, collegial environment.” Hamilton, a partner at Quarles and Brady, teaches electronic discovery law at the University of Florida’s Levin College of Law.

[lvii] I note with interest that one of the corporate players in the document review agency business is apparently reorganizing in such a way as to encourage their contract attorneys to become shareholders. What this development says about current market valuations, and whether this will otherwise help to change the working fortunes of document reviewers, only time will tell.

[lviii] See A New View, Part 2 in the ABA Journal, October 2011, an upbeat take on the business of document review in which the featured reviewer is quoted as saying “I enjoy the work. I love the people I work with…It’s not dull and boring.”

[lix] I have worked with published authors, aspiring playwrights, professional bicycle racers, rock saxophonists, and circus gymnasts. One placement company with which I have worked recognizes this in its marketing. On its website, Lexolution acknowledges that some candidates are available “because they need flexibility in their lives to pursue other careers…other passions. We applaud these pursuits and are committed to helping you to fulfill those dreams.” See

[lx] Kudos to Jeffrey Baldwin, Esq., an associate with Simpson, Thatcher & Bartlett, New York, and others like him, for providing proof of the effectiveness of an open communications style of project leadership.

[lxi] Careful listening, for example, takes into account that what people say is not necessarily what they mean. There is a classic instance in Zubulake IV where one of the UBS employees gave deposition testimony that she kept a separate archive file on her computer with documents pertaining to the plaintiff. UBS later admitted that for more than a year it misunderstood the employee’s use of the word “archive” to mean back-up tapes. Only much later did it finally became clear that she meant active data. Zubulake v. UBS Warburg LLC, 229 F.R.D. 422 at FN50.

[lxii] See Dr. Louis Weeks, Little Things Mean a Lot, at the blog of Duke Divinity School,

[lxiii] See Manfred Gabriel, Collaboration: A Requirement for Successful E-Discovery, The Metropolitan Counsel, 2008.

10 Responses to Contract Coders: e-Discovery’s “Wasting Asset”?

  1. Kristina G. says:

    Thank you for this wonderful perspective. As I write this from my iPhone (no Internet access for contract attorneys), sitting at a printer cart (real desks are a luxury at this firm), after having been called a “fucking animal” by one firm employee angry that we are taking up space here, it’s clear that not only are contract attorneys not considered attorneys, but we are not even colleagues. Six months at the same monotonous work, we receive little praise, only warnings. Bagels? Pizza? Coffee? Laughable, as they don’t even want us using the same refrigerator or microwave.

    All attorneys managing e-discovery projects should read this blog and all the evidence it presents. It not only highlights that proper treatment of contractors is the “right” thing to do, but also yields immense benefit to the quality of the review.

    • Ralph Losey says:

      Thank you for sharing this. The incident of abuse you report is shocking, and one I can’t get out of my head. What does this tell us about legal culture in the U.S. today? I am hoping this is a very rare and isolated experience. But perhaps I’m naive. If this is common, it could explain many of the colossal “mistakes” we read about in large scale review projects where obviously privileged and embarrassing emails are “accidentally” produced. Who could blame a slave for spitting in his master’s food?

      • Larry Chapin says:

        Especially in small organizations and groups, unhappy “leaders” tend to make others unhappy. Leader selection: its one more thing that lawyers need to learn how to do better. Thank you for sharing, Kristina. I hope that your people get smarter and things get better.

  2. Andrew says:

    Chapin’s article is a bit long winded and the payoff is questionable. While all the suggestions he makes would probably help some, the reality is that all of them will wear thin on the document reviewer population as long as the industry relies on temporary employment. If I could be fired the next day and in all likelihood will be fired in the next few weeks why should I get invested in the process? I know about ethical obligations to the clients, future temporary employment opportunities with the firm, a natural desire to perform effectively, to be of value, etc., but none of that is going to pay the bills, like the student loans that may be the is the size of a middle income house in middle America.

    On the flip-side if as the employer I make these people more valuable, I invest more of my emotion in them, and then I have to turn around and fire them in a week or month or two months and repeat this process three, four, five, or fifteen times a year, the emotional trauma on the employer becomes devastating. After all why did the military for many years allow its men to refer to the enemy as “redskins,” “Gooks”, “Nips,” “Huns,” and “Krauts?” To make them less human, less of a full person and easier to kill. By not investing in the temporary employee anything but money it is a lot easier to let them go.

    In short the legal profession generally and temporary legal employment model most especially has inherent contradictions to which Chapin’s suggested solutions can only provide window dressing.

  3. Larry Chapin says:

    Thanks for your note Andrew. I remain hopeful that genuine mutual respect will come to be seen as more than window dressing.

  4. […] Contract Coders: e-Discovery’s “Wasting Asset”? (e-Discovery Team) […]

  5. […] Webber’s findings are the result of something that is not a secret to anyone who has ever been involved in a large search project, that all reviewers are not created equal. Some are far better than others. There are many good psychological, intelligence, and project management and methodology reasons for this, especially the management and methodology issues. See eg the must read guest blog by contract review attorney Larry Chapin, Contract Coders: e-Discovery’s “Wasting Asset”? […]

  6. […] Larry Chapin is an attorney with 30+ years experience, including corporate Wall Street law, who now works as a contract review lawyer in New York City. Larry has taught at the New School for Social Research in NYC and currently serves on the Board of Directors for an asset management company in Stockholm Sweden. Larry is the first graduate of our e-Discovery Team Training program. He contributed a must-read blog here earlier this year entitled Contract Coders: e-Discovery’s “Wasting Asset”? […]

  7. MRR article says:

    MRR article…

    […]Contract Coders: e-Discovery’s “Wasting Asset”? « e-Discovery Team ®[…]…

  8. […] Webber’s findings are the result of something that is not a secret to anyone who has ever been involved in a large search project, that all reviewers are not created equal. Some are far better than others. There are many good psychological, intelligence, and project management and methodology reasons for this, especially the management and methodology issues. See, e.g., the must read guest blog by contract review attorney Larry Chapin, Contract Coders: e-Discovery’s “Wasting Asset”? […]

Leave a Reply