OpenAI’s Best Practices For Using GPT Software

June 16, 2023

The OpenAI’s new website GPT Best Practices provides six strategies and tactics to maximize the effectiveness of Generative Pre-trained Transformers (GPTs) like ChatGPT-4. The information provided is very detailed with many technical suggestions. The key message of OpenAI’s best practices guide is that while GPTs are always capable of generating intricate, human-like text, user input and guidance are vital to attain the best outcomes. The better prompts you make, the better answers GPT will provide. Success in large part depends on you.

Success of Your AI Output Depends on You

This blog will summarize the six main strategies outlined by OpenAI and include my original Midjourney photoshopped images for right-brain impact. ChatGPT-4, web-browsing pro version, helped me to write this, so did my WordPress software. It is all one big hybrid, multimodal, effort.

Best Practices to Add an AI to Your Team

Here is a synopsis of the six fundamental strategies provided by OpenAI for obtaining optimal results from GPTs:

  1. Write clear instructions: The AI cannot infer user intent, hence the need for clarity. If users require shorter responses, they should ask for brevity. For more technical outputs, they should request expert-level writing. If a particular format is desired, the user should demonstrate that format. Essentially, the clearer the instructions, the more accurate the GPT’s output.
  2. Provide reference text: GPTs can sometimes fabricate answers, especially when dealing with complex or unfamiliar topics. Providing reference texts can help guide the GPT to produce more accurate and reliable responses.
  3. Split complex tasks into simpler subtasks: GPTs are better at handling simpler tasks, which have lower error rates. A complex task can be broken down into a series of simpler tasks, with the output of earlier tasks used to construct the inputs for subsequent ones.
  4. Give GPTs time to “think”: GPTs can make errors when required to provide instant responses. Asking for a chain of reasoning before an answer can help GPTs reason their way to more accurate conclusions.
  5. Use external tools: To compensate for GPTs’ limitations, the outputs of other tools can be utilized. If a task can be done more reliably or efficiently by another tool, it should be offloaded to that tool.
  6. Test changes systematically: To improve GPT performance, any changes made to a prompt should be tested systematically. A modification might improve performance in some instances but worsen it in others, so it’s crucial to test these changes across a range of examples.
Give Your GPT External Tools

These strategies can be implemented through specific tactics, each tailored to the particular strategy:

  1. For writing clear instructions, tactics include providing important details in your query, asking the model to adopt a persona, using delimiters to distinguish different parts of the input, specifying the steps required to complete a task, providing examples (known as “few-shot” prompting), and specifying the desired output length.
  2. When providing reference text, include sufficient additional context or source material for the GPT to understand the reference.
  3. To split complex tasks into simpler subtasks, the approach is to break down the task into a workflow of smaller, more manageable tasks.
  4. For giving GPTs time to “think”, the article suggests asking the AI for a chain of reasoning before providing an answer, allowing it to work out a more accurate response.
  5. In using external tools, the idea is to use the outputs of other tools to complement the abilities of the GPT. For instance, a text retrieval system or a code execution engine can be used to augment the GPT’s abilities.
  6. For testing changes systematically, the suggestion is to develop a comprehensive test suite, also known as an “eval”, to measure the impact of modifications made to prompts.
Give your AI time to think things through

The bottom line of the information provided here is that although GPTs are capable tools, the quality of their output depends on: the clarity of the instructions they receive; sufficient context to any reference text provided; the ability to decompose complex tasks into simpler ones; having the time to “think”; the use of external tools when necessary; and, the systematic testing of changes. If you learn to use these strategies and tactics, you can significantly enhance the effectiveness of your interactions with all of OpenAI’s GPT models.

Make Sure Your Instructions Are Clear


Implementing the strategies and tactics suggested by OpenAI can, indeed, help users get the most out of ChatGPT. Seems to me OpenAI should have provided these instructions upon launch. Maybe then so many newbies, and really we are all newbies at this new software, would not have complained so much about the accuracy, relevance and quality of its outputs. Basically OpenAI is invoking the old saying, “Garbage In, Garbage Out.”


I hope we see many more instructions like this from Open AI in the coming months. In the meantime, there are hundreds of software hackers who have attained some level of prompt engineering skills and are already sharing their prompting tips. I have even ventured into this territory by sharing some of my more interesting prompt experiments, such as: Prompting a GPT-4 “Hive Mind” to Dialogue with Itself on the Future of Law, AI and Adjudications; and ChatGTP-4 Prompted To Talk With Itself About “The Singularity”.

“The Singularity” May Arrive Someday, But We Are Nowhere Near That. AI Still Needs Our Help For It To Better Help Us.

As OpenAI points out, the Six Strategies and Tactics given here can be used in a variety of ways. It all depends, as lawyers love to say, on the particular use case. It also depends on the capabilities of the particular GPT model you use. There are already many variations, with 3.5 being the first and weakest.

As always I encourage everyone to go hands-on with this. Hack around with this new software yourself. If you are a lawyer or other professional with client confidential information, be extremely careful in its use for all client work. Make sure you engage privacy settings and do not expose client confidential information. Legal ethics and common sense also require that you verify very carefully all of the output of GPT, especially in these early days. Your trust level should be low and skeptical level high.

Emerge and Be “Hands” On, But Remain Vigilant

So go ahead, experiment and adapt these six strategies to suit your needs and requirements. Just remember it may seem like you are dealing with a great savant here, but never forget ChatGPT is an Idiot-Savant. Just a child really, but with a big vocabulary. It is prone to forgetfulness, memory limitations, hallucinations, outright errors, ethics jailbreaks, and many, many other humanlike foibles. It may seem like a genius in a box, but it is not. It is more like a bottom of the class law student that somehow sounds smarter than he is, especially to non-experts. Still, he did somehow get into law school and might be able to pass your state’s Bar Exam.

See my blog for many articles about ChatGPT’s many unique foibles. Finally, note my e-Discovery Team blog now has a handy new, easy to remember HTML address – EDISCOVERY.TEAM. Yup, team is a domain name and you don’t have to remember to put a hyphen between the e and d. Yes, we humans are prone to forgetfulness too.

Best Efforts Are Diverse Team Efforts, Including an AI Team Member

Ralph Losey 2023 Copyright — ALL RIGHTS RESERVED

Second Grader Version by Chat GPT of the Blog Earlier Today “Four Generations of Predictive Coding Software, from 2009 to Today”

February 11, 2023

Dall-E 2 created image of great grandfather and 4th generation baby

The summary written here by the AI is slightly wrong in some respects, but hilarious. In some ways the rewrite is very insightful, maybe better. This second-grader version is certainly much more fun to read than the original overly-adult Tar-Class it summarizes. (Note: I have a grandchild in second grade.) I love it and hope you do too. It might even cause you to read my grown-up version. By the way, the images here are all by OpenAI’s Dall-E 2.

A long time ago, in 2009, a company called Recommind made a computer program called Axcelerate to help lawyers do their job. The program used a special way of learning called “machine learning” to help sort papers into two piles, important and not important, based on just looking at some of the papers. Lawyers used this program to help them make decisions about what papers were important in a big case. They worked with the computer program to teach it what was important, then the computer program did the rest. Later, another company bought Recommind and the Axcelerate program isn’t used much anymore.

A lawyer named Losey didn’t like how the program worked because it kept some parts secret from him, but he wasn’t sure how to argue against it. He tried talking to the people who made the program, but they didn’t understand how lawyers work. So, he just made the program work the way he wanted it to. The people who made the program didn’t understand that lawyers are important and have a hard job to do, so they were making things difficult for the lawyers. But some other people, like Gordon Cormack and Maura Grossman, were nicer and helped the lawyers understand how to use the program.

Four Generations of Predictive Coding

Predictive coding is a type of software used by lawyers to help them find important papers. The first version of this software had two steps: teaching the software how to find important papers and then actually looking at the papers the software found. But later, people started to use a new version where you keep teaching the software as you look at the papers. This is called “Continuous Training” or “Continuous Active Learning.”

A man named John Tredennick started using this new way of using the software and he called it “Predictive Coding 2.0.” He said it was better to keep teaching the software instead of just doing it in two steps because then you can keep finding more and more important papers. Some other people also started using this way of doing things, but they still used a secret set of papers to help teach the software. But Losey did some experiments and found that using random papers was not very helpful. Now, Losey uses a different way to teach the software, which is called “hybrid multimodal” predictive coding 4.0.

Predictive Coding 3.0 is a way to search for information that is different from older ways. Some people used to use something called secret control sets, but now they don’t use them anymore. This is because control sets don’t work well in legal search. Legal search is when lawyers look for information to help them with their work. The control sets would give false information because what is considered important or not important can change over time.

Scientists made the old versions of predictive coding, but they didn’t understand how lawyers work or how they search for information. They only knew about statistics and didn’t understand the way relevance can change. Nowadays, things are better and many companies make better predictive coding software, but some still make the old version.

At the beginning of a search, it can be hard to know what is important and what isn’t. As the search goes on, the understanding of what is important can change. The control sets would be based on what people thought was important at the start, but by the end, that might not be the case. This is why the control sets don’t work well in legal search.

Some experts in information and technology, like William Webber and Douglas Oard, learned about these problems and started to help make better ways of searching. They learned from people like Jason Baron who is an expert in legal search. They believe that only at the end of a project can we really know what is important and have a true measure of what we found.

Control sets are a way some lawyers use to help them find important papers, but it’s not a good way. It can cause their work to not work right and make them waste time. Lawyers may not like it because it’s too complicated and it seems like fake magic.

Some lawyers don’t want to spend a lot of time going through papers that aren’t important, so they don’t want to use control sets. They would rather do other things with their time.

People who make the software want to stop using control sets because it causes problems and doesn’t work well. They made a new way called Predictive Coding 4.0 to help make it easier and work better.

Using control sets doesn’t work well because the papers that are important are usually rare, so it’s hard to find them. And if they did find them, they would have to look at a lot of papers that aren’t important, which would take a lot of time.

In the end, control sets can cause more problems than they solve, so it’s best to not use them and try Predictive Coding 4.0 instead.

Predictive Coding 4.0 is a way to help computers understand and sort information. It’s still the best way to do it in 2023. This method is like putting two steps into one and it doesn’t keep secrets. Now, there are many ways to do it. But Losey only knows one way that he made and changed in 2016. He doesn’t own the idea, but two people named Gordon Cormack and Maura Grossman do. They also teach this stuff and they’re married now. They made software that uses this method but they don’t sell it.

Gordon and Maura wrote about problems with other ways of doing this and said their way is better. They said the old ways were too hard to use and the computer wasn’t doing a good job. But their way is better and easier. They also said that the old ways didn’t work well when you have a lot of information and only a little bit is important.

Their way doesn’t use secret control sets like the old ways. They wrote about this in a special paper called a patent. They said they made their way better by making it easier to use and not having to use secret sets. They also made their own software that does this, but it’s different than Losey’s.


Predictive Coding is a way to help people find important information in a big pile of papers or documents. There have been many different ways to do this over the years.

The first type of software was really hard to use and needed a lot of help from experts. That’s why we don’t use it anymore.

The second type was a little better because it helped you learn as you went along, but it still had some problems.

The third type was a big change because it got rid of the secret rules and made it easier for the experts to do their job. They didn’t have to spend as much time on it and the information they found was more accurate.

The fourth and latest type is called Intelligently Space Training. It’s like a smart way to find the important information, and it’s what we’ll be learning about in this class. It’s a lot easier to use and helps the experts find the information they need even better.


The way we learn about finding important papers is called Predictive Coding 4.0. We don’t use secret ways to pick papers anymore. Instead, we use random selection to help us pick the right papers and make sure we find all the important ones. We do this with the help of contract lawyers and without much help from SMEs. After we’ve gone through all the papers, we check again to make sure we found everything important.

In Predictive Coding 4.0, we start with a big or small set of papers and begin our training right away. There are no rules for how many papers we start with or what kind they are. We will learn more about this in our TAR Training class.

Four Generations of Predictive Coding Software, the Background and History of Predictive Coding

February 11, 2023

To understand the current state of the art of any field, you need to understand what came before. This is especially true with legal technology. This blog post also mirrors an updated version of the first class of my 2017 TAR Training Course. I am in the process of refreshing all classes. The other seventeen classes in the training program are shorter and easier. So do not let this first class discourage you from trying the rest of the course, besides it is free.

The First Generation of Predictive Coding

The first generation of Predictive Coding software for lawyers was called Axcelerate by Recommind. It was released in 2009 as the first commercial program to use active machine learning for legal search. Leading End-to-End eDiscovery Platform Combines Unique Predictive Coding Technology with Random Sampling to Revolutionize Document Review (2009 Press Release). From the earliest versions, the software was able to predict the relevance or irrelevance of a large collection of documents by manually reviewing only a portion of them. It sorted all documents into binary categories with weighted values. At the time this was a revolutionary for lawyers. Below is a diagram from 2009 explaining what it did.

In this version 1.0 of predictive coding methods lawyers began the review with a Subject Matter Expert (SME) who reviewed a random selection of documents, including a secret control set. The process was divided into two stages: training the software and actual review of documents predicted to be relevant. This is the software and initial process we followed in the landmark case, Da Silva Moore, in 2011. On April 26, 2011, Recommind was granted a patent for predictive coding: Patent No. 7,933,859, entitled Full-Text Systems and methods for predictive coding. The search algorithms in the patent used Probabilistic Latent Semantic Analysis, an already well-established statistical analysis technique for data analysis. (Recommind obtained two more patents with the same name in 2013: Patent No. 8,489,538 on July 16, 2013; and Patent No. 8,554,716 on October 8, 2013.) In 2016, OpenText corporation acquired Recommind and, I recently discovered, is now using a much revised and improved version of Axcelerate Software.

As the title of all of these patents indicate, the methods of use of the text analytics technology in the software were key to the patent claims. As is typical for patents, many different method variables were described to try to obtain as wide a protection as possible. The core method was shown in Figure Four of the 2011 patent.


This essentially describes the method that I now refer to as Predictive Coding Version 1.0. It is the work flow I had in mind when I first designed procedures for the Da Silva Moore case. In spite of the Recommind patent, this basic method was followed by all vendors who added predictive coding features to their software in 2011, 2012 and thereafter. It is still going on today. Many of the other vendors also received patents for their predictive coding technology and methods, or applications are pending. See eg. Equivio, patent applied for on June 15, 2011 and granted on September 10, 2013, patent number 8,533,194; Kroll Ontrack, application 20120278266; Cormack December 31, 2013, 8,620,842, Cormack; April 29, 2014, 8,713,023, Grossman and Cormack, September 16, 2014, 8,838,606,

Losey in 2011 when he first started arguing against the methods of version 1.0

I also remember getting into many arguments with the technical experts from several companies back in 2011, including especially Kroll Ontrack. That was when the predictive coding 1.0 methods hardwired into their software, as well as Recommind’s, were first explained to me. I objected right away to the secret control set. I wanted total control of my search and review projects. I resented the secrecy aspects. There were enough black boxes in the new technology already. I was also very dubious of the statistical projections. In my arguments with them, sometimes heated, I found that they had little real grasp of how legal search was actually conducted or the practice of law. My arguments were of no avail. And to be honest, I had a lot to learn. I was not confident of my positions, nor knowledgeable enough of statistics. All I knew for sure is that I resented their trying to control my well-established, pre-predictive coding search methods. Who were they to dictate how I should practice law, what procedures I should follow? These scientists did not understand legal relevance, nor how it changes over time during the course of any large-scale review. They did not understand the whole notion of the probative value of evidence and the function of e-discovery as trial preparation. They did not understand weighted relevance, and the 7+/2 rule of judge and jury persuasion. I gave up trying, and just had the software modified to suit my needs.

Part of the reason I gave up trying back in 2011 is that I ran into a familiar prejudice from this expert group. It was a prejudice against lawyers common to most academics and engineers. As a high-tech lawyer since 1980 I have faced this prejudice from non-lawyer techies my whole career. Many assume we were all just a bunch of lawyer weasels, not to be trusted, and with little or no knowledge of technology and search. They have no idea at all about legal ethics or professionalism, nor of our experience with the search for evidence. They fail to understand the central role of lawyers in e-discovery, and how our whole legal system, not just discovery, is based on the honesty and integrity of lawyers. We need good software from them, not methods to use the software, but they knew better. It was frustrating, believe me. So I gave up on the control set arguments and moved on. Gordon Cormack and Maura Grossman were not part of this stubborn group. They never included control sets in their methods.

In the arrogance of the first designers of predictive coding, an arrogance born of advanced degrees in entirely different fields, these information scientists and engineers presumed they knew enough to tell all lawyers how to use predictive coding software. They were blind to their own ignorance. The serious flaws inherent in Predictive Coding Version 1.0 are the result. The patents of Gordon Cormack and Maura Grossman were in part a reaction to these flaws and mistakes, as is shown in the body of their patent.

Still, even with poor methods, Predictive Coding software in 2009 disrupted electronic discovery by providing a tool that learned from a small set of coded documents how to predict the relevance of all of them. The new feature ranked all documents in the collection according to predicted probable relevance. It sorted all of the documents into binary categories with weighted values by using complex multidimensional mathematics and statistical analysis. We will not go into the black box math in this course, only how to use this powerful new capabilities. But see: Jason R. Baron and Jesse B. Freeman, Quick Peek at the Math Behind the Black Box of Predictive Coding (2013).

The methods for use of predictive coding software have always been built into the software. The first version 1.0 software required a user to begin the review with a SME, usually a senior-level lawyer in charge of the case, to review a random selection of several thousand documents. The random documents they reviewed included a secret set of documents not identified to the SME, or anyone else, called a control set. The secret control set supposedly allowed you to objectively monitor your progress in Recall and Precision of the relevant documents from the total set. It also supposedly prevented lawyers from gaming the system. As you will see, many of us think that the use of control sets in Predictive Coding for e-discovery was a big mistake. The control was an illusion and made the whole process unnecessarily complicated and time-consuming.

Eventually I came to understand that the experts who promoted a two-step process, train with what they called a seed set and a control set and then review with use of a secret control sets to calculate recall, did not understand a lot about document review in legal cases, For one thing, they did not understand the low prevalence of relevant documents in the data typically searched in legal cases. They also did not understand the impact of concept drift, where the SME concept of relevance will develop and improve over the course of a review. They also did not grasp the need for transparency in legal proceedings. There were enough black boxes in the new technology already.

My arguments with the software vendors were of no avail at that time. One vendor, Kroll Ontrack, was forced by me, as a major customer ,to modify a version of their software so that it could be used as an ongoing one-step process, without control sets, at least for my projects. Eventually, they saw it worked and gave me access to playgrounds they set up for me to experiment with new methods using the Enron database. I am grateful to them, and my law firm at the time, Jackson Lewis. I wrote and spoke publicly about this continuously. Other experts looked at the peculiarities of legal search and agreed with my analysis and attack of secret control sets and a two-step process. Chief among was Professor Gordon Cormack, who was well taught on the law by working very closely on review projects with attorney Maura Grossman. He also provided her with his own predictive coding software, which has never been offered for sale, but has been patented. The two of them, Grossman and Cormack, were, and still are, the dominant serious scholars and also tireless promoters of active machine learning for document review.

Second Generation Predictive Coding, Version 2.0

The original versions of most software available to attorneys, including the initial market leader Recommind, had two distinct stages: training the software to predict relevance, using control and seed sets (initial training set), and then actually reviewing the documents that were predicted to be relevant. We used it at first in the landmark Da Silva Moore case with Judge Peck, but later quickly abandoned this method in later cases in favor of a single ongoing process that combined training and review. That is the dividing line for version 2.0 of predictive coding.

In two of my ongoing ENRON experiments using Kroll Ontrack software in 2012, I did not follow this two-step procedure. I just kept on training until I could not find any more relevant documents. A Modest Contribution to the Science of Search: Report and Analysis of Inconsistent Classifications in Two Predictive Coding Reviews of 699,082 Enron Documents. (Part One); Comparative Efficacy of Two Predictive Coding Reviews of 699,082 Enron Documents(Part Two); Predictive Coding Narrative: Searching for Relevance in the Ashes of Enron (in PDF form and the blog introducing this 82-page narrative, with second blog regarding an update); Borg Challenge: Report of my experimental review of 699,082 Enron documents using a semi-automated monomodal methodology (a five-part written and video series comparing two different kinds of predictive coding search methods).

In my 2012 published Enron experiments, I was using what was later promoted by others as Predictive Coding 2.0. In this 2.0 version you combine training and review. The training is continuous. The first round of document training might be called the seed set, if you wish, but it is nothing particularly special. All rounds of training are important and the training should continue as the review proceeds, unless there are some logistical reasons not to. After all, training and review are both part of the same review software, or should be. It just makes good common sense to do that, if your software will allow it. If you review a document, then you might as well at least have the option to include it in the training. There is no logical reason for a cut-off point in the review process where training stops. I really just came up with that notion in Da Silva because it was built into the Recommind software.

In predictive coding 2.0 you do Continuous Training, aka Continuous Active Learning (depends on the perspective, human or machine) or CAL for short. It just makes much more sense to keep training as long as you can, if your software allows you to do that. In addition to Cormack and Grossman, one vendor who quickly developed predictive coding software after Recommind, Catalyst, also did not use control sets. It was then owned by John Tredennick, a one-time trial attorney, tech expert, turned e-discovery entrepreneur. He often wrote then, and continues to do so now, about predictive coding methods. He was one of the first to follow Cormack and Grossman’s many writings on the subject and their description of “Continuous Active Learning” or CAL. The Grossman and Cormack team were at the time, and still are, the leading writers on predictive coding from a serious, scientific perspective, and a popular view too. Unlike me, they still travel the country to promote their methods of use (they still don’t offer Cormack’s software for sale or use).

John Trednnick quickly picked up on the Grossman and Cormack terms, specifically CAL, and began promotion his software as Predictive Coding 2.0. He combined, as I had been doing since 2012, the two-stages of review into one. It was no longer train, then review, but instead, the training continued throughout the review project. This continuous training improvement was popularized by Maura Grossman and Gordon Cormack and they first called that method continuous active learning. Under the CAL method, which again started to be built into the software by various vendors, the training continued continuously throughout the document review. There was not one stage to train, then another to review the predicted relevant document. The training and review continued together seamlessly. Here is a diagram that John Trednnick created to explain the then new process.

The main problem with version 2.0 of some vendors’ predictive coding software, not including Trednnick’s or Comack’s, was that the use of a random secret control set continued. Further, many promoted the use of randomly selected documents as an important part of the CAL process. Most vendors who moved from version one to two, and adopted some kind of continuous learning, instead of a two step process, still used random selection of documents for an initial control set and for part of the ongoing training. I did many experiments to test out the effectiveness of random selection for training (I never believed in control sets), and found some value to random as part of CAL, but only limited. It became just a small part of the multimodal tool set, the “hybrid multimodal” predictive coding 4.0 method now taught in my Tar course.

Third Generation: Predictive Coding 3.0 and the Abandonment of Secret Control Sets

Although I never used or believed in secret control sets, and neither did Cormack, Grossman nor Trednnick, most of the other vendors and writers on the field held onto the use of control sets for many years. Some still use them. The elimination of the use of control sets in CAL is my bright line between Predictive Coding versions two and three. Version Three software does not use control sets.

Although the use of a control set is basic to all scientific research and statistical analysis, it does not work in legal search. Control sets fail in legal search for multiple reasons, including the fact, which seems to elude most black and white minded engineering types, that the SME’s understanding of relevance evolves over time. The SMEs never know the full truth of document responsiveness at the beginning of a project as the two-step control set process assumes. Control set coding decisions are made before the SME is familiar with the collection and the case may be unreliable. Control sets can also lead to overtraining and overfitting of the document types, resulting in poor recall and precision.

Control sets are a good idea in general, and the basis of most scientific research, but simply do not work in legal search. It was built into the version 1.0 and 2.0 software by engineers and scientists who had little understanding of legal search. The secret control set does not work in real-world legal review projects. In fact, it provides statistical mis-information as to recall. Thta is primarily because in the real world of legal practice relevance is a continually evolving concept. It is almost never the same at the beginning of a project, when the control set is created, as at the end. The engineers who designed versions 1.0 and 2.0 simply did not understand that. They were not lawyers and did not appreciate the flexibility of the relevance. They did not know about concept drift. They did not understand the inherent vagaries and changing nature of the search target in a large document review project. They also did not understand how human SMEs were, how they often disagree with themselves on the classification of the same document even without concept drift. As mentioned, they were also blinded by their own arrogance, tinged with antipathy against lawyers.

They did understand statistics. I am not saying their math was wrong. But they did not understand evidence, did not understand relevance, did not understand relevance drift (or, as I prefer to call it, relevance evolution), and did not understand efficient legal practice. Many I have talked to did not have any real understanding of how lawyers worked at all, much less document review. Most were scientists or statisticians. They meant well, but they did harm nonetheless. These scientists did not have any legal training. If they were any experienced lawyers on the version 1.0 and 2.0 software development teams, they were not heard, or had never really practiced law. (As a customer, I know I was brushed off.) Things have gotten much better in this regard since 2011, but still, many vendors have not gotten the message. They still manufacture version 1.0 and 2.0 type predictive coding software.

The theory behind their use of secret control proponents is that the initial relevance coding of these documents is correct, immutable and complete; that it should be used to objectively judge the rest of the coding in the project. In practice many documents determined to be relevant or irrelevant at the beginning of a project may be considered the reverse by the end. The target shifts. The understanding of relevance evolves. That is not because of a bad luck or a weak SME (a subject discussed later in the Course), but because of the natural progression of the understanding of the probative value of various types of documents over the course of a review.

The ground truth at the beginning of a search project is quick sand. The understanding of relevance almost always evolves as the search progresses. The main problem with the use of the control set in legal search is that the SMEs never know the full truth of document responsiveness at the beginning of a project. This is something that evolves over time in all but the simplest projects. The understanding of relevance changes over time; it changes as particular documents are examined. The control set fails and creates false results because “the human-selected ground truth set and used as a benchmark for further statistical measurements” is never correct, especially at the beginning of a large review project. Only at the end of a project are we in a position to determine a “ground truth” and “benchmark” for statistical measurements.

This problem was recognized not only by Professor Cormack, but other experts who became involved in legal search. One of the most important for me was information retrieval expert, William Webber, PhD. William has been kind enough to help me through technical issues involving sampling many times. William gained experience with legal search by working with his professor, Douglas Oard, PhD. Professor Oard in turn had learned about the unique issues in legal search through working with an attorney with significant expertise in legal search, Jason R. Baron, who also taught me a great deal about the subject. Anyway, here is how Dr. Webber puts it in his blog Confidence intervals on recall and eRecall:

“Using the control set for the final estimate is also open to the objection that the control set coding decisions, having been made before the subject-matter expert (SME) was familiar with the collection and the case, may be unreliable.”

William Webber

Jeremy Pickens, Ph.D., who was Catalyst’s in-house information scientist, agrees with this assessment of control sets. See Pickens, An Exploratory Analysis of Control Sets for Measuring E-Discovery ProgressDESI VI 2015, where he reports on an his investigation of the effectiveness of control sets to measure recall and precision. Jeremy used the Grossman and Cormack TAR Evaluation Toolkit for his data and gold standards. Here is his conclusion:

A popular approach in measuring e-discovery progress involves the creation of a control set, holding out randomly selected documents from training and using the quality of the classification on that set as an indication of progress on or quality of the whole. In this paper we do an exploratory data analysis of this approach and visually examine the strength of this correlation. We found that the maximum-F1 control set approach does not necessarily always correlate well with overall task progress, calling into question the use of such approaches. Larger control sets performed better, but the human judgment effort to create these sets have a significant impact on the total cost of the process as a whole.

Jeremy Pickens

In legal search the target is almost always moving and small. Also, the data itself can often change as new documents are added to the collection. In other areas of information retrieval, the target is solid granite, simple Newtonian, and big, or at least bigger than just a few percent. In other words, the prevalence of targets is higher. Outside of legal search it may make sense to talk of an immutable ground truth. But with legal search it does not. In legal search the ground truth of relevance is discovered. It emerges as part of the process, often including surprise court rulings and amended causes of action. It is a moving target. With legal search the truth is rare, the truth is relative.

The control set based procedures of versions one and two of predictive coding software were over-complicated and inherently defective. They were based on an illusion of certainty, an illusion of a ground truth benchmark magically found at the beginning of a project before document review even began. There were supposedly SME wizards capable of such prodigious feats. I have been an SME in many, many topics of legal relevance since I started practicing law in 1980. I can assure you that SMEs are human, all too human. There is no magic wizard behind the curtain.

GPT generated image

Moreover, the understanding of any good SME naturally evolves over time as previously unknown, unseen documents are unearthed and analyzed. Legal understanding is not static. The theory of a case is not static. Experienced trial lawyers know this. The case you start out with is never the one you end up with. You never really know if Schrodinger’s cat is alive or dead. You get used to that after a while. Certainty comes from the final rulings of the last court of appeals.

GPT generated image

The use of magical control sets doomed many a predictive coding project to failure. Project team leaders thought they had high recall, because the secret control set said they did, yet they still missed key documents. They still had poor recall and poor precision, or at least far less than their control set analysis led them to believe. See: Webber, The bias of sequential testing in predictive coding, June 25, 2013, (“a control sample used to guide the producing party’s process cannot also be used to provide a statistically valid estimate of that process’s result.”) I still hear stores from reviewers where they find precision of less than 50% using Predictive Coding 1.0 and 2.0 methods, sometimes far less. Our goal is to use predictive coding 4.0 methods to increase precision to the 80% or higher level. This allows for the reduction of cost without sacrifice of recall.

Here is my argument against control sets and rant against vendors still using this method in a video made in 2017. I have chilled out a bit about this since then.

The use of Control sets, a practice that sill continues today for some, despite by best efforts to stop it, has created many problems. Attorneys who worked with predictive coding software versions 1.0 or 2.0, have seen their projects overtly crash and burn, as when missed smoking gun documents later turn up, or where reviewers see embarrassingly low precision. May lawyers were suspicious of the results at first. Even if not suspicious, they were discouraged by the complexity and arcane control set process from every trying predictive coding again. As attorney and search expert J. William (Bill) Speros likes to say, they could smell the junk science in the air. They were right. I do not blame them for rejecting predictive coding 1.0 and 2.0. I did too, eventually. But unlike many, I followed the Hacker Way and created my own method, called version 3.0, and then in later 2016, version 4.0. We will explain the changes made from version 3.0 to 4.0 later in the course.

As legal understanding is not static, experienced trial lawyers know that the case they start out with is not the one they end up with. Many predictive coding projects fail due to the use of magical control sets, resulting in low recall and precision. Webber notes that using the same control sample to guide the process and evaluate results is not statistically valid. Despite this, some attorneys may not have seen their projects crash and burn, yet still reject predictive coding 1.0 and 2.0 due to complexity and the smell of ‘junk science’. To ensure better precision and cost reduction, predictive coding 4.0 methods were developed in 2016.

The control set fiction put an unnecessarily heavy burden upon SMEs. They were supposed to review thousands of random documents at the beginning of a project, sometimes tens of thousands, and successfully classify them, not only for relevance, but sometimes also for a host of sub-issues. Some gamely tried, and went along with the pretense of omnipotence. After all, the documents in the control set were kept secret, so no one would ever know if any particular document they coded was correct or not. But most SMEs simply refused to spend days and days coding random documents. They refused to play the pretend wizard game.

Another GPT generated image

Every day that vendors keep phony control set procedures in place, is another day that lawyers are mislead on recall calculations based on them; another day lawyers are frustrated by wasting their time on overly large random samples; another day everyone has a false sense of protection from the very few unethical lawyers out there, and incompetent lawyers; and another day clients pay too much for document review. I continue call upon on all vendors to stop using control sets and phase it out of their software.

The best SMEs correctly intuited that they had better things to do with their time, plus many clients did not want to spend over $1,000 per hour to have their senior trial lawyers reading random emails in a control set, most of which would be irrelevant. I have heard many complaints from lawyers that predictive coding is too complicated and did not work for them. These complaints were justified. The control set and two-step review process were the culprits, not the active machine learning process. The control set has done great harm to the legal profession. As one of the few writers in e-discovery free from vendor influence, much less control, I have been free to blow the whistle, to put an end to the vendor hype. No more secret control sets. Let us simplify and get real. Lawyers who have tried predictive coding before and given up, come back and try Predictive Coding 4.0.

Another reason control sets fail in legal search is, as mentioned, the very low prevalence typical of the ESI collections searched. We only see high prevalence when the document collection is keyword filtered. The original collections are always low, usually less that 5%, and often less than 1%. About the highest prevalence collection ever searched in open testing of competing search methods was the Oracle collection in the EDI search experiment. There, although not announced, it was obvious to most participants that it had been heavily filtered by a variety of methods. That is not a best practice because the filtering often removes the relevant documents from the collection, making it impossible for predictive coding to ever find them. More on that later in the course. Grossman and Cormack have also written extensively on the issue of keyword filtering.

The control set approach also rarely works well in legal search because of the small size of the random sample usually taken by most vendors. The sample is rarely large enough to include a representative document from each type of relevant documents in the corpus, much less the outliers. It is not even close. For that reason, even if the relevance benchmark is not always evolving during the review, concept drift, the control set would still fail because it is incomplete. The result is likely to be overtraining of the document types on those that happened to hit in the control set, which is exactly what the control set is supposed to prevent. This kind of overfitting can and does happen for a variety of reasons, not just do to improper training based on an incomplete control set. That is an additional problem separate and apart from relevance shift.

Not only that, many types of relevant documents are never included in the control set because they did not happen to be in the random sample. The natural rarity of relevant evidence in unfiltered document collections, aka low prevalence, makes the control set useless and worse, leads to over training in the CAL process of the few relevant documents caught in the random control set.

These are problems mitigated, if not solved by the hybrid multimodal search aspects of predictive coding in version 4.0 that I teach.

Lawyers still using version 2.0 methods should be wary about the use of control sets to certify the completeness of a production. Separate samples should be preferred for making final assessments of production completeness. Yes, there is still a place for random selection of documents, but only for quality control sampling, not for initial training. As is explained in my Predictive Coding course, a separate random sample should be used at the beginning and end of a project, but only to assess production completeness, not as a first-guess control set. Random sampling is critical for project evaluation, but it is a terribly inefficient guide for training. In version 3.0, random sampling remained, but was used in an entirely different way. It was the third step in the eight-step  process of both my versions 3.0 and 4.0, but the use of a secret set of random documents, the control set, was eliminated. That represents the dividing line for me between versions two and three.

Fourth Generation: Predictive Coding 4.0 – Still State of the Art in 2023

The next method of Predictive Coding, version 4.0, builds on the prior methods. It still combines the two-stages into one, the Continuous Training technique, and still does not use secret control sets, but now it also uses a variety of other methods in addition to predictive coding. This is what I call a multimodal method and a man-machine hybrid approach. There are several flavors of this available today, but my TAR course explains the the techniques I have developed and tested since 2012, I further refined these techniques in 2016 to include a new variation of CAL.

We do not claim any patents or other intellectual property rights to Predictive Coding 4.0, aside from copyrights to my writings, and certain trade secrets. But Gordon Cormack and Maura Grossman, who are both now professors (and also now married), do claim patent rights to their methods. The methods are apparently embodied in software somewhere, even though the software is not sold. In fact, we have never seen it, nor, as far as I know, has anyone else, except their students. Their patents are all entitled Full-Text Systems and methods for classifying electronic information using advanced active learning technique: December 31, 2013, 8,620,842, Cormack; April 29, 2014, 8,713,023, Grossman and Cormack; and, September 16, 2014, 8,838,606, Grossman and Cormack.

The Grossman and Cormack patents and patent applications are interesting for a number of reasons.  For instance, they all contain the following paragraph in the Background section explaining why their invention is needed. As you can see it criticizes all of the existing version 1.0 software on the market at the time of their applications (2013) (emphasis added):

Generally, these e-discovery tools require significant setup and maintenance by their respective vendors, as well as large infrastructure and interconnection across many different computer systems in different locations. Additionally, they have a relatively high learning curve with complex interfaces, and rely on multi-phased approaches to active learning. The operational complexity of these tools inhibits their acceptance in legal matters, as it is difficult to demonstrate that they have been applied correctly, and that the decisions of how to create the seed set and when to halt training have been appropriate. These issues have prompted adversaries and courts to demand onerous levels of validation, including the disclosure of otherwise non-relevant seed documents and the manual review of large control sets and post-hoc document samples. Moreover, despite their complexity, many such tools either fail to achieve acceptable levels of performance (i.e., with respect to precision and recall) or fail to deliver the performance levels that their vendors claim to achieve, particularly when the set of potentially relevant documents to be found constitutes a small fraction of a large collection.

They then indicate that their invention overcomes these problems and is thus a significant improvement over prior art. In Figure Eleven of their patent (shown below) they describe one such improvement, “an exemplary method 1100 for eliminating the use of seed sets in an active learning system in accordance with certain embodiments.”


These are basically the same kind of complaints that I have made here against the older versions of Predictive Coding. I understand the criticisms regarding complex interfaces, that rely on multi-phased approaches to active learning. If the software forces use of control set and seed set nonsense, then it is an overly complex interface. (It is not overly complex if it allows other types of search, such as keyword, similarity or concept, for this degree of complexity is necessary for a multimodal approach.) I also understand their criticism of the multi-phased approaches to active learning, which was fixed in 2.0 by the use of continuous training, instead of train and then review.

The Grossman & Cormack criticism about low prevalence document collections, which is the rule, not the exception in legal search, is also correct. It is another reason the control set approach cannot work in legal search. The number of relevant documents to be found constitutes a small fraction of a large collection and so the control set random sample is very unlikely to be representative, much less complete. That is an additional problem separate and apart from relevance shift.

Notice that there is no control set in the Grossman & Cormack patent diagram as you see in the old Recommind patent. Much of the rest of the patent, in so far as I am able to understand the arcane patent language used, consists of applications of continuous training techniques that have been tested and explained in their writings, including many additional variables and techniques not mentioned in their articles. See egEvaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic DiscoverySIGIR’14, July 6–11, 2014. Their patent includes the continuous training methods, of course, but also eliminates the use of seed sets. I assumed this also means the elimination of use control sets, a fact that Professor Cormack has later confirmed. Their CAL methods do not use secret control sets. Their software patents are thus like our own 3.0 and 4.0 innovations, although they do not use IST, as will be explained in the course.

Recap of the Evolution of Predictive Coding Methods

Version 1.0 type software with strong active machine learning algorithms. This early version is still being manufactured and sold by some vendors today. It has a two-step process of train and then review. it also uses secret control sets to guide training. This usually requires an SME to review a certain total of ranked documents as guided by the control set recall calculations. Beware and avoid this software.

Version 2.0 of Predictive Coding eliminated the two-step process, and made the training continuous. For that reason version 2.0 is also called CAL, continuous active training. It did not, however,  reject the random sample step and its control set nonsense. Avoid that type of software too.

Predictive Coding 3.0 was a major change. Control sets are excluded. It built on the continuous training improvements in 2.0, but also eliminated the secret control set with mandatory initial review of a random sample. This and other process improvements in Predictive Coding 3.0 significantly reduced the burden on busy SMEs, and significantly improved the recall estimates. This in turn improved the overall quality of the reviews.

Predictive Coding 4.0 is the latest method and the one taught in my TAR course. It includes some variations in the ideal work flow, and refinements on the continuous active training to facilitate double-loop feedback. We call this Intelligently Space Training (IST), instead of CAL. It is an essential part of our Hybrid Multimodal IST method. All of this is explained in detail in my course. In Predictive Coding 3.0 and 4.0 the secret control set basis of recall calculation are replaced with a prevalence based random sample guides, and elusion based quality control samples and other QC techniques. These can now be done with contract lawyers and only minimal involvement by SMEs. See Zero Error Numerics. This will all be explained in the Course. The final elusion type recall calculation is done at the end of the project, when final relevance has been determined. See: EI-Recall.


Ralph in 2022, hoping he’s finally won his long battle against secret control sets in document review

The method of predictive coding taught in my predictive coding training course is Predictive Coding 4.0. It includes refinements on the continuous active training – Intelligently Space Training (IST) – and our Hybrid Multimodal IST methods. Predictive Coding 4.0 replaces the secret control set basis of recall calculation with prevalence based random sample guides, elusion based quality control samples and other QC techniques. These can be done with contract lawyers and minimal SME involvement. Final elusion type recall calculations are done at the end of the project, when final relevance is determined. Additionally, in version 4.0 of Predictive Coding, sample documents taken at the start for prevalence calculations are known and adjusted as relevance changes.

A secret control set is not a part of the Predictive Coding 4.0 method. Although we still use random selection reviews for prevalence and quality control purposes – Steps Three and Seven – but the documents are not secret and they are used for training. Moreover, 4.0 eliminates first round training seed sets, random based or otherwise. The first time the machine training begins is simply the first round. Sometimes the first training set of documents is large in number and type, sometimes it is not. There are no rules for the first training set. These topics are further addressed in the TAR Training course.

AI Analysis of the Top Five Cases in 2022 Shows Major Flaws in Use of Open AI’s GPT Software for Legal Research and Analysis

December 29, 2022

Ralph is now Skeptical of AI for legal research and analysis

As mentioned in my blog on the top 2021 cases earlier this week, my AI helper, Open AI’s GPT-3, made some unexpected selections. See the disclaimer and more extended discussion of this caveat in my last blog, Surprise Top Five e-Discovery Cases of 2021. When I later asked it to identify the top five cases for 2022, the AI performed very poorly and made me look bad. I published the AI’s report last night after cleaning up the words only, and not actually reading each case (it cited to WestLaw and I have Lexis and, good grief, its a Holiday and I’m only human). I gave the robot too much slack.

An astute reader, Maura, noticed right away that the alleged Top Five 2022 cases cited were not all in 2022. You could easily tell that from the Westlaw citations the AI gave. I checked further this morning, as I should have done last night. I then realized that the AI “emperor had no clothes,” the AI was totally off and I had been fooled. As soon as I received Maura’s helpful comment, I took my blog offline. In fact, most of the cases were much older than 2022, and a few appeared to have been made up out of whole cloth. Stupid me for not noticing before. I blame a concussion, and my over-trust of the new, much hyped GPT 3.5. I will be more skeptical henceforth. When it comes to legal research Text-Davinci-003, the Open AI tool that I used, is definitely not ready for primetime.

I did not use Chat GTP 3.5 for any of these queries because that database ends in 2021. Although I am not completely certain about that, and if I did, that would explain some of the errors. Instead, I believe I used the Open AI playground found at It’s data is supposedly up to date. Guess my first error was to not pay enough attention to the beta part of the hyperlink. To be more specific, I used what Open AI calls the Text-Davinci-003. In defense of OpenAI, the Chat GTP, where the database ends in 2021, is supposed to be a big improvement over Text-Davinci-003. If you do not care about 2022 cases, and apparently few do, you could used Chat GPT 3.5.

AI Enhanced Ralph

The introduction the AI wrote for the first draft of the Top Five 2022 article that I imprudently published last night looked pretty good:

Introduction. As technology continues to evolve, so must the legal landscape. As my regular readers all well know, in the United States, electronic discovery is an increasingly important field of law that governs methods and processes for obtaining, preserving, and exchanging electronically stored information (ESI) in litigation. As such, it is critical that lawyers stay informed on recent legal opinions concerning e-Discovery. In this article we will discuss the five most important United States district court judge or magistrate opinions on electronic discovery in 2022 and explain why each decision is important.

But really, it doesn’t say much, does it?

Then, the day before yesterday, the AI picked the following cases as the top five for 2022. Note the names and dates:

  1. Toth v. Johnson & Johnson, 1:20-cv-00591, 2020 WL 5444439 (C.D. Cal. Sept. 11, 2020). (Note: The AI claims the opinion was written by Judge Banikian. I can find no such judge!)
  2. Google LLC v. Oracle America Inc., 5:20-cv-03624, 2021 U.S. Dist. LEXIS 66448 (N.D. Cal. Apr. 9, 2021)
  3. Schulman v. Smith & Wesson Corp., 0:20-cv-60780, 2020 U.S. Dist. LEXIS 148034 (S.D. Fla. Aug. 14, 2020)
  4. Oracle America Inc. v. Google LLC, 786 F.3d 1179 (N.D. Cal. 2014)
  5. Microsoft Corporation v. Quinnipiac University, 3:20-cv-01705, 2020 WL 6079805 (N.D. Cal. Oct. 14, 2020)

This is the list that Maura questioned. None of these case is even a 2022 case by the AI’s own dubious report. WTF!

No doubt the AI will blame human error in operation. So I asked it again this morning, December 30, 2022. Anyway, the query I input was: Please identify the top five most important legal cases in 2022 on electronic discovery. Here is my response this time, and note at least the dates appear right.

  1. In re: Google LLC v. Oracle America, Inc., No. 18-956 (U.S. Supreme Court 2022).
  2. Microsoft Corp. v. United States, No. 1:16-cv-02985 (D.D.C. 2022).
  3. U.S. v. Microsoft, No. 1:17-cr-0019 (E.D.N.Y. 2022).
  4. In re: Apple Inc., No. 5:19-cv-05847 (N.D. Cal. 2022).
  5. In re: Facebook, Inc., No. 5:19-cv-07189 (N.D. Cal. 2022).

Well, this looks a lot better, so I asked the AI the exact same question again to see if I would get the same answer. I didn’t. Here is what the AI came up with this time. It added a little explanation for the cases too. It did not do that the first time. I have no clear idea why. I asked the exact same question.

  1. United States v. O’Keefe (11th Circuit) – This case involves the legality of the government’s use of an electronic discovery technology to access data from a suspect’s computer, as well as the scope of the Fourth Amendment’s protection against unreasonable searches and seizures.
  2. In re Google (California Supreme Court) – This case considers whether Google must comply with a discovery request for emails stored on its servers.
  3. EEOC v., Inc. (7th Circuit) – This case examines the scope of an employer’s obligation to preserve electronically stored information and the admissibility of such evidence in a case alleging discrimination.
  4. Doe v. Facebook, Inc. (9th Circuit) – This case considers the scope of an individual’s right to privacy with respect to their online content, including emails, photos, and other communications stored on Facebook.
  5. National Labor Relations Board v. Apple, Inc. (2nd Circuit) – This case examines the scope of the National Labor Relations Board’s investigative powers to access and review electronically stored information held by employers in the course of an investigation.

This actually looks like a better list. But I am very wary now. The AI did not include any kind of citation this time, unlike the past time with the same question. Again, I have no idea why. So I asked the follow-up query, Please provide the full legal citation for these five cases. Here is the AI response.

  1. United States v. O’Keefe, No. 19-14883 (11th Cir. 2022).
  2. In re Google, No. S251764 (Cal. 2022).
  3. EEOC v., Inc., No. 19-2180 (7th Cir. 2022).
  4. Doe v. Facebook, Inc., No. 19-15645 (9th Cir. 2022).
  5. National Labor Relations Board v. Apple, Inc., No. 19-1718 (2nd Cir. 2022).

Ok, better, but human Ralph is still skeptical, so I asked a third time, Please identify the top five most important legal cases in 2022 on electronic discovery and provide full legal citations. Here is the robot answer this time.

  1. In re: Intuit Inc. Consumer Data Security Litigation, No. 5:20-cv-07982-EJD (N.D. Cal. 2022)
  2. National Bank of California v. KPMG, LLP, No. BC716987 (Cal. Super. Ct. 2022)
  3. In re: Google LLC Consumer Privacy Litigation, No. 2:20-cv-07837-JAK (C.D. Cal. 2022)
  4. In re: Apple Inc. iPhone Privacy Litigation, No. 5:20-cv-07999-EJD (N.D. Cal. 2022)
  5. In re: Facebook, Inc. User Data Security Litigation, No. 5:20-cv-08012-EJD (N.D. Cal. 2022)

You know the old expression, about consistency being “the hobgoblin of little minds.” So I asked again, a fourth time, although it is getting somewhat boring now. This time my query was: Please identify the top five most important legal cases in 2022 on electronic discovery and provide full legal citations and a brief explanation of why each case is important. Here is the result:

1. Doe v. Google LLC, No. BC641932 (Cal. Super. Ct. 2022): This case will be important because it will address the scope of Google’s duty to preserve and produce electronically stored information (ESI) under California law. The court’s decision on this issue will provide guidance on the legal expectations for ESI preservation and production across the country.

2. United States v. Apple Inc., No. 1:22-cv-04106 (N.D. Cal. 2022): This case will be important because it will determine the scope of Apple’s legal obligations when it comes to producing ESI in response to a federal criminal investigation. This case has the potential to set a precedent for how technology companies handle ESI in criminal proceedings.

3. In re Facebook, Inc., No. C-22-03763 (N.D. Cal. 2022): This case will be important because it will determine the scope of Facebook’s legal obligations when it comes to producing ESI in response to a civil discovery motion. This case has the potential to set a precedent for how technology companies handle ESI in civil proceedings.

4. Microsoft Corp. v. Twitter, Inc., No. 2:22-cv-0134 (W.D. Wash. 2022): This case will be important because it will determine whether Twitter must produce ESI in response to a civil discovery motion. The court’s decision on this issue will provide guidance on the legal expectations for ESI production in both state and federal civil proceedings.

5. In re Comcast Corp., No. C-22-04341 (E.D. Pa. 2022): This case will be important because it will determine the scope of Comcast’s legal obligations when it comes to producing ESI in response to a civil discovery motion. This case has the potential to set a precedent for how technology companies handle ESI in civil proceedings.


Maybe AI’s Text-Davinci-003 remembers it’s prior answers and comes up with a different list each time on purpose. Maybe that is a good way to create a top 20. I have many questions. I will keep working with it, when I get time, and let you know. GPT software is very interesting and shows a lot of potential. But be warned, it should not be used at this time without a great deal of skepticism and close human supervision. Hybrid, human and computer, working together – IMO that is the answer to for the proper use of AI for the foreseeable future. I am still very pro AI, but for now at least, the human needs to keep a close eye on the robot.

%d bloggers like this: