The Legality of Generative AI—Part 2

I’m sorry, User. I’m afraid I can’t do that.

September 2023 By Colin E. Moriarty

This is the second in a series of articles discussing the legal implications of generative AI. This installment discusses potential risks to end users of AI in the commerce context.

Just as pocket calculators lighten the mental weight associated with arithmetic, generative AI promises to do the same for writing, art, and other creative endeavors. Rather than spending decades developing an artistic style, individuals can spend an afternoon describing to a computer program the kind of art they want to see and then select the best results from among the hundreds of output images. Rather than wrestle with essay structure and topic sentences, a writer can generate a grammatically correct first draft in a few minutes of prompting. In addition to producing output for the unskilled, this capability may also speed up the output of skilled professionals. But this potential increased efficiency is not without consequences. This article addresses some of the apparent risks to the generative AI end user, with a focus on use in commerce.1

An Efficient but Unreliable Business Partner

Some studies suggest that the use of generative AI has already increased worker productivity by 14%.2Combined with the ability to work remotely, some workers report becoming so productive that they can work several jobs simultaneously.3 The technology may be new, but businesses are already adopting generative AI, particularly large language models (LLMs), into their workflow.4

This potential utility comes with costs, of course.5 Some of these costs are to society as a whole. How does society change when individuals need not invest the personal growth required to compose, write, or draw at a certain baseline level of competence?6 Generative AI may harm those whose jobs are changed, replaced, or devalued, as is often the case with automation.7 Other risks are specific to individual users who may wish to integrate generative AI to generate content, interact with third parties, or make decisions.

Of course, it is possible to intentionally use generative AI to advance bad goals. Cybercriminals use generative AI to create viruses and malware.8 Scammers use AI to learn and replicate family member’s voices to place phone calls asking for money.9 Counterfeiters use AI to learn and copy the literary or artistic style of another.10 Political actors use it to create fake videos for campaign purposes.11 Generative AI is used to create fake data to stymie researchers.12 These are serious problems keeping businesses and law firms on their guard. But while generative AI may make it easier to commit or harder to detect fraud, these kinds of intentional bad acts are not alien to the law.

Software that can misbehave unbidden is more novel. The first article in this series explained why, at a very fundamental level, it is not possible to perfectly predict the behavior of generative AI.13 To recap briefly, generative AI models are not directly programmed with a series of instructions by human beings. Instead, the programmer provides a training framework and a large set of training data to a model, and then repeatedly tests how well the model performs compared to its training data. The programmer then adjusts the model to perform slightly better the next time until the model ends up generating a good internal map between prompts and the desired output. No one knows exactly what kind of internal algorithms the model ended up using. Moreover, the training process is not identical to the real world, and behavior that may have worked well in training may produce erroneous results in practice. Finally, though the software often appears omniscient when it comes to information in its training set, it is a far cry from omnipotent and will often produce vague and nonspecific output unless carefully prompted or managed to do otherwise.14

In its current state, therefore, the software behaves like an untrained, entry-level intern with full access to the Internet and great technical writing skills but without experience, context, or particular loyalty to your company. If a business would not entrust a task to such an individual, it probably should not entrust the task to AI.15

User Concerns Regarding Intellectual Property

A natural role for generative AI in a business context is to generate content.16 Automating the human creative process brings with it several new risks, however. Without more involvement from a human creator, the work product of AI is probably not protected by either copyright or patent law, leaving a business unable to protect what it created. And, as a golem without loyalty or context, AI could create legal problems if it infringes on prior works without the user’s intention or knowledge.

Difficulty in Protecting Intellectual Property in Works Generated by AI

While users may be prone to anthropomorphize, the law is not. Works wholly created by generative AI are likely not protected by intellectual property law because they are not human. Sometimes, non-human entities do have rights under the law. Corporations, governments, boats, and others can own property and exercise rights.17 Colorado law, for example, expressly conveys rights on corporations,18 and many state statutes expressly include entities in the definition of “person.”19 Where non-humans have rights, however, it is typically the result of an express exception to the normal assumption that laws apply only to natural persons. Courts have sometimes expanded on the rights afforded to entities,20 but premised on the entity as a vehicle for human constitutional rights.21

The judicial presumption appears to be that laws are intended to apply to human beings except as otherwise stated. So, for example, “the world’s cetaceans” (whales, porpoises, dolphins, etc.) do not have standing to bring claims under the Endangered Species Act or similar laws because, though Congress could have chosen to authorize suits by animals, it did not do so.22 Absent a law to the contrary, “[a]nimals are simply not capable of suing or being sued . . . .”23

Thus, even though the Copyright Act does not expressly state that an author must be human for a work to qualify for copyright protection, courts have held that this is so.24 In Naruto v. Slater, a crested macaque discovered a wildlife photographer’s camera and took several photographs of itself.25 The photographer published the photos and was sued by PETA, on behalf of the monkey, for violating the monkey’s alleged copyright.26 The Ninth Circuit noted that the Copyright Act “does not expressly authorize animals to file copyright infringement suits” and explained that the Act’s use of human family terms such as “children . . . legitimate or not, . . . widow, and widower, all imply humanity and necessarily exclude animals that do not marry and do not have heirs . . . .”27 In Urantia Foundation v. Maaherra, the Ninth Circuit also refused to acknowledge copyright rights for a book allegedly “authored by celestial beings” and instead based its analysis on the humans who arranged it and wrote it down.28

The US Copyright Office interprets the term “author” in the Copyright Act to “exclude non-humans,” including generative AI.29 It requires that any work be the product of human authorship to be eligible for copyright protection.30 With respect to generative AI, the Copyright Office weighs the specific facts of the creation and “will consider whether the AI contributions are the result of mechanical reproduction or instead . . . an author’s own original mental conception . . . .”31 The question appears to be whether the generative AI is responsible for the creative work or is merely being used as a tool by a creative human.32 At one extreme, the office refused to grant a copyright in an image that was entirely “autonomously created by a computer algorithm,” according to its author.33 In a more recent case, the office took a more nuanced approach with a comic book written and arranged by a human, but where the art was entirely the creation of generative AI.34 In this case, the office decided that the art could not be copyrighted, but the other creative elements could be since they were the product of a human.35

Patents, too, must be the invention of a natural person to warrant protection.36 In Thaler v. Vidal, an individual claimed to have developed AI systems that generate patentable inventions and attempted to patent two outputs of his AI.37 Despite prompts from the US Patent and Trademark Office to identify someone as the inventor, he insisted that the AI was the inventor.38 His patent was denied because “a machine does not qualify as an inventor,” and the Federal Circuit affirmed.39 The court reasoned that the use of the word “individual” in the Patent Act ordinarily meant a human being and, absent an indication that Congress intended a different result, the meaning was plain.40

While works created wholly by AI are therefore unlikely to be protected, there is probably some level of human involvement that can likely result in copyrightable or patentable work. Exactly how much human involvement is needed is an open question, but the answer probably lies in how much and what kind of creative work is performed by humans after receiving a result from the software. The Copyright Office suggests that feeding a text prompt into the software is not enough.41 It explains that “prompts function more like instructions to a commissioned artist—they identify what the prompter wishes to have depicted, but the machine determines how those instructions are implemented in its output.”42 Whether instructing a human or AI software, the user describes the work in text and gets back an image, but the resulting work is only eligible for protection when a human artist is involved. Without more, the raw output from generative AI may fall into the public domain.43

So, the human input needed to elevate an AI-generated work may be the same as that needed to establish copyright for a work derived from something in the public domain. Doing so requires that the final work be “original,”44 meaning it must “contain some substantial, not merely trivial originality.”45 While triviality may be in the eye of the fact finder, it is not commonly held to be a high bar. The test of originality is “concededly one with a low threshold.”46 Some decisions suggest that originality “means little more than a prohibition of actual copying.”47 Thus, it may be that only moderate reworking of the output of generative AI by an artist48 will allow protection for the final derivative work even if the original AI output remains unprotected.

It is conceivable that, as the capabilities of AI grow, new statutes may allow individuals or entities to obtain intellectual property rights in the output of generative AI. Commentators have proposed giving artificial intelligence personhood, at least indirectly, through the establishment of an “autonomous entity.”49 The scheme involves having a human set up a limited liability company, convey property to the LLC, establish an operating agreement requiring the LLC to act at the direction of a computer program, and then disassociating from the company so that it has no (human) members.50

There does not appear to be any current law dealing with the propriety of an algorithm functioning totally autonomously as a legal person, but there are clues that the law is not quite ready to dispense with the need for human creators entirely.51 Four states have experimented with allowing digital management of entities, known as decentralized autonomous organizations (DAOs), but these entities ultimately are owned by human beings.52 When faced with legal questions concerning DAOs, a court may still seek out the humans who created or operate the software.53 In one case brought by a government regulator against a DAO, the US District Court for the Northern District of California determined that this was an “unincorporated association” of the DAO’s human affiliates under applicable law for purposes of service of process.54 Even if this tendency to seek out the humans in charge of an algorithm relaxed, that would not solve the copyright problem. An entity can become the owner of works assigned to it55 or even be considered the author if the artist was doing work for hire,56 but both of these doctrines still assume the original creation was the product of a natural person.

Unintentional Infringement Through Use of Generative AI

A user who explicitly prompts generative AI for a new Star Wars screenplay or a Harry Potter novel would probably find no protection in the fact they used ChatGPT instead of Microsoft Word to write it. A standard word processor, however, will not generate an infringing work unbidden. Generative AI can, for the same reason that the Copyright Office does not believe the creations of generative AI can be protected: it mechanizes the human creative process. The output of generative AI might infringe on a work that the user has never seen.57 If it does, would the user of the software be liable? Perhaps.

To begin with, the user’s subjective intentions are probably not relevant. Copyright infringement is a strict liability claim.58 As to state of mind, all that is required is to show that the user engaged in a volitional act related to the infringement. So, for example, an Internet provider was not liable where a user posted an infringing image because the provider took no volitional act toward the infringement.59 In the case of generative AI, however, the user is typing in a prompt that calls the text or image into existence. This act may be sufficient to incur liability.60

The determinative question will likely be whether the user can be said to have had “access” to the original work. As part of showing infringement, the plaintiff must show that the defendant copied original elements of the first work.61 Showing “copying” is important because independent creation is a complete defense to copyright.62 It is not infringement for two creators to create similar, even identical, works without ever having seen the other.63 In the absence of direct evidence of copying, a plaintiff can show copying by demonstrating that the defendant had “access to the plaintiff’s work and that the two works share similarities probative of copying.”64 Showing prior access merely requires evidence that the defendant had a reasonable opportunity to view or opportunity to copy the prior work.65 This can be direct, such as proof that the defendant specifically viewed the original work,66 or a more general showing that the prior work was widely displayed in close geographic proximity to the defendant.67 Since intention is not relevant, once prior exposure is shown, even subconscious copying is sufficient to support infringement.68

In some cases of infringement, such as ones involving widely known characters or stories, it may be possible to show that the user had access to the original works through the general popular media. But, using generative AI, it is possible for a user to create a work that is similar to a more obscure existing work without ever having seen it. Perhaps the model “saw” it, though. Is that enough? There appears to be no guidance in the existing law. After all, never before has a writer had to worry that her typewriter might have read a story she did not. It remains to be seen how courts will resolve this issue. Perhaps the same arguments that suggest human involvement is required to have a copyrightable work in the first place might similarly suggest that exposure to human memory might be required for infringement.

Even if the courts determine that “access” can be shown by demonstrating that the model was trained on a dataset including the offending work, there may be other unique challenges to a plaintiff seeking to prove infringement. A copyright plaintiff likely will not have access to the training data used by the generative AI software employed, and the software developer has no incentive to voluntarily disclose the details of the training set.69 Some plaintiffs are arguing that they can divine the inclusion of their work in the training process by showing that the generative AI is able to summarize that work.70 But since this might only indicate that the model was trained on summaries or reviews of the original work and not the original work itself, it is unclear if that strategy will be successful. Attempts to prove “access” based on the model’s training may thus flounder upon difficult third-party discovery.

For the same reason, a user wishing to avoid infringement might have a hard time knowing in advance whether the model was trained on a particular work because that user also has no access to the underlying training data. Nevertheless, a business that is serious about using generative AI may be able to mitigate the risk of accidental infringement by checking the output for similarity to existing work. The best course would probably be to hire a copyright lawyer to conduct due diligence, but there are budget-friendly options. One option might be to employ generative AI to solve its own problem. Midjourney, a generative AI for artwork, has a “/describe” command that essentially runs the model in reverse to generate possible text prompts from an image.71 This can include the names of artists who the model believes create similar work. The user could then inspect other works by those artists for similarity. Google, too, has a “reverse image search” that allows users to upload an image and use Google’s search engine to identify possible similar images online.72

Providers of generative AI can and are taking steps to minimize this problem, but it remains to be seen how effective those steps are. The model does not exist in a vacuum but relies on other code to obtain input and provide output to the user. Providers can implement external software that looks for improper behavior and stops it. Recently, users have observed changes to Open AI’s ChatGPT that abort responses that appear to be providing copyrighted material.73 Microsoft may have implemented some form of copyright monitoring and warning system in its image-generating software.74 Google’s music-generating AI, MusicLM, appears to reject prompts that use the name of an existing song, artist, or other intellectual property.75 As the technology improves, it may become more commercially viable to rely on providers to filter out potential infringement.

Misrepresentations and Misstatements Related to AI in Commerce

The danger to a business using generative AI does not end with intellectual property law. In any situation where an untrained, disloyal employee could cause mischief, generative AI potentially could, too. A business should not lie about the skill level of its employee. It should properly train and supervise its employees. If the employee lies to third parties within the scope of employment, the business may find itself with legal exposure or worse. And, of course, a business cannot suggest that its employee can perform services that can only be performed by a licensed professional. So too with generative AI.

Misrepresentation About Services or Products Containing AI

As of the writing of this article, excitement about the possibilities of generative AI has reached a fever pitch.76 There may be a temptation to cash in on the hype by misrepresenting the details or capabilities of systems. The Federal Trade Commission (FTC) has been quick to issue warnings about the same. The FTC issued a statement in which it warns businesses that “artificial intelligence” is “an ambiguous term with many possible definitions.”77 It warns against two specific forms of possible misrepresentation: (1) misrepresenting whether a business actually uses AI at all, and (2) exaggerating what the technology can do.78 The FTC warns that it can “look under the hood” to see whether AI is actually being sold, and further notes that using an AI tool to develop a product does not mean the product “has AI in it.”79

If a business makes misleading claims about the functionality of its services, it may be liable under state and federal law. The Colorado Deceptive Trade Practices Act prohibits false representations about the quality, characteristics, uses, or benefits of goods or services.80 The misrepresentation is equally actionable whether it was done “knowingly” or merely “recklessly.”81 Under the Federal Trade Commission Act, any “unfair or deceptive acts or practices in or affecting commerce” are unlawful.82 The Act defines “unfair” conduct as conduct that “causes or is likely to cause substantial injury to consumers which is not reasonably avoidable by consumers themselves and not outweighed by countervailing benefits to consumers or to competition.”83 A representation is “deceptive” if it is likely to mislead consumers acting reasonably under the circumstances and is material.84

A business should be careful about advertising the use of “AI” at all. In the recent past, the discussion over AI has had a lot to do with generative AI, namely, software that is able to produce creative works such as text, art, code, or music. But the term “AI” is older than 2023 and has changed over time. It was coined in the 1950s to refer to a computer performing tasks that were previously only capable by beings already recognized as intelligent.85 Early computers like ENIAC first tackled the problem of arithmetic, which had previously been something only a human could do.86 Over time, each time engineers figure out how to automate a task that was previously assumed to require “intelligence,” that task tends to be removed from the scope of what is considered “true intelligence,” and the popular definition of AI retreats to accommodate.87 Today, a pocket calculator is not considered AI even though it is doing an intellectual task only humans could do before its invention.

Because the definition of AI is fluid, the term is vulnerable to being used in a misleading way.88 It may not be possible to determine precisely what can and cannot be called AI, but at least some legal commentators suggest that in the modern context, AI refers to a computer program that does not “operate based on a definite set of pre-programmed instructions” but instead is “trainable and able to learn from experiences” and thus produce output “not contemplated by the human-in-interest.”89 In other words, a company marketing its product as AI in 2023 may need to consider the present context, in which “AI” usually refers to software that runs on a model created by machine learning and has some level of unpredictability, as opposed to a piece of software that was explicitly written with conditional statements to produce a given output for a given result. So, for example, it may be a deceptive trade practice for a business to claim it is using “artificial intelligence” to generate documents if it merely runs a website that fills in the blanks on a form in the fashion of a Mad Lib.

Even if the service offered does use the modern version of AI, businesses should be truthful about its capabilities and limitations. Generative AI software is unpredictable and may generate improper or incorrect output.90 In the case of LLMs, some of this unpredictability is inherent in the way the models are trained and refined. An LLM is generally first trained to accurately predict the next word by testing it based on its training data. Then, the currently popular LLMs are also taken through a period of human-assisted reinforced learning involving a human subject ranking how much they like the output of the LLM.91 As a result, the model ends up with a good ability to predict the next word based on its training data and human-based feedback, not based on what is objectively true.

In several well-documented cases, LLMs are prone to “hallucinate,” or make up false information, providing a confident and well-written response that is not justified by the training data.92 The model’s training process encourages the LLM to provide some kind of response that sounds like what the LLM was trained on and would be pleasing to a human. LLMs can be asked to provide an explanation for a scientific phenomenon that does not exist,93 provide incorrect biographical information,94 agree with false or fake claims,95 get the date wrong,96 invent false court decisions,97 or become stuck in a loop of unhinged ranting.98 Users must bear in mind that LLMs were not trained to navigate the real world, only to predict the next word. Currently, LLMs exist in Plato’s cave,99 and all they know about how the real world operates is from the shadows reflected in the language upon which they trained. Blind faith in the veracity of current LLMs, like blind faith in an unskilled entry-level intern, is a recipe for disaster.100

So too would be promises of infallibility by businesses offering generative AI. To the contrary, businesses should provide prominent disclaimers concerning the risks and instability of the software. This is the approach taken by OpenAI, which states in its disclaimer that it “takes no responsibility or warranties about the completeness, accuracy, or reliability” of its information101 and notes in its terms of use that its software “may in some situations result in incorrect Output that does not accurately reflect real people, places, or facts.”102 Google, likewise, warns its users that “LLM experiences (Bard included) can hallucinate and present inaccurate information as factual.”103 A disclaimer is probably the bare minimum a company can do, however. It may be better to ensure that customers enter into user agreements or similar contracts before getting access to tools powered by AI, in which customers are not only warned of the limitations of the software but also affirmatively agree to shift the responsibility for those errors to the customer or otherwise hold the business harmless.

Misrepresentations or Mistakes Made by AI

Instead of using generative AI internally, some businesses may want to give their customers direct access to generative AI. LLMs take human language as input and output, code, and data; customers used to texting or typing emails can easily interact with LLMs.104 Companies are exploring use cases where customers obtain access to a trained LLM as the core product, including online tutoring,105 therapy,106 weight loss guidance,107 small talk for romantic partners,108 or playing the role of an AI romantic partner itself,109 and as a supplement to an existing product, such as customer service chatbots.110 Any time a business is connecting generative AI to its customers directly, however, it must be concerned about the AI itself making mistakes or misstatements that could get the business into trouble.

Federal law likely provides no safe harbors. The normal safe harbors that protect companies that provide access to information online probably are not going to help when a business gives customers access to generative AI. There are two major safe harbors in federal law for online providers: the Digital Millennium Copyright Act (DMCA)111 and the Communications Decency Act (Section 230).112 Both of these safe harbors apply only where the offending material was provided by a third party, not where it was generated by the service provider itself. The DMCA provides that a service provider shall not be liable for copyright infringement by reason of the provider’s transmitting, routing, or providing transient storage of infringing material under certain conditions.113 Among other things, those conditions require that “the transmission was initiated by or at the direction of a person other than the service provider.”114 Section 230 offers broader protection to service providers, not just protection from copyright infringement. It states that “no provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider.”115 As with the DMCA, it only offers protection against information “provided by another,” not against information provided by the service provider itself.116

Whether DMCA and Section 230 protect a business against bad behavior by customer-facing LLMs may depend on how the LLM is being used. If a chatbot is acting merely as a “neutral tool” to retrieve information from another provider or user, the business might be able to claim Section 230 protection.117 This argument could arise for a provider that, for example, allows users to train chatbots that then have preliminary conversations on the user’s behalf with potential dates118 or uses a chatbot to facilitate research into articles published by third parties.119 Here again, though, the peculiar way in which generative AI works may be its downfall. In Fair Housing Council v. Roommates.com, LLC, the Ninth Circuit had to consider whether a website was involved in developing information provided by users about sex, sexual orientation, and the presence of children, because the website required users to provide that information.120 The court found that by “requiring subscribers to provide the information . . . and by providing a limited set of pre-populated answers,” the website became more than a passive transmitter and was actually engaged in developing the content.121 Thus, the website had no immunity. In the case of generative AI, too, the provider is probably not able to merely provide naked access to third-party data. That kind of functionality does not require a chatbot. Instead, the LLM will create new text based on what relationships in its model are triggered by the user’s input. If merely posing multiple choice questions is enough involvement to lose Section 230 protection, then it is hard to see how generating entirely new explanations or paraphrases would not also meet that threshold.122

Defamatory statements by generative AI. A misbehaving chatbot could cause trouble for a company in basically the same way as a rogue employee chatting with customers. If the LLM publishes false and scandalous biographical information about a third party to a user, the subject of the information might have suffered actionable defamation. One such case was filed this summer in Georgia, in which a reporter allegedly asked ChatGPT to summarize the accusations in a complaint filed in the Western District of Washington.123 In response, ChatGPT hallucinated accusations that one Mr. Walters was “accused of defrauding and embezzling funds.”124 Walters, upon learning of this, filed a defamation lawsuit. In Colorado, defamation is “a communication holding an individual up to contempt or ridicule that causes the individual to incur injury or damage”125 that is false 126 and published to a third party.127 Certain kinds of accusations, including allegations of criminal conduct, are per se defamatoryand are actionable even without showing damage.128 If a chatbot spouts defamatory statements to the public, especially if it slurs business competitors, a business may risk becoming a test case for defamation liability.

The risk of defamation claims might be mitigated with a disclaimer to the effect that the LLM output is inaccurate, non-factual, or not the opinions of the business, but it is unclear if such a disclaimer will eliminate the problem. On the one hand, simply labeling a work “fiction” does not prevent it from being defamatory.129 In Muzikowski v. Paramount Pictures Corp., the little league coach featured in the book on which the movie Hardball was based sued the movie studio for defamation.130 The Seventh Circuit reasoned that even though the movie was labeled “fiction,” the coach was still entitled to present evidence that a character in the movie was actually intended to portray him.131 On the other hand, Fox News was recently successful in dismissing defamation claims brought against it based on statements of one of its hosts by arguing that, in context, the host’s statements could not be construed as provable facts but mere hyperbole.132 The court considered the context of the host’s speech as being part of an ongoing political commentary and decided that the “general tenor of the show should then inform the viewer that he is not stating actual facts” and that any reasonable viewer must approach “with an appropriate amount of skepticism.”133 If a chatbot spouts defamatory statements about a third party but there is a prominent disclaimer not to believe those statements, is that more like a cursory note that the information is fiction or more like political partisan spouting arguments in a subjective forum?

Discriminatory statements by generative AI. A misbehaving chatbot might also make discriminatory statements concerning race, gender, national origin, or other protected classes. An LLM’s output is only as good as the text that trained it and the human preferences that refined it. If the training process included biases or misinformation, the model will include associations to those patterns.134 For example, some have reported finding evidence that certain LLMs identify certain jobs, like “flight attendant” or “secretary,” as feminine and others, like “lawyer” or “judge,” as masculine.135 This problem can be exacerbated by users using “prompt injection,”136 such as telling the LLM to answer in the persona of someone who would say discriminatory things.137 At least in the view of many federal agencies, including the Equal Employment Opportunity Commission (EEOC), businesses are liable for actionable discrimination carried out by their computer programs.138 In a press release describing enforcement action against a tutoring company that allegedly used an algorithm to reject any applicants above a certain age, the EEOC stated: “Even when technology automates the discrimination, the employer is still responsible.”139

In Colorado, discriminatory statements by a chatbot might give rise to problems under several laws. Discriminatory responses from LLMs in the employment context may cause violations under the Colorado Anti-Discrimination Act.140 This is true even if the LLM is not actually making any decisions. The Act prohibits an employer from “caus[ing] to be printed or circulated any statement” in connection with prospective employment that “expresses, either directly or indirectly, any . . . discrimination as to disability, race, creed, color, sex, sexual orientation, gender identity, gender expression, religion, age, national origin, or ancestry.”141 It also prohibits “harass[ment],” which means creating a hostile work environment based on a protected class.142 Colorado has several unique laws governing discrimination by AI, including prohibiting insurers from using algorithms to unfairly discriminate143 and requiring companies that handle a large amount of private customer data to conduct annual reviews for fairness and disparate impact for certain decisions that produce legally significant effects on a consumer.144

Contracts or promises made by generative AI. Another potential risk for businesses is that a customer-facing chatbot might make promises or representations that could later bind the company. If a customer, through prompt injection or inadvertence, coaxes a chatbot to promise to sell goods at a particular price, for example, would that create a contract binding on the company? This problem has already arisen in the field of smart contracts that incorporate algorithms into their terms.145 It might be possible for a chatbot to produce written words communicating to a third party that objectively show an offer to enter into a contract, and some commentators suggest that may be enough.146 There are cases supporting the idea that a company can be bound by errors in its algorithms. In Bristol Anesthesia Services., P.C. v. Carilion Clinic Medicare Resources, LLC, the US District Court for the District of Tennessee found a triable issue of fact for an implied-in-fact contract claim where an invoice was generated and paid in the wrong amount due to an error in the algorithm.147

Even if the words generated by AI are not found sufficient to demonstrate an actual meeting of the minds between a customer and a business, promises or representations might still cause problems. Promissory estoppel, for example, does not require actual acceptance.148 Instead, it requires that (1) the promisor made a promise to the promisee, (2) the promisor should reasonably have expected that the promise would induce action or forbearance by the promisee, (3) the promisee reasonably relied on the promise to the promisee’s detriment, and (4) the promise must be enforced to prevent injustice.149 If a chatbot hallucinates a statement that an uninformed customer might reasonably interpret as a promise to provide goods or services at a certain price or in a certain amount and then relies on that promise, an estoppel claim might arise.

As with accidental infringement, improvements to the technology may minimize the risk of generative AI misbehaving, but it is unclear whether it can ever be entirely eliminated any more than human employees can be guaranteed to always act perfectly. As mentioned above, a business employing customer-facing generative AI would be wise to ensure that customers sign a user agreement expressly noting the limitations of the software and making it clear that the software is not an agent with authority to bind the company. A business would also be wise to carefully supervise and monitor any such generative AI.

Practicing Without a License

Because the most popular LLMs are trained on a massive corpus of human writing, including specialized professional knowledge, the models end up learning relationships between words common to those professions. This means, as a practical matter, that they can answer medical, legal, or scientific questions. Some can do this sufficiently well to pass law school tests,150 the bar exam,151 and medical licensing tests.152 This capability may tempt some companies to market generative AI as a substitute for a professional in a specialized field. Some professions, however, require a license. These include attorneys,153 doctors,154 certified public accountants,155 public adjusters,156 real estate brokers,157 and others.158 The model may be capable of producing specialized information and even sometimes providing correct advice, but a business is asking for trouble if it sells an LLM for this purpose.

Marketing or using generative AI as a substitute for a licensed professional likely violates Colorado law. The practice of law, for example, includes “act[ing] in a representative capacity in protecting, enforcing, or defending the legal rights and duties of another and in counseling, advising and assisting [another] in connection with these rights and duties . . . .”159 More broadly, practicing law involves “the exercise of professional judgment, calling upon ‘legal knowledge, skill, and ability beyond [that] possessed by a layman.’”160 Charging a fee to prepare legal documents for another can also constitute the practice of law.161 There is no reason to think that the Colorado Supreme Court would find the artificial judgment of generative AI is permitted to engage in this kind of conduct.

Some jurisdictions have recognized a so-called “scrivener’s exception” that allows unlicensed individuals to merely record information that another provides so long as the individual exercises no judgment at all.162 It seems unlikely that generative AI could fall under this exception because LLMs do more than merely record information; they use a model of relationships between prompts and text output to predict the appropriate legal advice or document being requested. In Conway-Bogue Realty Investment Company v. Denver Bar Association, the Colorado Supreme Court determined that unauthorized practice of law specifically includes the preparation of promissory notes, deeds, mortgages, releases, leases, notices, and demands for particular clients.163 This implies potential liability for any company marketing generative AI as a replacement for lawyers.

Early adopters of generative AI for this purpose are indeed facing peril. A company named DoNotPay marketed its software as “a robot lawyer” on behalf of customers,164 is developing apps that use LLMs to provide services like contract negotiations,165 and planned to provide an earpiece to allow a pro se defendant to rely on an LLM to present argument in court.166 The last of these led to warnings from “multiple state bars” of possible “prosecution and prison time,” according to the company’s owner.167 It is unclear whether any prosecutions have actually commenced, but at least two civil lawsuits alleging unauthorized practice have already been filed.168

Of course, if a licensed professional is involved, that professional probably may use generative AI to provide professional advice.169 At least in Colorado, the professional might not even have to be licensed in the same practice area as the advice is given. In Conway-Bogue, the Colorado Supreme Court ultimately permitted real estate brokers to practice law in the form of drafting documents because the brokers were properly licensed in their profession, limited the drafting of paperwork to customers who had hired them for real estate work, and charged no fee for this work.170 Could a realtor use ChatGPT to write those contracts? The scope of a professional’s ability to use generative AI to supplement their core profession with skills from other licensed trades has yet to be determined.171

Breach of Confidentiality

A business using generative AI to supplement its existing workflow rather than as a core part of the goods or services offered to its customers still must grapple with risks in using the new technology. Among these is the risk that using generative AI may endanger confidentiality.

At the moment, the most powerful and common generative AI programs are owned and operated by third parties like Google, Microsoft, and OpenAI. Few users download and run their own, local generative AI programs, though this is possible.172 If a business uses the generative AI of a large tech company, it has to send its prompts or other data to that company for the model to process. Businesses often deal in confidential information such as trade secrets, protected health information, student information, or customer private data. This has already caused problems for businesses allowing employees to use LLMs. Samsung employees copied proprietary code into ChatGPT to help fix errors.173 When this information is provided to generative AI, does it remain confidential?

The terms and conditions of some generative AI explicitly state that prompts and responses to the same may be used by the provider to help develop and improve the models or use the information in some other way.174 Models can continue to train and improve over time, adjusting internal relationships, with the result that the prompt and output can be “gobbled up” and used as “fodder for pattern matching . . . .”175 The model may subtly adjust the pattern of relationships that allow it to predict text to account for new prompts. Some commentators suggest that this process may work to undermine confidentiality, including trade secrets.176 There is some truth to this claim, because the definition of trade secret in Colorado requires that the information be “secret” and that the owner “took steps to keep it secret.”177 But the outcome is probably not that simple. Generative AI models do not retain perfect copies of all information from training. Rather, the models are subtly adjusted by each new training iteration. The precise prompt and response are not copied into the model so much as added to the mathematical matrix making up the weights in the model. Thus, disclosure to ChatGPT may not automatically destroy a company’s trade secrets, but it may undermine such protection.178

Businesses wishing to avoid this risk have a few options. There are already models available that are small enough to run on a local computer and could, in theory, operate without any disclosure outside of the local machine. Doing this, however, means the business loses the potential protections imposed by large LLMs on misbehavior. Moreover, the business would have to be careful to vet whether these models came from legitimate sources as opposed to improper leaks of intellectual property and are free of viruses or malicious code. Another option is to hire a company that has established its own instantiation of a generative AI model, independent of a public model, and has taken effective steps to prevent any injection of user data.179 It appears likely that if generative AI shows it is useful in a particular application, third-party services to accommodate this need and simultaneously preserve confidentiality will likely emerge.

Conclusion

Businesses are likely to experiment in the coming years with the best uses for generative AI and, while they do, judges and legislatures will be sorting out the legal implications of those uses. For now, caution is likely the best policy. If a business would not entrust a task to an entry-level intern, it should not entrust the task to generative AI. Rather, the software should be supervised carefully by a skilled employee and, where appropriate, a licensed professional. The software should not be overmarketed or overpromised, nor entrusted with any secret information. This will help avoid misbehavior and help ensure that the work product of the generative AI may best qualify for intellectual property protection. Where a business interfaces with the public using generative AI, it should ensure that those engaging with the software are provided disclaimers or user agreements to minimize the chance of misunderstanding or reliance. When advising business clients, attorneys should examine any use case with a clear view of these risks and advise clients carefully.

---

Explore This Issue

The Noble Profession

President’s Message

By Nathaniel Baca

Finding Open Access Legal Scholarship

Legal Research Corner

By Marla Morris

Simplifying Our Writing

Modern Legal Writing

By John Hiski Ridge

Related Topics

Notes