AI Flash Update: California Governor Decides Fate of Key AI Bills

We have seen a flurry of significant legislative developments in California this month related to artificial intelligence (AI). The California legislature, at the conclusion of its 2023–2024 legislative session in August, approved and sent several AI bills to Governor Gavin Newsom for his signature. The governor has now vetoed one bill (SB-1047) and signed into law several others. In vetoing SB-1047, the governor signaled that a “California-only” approach to regulating AI models may be warranted, “especially absent federal action by Congress,” leaving open the door to additional legislation.

The following provides a brief overview of some of the AI bills approved and vetoed by the governor.

The governor vetoed SB-1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, legislation introduced by Senator Scott Weiner that would have regulated large AI models trained using a certain quantity of computer power at a certain cost. The bill imposed specific compliance obligations on developers of large AI models and operators of computing clusters used to train these models. In addition, the bill established labor protections for employees of developers, contractors, and subcontractors, created a Board of Frontier Models within the Government Operations Agency to issue regulations on, among other things, thresholds for covered models and audit requirements, and authorized the California attorney general to bring civil actions for law violations.

According to the governor’s veto message, “[b]y focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 – at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.” The governor went on to state that “[w]hile well-intentioned, SB 1047 does not take into account whether an Al system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions — so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology. . . . A California-only approach may well be warranted – especially absent federal action by Congress – but it must be based on empirical evidence and science.”

The governor signed into law several AI bills that will impact the following entities and/or activities:

• developers of GenAI systems or services;
• providers of GenAI systems that have over 1 million monthly visitors or users and are publicly accessible within California;
• healthcare service plans or disability insurers, including specialized healthcare service plans or specialized health insurers, that use an AI, algorithm, or other software tool for the purpose of utilization review or utilization management functions;
• health facilities, clinics, and physician’s offices that use GenAI to generate written or verbal patient communications pertaining to patient clinical information;
• professional services agreements involving digital replicas;
• large online platforms and the posting of election-related content before and after an election.

1. AB 2013, Generative AI: Training Data Transparency requires that, on or before January 1, 2026, and before each time thereafter, when a generative AI (“GenAI”) system or service or a substantial modification to a GenAI system or service (released on or after January 1, 2022) is made available to Californians for use, a developer of the system or service must post documentation regarding the data used by the developer to train the GenAI system or service on its internet website.

AB 2013 requires that such documentation include a high-level summary of the datasets used in the development of the GenAI system or service, including, but not limited to:

• the sources or owners of the datasets;
• a description of how the datasets further the intended purpose of the AI system or service;
• the number of data points included in the datasets, which may be in general ranges, and with estimated figures for dynamic datasets;
• a description of the types of data points within the datasets;
• whether the datasets include any data protected by copyright, trademark, or patent, or whether the datasets are entirely in the public domain;
• whether the datasets were purchased or licensed by the developer;
• whether the datasets include personal information and aggregate consumer information;
• whether there was any cleaning, processing, or other modification to the datasets by the developer, including the intended purpose of those efforts in relation to the AI system or service;
• the time period during which the data in the datasets were collected, including a notice if the data collection is ongoing; and

• whether the GenAI system or service used or continuously uses synthetic data generation in its development.

2. SB 942, California AI Transparency Act requires a covered provider (i.e., a person that creates, codes, or otherwise produces a GenAI system that has over 1 million monthly visitors or users and is publicly accessible within the geographic boundaries of the state) to:

• Make available an AI detection tool at no cost to the user that meets certain criteria, including:
• that the tool allows a user to assess whether image, video, or audio content was created or altered by the covered provider’s GenAI system;
• that the tool does not output any personal provenance data, as defined, that is detected in the content; and
• that the tool is publicly accessible (however, a covered provider may impose reasonable limitations on access to the tool to prevent, or respond to, demonstrable risks to the security or integrity of its GenAI system).
• Collect user feedback related to the efficacy of the covered provider’s AI detection tool and incorporate relevant feedback into any attempt to improve the efficacy of the tool.
• Offer the user an option to include a manifest disclosure in image, video, or audio content created or altered by the covered provider’s GenAI system that meets the following criteria:
• the disclosure identifies content as AI-generated content;
• the disclosure is clear, conspicuous, appropriate for the medium of the content, and understandable to a reasonable person; and
• the disclosure is permanent or extraordinarily difficult to remove, to the extent technically feasible.
• Include a latent disclosure in AI-generated image, video, or audio content created by the covered provider’s GenAI system that meets specified criteria, such as: (i) to the extent technically feasible and reasonable, the disclosure conveys certain information (e.g., name of the covered provider and the time and date of the content’s creation or alteration), either directly or through a link to a permanent internet website; and (ii) the disclosure is consistent with widely accepted industry standards.
• Require by contract, where a covered provider licenses its GenAI system to a third party, that the licensee maintain the system’s capability to include a disclosure, as required above, in content the system creates or alters.

SB 942 prohibits a covered provider from:

• collecting or retaining personal information from users of the covered provider’s AI detection tool except as provided in the law;
• retaining any content submitted to the AI detection tool for longer than is necessary to comply with the requirements of the law; and
• retaining any personal provenance data from content submitted to the AI detection tool by a user.

This law does not apply to any product, service, internet website, or application that provides exclusively non-user-generated video game, television, streaming, movie, or interactive experiences.

SB 942 provides for civil actions by the attorney general, a county counsel, or a city attorney and makes a covered provider that violates these provisions liable for a civil penalty in the amount of $5,000 per violation.

3. SB 1120, Health Care Coverage: Utilization Review requires a healthcare service plan or disability insurer, including a specialized healthcare service plan or specialized health insurer, that uses an AI, algorithm, or other software tool for the purpose of utilization review or utilization management functions, or that contracts with or otherwise works through an entity that uses that type of tool, to ensure compliance with specified requirements, including, for example:

• that the AI, algorithm, or other software tool base its determination on specified medical and clinical information;
• that the AI, algorithm, or other software tool not base its determination solely on a group dataset;
• that the use of the AI, algorithm, or other software tool not discriminate, directly or indirectly, against enrollees in violation of state or federal law;
• that the AI, algorithm, or other software tool be fairly and equitably applied, including in accordance with any applicable regulations and guidance issued by the federal Department of Health and Human Services; and
• that the AI, algorithm, or other software tool’s performance, use, and outcomes be periodically reviewed and revised to maximize accuracy and reliability.

4. AB 3030, Health Care Services: AI requires a health facility, clinic, physician’s office, or office of a group practice that uses GenAI to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include both: (1) a disclaimer that indicates to the patient that a communication was generated by GenAI; and (2) clear instructions describing how a patient may contact a human healthcare provider. The  law exempts from this requirement a communication read and reviewed by a human licensed or certified healthcare provider.

5. AB 2602, Contracts Against Public Policy: Personal or Professional Services: Digital Replicas is intended to help performers and other individuals protect their digital likeness in audio and visual productions and requires contracts to specify the use of AI-generated digital replicas of a performer’s voice or likeness. Specifically, the law provides that a provision in an agreement between an individual and any other person for the performance of personal or professional services is unenforceable only as it relates to a new performance by a digital replica of the individual if the provision meets all of the following conditions:

• the provision allows for the creation and use of a digital replica of the individual’s voice or likeness in place of work the individual would otherwise have performed in person;

• the provision does not include a reasonably specific description of the intended uses of the digital replica (except that failure to include a reasonably specific description of the intended uses of a digital replica does not render the provision unenforceable if the uses are consistent with the terms of the contract for the performance of personal or professional services and the fundamental character of the photography or soundtrack as recorded or performed); and

• the individual was not represented:

• by legal counsel who negotiated on behalf of the individual licensing the individual’s digital replica rights, and the commercial terms are stated clearly and conspicuously in a contract or other writing signed or initialed by the individual; or

• by a labor union representing workers who do the proposed work, and the terms of their collective bargaining agreement expressly addresses uses of digital replicas.

A “digital replica” is defined to mean a computer-generated, highly realistic electronic representation that is readily identifiable as the voice or visual likeness of an individual that is embodied in a sound recording, image, audiovisual work, or transmission in which the actual individual either did not actually perform or appear, or the actual individual did perform or appear, but the fundamental character of the performance or appearance has been materially altered.  

6. AB 1836, Use of Likeness: Digital Replica makes a person who produces, distributes, or makes available the digital replica of a deceased personality’s voice or likeness in an expressive audiovisual work or sound recording without specified prior consent liable to any injured party in an amount equal to the greater of $10,000 or the actual damages suffered by a person controlling the rights to the deceased personality’s likeness.

7. AB 1008, CCPA of 2018: Personal Information amends the definition of “personal information” under the California Consumer Privacy Act (CCPA) to clarify that personal information can exist in various formats, including, but not limited to, all of the following:

• Physical formats, including paper documents, printed images, vinyl records, or video tapes;
• Digital formats, including text, image, audio, or video files; and
• Abstract digital formats, including compressed or encrypted files, metadata, or AI systems that are capable of outputting personal information.

8. AB 2655, Defending Democracy from Deepfake Deception Act of 2024 requires a “large online platform” to block the posting of materially deceptive content related to elections in California, during specified periods before and after an election. A “large online platform” is defined as a public-facing internet website, web application, or digital application, including a social media platform, video-sharing platform, advertising network, or search engine that had at least 1 million California users during the preceding 12 months.