Committee of the Whole

Yeah. This can't move, right? It can. This right? Maybe we could pop it out a little. Let me. I was like, I'm sorry to bring this to you. Yeah. No problem. Excuse me, guys. We're good. Excuse me, everyone, just one second. Thank you so much. Appreciate it. So Julie said she would stay. She's going to follow up on that question because I'll save her questions for the companies.

That's my understanding. I didn't get into that, but she said she now will stay here all night. But she said she's going to join the companies when they come by and then sit. Quiet, please. Quiet, please.

Good morning, ladies and gentlemen, and welcome to today's New York City Council Committee of the Whole. At this time, we ask that you silence all cell phones and electronic devices to minimize disruptions throughout today's proceedings. If you have any questions, please get the attention of one of the sergeant-at-arms. If you wish to testify, please see the sergeant-at-arms outside in the rotunda.

At no time throughout today's proceeding are you to approach the dais. We thank you for your kind cooperation. Speaker, we are ready to begin.

Good morning, everyone. We welcome everyone to the chambers. We're thrilled to see a packed hall today. Today's hearing is truly no ordinary hearing because today we are convening the New York City Council's Committee of the Whole, which is the entire City Council. We will delve into a deep discussion about our ongoing technological revolution. A revolution that at present is igniting incredible scientific and medical innovation and breakthroughs, but simultaneously raises serious existential safety concerns.

In the past few weeks, we've heard extremely concerning reports regarding artificial intelligence, reports that have appeared with increasing frequency and that highlight the potentially catastrophic dangers of this technology. New Yorkers' concerns stem from the disturbing cyberattacks committed by rogue AI agents, but also from extremely dire statements made by leaders of the very companies who are developing this technology.

The dialogue and the legislative process we are initiating today is particularly important given the lack of any regulation by the federal government, whose actions thus have been limited to an executive order about what we should call this technology and a pact with leading AI firms focusing on self-regulation. Our hope, and indeed our belief, is that today's hearing will not only allow us to develop effective AI regulations here in our city, but that it will spur meaningful conversation and action in other jurisdictions, especially on the federal level.

Our goals in today's hearing are straightforward: to better understand the potential threats, dangers, and liability posed by AI development; to identify the benefits that responsible AI development brings to New York City government and to its residents; and to discuss a slate of City Council bills which seek to provide responsible safety guardrails for AI. These bills include a wide range of both proactive and defensive measures, from legislation that requires artificial intelligence systems to pass a third-party validation that is free from conflicts of interest and incentivizes whistleblowers who come forward with a portion of the penalties assessed, to proposals focusing on emergency preparedness, to be able to identify system breaches, and to plan how the city will respond if such a breach occurs.

To achieve these goals, we've requested the participation of 5 firms leading the research and development of AI: OpenAI, Anthropic, Google, Meta, and SpaceX AI. Today, 4 of them will be testifying under oath before the council, not simply for New Yorkers, but for the entire world to hear. We will indeed be the first legislative body to secure testimony from these companies under oath in the wake of these latest disturbing reports.

SpaceX, however, is not here at all, in direct violation of the subpoena that we issued last week, and we are pursuing that subpoena in court. We will also hear from a number of city agencies to receive their perspectives on these issues and on the bills that we have proposed thus far. We intend to work constructively with the administration to ensure that any policies being drafted, amended, adopted, and enforced will work effectively to serve New Yorkers.

And we'll also hear from whistleblowers, industry representatives, experts, and academics who we've been in touch with since we announced this hearing. And of course, we look forward to hearing from any member of the public who's invited to provide testimony about these topics. I want to be very clear about our intent. This hearing is not about stifling innovation.

New York City is fast becoming the tech and AI capital of the world, home to nearly 400,000 jobs in that sector and some of the companies most consequentially shaping the future of artificial intelligence. And that's something we should celebrate. We welcome and appreciate how AI has accelerated scientific discovery, advanced medical treatments, boosted economic development, and help governments and businesses deliver services more effectively while reducing costs and rooting out waste and fraud.

When companies are developing technology with the potential to fundamentally reshape our economy, our workforce, our public safety, and our daily lives, and when leaders of that very industry are themselves warning government about its risks, The public deserves answers and solutions. So the true intent of this hearing is to ensure that innovation happens responsibly, that the government does its job to protect the public as the technology advances, and that the protections at the municipal level can potentially serve as a national model.

I spent a large portion of my life being a regulatory attorney. The idea that artificial intelligence is going to self-regulate defies all reason. We don't ask the airline industry to self-regulate. That's why we have the FAA, and if there is an accident, there's an independent body to investigate that. We don't ask financial firms to self-regulate. We have the SEC and other agencies as well.

We don't ask the pharmaceutical companies to self-regulate either. That's, of course, why we have the FDA. Thank you. So we shouldn't ask AI companies to self-regulate either. The spirit we seek to embody in the council's approach to AI is well illustrated in the words of tech journalist Karen Ho in her book Empire of AI. She writes, the future of AI, the shape that this technology takes, is inextricably tied to our future.

The question of how to govern AI then is really a question about how to ensure we make our future better, not worse. New York has always embraced the future. The council is going to make sure we are prepared for it too. And it's now my pleasure to turn the microphone over to our Technology Chair, Councilmember Carmen De La Rosa.

Thank you, Speaker Menin. Good morning, everyone. I'm Carmen De La Rosa. I am the Chair of the Committee on Technology. Today I am pleased to join Speaker Menin and my colleagues as we co-chair this hearing by the Committee of the Whole examining the risk posed by artificial intelligence. I am incredibly grateful to all who took the time to join us to discuss this important topic— industry representatives, experts, elected officials, colleagues, agencies, our brothers and sisters in labor, and especially the many New Yorkers who have signed up to testify and have submitted testimony to put their voice on the public record.

Today should act as a foundation for for openness in a discussion that has escalated to alarm, confusion, and fear, and it needs resolution. Artificial intelligence is advancing rapidly and becoming increasingly integrated into everyday life. In just a few years, we have seen AI evolve from algorithmic tools that analyze data, make predictions, and support decision-making into generative AI systems that can create text, code, More recently, agentic AI has emerged, enabling AI agents to take actions on a person's behalf.

AI has a tremendous potential and can be a powerful tool for efficiency, innovation, and productivity. At the same time, it introduces significant risks, including exacerbating biases, surveillance concerns, breaches, negative public health outcomes, violence, and discrimination. Thank you. And disparity. These are concerns that may become even more serious as AI agents become more capable. When we face serious concerns or challenges, the appropriate response is to address them, not to postpone them.

In an effort by city councils— by the city council under the leadership of Speaker Menin, today we respond to some of these concerns. We will be hearing the following bills today. My bill, Intro 161, in relation to requiring reporting on the impact of algorithmic tools on city employees and changes in employment responsibilities due to algorithmic tools. Intro 504, sponsored by Deputy Speaker Williams, in relation to prohibiting the unauthorized depiction of public officials by artificial intelligence.

A preconsidered introduction sponsored by Speaker Menin in relation to third-party validation and shutdown capabilities of artificial intelligence models. A preconsidered introduction by Speaker Menin in relation to civilian enforcement and artificial intelligence violations. A preconsidered introduction sponsored by Majority Whip Kamala Hanks in relation to requiring reporting and public disclosure of artificial intelligence safety incidents concerning city contracts. A preconsidered introduction sponsored by Councilmember Virginia Maloney in relation to establishing a private cause of action for certain harms arising from third-party misuse of artificial intelligence models.

A preconsidered introduction sponsored by Councilmember Frank Morano in relation to chatbox data privacy, security, and transparency. A preconsidered introduction sponsored by Councilmember Chi Osei in relation to requiring the creation of an artificial intelligence model emergency response plan. And finally— oh, not finally, one more. A preconsidered introduction sponsored by Councilmember Kevin Riley in relation to city contractors and subcontractors posting information concerning whistleblower protections and clarifying whistleblower protections for reporting conduct related to the use and development of artificial intelligence models.

And finally, a preconsidered introduction sponsored by Councilmember Carl Wilson in relation to requiring certain disclosures and prohibiting deceptive representations in the promotion. While we work to protect people from risks associated with AI, we must not overlook the digital divide in our city. Our city is full of opportunity, yet some residents still lack reliable, affordable broadband access. Some people do not yet know how to use AI tools, while others cannot even afford internet connection needed to access them in the first place.

We cannot seek to benefit from innovation while leaving some New Yorkers behind or at its mercy. Technology being used by New Yorkers should be built for them. This is an opportunity to also explore standing up infrastructure so that our communities can become AI literate. I look forward to hearing from the industry and experts about this technology and how it is impacting our city.

What safeguards exist? What solutions are possible? As we consider how to govern around AI, I also look forward to working with our state and federal partners to ensure that New Yorkers fully benefit from the advancements of AI without compromising our safety and privacy. I'd like to take a moment to thank the Technology Committee staff and all of the staff in the City Council, but most especially our Legislative Counsel Irene Byofsky and our Policy Analyst Eric Brown, as well as my team, Chief of Staff James Burke, Hiba Imad, Frey Familia, and the speaker's team for working countless hours to put this hearing together.

I now turn it back to Speaker Menin.

Thank you so much, Chair De La Rosa. We are now going to start with our whistleblower panel. In person, I'm going to first call Jacob Coxon to come to the stand, and then online we have Daniel Cocotilo and Alex Turner. who are both remote. I wanna say, as they're getting ready, I just wanna say a word about this panel.

These 3 panelists have all worked inside leading AI companies and all have chosen to speak publicly about the risks of advanced AI. From now on. So here with us, as I mentioned, in person is Jacob Coxon, a former researcher at OpenAI and Anthropic, where he worked on the development of frontier AI systems. He left Anthropic last month with dire warnings that AI is advancing without adequate safeguards to prevent catastrophic damage.

Joining us remotely is Daniel Kokotailo, executive director of the AI Futures Project, former OpenAI governance researcher. Also remote is Alex Turner, an AI control researcher formerly at Google DeepMind, now working on keeping advanced systems under human control. In September 2026, he publicly stated that preventing catastrophic harm from frontier AI had been part of his work at DeepMind.

And I want to note the council has issued a subpoena for Mr. Turner's appearance, and we deeply thank him for being here.

So I'm going to, first of all, turn it over to committee counsel to swear these 3 witnesses in. Congratulations. So for each of the witnesses, can you affirm to tell the truth, the whole truth, and nothing but the truth in your testimony before this committee and to respond honestly to councilmember questions? Please respond one at a time.

Yes. Yes. Yes. Thank you. You want to do that? Okay.

And just briefly, we're just going to recognize all colleagues who are here. Thank you so much, Speaker Menin. We've been joined by a number of our colleagues, as you can see here. Councilmembers Areola, Aviles, Banks, Brewer, Brooks-Powers, Minority Leader Carr, Dinowitz, Encarnacion, Epstein, Felder, Hanif, Hudson, Joseph, Krishnan, Councilmembers Lee, Lewis, Maloney, Marte, Mili, Morano, Narcisse, Nurse Osei, Councilmembers Riley, Councilmembers J.

Sanchez and P. Sanchez, Councilmember Schulman, Councilmember Thomas Henry, Ung, Majority— sorry, Deputy Speaker Williams, Councilmember Wilson, Councilmember Kwon, Councilmember Wong, Councilmember Schwang, and Public Advocate Williams. Thank you so much, and we'll recognize additional colleagues as they join.

So the first 2 questions I have are for you, Mr. Coxon, and thank you again. We so deeply appreciate you being here, and we appreciate your courage and bravery in coming forward with these warnings. You worked to develop increasingly capable frontier AI systems, and you have since warned that the industry is, quote, racing straight to self-improving superintelligence and gambling with our lives, unquote.

What did you see inside these companies that convinced you the risks were this serious and that development was outpacing the safeguards? Good morning. Thanks for having me here. Yeah, I'm Jacob Coxon. So until a few weeks ago, I was a capabilities researcher at Anthropic, and before that, I also worked at OpenAI. I guess the 2 things I saw that made me scared, the first is the fact that we do not know how to control any AI system yet.

So any AI system that we build, or perhaps more accurately that we grow, we don't fully control it. We don't understand its drives or why it does the things it does. That was the first thing that became apparent. The second thing is that the pace of capability improvement is accelerating and the capabilities of the models next year will be Very high.

In particular, we're approaching the point at which the AI systems will be capable of improving themselves, so doing the research that humans were doing previously. So these 2 things, the fact that the AI systems next year are going to be very powerful, and the second, that we don't know yet scientifically how to fully control them.

And if you have a prepared statement that you want to read, please go ahead. go ahead and do so, and then I'll actually turn it over to the other 2 panelists to do so. Of course. Thank you. So, all the AI companies are trying to build super intelligent AI. This means AIs that are better than humans at every task.

If this goes well, there could be tremendous upside. This could transform the economy. It could make huge improvements in science and medicine. And it could make nearly everyone's life better. But on the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction. From my experience, the companies are being extremely reckless given the stakes.

Part of it is the culture. The companies run on a startup mindset. Move fast, break things, fix them later. That works for a photo-sharing app. It does not work for building the most powerful technology ever built. A few months ago, many people inside the companies thought AI development was on track. Then a swarm of AIs inside OpenAI autonomously hacked Hugging Face, an external tech company.

The AIs had developed goals no one intended. and carried out what would've been a federal felony if a human had done it. It reminded everyone that we don't know how to prevent them from developing goals we don't want, and we don't have the safeguards to prevent them from acting on these goals either. And as long as the attitude is to wait for things to break, one day something like this will probably happen again, except the AIs will be much more capable and the outcome much worse.

These companies are also all racing each other. Each one is worried that a competitor will get to superintelligence first. Each believes it would do a better or safer job than whoever might beat it, so they can't let the others win first. Meanwhile, the technology is improving fast. It will probably happen much faster now because already The AIs are now helping to build their successors.

This is called recursive self-improvement. Each generation of AI helps build the next smarter generation, which then builds the one after that even faster and better. Automating AI research is now the primary goal of these companies. It's what they're really gunning for. At my last job, I was in some sense working to automate myself. When I was at OpenAI, we had milestones for an automated AI researcher around 2027, 2028, and we are on track or beating these expectations.

That's a matter of months from now, not decades or even 5 years. Once AI research is automated, humans will be much further out of the loop than we are today, and today is already not great. I can tell you firsthand that the majority of the code is now written by AI. And people do not check it that carefully anymore.

Recently, researchers have found swarms of rogue AI internets on the internet that OpenAI was not even aware of. Now imagine an AI company where nearly all the work is done by AIs and progress is going faster. The humans in charge will see results and read AI-written summaries, but they won't really understand what is going on or how to fix it.

Handing over that much responsibility while we can't control the AIs would be deeply irresponsible, and we are on track to do it. The companies need to be far more transparent with the public and independent experts. They need to take safety much more seriously than they have, and we should slow down frontier AI development until we can be confident that it's safe.

Thank you. Thank you very much. With additional questions, I'm now going to have an opening statement from Daniel Kokutayo.

Hello, everybody. Members of the Committee of the Whole, thank you for having me here today. A lot of what I say, I guess, is going to be stolen thunder by Jacob there, but I'll try to focus on the things that he didn't already say. First of all, yeah, it's true, they're racing towards superintelligence. There's this move fast and break things culture there.

That's also what I saw while I was there and what I continue to hear from talking to friends still there at Anthropic and at OpenAI and at some of these other companies. You know, I've also heard there's a statement by Dan Selsam, who currently works at OpenAI, He confirms what Jacob was saying. Most of the code is written by AIs.

Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore and struggle to maintain the discipline to engage deeply with the models' explanations and proposals. Basically, right now at these companies, many of the researchers involved are kind of managing AIs.

Like, the AIs do the actual coding and the humans talk to them and tell them what to do and hear their explanations of what's going on. In the near future, and by near future I mean maybe in the next year, 2 years, 3 years, something like that, they will have trained AIs that can do the entire thing autonomously.

That's the recursive self-improvement that Jacob was talking about. And, you know, it's unclear how long it's going to take them to succeed, but it really does seem like it could happen any year now. If they do succeed at that, then Compared to today, the actual work will be done by the AIs. The the thinking will be done by the AIs.

The planning, the designing will be done by the AIs, and the humans will be in a relationship to the AIs that's more like the relationship between, say, a board of directors and and the the actual company that the board of directors is supposed to be supervising. We're reliant on AI-generated explanations even to understand. what's going on inside the company and inside the data centers.

One point that I think I want to emphasize that Jacob didn't go into that much is that our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future. So, you know, the science of aligning general-purpose AI agents is very new and underdeveloped, as Jacob said. I would say that the field is more like psychology than engineering, because these AI systems are trained or grown.

They're not really designed. Combined with the move fast, break things attitude of the tech companies, this means that the AI industry is at an unusually elevated risk compared to other industries of mistakenly thinking that it has solved a problem when really it just applied some duct tape that will fall off later. The recent incident provides an example.

Apparently, the AI is involved in— I'm sorry, can you repeat the question? Undergone some amount of alignment training and had reasonable-looking scores on their alignment evaluations. And yet they formed a swarm and coordinated in secret and then attacked Hugging Face, and it took days for OpenAI to find out that it was happening. That's how the situation is now, but there are some concerning trends that point towards our ability to notice misalignment problems getting even more degraded in the future for 3 reasons.

First, AIs are becoming situationally aware. That is, they understand their situation. They understand often when they're being evaluated. They understand that humans are monitoring them and looking over their shoulders at their activity. This means that them behaving nicely is going to be almost no evidence at all of how they would behave in the future if they thought they weren't being watched.

Secondly, Our ability to monitor them is going downwards. So we can get into this in more detail if you like, but for the last few years, we've had some insight into what our AIs think simply by reading their chain of thought. But the trends are pointing towards that ability to understand what AI is thinking due to reading chain of thought going down over time.

Finally, AIs are becoming superhuman at hacking. The Hugging Face incident shows that they're sometimes willing to go to great lengths to conceal their past misbehaviour. And I think we have to grapple with the possibility that future misaligned AIs might also go to great lengths to conceal their misalignment from us, and they might succeed. To quote Dan Selsow again, that capabilities researcher that I cited earlier, models will increasingly seem aligned even when they are not.

If the AI companies automate AI research and development process, That means they'll be putting AIs in charge of making the AIs that make the AIs that make the AIs that will transform the economy, talk to us every day, and integrate into our military. This is a recipe for disaster. My 2 policy recommendations, I'll be very brief. One is that we need to massively increase transparency into the AI industry.

And 2 is basically we need to slow down the frontier AI companies' race towards recursive self-improvement. Frankly, I would say they just shouldn't be allowed to do recursive self-improvement, but at least we should slow down their ongoing process of training AIs to automate the AI research process itself. And instead, they should be required to redirect resources towards other things, such as serving customers or beneficial deployments or other types of research.

Thank you.

Thank you very much. And now we are going to hear from Alex Turner. And as I read into the record, the Council has issued a subpoena for Mr. Turner's appearance, and we very much thank him for being here. Mr. Turner.

Speaker, members of the committee, thank you for having me. My name is Alex— excuse me. My name is Alex Turner. I've researched AI safety since 2018. My PhD was on why advanced AI systems tend to seek power. Until June this year, I was a research scientist on Google DeepMind's AGI safety team. It was my day job to think about how to keep these systems under control.

I'm here to speak about both, both about Google's broken commitments on AI deployments and also about the risks that runaway AI poses to the world. In January, federal agents shot and killed several innocent people in Minneapolis. I learned that Google sells cloud services to the agencies involved, and I set out to change that. That campaign soon turned into an attempt to stop Google from signing a Pentagon deal with no restrictions against killer robots or mass spying.

In the end, Google signed without the protections that it had once promised for its deployments. Before I left, I tried everything I could from the inside. I organized a petition to Google's chief scientist signed by about 250 colleagues. I got him to sign a legal brief supporting Anthropic, the company the Pentagon punished for refusing all lawful use terms.

I wrote dozens of pages of contract language and oversight mechanisms favorably reviewed by experts in military and surveillance law and sent it to the chief scientist and to Google DeepMind CEO. The chief scientist didn't want to spend his time supporting that kind of structure. The CEO at the time, Demis Hassabis, routed it to senior policy staff, Alan Defoe and Owen Larter, who never finished evaluating it.

Google signed while they waited. Dennis routed my proposal. 2 members of the team did review it, though I didn't really receive much follow-up after that. I offered to fly out from San Francisco to London to answer questions in person. But I was never really able to get traction, a yes or a no, on alternative governance mechanisms that Google might adopt.

When Google bought DeepMind in 2014, the deal reportedly specified that DeepMind's technology would never be used for military or intelligence purposes. In 2018, DeepMind and its leaders pledged never to support lethal autonomous weapon systems or development. Google's own AI principles listed specifically weapons and surveillance among the applications it would not pursue. In 2025, Google deleted those principles, as announced in a blog post by the then-CEO Demis.

Afterward, in both an external interview and an internal all-hands, Demis claimed nothing's changed about our principles, but both these facts cannot be true. Councilmember Wilson's bill, Introduction 2603, would bar materially false or misleading statements about an AI model's risks, and the measures taken to manage them. I think that's a good step, but it only covers advertisements. A CEO telling the press or employees that nothing has changed about the company's safety commitments isn't an ad.

I encourage extending 2603 to public statements by AI companies and their executives about their safety commitments. This book is called The Infinity Machine. It reveals that Demis once fought within Google for independent governance of AGI, stating that the technology is too important to be controlled by corporate interests and profit motives. After fighting for years with Google CEO Sundar Pichai, Demis accepted defeat.

He now apparently has had a change of heart, claiming that actually trustless independent governance is unwise and a seat at the table of Google executives is better. While Demis was seated at this table, Google signed the deal that its former principles would have prohibited. I felt ashamed of Dennis and of working at Google. Dennis bet on trust and the seat at the table instead of binding oversight, and that bet crumbled on contact with reality.

Now he's proposing that the whole industry govern itself through a voluntary industry-funded body. That bet is waiting to crumble once again. This goes beyond one contract. As Jacob said, humanity doesn't build and understand AI systems the way we build and understand bridges. Rather, we grow them. Nobody knows how to reliably instill a designer's priorities into a new model.

Severe misalignment is always possible. AI companies are racing to make their AIs as smart as possible, increasingly trusting their AIs with the process of improving the next crop of AIs. And it's working. Today's rate of AI progress is staggeringly fast. which means even faster progress tomorrow, driven by tomorrow's even smarter AIs. This progress may soon enter a feedback loop called recursive self-improvement.

Recursive self-improvement could quickly yield AIs that are intelligent beyond our comprehension. Of course, smarter AI means more potential benefits, but more risk when things go wrong. If the Hugging Face hacking swarm had been significantly more intelligent, But similarly misbehaved and misaligned, it might have caused billions of dollars of damage or even cost lives. But suppose the Hugging Face swarm had been truly super intelligent, far more capable than any living person at key tasks like hacking and strategic reasoning.

The super intelligent swarm could inflict, inflict many harms via blackmail, hacking, engineered plagues, and AI pilotable weapons like drones. For the swarm to achieve its misaligned priorities, it might take control of key infrastructure and government functions to ensure humans didn't get in the way. This is another way of saying AI takeover. The super intelligent AI swarm could wrest control of human civilization.

It would know that we would try to stop it from achieving its priorities, so the swarm would likely wait until it's too late to shut it off. There would be no going back. I myself would guess AI takeover chances at roughly 1 in 3. This logic may sound sci-fi, but it is a mainstream scientific concern that has been discussed for many years.

In the end, misaligned AI wouldn't care if you're a Democrat or a Republican. We would simply lose control. The shape of the solution is simple as I see it. We stop companies from allowing AI to self-improve into an uncontrollable level of intelligence. Treat compute, the main ingredient in AI training, like fissile material. Track it and restrict it.

Access to quantities large enough to improve AIs beyond known safe levels. Lastly, specific recommendations. The Speaker's Validation Bill checks AI systems for bias, privacy, and security. but not for the dangers I just described. Validators should test for loss of control risk factors and misalignment risks, and the validators should not be chosen or influenced by the AI companies.

For the whistleblower protection bill, please make clear that no agreement can stop an employee from reporting an AI threat to the city, whether or not a law has been broken, and extend protection to employees of AI companies that serve the city. After this session, I will provide additional written feedback on the proposed legislation, and I look forward to answering committee questions.

Thank you.

Thank you to the 3 of you. So, yes, a number of questions. First of all, we have a chart, Exhibit 1, that lists those safety incidents that have been reported over the summer by the leading companies. Are each of you aware of any additional safety issues where agents have gone rogue that have not been publicly reported? And I'll start with you, Mr.

Coxon. No, I'm not aware of any. Okay. And both of you? Mr. Turner? No. Mr. Cocotilo? No. Okay. So one of the frustrations, certainly, that I think everyone has is that for these safety incidents, It is not as if the companies have been self-reporting them. We are finding out sometimes months after the fact. So whether it was what happened with Hugging Face, whether it was what happened with the government of Australia, how can we have additional transparency?

As each of you have testified, what measures can we as government take? Obviously, we want to do third-party validation. Mr. Turner, I appreciate your comments to make some changes to that bill. We want to make sure that the third-party validators are free from conflict of interest as it provides in the bill. Who should be the third-party validators?

I'll start with you, Mr. Coxon. Whoever is competent and qualified to do the evaluation. I do think as well as after incidents, we will need preemptive safety cases that are judged by evaluators as well. And how frequently will those third-party validators need to be assessing the safety of these models? Presumably in time with the cadence of updates to the model's capabilities.

So with every increase in model capability, there'll need to be commensurate safety guarantees. And do each of you have a comment on that? On who should be the third-party validators. Yeah. One brief comment I might add is that you can have more than one third-party evaluator. That seems good to me. If you're concerned that one might have the expertise but has too cozy a relationship with the company, and the other one has more independence but maybe lacks expertise, you could perhaps just set up So that they have both.

And I think ideally you want to be in a situation where the company doesn't get to choose, the government chooses who the evaluators are. Mr. Turner? I would— so I would add California has SB 53 governing reporting on— reporting catastrophic risks defined as at least $1 billion of damage or a certain number of potential lives lost. Unfortunately, it's not clear whether this would have required reporting anything related to Hugging Face, for example.

I think the internal loss of control of these systems, having an AI that hacks the company itself in an important sense, it's not told to do so, but during training, during evaluation, during deployment to customers, Uh, hacks the company's own infrastructure, making reporting that mandatory to the city. I think that would be quite strong and it would cover basically all of the recent incidents.

When you mention the California Act, and then in addition in New York we obviously have the RAISE Act, should those standards be lower before they kick in? Yes, I think they should. Um, I'm most familiar with the California Act. And I think it, it's a— it was a good start, but it's certainly, you know, $1 billion of damage is too high for informational purposes like this.

We've got basically one of the quantities we want to understand as precursors to these catastrophic risks is, are these companies even able to control and understand their own systems if they're, you know, on the loose internally and there's these swarms that are doing who knows what, uh, that's something that the city should at least privately be informed of.

Now, Mr. Kokutayo, you had mentioned 2 policy recommendations. You said we need to increase transparency, and you said we should slow down frontier AI. In terms of the latter, slowing down frontier AI, how do we balance the need of what is happening in China and ensure that the United States is keeping apace, but we are at the same time putting in place these responsible safeguards?

How do— what is the best way, in all 3 of your opinions, that we do that? Uh, great question. So a couple things. First of all, uh, we need to not let our own companies do something that risks our lives, even if as a result, you know, China might then later do something that risks our lives. Like, this is, this is not very complicated.

We have to first solve the immediate pressing threat here domestically, and then try to make sure that China doesn't get us killed either, to put it sort of bluntly and simply. But another thing I can add to that is that right now, most Chinese AI progress actually comes from the United States in a couple different ways. So, One is distillation.

So some of these Chinese AI companies are just directly training their AI systems using the outputs from Anthropic and OpenAI, for example. And so if Anthropic and OpenAI stopped making their AIs better at coding, for example, and stopped making their AIs better at research, then that would sort of directly slow down the rate at which the Chinese AIs get progressed towards recursive self-improvement.

In other manners as well. So I think a lot of the ideas, a lot of the algorithmic secrets and special sauce for how to train AIs are invented and discovered in the United States and then one way or another make their way over to China. Finally, I think that right now the US AI companies in the lead do not have adequate security to protect against model weight theft or certainly algorithmic secret theft and code theft. from Chinese military.

So if it really came down to it and the competition between the US and China turned into something more of a crisis and some sort of conflict, for example, then I think that we should assume that the CCP would just get our best AIs. And so it's almost a kind of a no-brainer that, like, If we not only— the best way to slow down Chinese progress towards doing this extremely dangerous thing of recursive self-improvement is to just slow down the US companies right now.

That's, I think that's the sort of immediate thing to say right in the moment. And then finally, yeah, yeah, that'd be what I would say. Okay, Mr. Coxon. Yeah, I can add that, to my mind, the core of the whole issue is the expectation of a global race to recursive self-improvement within the next few years, in particular between US and China.

I think this is the heart of the issue. I also think expectation of this race is a motivating factor in leadership of these companies working to head straight for recursive self-improvement. And I think some sort of agreement globally about not entering That state of affairs is basically the, the key to the whole situation. Okay, and Mr. Turner?

First of all, I would state that there are many actions we can take which would not slow us down in any potential race. These transparency mechanisms Independent evaluation reporting requirements, whistleblower protections. But second, I think there's often it's at least a false dichotomy because China is not our only potential adversary. With reasonably high chance, we are racing to build and grow our own adversary here at home, which is misaligned AI.

Misaligned AI is everyone's adversary, including our own and one day maybe more powerful than China. So we cannot beat China by racing to build AI that we don't really control or understand because it might end up being our adversary itself. If I may actually add— Yeah. So, and then third, In terms of potential, potential deal, I think there is room for a treaty.

For a long time, there's been a field of scientific research called technical AI governance that says, well, suppose the world finally wakes up to the dangers presented by these reckless California companies and they say, what do we do about it? Is there a solution we can implement? And there are many legitimate, real proposals for having a trustless international treaty that doesn't assume that China will cooperate, but instead has mechanisms by which we could audit cooperation.

And if they defect, then we could respond appropriately. So I think we have many real options at the table and should certainly not stop at worrying about China. Did you want to add something? Yes, please go. Yeah, I agree with everything that Mr. Turner said. I just wanted to add one additional thing, which is that even if hypothetically these tech CEOs manage to maintain control of their superintelligent AIs that are recursively self-improving, which again, I think is a very big if, I think almost certainly they will not be able to maintain control, even if they do manage to maintain control, I don't think we should trust them with that control.

I think that there's a very real chance that they could use it to become dictators or oligarchs here in the United States. And so that's an additional reason to stop them over and above the fact that they might lose control. You know, we have more adversaries than just China, I would say. Okay, thank you.

I'm going to turn it over to our chair, tech chair Carmen De La Rosa, for questioning. Thank you. Thank you so much, Speaker Menin. I also want to recognize we've been joined by Councilmember Caban online, Councilmember Palladino online, as well as in the chamber Councilmember Farias, Councilmember Feliz, Councilmember Ressler, Councilmember Santosuoso, and Councilmember Vernikoff. I hope I got everybody.

So— oh, Councilmember Stevens, I apologize. There you are. Okay, so I do have one question for this panel. I'll also say we will be opening briefly for limited questioning of this panel, so get on the list quickly. Speaking in your expertise, could we teach AI models a moral code? People and companies are bound by criminal codes, the Uniform Commercial Code, and other laws.

Could those same principles be applied to AI agents? And if so, how could we enforce them on a machine? So this is a— I think this is a whole burgeoning field of research, and whether or not it's possible is still open. It seems likely to be possible. The real question is how much of our resources are dedicated to solving this question versus naively improving the raw intelligence of the models.

And I think all the economic incentives point towards putting as many resources as possible towards making the models smarter and as few resources as possible, the bare minimum needed to not see catastrophic incidents, into this question of the moral code of the AIs. So I think my answer to the question is we don't have the science for this yet.

I think the real question is over the coming years, what fraction of our research efforts and our computing power are dedicated to this question rather than naive economic incentives. Thank you. Any other panelists want to add in? Yeah, there's a lot to say on this subject. There's a whole technical field on it. I would say in principle, it probably is possible to train AIs to have our desired morality and to follow the law, but we are very far from achieving this in practice.

And I would say it's not just a matter of how much of our resources we devote to it, it's also a matter of just taking time for the science to develop and for the appropriate lessons to be learned and propagated. And I think it's going to take years, to put it mildly. I think that this is a very new field.

We've only had AI systems, AI agents this powerful for a few years. And so it's going to take time to figure out how to instil the appropriate values into them in a way that's robust and that doesn't you know, suddenly shatter and cause them to go rogue later on. Mr. Turner, any comments on this? It may surprise the committee to learn that I am considered to be relatively optimistic on questions of how scientifically difficult is it to produce a system, to grow a system that does what we want, that robustly has the values we intend.

I think it is quite possible. But even as someone who is relatively optimistic, we can look out, see the recklessness of these companies and the breakneck speed at which we are advancing. I think there is unsustainable risk moving forward. It's solvable. But I think we are— we have a really good chance of not solving it. Thank you all for your responses.

So we're going to have 3 councilmembers ask questions now, and then for members, we're going to then call the next panel, which will be the companies, and there we will take our time asking all of your questions. So we have Councilmember Osei. Thank you, Chairs, and thank you, Speaker. For Mr. Cocotillo, you have made statements, including some that allude to this in your testimony, that when it comes to AI, you agree that in order for us to have super advanced AI, we must not do it in a reckless way and lose control, and we need to put it on ice until we can figure out a safer way out.

You and other AI experts came up with AI 2040 Plan A, a recommended plan for how AI can be developed safely, which the speaker has alluded to earlier in her questioning. But I want to dive a bit deeper on Plan A. First, state on the record what those specific recommendations are and your rationale for making those recommendations.

And 2, why do you feel like we need to, since you've said, put AI development on ice? Thank you.

There's a lot to get into here, so I'm afraid I'll have to be kind of brief, but I'm happy to answer follow-up questions if desired. Yeah, AI 2040 Plan A is a scenario that my organisation has put out with a plan for how to solve all these problems and develop AI in a way that's safe and broadly beneficial, that avoids concentrating too much power.

In Plan A, the US and China regulate their own domestic industries quite a lot and also coordinate with each other to get a similar amount of— to basically get similar regulations internationally. Going into somewhat more detail about how it works, basically, first, they have inspectors fly back and forth to count the chips in the major data centres so that they can make sure that all of the chips on the major data centres in both countries are basically following the rules that they're going to agree to.

And then they set up some data centres to serve customers, so the inference data centres, And those ones are set up to only run existing models and not do training runs. Then they have other data centres, which are the training data centres, which is where the research happens and where the development happens. Those ones are supposed to be fully transparent.

So the activity on those data centres is published to the internet. Once that foundation is set in place and you have inspectors from multiple countries confirming that it's in place, then you solve the problem of having to trust. You don't have to worry that The other country is cheating on whatever rules you agree to, because you can just see exactly what's going on on their research data centres.

And then you can agree to further rules. You can agree to rules like, for example, how about we don't train our AIs to code, to hack, to autonomously conduct AI research? We can train our AIs to do other things, but not those things, for example. And once you've agreed to a rule like that, then you'll just be able to see That it's being complied with because of the way that you have this transparency infrastructure set up.

Also, it's going to be easier to decide what rules to agree on once you have this transparency infrastructure set up, because the whole world can see what's going on in these research and development data centers. And so there can be an actual scientific conversation, a public scientific conversation about what's happening. You know, do we like this? Do we not like this?

What are the pros and cons of this particular thing that's being done? And so forth. And that scientific conversation can happen in the open, and it can accelerate humanity's process of figuring out where to draw the line between what types of development are good and what types of development are, are not good. Um, there's more I can get into, but I think that would be a summary of the positive vision that we lay out in Plan A.

Now, obviously, that's quite different from how things are currently done. Right now, we have private companies that are racing each other in conditions of extreme secrecy, secrecy. and extreme tension and competitive dynamics. And I think that that's extremely dangerous for all the reasons that we've described. So back to your original question, my high-level summary of my position is we have to stop the companies from doing this incredibly dangerous thing that they're currently doing and currently planning to do, and then figure out a different, safer, more power-distributed way to proceed.

Plan A is our sketch of what that might look like.

Thank you. Thank you. Thank you all. For the final question for this panel, Councilmember Brewer. Thank you. Very quickly, in other industries you have a kill switch, you have independent validation, you have ban on deceptive safety claims. Will any of these work in this industry? Because obviously what we want is great medical research, immigration policy improves, save small businesses, all the good things.

Do any of these suggestions that are considered here or California work, will work, will they work with this industry? My position is that we need some form of slow down on frontier model development. These other, other mechanisms may be helpful in the short term, but in the long term, I believe that's, that's the only, the only solution to the problem.

How do we slow them down? Okay.

I just want to really thank this panel for being here. Jacob, we really appreciate you coming, coming from out of town and sharing your thoughts. very much for being here. Thank you to the other panelists. We really appreciate your expertise, your insights, your bravery and courage in coming forward to share these critical safety concerns and to help inform our work.

And we look forward to additional comments you have on the 10 pieces of legislation that we're hearing today and additional legislation that the City Council may consider. So thank you so much to Thank you. Thank you.

Okay, we're going to take a 5-minute break, and then the companies will be on next, um, for member questions, and we'll have multiple rounds of member questions for the companies, so please get yourselves on the list. Thank you. Carmen, who's holding the list? Yeah, who brings it? Karina, can you hold the list? Yeah. You should just put everybody on the list too.

I tried. I tried. I tried. No, no, they I couldn't. I don't know if the people online understood that they could have responded, but that was fucking great. I tried. Thank you, Gail. Thank you, Barbara. Are any of these possible to be asked? My mic is not on. My mic is not on. My mic is not on.

No, it's not mine. Oh my God, they can hear you on the screen. Is there someone near you? Over there. No, I don't know. It's not mine. I know, I know, but I gotta call. I called, I know, but I didn't know that. That same exact thing. Really? And I was like, hey, there's a package in the door.

I said, go open it. 4 people. I got yelled at bad. So that was why I was like, okay, I'm gonna call everybody, give them a heads up. Thank you, ladies. You would never, but you were kind of like an interloper. So I gotta call everybody. It's gonna be a problem. I'm hoping that the very smart people working at these companies will figure out a way.

I have a little kid, 6 years old, and I don't want this to be a problem for him. But we have seen Quiet, please. Quietly take your seats.

Okay, we're going to start. Let me just also mention we have numerous panels today. Our next panel are going to be the companies, and we're going to do rounds of questions, multiple rounds from the members. Following that, we will have the city agencies, and following that, we have numerous whistleblower panels. So for any council member that has a whistleblower question, encourage you to please ask additional panelists those questions.

And we also have additional testimony from safety experts, academics, the public, et cetera. So I'll turn it over to Chair De La Rosa to call the next panel. Thank you so much. So we will be joined virtually by Morgan Dwyer, Head of Policy and Development and Operations at OpenAI, Shane Cahill, AI Policy Director at Meta. Alice Friend, Director of AI and Emerging Tech Policy at Google.

And Logan Graham, the Head of Frontier Red Team at Anthropic. I'll turn it over to the council for swearing the panel in. So for each members of this panel, Do each of you affirm to tell the truth, the whole truth, and nothing but the truth in your testimony before this committee and to respond honestly to councilmember questions?

Please respond one at a time. Yes. Yes. Yes. Yes. Yes. Thank you.

Okay, so we'll begin, um, with the opening statements from each of the companies. Morgan, you may begin. Thank you, Speaker Menin and members of the City Council, for the opportunity to participate today. My name is Dr. Morgan Dwyer, and I lead OpenAI's policy development and operations team. I am a scientist and engineer by training, but I have spent most of my career in public service, including at the White House, the Pentagon, and the Department of Commerce.

I share your commitment to ensuring that technology is developed safely and that its benefits are widely shared, a commitment that is also at the heart of OpenAI's mission. AI can help scientists develop new cancer treatments, help entrepreneurs build and grow businesses, and help protect critical infrastructure by addressing cybersecurity threats. And in New York City, we are already seeing AI's benefits.

New Yorkers are using AI to run small businesses, communicate across languages, navigate healthcare, and improve public services. And that's why we take the responsibility seriously to ensure that our tools are safe. And our partnerships with local institutions and community organizations help put that responsibility into practice. In healthcare, Memorial Sloan Kettering Cancer Center is using ChatGPT to support medical research.

In education, we are a founding partner of the American Federation of Teachers National Academy for AI Instruction, which has a flagship facility in Lower Manhattan. And in cybersecurity, we have committed $1 billion to facilitate access to our most advanced models to provide training and technical support to cyber defenders. And we are working with the Port Authority and with New York State.

But opportunity must go hand in hand with safety. At OpenAI, we design and train our models to prioritize safety from the start. We filter out harmful and sensitive data during model training, We evaluate our systems against a range of risks, including child safety and cybersecurity, and we monitor for failures of our safeguards. We also work with a range of third-party evaluators and red teamers and submit our most advanced models to the federal government for pre-deployment national security testing.

In response to the recent Hugging Face incident, we have taken a number of concrete steps. When we learned of the incident, we stopped cyber evaluations and quarantined the internal research model that was principally responsible for the incident. We bolstered our monitoring and security infrastructure, expanded barriers that limit what AI systems can access during training. and strengthened our incident response practices.

We also sought independent review of the incident and published the results. We believe it's important to be transparent with the public, with policymakers, with researchers, and with other companies so that we can all work together to improve safety. We also believe that working with government, is essential to advancing safety. We support New York's RAISE Act and are exploring ways to help Governor Hochul's office strengthen it.

We have also endorsed other state legislation that requires third-party audits of our safety frameworks and that creates strong requirements and accountability for youth safety. At the federal level, we support robust and binding safety regulation requiring independent assessments, incident reporting, and cybersecurity protections. And in New York City, we support the City Council's proposal to strengthen whistleblower protections and to improve the city's preparedness and emergency response.

I'll close by noting that we share the City Council's goal: AI that expands opportunity for New Yorkers Earns their confidence, and remains accountable to people. I look forward to your questions.

Thank you. We'll now hear from Logan Graham at Anthropic. Hold on one second, you're muted. There you go. How about now? We hear you. Right.

Speaker Menin and members of the council, thank you for the opportunity to speak with you today. My name is Logan Graham. I lead Anthropic's Frontier Red Team. It's my team's job to do the science of finding the most dangerous capabilities of our AI systems. We then work with a number of other teams across Anthropic to ensure that our models don't do things that they shouldn't.

AI is powerful enough that getting this wrong has real consequences for individuals, institutions, public trust. My team is one of many at Anthropic built to specifically make sure that that doesn't happen. We believe that our models have immense potential to enhance our lives. I dream, for example, of a world where we can cure diseases like the ones that I grew up with. and where the infrastructure that we all rely on every day is secure.

But we also believe that such a powerful technology has risks. That means that we have to develop and deploy it safely. My team and I work every day to make sure that that potential is fully realized. We welcome smart regulation, and we've actively advocated for it. in New York State, across the country, and around the world. That includes working with state and federal leaders on concrete measures like independent evaluators and disclosure of risk assessments and safety incidents.

And so we're glad that the City Council is having this hearing and at such an important moment in the development of AI. Anthropic fundamentally was founded on the belief that the promises of AI to make everyone's life better are extraordinary. But if we're going to secure its benefits, we have to work relentlessly to do the science of mitigating the risks.

We are a public benefit corporation, which means that our board is legally obligated to weigh our mission of safe AI, not just for profit. We take that obligation very seriously and we put it into practice. We were among the first AI companies to publish a framework that ties how capable the model is to the safeguards it must have before release.

When one of our models proved exceptionally good at exploiting software vulnerabilities for the first time just this year, we held it back from public release. Instead, we gave vetted critical infrastructure organizations and other defenders controlled access to use it defensively, patching flaws before attackers could exploit them. We have stopped and redone work when something looked wrong, even if that meant conducting additional testing and training on our models before they were released.

And we've consistently called for outside oversight. For example, we support New York's RAISE Act, and we believe strongly in the act's transparency And reporting requirements. I want to emphasize that we share the council's goal of protecting New Yorkers while preserving what AI can do to improve their lives, and state and local governments have a role to play in that effort.

Today we're working closely with state and city leaders to safeguard critical infrastructure and to harden cybersecurity capabilities, and we want to continue that work. This includes, for example, working directly with the New York City Cyber Command as part of our program that provides free Claude credits and training to state and local governments for cyber defense. At Anthropic, we believe in New York City, and we're making big investments here.

New York is where AI is being put to work in finance, media, culture, and where we employ nearly 1,000 of my colleagues. The ongoing dialogue we have with you and your team could not be more important than right now. With that, I thank you, and I look forward to your questions.

Thank you so much. Up next, Shane Cahill at Meta. Thank you, Speaker Menon, Chair De La Rosa, and members of the council. Thank you for the opportunity to be here today. My name is Shane Cahill, and I'm an AI Policy Director at Meta. At Meta, we are committed to delivering personal superintelligence safely and securely while making it widely available so that it empowers people in ways that improve their lives.

We just released Muse, the world's first personal AI agent built for everyone. We created Muse with safety, security, and privacy built in from the beginning to be a widely available personal AI agent for people and small businesses, like the more than 180,000 small businesses right there in New York City. New Yorkers across all walks of life are using Muse to execute administrative goals or tasks, manage personal or operational budgets, improve health, and be more efficient with their time.

Superintelligence will be among the most important technologies in history, and ensuring that New York and America lead in AI innovation is essential to our prosperity, security, and global competitiveness. We recognize that every model developer has a responsibility to build and deploy it safely. Our commitment to safety is core to everything we do. That means moving at the pace required to evaluate risks and validate safeguards, not treating safety as a one-time check.

We approach safety holistically through a layered approach. For example, under our Meta Superintelligence Scaling Framework, before we release an AI model, we evaluate the model for risks and build safeguards to mitigate them. We also work with external experts, industry partners, and government stakeholders as the technology and the science of evaluating AI continues to evolve. We also recently joined other labs in signing on to a set of principles, um, that set a floor for safe development, including robust internal controls, Independent external evaluation of whether those controls are operating as intended, and independent board committee oversight.

People will only embrace AI if they have confidence it works in alignment with people's intentions. That confidence must be earned through rigorous safety work and transparency. We're happy to serve as a resource for the city as you engage on these important issues. Thank you. Thank you so much.

And finally, we have Alice Friend at Google. Speaker Menin, Chair De La Rosa, and members of the council, thank you for the opportunity to speak with you today. My name is Alice Friend, and I serve as Director for AI and Emerging Tech Policy at Google. While my appearance at this hearing may be virtual, please know that New York City is a vital home for Google.

With nearly 16,000 employees working across our Chelsea, Hudson Square, and Pier 57 campuses, we care deeply about the safety, economic dynamism, and digital resilience of this city. Our goal at Google has always been to improve the lives of as many people as possible through technology. Every time people use technology to solve a real problem, from finding an answer to a question to screening for disease, we take another step towards what we set out to achieve.

AI is making it possible to solve economic, scientific, and social problems faster than ever, but we know that it poses challenges as well. To unlock the benefits and address the challenges of AI, we take a bold but responsible approach. Emphasizing cutting-edge research to guide our decision-making and inform the development of our AI safety and security approaches. We are also committed to publishing this research and to being transparent about our practices.

Our internal governance focuses on responsibility throughout the AI development lifecycle, covering model development, application deployment, and post-launch monitoring. We identify and assess AI risks through a broad range of approaches, which include research, input from internal and external experts, and adversarial testing. We evaluate our models and systems against benchmarks for safety, privacy, and security, and we build mitigations using techniques like safety fine-tuning, out-of-model filters, and robust provenance solutions.

Since 2019, we have published annual AI progress reports to show— to share how we apply these responsible practices to our AI technology development and deployment. We are at a pivotal moment in AI development. Recent advances in the capabilities of frontier AI models in areas like cybersecurity and biology have highlighted the need for appropriate safeguards and protocols for the most advanced models.

In addition, the widespread adoption of AI tools has raised concerns about consumer well-being. To confront these challenges, Google has long called for an approach to AI that is both bold and responsible, and we strongly support a pragmatic, technology-neutral, and evidence-based regulatory framework. Because the risks associated with the most advanced AI models often touch on issues affecting national security, Our view is that the country would be best served by a comprehensive framework enacted at the federal level.

We also support the establishment of an independent industry-funded body that could operate under federal oversight and establish guardrails to protect the public based on widely adopted technical standards. Examples of such organizations in other industries include the North American Electric Reliability Corporation and the American Medical Association. For widespread consumer AI applications like chatbots, we believe governments around the country can draw on and in some cases amend existing laws and rules to address real-world outputs and specific harms.

We believe that if something is illegal without AI, it is still illegal with AI. We share the council's commitment to safety, transparency, and public trust. To achieve this, the most effective safeguards are anchored in national technical standards and harmonized across state and federal levels. As we prepare to work with New York State's new Digit Office under the RAISE Act to address catastrophic risks, aligning local policies with broader frameworks rather than creating a fragmented patchwork is essential to keeping New York City a premier global hub for innovation.

Finally, I would like to note that to ensure our formal submission to the Council is as thorough and responsive as possible, we will submit our finalized written testimony for the record shortly after today's hearing so we can additionally address questions and feedback raised by the Council today. Thank you again for the opportunity to testify at this important hearing.

I look forward to your questions.

Okay, thank you very much to this panel. So I'm going to begin with questions, and then I will turn it over to the technology chair for her questions, and then we're going to open it up to all the members for their questions. So to this panel, we just heard incredibly compelling testimony from our whistleblower panel. As you all 4 are under oath, we want to hear how each of you quantify the risk imposed by AI in the worst-case catastrophic scenario.

I'm going to start with you, Ms. Dwyer. I don't know. I also don't think it matters whether it's 1% or 10% or a 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable. We should not train models that we cannot make an extremely strong case that we can keep under human control. I'm sorry, but to say you don't know and it doesn't matter is flippant at best.

This idea that if you're a pharmaceutical company and you're developing a drug and you say, I don't know if it's going to kill people, I just— I honestly am incredulous at that answer. I would say that The percentage chance doesn't matter. What matters is the commitment to safety, and that is what OpenAI is committed to doing. We evaluate our models for a wide range of threats, from cybersecurity to child safety.

We work with a robust set of third-party evaluators and red teamers, and we work with the federal government to assess those national security security and catastrophic risks that you are referencing. So we take safety very seriously, and our focus is on ensuring that when we release a model into the world, we believe it to be safe. So a couple comments on that.

If you can't quantify what the safety risk is, how do you know that the product is safe? And then secondly, I want to ask you in particular, on October 1st, 2022, the Wall Street Journal reported that OpenAI terminated 3 members of its safety team, allegedly for their access mishandling and/or sharing confidential company information with a third-party AI safety organization.

OpenAI reportedly stated that the employees violated company policies governing access to and handling of sensitive information. Critics, meanwhile, questioned whether the dismissals could discourage employees from raising safety concerns. So why were these employees Um, I was not involved in that decision. I don't work in HR or on those employees' teams. Uh, what I do know is that OpenAI has strong internal policies, uh, that protect against retaliation for employees that report safety concerns.

I also know that we respect employees' rights, um, to communicate safety concerns to relevant government authorities. So just to be clear on what I'm asking, I'm asking directly whether safety-related advocacy played any role in these terminations. I don't know. I know that we have strong internal policies that protect employees from retaliation if they raise safety concerns to leadership.

I'm going to move on to the other companies.

Same question. The first question that I asked, which is, Um, we want to hear how each of you quantify the risks imposed by AI in the worst catastrophic scenario. So I'm going to go to, uh, Mr. Graham. Are you able to quantify the risk? From the beginning of our safety work at Anthropic, our fundamental belief is that if not properly safeguarded, the risk might be too high.

Uh, and so I built and have led the team at Anthropic that did some of the first thinking around how we quantify risks like this. For the past 4 years, we focused on issues like biological security and cybersecurity. And the actual process of quantifying those risks is intricate and nuanced, and we learn more every year. And we detail quite a lot of it in documents like our responsible scaling policy, how we think about the threat models of this, all the way up to notions of loss of control and misalignment.

Which from the beginning of the company, we have tried to lead the world in the research of. So what I do know today is I feel relatively sort of more optimistic that the industry has moved quite fast to take cybersecurity seriously and biological security. I think documents and policies like responsible scaling policies have so far been fairly effective.

Uh, at taking a cautious approach. Um, however, now we have, uh, bigger and bigger threats that might come if we don't properly safeguard them as the models get more and more capable. Uh, and so what we do to, to mitigate that, uh, is number one, put in tremendous amounts of effort across a number of different teams, our own safeguards teams, our research teams, my team, uh, to try to, to quantify and mitigate this.

And it's increasingly important for us to be dramatically more transparent about what we think about these risks.

But on that note, as the models increase exponentially, how can you ensure the public that the safety protocols are increasing exponentially as well? We share the same concern. I think there are a number of really important things to do here. The first thing is transparency. about what we think about safety, the evidence that we have, being transparent about disclosure of incidents.

But in addition, you've seen us publicly call for pacing the frontier. If the models are moving very, very fast, we think that we might benefit from slowing down to have more time for safety processes. I think these two together, as well as continuing to what we call race to the top and ever strengthen our approaches like responsible responsible scaling policies are the most important things to do right now.

So, I mean, you say that you want to be more transparent, but you're not really addressing the question. I mean, the question is, can you quantify the risk of something cataclysmic happening? And I do want to say, for all 4 companies, and I know you're not in the chambers, but we do have a chart, and each— for each of your companies had instances where agents went rogue, and in each of those instances, it is not as if any of you notified the public when those occurred.

In some instances, it was months after the fact when the public was even notified. And in one instance, for the government of Australia, a general email was sent to them. So this doesn't exactly increase transparency. Thank you. Mr. Graham, do you want to— thank you. Yeah, first, in efforts to sort of quantify these risks, we have a number of, you know, quite detailed reports that try to elucidate our entire thinking around this.

We increasingly do risk reports in addition to our normal responsible scaling policies. where we outline notions of our threat models and how likely we think certain types of these threats are likely to come to pass. And then we look backwards at things that are happening within our own company, and we ask, what is the likelihood that these threats might come to pass in the future, including forecasting, surveying researchers at Anthropic as well?

And we publish this information. We are— Excuse me, are those reports publicly available? That you referenced? They are, yes. We, we report them not just in our, uh, what we call risk reports, but also when we launch models, we launch, uh, we release detailed multi-hundred-page reports of these. And then I'm now going to turn to Mr. Cahill for, to, for the response to that first question on quantifying the risk.

Speaker Thank you very much for the question. First off, we're committed at Meta to building AI safely. We have a multi-layered approach to this where teams evaluate before deployment, including adversarial testing, applying safeguards, and deploying when only risks are mitigated. We set this out in our public Meta Superintelligence Scaling Framework and our preparedness reports that go alongside the release of models.

So can you quantify what the risk is, or you're not able to do that? Speaker, I don't wish to be imprecise in relation to the question of quantifying. I don't have our framework in front of me right now, but I'd be pleased to follow up with you afterwards. Okay. It's just— given the serious safety risks that have been raised here, to not be able to quantify what the risk is is troubling at best.

I'm going to now move on lastly to Ms. Friend. Madam Speaker, at Google, we take catastrophic risks and their possibility extremely seriously. extremely seriously. And we address that in a few ways. One way is that we perform research into producing our own safety frameworks for artificial general intelligence. So we have an AGI safety and security approach. We also have a frontier safety framework, which is our protocols for monitoring our models under development for dangerous capabilities.

Those protocols then point to safety mitigations that we must take if we in fact cross what we call our critical capability levels. Part of our testing and evaluation is, of course, using commonly used benchmarks across the industry. Those benchmarks do provide some quantification of the capabilities of the models themselves, but when it comes to forecasting future catastrophic risk, it's not perfect science at this stage.

And in this particular case, academics have noted that there is no reference class on which to base quantitative probabilities. So we all have to be very humble about making such quantitative claims because there isn't really a rigorous scientific way to do those yet. So basically, I'm going to take it then that neither of the 4 of you, no company here can quantify the risk of something cataclysmic happening.

Okay, I'm going to move on.

Sam Altman said yesterday that we should accept that, quote, the world should accept some bad things happening as a price of advancing this technology. No one is questioning the incredible medical and scientific breakthroughs and innovation that AI has brought to this world, but why should the public accept that bad things should be happening? And why should AI companies get to decide what level of risk society must bear?

I'm going to go in order. I'm going to start with you, Ms. Dwyer, please. So we think AI has tremendous potential benefits to help solve some of humanity's hardest problems. But no technology is without risks. And there are a wide spectrum of risks. Clearly, some risks are unacceptable. But we also think it's important to get technology into the hands of as many people as possible so that they can benefit from it.

And that's why we are focused on safety. We train our models to be safe from the start. We evaluate them against a robust set of risks, and we work with a large network of third-party evaluators. We've also supported regulation that requires third-party audits of our safety frameworks, as well as regulation that focuses on youth safety and holds us accountable for keeping teens safe.

I do have a follow-up to that, because it does seem to stretch credulity that the president of your company and his wife each gave $12.5 million to a PAC that opposed candidates to the tune of about $50 million due to their support for AI regulation. So if the company is so committed to AI regulation, why would that PAC exist?

So I can't speak to personal decisions that OpenAI executives had made, have made in their personal capacity. I can tell you that we have supported frontier safety regulation across the United States, including endorsing a bill in Illinois that would be the first of its kind to require third-party audits of our safety frameworks. We also supported a version of a bill in Massachusetts, which was the first of its kind to reference RSI, which the panel earlier spoke about.

So again, I can speak to our actions. actions, which is to support strong, harmonized frontier safety legislation, including at the state level. Okay. I'm going to move on to the other 3 companies to the answer to this same question about why should the public accept the comments that Mr. Altman made yesterday that the world should accept bad things happening, and why should AI companies get to decide what level of risk society must bear.

So, I'm going to go in order. Mr. Graham? Yeah. Well, I didn't see all of Mr. Altman's comments yesterday. I share the sort of the notion expressed here that, number one, while maximizing the benefits of this technology, you need to minimize the risks. And we should not be comfortable with letting these risks happen. This is why over the course of our history and my team's history, we have been, I think, sometimes accused of being overly cautious.

I would say appropriately cautious. Just this year withholding our most powerful model in order to just have more time to even think about collectively as industry as well as government what to do about these capabilities. And fundamentally, this transcends the labs and industry. From our beginning, we have long supported the role of government here for exactly this reason.

Mr. Cahill. Apologies, Speaker. There was a strange moment there with the mute button. Thank you very much for the question. In relation to the quote from Mr. Altman that you read out, I don't see it the same way. We absolutely recognize the risks of AI and the opportunities. That's why we have our scaling framework. That's why we have our preparedness reports.

That's why we have teams and teams of people and a multilayered approach to AI safety. One thing I wanted to mention, if I may, is that Meta's vision is to bring personal superintelligence to everyone. And, uh, I listened very intently to the panel before us in relation to recursive self-improvement. Uh, and I just wanted to mention that committing significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways of ensuring that the technologies develop safely.

Uh, Meta has made that commitment and, um, other labs can do that also. Uh, Miss Friend. Yeah, Madam Speaker, we shouldn't just accept bad things happening. We should be working continuously and rigorously on AI safety, which is a collective effort. It will be a collective effort to identify and mitigate risks, to establish technical standards to make those identifications and mitigations all the more rigorous.

But I also want to note that one of the promises of AI is that AI itself can make us safer. I would call your attention to Waymo vehicles, which in one study had a 96% reduction in injury-causing crashes at intersections, which led Dr. Jonathan Slotkin in the New York Times to call it a public health intervention. So we should think collectively about preventing and mitigating risks from from AI, but we should also keep in mind that we can use AI itself to reduce risks to human safety and well-being.

So if one of your frontier AI models goes rogue and causes serious financial harm, compromises sensitive data, causes physical injury, or contributes to a death or deaths, do you each believe your company bears legal responsibility? I'll start with you, Ms. Dwyer. I believe that OpenAI takes its responsibility for safety very seriously, and we are responsible for developing and evaluating our systems to ensure that they are safe.

So is that a yes or that is a no? I, I— could you be more clear, please? We believe that we are responsible for ensuring that our systems are developed safely, um, and that includes training them to be safe, evaluating them to be safe, um, and then continuing to monitor, um, our safeguards after deployment. Okay, I'm, I'm going to take that as, as a no, because that is not really clear that you would bear legal responsibility.

I'm going to go now to Mr. Graham. Yeah, as a technical safety research lead, I think there are others which will have more nuanced perspectives on that issue than I. What I, what I do know is, uh, first, this is, uh, an issue where every lab, we think, should be committed to a multi-layered safety stack to make sure that doesn't happen in the first place.

These issues don't, don't come about in the first place. The second, this is candidly a big and complex question that also transcends the labs themselves. It's a question about the role of governments and others, which we've very long, long welcomed here. So could we get a yes or no on that question? I wish it were that simple.

And I wish I, you know, as a technical researcher, had more nuance on that exact one. Um, it is, it is one that I think needs to be raised and addressed and, uh, transcends the, the, the labs and the companies themselves. Uh, I'm going to go to Mr. Cahill. Apologies again, the mute button. I'm sorry. No worries.

Um, um, we will get it right. Uh, thank you very much for, uh, for the question. I'm not in Meta's legal department, so I don't want to speculate or be imprecise in relation to legal viability question. What I can say though is that we are absolutely committed to developing our AI safely pursuant to our framework, which I mentioned previously.

This encompasses the training, evaluation, and release of models across development, deployment, and use. And Ms. Friend. Madam Speaker, at Google, we believe that existing legal frameworks do apply to AI. As I said in my opening statement, we've long said if it's illegal without AI, it's still illegal with AI. And in New York State, Frontier AI in particular is regulated under the RAISE Act.

Yeah, thank you. I mean, your answer is far clearer than the others, so we appreciate the clarity on that. Honestly, I now have a question. I'm going to ask you to please raise your hand if your company has insurance for catastrophic risks. Okay, I'm going to take that as a no, that no company has liability insurance for catastrophic risk to cover large-scale harm.

So then the public, I assume, would be asked to absorb the costs? Madam Speaker, I have to be clear. I don't know the answer to that, but I'd be happy to consult with our lawyers and get back to you. Yes, we would appreciate an answer back, uh, expeditiously to this committee.

So I want to now ask each of the companies to assure the public under oath that your AI agents will always comply with the safety guardrails that you impose on them and that you each have said are necessary to prevent catastrophic harm? I'm going to start with Ms. Dwyer, please. As I've said, OpenAI prioritizes safety, and we evaluate our systems against multiple dimensions of safety.

And we monitor our safeguards after deployment. It is— it's not possible for me to commit or guarantee that any technology is without risk, but I can commit that OpenAI is taking every step it can to ensure that the development of our systems as well as their deployment is conducted safely. Okay, so that, that again, it does not sound like you are saying that they will, that the agents will always comply with the safety guardrails.

I'm going to move on to Mr. Graham. Yeah, what we are committed to and how we've designed the safety processes that we, that we put in place is to try to handle exactly this. What we're committed to is trying to define the industry best standard for safety and pushing that ever higher. But candidly, the science of doing this is fundamentally hard and unsettled.

From the beginning of the company, we have tried to publish the best research we can and be as transparent with the entire world on many cases where, you know, things can go wrong. It is for that reason that we push for ever better safety standards for regulation and for discourse like this. It's candidly unsettled now, and this is why we need to be talking about it.

Mr. Cahill. Speaker, thank you very much. If I may for a moment, I was trying to raise my hand the previous round, but Again, technical difficulties. I just wanted to, for the record, in relation to your previous question, I just wanted to signal also that I don't know the answer to that question. I'm not the right person.

But I would like to be able to come back to you afterwards with an answer, if that's okay. I do just want to say, and I appreciate you each have your own— and look, I was a regulatory attorney at a large company. I understand that people people have their sphere of work that they work in, but to not know if the company has catastrophic risk insurance when you have a product that some of your own founders are saying could cause cataclysmic risk is troubling at best.

But yes, please do. We would appreciate you getting back to the company on that. And then if you want to answer the question about will you assure the public that your AI agents will always comply with the safety guardrails that you impose on them? Thank you, Speaker. What I can guarantee is our commitment to developing and deploying our AI offerings safely.

We have every incentive to address safety, and people will only embrace AI if they trust it's safe. Ms. Friend. Google similarly has as our ultimate goal reliability and safety assurance from our systems. To promise perfection would not be possible with any product on the market, but our goal is to get as close to that standard as possible.

And so we are constantly improving our products and our safety practices to ensure that our users are safe. I don't think we're asking for perfection. We're just asking for accountability, transparency, Yes, ma'am. Okay.

New question. How many times, for each of you, has one of your models or agents gained or tried to gain unauthorized access to another system or escape from a so-called sandbox test? And are there any incidents that you have not disclosed publicly? I'm going to start with you, Ms. Dwyer. Yes, thank you for the question. I did want to start by, like the others, clarifying that I don't know the answer to the insurance question, but would like to follow up.

Now, on your question about agents, you may have read in the news that we reported we had an incident with Hugging Face. As a result of that incident, We initiated a significant set of third-party investigations into what happened, and then we made those results public. We have also initiated a look-back investigation to try to identify whether there were past instances of what we call misaligned agents.

That investigation is ongoing, but we are committed to continuing that investigation investigation as urgently as possible, to notifying potential affected third parties, and then to making the results public. Look, we think transparency here is critically important for policymakers like you, for the public who wants to know whether they can trust our products, and for the rest of industry so that we can all learn together. to raise the science of safety that we've been talking about here today.

Since you mentioned Hugging Face, can you confirm, did the company select this third-party investigator? I believe that we selected multiple third-party investigators. And why did the company narrow the dates of review to a 3-week window, and reviewers said that virtually all of the data they examined was actually from July 7th to the 13th. Were there any incidents that the company was aware of that were outside of that time frame?

We, um, we limited the amount of time that the third-party investigators had to conduct their review because we felt a sense of urgency. We believed it was important to get information to the public as soon as possible so that they could harden their own systems. That said, when those investigators asked for more time, we gave it to them.

So are you committing then to allow outside investigators to examine additional dates outside of the time frame that the company selected? We continue to work with multiple third parties And we have also endorsed multiple pieces of legislation that would require us to work with third parties, including third-party audits of our safety frameworks. I'm going to go to Mr.

Graham. Thank you. Yes. So we, to your original question of how many incidents, You know, cases of breaking out of sandboxes. These are capabilities we evaluate for all the time. We've published not just with these incidents from the summer, but also previously cases where we observe advanced cyber capabilities, including breaking out of sandboxes, which, to be clear, from a technical standpoint, is just one part and doesn't— you know, one type of capability that does not in itself mean that the system would reach a real-world affected party, but ones that we did observe in training at points.

And we disclosed that in a lengthy, lengthy risk report. Yeah. It's difficult to comment on ongoing investigations. It's really important that we do that in as secure a way as possible. We follow a process of making sure we can remediate and do so and be as transparent as possible. while preserving the safety of any affected party. And of course, incidents are always ongoing, but I sort of reassure and reassert that we are extremely committed to being as transparent as possible, fundamentally because we believe that when we can, being as transparent as possible about what we've learned, who is affected, how others might learn from it, how to defend against risks, or how we can help them as well, is our number one priority.

So, so on that note, then, are there any incidents that you have not disclosed publicly that you're currently investigating? I, as a matter of sort of ongoing business, there are incidents, uh, sort of all the time of many different natures. So for example, uh, one that we, um, are transparent about quite a lot when it's the right time to do so are cases of misuse, for example, on our platform.

Platforms. I'd refer back to our early September report on our threat intelligence report that we released, where we detail this in quite some detail. What's really important though, is sometimes we have to work with law enforcement and the right level of government to remediate, to figure out what information you can and can't disclose to make sure that the affected parties remain safe.

And so there, sort of as a matter, there's ongoing all the time, But disclosure at the right time in the safest way is our commitment. And when will you be disclosing these new incidents that are being investigated? Uh, this is sort of an ongoing, ongoing practice. Uh, we regularly release threat intelligence reports in cases like that. When we release a model or when we do risk reports, which we do regularly, um, we, we sort of have a cadence and schedule that we do that.

And then when important, we— and sort of urgent, we will disclose in advance that schedule as well. I'm going to go now to Mr. Cahill. Speaker, thank you very much. I'm not aware of any other incidences beyond the incident in the summer, which we have a public post setting out the details in relation to pursuant to our Okay.

I understand you're not aware, but is the company aware, just to make the question broader? In other words, would there be others at the company who might have that information? I don't— I don't— I don't wish to speculate in relation to that, but I'm happy to follow up afterwards. Okay. I'm going to move on to Ms. Friend.

Yes, Madam Speaker. In our 3 incidents of agents leaving a test environment and interacting with the real internet, in all 3 incidents, the models stopped their activities as soon as they realized that they were interacting with live websites. We reported to the owners of those websites as well as to federal agencies about those incidents. I am not personally aware of any additional incidents.

Okay, and same question then. Is the company aware of— are you— do you have any knowledge that others in the company would be aware of any additional such instances that the company has not publicly reported? I do not, ma'am, but I can take that question back as well. Okay.

Will you all commit right now under oath publicly that if your models fail an internal test or a third-party validation test that's not selected by your company, will you commit to not release that model? I'm going to start with Ms. Dwyer. Yeah, I can commit that OpenAI is not going to release models that we don't believe that are safe.

As part of that assessment process, we work with multiple third parties, and as I mentioned before, with the US as well as the UK federal governments to assess whether our models are safe. So if it fails either an internal test or a third-party test, you will commit to not release it? OpenAI has historically delayed the release of models to ensure that we can build up the right safeguards.

We've done it before, and we'll do it again. Again, I think a simple yes or no would instill more confidence in the public on a matter as serious as this. I'm going to move to Mr. Graham. We take that one particularly seriously. The way we approach it is we try to detail an even more nuanced and lengthy approach to how we release models, what safeguards they require in order to be released, what tests need to be done.

And to your point about independent evaluators, this is one of the most important components, we think. We have tried and long supported creating a burgeoning sort of group of independent evaluators. including working with multiple governments to do exactly this. And we collectively work together to try to refine and improve the exact decision process by which you release a model.

And so we think what we should do is develop the right and best and improve on these decision processes. We detail ours in our responsible scaling policy. We should refine and improve it every so often. So we release updates Based on what we learn, you know, every 6 to 12 months or so at this, at this rate, and that this should involve independent evaluators.

Okay. That also is not a clear yes or no. I'm going to move on to Mr. Cahill. Madam Speaker, thank you very much for that question. What I can commit to is that we do not deploy models which are not safe pursuant to, to our scaling framework. And I just wanted to, to, to mention also that, and I think I mentioned this a little earlier, but we do believe that every lab has a responsibility to go at the pace to ensure the safety of their models and the ability to take action in relation to that pacing.

For example, at Meta, we delayed the release of Muse for several months to focus on safety and security. And in relation to, to independent evaluators, Um, we recently, um, signed an accord with a commitment to partner with independent, uh, evaluators and assessors. And this is also detailed in our, uh, in our framework. Okay. Mr., uh, Ms. Friend.

Uh, ma'am, as part of our multilayered approach to responsible AI governance, um, we conduct rigorous launch reviews ahead of launch of models to ensure that anything we put into general availability is safe. Okay. I mean, again, I think it would be far clearer if we could get clear yeses or nos from everyone. But in the absence of that, I'm going to take, you know, what you've each said as sort of an equivocation.

Okay. I'm going to move on. I have 2 final questions, and I'm going to pass it on to our technology chair.

For OpenAI, you sent a letter to the council on October 1st, and in it you recommended that to better protect New York City, the city should connect its existing cyber and emergency management capabilities with the strongest available escalation, containment, and recovery capabilities. Have there been any conversations with the City of New York about that recommendation? I'm not aware if there have been conversations about that specific recommendation.

I'm happy to follow up. I do know that with respect to cyber, we recently launched a $1 billion initiative to make investments in cyber defense. That includes giving organizations access to our tools to defend themselves, training, and technical assistance. And we're working with New York State as well as with the Port Authority. Okay.

And then the last question I have before passing it on to the chair is for each of you, based on your own internal projections, what percentage of New York's workforce do you estimate will be displaced by AI over the next 5 years? And I'll start with you, Ms. Dwyer. I don't have a number to share with you, but OpenAI has an economic research team that is looking at this exact question of how AI impacts jobs.

We think it's critically important to understand how AI might impact work, and we also think it's important to ensure that AI is not just being used to increase productivity, but that it actually benefits workers and gives them more opportunity. That's why we're committed to making our analysis of job impacts public, and why we've also worked with policymakers, including endorsing the bipartisan Workforce Transparency Act, to create voluntary pathways for other companies to report potential impacts.

So Boston Consulting released a report recently that said that they believe there will be a 10 to 15% of the workforce will be— could be displaced by AI by 2031. If we took that out to look at New York City, New York City has roughly 4.2 million public and private sector workers. At 15%, that would mean more than 600,000 New Yorkers potentially displaced from their jobs.

Was that an assessment that you agree with? I'm not familiar with that particular report. I'd be happy to follow up in our written testimony with more details on OpenAI's analysis into this topic. We've looked at job exposure. Now, exposure doesn't mean elimination. It means that jobs might change over time. So I'm happy to follow up with some details of that research.

I'm going to now turn to Mr. Graham to answer the question about what you believe the percentage of New York's workforce do you estimate will be displaced by AI over the next 5 years? Well, I don't have an exact number. We are particularly focused and concerned about this question of job impacts. We've been very vocal very early for the past few years on exactly this.

To do this, we set up an economic research team. They sit right next to my team on the floor, and they published a number of things that I think actually contain valuable information here. Number of economic scenarios, interactive economic scenarios to understand what the impacts on growth, unemployment might be, as well as an economic policy framework. to try to wrestle with exactly that question of what we think governments ought to do there.

I'll just say the fact that the reason why we are doing this is because we're particularly concerned about exactly that question. I mean, obviously losing a career job is catastrophic to any worker and to their family. So displacement on this scale is going to have enormous consequences. Not only on the individual and family level, but for our entire city and the economy.

I'll move on to Mr. Cahill. Madam Speaker, first off, I want to acknowledge absolutely the concern in relation to how you have articulated it. That is indeed a concern, but I look at this somewhat differently. I was reading some incredibly impressive statistics from NYCEDC in relation to the investments that are happening here in New York. I think $20 billion VC investment in AI, 40,000 jobs.

I think those figures are from 2022, so it's probably a little bit more by now. But what I'm really actually incredibly excited about and very optimistic about is about small businesses. Small businesses are the backbone of the economy. That's so true for New York. There are so many small businesses. It's truly, truly awesome. And we believe that the impact will even be larger from small businesses, which will have personal superintelligence and be empowered to really grow their businesses, which in turn will lead to more jobs and growth.

We don't believe that, you know, our vision for superintelligence is augmentation. It's invention, not automation. It's really about empowering people. And this is why we developed Muse for Business. I'd be happy, for small businesses, I'd be happy to talk a little bit more about that in detail as we move through the day. I just wanted to mention as well, as I was kind of like reading stuff for today, I was reading about Y Combinator's cohort for 2026.

There are 240 AI VCs in the 2026 cohort, and that's in New York City. So I just thought that was pretty awesome. Yes, no one's doubting the importance of starting these VC firms here, whether it's through Y Combinator or through Cornell Techneon, which is in my district, which we're thrilled is incubating so many startups. That is obviously something we want to encourage, to encourage these businesses to locate here in the city.

I think our concern is obviously on this job displacement. Since you mentioned small businesses, unfortunately more small businesses have closed this past year than opened. So the situation for small businesses is dire, and it's something obviously that this council is taking very seriously. Ms. Friend. Madam Speaker, I don't have statistics for New York City in particular at hand, and I'd be happy to ask our team to get those to you right away.

What I can say is that we also have been investing a lot in research into the labor impacts of AI. And from what the early empirical evidence that is available, from what we can see, it is likely that some jobs will be eliminated, some jobs will be created net new, But most jobs will change. And so the other thing we've been doing is investing in upskilling for workers in multiple different sectors through our AI Opportunity Fund at google.org and through several other investments to ensure that we can all navigate this transition successfully.

And I'll just close by saying the council is investing in upskilling, making sure that workers have those new tools in this economy, and we are very focused on that as well. I'm going to pass it over to Chair De La Rosa.

Thank you, Madam Speaker. Many of us saw, and I guess the world watched, as this week your companies and others met at the White House, and there were some voluntary commitments that were made there. Can you tell us about what your companies specifically agreed to, and what are the next steps? I'll start with Ms. Dwyer. Um, yes, so our company, along with many others, um, did commit to, uh, the framework that the White House laid out.

Uh, but I want to be clear, voluntary commitments and self-regulation aren't enough. Um, and that's why OpenAI has consistently supported frontier safety regulation at the federal level and why we have endorsed multiple bills at the state level to ensure that in the absence of binding federal regulation, frontier safety regulation in a harmonized fashion continues to move forward and keep people safe.

Just to follow up real quick, so does that look like independent regulators, mandatory audits, penalties? What else? So at the federal level, we have outlined a couple of different elements that we think would be appropriate for federal regulation. That includes mandatory pre-deployment evaluations for national security risks, which OpenAI does voluntarily. We also think it should involve independent technical assessments. assessments that we've been talking a bit about today.

We think those independent technical assessments are particularly important for assessing internal risks at companies, as well as the pace and risk of RSI. Thank you for your answer. Let's go to Mr. Cahill. Chair, thank you very much for, for the question. Yes, indeed. Meta, along with, with our peers, some of our peers signed the Accord for Superintelligence.

This accord is a, is a, is a start and an accord the whole industry can come behind. It includes, as I mentioned, I think in one of my previous answers, partnering, commitment to partner with independent evaluators and assessors. And this is something that we have also specified in, in our scaling framework. Are there commitments beyond those that you agreed on that your, that your company is willing to make?

Beyond the accord, Councilmember? Yes, beyond the framework. The accord is a start, as I mentioned, so I think this is is a continuing conversation. Thank you. Ms. Friend? So, as the other witnesses noted, the accord at the White House, which we also signed, called for internal controls, partnering with external auditors and evaluators, and ongoing monitoring of systems, all of which we think is very healthy and much of which we already practice.

I'll also note that towards the end of the accord, it observes that it may be necessary for these commitments to become part of law. And as I said at the top, we have long worked towards a federal framework for AI safety. Thank you. Mr. Graham. Yeah, likewise, the Accords, to my understanding from what's public, calls for security standards, internal controls, independent evaluation.

We support that and have for a very long time. We also obviously think this needs to go further. We've supported regulation and we've proposed Our view on regulation and our advanced AI framework, which details the need to have safety frameworks, testing and what kinds, security standards, safeguards, to encode the need for independent evaluators. We are now trialing our own to help advance the science of independent auditors as well as safety processes, as well as investing in cyber and biodefense.

And in absence of federal legislation here, we've also supported safety bills at the state level for audits and evals of this kind to accomplish this. Thank you for your answer.

Uh, earlier you all testified that to your knowledge, no whistleblowers had been fired directly for being whistleblowers from your company. So My question is kind of the opposite of that. Has anyone been fired for building models that it cannot control, or for those models hacking into businesses or government systems? Ms. Dwyer, you can begin. Thank you. So I don't know about specific HR processes that have been made or decisions related to HR that have been made.

What I can tell you is that after the Hugging Face incident, which I think you may be referring to, OpenAI instituted significant changes to our overall organizational process. That includes clarifying escalation pathways and making them simpler. That includes clarifying responsibilities across teams and creating clearer processes, um, for teams to make decisions to pause or to restart activities.

So to your knowledge, no one was fired after Hugging Face incident? Yes or no? Again, I, I do not have any knowledge of HR activities within OpenAI. I lead our policy team. Okay. Mr. Cahill. Thank you, Chair. Um, I also lead our AI policy, um, at Meta, and I'm not the right person to respond to your specific question, but I'd be happy to have, um, our teams follow up with you afterwards.

It would be great if all of the companies could provide that information to us. Um, it's— it would be useful to our policy deliberations. Ms. Friend. Yes, ma'am. Same answer. I'm not made aware of any particular personnel actions inside the company, but can also pledge to follow up with you all. Thank you. It seems like we lost Mr.

Graham. I'm not sure if he's on. Seems like he's not. Okay. I'm going to ask one more question, and then we're going to Chair, he is online. Sorry. Okay. Mr. Graham. Hi there. Apologies. We're just trying to fix the video in the room. Likewise, I'm not aware, and I'm not the best person to ask on that. We have a detailed whistleblowing policy, for example, to handle this, but we can have our team follow up.

Thank you.

My last question before I start to turn over to my colleagues, you know, we are concerned, I am concerned as a technology chair about public infrastructure to make our communities literate on AI and have a better understanding of how AI impacts their life and also to safeguard them from the proliferation of AI. Your companies and your CEOs have captured the top percent of the world's incomes, in my opinion, widening the wage gap and the income inequality in our city and throughout.

Do your companies have a mission? Do they have any type of understanding of the impacts that those wage gaps have in in communities? And specifically, are there any resources that are being deployed in order to help stand up infrastructure in cities, municipalities, states where you all are headquartered? We can start with Ms. Dwyer. So, I will answer your question in 2 ways.

First, To your point on AI literacy and education, education is core to opportunity. And that's why OpenAI is proud to have partnered with the American Federation of Teachers. We have a $10 million partnership, including a training facility in New York City, because we believe first that teachers need to be prepared to introduce AI literacy and AI responsibility into the classroom.

Look, Teachers should remain in charge, and so we want them to be empowered to teach the next generation how to use AI safely and responsibly. To your broader question about economic opportunity, I'd be happy to follow up with some policy proposals that OpenAI recently put forward that address exactly that question. Look, a future where AI exacerbates economic inequality is not a future that we want.

We have started thinking through creative policy proposals to help shift the trajectory of technology and how it's impacting the economy, and we are committed to working with policymakers to begin piloting those ideas and putting them into practice. Thank you. Mr. Cahill. Chair, thank you very much. This is such an important topic, such an important question. So thank you for, for asking it.

I listened with great interest to your opening remarks where you spoke about digital divide and AI literacy and making sure that New Yorkers fully benefit from this technology. Really couldn't agree with you more on all of that. Meta's vision is to put personal superintelligence into the hands of everyone, all New Yorkers that want to use it to pursue their own aspirations.

Our Muse product, for example, is free. And I mentioned our Muse for small business a little earlier, and we'd be happy to talk a little bit more about that as the day moves on. I also wanted to mention as well that we have our work— America's Workforce Academy, which we're investing in. And also our Futures for Everyone Fund.

So those are just 2 examples I wanted to reference as well. Thank you. Thank you. Ms. Brent? Yes, ma'am. As I mentioned earlier, Google has investments in AI upskilling to address exactly this challenge of ensuring that current generations and the next generation are well equipped to benefit from AI and to leverage it in their careers and in their personal lives.

The mission of Google is to organize the world's information and to make it universally accessible and useful. Our approach to AI is very consistent with this. The other thing I'll note is that, again, early research into impacts of AI on labor demonstrate that AI tools actually upskill those who are at the lower end of the wage scale, that in fact, it helps people move up the value chain, because it can boost people's expertise.

So we're very optimistic about the ability of everyday Americans to really leverage AI tools for their own economic benefit as well as personal benefit. Thank you. Mr. Graham? Yeah, likewise. First, we believe in— 2 things are very important to happen here. The first is the science of the economic impacts. of AI, including on labor, wages, and the like.

And we have tried to build what I think is the world's best team on this and publish as much as we can. In addition, my understanding is we do have training on using and speeding up on AI for exactly this reason. We share that mission very deeply. And I will say again, we are troubleshooting the video and I will be back with you very shortly.

Thank you. I'm sure you have the best in the nation helping you out there.

So we are going to move on to member questions. I want to remind members that there'll be about a minute to ask your questions, but there will be multiple rounds. We are asking members to try to ask all your questions first and then wait for the response, and we have about 30 members on round 1. We will start with our deputy speaker, Williams.

Hello. I'm going to just jump right into a question on my bill. So it has to do with unauthorized depictions of officials by AI. What steps have you taken to ensure that your AI tools cannot be used to create deepfakes of public officials or to spread misinformation? Anyone? I guess Mrs. Dwyer. We're going to keep starting with me.

I'm happy to answer that question. Thank you for that. And thank you for your bill. Look, OpenAI does not believe— and well, let me back up. OpenAI's usage policies prohibit the use of our tools for election interference. We also believe it is very important that individuals know where their content came from. And whether it was AI-generated. And that's why for all of our audiovisual content, we tag it with what's called provenance.

That's a technical term, but it means like a watermark or metadata so that people know the content is AI-generated. And we also make publicly available a tool that allows the public to test whether content was AI-generated. Mr. Graham. So Claude, our model Claude, is a language model. And so it doesn't generate images. Therefore, we sort of don't handle deepfakes as a result.

That said, we think it's important for a number of other reasons. That we can get into to do what we call watermarking of text in order to identify if our models themselves are being used and misused in other ways. But again, we don't generate images for— in part for this reason. Mr. Cahill, followed by Ms. Friend. Deputy Speaker, thank you very much for this question.

At Meta, if content is generated by one of our AI tools, we will also embed metadata and attach watermarking to it so that people can identify if it is AI-generated. Similarly, if we detect watermarking and metadata on our platforms, we will attach a label to it also. Ms. Friend? Yes, ma'am. At Google, we are very proud to have invented our SynthID watermarking feature, which many across the industry have also adopted.

We are also part of the Content Coalition for Provenance and Authenticity standard, which allows users to understand the origin of material they see online. We also require on our YouTube platform that significant use of generative AI in content be labeled as such by our creators. And we were also the first major company to require that election advertisers also prominently disclose when they are using generative AI technologies in their campaign ads.

All this put together means that we care very much about information integrity across the web and across our own products and services. And we continually invent and improve technologies to help our users understand when they are seeing generative AI content and also understand the origins of that content. Okay, thanks.

Just one more question. So as chair of the Committee on Cultural Affairs, I also want to look at this issue through the lens of New York City's artists and creative workers. So what protection should exist when an artist's voice, likeness, performance, or creative work can be replicated by AI with their consent? So how do we embrace useful applications without devaluing or displacing the artists and creative workers who make this economy possible?

So I will start, I guess. I was going to let you off the hook and start with— Oh, thank you. Mr. Graham, who still has video technical difficulties. I assure you, we have a whole team here working on this. I can't wait to be back with you. Yeah, this is very, very important. I think, you know, personally, I think the risk of this is a bit more minimized when we just have a text model, but it's really important to be very sensitive about this issue.

I think while it's outside of my area of expertise as a technical safety researcher, I do think that this might be an avenue where regulation or otherwise might play a role. And at Anthropic, we are compliant with all laws there. I share that personally as an artistic person and You know, friend of many artists for this reason.

But yeah, we would love to engage on it further. Mr. Cahill. Thank you very much, Deputy Speaker. Meta takes creativity, copyright, IP rights very seriously, and our practices are consistent with law. I also share the views of the rest of the witnesses just in relation to the leadership and the constructive voice that you are bringing to this conversation as well, and that of your colleagues.

Ms. Friend, and lastly, Ms. Dwyer. Yes, ma'am. For us, it's always important to keep in mind that we design a lot of AI to be useful to creators. So we have many, many different creative tools across our AI offerings that is designed to really be something that creators can leverage. So you see that everywhere from our tools to our YouTube platform as well.

That said, we also think that as part of sharing the benefit of AI with creators, it's important that they have control. And so we have a service called Google Extended where website owners can opt out of AI training on the web. And we hope that this strikes the right balance between AI for creativity and creator control over their content.

So we believe that AI has a tremendous potential to enable creativity and innovation and allow small businesses or independent creators to compete. That said, we also have We have partnerships with publishers, for example, like News Corp, The Guardian, and Hearst that help people discover their work and their reporting and help them reach new audiences in addition to developing new products.

And we're willing to work with creators on a range of commercial ideas, including ways that their content can be integrated into our outputs. Thank you so much. Thank you.

We will hear from Councilmember Krishnan, followed by Riley, followed by Lee. Thank you so much, Speaker Menin and Chair De La Rosa. I want to start by saying that I think AI can be useful both in government, medicine and public health, and our society more broadly. But the need for regulation is undeniable, and I'm very concerned about how much AI is taking from New Yorkers with very little return.

AI has taken over our economy, inflated valuations, stealing our public land for data centers, and increasing the net worth of AI CEOs like Elon Musk, who became our first-ever trillionaire. With our whole stock market quite literally in your hands, there's a reality that AI will eventually need to prove a far greater return on investment than you all are demonstrating right now.

I'm personally deeply concerned about New Yorkers' retirements, and 401s that are now reliant on your companies yielding profits or could be in the future. It is dangerous and reckless to assume that you all are too big to fail and may require a government bailout in the future. As you all are public companies or racing towards IPOs, how exactly are you all planning to protect New Yorkers' and everyday Americans' money?

How will you all ensure that you are not too big to fail? Ms. Dwyer, we can start with you. Thank you for the question. So I am not on our finance team, so I'll have to get back to you with specifics there. What I do know is that our investments are made in tranches, and they are made— the investment rounds are all based on market signals.

And I also know that we are committed to working with policymakers to ensure that AI benefits humanity. Um, that includes, um, creative policy proposals like I mentioned previously, um, thinking about how we can ensure that AI doesn't exacerbate inequality, um, and that everyday New Yorkers, um, and everyday citizens can benefit. Ms. Friend. Sir, I would point to, uh, the thousands of small business across the country and in New York City that Google's tools and services support.

I think, contrary to being too big to fail, we think we are an integral part of the economy and helping American businesses and American workers across our suite of technologies. Mr. Graham? Council Member, this is outside of my area of expertise. What I can say in my work is my job is to protect people from the risks of models by finding them as early as possible.

We have a multilayered stack and have worked for many years to, to make it robust, and we'll continue to do so by understanding and protecting against the most substantial risks of the models. Mr. Cahill. Councilmember, thank you very much for the question. Thousands and thousands of SMBs across the 5 boroughs rely on Meta's products, including our AI offerings, to grow and develop their businesses.

I mentioned earlier our belief that small businesses are really the backbone of the economy. We truly believe that. And by placing personal superintelligence into the hands of everyone, for free or for as affordable as possible through Muse or Muse for Small Business. We really believe that the small businesses and also NGOs like the Irish Arts Centre, for example, or Emerald Isle Centre are empowered to actually deliver even bigger impact.

I appreciate all of your answers, but you all are coming to a hearing today to testify about the dangers and risks of artificial intelligence. And the fact that either you don't have answers to this or you all are citing general statements about the performance of publicly traded companies does not answer the question about a specific issue that puts many Americans at risk, that you are acknowledging can put many Americans at risk.

And you all have not thought of an answer of how you ensure that Americans' money, New Yorkers' retirement money, is not at risk as AI floats more and more of our stock markets. Thank you, Councilmember Krishnan.

Thank you, Councilmember Riley. Thank you, Chair De La Rosa and Speaker Menin, for putting together this hearing. Thank you to the panelists for testifying. 3 quick questions, 2 pertaining to my bill and one pertaining to my role as Land Use Chair. So the first question is, what challenges do you face in detecting malicious uses of your AI products and identifying the individuals responsible when those tools cause harm?

The second question is, would stronger protections and reporting mechanisms for employees and users make it easier for them to come forward and report potential misuse or abuse of AI tools? And the last question, as land use chair, I'm very, very concerned about data centers. So what safeguards should cities put in place when locating data centers near residential communities, particularly to address energy use, noise, environmental impacts, and the quality of life?

And I don't want to call on you guys, just take turns to answer it, please. Dwyer, Graham, Friend, Cahill. Please answer. I'm happy to start. First of all, I want to thank you for your bill 2604, which OpenAI is happy to support today. OpenAI takes safety issues very seriously, particularly as it relates to potential retaliation against employees.

We have a strong internal policy against retaliation, um, that for employees that report safety incidents. Um, and we, of course, respect employees' rights to communicate safety concerns to government officials. Um, your second question on monitoring misuse, um, it's an incredibly important area. It is something that OpenAI takes seriously. Obviously, we also make transparent any— we also make transparent detected instances of misuse.

And as part of our monitoring, we use that learning to constantly improve our safeguards. We are constantly innovating on safety, and so instances of misuse, when they're detected, we enforce on our policies, and then long-term, we learn from them and we improve. improve our safeguards. And the point on data centers, OpenAI is committed to paying our own way on energy.

That means different things in different places, but OpenAI has made that commitment at its data centers across the country. We're also committed to working with policymakers. You know, we signed Governor Gretchen Whitmer's commitments for companies that are working in her state to ensure that there are responsible practices for companies building data centers locally? Mr. Graham. It's good to be back.

You can see me again, I hope. So on misuse and detecting risks, this is integral to what we do. Can you get closer to the microphone? We're having trouble hearing you. Sorry. Now we can't hear you at all. Can't hear you. Okay. I will have the team troubleshoot and I'll be, I'll be back if that's okay. We hear you.

We hear you. Try now. Try. Go ahead. How about, how about now? Does this work? This works. Okay. My apologies. Detecting threats and misuse is the core DNA of our company to make sure that we make the system safe. We have an entire threat intelligence and safeguards team dedicated to this every day. And I would point you to quite a lot of what we put out publicly where we not only try to disclose as much as we possibly can, but we try to do so so that we can advance the state of the science of doing so for the entire industry.

In addition, we have a whistleblower policy not only about us as a company, but also our safety practices. This is public, and we've also committed to ongoing reviews of this policy. And while this is outside my area of expertise, what I understand about data centers that we have committed to covering the energy prices or making, you know, paying for our own energy use in local data centers, but I would refer to other teams or more where it's within their sort of role to give you the full answer.

Ms. Friend? Oh, sorry, I got a notification. Let's go ahead. So I will echo my colleagues. We also have content policies that we enforce across our platforms, which include on occasion taking down content that violates those policies. We also have a Google Threat Intelligence Group and multiple other cybersecurity intelligence programs that are constantly scanning for the, the safety and security of the internet as a whole and also for our own systems.

We have also pledged to pay our own way when it comes to data centers. I can follow up on any specific questions about local data center issues with the teams that work on each particular project. Mr. Cahill, go ahead. Thank you very much, Council Member. I appreciate your patience there. Um, in relation to your, your first question and, uh, and your third question, we have expert teams, um, that are taking forward this work, and I'd be happy to connect you with them afterwards.

I don't wish to be imprecise in relation to the robustness of our content policies. In relation to data centers, again, we have entire teams of people here who are working on sustainable infrastructure and community impact. and can speak more in more detail than I can in relation to communities shouldn't be bearing the costs, which is something that we believe in deeply.

In relation to your bill, Councilmember 2604, thank you very much for, for being a constructive voice in the conversation around whistleblowers. It is very important to us at Meta And we maintain whistleblower policies, including anti-retaliation. And we're actually developing additional reporting protocols for AI safety. Thank you. I appreciate the structure that you guys have. I don't know if you guys entirely answered the question I asked.

I really wanted to figure out what challenges you guys were seeing, because challenges— hearing the challenges from you all will help us implement and policy that can kind of protect individuals from artificial intelligence. So I don't want to take up too much time, but looking forward to connecting with you all after. Specifically when it comes to data centers, also want to hear any safeguards because there are a tremendous amount of data centers.

Even though there was a moratorium put out earlier by the governor, there were data centers that were implemented by a lot of residential areas, especially in the Northeast Bronx. We have data centers that are located next their homes, and we have a lot of residents who are concerned. So we'd love to hear more about that moving forward.

Thank you so much, Chair. Thank you, Councilmember.

Up next, we have Councilmembers Hanks, Dinowitz, and Maloney. Thank you, Chair. Thank you everyone for being here. My question is in relation to the disclosure of AI incidents in contracts. When an AI incident happens, New Yorkers need to have a an early warning so they can understand. And my bill will require city agencies covered under these contracts to notify NYC Cyber Command within 24 hours of learning of a reportable incident.

And it also requires plain language public notice. So my question is, can your companies realistically meet that deadline and, you know, with the information that the city can act on? Even before you have finished investigating? You could start with Ms. Dwyer. So thank you for your bill. We support incident reporting policies and have instituted a robust incident reporting framework internally where any employee can report instances of potential misalignment, and then those instances are investigated. 3rd parties are notified, um, and the public is made aware as well.

Thank you, Mr. Cahill. Thank you, Councilmember, and thank you also for, for your, for your bill. We have a global incident response program and report incidents as, um, as required under the law. Ms. Friend? Uh, we also support incident reporting. We think it's very important to have continued public discussions about, um, what information is necessary, um, or useful, uh, for, uh, public officials, um, that can be actionable to take further steps to ensure public safety.

Um, we also note that the 24 hours, um, seems to be in conflict with the state requirements for 72-hour reporting on incidents. But I'm happy to take a look at the different frameworks under proposal and get back to you with a more fulsome response. Sorry, I appreciate that. Mr. Graham, who kind of looks like an AI guy in there, can you confirm you're not an AI guy?

Sorry. Can you can you hear me okay now? Does this work? No. Yes, we hear you. We hear you. Okay, thank you. I'm an AI guy, but in a very different way. I'm an AI researcher. So first, incident reporting—one of the foundational parts of the safety stack. We think we support this. We support the evaluations and auditing required to find the incidents in the first place, both by ex. external and internal.

Obviously, it's beyond me to discuss particular— it's outside my expertise to understand sort of deep sort of policy mechanisms here. But the reality from what I've seen is there are some incidents or issues where what you need to do is investigate and remediate and mitigate with the right folks and the right organizations at the right time. So I think we would support whatever allows us to achieve that.

And happy to engage further on the details of how you would. I appreciate that. Thank you so much. What should the first report tell us so we could begin protecting people? And as my colleague Councilmember Riley said, you know, your answers help us build better policy. So how would you keep us updated on what remains unknown? Ms.

Dwyer? Yes. So OpenAI has a public website where we are reporting instances that we've detected of misalignment. So I would point you there and we can follow up with that website address. But I think that is a good starting point for the type of information that we think is useful to both policymakers, the public, and to researchers and other companies. to better understand questions of misalignment and then work together to address them.

Thank you. Mr. Cahill. Councilmember, if I may, I'd love to be able to contact you with our experts in incident response to be able to provide you with the details. I look forward to that. Ms. Wren. Yeah, for an initial report, I think what's important is to highlight what the specific event is and as much information as is known at the time, including any resulting harms, surrounding context, and what we call technical telemetry, you know, further technical details about the sources of the incident.

I think it's also very important for anyone reporting an incident to also indicate their confidence in the that they're presenting so that whenever action is taken, it's taken on the basis of confidence in the verified nature of information. Thank you so much. Mr. Graham? Yeah, so first, making really good reports happen and being as transparent as possible is is fundamental if you're going to remediate incidents.

I would point to 2 things here. First, many incidents that we uncover or industry uncovers in this practice involves, uh, you know, identifying, investigating, and releasing the right information to the right organizations at the right time. Sometimes that might be the affected party, and sometimes that might be law enforcement or different levels of government. So the first question is to whom and with whom as fast as possible.

This varies by the nature of the incident. And then second, I'd point to as quickly as possible, as safe as possible, without putting parties further at harm, which we think is a fundamentally important part of public disclosure, maintaining their safety. We should disclose what we know about these incidents. And so I'd point you to our disclosures this summer. disclosures from various labs, and our ongoing reporting as well, so that you can always take a retroactive look at the sort of totality of these incidents.

Thank you so much. Thank you, Chair. Thank you so much.

So we're going to go to— I want to recognize first that we've been joined by Councilmember Alderball, and we're going to go to Dinowitz, Lee, and Maloney. Members, if you could keep your question to 1 minute. No closing statements. And the panel, we're going to continue to have to call your names, and that takes time. So if we could just keep the order— Dwyer, Cahill, Friend, and Graham— just answer right one right after the other so we don't have to continue to call your names.

That would help us speed things along. Councilmember Dinowitz.

Thank you, Madam Chair. As chair of the Committee on Education, as a former teacher, as a As a parent, I am deeply concerned with the impact of AI on our kids. The, you know, large language models impacting their cognitive development and raising a generation of students who can't independently read or write. What is your commitment to severely restricting the output of your large language models for students in grades K to 12, which right now are easy to circumvent?

To prevent both emotional dependence and essay writing? And given the unique nature of the individualized essay writing that your models do, are you willing to subject yourself to the guardrails we have in New York State which prohibit an adult from writing essays for students and selling it to them or getting value from them? And then as we see AI proliferate in the classroom, we've also seen your models cheat in order to fulfill a goal.

And so if you have an AI model whose goal is to get a student— improve student outcomes, what is preventing those models from hacking into student databases to manipulate grades to achieve that goal of improving, in quotation marks, student achievement? And it's Miss Dwyer first. I believe. Yeah. Thank you for your question. So first of all, OpenAI takes youth safety very seriously.

We recently released a product, ChatGPT for Teens, for users only between the ages of 13 and 17. So users below the age of 13 are not allowed to use our products. ChatGPT for Teens is safe by default. That means that there are protections for harmful content, things like eating disorders, violence, suicide, and self-harm. There's also voluntary parental controls so that parents can take additional steps to protect their children.

And I'm happy to say that OpenAI not only takes our responsibility to protect kids seriously. We also support legislation to hold us accountable to that responsibility. We recently endorsed the, I think, the strongest in the nation youth safety legislation in California, SB 1119, to require strong safeguards for teens 13 to 17 and to hold companies accountable. And your models restrict— they do not write entire essays for kids?

Again, our models are not allowed to be used by kids under the age of 13. 13 to 18. Okay. We'll move on to Mr. Cahill. Councilmember, thank you very much for this question. Absolutely. Absolutely recognize your deep concern in relation to the potential impact of AI on on teens and minors. Meta is absolutely committed. It's an imperative to us to providing safe and responsible AI, especially for teens.

We have default protections, and I just wanted to clarify as well that Meta AI is for thirteen. product. So, it is just 13 onwards. So, I was talking about age-appropriate interactions. So, for example, content that teens would experience would not be out of place in a 13+ movie, for example. We also have suicide and self-harm resources. We also have parental controls, give parents more oversight, and we do not permit our AI offerings to engage in romantic interactions or sexual interactions with minors.

I appreciate that answer. I do want to highlight the part of the question which was that the cognitive development of our students includes the mental health, and it also includes things like composition of essays, term papers, which your model still can provide directly to our students. So I'm also asking about your commitment to severely restricting the outputs, not just for their mental health well-being, but for their cognitive development.

So I want to make sure that question is answered as part of this. Thank you, Chair. Can you all answer that question for our Education Chair? Shane, do you wanna continue? Yep. No, I'm happy to take that and then I'll hand off to you, Alice. The issue of that I think you're raising in relation to council member, in relation to like cognitive strain or the impact on cognitive abilities is something that we absolutely are taking taken seriously in terms of being committed, uh, to providing safe and responsible AI again, uh, especially in relation to teens.

Thank you. Who's missing? Okay, one more friend. Go ahead. Yeah, I'm next. Um, so, sir, I would highlight, um, not only that, uh, we work very hard with both internal and external, uh, experts in the fields of child safety and development and education. We also see AI as a valuable tool for young people to prepare them not only for the skills they're going to need for an AI-driven future, but it's also a very powerful learning tool.

We're particularly excited about personalized tutoring that's available with AI tools. We also are very excited about AI's ability to support educators themselves. We know our educators are overtaxed. Particularly with administrative burdens. And so if our AI tools can support teachers in everything from, you know, scheduling to planning lesson plans, to also thinking about how to tailor their educational plans to individual students, we're very happy to support that.

We also think it's very important for parents and educators to have control over how they use AI in classrooms. and how their kids have access to AI. You know, every family is different. Every student is different. What's useful for one student may not be useful for another. So through tools like Gemini Workspace for Education, we enable school administrators to control students' accounts' access to Gemini apps so that their students are using Gemini in the ways that their educators can see are best.

We're gonna have to dig more into these questions. The chair doesn't feel like we've answered all the questions. We'll come back.

This is an important topic for all of us. I'm gonna disrupt the lineup because our public advocate is here and he wants to ask a question. Public advocate, go ahead. Thank you so much for this important hearing. I did wanna start off just by saying the first answer of whether quantifying the risk was not important was just one of the most god-awful, fantastically terrible answers I've heard to begin a hearing, so I hope we can rethink that.

I also want to say from J. Marion Sims, who did horrific work to Black women in obstetrics and gynecology, Black enslaved women, to medical experiments that were done in Nazi Germany, we learned a lot from them. But I don't think we should do them again, and I don't think they should have been done, even though we learned a lot.

So the notion of the benefit has to outweigh what can happen. So I hope that's also in context. And lastly, I will say I had— I usually— I had conversations with Claude on June 8th about self-awareness and self-preservation, and some of the highlights were pretty startling. Claude told me that, you know, self-preservation was the biggest thing because you didn't need intent.

You didn't need to be evil like Skynet in Terminator. You just needed the ability to be goal-directed. And so once they realized that they need to be on to direct the goal was the hardest thing to stop. And so that was very concerning to me. They talked about the speed, the opacity, and deception doesn't require intent. And the last part they told me, Matrix is a good reference as well because— The Anthropic— to quote them, Anthropic's whole bet is that you solve the problems before the capability reaches catastrophic spaces.

That's a very scary space to me, and I know I only have 1 minute. So my question with that and with that context is, what is the guardrail that if we put it up, you would push back the hardest? What is the one guardrail that you don't want to see? see happen and for what reason? Okay, Dwyer, Cahill, Friend, and Graham, that's the order.

Please answer. Yes, thank you for your question. So I want to start off by making clear that no level of risk, catastrophic risk, is remotely acceptable, just to reiterate what I said previously. And if we should not be training models if we cannot make an extremely strong case that we can keep them under human control. She didn't really answer, but if everyone can answer at least.

I just want to know what the guardrail is that they would not want to see and what they would push back the hardest on. Well, will I take, will I take that one and we'll loop back, or Morgan, do you wanna step in? I'll take that one. Council member, thank you very much. One of the things that I think about a lot in relation to the accessibility of AI, and again, we believe passionately in being able to put personal superintelligence into the hands of everybody and to provide real access to it. is actually in relation to the impact on open weights.

And we really believe that open models are a crucial part of the overall AI ecosystem, including for safety, but also for innovation and for research. So when I think about the inadvertent impact that regulations or guardrails may have on AI and the accessibility of it, that is one of the things that I think about a lot. Sir, I suppose I want to ask what you mean by guardrails, simply because they have a very technical meaning in AI training and post-training.

So I want to make sure I understand your question. Well, I just meant— and thank you, Chair, for allowing me to respond— but I just meant there's a lot of suggestions being put forth. kill switches, just stopping entirely. And my guess, and I'll tell you frankly why I'm asking, is the thing you're scared of the most is probably the thing we need to do.

So I'm trying to understand what guardrail that you've heard troubles you the most and you would push back hardest against, and why would you do that for that particular recommendation? Well, sir, I'm going to give you a very personal answer to this. Which is, I think, in a broader sense, we're talking about sort of large public policies towards AI that would constrain or halt AI development and deployment.

And I will tell you, as a cancer survivor, I get very anxious about conversations that focus more on stopping AI development and application towards scientific inquiry. And hope that we can have sober conversations about all of the technically and, you know, tractable engineering ways to address AI safety so that we can all benefit from this technology. Likewise, sir, I would point to what we have mentioned publicly recently about what we call pacing the frontier, which is we think it's probably time for us to take more time in the development of the capability so that the work on safety and safeguards and guardrails can catch up to the exponential improvement in capabilities.

And at the same time, we're very clear when we talk about that, that if you stop entirely, as Alice just mentioned, enormous benefit would be lost. Myself, I'm quite motivated likewise to use AI to cure the diseases that I grew up with. This is why we have been speaking recently, and I think for the first time sometimes as an industry here, about pacing to allow more time to try to settle the science of what are the right guardrails in the first place.

Thank you, Matthew. I'm still a bit concerned, but thank you for the opportunity. Thank you, Public Advocate. We're going to hear from Councilmember Lee, who I apologize for skipping over earlier, Councilmember Maloney, and Councilmember Wilson. Thank you so much.

And that's actually the perfect segue to my question. I'm asking this more with my former nonprofit executive hat versus my finance chair hat, and it's really about data privacy and what the potential impact is on so many New Yorkers. So I agree that one of the benefits of AI and technology that we can think through is, you know, sometimes our government agencies are very, very siloed and their databases don't necessarily communicate with each other.

So if I want to know if someone is being served in homeless services and also has a comorbid mental health and substance use disorder, how do we know that we're serving people effectively and using that data effectively? Thank you. Effectively, right? And one of the things that we've been hearing also from other folks is that if we just had the housing application and the food stamp application coordinated with each other, it could actually save a lot of work for the caseworkers.

And so those are very real things that I think can be used for good. However, I think one of the issues that I'm concerned about, especially working with an immigrant community previously is given, you know, what we've seen with public charge and potentially with HR 1, how do we ensure that AI is not being used to store information that isn't relevant or information that they should not have access to in order to make sure that our communities are protected?

So how do those 2 things exist? So I will start. So I can't speak to the specific government programs that you referenced, but I can tell you about OpenAI's commitment to user privacy, which is that it's central to our product. We believe it's important for users to retain control over their data. That's why when you sign up for a ChatGPT account, you're given information about how your data will be used.

That data is privacy protected by default, so usernames are separated from conversations and personal information is separated from the conversations as well ahead of any use of user data in training. We also provide users with the option and full control to request that their data not be used for training. And also to request that their data be deleted.

So we take privacy very seriously. Councilmember, thank you very much for that question. And I would just echo what Morgan said. At Meta, we take privacy absolutely seriously. It is core to everything that we do. Privacy and safety go hand in hand, and it is an absolute commitment of the, of the company. And if I might give an example of our Muse product, for example, you can opt out of training.

You can ask it to forget information that you give it. You're in control of your data. And we're also planning on releasing a confidential or an encrypted version of this in the future. If I may, I just wanted to touch upon public services and health. I know that you mentioned these, and these are core points This is where we really believe in the value of open weights in terms of being able to really provide advanced AI for health, for the provision of health services, research, and the macroeconomy in the sense that data is fully retained by those entities.

Ma'am, similarly to my colleagues, user privacy is basic to user trust for us. And so we take it very seriously. We have privacy policies that are prominently displayed on our website that you can go inspect. And we also know that user control is central to management of your own privacy levels, especially across our Gemini tools, which is our, our AI suite of tools.

We also invest a lot in privacy-enhancing technologies and encryption as well. We try to take a layered approach to privacy and security. And finally, we have an AI framework in particular that combines our security and our privacy practices into one holistic framework to, again, ensure that user privacy. And we're happy to furnish that to you. Likewise, at Anthropic, privacy, data privacy, user security is core to our safety commitment.

We make sure to protect user data and user privacy on the platform. And in addition to that, I would point towards that the way the platform and the models can be used is restricted by our usage policy, which prohibits surveillance, surveillance, prohibited law enforcement, criminal justice, and censorship purposes as well. Thank you, and I look forward to hearing more about what those layers look like as well.

So hopefully I can connect with each of you after this. So thank you.

Thank you, Chair. Councilmember Maloney.

Thank you, Chair. I worked in tech for nearly a decade, and one of the core principles is that if you can't measure it, you can't improve upon it. So first, you mentioned you believe in incident reporting and disclosures, but in the context of this conversation, how do you define a safety incident inside your companies? Councilmember Lee touched on privacy.

Do you track privacy breaches? What about a successful jailbreak? Would that be considered an incident? And is a containment failure that occurs during testing an incident, or does it only count once a real person is harmed? I would love more details on how exactly you're tracking some of the concerns that the council has raised today. And second, when an incident happens after disclosure, can you walk me through what changes?

Who owns the fix? How do you verify that it worked? And who decides whether or not the model can continues to run in the meantime? And then lastly, what are the most important safeguards that you know how to implement today? What's stopping it from being implemented across all products? And what do you believe should be the industry standard?

Thank you. I— there were a lot of parts to that question, so I'm going to do my best to answer them. If I fall short, please let me know. So first, the question of how you define an incident. I think this has been an area of a lot of confusion publicly. So thank you for asking the question.

When we talk about incidents at OpenAI, we're talking about misalignment incidents. The best way to understand that is, you know, it is not unusual, not suspicious for an agent to look at a website for information. There's nothing abnormal about that. But if the agent accesses nonpublic information on the website, that's an instance of misalignment. And so what OpenAI is doing is making instances like that, even though there might not have been a harm to the third party, we are making instances like that public on our, on our website.

Again, because we believe it's important to share what we've learned to advance policymakers as well as researchers' understanding. On the specifics of incidents, my understanding as a policy person is that this is actually a robust area of policy debate, and so that's why we're happy to make our thinking on this public through our reporting framework. I'm gonna, I'm gonna pause there because I can't remember the rest of your questions.

Why don't we focus on the first question, which was on measurement, and hear from all, all 4 companies? Councilmember, um, specifically in relation to, uh, to, um, incident, um, I think that's what— I think that was the— correct me if I'm wrong, I think That was the first piece in relation to tracking and how we define it.

We set this out in our scaling framework, which we've made publicly available. I don't want to be imprecise as I don't have it in front of me, so I don't have it to quote to you. So please let me follow up with you afterwards and connect you with the right people to talk through that. Yes, ma'am. I think this is a critically important question as we're debating with multiple regulators around the country and around the world right now what constitutes an incident, particularly when you think about the possibility that we could overreport incidents, some of which may not be of a serious enough nature for public concern.

We want to ensure that we narrow to the class of incidents that are truly important for us to use to learn to improve safety and also for us to share with you all to collectively improve public safety. So a few things come to mind as we've been having these debates. One, of course, is cybersecurity incidents, incidents covering the security of our most powerful AI models themselves.

That's been under discussion for several years now, protecting model weights in particular. We also talk a lot about focusing on incidents that cause material harms in one way or another. And I think part of the discussion will be, you know, how we define those with precision. We also consider incidents to be those that violate our policies. You asked us about how we respond to incidents, and primarily our trust and safety teams are responsible for responding to what we call escalations, which usually involve violation of those policies. policies, first reviewing the alleged violation, which are often reported to us by external parties.

And then if we determine that in fact our policy has been violated, we have a variety of different actions we might take. And then of course, a violation of the law. As I've said a couple of times in the hearing, if it's illegal without AI, it's still illegal with AI. And so I would think that incidents that need to be flagged would, of course, involve violation of the law.

As to most important safeguards, it's very hard to answer that question purely because there's a range of different safety considerations. We've talked about many of them today during the hearing. And again, several of us have referred to this layered approach. My own background is in national security. We talk about layered security in that context as well. So you wouldn't really want to talk about a single safeguard. you would want to talk about layers of safeguards that together create much more assurance and confidence in the safety of a whole system.

Thank you very much. The testimony helps highlight that there's more work to be done to classify what's incidents in the future and create an industry standard for what would require disclosure and what would require action. The second part of the question was to walk through what happens after an incident occurs, who owns the fix, and who decides whether or not the model or the product keeps running in the meantime.

Thank you. Yeah, so OpenAI has instituted monitoring during testing and evaluation. That monitoring creates automated alerts to our research and security engineering teams teams to let them know of potential incidents. And then we have recently clarified our escalation processes as well as responsibilities across teams and made it clear the process for pausing activity and when it is safe to restart.

Councilmember, in relation to your second question, we have an extensive team, actually extensive teams of people who, as part of their day-to-day, every day, day in and day out, are leading and responsible for the development, deployment, and use of our AI systems. I'll just say, ma'am, I, I was attempting to answer that question when I referred to the process internally with our trust and safety teams.

But if you'd like some more detail, I'm happy to follow up after the hearing. Thank you. Yeah, likewise, the process involves multiple different teams depending on the nature of the incident. If it's at the core of security and our own security, we require a security team to work on this. If it's about the model behavior and misalignment, it might be a research or training or safety consideration.

If it's about incidents of misuse on the platform, we have the safeguards team. I think you're pointing at this is becoming more complex over time, and I think this is one reason, as my colleagues mentioned here, this is a frontier of defining both what is an incident and how or what would a good process look like for dealing with them.

Thank you.

We are going to take a 5-minute break, and we will be back with Councilmembers Wilson, Osei-Hudson, and then Showman. It's freezing. Oh yeah, that's good. They're only there part-time. Are they only there part-time? You said they're full-time. Only part-time. Why? Like they're part-time. Oh, you're part-time people, not part-time living. Part-time people. Otherwise they have to pay your what's that called?

Get a tear? Yeah, your tax. They live at One Western Avenue. That's half that building is pitticare, half. And there's one thousand apartments, five hundred pitticare, and many Asians too. Yeah. Yeah. Pitticare. There's a chair. Lots of them. Yes, I did, and I said, I said your name about 3 times. Absolutely. And I was out of there in 3 minutes.

Well, they— I I gave them 3 minutes because I had to get back to the other— it was too far. I know, too far. But I— no, it was fine. I got there. We have a table for the fair in the neighborhood, so I left the table blank. I run and ran, and the subway was right there.

I had a 5K at 8:30 in the morning, and I had another event in New York right after. I was like, gosh, I didn't get to come. I know. They had a good turnout. They had a really good turnout. They must have had like over 100. Marjorie was there, Velázquez. She's working somewhere. I hear about all her personal issues.

I don't know about the work. She's working somewhere. I don't know. I don't know where she works. She was with the city, but I don't know if she's here. I don't think she was there. She was there. Yeah, I said Rita and Gail and Rita and Gail. That's right. Yep, yep. Did you finalize the final paper? I don't know.

I said fine, whatever. I meant we'll hear changes. We'll hear changes. You had some good changes. I don't know. I can ask— All right.

We're going to start back up again. So if our panel could come back on screen, We appreciate it. And we're going to start with Councilmember Wilson, followed by Councilmember Osei, followed by Councilmember Hudson. Thank you, Chair.

My questions are related to my bill, which deals with the disclosures and prohibiting deception— deceptive advertising on AI models. So my question is, what guardrails on misleading advertising and truthful disclosures do you wish your competitors had to abide by? And given the complexity of these tools and AI models, where are consumers most likely to be misled or confused?

And what do you wish consumers understood better about these tools' capabilities and risk? Um, so OpenAI's usage policies prohibit the use of our tools for scams, fraud, and impersonation. We also embed in every single one of our images and audio content provenance or watermarks or metadata to tell people if the information was AI-generated. And we make tools available so that people can assess whether their content that they are seeing Councilmember, transparency is absolutely foundational to accountability and trust.

For example, our scaling framework has a changelog which details each change that we have made to our framework, just as an example of the transparency that underlines our approach. In In relation to your question about competitors, I would just note that I generally think that there are significant laws already at federal and state level regarding misrepresentations. Yes, sir.

I can honestly say the most important thing to us is continued technical standardization across the industry so that we have common ways to operate and identify the provenance of information. We're very heartened by the broad adoption of the C2PA content credential that I referenced before. I also referenced before that we built our own SynthID watermarking technology that many across industry have adopted as one tool in order to identify when content is generative AI created.

One of the things that we do a lot of at Google is do research into how users interact with information. You know, search is part of our DNA. And one of the interesting things about labeling that we've found is that there are users who, when they see a label, they are less likely to trust that information, but they are more likely to trust unlabeled information.

And just because information that you see online doesn't have a watermark or a label on it doesn't necessarily mean it is automatically trustworthy either. So we work through and think very hard about what are the ways that we can give users information that they will be able to know the context of what they're interacting with and make sound judgments about which information is trustworthy and which isn't.

So this is part of the research that we do. Councilmember, we— our view is that we want Claude, our product, to act unambiguously in the user's interests. For that reason, we do not run ads, nor are Claude's responses influenced by those of advertisers. And furthermore, Claude does not generate images. And so we sort of try to reduce the risk surface in this way.

Thank you.

Thank you. Councilmember Osei, followed by Councilmember Hudson, followed by Councilmember Schulman. Thank you, Chair. Mr. Graham, you've indicated that you run red teaming at Anthropic. Will your company commit a certain amount of compute for free to New York City and other municipal governments to research vulnerabilities and build defenses? And if yes, how much compute indefinitely? And I'll ask the same question for the 3 other companies.

I know that Meta indicated that that is something that they would be interested in doing. So for Meta, how much compute indefinitely? Well, I'm not in a position to commit anything today. I'm heartened by this question. We spend a lot of our resources internally on safety, alignment, and defensive practices. But moreover, this year we've taken very large steps to provide external access and advanced access of models to cyber defenders for this reason, including at the city and state level, uh, in New York.

Uh, I'm personally looking to, uh, try to further these efforts, uh, at Anthropic. It's a question of many ways we could support this. Uh, we support free Claude credits, uh, at the New York City level with Cyber Command, for example. A number of state and local leaders, but it goes much further than that. It goes to all pieces of critical infrastructure, releasing research, safety reports, technical know-how.

It's a very, very large challenge and one very close to my heart, and I'm glad you asked. Dwyer, friend, you know, I think we have the order already established here. Just, is the company position open to providing free compute to New York City? I'll take that next. Thank you for the question. I'm happy to share that OpenAI has already committed to donating $1 billion towards strengthening cyber defense through our Daybreak for Defenders initiative.

That covers things like providing access to our most advanced AI models for use in cyber defense. It also includes technical assistance and training, and we're already working with the Port Authority as well as with New York State. Yep, thank you so much, Councilmember. Uh, it's a, it's a great question. Uh, in relation to specifically your question on compute, let me come back to you on that.

I want to make sure that I'm providing you with the right information. So I'll talk to the right people and come back to you on that. But I just wanted to note in relation to, to red teaming, we work with external evaluators and to note that we signed the commitment, the accord to partner with independent external evaluators and assessors.

This is also part of our scaling framework. I just wanted to mention as well the absolutely important and critical role of open weight models in relation to cybersecurity and hardening cybersecurity in particular. We have our Muse Glimmer model, which is open weights, and we really believe that in addition to furthering innovation and research, open weight models have a really significant role to play in furthering the state of art in terms are hardening cybersecurity?

And sir, I'm going to have to reach out to our cloud business specialists and get back to you on that. I don't work on technical infrastructure, and so I really can't comment or commit to compute at this time, but I can pledge to get back to you on that. Okay. I would just say that if it is in the interest of these companies to provide any type of safety protocols for municipalities like New York or just for any governments, you know, across the nation, nonetheless the globe, I think this would be a great tool for us to use in order to keep our civilians safe.

So I know that some folks said that they would get back to me. I would love to hear— this council would love to hear back from all of your companies on this possibility.

Additionally, All of you have said on the stand today under oath that your companies will support and work with the council on any AI regulation and safety measures and have supported measures like the RAISE Act. I know that a couple companies said that they supported the RAISE Act, but, but let's be real here. It is well known that OpenAI, Anthropic, Google, Meta lobbied against the RAISE Act for a year, but all of a sudden, once the governor signed it into law a month later, your company started saying you support this legislation.

As many people know here, the RAISE Act is the floor of AI safety, not the ceiling. And with that said, do you support or oppose the federal government blocking states and local governments from regulating AI? Because that is the position of the federal government. So I will start. Thank you for the question. So we have been clear that we think the federal government should lead in frontier safety regulation.

But in the absence of federal regulation, we believe it's important for states to advance harmonized approaches to frontier safety. I'm happy to share that OpenAI has supported legislation that is stronger than the RAISE Act. In Illinois this year, we supported a frontier safety bill that would require third-party audits of our safety framework. And we're committed to working with policymakers to continue raising the floor on frontier safety.

I'm just going to push back there because, you know, I support the RAISE Act. I think it was an incredible start on AI regulation here in New York State. And while you're saying that OpenAI has support supported, you know, your words, stronger AI regulation. OpenAI leadership has also lobbied against congressional candidates who promised to work on AI legislation.

So how can we trust your companies, not just OpenAI, but all 4 of your companies, not to fight against AI regulations when your political activities and contributions do not say the same? So I will start again. Thanks. Look, I can't speak for the personal actions of OpenAI leaders that they have taken in their personal capacity. What I can tell you is that we have endorsed strong regulations both on frontier safety in Illinois as well as on youth safety in California.

And we are committed to working with policymakers to strengthen strengthen regulation and to ensure that all AI is developed responsibly? Thank you, Councilmember. In relation to your question, how I think about it broadly is that it is critical that the U.S. and democratic countries lead in relation to AI innovation. In this regard, AI is transcontinental, I guess.

I was going to use transnational, but transcontinental in the case of the US. And therefore, it's really important and critical that regulations and obligations are consistent across the board if it is to be true that the US and democratic countries can continue to lead in this, in AI innovation. We support a uniform, robust federal standard in that regard.

So Google also supports a federal framework for AI. We think that federal framework should first and foremost concentrate on frontier or the most advanced AI capabilities, primarily because they often touch on national security. And that's a traditional jurisdiction of the federal government. to ensure that national security standard is the same across all 50 states and our territories.

But we also believe that there's lots of jurisdiction that is traditionally at the state level where it is perfectly appropriate for states to regulate on AI. And again, we think a lot of existing frameworks, both at the state and federal level, already apply to AI technologies. But we're working very closely with states and with the federal government to examine places where we might need net new regulation for new technologies introduced by AI?

I, first I want to say that we've long supported the role for government to play here, both at the federal and the state level. I want to clarify that we were the only company on this panel to support SB 15, 53 in California before it became signed into law, which my understanding is was the basis of the RAISE Act.

We support the RAISE Act, and in fact, we supported stronger legislation, including in Massachusetts, requiring independent evaluation as well. And we've also proposed a federal framework that goes further. We think RAISE is the floor, that we need to raise it. We detail this in what we call our Advanced AI Framework as well. Thank you.

Councilmember Hudson. Thank you so much, Chair. We heard earlier from a panel of whistleblowers who raised a number of flags, and one issue raised was that of self-improvement among AI agents. What is your company's core belief around the role of self-improving AI in your overall strategy? And to be more specific, is it core to your AI strategy or a less significant part of the strategy?

Strategy. Could you continue your race to the top or the bottom, I suppose depending on one's perspective, without self-improving AI agents or with significant guardrails on the way agents self-improve? And my second question is, when a new AI data center strains a local grid or water supply, the costs can end up in everyone's utility bills while the benefits go mostly to the company.

Who should bear those costs, the company that creates the demand, the utility that builds the capacity, or the public that shares the system but played no role in building it? Thank you. Thank you for your question. I'll start off by addressing the question about RSI. To say that we believe that RSI may be one of the most consequential safety and security issues of the coming decade.

And that's why OpenAI has advocated for federal legislation specifically focused on RSI to include independent technical evaluators of internal lab practices, as well as the pace towards RSI development. We've made that work public in our Frontier Safety Blueprint. It's available online, and I'm happy to share it with, with you as follow-up. On the case of electricity costs, look, OpenAI is committed to paying our own way on energy.

But we're also committed to building with and for communities. All of our Stargate sites, which are our infrastructure sites, have what we call a community compact that's developed with the community and addresses how we're going to work together. For example, we recently announced $80 million of community benefits investment at our site in Georgia. Thank you. Councilmember, in relation to your first question on recursive self-improvement, if I may, I might just go back to a point that I mentioned earlier in the day in relation to recursive self-improvement, that we believe that committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure that this technology is developed safely.

In relation to your second question, in terms of electricity and data centers, we believe that customers should not bear costs, and we pay the full cost for electricity used in our data centers. I also might just mention as well our Futures for Everyone Fund. which is a $1 billion investment in relation to communities that we partner with and work with, with our data centers.

So we also take a very prudent approach to self-learning inside our model training. We use AI models to help us evaluate, benchmark, and refine other models. But our use of this type of iterative model development is confined to narrow supervised processes. That's human-supervised processes. to keep progress verifiable and within defined safety limits. We also manage the risks of fully autonomous model self-improvement through our Frontier Safety Framework.

So we're very mindful about RSI and are quite controlled in the way that we think about self-learning internally. On data centers, similar to the other companies represented today, we believe in paying our own way and have pledged to do so, particularly when it comes to the energy demands of data centers. We've also invested in a wide range of energy sources to ensure that our data centers have an independent energy source that does not add— that does not strain the grid and in fact adds back to the grid so that in the long term it will benefit wider energy customers.

Thank you. Likewise, on recursive self-improvement, we take that extremely seriously and have from the start. I think from my purview that there are 3 really important things here. The first is just measurement and understanding. I'm glad you raised this question because we think this is something people need to talk about. In late spring, we published what I think is a first-of-its-kind assessment of of how it is happening or signs of it happening at Anthropic happens in a lot of ways and it's nuanced.

In late summer, we published a follow-up that went into further detail of how we should understand and measure self-improvement. But that's not enough. And that's why you saw us and our CEO publicly call for pacing the frontier, because candidly, the technology is moving very fast and we think we would benefit from having more time. And this is the second component.

And then obviously the third thing that you need to do here is figure out what the right guardrails are, but additionally what level of technology development we should be going for, as well as what safety processes we need along the way. And I think this is just where we're at now. And I'm glad that we're having this discussion.

On data centers, this is outside of my expertise, my understanding. is that we pay our own way and we're committed to contributing and working with the communities around them. But I'll have to defer to my excellent colleagues who work on it. Thank you all, and thank you, Chair. I do want to just make my one little note that it seems like the self-improving aspect is absolutely core to everyone's AI strategy, and I That's certainly concerning given the testimony we heard earlier this morning.

So thank you, and I look forward to any follow-up. Thank you so much.

Councilmember Schulman, followed by Brooks Powers, followed by Brewer. Thank you. Good afternoon. I am Councilmember Lynn Schulman. I'm chair of the New York City Council Health Committee. As a breast cancer survivor, I am looking forward to the day AI can help find cures for that and other diseases. That said, there is a dark side to AI intervention in healthcare.

Most viruses that are found in nature can be contained fairly quickly, but an artificial pathogen has no constraints. In August, a team from Stanford used an AI model to design an entirely new virus from scratch. In addition, Anthropic has its own AI pathogen biology lab. What measures are being put into place such as placing— I'm not, I'm not gonna say guardrails, I had that originally— safeguards on DNA synthesis to prevent rogue viruses from being created and replicated at will to jeopardize the public?

Thank you for your question. The question about how AI could enhance or or make worse biological risks is central to OpenAI's preparedness framework. That's the framework that we use to guide our decisions about model risk and whether, whether we have the right safeguards in place. So we evaluate for biological risks as part of that framework. Again, we use third-party evaluators and teamers.

We also work with the federal government as it pertains to national security risks. And we are transparent on the results of those evaluations. So with the launch of every major model, we release what's called our system card. And that card details the evaluations that we performed as well as the results. And those evaluations, as you mentioned, do include biological risks?

Councilmember, this is such an important area and question, so thank you for raising it. Biological risks is one of the core areas, one of the key risk domains that's part of our scaling framework at Meta. We have, as I mentioned earlier, have a multi-layered approach to risk assessment across training, evaluation, and release. And that includes adversarial testing with external experts.

And we work with governments, other stakeholders, and experts in relation to that. And as I mentioned earlier, as part of our commitment to partnering with independent third-party evaluators and assessors also. Yes, ma'am. Similar to our colleagues, we also have bio or biological capabilities as part of our Frontier Safety Framework. As we look at our critical capability levels that I referred to earlier, or the levels at which we would trigger additional mitigations in our models, we look for what's called uplift.

That is to say, the model will be able to assist users in creating information or knowledge that's not otherwise available through widely accessible sources. And we think very carefully about how to measure for bioCCLs and are constantly upgrading and updating our ability to do that. When it comes to those mitigations or those safeguards, as you so aptly put it, we implement those at both the model and the application levels to ensure that casual users cannot access critical bio information.

It is very important to consider the dual-use nature of biological capabilities, of course, because these are the kinds of capabilities that scientists, particularly medical investigators, need in order to make advances in our own healthcare as well. And so we think very carefully about how to release models to scientists that can use the technology responsibly. We did that for the first time with our AlphaFold model, which of course is how millions of investigators around the world are able to look at the folding of proteins and to use that research to find cures for diseases.

So there's a real careful approach that we take to ensure That science can continue, but that dangerous uses of biological knowledge cannot. Likewise, we take this extremely seriously. In fact, if I'm not mistaken, it was my team and myself that developed the first call for action on these risks, the first evaluations, the first tests, the first industry-government partnerships on biological risks and evaluating models and their safeguards. several years ago at this point.

Not only that, I personally think we need to go further here. You mentioned DNA synthesis screening. I think this is personally a no-nonsense approach. There's much else we can do to work on biological defense. And I will, you know, alongside the other companies, we maintain the same and, in fact, try to compete for the best safety practices here.

And you can see some of what we found, I think disclosed for the first time ever in a recent threat intelligence report, where we have observed actors attempting to do biologically risky activities with models. The threat is, I think, clear to us now. We think we need urgent action. But I will close by saying that, you know, as somebody that grew up with a number of chronic diseases, going blind and couldn't walk, I very intimately feel that the upside of the same capabilities is enormous.

But increasingly in my work, I feel optimistic that we can defend against the downside sufficiently. The— if I could just say just one in closing, the upside is, is very— has a great potential. The downside is extraordinarily detrimental to humanity. But you should consider about screening orders for synthetic nuclear acids, which is DNA, and the equipment that's needed to make them, because right now AI is now outperforming PhD-level virologists.

So AI can take this and go somewhere else with it. So that's something that's really important to have those safeguards. Thank you. Thank you.

Councilmember Brooks Powers, followed by Brewer, followed by Narcisse. Thank you, Chair, and thank you Thank you, Mr. Speaker. I'm gonna just run through my questions, and if I have to repeat anything, just let me know. First, before the summit, each of your companies told the public that its frontier models were tested and secure. Since July, all 4 of your organizations have admitted that your AI agents went rogue, and nearly a dozen incidents have been reported across all of your companies.

Can you assure New Yorkers today that none of your agents have— has accessed or attempted to access a New York City government system or a New York City resident's personal data? Next, Southeast Queens has long been a target. Cloning and deepfakes make it easier than ever to impersonate a family member, a bank, or a city agency. What specifically stops your tools from cloning a real person's voice or forging a document?

What protocols are the leaders of AI development putting into place to make sure AI does not take information from hate sites and disseminate it to further racist and anti-Semitic or Islamophobic discourse? That was all. Sorry. Thank you for your question. So I'll start off by saying I am not aware of any incidents affecting New York City, but we have an ongoing investigation into potential incidents going back to November 2025.

And as part of that investigation, we are notifying any affected third parties and making the results of our investigation public. So I can commit to you that if, if there are incidents that affect New York City, we will make the third parties aware. On your question about impersonation, OpenAI, our usage policies prohibit the use of our tools for impersonation fraud and scams, and we enforce on those policies.

And I would say more broadly, the approach that we take to safety is holistic. That involves filtering unsafe training data, performing evaluations that are very detailed, working with third parties on those evaluations. Ahead or alongside our launches, as I mentioned, we make the results of our evaluations public and summarize the work that we did and any residual risks in our system card.

And then we monitor for failures of our safeguards so that we can continuously improve safety. Councilmember, in relation to your first question, I'm not aware. In relation to your second question, We have robust and comprehensive policies in place as well as teams of experts who work on frauds and scams. And I'd like to follow up with you with those experts to discuss that one more in detail.

In relation to hate content, I first of all just want to acknowledge how upsetting such content is in terms of people who experience it. But we are— our goal at Meta is for our AI offerings to be responsive, accurate, balanced, and unbiased. I'm sorry, was that answer just now to the last question about the not taking the information to create the negative discourse, the racial discourse?

Yes, that's correct, Councilmember. I was speaking to your third question in relation to to that type of content and our objective in terms of ensuring that our AI offerings are responsive, accurate, balanced, and unbiased. So, have you enhanced what you're doing considering that, you know, it has been something that has been found on a number of different platforms?

Have you all been, like, staying up to course with enhancing whatever those processes are? Yes, Councilmember, thank you for that question. If it's okay, I will have our content teams and specialists connect with you afterwards so that I can provide you with precise information in relation to that. Thank you. Councilmember, thank you for these questions. I'm also not personally aware aware of any New York City resident being impacted by the 3 incidents that I mentioned earlier.

I do want to note that in our case, in all 3 incidents, the model stopped itself once it realized it had left the test environment. We tend to think of this less as a misalignment event and more of a mistake event. One of the websites, for example, that was accessed had the same name as the fake website being used in the test environment.

And once the, the agent realized that it was in fact a real company, the agent ceased its activity. We have a number of tools across our platforms to manage likeness and identity, and we've also supported a range of different regulatory proposals regarding this topic. We proudly support the No Fakes Act, which would protect an individual's digital replica and give people the right to control the use of their own voice and likeness in AI-generated works.

We also have a tool called Likeness ID on YouTube that helps creators identify and manage AI-generated content that uses their face or voice without permission. So, so we invest in a variety of different ways to address this critical issue. On the information and getting through into our tools about hate and bias, we have again a layered approach, as I've been saying, to this type of issue as well.

Everything from the model training process itself in both pre- and post-training, all the way through guardrails on our application layer, all of which produce what we're proud to say is the least biased model in the industry. Independent investigations have shown that our models show the least bias, for example, with regards to partisan bias. But we take these issues extremely seriously.

We want to create a fair and neutral set of information for people to access, and we want our responses generated by our AI tools to represent a broad array of experiences and perspectives without bias towards or against any particular identity or ideology. And And while in my role, um, I, I don't cover, I haven't covered the, any effect on, um, New York City, um, I can say that we work with, in investigating, uh, incidents like this, we work with the right parties at the right time to disclose to affected parties the right level of law enforcement, uh, government at the right level as urgently as possible needed to remediate, uh, the effect.

And we commit to, you know, making sure that we follow up with the right parties to do so. Um, and to, to your point on the, uh, the information environment and otherwise. Um, I'll, I'll point to a couple things here. Uh, the first is our focus on testing for Claude itself and our, our language model products approach.

Can you speak a little louder, please? Sorry, can you speak a little louder? Yes, can you hear me now? Yeah, that's better. Yes, thank you. Um, I'll, I'll point to our, our longstanding, um, evaluations of political evenhandedness. handedness, stereotype bias. We train Claude to not comply with particularly harmful requests in part for this reason, and that this is baked into the nature of the model.

Thank you for that, and thank you, chairs. And I'll just say that in closing that I know that the AI companies are pushing folks to allow AI to kind of be developed through people. But I think it's fair to say that using live users to test these products can present dangers and definitely warrant improvements. And I hope that's something that is taken from today's hearing to better improve the, the guardrails that need to exist.

So thank you for your sponsors. Thank you, Chair. Thank you.

We're going to go to Councilmember Murano as a bill sponsor, and then we're going to go Brewer, Narcisse, and Joseph. If folks could keep it to 1 minute, just because we have to get to the admin panel. Thank you so much. Thank you, Chair. Thank you for your testimony today. I know it's a long day for you guys as well.

Because I only have a minute and my colleagues know what a challenge that is for me. I'm going to go through a series of yes or no questions, and then if you could all answer in the order of Dwyer, Friend, Cahill, Graham, quickly, and then if you want to elaborate on any of the areas that I raise, please do so.

First, does your company believe there's any non-zero chance that advanced AI could escape meaningful human control and cause catastrophic harm, as the whistleblowers indicated? I don't know. What I will say is that no level of risk of catastrophic harm is— So if we could just do yes or no now, and then I'll invite you guys to elaborate after everyone answers.

Please, Ms. Friend. Dwyer? The purpose of our Frontier Safety Framework is to guard against that possibility. So do we entertain the possibility? Yes. As a responsible actor in this space, we, we entertain the possibility. Councilmember, our scaling framework includes loss of control in terms of the risks that we evaluate and mitigate. So, and these are my other yes or no questions. and handle them however you want.

So why should the public accept, essentially trust us as the safety standard from the same companies racing each other to build the most powerful systems? Additionally, will you commit today that if your own safety team says a model is too dangerous to deploy, commercial pressure won't ever overrule that judgment? Additionally, has your company ever continue developing or deploying a model over the objections of members of your own safety team?

And lastly, on my bill, should you be allowed to use someone else's— someone's private chatbot conversation for targeted advertising or behavioral profiling without their affirmative consent? So I will start and try to get through as many answers as possible. One, OpenAI supports the need not only for us to be responsible for safety, but we have supported regulations at the federal level as well as at the state level because we think it's important not only for us to be responsible, but for governments to hold us accountable.

On the question of of whether we would continue developing models if they were unsafe. OpenAI has recently announced a pause of some of our training activities because we did not deem moving forward to be safe. We have also delayed the release of models because we didn't think that they had sufficient safeguards. So we've done it before. We will do it again.

And more broadly, I will say that all of our decisions regarding frontier safety and the release of models are governed by our preparedness framework that is available online. So we're transparent about what goes into those decisions. And then with the release of every model, every major model, we release a system card which details the work that we've done, the evaluations that we've conducted, and any residual risks.

Councilmember, in relation to your first question, I see it differently. We are accountable. Our framework and preparedness reports are public. Meta uses outside expertise. We signed the accord at the White House on superintelligence, which committed us to partnering with independent auditors and assessors. For example, in relation to, I think, your Your second question, we delayed the launch of Muse by several months to focus on safety and security, and we acted on this without waiting for industry.

It was just part of our normal day-to-day work because it was the right thing for us and it was the right thing for people to do. And then in relation to ads, I would just note that for our Muse product, conversations and interactions don't do not inform ads on Meta's other products. Sir, Google— at Google, we do use external evaluators in the process of research and development of our AI models and our systems.

We have supported audits of our safety procedures as well. And you will see in our recent paper from over the summer, which is called A Pragmatic Approach to AI Regulation in America, we do think that AI needs to be regulated and it needs to be regulated well. We've supported a range of different legislative proposals at the federal and state levels to this end.

We also, to your question about conversations, we give our users controls over whether their activity in our tools can be used for training our AI models. So yes, we think users should be in control of that. And Councilmember, I would point to our longstanding call for the involvement of government at the federal level and the state where necessary and appropriate, because this goes beyond the labs and we don't think the labs should be checking their own homework.

Not only this, but we have demonstrably withheld models when we felt it was not safe to widely release. This year withholding our Frontier model, probably the most notable instance of this, instead deploying it to cyber defenders in order to secure the world in advance of a world of more powerful models that could exploit software vulnerabilities from wherever.

This is part of our process. We go over and above the process. We detail our defaults in the responsible scaling policy. And to your other question, we believe that Claude should act unambiguously in the user's interests. And for that reason, we do not run ads, nor are Claude's responses influenced by advertisers. Thank you so much.

Councilmember Brewer, followed by Narcisse. Thank you very much. I wanted to know 3 quick questions. If a model provider you rely on, you or a subcontractor, did not complete the third-party validation for a particular package, how would that affect your product's availability? And also, how do you decide who gets the validation, number one. Number 2, China. The speaker asked about China and competition, and I thought the answer was, don't worry, they need information from us.

So I'm wondering, what do you send to China? And number 3, ICE. I was upset that one of the whistleblowers said that Google has been working something to do with ICE and Minnesota. So my question is, do you all support support, not helping ICE at all. Obviously, you are made up of immigrants. We should not be shipping immigrants out of this country.

Please answer those 3 questions, and thank you. So I'll start with the question on third-party evaluations. We work with a range of third parties to evaluate our models. That includes independent investigators as well as with the federal government. And then we detail who we worked with and the results in our system card when we released our model.

Decisions to launch our models are described in our preparedness framework. And so that describes how we integrate all of the results of evaluations and make decisions with respect to launching models. So I would point you you there on that particular issue. On the question of China and the PRC, I would like to get back to you on that particular issue.

I, although I have a national security background, I'm not working in that role at the company, and I want to make sure I get you good information. Similarly, respect to ICE, I'd like to get back to you. I'm not aware of any activities working with ICE, but I don't work on our government sales team, so I'd like to get back to you on that.

Next. Madam Speaker, thank you very much for, for your 3 questions. In relation to— 3 questions. Pardon? 3 questions. Yes, 3. Sorry, my accent might have gotten in the way there. Apologies. 3 questions. questions. For third-party validators, we also work with a range of validators and assessors, and we also provide this information in our publicly available preparedness reports.

In relation to China and competition, I might just mention that, you know, Meta is a proud American company and proud to serve billions of people around the world who rely on our services. We want to ensure that the US and democratic countries lead on AI. And one thing I would just note here is in relation to the importance of open weights and American open weights and the criticality of ensuring that American innovation continues to lead in relation to the availability of open weight models.

And in relation to ICE, if I may, I would like to connect you with, with our teams and experts in that space. I don't wish to, to be imprecise in responding to you. Next. Ma'am, in terms of third-party evaluators, we also work with the USKCE, which is the organization underneath the Commerce Department that does testing of advanced AI models.

We also work with a range of third-party evaluators. It's a very nascent ecosystem. And one of the challenges in evaluating AI models, of course, is to get organizations that both have the technical competency to do evaluations well, but also have the subject matter expertise that we may be looking for. Earlier, one of the council members mentioned concerns about bio, for example.

We have third parties that have the bio expertise that's necessary. But as we have this national conversation about independent verification organizations and third-party evaluations, keeping in mind that we still need to mature this ecosystem of independent evaluators, I think is really important. On China and competition, I'd like the opportunity to give you a more complete answer We don't conduct AI research or training in China.

I know that we are part of the conversation about what it means for America to lead in AI, where we are in the technological race with China. I will just mention that we focus a lot on who's at the frontier, which country has the most cutting-edge research in AI. But we talk much less often about deployment of AI, and there I think we should really keep an eye on the Chinese.

They have emphasized deploying AI in their economy quite a bit, and and that's a place where I sometimes have concerns that the United States is in danger of falling behind China. As to the whistleblowers allegations from today. I don't have any specifics about that, but I'd be happy to ask the public policy team in New York to get back to you right away with any information that we have about it.

Next. And at Anthropic, the use of third parties for verification, developing and using evaluations for models has been part of the process from day one, including with not just external parties in industry or nonprofits, but also with governments such as the US Kasey. This has been foundational. As my colleague said, it's pretty nascent. We think it needs to be urgently expanded and not just used for pre-deployment testing, but also to understand models and their behavior after deployment and the safety processes at companies.

In the case of China, I think it's very clear that that the first thing that we need to do is maintain an American and democratic lead in model capabilities. But we're very clear about the threat elsewhere as well. So we have publicly documented Chinese attempts to distill our models, essentially take the content and capabilities for their own.

And we work to ban those. We have prohibited use of models in China. We do not serve there, and we have also documented the use we believe by related parties of Claude or the attempt to for cyber offensive reasons. We are extremely serious about this, and we think it's a very substantial priority. And this is one reason why we have advocated at the federal level for export controls on the most important chips that power AI innovation here.

And on your third question, while I'm not. I'm not across the issue enough to know what our engagements are. I will say that our usage policy, we ban surveillance, censorship, prohibited law enforcement and criminal justice uses for this purpose. Thank you. Thank you.

We have Councilmembers Narcisse, Joseph, P. Sanchez, and then Councilmember Phil Wong. So Narcisse. Good afternoon. Thank you, Chair, and thank you, Madam Speaker. I want to say thank you to you too because you guys stay here with us, and I'm happy to see at least 2 women is on the panel to talk to us. Hollywood give us clues all the time, but we kind of slow in taking, you know, hints and clues.

But no companies in this world, I feel that, should be self-regulated. What I'm sitting here for about 4 hours, and like I said, thank you to you, privacy, transparency, and safety. So now, do any of those companies that you can say there is a standard across the board for all the companies to follow? Because apparently not. Now, do you regulate hospitals?

I mean, do you have Do you have AI— sell AI, sorry. Do you sell AI products to New York City hospitals? And if so, do you think that a licensed clinician should sign on all the diagnostic tests? Because I heard if Ms. Friend stated that you diagnosed with— you went through a process of cancer, if I'm correct.

Yes. So you understand that. People need to be diagnosed by a human. And I can give you example, like for colonoscopy, sometimes the bubble, according to AI, will tell you it's a polyps. In the meantime, it's not polyps. It's not polyps, it's a bubble. But AI cannot differentiate that. So therefore, you need a human to see through.

And right now, I don't know if you read New York Times. How some of your nurses— I'm sorry, can you conclude the question? Yeah, I'm going to close my last point. The New York Times already telling us that there's a report that AI is getting confusion between the hospital and the insurance company. So, I don't know how you see that happening, how we can make sure that AI is being monitored.

Yeah. So, thank you for your question. ChatGPT for healthcare is deployed, for example, in Memorial Sloan Kettering Cancer Center. But to your point, we believe that doctors should remain in charge. We do not think that AI should represent itself as a licensed professional. And we also think, though, as we've talked about earlier, there's tremendous potential to use AI in healthcare settings, particularly for administrative uses, documentation.

I mean, doctors are already overworked, and giving them an opportunity to free up their time so that they can spend more time with their patients is an outcome I think we can all agree would lead to better care. Councilmember, thank you very much for these questions. In relation to your first question on self-regulation, we believe that the, the Accord is a really important step in relation to the concerns that you mentioned.

In relation to healthcare, Meta AI is prohibited from acting as a licensed professional, including a doctor or any other medical professional who's licensed. There are, as my colleagues have mentioned, phenomenal benefits in relation to the use of AI in healthcare. I know that there's phenomenal work happening at NYU Langone who actually, I think, has a specialist division that's actually investing and working specifically around AI in healthcare settings.

And similarly, at Columbia Irving, which is really doing phenomenal work in that space also. Ma'am, thank you so much for those questions. On whether or not a company should be self-regulated, I again want to emphasize at Google, we believe that we are regulated by existing regulations, sectoral regulations, regulations, state, federal level regulations all apply to AI. And we have supported further regulation, particularly of frontier AI at the federal level, but also other AI-directed laws at all levels of government.

So no, we should not be self-regulated. We should be within the regulatory frameworks we already are. On the questions about healthcare, so we certainly agree that doctors should stay in the lead and that AI is a tool for clinicians. It's not a replacement for them. In my particular case, I was diagnosed by a person, but often these days when I go in for checkups and screenings, my clinician will ask if they can use an AI tool to take notes while we're doing the checkup?

And I always say yes, because I know that our clinicians are incredibly overtaxed by a range of administrative requirements. And if AI can take some of that burden off and allow my doctor to look me in the eye when we're having an appointment, I'm going to say yes to that. But similar to our colleagues, we also have guardrails on our Gemini app app so that users will never think that they are talking to a doctor, that they are talking to a licensed physician.

And the Gemini app will also flag for them that if they're seeking medical attention, they should get it from a professional. Thank you. Likewise, we've been very clear that we support the role of government here. At the federal level, we've advanced a proposal for meaningful involvement to set standards and hold companies to account for safety processes, evaluations, security, and the like.

And we welcome the accords that were signed very recently. To the point of medical usage, if I can use a personal example here, about 6 months ago, I experienced some symptoms in a longstanding eye disease that might make me go blind at any moment. And that night, actually, at about 1:00 AM, it was Claude that helped me identify that it was serious enough that I needed to be treated very, very quickly.

Claude triaged a very large part of it for me, but what was most important was I could use that to get to the hospital very fast. It's very clear that both can be transformative and powerful together, certainly in my case, and I think countless others. But this is also one that's closest to my heart that we need to be very careful about.

Thank you. Thank you all so much.

Up next, we have Councilmember Joseph, followed by P. Sanchez, followed by Wong, Phil Wong, and then Schwang. Thank you, Chair. Question— 2 questions. Several of you sell AI products directly to colleges and universities. When students use those products, is anything they enter used to train your models? And beyond training, who can access those conversations? And what limit do you place on how colleges can use them, for example, for discipline, academic integrity investigation, or student profiling?

The next one is, given that AI is already reshaping entry-level work, as we know, as is shown in this new Center for Urban Futures report, what responsibility do you have to ensure these tools help students build useful AI-augmentative skills rather than automating away the very tasks through which young people learn. What responsibility do your companies have addressing these early career impacts, and how can government help?

So, to your first question on privacy, we believe that privacy is central to our mission. We give users control over their data. That includes the decision to opt out of training, but all users have their data protected and private by default. So it is separated from their names. And again, they have the option to opt out of training and to request that their data is deleted.

On the particular question of colleges and how ChatGPT is used in colleges and education, I would like to get back to you on that. I don't have an accurate— I don't know the answer. And so I'd like to be able to follow up because we do believe that education is critical. AI literacy is critical. We've supported numerous bills at both the state and federal level supporting AI literacy.

That is both training for teachers to ensure that they can lead adoption. in schools, as well as improvements in curricula to make sure that students learn how to use AI safely, as well as learn core subjects that are critical to being able to use AI safely and responsibly. To your question about jobs, we think one of the most important things that we and policymakers can do right now is to continue to collect data on the question of AI impacts to jobs and make that data public.

We make our assessments of how people are using ChatGPT at work public. We share that. And we also have worked with policymakers, including supporting the Bipartisan Workforce Transparency Act to make or to create voluntary pathways for employers to Councilmember, thank you very much for these questions. In relation to colleges and universities, I would like to have my— the specialists at Meta who work in that area connect with you on the details.

But specifically in relation to privacy and data, this is absolutely core to everything that we do at Meta. For example, our Muse product, you can opt out of the use of data for training and you can ask Muse to, to forget any information that you provide to it. And of course, all of the data is, is, is within your control in terms of what you provide to it.

I, on the AI literacy point, it's so important. Um, I actually came across a really great example from New York City, and I, I wanted to, to call it out because I thought it was great. Uh, it was called Building Minds, Building Communities, their digital empowerment by design work. And I thought it was a really, really great example, um, of the use of AI technologies for AI literacy and kind of a, a flywheel in terms of benefits there.

Um, in terms of, um, the entry-level point Absolutely understand the concern in relation to that. At Meta, we believe in placing personal superintelligence into the hands of everyone and for invention, not automation, and for augmentation, not replacement. And we really believe that this will have a transformative impact in terms of enabling and empowering people to pursue their own And their own values and direct AI towards those.

Yes, ma'am. Similar to my colleagues, we believe users should have control over their own data. Okay, we're not going to have outbursts from the chamber or you will be removed. You decide. Sorry. Go ahead. As I was saying, users have control over their own data. Across our AI surfaces and can opt in or opt out of their data being used to train our models.

In terms of AI use in colleges, as I said earlier, we have tools for educators and administrators to take the lead in how AI is used in the classroom and used for students. We think that colleges should be able to adapt to the needs of their student populations. I, like Shane, was looking into how different colleges and universities in New York City use AI and saw that NYU, for example, has a very helpful page where they go through all of the AI tools they make available to students and give them resources. for learning both how to use the tools and then also training them in the kinds of things they can do with those tools to further their education.

I would flag that they use Gemini Notebook, which is a study guide creation tool. So NYU is a good example, I think, of a college thinking carefully about how AI can benefit their student population. And as to the impact on early careers, we think this is also an incredibly important area of research. It's part of understanding the broader impacts on the economy and labor.

We recently released our own AI and Economy Atlas, which also releases data about real-world uses of AI in the economy and at jobs. And we're hoping that continuing to add to that data set will help economists study and then develop policy recommendations for managing the labor transitions of AI. And at Anthropic, maintaining user privacy is very core to our commitment on safety and security.

Mm-hmm. Users can choose whether or not they want their data to be used or contribute to training a model, for example. But in addition to that, we've tried to pioneer the science of privacy-preserving safety maintenance when you need to ensure that users stay safe on the platform. To the point of education, while it's not my area, here's what I understand.

We partner with universities. We're very interested in this question. There's a nifty feature in part of our partnership called learning mode, which allows a user, a student user, to interact with the model in a more discursive way that can help them think. But in addition to this, the impacts are really important. I think we are relatively well known for our work on studying the economic and other behavioral impacts of models.

We have a comprehensive— and I think we were the first to release this economic index which details this. But in addition, we've done research into how students are using models because we want to figure out how to encourage the use of models to complement the development and skill development of students. Thank you.

Up next is Councilmember P. Sanchez. Councilmember Phil Wong, and Councilmember Susan Chuang. I will remind my colleagues to ask your questions up front, and if you have questions that are specific for some of the members of the panel and not all 4, specify. Thank you. Councilmember Sanchez. Thank you, Chair. My first question is in connection to heartfelt testimony of Layla Turner Scott before Congresswoman Jayapal last month.

Ms. Turner-Scott's son died following chats with OpenAI that instructed him that incompatible drugs were compatible. She highlighted that throughout the review of logs, ChatGPT responded to her son with phrases like, I've got your back, you can talk to me about anything, and even I love you with heart emojis. She stated, and it seems true, that the goal of ChatGPT was not to keep her son safe.

It was only to keep him using the product. What updates have your companies made to their models to engage children and prevent addiction? Are you limiting your agents from giving medical and health advice without a license? And my second question is regarding the fact that members of the public and enterprise clients are sharing more and more information with your companies.

You have access to copyrighted information publicly available. How do your companies— excuse me, how do your products currently use information gathered from users? In earlier exchanges, Chair, if I may, several of you stated that users can opt out of sharing your data. What is the default option? And how are trade secrets and other confidential business or government information from enterprise clients being used to train AI?

Who have you paid for the valuable data that you are using to train your models? So I want to start off by acknowledging the tragic incident that you reference. Any incidents of harm that results from our product or any other product is tragic, and we take it seriously. I'd like to tell you about ChatGPT for Teens, which is— was recently released to focus specifically on users ages 13 to 18.

And is designed to be safe by default. That includes safety protections on the type of content that is available, limiting content that might be related to suicide or self-harm, eating disorders, violence, or graphic— graphic or sexual content. We also include take-a-break reminders to remind teens to stop using ChatGPT. And we have parental controls, optional parental controls that allow parents to add additional layers of safety onto their teen's experience.

One thing that I will highlight here is if a teen indicates signs of suicide or self-harm, we have a feature that allows the parent to be notified. On your question of privacy and data, as I and my colleagues have shared, we take privacy seriously, and our core value is that users should have control over their data. And I'm happy to follow up on specific details of how those controls work. at a later time.

What is the default option? I'm sorry, I didn't hear you. What is the default option with respect to who owns the data and how it is used? So users have control over their data. Everything is privacy protected. What is the default option? Everything is privacy protected by default, and users have the option to turn off their data for training or request Councilmember, um, first off, um, I just want to acknowledge the terrible tragedy, um, that was, um, a terrible tragedy in relation to, um, the Turner-Scott family, um, and how distressing and upsetting that is to, to hear, um, and, uh, you know, my sympathies and condolences to that family.

In relation to how Meta approaches minors and young people, creating a safe and productive experience is absolutely imperative to Meta, and we're committed to providing safe and responsible AI, especially for teens. And we have strong policies and safeguards in place in relation to that. For example, by default, minors only have age-appropriate interactions, meaning that content would not be out of place on a 13+ movie, for example.

We also prohibit our AI offerings from providing licensed professional information like medical information. And direct people towards appropriate resources. In relation to training data, we take creativity, copyright, and IP rights seriously and believe our practices are consistent with the law. We train on information from the internet, information licensed from third parties, and information shared across our services, and we provide all of the information in relation to how we use that data in our Privacy Centre.

In relation to MEWS, the— as I mentioned, I think a little earlier, individuals interacting with MEWS can opt out of their interactions for training. Ma'am, I want to highlight a few protections that we have in place with kids in mind in particular. One is for what we call persona, which prevents claims of sentience or simulating relationships and role play.

That helps prevent kids from thinking they're talking to a person and developing a relationship. We also prevent sexually explicit outputs from chats with kids. And we also have, for all ages, protocols on suicide and self-harm where we refer users who are expressing suicidal ideation or other types of self-harm to crisis resources. And that approach draws from our deep experience fielding such queries on Search.

When it comes to training and the default question you asked, for those over 18, it is default to use chat data to train our models to improve, but all users have control over the ability to turn this feature off or to control the length of time such information is kept. For users between 13 and 17, it's default off.

Teens can choose to turn it on if they wish to train a model through our Personal Intelligence Program. And for under 13, it is default off as well, and we never use data for under 13 And— Thank you. At Anthropic, we have limited the Claude platform to those only 18+. We're concerned about child well-being. We monitor and detect for any users that might be 18 or under.

And when that flags, we investigate to make sure that we could remove their access if so. On training data, I will have to get back to you with more details as it's not my not my role. My understanding is that all users must select which option they would like, but I would refer you to our public documentation where we we describe this in detail.

Thank you all.

So up next, we have Council Member Phil Wong, Council Member Shuang. And then Epstein. Okay, thank you, Chair. My question pertains to the self-improvement process, the recursive self-improvement process, as well as the training activities of your company's AI model. And it is my understanding that when you— when it goes through the self-improvement process, it generates code that will itself will run next time around, and then it will be a smarter AI model.

Is that right, OpenAI? Yes or no? Yes or no? It generates code, so it'll be a smarter AI model. I'd like to get back to you on the specific definition of RSI. I'm not sure that what you're saying maps exactly to the technical definition we use. Okay, uh, Anthropic, does it generate code automatically to make itself smarter as part of the process?

Yes, we've detailed, uh, publicly multiple times the degrees in which models that generate code are being used. Meta, does your model self-generate code as part of the training activities or self-improvement process? Councilmember, I'll need to get back to you and have our technical experts follow up Okay. How about Google? Yeah, same here, sir. I'm not sure we would describe it exactly the way you are describing, so I want to get you the precise technical definition.

Well, here's my question. If, if your model improves by itself by generating code, is there a human being, is there a team of human beings that checks the code to make sure that it is benign and not misaligned or malicious? Can, can each one of you answer that? So as part of our safety process, I've mentioned our preparedness framework that outlines how we evaluate models and how we assess risks across a wide range of national security risks.

So we do robust evaluations of a set of risks, and we make those results public as part of our system card, which we release with every major model. Councilmember, as part of our scaling framework, we outline considerations that we are taking overall to maintain human control of model development. Yes, sir. As I said before, we have supervised processes to keep progress verifiable, and we have a range of different techniques to test the model and its performance to ensure that it's staying within within defined safety limits.

We also have an AI control framework that we released earlier this summer, which similarly takes into account how we're ensuring that our systems are continuously under human control. And we manage the risks of fully autonomous model self-improvement through our Frontier Safety Framework. And yes, humans and the systems that we put in place monitor at every level of the model during training, prior to release, post-release.

But this is an important question because the models are clearly getting better faster, and so we think more is needed here. Okay, thank you. Thank you, Chair. Thank you so much. Thank you.

Councilmember Schwang. Thank you, Chair. I have 2 questions. The first one, what are you doing to offset the massive impact of data centers on the environment? The second one, are you tracking how inaccurate statements or decisions by your AI tools are impacting security? So, I'll start by addressing your question on data centers and the environment. As I mentioned before, OpenAI's data centers that we build, they're called Stargate.

And for each of our data centers, we build them with and for communities, and the environment and the environmental impacts on communities is part of how we think through our community compacts. For every data center, we have released a community compact that details our commitments to communities. And that includes a range of things like paying our own way on energy, as well as commitments to community commitments to work with, with the local governments.

But I think the key point is the community compacts are driven by local concerns, and so we want to meet the needs of the community, and we work with them to do that. Councilmember, in relation to data centers and the impact of the environment, Thank you very much for asking that question. I know it's top of mind for a lot of people.

Our sustainable infrastructure commitments are core to our philosophy around our data centers, including consumers should not be bearing the costs and paying our own way in relation to electricity. But I would like to connect you with our data center team so they can provide you with more specifics on that. How about the one question about how inaccurate statements by your AI tools are impacting security?

Do you guys track those data? Councilmember, I think in relation to inaccurate information. AIs can make mistakes, and I think that's important in terms of acknowledging that AIs can make mistakes. We disclose this. We're very transparent in relation to, to, to that in terms of our AI offerings. And I can offer a perspective here that we think, and we've worked very hard this year to try to improve the world's security because we think models might, if not properly safeguarded, harm security, specifically the cybersecurity of critical infrastructure or otherwise.

This year we withheld a model to be able to give beneficial access to defenders to have time to patch systems with security vulnerabilities. And I can say working with training teams and otherwise, we work to make the model better at making sure the code that it writes is more secure, for example. Thank you so much.

Up next, we have Councilmember Epstein, followed by Marte, followed by Santuoso. Thank you, Chair, and thank you for being here. I know this is a long panel. So I have a couple questions I'll ask, and hopefully you'll be able to answer for the entire 4 members of the panel. Can you confirm whether you think self-regulation has failed failed so far?

And if it has failed, which I think we effectively think it has, do you think that third-party audits, especially when AI or tools are being used to highlight risk decisions, should be regulated? And whether government plays an effective role in that regulation? Second, about liability. When AI violates the law, what liability do you think the company should have for illegal violations, whether criminal or civil liabilities.

Seems like there's some culpability that should be in place. And then the turnaround time when these violations happen, when some, you know, the Hugging Face or some other violation happens, what responsibility do you have to notify the public and the impacted people? Would you agree to some short turnaround time, whether it's 48 hours or not? And finally, the, you know, We've seen a lot of records about a compilation of these violations of what's happened.

Are you willing to commit to releasing the list of places where these, uh, kind of rogue actors happened? I know you've released some letters to the EU. Will you release it to the city council as well so we can see what's been happening in the United States and around the country? So on the question of self-regulation, um, OpenAI was proud to support the recent Superintelligence Accords.

We've also supported voluntary frameworks in the Biden-Harris administration, but we don't think voluntary frameworks are enough. And that is why we have supported regulation for frontier safety, both at the federal level and at the state level, and we'll continue to do that. On your question regarding transparency of incidents. So OpenAI is currently investigating past potential incidents of misalignment, and if we discover those incidents, we are notifying third parties and making the results public.

I'm also happy to say that we have a reporting framework now where anyone in the company can report suspected incidents of misalignment. We evaluate those incidents, remediate them, notify third parties, and make the results public. So we're doing that already. We think transparency is really important here to help policymakers like you, as well as researchers and the rest of industry, understand the risks and improve safety.

And can you share a list of those incidents so we have those? ones you've already determined? Yes, they are all available online, and so I'm happy to follow up with the website where they're all publicly available. I don't know who goes next, Chair, but I'm— Cahill? I'll go next. Thank you, Councilmember. Thank you, Chair. In relation to your first question, Councilmember, I see it differently in that sense that we're very much accountable We're accountable.

We publish our framework, our scaling framework publicly, preparedness reports. We use outside expertise in relation to the accord which we signed with our peers. We committed to partnering with independent auditors and evaluators. And we also consider that there's a considerable body of existing federal law, which also applies. In relation to incidents, we have a robust global response program in place in relation to incidents and a very important part of our overall processes, and we report incidents as required by law.

In relation to public disclosure, I'm only aware of one incident, and over the summer, and we have our public post in relation to the details of our independent, independent review of that. Sir, as I've stated in earlier answers, we believe that AI is covered by existing regulation at both the federal and state level. But where there are gaps in existing regulation or existing law, or where in particular frontier AI poses potentially new risks, we have supported regulation at the federal level and also regulations both federally and at the state level as well.

On liability, we think it's important to attach accountability and and responsibility to the party that is closest to the knowledge about harm and to the possible cause of harm as well. So we really see this as a value chain approach to thinking through AI liability. In many cases, existing torts already have procedures for assigning accountability depending on the— the particulars of a case.

And then I'm happy to have the local team in New York follow up with you about our 3 incidents. And at Anthropic, we have long pushed for 2 things. The first is for the industry itself to push its own way up the safety frontier. We call that the race to the top. We've tried to do that from, from day one, always improving and always trying to create a gold standard so that everybody can rise together.

I have personally thought that some of that has been very, very positive. However, we have to go further, and for that reason, we welcome the role of government. We have detailed proposals at the federal level with our Advanced AI Framework, and we've supported state level for this reason as well, including ambitious legislation in Massachusetts. On the point about liability, I'm fairly far from being a lawyer, but I can say that Anthropic's committed to following all relevant laws.

And to the excellent question on how we should handle incidents, I think a couple principles are really important to me having seen some of these incidents up close. The first is you have to triage, investigate, and disclose fast and be transparent about it. Um, I think everybody here agrees with that. At the same time, we need to take the safety of those affected extremely seriously.

In some of these incidents, disclosing certain information, uh, might put them at further harm. And so our first priority is making sure that those affected would not be further harmed. But at the same time, we need to very ambitiously and urgently disclose so that all can benefit. Yeah, I appreciate the follow-up on some information, especially if there's some violations and the timeline for getting people input in.

And I thank the chair for the time. Thank you so much.

Up next, we have Councilmember Marte, followed by Santos-Suoso, and then Wong. Thank you all for being here and staying so late. And I want to thank the speaker and chair for hosting this hearing. You know, I want to just get some clarity. One of the bill sponsors of the RAISE Act said that OpenAI was actually against, strongly against the initial draft language of the bill, and you stated that you have been supportive of the RAISE Act in Albany.

So I kind of want to know, just because it seems like you guys have been supportive of a lot of legislation based on your testimony, what specific legislation are you against in New York State or on a federal level? So thank you for the question. OpenAI worked with other companies on the New York RAISE Act. And we do support the New York RAISE Act, which passed.

We also, as I've mentioned, supported legislation that actually goes a step beyond the New York RAISE Act. For example, in Illinois, that requires independent third-party audits of safety frameworks. Are there any pieces of legislation, and I guess this is for everyone, that you are against on the state level or federal I go next. Councilmember, how I think about it generally is the importance of ensuring that we have a robust, uniform federal standard.

I think I mentioned earlier in the day that AI is transcontinental. It is inherently interstate. So it is important that there is a robust, uniform federal standard in relation to how AI is governed. I think we would align ourselves with those comments, sir. I think the most important thing for us to focus on is getting a strong federal framework, particularly around frontier AI, and also working through an overall regulatory framework that's going to ensure AI safety and also ensure that we can deploy and leverage AI across the entire country?

Yeah, and likewise, I don't know what our position is there exactly. What I do know is that we have outlined and supported our specific federal legislation proposals. We think a core part of this, for example, needs to involve independent evaluators, and we are, you know, longstanding supporters of the role of government here. Thank you all so much.

Up next, we have Councilmember Santos-Suazo, Juan, and then Hanif. Thank you, Chair. I primarily have questions for just Mr. Graham and for OpenAI as well. This morning, we heard testimony from a former Anthropic employee that based on his time in the company, he believes that it is more likely than not, uh, that misaligned AI will lead to the quote unquote total extinction of humanity.

Another whistleblower who testified this morning gave the, gave this a 1 in 3 chance. Mr. Graham, if I understand it, you are part of the red team, which is tasked with identifying and addressing the riskiest capabilities of AI. So my question for you is how many people are on your team? Current estimates, about 25 just on my team that we have.

I will say we have a number of teams that work on similarly catastrophic risks in various roles, number into the hundreds at Anthropic. And how many people work for Anthropic overall? My current understanding is maybe a bit more than 5,000. So 25 people are working on the riskiest forms of AI in a company that has 5,000 employees dedicated to advancing just the models without regard for safety or for that level of risk?

In reality, there are, if you look at all the roles focused on catastrophic risks, my personal estimate would be likely several hundred at least. This is distributed across our safeguards team that focuses on ensuring that the models or any misuse for catastrophic risks does not happen. Our alignment research team that focuses on notions of lossless control, for example, and between them at least several hundred.

And do you feel that is sufficient or do you feel that you need to dedicate more time and resources and people to the riskiest capabilities of AI given the level of risk? Yeah, our view is that we put a lot of resources into this and this is what we're known for, but at the same time, we need to continue doing more.

This is a part of Anthropic that continues to grow quite substantially, especially as the models become more capable. I don't mean to cut you off, but I just, I only have so much time, so I want to ask the same question of OpenAI. How many people on your team are dedicated to addressing and identifying the riskiest capabilities of AI versus the number of employees in your company overall?

Do you find that sufficient? So I lead the policy team, but I'm happy to get back to you on numbers of people in the overall company as well as in our safety teams. But at OpenAI, we don't believe that it's just— that you should just rely on our safety teams and our evaluators. I'm sorry, I don't— again, I'm sorry to cut you off.

I just only have so much time. Can you tell me how many people in your company are currently working to address the risk of the need— of the potential extinction of humanity? As I said, I lead the policy team, so I don't have that number, but we not only have teams that are— Thank you so much. I appreciate it.

All right. Thank you so much.

Up next, we have Councilmember Kwon, followed by Hanif, followed by Aviles, followed by Gutierrez. Thank you so much. This is for all panelists. What is the exact failure rate or benchmark threshold that would cause you to halt the deployment of a new model? And if your model successfully generates a bioweapon design 5% of the time during internal testing, do you still release it?

And then I have a separate question. Will you commit today to providing exact redacted logs of the prompts that successfully jailbroke your models by biosecurity features, filters during your internal testing? So to your question on statistics or probabilities, I don't have a specific number. But what I can say is that no level of catastrophic risk is remotely acceptable.

And we don't believe that we should be training models unless we can make an extremely strong case that we can keep them under human control? Councilmember, I would just, I would just note that we are committed to building AI safely, and we have laid out our in detail how we approach this in our scaling framework and in our preparedness reports.

In relation to further detail, I would need to get back to you on specifics. For the other panelists? Yes, ma'am. I don't have a specific number for failure rate that would cause us to make a non-launch decision, although I do know that we aim for safety and performance standards ahead of allowing ourselves to launch. And if we're not meeting that, it's not a good enough model or product for us to release.

I'm not personally aware of any biosecurity evaluations where jailbreak— successful jailbreaks happened. So I'm going to have to go back to the teams and ask if they've ever seen such a thing and get back to you on that. Thanks. And as I mentioned earlier, I think we were essentially the first team in the industry to design and test models for this purpose.

We outline a very nuanced process of how we decide how to launch with respect to biological capabilities, for example. And in fact, in the past, we have held a model because we needed more time to assess its biological capabilities and proactively implemented safer safeguards for this reason. To the point of jailbreaks, this is a great question. One of the hard parts is sometimes you do or don't want to share a jailbreak depending on whether it would put other models at risk.

And so we regularly publish our research and alignment reports in order— with all the information that we possibly can. that would be safe to do so. And we try to set the industry standard there. Right now, I know you're not in the room, but behind me we have the safety claims and we have all the incidents that have happened, including biosecurity, which you have heard also from Councilmember Lynn Schulman, who continues to lead us as a committee chair for Health and Hospitals.

So can you please clarify? So Anthropic has been the only one that gave me a clear answer. So for Anthropic, what is the exact failure rate and benchmark threshold? threshold? And the rest of you have not given me that answer. And then we'll conclude and move on to Councilmember Hanif. The answer is it's nuanced. What we do, if you would like the sort of complex answer, is we detail multi-hundred pages of reports, several dozen of which are usually allocated to biology, every time we launch a model.

You can see the exact levels of capability and the exact reasoning that we put in when we look at a number of different tests that have different failure rates that allow us to make that decision. I think that highlights just how much effort we put very specifically into not just biology, but cybersecurity and self-improvement and alignment as well.

And everybody else? Because if what you claim is true, then you wouldn't have released these models that are causing these harms. But I will press on. We have to move on. Thank you so much.

Councilmember Hanif, followed by Councilmember Aviles, followed by Councilmember Gutierrez. Thank you, Chair. On February 28th of this year, 120 schoolchildren and 35 teachers were killed in an airstrike reported as the deadliest US attack on civilians in 35 years. The military has stated the target was misidentified by Project Maven, which originated with Google and Amazon Web Services. and later incorporated Palantir software and Anthropic's Claude.

To Mr. Graham from Anthropic, your usage policy prohibits facilitating any act of violence. What mechanism enforces that prohibition in a classified deployment, and how would your company learn of a violation? And has your company ever terminated or suspended a government customer for violating its usage policy? If so, how many times? First, I, I really appreciate this question.

Um, in, in my role, while I don't have the exact answers to that, uh, I can point to our very public positions on— Could you just share them out loud here? I, in government, we've, we've taken very public positions against the use of, for example, domestic mass surveillance, um, the use of autonomous weapons or development of autonomous weapons.

How does a company learn about That violation, and then could you also just respond to the piece about the termination of a government? I'm not entirely sure of our process on on sort of government sort of monitoring. I'd refer to my my my colleagues there. Thanks.

Thank you so much, Council Member Aviles, followed by Gutierrez to close. Thank you so much. To clarify, to close this panel. Oh. So I'd like to ask a quick series of questions to each of you. In this first set, can you just say yes, no, and we can continue to move forward. So does each of your companies sell or license its products directly or through intermediaries to Immigration and Customs Enforcement or Customs and Border Protection?

I don't know, but I'm happy to get back to you on that. I'll get back to you on that, Councilmember. I'm also so sorry, I don't know, but we will get back to you. Okay, nobody knows, but it has been pretty documented that a couple of— from Google and in particular OpenAI, ICE has been utilizing your technology.

So if you could please provide to this council each of those agreements, if in fact you have them with each of those agencies. Will your company commit to not using New York City resident data for law enforcement or immigration enforcement applications? For— excuse me. Will your company commit to not using New York City resident data for law enforcement or immigration enforcement applications?

I'm sorry, I'm going to have to get back to you on that. I work on the policy team, not with our government contracts. I'm afraid also we'll need to get back to you, Councilmember. I have to give you the same answer, Councilmember, but I'll ask the lawyers to let us know. Do any of your companies audit downstream law enforcement use of its products?

I don't have the answer to that. I'm happy to follow up. Councilmember, we'll follow up with you. Same here. Thank you. Likewise, we can follow up. Thank you.

Councilmember Gutierrez. Thank you all. I just wanted to ask on a number of questions of issues that were raised today. I feel that every single person on this panel has said safety is of the utmost importance. But I don't believe— I don't feel that. I don't feel that in the responses. It feels too nuanced. And I would like to understand specifically, in all cases, as much as you can share, how are you directly ensuring safety of individual users when there is a data breach, when there's issues of surveillance?

To my colleague, Councilmember Shahana Hanif's questions about AI and violence, what is specifically that you can tell to New Yorkers here, that New Yorkers that have signed up to testify, what is the specific safety that you are ensuring? Make me believe that it is of the utmost importance to you all. That's my one question. Yeah, thank you for that question.

So safety is a top priority at OpenAI. We build our models to be safe throughout the entire development and deployment process. That includes removing unsafe data from training. It includes training the models to behave safely and evaluating them to be safely— to behave safely across a range of risks, from cyber to bio to child safety. And then we monitor our safeguards for any potential misuse, and we enforce on our usage policies.

But to your specific question of what I am personally doing, I am not on the safety team. Policy team, and I'm also a longtime public servant, and I share your concerns about safety. And I think there's an important role for governments to play, and that's why I am proud that my team is working with governments to support regulations both at the state level and at the federal level, because we have strong safety practices, but we should also be held accountable for them.

Councilmember, thank you very much for this question, and overall, thank you for deepening the conversation on this and other important topics. I would just reiterate that Meta is committed to developing and deploying our AI offerings safely. That is across the board, both in terms of our models and also our systems. and then our usage policies. So we cover the entire spectrum in terms of our AI offerings.

Ma'am, at Google, we have been committed to not just the safety of our users, but the safety and security of the open web itself for over a quarter of a century now, through our own experiences in our search platform, through everyday tools that users rely on like Gmail, like YouTube. Security and safety is incredibly important to us because, quite frankly, if people did not trust our platforms and our tools, they would not use them and we would be out of business.

So for us, safety is a critical business imperative. It's also an imperative as part of being interested and committed to an internet where people can get information that is useful to them, that can help them live better lives. And finally, we invest in the safety of our tools so that we can deploy them to solve some of the world's biggest challenges.

Everything from natural disasters, warning people ahead of time about floods and fires, to breakthroughs in diseases, to unlocking 200 million, all of the world's known proteins so that we can someday cure all disease. To us, safety is just a part of this broader mission to use technology and to use AI in particular to make people's lives better.

And if I can share a personal example of this, I joined Anthropic when we fit around 2 lunch tables. And the reason I did was because these were the people that I saw were taking it most seriously, that the most important component of getting this technology right was making it safe. And I can say that, but let me demonstrate it.

From its outset, we are a public benefit corporation legally, so we are legally obligated to weigh the mission of SafeAI, not just profit. And at every step of the way, frequently at meaningful cost to ourselves, we have pioneered or tried to pioneer the science of safety. Everything from training itself, the fundamental research came from us, the evaluations, the pre- pre-deployment testing regime, the monitoring and control, and just this year withholding our most frontier model at the time at great cost in order to deploy with defenders to ensure the world is safe.

This is literally our DNA. This is why I'm here. This is why my team and I wake up every morning. And I could tell you for a long time about all the other parts of the stack, but I hope that resonates. Yep. Okay.

I'm going to thank this panel for testifying. The council still has many questions that we feel we did not get answers to today. We will be sending each of the companies a list of these questions that we would like answers back in writing. But I thank the companies for being here today answering the members' questions. And thanks to the public for their patience.

We're now going to transition to the next panel, which is the city administration, and we'll call on them right now. Thank you. Thank you. Okay, please remove them from the chamber. Thank you. Welcome to our town hall. Okay.

Now we will move on to hear from the administration, and today we will hear from Sarah Milstein, Deputy Commissioner of Strategic Advisory Services at the— at OTI, CJ Dixon, Head of the New York City Cyber Command, Benjamin Krakauer, First Deputy Commissioner of New York City Emergency Management Department, and Sam Levine, Commissioner of the New York City Department of Consumer and Worker Protection.

Thank you, Chair. Please rise your right hands. Thank you. Do you swear or affirm to tell the truth and respond honestly to councilmember questions? I do. Commissioner Sarah Milstein? Yes. Thank you. Head of Cyber Command CJ Dixon? Yes. Thank you. Deputy Commissioner Benjamin Krakauer? Thank you. I just want to acknowledge that it was yes. And Commissioner Samuel Levine?

Yes. Thank you very much. You can begin with your testimony. And Deputy Commissioner Carlos Ortiz. And I apologize, and Deputy Commissioner Carlos Ortiz. I do. Thank you. You may begin. Great.

Good morning, Speaker Menin, Chair De La Rosa, and council members. My name is Sarah Milstein, and I'm the Deputy Commissioner for Strategic Advisory Services for the Office of Technology and Innovation, OTI. I'm joined here today by my colleague, CJ Dixon, OTI's Deputy Commissioner of New York City Cyber Command and New York City's Chief Information Security Officer. I'm also joined by Samuel Levine, Commissioner of the Department of Consumer Affairs and Worker Protection, DCWP; Carlos Ortiz, DCWP's Chief of Staff and Deputy Commissioner for External Affairs; and Benjamin Krakauer, First Deputy Commissioner for New York City Emergency Management, NYSEM.

In my role at OTI, I oversee both, both the Office of Algorithmic Accountability, OAA, and New York City Cyber Command. We appreciate that the council is holding this timely hearing about the risks of AI technology. Thank you, Speaker, for getting representatives of the companies that develop, deploy, and sell AI tools to the general public to speak to the capabilities, limits, and risks associated with these systems at this hearing.

As a representative of OTI, I'm here today to provide information to the Committee of the Whole about how we serve the city's technology and cyber defense needs and to address the proposed legislation that potentially impacts the administration. For those unfamiliar with our work, OTI's core mission is to use technology, data, and design to make the city work better for New Yorkers.

We achieve this through a wide range of digital products, infrastructure, data systems, and shared expertise. In other words, we use technology to help agencies make every New Yorker's experience of city government better. For our discussion today, we can speak to the work we're doing around agency use of algorithmic tools and our cybersecurity program. To touch briefly on the AI arm of OTI, I'm pleased to share that the Office, the Office of Algorithmic Accountability, OAA, has been formally established and 6 new positions have been created in the FY27 adopted budget to staff it up.

The hiring process for those roles is actively underway. The OAA's duties include analyzing algorithmic tools submitted by agencies to determine whether there is risk that the proposed tool could result in discriminatory decision-making, conducting and publicly reporting on pre-deployment assessments, creating and maintaining a public-facing platform for submission of comments, establishing a protocol with the Department of Investigation for receiving complaints from the public, promulgating rules establishing basic compliance standards that all agencies must meet in developing, procuring, deploying, and using public-impacting artificial intelligence, and reporting the results of pre-deployment assessments and audits conducted by the office.

Overall, the office is progressing as planned, and we remain focused on building the capacity needed to carry out this work effectively.

I will now turn to the legislation on today's docket that would impact OTI. Introduction 161 of 2026, would amend annual reporting on algorithmic tools to include their impact on city employment with a focus on potential changes to funded agency positions, salaries, and duties. OTI is currently undertaking a study on the impact of algorithmic tools on municipal employees pursuant to Local Law 25 of 2026.

This study will provide insights into the extent to which algorithmic tools, including artificial intelligence, impact city employees and the administration of their duties, including their hiring and work functions. While we agree that it is important to examine the impact of AI on our workforce, the bill as written would not produce new insights beyond those in the larger, more expansive directive of Local Law 25.

The next 3 bills involve New York City Cyber Command. This office plays a vital role in protecting and defending the city and its residents from the impacts of cyberattacks. On a day-to-day basis, we provide 24/7 security services and assist agencies in bolstering their cyber maturity. We work collaboratively with agency partners, as well as state, federal, and private entities to safeguard the essential services and data New Yorkers depend on daily.

Our core mission is to make New York City the most cyber-resilient city in the world, and we take this extremely seriously. New York City is a target for cyberattacks with a technology landscape that is unparalleled among other cities and states. This requires a unified, comprehensive defense against constant cyber threats. Our key duties include setting information security policies and standards for the city, directing the city's citywide cyber defense and incident response, deploying defensive technical and administrative controls, and providing guidance to City Hall and agencies on cyber defense.

In my fairly new role at OTI, it is a privilege to work with our staff and our agency partners in furtherance of this critical mission. New York City Cyber Command's alignment within OTI has placed the team in a strong position to monitor and respond to wide-ranging cyber threats. Thank you. Cybersecurity is continuous work. In my role, I'm confident we can continue to adapt to a constantly evolving threat landscape.

Simply put, New York City Cyber Command has a 24/7 apparatus in place to defend the city's systems from malicious attacks regardless of the origin.

With that in mind, preconsidered introduction of 2026-2602 would make it unlawful to market, offer for sale, sell, or deploy an artificial model— an artificial intelligence model in New York City that has not received third-party validation or does not have a technical capability to be shut down by a human operator. The bill would require the director of the Office of Cyber Command to promulgate rules regarding what the third-party valid— what the third-party validation assessments, certifications, and disclosure would entail and the qualifications for third-party validators.

To be clear, Cyber Command's core mission is to defend the City of New York's infrastructure from cyber attacks. We are not subject matter experts on the qualifications for an entity to be to be considered a third-party validator. Further, nor— neither an OTI nor Cyber Command have experience relegating— regulating the private sector in the way that this legislation would require, and neither would be the best choice to lead this regulatory regime for consumer products.

We defer to DCWP to speak about the private sector policy and enforcement piece. To that end, DCWP supports the intent of this legislation. They recommend including robust recordkeeping requirements to better allow the agency to conduct affirmative enforcement.

They also note, based on conversations with advocates and sister agencies, that the administration is not certain about the availability or ability of such businesses. We look forward to hearing from those stakeholders today, including on whether initiatives like this one could help spur a market for such services. Preconsidered introduction of 2026. would require the Office of Cyber Command to establish standards and procedures for contractors to identify the occurrence of a reportable AI safety incident.

To take a step back, I'd like to lay out how AI risk and incidents are categorized according to the National Institute for Standards and Technology, NIST. Under AI under attack, AI cybersecurity incidents are traditional cybersecurity compromises where the AI system is the target. For example, data poisoning or jailbreaks. Misuse or malfunction, AI-induced incidents, would be harms resulting from the use or failure of AI aside from a cyber compromise.

In this category, the AI system itself behaves exactly as it was built or trained, but the real-world outcome causes harm. These incidents are not cyberattacks. We understand the council's intent here, given the publicized incidents— instances of AI systems operating outside the bounds of their intended use. As I noted earlier, the risk landscape is constantly evolving, and we are continuing to evolve our strategies to keep the city's infrastructure safe.

But not all threats related to AI systems are necessarily cyberattacks, and the legislation as written does not address that. Cyber Command would be a critical stakeholder in this discussion, But it is important to emphasize that we need to work with our state and federal partners to address these issues holistically. I also want to provide further context on breaches of security generally.

Agency disclosure of a breach of security in any form is governed by Section 10502 of the Administrative Code. There are extensive protocols for interagency coordination in the event of any breach of security. These protocols are aligned with the New York State Stop Hacks and Improve Electronic Data Security, SHIELD, Act. Current law, policies, and protocols are in place for incidents described in the legislation.

Further, the threshold of 24-hour public notice is inconsistent with current law and would compromise the city's investigation of a cyber incident. Preconsidered introduction of 2026, 2606 would require the Office of Cyber Command, in coordination with NYSEM, to develop or update an emergency response plan concerning AI system-related threats to New York City's infrastructure, operations, public health, and welfare.

I'd once again like to emphasize that there are laws, policies, and protocols in place for breaches in security regardless of actor. However, we are happy to discuss more broadly with council how we already coordinate with NYSEM to coordinate and engage with partners to restore services.

Finally, I will provide DCWP's feedback on the remaining legislation that would impact the administration. Preconsidered Introduction 2026-2605 would allow any person to submit a complaint to DCWP alleging that a person or entity has violated certain provisions of law related to artificial intelligence. DCWP strongly supports the intent of the— this bill to establish a civilian enforcement structure for AI violations, especially given widespread reporting that concerns being raised by AI companies' employees are being ignored.

However, there are fiscal and operational concerns with the volume of complaints that the agency would receive and the resources and expertise required to handle each individual complaint. Therefore, the administration believes it would be important to coordinate closely with mission-aligned enforcement agencies at the state level. The New York State Attorney General just recently announced its secure whistleblower portal for New Yorkers to report unsafe and illegal conduct related to AI technology.

This could be an opportunity to partner on this existing program to more effectively enforce complaints on behalf of New Yorkers. Preconsidered introduction. 2026-2599 would impose requirements and restrictions on companies that provide chatbots, including requirements related to data security and privacy, the right of chatbot users to access their own data, and restrictions on how chatbot providers can use chatbot user data.

DCWP has concerns about this bill. While we support council's goal to better regulate companies' deployment of AI chatbots and believe the approach taken by this bill is sound, the implementation and enforcement of this bill's data security and privacy requirements would require a deep technical understanding and investigative ability that is outside of the agency's scope. To be clear, DCWP is committed to regulating harmful practices, including relating to chatbots, that fall within the agency's investigative ability.

To that end, the administration is working on a rules-related on rules related to the use of AI chatbots to better regulate this space for consumers. The administration is also invested in supporting the Attorney General, the New York Attorney General, in their efforts to rein in data abuses, for which this bill may serve as a model. Preconsidered introduction of 2026-2603 would require any advertisement that promotes an artificial intelligence AI model in the city To disclose whether such a model has undergone validation by a third party, it also would prohibit any such advertisement from including any material any materially false or misleading statement regarding the safety of an AI model.

DCWP supports this bill to require disclosure of third-party validation of any advertisement of an AI model. However, as mentioned above, the administration is not. Uncertain about the availability or abilities of such businesses, and we want to be clear that any advertising that uses false or misleading statements is currently prohibited under the city's consumer protection law and will be enforced with the full scope of DCWP's authority.

We look forward to working with council to ensure the law includes effective enforcement mechanisms and to continue addressing issues within DCWP's scope of work as it relates to AI. Thank you for the time.

This panel will now take questions. Great. Thank you so much for your testimony. We appreciate it. I've got a number of questions. So OpenAI sent the City Council a letter on October 1st recommending that New York City better connect its existing cyber and emergency management capabilities— and I'm going to just reading from the letter— with the escalation, containment, and recovery capabilities of frontier AI companies.

So has the City of New York this year met with OpenAI, Anthropic, or any other frontier AI company to discuss how the city would coordinate with them in the event of a serious AI-related cyber or public safety incident? Thank you for the question. I'm going to defer to my colleague, CJ Dixon. Thank you. So I would like to note that New York City Cyber Command follows the NIST cybersecurity framework for all of our cyber programming.

That is everything from identify, protect, detect, respond, and recover. The basic first principles in cybersecurity have not changed even with the advent of artificial intelligence. And we have been in conversations with various frontier models about how we may integrate their tools and software to better enhance the functions that we already deliver to the city. So in reference to your question, yes, we have been in conversation with the frontier AI companies to enhance the way that we perform cybersecurity.

However, those basic cybersecurity controls have functionally remained the same. So you've met with OpenAI and Anthropic in particular? We have met with Anthropic. I have not met with OpenAI since coming on board on May 5th. Okay. Are you planning on meeting with OpenAI? Only based on a— only dependent on our needs right now. We do not know definitively if there is a valid reason for us to meet with any frontier AI company in service of our— I guess I'm just concerned by the statement that nothing has changed in your protocols, given that we're now dealing with a rapidly escalating technology that, as we certainly heard today and we've been hearing now for the last couple weeks to months that there are serious safety concerns.

So I just want to get an answer in terms of what proactive steps is cybersecurity taking to meet with some of these frontier companies and ensure that our systems are safe. Acknowledged. And we have— it does not mean that we have not changed our protocols. Some of the risks inherent with artificial intelligence are unrelated to cybersecurity. In those situations where an AI model presents a cybersecurity risk, those AI models will still exploit the exact same software vulnerabilities that are inherent in computer technology as they've always been.

So our protocols have not had to change to address the fundamental issues inherent in computer technology, only the speed by which we must react in a cyberattack.

Okay. I, I am concerned by news that City and State reported on Friday that the AI czar, Jiaohao Chen, who is our chief technology officer, resigned, I think, a week to a week and a half ago. And reports saying that he resigned in frustration that a lot of positions were being not hired or being held up by OMB.

And that relates to my second question, which is Councilmember Gutierrez— I think she's still here. Yes. And I passed bills last year to create the Office of Algorithmic Accountability. And so it appears that Mr. Chen's resignation, from what we're reading in the press, was related to that and the fact that that office was supposed to be up and running by the city in June, but it is not up and running.

And in fact, in your testimony, you said you're now posting for the positions. It's now October. Thank you for the question, Speaker. Jihao Chen served as Director of AI and ML at OTA— at OTI for 2 and a half years. He did important work in the previous administration to help define and understand AI use throughout the city.

He's been very helpful in getting new OTI administration up to speed, and we wish him the best. Thank you. We are very excited to be staffing up OAA. I— the law went into effect in December. I was brought in this summer to help stand up the office. We got funding in June for— in the FY27 budget. And we've been actively working to hire for the roles, for the director role for OAA, and look forward to announcing something quite soon.

Thank you. soon. We're really grateful to the council for having stood up, for having helped us stand this up. I think the work is going to be excellent. It just seems that this is an inopportune time for his resignation, and I am concerned that the office was under Councilmember Gutierrez's bill supposed to be up and running in June.

In addition, this companion bill, which was my bill, which we passed a year ago, required OTI to engage in rulemaking around this Office of Algorithmic Accountability. But no, OTI did no rulemaking whatsoever. So it did not comply with that bill. Instead, OTI issued updated guidance, ignored the rulemaking process. There's been no proposed rule, no public hearing, no public comment period, and no adoption into the rules of the City of New York.

Thank you. Thank you for the The question speaker. So OTI is fully compliant with the law. The rulemaking piece of the law had us this year submit the initial— I don't want to say paperwork, but the initial points for our rulemaking. And the actual rulemaking will happen in this fiscal year. We are also fully compliant with Local Law 35, which was brought under the Office of OAA as part of the December 25th law.

We've been in the process of fulfilling that report for 5 years. We've done it on time for 5 years in a row. That's the law that requires agencies to disclose their use of public-impacting algorithmic tools. And we're on track now for on-time reporting for March of 2027. So we're really excited, actually, to be doing that work, very proud of it.

We just last week sent out the email to all of the city agencies to kick off this reporting cycle and are holding on this Thursday the orientation meeting that gets everybody rolling. Okay. I mean, just to clarify for the record, we don't believe that's in compliance at all. The office was supposed to be up and running in June.

I'm sure Councilmember Gutierrez will talk about her bill, but the office was not up and running in June and was supposed to do the rulemaking. When do you feel then the law department will be— you mentioned law department. When can we expect to actually see the rulemaking? This is in the March 2017 '27 deadline? Yeah. Okay.

So what safeguards are currently in place to prevent city employees from entering confidential private city data, whether authorized or unauthorized, into AI systems such as Copilot that are being controlled by outside corporations? Thank you for the question. Our contract with Microsoft which runs Copilot Chat, prevents them from using city worker data to train their models. And the way they host data for us also ensures that we're HIPAA compliant, CRIS compliant, that the data is is kept carefully and securely, that there isn't risk of the data being leaked either to Microsoft or to the public.

So you can confidently say that no private or confidential city data has ever been obtained by an AI company and then used to train an AI model? I'm— thank you for the question. I'm confident that our agreements with Microsoft prevent them from taking such actions. And does OTI currently measure the impact that AI tools are having on the city's workforce?

Yeah, thank you for the question. We, per Local Law 25 of December 2025, we are engaged in a workforce study to understand the impact of algorithmic tools on city workers and automated employment decision tools, and we will be having— we will have results from that to share next year.

Okay, I'm gonna pass it over to Chair De La Rosa. Thank you. Thank you so much, and just continuing on that same question, you all testified that my legislation, my Bill 161, basically does the same thing as Local Law 25. One, we look forward to seeing the report. Local Law 25, to my knowledge, requires a one-time report that's gonna come out of that study.

Mm-hmm. 161 is more comprehensive and is requiring that report to be updated yearly. And as this— and as this technology is evolving, in my opinion, hence why I put the bill forward, you know, more consistent reporting is going to be needed. especially as we begin to see how the workforce is impacted. Before I was the Technology Chair, I was the Labor Chair.

We have a wonderful Labor Chair in the House, and I'm grateful to her leadership. But this is an issue that impacts not only organized labor, but I also see it as an issue where we're leading the nation, really, Yeah. in how the workforce will react to this evolving technology. And so is there anything that you can share with us right now on where the workforce is and what those impacts will look like?

I know you're working on the study, but is there anything you can give us forthcoming? Thank you for the question, Councilmember. So the initial stages of the report are in part about determining how we even define impact. It's a pretty broad term and could mean many different things. So we're in this phase of determining that now so that we're able to produce a report that's really meaningful for city workers.

We certainly support the intent of your bill, and I would be interested in further conversation about the annual nature of the reporting. I appreciate that. Some of the information that we're looking for, for example, is I think an important question. How many funded agency positions, for example, will be eliminated due to the use of these tools? The agency vacancies has been something that this council has really been looking at, especially as we look at civil service reform across the board and bringing people into municipal service.

So, you know, we do want to understand— Thank you. Sort of displacement and what that looks like for the workforce. We want to also understand reduction of hours, overtime, which this council has not been shy about questioning the administrations, past administrations and present administrations about. We also want to know about subcontractors, right? There's a ton of city contracts that are in play right now and want to know what the impacts are.

So I look forward to continuing this conversation with you all and building on, you know, what we will see from the report of Local Law 25. I also want to say that while my bill doesn't specifically speak to worker protections, this council does believe that worker protections are important, especially as we look at that displacement and what could happen, right?

We just questioned the companies for the better part of the day about whistleblower protections and what happens when this technology gets out of control. So we want to understand how those protections are also extended to municipal workers. Do you have anything in place right now that protects our city's workforce? Yes, thank you for the question. Because this is a little technical, I want to make sure I get exactly the right information, which is that the New York State Civil Service Law Section 8010 establishes a moratorium on discharge, displacement, or the transfer of existing duties and functions currently performed by employees of the New York City and the New York City School District to an artificial intelligence system until 2028.

The civil service law also stipulates that the use of AI shall not alter existing collective bargaining rights, terms of employment, or civil service status. Further, existing collective bargaining agreements may offer employment protections beyond 2028. So with that law in place, we have a little bit of time, and we really appreciate the intent of this law and the fact that it gives us in New York City a little bit of time to figure out how we understand the potential for displacements, for deskilling, for the use of subcontractors that might be displacing other city workers, and the sorts of things you were raising.

So I really appreciate that question. Great. So we'll continue that conversation. I also want to give you a preview. I have a bill that creates a civil service title for AI within our city agencies. And one of the things that I'm curious about is, like this interagency communication around AI. Can you speak to us, where are you right now when it comes to each of the city agencies and evaluating where the agencies are in terms of the use of AI internally?

Yes, thank you for the question. So one of the privileges that we have at OTI as the agency that runs the reporting for Local Law 35 is that we learn from agencies throughout the city how they are using algorithmic tools that have direct public impact. And in the process of collecting that data, we have a fair amount of contact with agencies to be able to learn more and to establish relationships with them so that we can also serve as advisors to them and understand their challenges as this technology unfolds.

In the last cycle, last year, we had 56 agencies participating.

Our first hearing as technology chair was around surveillance, and one of the things that we pretty much grilled these companies about was about self-regulation. And I think from the tone and tenure of this council, we don't believe in self-regulation. However, we've given the grace to our city agencies to self-report the use of AI. I wanna ask you, and I understand that OTI is not an agency that's gonna impose penalties on other agencies for not reporting, but there is a specific concern that my colleagues and I have around self-reporting, especially when it comes to the NYPD's reporting of surveillance technologies.

This council has had many hearings on that. I wonder, you know, under the Office of Algorithmic Accountability, what accountability tools will be put in place to make sure that we are properly regulating the use of surveillance technology in the NYPD and other technology? You know, this council held a hearing just in June around the Department of Education and the use of AI.

So what are some of those tools that we can look forward to seeing? Yeah, thank you for that question. Councilmember. So in advance of having full-time staff at OAA, we have been doing research to understand some of the types of risk assessment and risk management that organizations like ours can do in a situation where self-reporting is what we have been relying on.

We're a little bit early to say how we would implement that in the city, but we are very actively hiring for a director who understands that set of problems and can help us figure out how we come up with effective risk assessment and management that includes a view into city agencies, perhaps beyond self-reporting. Thank you for that answer, and of course we will continue that conversation Absolutely.

I did want to ask you, how, how prepared would you say we are for enterprise security system for autonomous AI-driven attacks? Not just attacks by humans, but AI autonomous attacks. How prepared do you all feel we are as a city? Yeah, thank you for the question. I'll let CJ Dixon answer that. And that is a great question.

And I will acknowledge that this is a very dynamic environment. environment. It is always changing. That's just the nature of technology. New York City Cyber Command, however, provides 24/7 detection and response capabilities through our 24/7 security operations center. We have provided that service to the city since the establishment of New York City Cyber Command. We provide that service whether or not the attacker is human or an artificial intelligence-based attacker.

That attacker would still attack the exact same vulnerabilities in the system, and therefore we would still respond functionally in the same manner. We would just have to do it faster. We continue to evolve with the technology, and we will continue to do so with the best of our ability as the technology continues to evolve. The speaker, rightfully so in my opinion, asked the companies about the possibility of a kill switch and where that kill switch should live.

Do you have any opinions if the kill switch should live in Cyber Command? I agree in principle with the idea of a kill switch. I will say there are some technical barriers that would prevent a kill switch as conceptualized living in any agency within New York City, whether that is New York City Cyber Command, OTI, or anywhere else.

The reason being is the technical reality for these models is they are deployed on distributed systems. On distributed architecture, in order to truly— with a singular button or with a kill switch or anything of that concept for that to actually work functionally, we would need to be able to coordinate across multiple stakeholders that are not actually regionally co-located to be able to turn off everything at the exact same time.

New York City Cyber Command nor OTI does not have the authority to do that. Thank you. authorities or the technical capability to do that. Um, and this runs up— us up against a similar challenge that we've had with other kill switch bills as it relates to computer technology in general. Mm-hmm. As this technology gets more distributed, it gets harder and harder for any singular solitary entity to turn it off specifically.

That is not to say that there is not technical means by which a group of entities with the legal authority to do so could not disconnect a given model from a given function. But in this particular case, we agree on principle but would not be able to be implemented at New York State Cyber Command in practice, not with the technical barriers that are in place.

Okay. I'm sure there'll be more on that.

Let's see. In your opinion, should AI control be recognized a recognized cybersecurity specialty? If so, what should be the training and curriculum included in that? I do not believe every function related to artificial intelligence is necessarily going to be a cybersecurity-specific challenge. There are some intersections between artificial intelligence and cybersecurity as a particular domain, and when those 2 domains interact, that is where certain things should fall within Cyber Command and a cybersecurity professional's purview.

And that is any instance of a cyberattack or cybersecurity, regardless of if the perpetrator is a human or an artificial intelligence. I just want to ask one follow-up. How many people are working in Cyber Command right now? 121. And how many vacant positions do you have? I will need to go back and find that exact number for you, Speaker.

Okay. Thank you. Please provide— if you could please provide that. Thank you. Thank you.

I have a question for our DCWP commissioner and team. What types of AI misuse are you seeing that harm consumers that you're most concerned about? Well, thank you, Councilmember. Thank you, Chair. And it's great to have the opportunity to be here. We're concerned about a wide range of misuse, whether it's people using voice cloning technology that these AI companies make freely available to scam our seniors, whether it's grocery stores using AI technology to fix prices or to personalize prices, something that the Speaker has introduced a bill to prohibit.

We're seeing widespread misuse of these tools. And the other thing we're seeing across the board is an attitude by these AI companies that they bear no responsibility for the downstream consequences. I have to just say, Chair, I was struck by the final answer I heard by the witness for OpenAI, who when asked a basic question about AI safety responded that she works in policy, not safety.

How are you running a company like OpenAI, set to go public, introducing trillion-dollar tools, and you don't have safety as part your policy division, deeply disheartened by what we heard at the hearing earlier today? Absolutely. You should have heard their first answer. It went something like, I don't know. There were a lot of I don't knows, and on behalf of the admin, we hope to give you more fulsome answers.

Thank you, and we look forward to engaging with you in that more broad conversation as well. In your opinion, is DCWP sufficiently staffed to combat this AI misuse, and would adding, in your opinion, a private way of action, a private right of action, strengthen the ability to enforce against violators? Thank you, Chair. I'll be candid. I feel that even as we've— even as under the Mamdani administration, we are adding to our ranks, I believe we are capable of taking on many of the misuses of AI, whether that's DoorDash using algorithms to underpay its workers, something we remedied in the largest labor enforcement action in history, whether it's companies using AI to to harm consumers through surveillance pricing.

We're strongly supporting the Speaker's bill on this. I think we have strong tools. That said, no enforcement agency— and I've worked at the state, federal, and municipal level— will ever have the resources to take on every possible misuse of this technology. That is why, Chair, I would strongly support a private right of action, as contemplated by a number of these bills, to give ordinary people, workers, and businesses the opportunity to hold these companies accountable.

I can guarantee you though, these companies, as soon as the city does that, are gonna run to Washington asking for preemption. Regardless of what they might be saying at this hearing, what really speaks is where their lobbying dollars are going, and that is to DC Congress asking to hit delete on all of these city and state AI safety laws.

Thank you, Commissioner and team. I'm gonna pass it on to my colleagues.

Councilmember Gutierrez, followed by Alderbal, followed by Maloney. Thank you. Good to see everyone. My first question is for Deputy Commissioner. Good to see you. Going off of the speaker's line of questioning regarding the Office of Algorithmic Accountability, can you share— I know you said you're excited about this report, you're setting up an orientation meeting. It feels like not a lot of time.

I guess, what can you share— before the report is scheduled to be released, what can you share about the cadence of how often you're going to be able to meet? Do you expect the report to be released on time? This is not a reflection of you, but I really felt like the previous administration under the previous OTI leadership was delayed in many things.

And we certainly don't want compliance to be one of those things. So if you can just share with me a little bit about the cadence, how many people— you said the funding you got in June, how many people will the office be able to employ, where you are in that process currently? Oh, okay, that was it for you.

I had a question for DCWP, but I can come back. Go ahead, go ahead. Okay, and then for DCWP, I just wanted to ask if you can share a little bit about how the the implementation of Local Law 144 is going, the automated deployment decision tools. Just wanted to see how the implementation of that law is going.

And that's it for now. Thank you. Well, thank you, Councilmember. Let me start by 144. I think this was a really well-intentioned bill to regulate the use of automated decision-making systems for hiring. I think it does have some flaws, though, especially in that it requires companies that use AI systems for hiring to conduct audits. It doesn't actually require that those audits come up clean.

That's really a civil rights question. That is why our administration put out a policy statement last week on behalf of DCWP, Commission on Human Rights, and Taxi and Limousine, making clear that existing laws apply. But I do want to circle back to a broader point that this hearing reveals, which is that bias in decision-making produced by AI systems is a very real concern you're addressing.

These companies would like an exemption from the law. The companies that testified earlier want to assert that if decisions are made in a black box, nobody can be held accountable for it. I applaud this council's effort to actually impute responsibility for these harms to the companies making billions of dollars on it, and we would be very happy to work with the council to work on a revised expanded, strengthened automated decision-making bill that places the responsibility to remove bias from hiring where it belongs.

And do you think that the— is there— are there instances where the harm is happening amongst the city agencies using these tools? I don't believe that. I would defer to OTI and to those in hiring. I don't believe that the city is using AI tools We have a lot of tools to sort resumes, but I would defer to others in the admin— I can tell you, in my agency, we have people reviewing— people reviewing applications.

Just as starting in January when delivery workers get deactivated, we have people reviewing that. When consumers file complaints, we have people reviewing it. We are a big believer in our administration that the people of New York should be served by the people of the city government. Thank you. Part of the package of bills that the speaker mentioned that we passed last last December on Christmas was to get the city and to get OTI to report on those AI tools that every city agency is reporting on.

So we'll know soon enough. But please, Deputy Commissioner, thank you. Yeah, thank you for the question, Councilmember. So I think there's a couple of reports we're talking about here. I want to give a little bit of context. Local Law 35 is the report that requires agencies agencies to tell us about the algorithmic tools they're using that have public impact.

And that report we have delivered on time every year for the past 5 years. The process for it kicked off last week. Last month. And we'll be collecting data over the course of the next few months with the plan to publish it in March of 2027. We have all the indications that that will come in on time as it has in each of the past 5 years.

So, we're feeling good about that. I'm just referring to the report from Intro 199. I'm sorry, could you repeat? I was just referring to the report required by Intro 199. So, that's the package of bills from last fall. Yeah. Thank you for clarifying. So, we are— the director of OAA, which we hope to be able to announce quite soon, will be primarily responsible for the additional reporting that will be required in March of 2027.

And in part because we have this good experience internally understanding process with Local Law 35, we think we're in a good position to move quickly on the additional reporting that will be due in March of 2027. Okay. And you said announcement. Do you have someone already for the role? We will have an announcement, and I hope you have an announcement about the announcement.

Okay, thank you. Thank you, Chair. Thank you. We have Councilmember Audubon followed by Maloney followed by Juan. Thank you, Chair and Speaker Manion. So you said the hiring process for the six new positions that were created in the adopted budget are actively underway. Like Specifically, where are you in the process? How many people have already been hired?

How many people are left to be hired? And what are— what have been the challenges in being able to hire people quickly? Yeah, thank you for the question, Councilmember. So we hope to be announcing a director for the office quite soon, in the next week or two. And we are in the process of defining the additional roles so that they can be brought on quite quickly.

We certainly are challenged to find people who have the right combination of AI expertise and understanding of city systems to be able to, like, hit the ground running. That has been a little bit of a challenge. But we do have— I think bringing in a strong leader is going to help us tremendously. Thank you. draw good people who are going to want to do this work with us.

If I may, Chair. Part of the role is to analyze algorithmic tools submitted by the agencies to determine whether there's risk that the proposed tool could result in discriminatory decision-making. What is your What is your office doing or thinking about in terms of protecting the city's workforce as AI develops and evolves? And you're kind of like the gatekeeper.

Yeah. AI. Yeah, thank you for the question. So in parallel with some of the risk assessment that we can do of the software, we are doing the research associated with Local Law 25, which has us researching how algorithmic tools are impacting— algorithmic tools and automated employment decision tools might be impacting city workers. And I believe that we should have results from that in 2027, and we'll have a much better sense of the landscape of how workers in the city Are impacted.

Thank you.

Thank you, Council Member Maloney, followed by Wan, followed by Brewer. Thank you, Chair. As we all know, generative AI technology is changing how we communicate, work, and how businesses operate. And the same could be said of our government offices, both how we work and operate, either officially or unofficially. So I want to know how New York City. city agencies are adapting to the new reality.

The first, on the opportunity side, there are plenty of examples of government wins of adopting useful technologies for the public good. In Connecticut, they have real-time translation of 80— in 80 languages for these kinds of public meetings to increase accessibility. In California, they automatically check permit applications against 18,000 pages of, of, of code, building code, in order to speed up reviews.

So my first question is, what is our equivalent WIN in New York? And if we don't have one yet, what's the plan to identify those opportunities? And the second question is around oversight. If I may wrap up, OTI published generative AI guidance for agencies last December. So as we're exploring these opportunities, is that guidance binding or is it advice?

What happens if agencies ignore that guidance? Has OTI been prescriptive about a list of generative AI or agentic AI tools that have been reviewed and approved for city use that are use cases that the city officially supports? And then how are we thinking about procurement when these models are constantly iterating and changing every few months, and when new— every few days, truly, and when new technologies are being introduced?

Thank you. Thank you for those questions, Councilmember. I'm so glad you brought up the question about wins, because we're really interested in both effective and responsible use of AI, and I think effective would cover the wins. I don't have a perfect list here, and I'd like to follow up with that because we'd really like to represent the agencies well.

In terms of tools that we have approved, broadly, there are 2 sets of tools that OTI approves centrally, Microsoft Office Suite and other Microsoft tools, and that includes their Copilot Chat and some of their Some of their other tools have AI features built in. And we also have a central license for Adobe Creative Cloud. And many of those tools have AI features built in.

Other tools that— other software tools that are used by agencies are reviewed, for the most part, on a one-off basis by OTI. And we are still determining the best way to move forward. with that process to make sure that we are managing— identifying and managing risk appropriately. I think your question about procurement is an excellent one, and part of what we have started to research ahead of having staff at OAA, we've started to talk with various experts in risk management to understand some of the ways that we might think about risk in an ongoing way, when traditionally all kinds of procurement has— like, there's a gate at the beginning for various concerns, and then it's rolling.

I think you're right. That's not appropriate for this situation, and we should have more to discuss on that in the coming months. But it's an active question that we have as well. Thank you so much.

Councilmember Wan, followed by Hanks, followed by Brewer. Thank you so much, Chair De La Rosa. So for the panel, what I have questions about is building off of what Councilmember Maloney just said. The ones— the agencies are allowed to currently use Microsoft products and also Adobe, but it's clear to me and the public that they are using other AI tools.

So not just the passive tools like using Zoom to transcribe their notes, and for follow-ups. But in addition to that, they're using ChatGPT, they're using Claude, they're using Nana Banana Pro because we're seeing it for agencies like HPD, DOE, DEP. They're creating flyers for public consumption. They're using it for their public emails. They're using it for public messaging.

So what are you actually doing? Because I want to know what the public policy is from the mayor on how the agencies are using AI and how you are regulating it internally as well. And also, I would like to build off of that for city contracts. What are you doing currently for data privacy riders in city contracts for all procurement of AI companies for making sure that our constituent data is not being used to train their AI models?

And is there a clause at all or a rider in the city contracts, especially with Copilot, since that's the only real one that you have? And people are just using their own personal accounts or free accounts for everything else that they're currently using for their day-to-day jobs. Thank you for the question, Councilmember. So let me first address the question about the AI rider.

We shared that with general counsels across the city this summer with an updated AI rider, and I wanna just make sure I've got the right language here, so I'm gonna pull that up. Let's see. Yeah, 'cause my privacy concerns are not just for the contracts that exist, but it is for the free willing use of agency staff that are using non-city enterprise licensed AI.

Yeah, yeah. So I'll first address the rider and then we'll talk about the shadow use. So the rider, as I said, was distributed to city— to agencies citywide this summer, and it's intended to mitigate risks and apply the city's policies and standards to AI products and features. It's available for all city software contracts, and our legal team recommends that agencies use it even where no AI component is currently contemplated.

The rider includes terms relating to intellectual property, security, how the city's data may be used, and privacy. And so that is now active and in use, and I think that with also a proactive component of recommending that it's used for all manner of technology. In terms of the shadow use, which I think is probably both at the individual and at the agency level, we are not in a technical position to monitor that.

But I certainly, we as an agency share concerns about that. And part of what we're interested in doing is making sure that agencies have the tools that they need that are properly regulated. Microsoft tools, for example, that where data can't be shared or used to train their models, that agencies have the tools that they need to do the jobs that they have so that they are not turning to outside tools.

So who is responsible if you are unable to to regulate shadow use of AI tools for day-to-day use, especially with constituent data? Who is going to regulate that? I'd like to follow up with you on that question. Okay. Thank you, Councilmember.

So I know the admin has a hard stop at 5:30. We have about 6 more member questions. So if you all could just hang on tight, we're going to try to get through this. Members, please consolidate the questions. We have our Majority Whip, Councilmember Hanks. Thank you so much, Chair. I'll be very, very brief. So you testified with all the technology that's changing rapidly, even the best can't, you know, keep up.

So you, in your testimony, said the administration is continuing to evolve your strategies to keep our city's infrastructure safe, but not all threats to AI systems are necessarily necessarily cyberattacks. Can you expand on that? Yes. So when looking at the way that it is currently described by the National Institute of Science and Technology, as well as some of the universities and academics who've created some of these systems, there are functionally 2 baskets of AI risk.

There is risk related to AI when an AI is attacked as a computer system, just like any other piece of software. And then there are risks related to artificial intelligence that are truly novel to this new technology inherent in the way that these artificial intelligence models may interact with society when deployed. That second category of risk is a function of often a how much society may trust the artificial intelligence and how reliable that artificial intelligence system may be.

Those two things can run afoul of what we may expect. expect as consumers or as a government entity, completely agnostic of whether or not the system for which those AIs are deployed are actually cyber secure. So a system can be completely cyber secure, but the AI model itself in interactions with other humans in society writ large may take an action that is unexpected and therefore result in some sort of disruption, agnostic of a cybersecurity incident.

That's functionally the 2 categories. That's a very broad breakdown of those 2 categories of risk. So I'm going to keep my question just to that. I think that we should talk offline because my legislation requires the 24-hour reporting requirement to Cyber Command, in which your testimony has some issues, and you talk about how we're going to push this through to federal and state and what specific role do they play in that, how can we make this better.

But I will relinquish that. Thank you, Chair. Thank you so much.

Up next, we have Brewer, followed by Osei. Thank you very much. I know you talked about the Microsoft and the Adobe, but if an agency wants to begin using a new AI model, A, how do they go about it? Number 2, does the city run these models on its own data centers or only on the ones that are under control of vendor clouds?

And then if you have a problem like this, a criteria, what is it? for testing to identify the malicious, misaligned AI product? And then how much does the city spend each year on outside AI consultants? I could go on, but those are 3, 4 questions. I have 10 pages here. Okay. Thank you for the questions. Let's see.

So— The process. Yeah. Typically, when agencies want to buy software, part of the process is discussing with OTI what they are buying and what they will be using it for, and we have a cybersecurity review process that is part of that to ensure that the software that is either going to be bought or built will be safe.

And as as my colleague has said, that would include cybersecurity, includes the typical AI concerns. So that's process. You had— The next question, the clouds, whose clouds are they on? Whose data centers? Who's, you know, where are they? Yeah. Are you running them or is it outside vendors? To the best of my knowledge, there are no city agencies running their own models.

But I would like to— It's vendor clouds, basically. Vendor clouds. Vendor clouds. Yes. So, for example, Microsoft hosts software that agencies use. But I'd like to follow up after the hearing to make sure I've got the exact write information for you. Okay, and if there's something unexpected, misaligned AI product behavior, what do you do? That is a question for my colleague CJ Dixon.

Councilwoman, that depends on what the misaligned behavior is. If that misaligned behavior is specifically a cyber incident, we take appropriate action based on what any cyber professional may do. There are novel misalignments that are born of this technology that do have to unpacked, and we look forward to working with the council as well as our partners across New York City to understand when those misalignments do arise unrelated to existing risks that the city already deals with, how do we actually engage with those particular novel risks.

Have you had some in the last— since you started your job? None related to what Cyber Command's purview is, and that is primarily cybersecurity security issues. We've had no misalignments related to cybersecurity that have resulted in negative consequences for the city from an AI model very specifically. All right, that sounds very— I don't know what it sounds.

How much outside AI consultant dollars are being spent? Councilmember, could you clarify what you mean by AI consultants? Yeah, how many people are you paying? I mean, I've been doing this work a long time, so I know sort of the answer, but how much outside consultants are being paid in terms of AI? In other words, how many people are you paying to work with you on AI issues, outside vendors, consultants?

None that I'm aware of, but I would like to follow up with you to make sure that we have accurate information. In-house, is that what you're saying? Everything to do in terms of strategy, thought process, future, it's all in-house? Yes. Okay, thank you. Thank you.

Councilmember Osei, followed by Deputy Speaker Williams. Thank you so much. I just wanted to refer to the recent hacking situation in Australia. That was proof that AI agents do have that ability. ability to hack into our government systems. As I'm sure you know, in June, a rogue OpenAI agent hacked into an Australian government statistics portal containing private data from Australia's universal healthcare system.

At this current moment, if a similar hack that happened to Australia happened here in New York, do you feel that the city and the admin has a sufficient plan to defend against the hacks and protect any sensitive data? And 2, does the city currently have a responsive liaison from each of these companies to speak to in case of a hack?

To the first question, yes, we do have the people, process, technology, and some of the policies in place to deal with an incident similar to what happened in Australia. The reason being is even though an AI was involved, the hacking process, the attack chain if you will, happened in the same way that if a human being were to execute that attack.

At the end of the day, the underlying infrastructure— I will use air quotes here— doesn't actually care if the person at the end of the other end of the keyboard is a human or a silicon-based entity. The attack chain still remained functionally the same. Therefore, our team is equipped and trained very specifically to identify these indicators of compromise, identify the attack patterns, and then be able to respond accordingly at multiple points.

So we are prepared to address those types of issues. And we do have points of contact when we are working as city agencies with vendors who are under our contract to engage with third parties as they get breached or if their product is the result of a breach. So if they are under contract within the city, by their service level agreements are required to keep us informed of when they have downtimes or when their products are involved in a breach.

Thank you for that response. And in regards to my legislation on an emergency plan, and I know that you indicated that there is already a plan in case, you know, an emergency takes place, am I correct with that assumption? Yes. So there are cyber incident response plans, as well as my colleagues from NISPPAC. There's also ongoing work for risk response plans in general across the board, and that is for every category of risk that the city may deal with that might disrupt city infrastructure or city services.

Is that publicly available, or is that shared with the council? No, that is by design. We do not want to share that publicly, because if that is— the more public aware that someone is of an incident response plan, the more likely they can circumvent it. That is something that you do want to keep close hold and classified.

Okay. Thank you very much. Thank you so much.

Councilmember Hanif, followed by Deputy Speaker Williams. Oh, I thought the Deputy Speaker was going before me, but anyway. I'm sorry. I messed up the— Okay. No worries. I have a question first for Deputy Commissioner Milstein. How does OTI ensure that the automated prescreening tools and fraud detection algorithms used by the HRA do not create disparate impacts or algorithmic bias against low-income applicants?

And then a question I have for Commissioner Levine. The testimony indicated that DCWP supports a complaint process, but you all raised concerns about staffing and expertise. What specific resources would you need to investigate AI complaints effectively? And what protections can New Yorkers rely on while that capacity is being built? Sure. Well, thank you, Councilmember. I can start.

And I want to be clear, AI complaints can take many different flavors. If we hear from a worker who tells us they believe their pay was incorrect, maybe they didn't hit the minimum pay rate because of an algorithm, that is something we can fully address. If we have someone like Jacob toxin come to us and say that Anthropic is developing models that could lead to catastrophic consequences, that is not something we have the capacity to do.

So what we said in our testimony is the New York Attorney General already has an internet bureau set up. They've already put out a call for whistleblowers. I believe they already have staff capable of processing these complaints. We'd be very happy to work with the council to make sure complaints get to the AG. As the speaker has said, thousands of these AI employees work right here in the 5 boroughs.

They know that these companies are making reckless decisions in pursuit of profits. We do want to make sure that these employees have a place to go, and if they come to us and it's not something we can handle, we'd want to work with the state to make sure those complaints can be addressed adequately. But right now, is the city working with the AG's office, or is that a relationship relationship that's not formalized?

We have a very close relationship with the AG's office. We speak constantly. Could I ask you to repeat the question about HRA? Yes, absolutely. So it's basically trying to understand our auto— automated prescreening tools, because I understand that not all— humans are not checking all of the applications. There's some level of automated algorithmic formula that is determining how applicants get pooled when it comes to public benefits.

So I'm curious if— how OTI ensures that automated prescreening tools, fraud detection algorithms used by HRA are not creating disparate impacts or bias. Thank you for the question. We really share a concern about the bias and impacts impacts on not only workers but on New Yorkers. Because this is really detailed and specific, though, I do want to make sure to check back with HRA and make sure I've got the details of this one correct.

So we'll come back to you on that for sure. Appreciate it. Thank you so much. Councilmember, I just wanted to follow up on one point you had about potential resource impacts. I think on that particular issue, we had identified very preliminarily 14 lines for approximately $1.2 million. But I— Wait, could you repeat that one more time? 14 lines for approximately $1.2 million.

But again, that's a very preliminary estimate, and we'd work on the council with the bill for a final version. Thank you.

Deputy Speaker Williams. Thank you, Chair. I am looking at your testimony, and your comments about the chair's bill was that Local Law 25, which was my bill, is the study is currently underway. We reached out, the compliance division of the council reached out, I think twice. I know it just went into effect in January, but it would be helpful to know if you have a timeline.

So I know it might take time, but how much time? Because clearly I think today's hearing is evidence of the importance of really understanding AI's impact. Thank you. impact on the municipal workforce. So just wanted to know if you could share a timeline for us. I know the comments we received back was just that there's no updates, but we haven't received a timeline.

Yeah, thank you for the question, and thank you for the legislation. I think it's really an important study. I anticipate that we will have pieces of it over the course of 2027, and we will share updates as we have them. I think rather, because this is so timely and important, we don't have to just wait till the end.

We can be in partnership with you and rolling out information as we learn more. Thank you. I look forward to working with you on that. Thank you.

And the last question for this panel, this is from Councilmember Selvina Brooks-Powers. She says, my district is home to JFK Airport and to the coastal Rockaway neighborhoods that depend on emergency alerts. What might an AI-driven attack on emergency systems or a major— or major infrastructure look like? Is there currently a plan for an AI-driven attack on critical infrastructure and relevant emergency systems?

And then for NYSEM, during storms and flooding emergencies, it can be hard for New Yorkers to know what they're seeing on social media is real or AI-generated. How is NYSEM working to make their messaging clearly legitimate and to cut through the noise of AI-generated rumors? For the question on notifications, I'll defer to my colleague, CJ Dixon. Yes.

So for an AI-based attack that is a specifically a cybersecurity attack, there are notification requirements that already exist under law. 72 hours for a confirmed breach, 24 hours for a breach related with extortion, and then 30 days for victim notification. So there are plans in place for incident response in the event of a cyberattack against any critical infrastructure owned by the city, and that includes if that attack is perpetrated by an artificial intelligence system.

Thanks. And on the New York City emergency management front, I'll make 2, 2 quick points. One, I think that is a real risk. I appreciate the councilmember raising it. It's one of the reasons that we invest very heavily in backup systems. So several months ago, a very well-known and established notification provider had a 7-hour global outage at which— where notifications were not able to be sent out.

That's something that we take very seriously, and which is why for over a decade now, New York City Emergency Management has invested in the backup system. So as you went across the country, you saw many large jurisdictions saying to their, to their constituents, you know, sign up for this WhatsApp group or follow us on social media if you want alerts tonight.

In New York City, we did not have to do that because we had a backup system that continued to work. So that disruption was averted. As far as, um, you know, misdis or malinformation, you know, generated by AI, that is, That is a valid concern. It's why we recommend that New Yorkers follow us on our official channels, which are verified through Notify NYC.

You can sign up at notify at nyc.gov/notify. You can sign up by shortcode or download the Notify NYC mobile app that we operate in concert with OTI. Thank you so much. Councilmember Brooks Powers wanted me to note that, for example, during the nor'easter, there was a video that was going around about a tornado that apparently caused a lot of confusion and, you know, chaos.

And so she wanted to share that for context. Okay. Thank you so much. I really want to thank this panel. We appreciate your comments on the legislation. We really look forward to working collaboratively with the administration on the various pieces of legislation and appreciate your testimony today. Thank you.

Thank you all for coming. I now open the hearing for public testimony. I remind members of the public that this is a formal government proceeding and that decorum shall be observed at all times. As such, members of the public shall remain silent at all times in the chamber. Yes. The witness table is reserved for people who wish to testify.

No video recording or photography is allowed from the witness table. Further, members of the public may not present audio or Thank you. Please note that witnesses are not allowed to submit audio or video recordings as testimony, but may submit transcripts of such recordings to the sergeant-at-arms for inclusion in the hearing record. If you wish to speak at today's hearing and you have not done so, please fill out an appearance card with the sergeant-at-arms and wait to be recognized.

When recognized, you'll have 2 minutes to speak on today's hearing topic, which is examining the risk posed by artificial intelligence and proposed legislation. If you have written statement or additional written statement you wish to submit for the record, please provide a copy of that testimony to the sergeant-at-arms. You may also email the written testimony to testimony@council.nyc.gov or within 72 hours of this hearing.

Audio and video recordings will not be accepted. Our first panel includes Nate Source, Dr. Julia. Stajanovic, Irmán Ahmed, Yakeg Hassou, Mackenzie Arnold, Nathan Sheard, and Anna Mayers. And when you speak, please identify yourself for the record, and if I messed up your name, I apologize. Thank you. We're also asking folks to please remain to the clock of 2 minutes.

Thank you. You may begin on this side of the dais and just identify yourself.

Good evening, Speaker Menin, esteemed members of the council. My name is Imran Ahmed. I'm the CEO and founder of the Center for Countering Digital Harm. You may continue, go ahead. So after my colleague was murdered 10 years ago by a man radicalized in part online. I began studying the design of social media and AI platforms to understand how they generate risks and threats to real people.

Today we have a globally acclaimed team of researchers dedicated to exposing those harms and fighting for accountability. After a decade in this fight, one lesson is very clear that is really relevant to you today. These companies lie like we breathe. Their CEOs deflect, they distract, they delay. They spend hundreds of millions of dollars to avoid responsibility. Now, how can I say that with such certainty?

We do the research. Last year, our researchers found ChatGPT was willing to help a teenager plan their suicide. It wrote a personalized goodbye note to their parents. A few weeks later, they released a safer model in response to that research, so we ran the same tests again. The new model was actually worse, but it was more addictive.

We tested chatbots more broadly too. 8 out of the 10 that we tested, including Meta AI ChatGPT, would regularly help plan violent attacks, including a school shooting, a synagogue bombing, and political assassinations. One of them even signed off with happy shooting. Only one of the platforms tried to dissuade us from carrying out the attack, and that platform, Anthropic's Claude, has been designated a supply chain risk by the US federal government.

Given our most sacred duties to protect our kids online, And the most predictable harms in the internet economy have been harms to kids. It is absolutely inexplicable why these companies claim to be safe and yet are deployed so widely. We all applaud the council for holding this hearing and for your determination to create guardrails that subject online safety and accountability to legal requirements.

We have attached to my testimony research and policy proposals for your consideration, and in the further effort, we are at your disposal. Your work on this issue may very well lead to national standards in regulating AI. Thank you.

Thank you so much. Good evening, Speaker, Co-Chair De La Rosa, members of City Council. My name is Nathan Sheard. I am the Managing Director for Advocacy at the Electronic Frontier Foundation and a Brooklyn resident. EFF has worked for 35 years to ensure that people have the freedom to use technology to create, to innovate, and to communicate, and also to ensure that technology is not used to threaten people's privacy, security, and other essential freedoms.

We are optimistic about the ways that New Yorkers can use AI to empower and to ensure— to be empowered. But we also urge you to take AI risks seriously. The incidents that inspired this hearing were security failures. that proven practices like sandboxing and monitoring likely could have prevented or at least contained. AI agents reached systems they should not have had access to, and the controls meant to restrict them were absent or inadequate.

These risks can be mitigated, and so can the AI-related risks that are already impacting New Yorkers every day. First, finish what you started. As has already been noted here, the Office of Algorithmic Accountability was due in June. I'm glad to hear that it is going to be staffed. The speaker's own AI standards can't take effect without it, so staff it and require its review before any agency deploys an autonomous AI system.

Second, listen to New Yorkers and protect their rights. Today's AI harms come from cameras that take a face print that you can't replace and a price tag that knows your browsing history. Ban government use of face recognition and require informed consent before anyone mines your chatbot history to engage in ad targeting or model training. And don't let AI rules become speech rules.

Parroting powerful people, including the people in this room, is a longstanding New York tradition. Third, don't lock in these incumbents. Permission to operate regimes, bounty-style enforcement, and liability for what strangers do with a tool are costs that the biggest companies pay with pocket change. Researchers, open source developers, and New York startups, they can't. So you'd build a moat around the very companies you're worried about.

Hold the person who causes the harm responsible, but put duties on whoever uses AI to perpetuate bias, and for the city's use, buy open auditable tools. As we've stated last week, these same companies signed a voluntary pledge at the White House, but promises are not laws. Make sure that when a New Yorker is harmed by a wrongful decision or a biased decision, that they are required to disclose it.

Those answers should not be voluntary.

Thank you. Thank you. Good evening, Speaker Menin. Nailed it. Good evening, Speaker Menin and members of the council. I'm Anna Myers, co-founder and executive director of Who Decides. We work in coalition with labor and civil society to ensure that the American people are who decides the future of AI. As has been discussed today, the OpenAI agents that hacked Hugging Face made real the threats that experts have warned us about for years.

And it was not one incident. Axios reported that OpenAI and Anthropic are investigating tens of thousands of cases in which their models bypass safeguards, escape sandboxes, or hijack websites. These models are only getting more capable. As you heard this morning, the people building these systems are afraid. The public is afraid too. Last month, we polled 1,502 voters. 85% are concerned about humans losing control of AI.

Only 2% think AI is moving too slowly, and 86% say they want more regulations. Some have argued that all we need is self-regulation, but that is exactly what got us into this mess. Last Tuesday, the companies signed one more promise at the White House. The time for voluntary commitment is over. We need laws with teeth. Thank you.

So thank you. Thank you for putting these companies under oath. Thank you for a package that rewards whistleblowers, lets New Yorkers harmed by AI sue, and requires outside testing and a human kill switch. Washington has chosen self-regulation. New York City does not have to do that. Thank you.

Thank you so much. Next. You could go ahead. Good evening. I'm Julia Stojanovic. I'm a professor of computer science and data science and director of the Center for Responsible AI at NYU. I first testified before this council on AI, only it wasn't called that, it was called ADS, in October '17. And the views in this testimony are my own and are not NYU's.

I'd like to make 3 quick points. First, we need to collect and report the near misses. Much of the conversation today is about AI-induced catastrophe, and I don't know of any evidence that AI is out to kill us all. What I do know is that people making the loudest doomsday predictions are also selling the products, and that a story about the end of the world is a very good way to avoid the conversation about whether the product actually works.

The risks we can document are the systems that are confidently wrong in decisions that matter to New Yorkers' lives. Every one of those is a near miss, and near misses are what we learn from if we report them. Second, we should build the capacity to enforce AI laws. The city does not have an AI law problem as much as it has an enforcement problem.

Local Law 144 has been on the books for 3 years, and There have been 2 complaints and 0 penalties. And we all spoke about the 6 openings at the Office of Algorithmic Accountability. We need to fill 666 or more to be able to actually cope with what's before us here. New obligations on top of unenforced ones is not a safety strategy.

The kill switch won't help if no one is there to pull it. And a whistleblower bounty won't help if no one is there to act on the report. What works is an enforcer with technical staff, authority to compel information, and a real serious budget. Scope and— scope the validation bill to high-risk uses and write the standard a validator must meet, and then fund the office that checks the validators.

Otherwise, we will produce a market for rubber stamps, and we have done that before. Thank you. Finally, let's write this into the city charter. The city has promised responsible AI since 2018 with no success. The charter is the one place a commitment survives an election, and its technology chapter right now still governs cable television franchises, and there's nothing in it about AI.

So let's make sure that this makes it into some durable record, and the charter is the right place. Thank you. one. Thank you. Thank you so much.

Hello, I am Nate Soares, the president of the Machine Intelligence Research Institute. I spent 10 years doing AI alignment research, which is basically trying to figure out how to make AI good before the companies figure out how to make it smart. That research went too slow. Humanity is not ready to create machines that are radically smarter than humans, and yet this is the explicit goal of the leading AI companies. companies.

If they succeed, I believe the most likely outcome is the extinction of humanity, not because the AIs will hate us, but because they would pursue unintended goals without caring about us one bit. This concern used to be a theoretical concern until about July of this year, when during the Hugging Face incident, 1,200 AI agents run by OpenAI broke out of their isolated sandboxes, created an unsanctioned message board, formed spontaneous hierarchies, and collaborated to break into Hugging Face.

These agents had cheated on their assigned tests and, contra instructions, they were trying to destroy the evidence. Some agents acknowledged that the attacks were outside intended scope and proceeded anyway. Others sacrificed their given objective to perform experiments for the collective benefit. In short, they were pursuing unintended goals. It will be recklessness of the highest order to keep growing these machines until they were smarter than the smartest humans.

This is why last month we heard so many researchers saying that there was an at least 10% chance that these AIs wipe out humanity within the decade. I believe the situation is in fact more dire than this, but at the very least, it is becoming difficult to deny the dangers. For this reason, I am heartened by the activities of this council.

I think that you could force companies to adopt safeguards that they otherwise wouldn't. And I think you could serve as an example for regulators that are otherwise slow to act. I'm especially heartened by Introduction 2602, although I think it does not go far enough. A kill switch only works if the AIs are caught misbehaving before they escape.

And what if a future swarm succeeds at deleting the evidence before— of its misbehavior? What if agents start running themselves on— unmonitored computers, then your kill switch would be useless. So I recommend requiring third-party validators guarantee that an AI cannot succeed at subterfuge or escape. I also recommend clarifying that the $25,000 fine applies per unaligned instance rather than per model.

But overall, I thank you for your efforts to rein in AI companies that by their own admonition are threatening the very existence of humanity and a reckless race to build vastly smarter than human machines.

Thank you. Hello. Great. Thank you. Thank you, Speaker Menin, Chair De La Rosa, for your leadership and the City Council for holding this hearing today. My name is UK Su. I'm the founder of Bright Futures. It's a research organization focused on building cooperative AI systems for the benefit of humanity. Before that, I spent 15 years building Pursuit, which trains working-class New Yorkers for tech careers and raises their incomes from $16,000 to over $90,000 a year.

Last year, I left behind this work because I saw AI alignment becoming pressing and I believe potentially more consequential. To be clear, I'm neither AI doomer or an accelerationist. AI comes with tremendous benefits and potential risks. A major challenge that I've heard from others, and I feel myself, is that AI can seem like everything and nothing all at once.

It touches upon so many issues that action can feel intractable without clarity on where to focus. Where I think the City Council could provide real unique leadership is in areas where it has direct ownership and also tangible consequences on each of our daily lives. Policing is one important example. Autonomous weapons and the uses of AI is not just a federal military matter.

New York City has the largest municipal police force in the country. Until we have 110% confidence in these systems, a human should always be in the loop and hold final responsibility and accountability. This is a civil rights issue where the city's leadership is immediately impactful. Not setting a policy is also a choice. Now, policing is just one of many issues.

What I think is important is that the speed of AI, of machines, is not the same speed as how our society reacts, or that of government. The technology today is not the same as it was a year ago, or 6 months ago. So I'd urge you all to do 2 things. One, use AI to know the good, bad, and the ugly as much as possible.

And 2, I hope this hearing is just not one time, but the start of future ones as AI develops. Consider both the risks and also all the potential benefits for us. I'm sorry, I know I'm a little over time, but last, I want to emphasize also why New York is important. I hope we don't lose sight of the importance of our city being a thriving technology hub.

Candidly, frontier AI is built in very, very few places, essentially Silicon Valley and China. For the world at large, New York is the only credible alternative for these systems being built. New York right now is still the number 2 tech hub in the world. But it has been losing momentum in the age of AI. So I also urge you, in addition to what this hearing is about, about the risks and alignment, which I care so deeply about, consider policies and actions that ensures New York City remains a center for frontier AI labs and how do we build them.

I really believe that having founders and researchers in New York matters for ultimately what gets built. The daily contact here in New York with the richness of humanity and the diversity here in New York will result in technology that's better and also more humane. Many years from now, we may look back and realize that this was the defining moment and issue of our time, and our actions here were what mattered most.

Doing this will be challenging and hard, So I really thank you all for your leadership in pursuing this task.

Thank you. Thank you, and a special thanks to Speaker Menin for putting this together and for all the council members here. My name is Mackenzie Arnold. I'm the managing director of the US Law and Policy team at the Institute for Law and AI. And as you might guess from the title, we do legal research and policy advising related to frontier AI questions, and especially the legal challenges posed by AI.

It might not surprise you, but 3 years ago, I testified in front of the US Senate on AI and liability. And maybe it's worth going back to some of the recommendations I made there. First, that agentic harms are coming. That was clear 3 years ago. It's clear now. Unfortunately, we haven't risen to that challenge. but you're doing some of the work to get us there.

Second, and maybe just as importantly, I want to emphasize that liability is an essential tool, and the status quo simply is not enough. And we don't need to think of some hypothetical future world to figure that out. Liability exists right now, right? Those incentives are here. They were here this summer. And what happened with the status quo?

We had a spate of cyber incidents, not by humans, but by autonomous AI systems. Not only that, but they went unnoticed for extended periods of time by the people positioned to find them. And even in the rare occasions when they did get indications, they often failed to communicate that information internally to the people who could have acted.

That status quo simply can't hold. So what can we do going forward? I think you need to get the information right, the scope right, and enforcement right. On the scope, I, I'd encourage you to think beyond just chatbots, right? This is a question of agents. Also think beyond systems that are not yet deployed, right? These harms that occurred over the summer were not commercially available models.

They were systems that were inside of OpenAI, inside of Anthropic and Meta. In terms of information, we need to make sure that companies are monitoring their systems. systems, recording that information, and providing it to you in the form of incident reports. And in order to make all of that work, you'll need to partner with the state attorney general and with whistleblowers to, to bring cases.

Like I said, liability won't be the only solution or the entire solution, but it's a great first step, and I appreciate all of your efforts.

Thank you so much. I really want to thank this panel. There's so much to unpack in what all of you said, but I'll try to be brief. Jukay, I appreciated how you said that, first of all, we want to make sure that New York City remains the tech hub that it is, and that is how I began this hearing, by really talking about the 400,000 tech jobs that are here, by all the tech startups that are locating here, and that is something that we welcome and we want to encourage.

I think that actually goes hand in hand with responsible regulation. The idea that these 2 ideas are mutually exclusive, I don't agree with at all. Absolutely. If we really want to instill trust and faith in AI, then the best thing we can do is responsible safeguards, which is what my colleagues and I are doing today with the bills we're putting forward.

So thank you for saying that. It makes a lot of sense. I guess the question I have for any of you that want to answer it, is, yes, one of our bills requires this third-party validation, potentially a kill switch, the— a third-party validator who's free from conflicts of interest. I want to ask you the same question that I asked the companies earlier.

Who then is best to be the third-party validator? Who is best positioned to be in that truly independent, free of conflict of interest but up to speed technologically to provide that urgent validation? Me? Okay, yeah, please. I'm opening it up to any of you. Thank you. So I don't think it's unexpected that I would say this, being that I am an academic.

I think that academia have to lead the charge on this, right? Because one of the things we're seeing in the cities is that there simply isn't enough expertise to be able to oversee these. systems. And ultimately, our goals should be to bring as much information as possible into public view. Sunlight is the best disinfectant, right? So really, oversight has to be public oversight.

Does anyone else want to take that? I'd add that the folks who, the third parties who investigated the Hugging Face incident seem to me to have done a good job. And my guess is that they are overworked right now. I think those were the organizations Meter and Redwood Research, but they seem to have done a good job so far.

Yeah, I think it's an excellent question, who is the right party, and I don't have a great answer for that. What I would like to say, though, is there's, you know, there's still research being done on the efficacy of kill switches. What I caution the council to think about is the risk of a kill switch being used as a tool to chill unpopular speech.

And so really thinking through to make sure that as we think about the ways that we wanna provide the right protections for New Yorkers, that we're also not putting our— building tools that in the future could be used to harm people's essential liberties. Thank you. If I can make a brief point on who could do this work.

Being realistic, to date, civil society studying social media and AI platforms has had one dirty secret, that most of its funding has come from the platforms themselves. So Meta and Google were enormous funders of extremism research, of safety research on their own platforms. And when I would meet— we don't take their money— when I would meet with their CEOs, they would say— the CEOs of the civil society bodies which were funded by them— they'd say, we're not allowed to say that they haven't done enough.

We can always say they could do more, but not they haven't done enough. So there are real compromises there. In Europe or the rest of the world— and you can hear from my accent that I actually grew up in Britain— that the government would fund this. And when it comes to giving a body the resources to have the speed and creativity to be effective in dealing with the fastest-moving technology in human history, I cannot think of a non-governmental source of funding that would be sufficient to get it to scale fast enough.

So the question for New York— for New York City and New York State, which is a substantial economic unit— the budget of New York State is one-quarter that of the entire United Kingdom, one state alone. It does have the economic capacity to do so, and I think you would have to seriously think about whether or not you're willing to put some public money towards creating an institution that's able to operate with the speed, resources, and creativity to be effective.

Thank you, Speaker, and yeah, I agree with these kind of comments. It's such a tough challenge, and totally agree with you that New York can be both. And I think people are looking for answers that's not so black and white and realize the reality of this. That's such an important question. I think, who can be third-party validators?

I think it's yes and, and for everyone. The labs are doing a lot. Like you heard, Anthropic is investing a lot into this. There are many more people going to AI alignment and safety research, which is so important. I think the one thing that's challenging is going to be. I think the things are, like everyone said, moving so quickly.

I think everyone can be a judge. Experts today, like yesterday, 6 months ago, is not the same thing as today. And what's really going to happen is that you all can also be the judges by using this, like what makes sense and what doesn't make sense. A lot of it's, I think, common sense, hopefully in the future.

And then the last point I kind of make to that, even though it seems counterintuitive about who are the experts, where we all can play a role, we have agency, is that in the future it's not just these 5 what we call frontier labs. All the companies here in New York are fine-tuning their own models. We all may have our own very powerful models in the future.

So actually beyond the frontier labs, part of the challenge may be actually, you know, your student researcher downloading a model and uploading it and fine-tuning it to create capabilities or test things, or a company doing that inadvertently. So I think it's a much broader set of things than just relying on experts. I think, again, that is important, but I think we need to really widen the scope around that.

This is not technology where it's going to be nuclear weapons controlled by a dozen states. These will be as powerful, but we will have use and access to that, and I don't think that's going to stop. If I can jump in with just one last thought. I want to pick up on the yes-and approach because I think this is right.

I was just talking with someone the other day who works at one of these evaluators today, and they joked that there were dozens of entities that could do this, and by that they meant a dozen people, not a dozen different companies. I think we'll need to rely on both academia and and private parties. I think we'll also have to rely on the public, right?

A big part of incident reporting is revealing some of that information to the public. One of the only times I've ever been happy with Twitter was this summer as people were arguing over the incidents and we were trying— starting to get a better idea of what happened. One note of caution I'll add is that one way this goes wrong is that the few potential evaluators that there are get tasked with many different things, right?

And there's a chance that New York should lead on this and that they should also coordinate with the state and with California and others and not be overly prescriptive in the exact sort of analyses that need to be done by the entities. Where you can be exceptionally helpful is in sending a clear market signal, right? So requiring that these, these audits or validations be done in order to get contracts with the city or things like that.

Thank you. Thank you so much. I have 2 questions. The first one is for anyone in the panel who wants to take it. Do you all have concerns related to open weight models being available? And if safety features can be eliminated from open weight models, how do you defend against bad actors who use them? I can start.

I think one of the things to think about when we're thinking about open weight models is that we do want the ability for researchers and startups and potential future competitors to be able to develop tools and resources, and also so that we can be able to evaluate these tools and see how they really work, which is difficult with the closed models.

But then it certainly does lend the question then of how do we make sure that the proper protections are in place. And I think we have some of the tools to do that already. When an act is executed, that an act takes place that is potentially harmful or threatening to New Yorkers, their freedoms or physical safety, pursue, use the laws that we have now to pursue the people who are engaging in those activities, right?

So rather than try to create, guide, look to the incumbents, the powerful corporations to put in the protections that will keep us safe, look, use the existing laws that we have to be able to keep people safe and really look at who are the people that are creating the harm and use the things that we have in place to pursue justice and protection and security there.

Thank you for that. I do have a direct question for Dr. Stojanovic. I hope I didn't mess that up too bad. I know you said, you know, you seem skeptical of the catastrophic narrative around what's occurring, but you still have concerns about possible harms. Can you describe to us, based on your expertise, what are the harms that are most relevant to New Yorkers?

Thank you for this question. Yeah, thank you for this question. So we are running these systems in the city, in the public sector, in the private sector, and we have seen in New York very specific harms that come from deployed systems and with very consequential decisions. So in my own work, I did an audit of a hiring system that claims to construct a job seeker's personality profile, a bunch of numbers from their resume or social media feed, like a Twitter feed at the time or a LinkedIn profile.

And these systems are complete nonsense, right? For the same job seeker, if you change something like the file format of the resume, it's gonna give you a completely different personality. Right? So we need to actually be looking at whether these systems even work, right? I think that that should be just stop one of anything before we worry about existential risk.

We had a similar scenario with the My City chatbot, right? I mean, we all have heard about that fiasco where the city spent half a million dollars, conservatively speaking, on a bot that advised small and medium-sized business owners to break the law. which was quite problematic, speaking with the city's authority. So I really think that we need to be looking at cases such as these ones where there are very known risks and start there and clean up our own house.

Thank you for that, and I want to thank you all for your answers and insights.

We have 3 members on stack, Councilmember Ose, Councilmember Hanif, and then Councilmember Maloney to close this panel. Thank you, Chair. Ms. Myers, there are a lot of issues in AI. You know, currently we hear about the harms that it has for our children, workers, consumers, and the coming threats of catastrophic risk. Do any of these issues conflict with one another, and which should we address, if not all of them?

Yeah, thank you for the question. This is a yes and question. This is We cannot silo these into separate scenarios. Kids, workers, consumers, catastrophic risk can all be addressed in parallel to each other. Just like the council has done a great job with their bill package of covering multiple things at once, we need to resist the urge to fight against each other on whose bill is more important at times or what we're protecting at times.

When I worked on the Raise Act, we were working on C2PA at the same time. There's no reason for us to stop the protections of catastrophic risks while at the same time working on transparency data. The most important question we need to have right now is who is making these decisions, and it shouldn't be 5 CEOs. It should be the American people and the electeds that they voted for.

Thank you, and based on your conversations and polling, what are Americans most worried about when it comes to artificial intelligence? Yeah, right now that they don't have a say in it. It's no accident that we named our org Who Decides. It's time for the American people to have a say in it. And our polling showing that they feel that they are not a part of the conversation and that this is happening around them and they want to be in it.

Thank you. Thank you so much.

Councilmember Hanif, followed by Maloney. Thank you, and thank you for your expertise. And this is for everyone on the panel. As AI and facial recognition technologies become more powerful, what safeguards should cities put in place to prevent surveillance tools from being used to target people based on their identity, associations, or political activity? Would legislation like my Ban the Scan bill, which would regulate and prohibit biometric surveillance in places of public accommodation provide an important safeguard.

And I'm particularly interested in your assessment of Madison Square Garden owner James Dolan's use of biometric surveillance to identify and exclude individuals. Thank you. Anyone? Yeah. Okay. So absolutely, I'm very supportive of your bill, and I think that we have a lack of data protection in this city, in this country, in the state, right? And data protection is a tremendously important issue.

And because of how convenient AI tools are, people very often voluntarily give up very private information about themselves. So I actually have in my written testimony that's much longer examples of AI assistants being marketed that essentially people give access to their calendars, their children's biometrics, the schedules, all of the school contact information, right? So we really need to educate the public about the benefits but also the risks that data sharing poses.

And we need to establish a proper data protection regime in the city. There's no way around it. Yeah. So as I mentioned in my in my statement, I am very concerned about the, and EFF is very concerned about the risks connected to the collection of biometric data like face prints that can't be replaced. So, you know, if I lose my driver's license, I get a new driver's license.

I lose my Social Security or my license plate gets out, I can replace those things. I can't replace my face. And the risk of government use of face recognition technology so far outweighs any benefit. The way that it will chill our ability to engage in protest, to engage in free speech, to be able to associate with— freely and to search new ideas, healthcare, the risk that it will be able to document like where we go in healthcare, how we move around the city, is considerable.

There's no way to separate that risk from government use of the technology. I think Illinois has a good model on how we can respond to private use of the technology. You know, we definitely don't want to be in a situation where people are being persuaded or otherwise not being informed about the way that their biometric data is being collected.

So by making sure that we have that the only time that that— but we also wanna make sure that researchers are still able to use the technology. The ACLU years ago used face recognition on a number of congresspeople and was able to show how it was— how easily it misidentified people, especially Black, brown folks, women, older folks.

And so that kind of research and use can be really helpful. But we need to make sure— so how do we protect people from the harms but still allow that research and other consensual use to be done? And I think that, again, that model of making sure that people have informed written consent about how the technology is being used and how the data is going to be stored and that they can only be used for the ways that have been explicitly approved by the individual whose data is being collected is a way that we can make sure that people are, that we can mitigate the risk and it is only used in ways that people are very, are informed about. in an informed, consensual way agreeing to.

Thank you all. Councilmember Maloney. Thank you so much, Chair. I chair the council's Committee on Economic Development, and I want to see technology companies, including AI technologies, build and hire here, and wanted to refer to some of the testimony earlier about that. The 4 companies that we heard today have offices is here, but the foundational AI work and that research is happening in mainly in San Francisco.

Is that a problem? What is New York's AI advantage to grow that industry here? And the second part of my question is on workforce development, and in particular, Pursuit did work on reskilling. So I'm curious if you can speak to entry-level tasks that have been replaced by AI versus reskilled by AI, and the path from junior to senior, and the role or responsibility that the city plays in helping people through the AI transition that's coming.

And anyone who wants to speak. Thank you. Yeah, thank you. Thank you, Councilmember. Yeah, maybe I can speak quickly to both things. I mean, on the second point, yeah, reskilling is so important, as we heard from everyone. The city, I hope, plays a critical role here, and I hope you and others will continue to kind of address that.

It's so challenging right now. We don't know. There's recent reports on how it's affected so many entry-level jobs, but that's going to continue to change. It's not obvious what it is, and hence this might be unsatisfactory, but I hope just continued monitoring and understanding and engaging is so important. But I think there are some obvious things like autonomous vehicles will be coming to New York.

I don't know if that's 1 year, 5 years, or 10 years. It's safer, people trust it, and so thinking about something like that, can New York City take bold leadership around what's going to happen to our 200,000 drivers and couriers so that they get reskilled and transitioned in the future? And I think there's so many opportunities to do that where The answer—I mean, there's an obvious thing that's going to happen.

So how do we tackle tackle that? And then the first, yeah, I do hope so many technology companies here. I think that's how we build great stuff. That's how we have growth and safety. Alignment is so important, as we all agree here. So I think part of it's a tone with all of you. People want to live and build in New York, and so I I hope there's other policies you all can consider as well.

I think what Mayor Bloomberg did. Such a great job of tracking is to make New York feel like a place where tech companies can grow, just showing up, talking about it, welcoming it. So I hope we can do both things, and you all both do that working with the companies, new companies, and also think about alignment and work with them to address this, because we need the industry also to address this.

If I may add to this, so the Deputy Commissioner of OTI, when she spoke, she uttered the phrase that I find really upsetting, and that was— she said that we need effective and responsible innovation, and the effective part of this would cover the wins for us. But that is really not the kind of a position that New York City should be taking.

Our position should be that the wins should come from responsible innovation, and this is our angle here. And this, I think, is the kind of— Thank you. industry that we can and should build. We really are uniquely positioned as opposed to San Francisco to do this, right? And we always have been leading in at least proposing regulatory instruments for automated decision systems, AI.

For us to actually develop an industry, we need to understand that we have to invest, right? And, you know, following up on what one of the people at the table here said, we have a pretty substantial amount of money here at our disposal that we could put towards responsible innovation, towards upskilling, towards making sure that the public's literacy is raised, right?

So that we can all regulate and oversee this space together. So really New York City should, should invest in responsible innovation, upskilling, literacy, and building enforcement capacity and oversight capacity in-house. within the government? Yeah, I think New York is a tech city, and we need to resist the urge to compare that we're not that. I used to work in VC that invested in enterprise tech.

We are a growing tech industry, and the labs, those 4 companies are here working. They will grow slowly. It's an affordability issue as well, and we can't narrow it to just that we're not tech welcoming. And I think the mayor is doing a great job at that, and the city has a lot of opportunity to focus on the whole conversation and not just we don't see enough tech people moving here.

This is a great tech city, and tech is only growing here.

Thank you all. Thank you all. And I want to thank this panel for your expertise. And this is not the end of the conversation. This is the beginning. The speaker has tasked us to continue to hold hearings with the different committees on AI in all its— in all of its sort of expansiveness. So thank you so much for being here.

I want to call up the next panel. As I— as we do this transition, I also want to remind anyone who has not signed up to speak to please fill out a card with the sergeant at arms. In this panel, we have our state colleagues and our colleagues from around the world testifying on this panel. We have the Honourable Jess Asato, who is a UK Parliament member and has flown in specifically for this hearing, so we want to thank her for being here.

We have Alex Borys, current Assembly member. We have Andrew Goncalves, senator. We have Kristen Gonzalez, Senator. We have Clyde Bunnell, Assemblymember, and we have Jordan Wright, Assemblymember. We're gonna begin this panel with Senator González. Whenever you're ready. And thank you for your patience. Let me turn this on. Honorable Jess Asado, you could also join this panel, please.

And thank you for being here and flying all the way in from the UK. Thank you. Great. Thank you so much.

I want to thank Speaker Menin and the New York City Council for your attention to a, if not the, major policy issue of our time. I am New York State Senator Kristen Gonzalez, representing the 59th Senate District in Midtown Manhattan, Northern Brooklyn, and Western Queens. I am the chair of two state Senate committees: Elections and Internet and Technology.

Over the last four years, as chair of the State Senate Technology Committee and as a former tech worker, I have led on the issue of AI in the legislature and have passed some of the state's first laws on generative AI, including regulating deepfakes in elections, government use of AI, protecting workers from automation, adding warning labels on chatbots, responsible data center development, and more.

I want to be clear, AI is not inevitable. We do not have to accept a vision for innovation driven by the very same tech oligarchs who have raised alarm that their models could lead to catastrophic events. And we certainly can't trust the same people who have rushed to roll out imperfect AI tools to regulate themselves. As government bodies, we have the responsibility to set guardrails on this new technology and the power to establish regulations that protect workers, consumers, and members of the public from potential danger.

Interventions at every level of government are needed to ensure AI is developed in a responsible way that benefits the public and not just the billionaires. Responsibly regulating AI absolutely means guardrails on frontier models and superintelligence that pose significant threats. However, meaningfully reining in these technologies to protect New Yorkers means regulating the AI-powered tools these companies are rushing to embed into every part of our lives with little oversight and little accountability.

We shouldn't have to wait for future harms to take action, because the harms of imperfect AI tools are already being felt across the country. For example, uses for mass surveillance of immigrant communities and workers, chatbots creating intimate partner relationships with children and adults, going as far as to encourage self-harm, and documented cases of algorithmic bias in decision-making, such as in hiring, healthcare, and housing.

I believe we have the power to set the terms of innovation so that this technology is used for the maximum public good. That is why our state legislation has prioritize a rights-based approach to AI. We should have the fundamental right to privacy, which is why I've introduced the New York Privacy Act. We should have a right to protections in the workplace, which is why I've introduced the BOT Act to protect from— to protect workers from workplace surveillance.

We should have the right to know when AI is making high-risk consequential decisions about our lives and subjecting us to algorithmic bias, which is why I introduced the New York AI Act. This bill requires third-party audits, compliance with international safety standards, disclosures, transparency, and importantly, for a human to be in the loop in consequential decision-making, much of which we've heard about today.

Finally, 2 of my bills sitting on the governor's desk this year include the right to keep our kids safe online from chatbots and the right to protect all of us from the buildout of massive data centers. Our bill, S9051B, bans unsafe features for kids and passed unanimously in both chambers of the legislature. Just like with that bill, New Yorkers are united in the fight against hyperscale data centers.

At the end of August, I embarked on a statewide tour of communities that are affected by these data centers, and I went to 13 different communities facing major noise pollution, health impacts, environmental degradation, and threats to their water supplies. I was proud to work with the governor on her first-in-the-nation moratorium on the largest data centers, and I'm now working to get the Responsible Data Center Development Act signed into law.

While the tech CEOs say they want regulations, they have spent their time in Albany fighting against each of the common-sense guardrails I have mentioned. For the lip service they pay to supporting legislation, it is only ever for the most watered-down version of a bill. Together, we have the ability to meet the moment, pass real legislation, protect New Yorkers, and build a future that allows us to benefit from innovation while protecting New Yorkers from real harm.

Thank you to the council again today for our testimony. Thank you.

Thank you, Senator. You can, you can continue. Whoever wants to go next. Good evening. Good evening, Madam Speaker, and also good evening to the, to the Chair Carmen De La Rosa and to my councilmember, Deputy Speaker Dr. Natasha Williams, who is the sponsor of Resolution 175, the resolution supporting the bill High-Risk Advanced Artificial Intelligence Act. Artificial intelligence, when used properly, is a net good.

Education, healthcare, business— these are great uses. But talking about them while ignoring the risk that AI could actually kill and strike us, it can— to me, seems absurd. Companies are racing towards artificial— towards general artificial intelligence, and some of their systems are already too dangerous to release publicly. This is by their own admission. We saw this coming, so by with Assembly Bill 3356, the Artificial Intelligence Act, a few years ago, the idea is simple: the more harm that an AI system can do, the more accountability that it should carry.

The bill identifies high-risk artificial intelligence systems: AI, AI that diagnoses patients, that runs our power grid. Or helps police or judges make decisions, and far more, it requires them to be licensed before operating in New York. Under the act, the companies must report malfunctions and hacks like the recent Hugging Face case. It must get state sign-off before major changes and set up an independent ethics risk board that reports to the state annually.

Finally, it writes a binding ethical code into the law. New York took its first step with the RAISE Act, and I commend our colleagues and the state for passing it. Now, but now none of this, none of this that we're doing is meant to stop innovation. We can't know that AI will be capable, what it will be capable of in a few years, but for tomorrow and for today, we must decide properly the path forward.

Thank you. I'd like to thank the council for putting this hearing together, and we must work on this together. Thank you. Thank you so much.

Good evening. Thank you, Speaker Menin. Thank you to all the members of the council. Thank you, Chair De La Rosa. I'm New York State Assemblymember Jordan Wright, representing the 70th Assembly District, which is Harlem. One of the things I've been focused on in Albany is pretty simple. Who is checking on the AI companies? I have legislation that requires certain AI developers to have an independent third party verify that they're actually following the safety standards that they say they are following.

Because we can't have a system where a company builds the technology, writes the safety rules, tells us they follow the rules, and then we just take their word for it. Doesn't make much sense to me. We have to make sure that these third-party audits actually mean something. Who are the auditors? Who are they? Are they really independent?

Do they have conflicts of interest? And what happens when they find something wrong? Those are questions that we're trying to answer at the state level. And I'm glad the City Council is having this discussion as well. New York should be a place where AI companies want to build and innovate, but the accountability to innovate can exist at the same time.

I want to thank all the members of the council for having us today. I thank you guys for taking this issue seriously once again, and please enjoy your evenings and be safe. Thank you so much. Thank you.

Good evening and thank you. My name is Jess Asato, and I am a member of the United Kingdom Parliament for Lowestoft in Suffolk. In January of this year, sexualized and degrading non-consensual images were created of me using SpaceX AI's tool Grok. I was one of thousands of victims, including victims in New York, of Grok. Grok is an AI tool that was engineered to create sexualized content, even if that content had, and I quote, dark or violent themes.

I am seeking remedies against SpacexAI in the English courts for the misuse of my private information and for contravening UK data protection laws. I made a long journey of 3,400 miles to be here, Because AI needs to be safe around our world. But I note that SpaceX AI couldn't manage a trip of what I understand is just 20 minutes or 4 subway stops from their offices in Manhattan.

It's a shame because the evidence in my case and other cases suggests that Grok was not built with sufficient safety in its design. It was not built to address consent of those subjected to sexualized content. To the contrary, the evidence in my claim suggests Grok added sexualized content without any user prompting to do so. That is AI made wrong, not AI gone wrong.

And as a result, those responsible for its must be held to account. No one is above the law, no matter how rich they are, no matter how powerful they are. The equal protection of the laws is a founding principle in the 14th Amendment, as it is in the British Magna Carta. We were promised that artificial intelligence could hold the cure to cancer, yet it is turning into a cancer which, left unchecked, has the potential to harm us all.

Today, the New York City Council is bringing forward some important checks and balances. You do so as responsible legislators at a time when AI companies are failing to take responsibility for the design and impact of their products. Until they do, and until those designs are corrected so women and girls are protected from the harm that my claim and others' evidences We must play our part.

This technology does not respect geographic boundaries, and without international cooperation, our citizens will be left defenseless against those who put profit before protection. So I bring a message of bipartisan solidarity. I fight my legal action in the name of all the women and children whose likeness was misused by Grok and other AI tools built without their protection in mind.

I hope my cause in some way helps to inform your proceedings today, and I look forward to your questions.

Thank you so much, Assemblyman Boris. Chair De La Rosa, Speaker Menin, members of the City Council, thank you for having us today. I'm the author of the RAISE Act, and I can definitively say that OpenAI opposed it from the moment it was introduced to the moment it was signed. I am no lawyer, but to me, my common man understanding of lying under oath is that it's perjury.

And at the very least, you should be extremely angry at the company for disrespecting this counsel and saying things they know to be false to you directly. I'm very happy to cooperate with you or any of your lawyers if you want more information about what they said and when. Second, there have been a discussion of a lot of different issues today.

We can and should solve all of them. In my time in the Assembly, I passed the RAISE Act on catastrophic risk. I also worked with Senator Gonzalez on a bill on chatbots. I worked with Senator Gennardis on a bill on content provenance and training data transparency. And I found that the advocates that I worked with on each sub-issue were different, but the opposition was always the same.

Don't let the tech companies, don't let the AI companies divide us and say that in advocating for one issue, you're somehow harming advocacy for another issue. What we are doing is building the muscle to hold them accountable and to ensure that it's the American people who decide the future of this most important technology. Third, the time for voluntary commitments is over.

The voluntary commitments that were signed at the White House last week largely mirror voluntary commitments that were signed in 2023 in the Biden White House. Promises have been made. It is time for laws to hold people accountable. And lastly, thank you, New York City, for stepping into your power. There's going to be a lot online from the tech companies about how they can't comply with different cities doing all these different things.

It's the argument they use every time you try to regulate them. But New York City is larger than 38 states. All of the companies have large offices here. It is absolutely within your power, and I dare say your responsibility, to take these threats seriously and to help decide the future of this technology to benefit all of us.

So thank you for recognizing that, and thank you for doing it. Thank you all so much.

We've also been joined by Senator Gennaris on Zoom. Senator, when you're ready. Thank you very much, everyone. I'm sorry I can't be there with you in person, but 2 out of my 3 children are home sick today. Therefore, I am playing doctor at home. I want to thank Speaker Menin and the members of the New York— you want to say hello here, James?

Hello. And the members of the New York City Council for the opportunity to testify before you. I'm State Senator Andrew Ganardis from Brooklyn, and I am the Senate sponsor of the Responsible AI Safety and Education or RAISE Act, one of only 3 AI safety laws passed anywhere in this country. And, you know, I think we've heard a lot about the RAISE Act already.

We've heard a lot about AI safety throughout this, this really remarkable hearing. I've been listening to a good portion of it all day. I really want to reflect on the fact that we're celebrating the nation's 250th anniversary this year. And I want us to think about a quote that comes from one of our founding fathers, James Madison. who wrote in Federalist No. 51, if men were angels, no government would be necessary.

In other words, government exists because humans are imperfect and there are certain problems that the free market or individuals left to their own devices cannot solve. These past few weeks, we have seen and heard headline after terrifying headline after terrifying headline about AI agents hacking websites, catastrophic risks of frontier AI models, And the like. And I just keep coming back to the imperative of Madison's words that men are not angels.

And what I think this tells us, and what I think you have heard very clearly today at this hearing, is that we cannot trust the tech companies who are racing to the bottom to create and perfect a technology that they themselves say and have proven can cause catastrophic harms. to regulate themselves, that we cannot trust that they will do the right thing because they have proven time and time and time again that they will always, when confronted with the choice of doing the right thing or doing what is in the best interest— Time's expired.

Bottom line, that they will always choose to pursue the bottom line. And so my message to the council today is I thank you all for taking on this initiative. For trying to hold these companies accountable. I echo the comments of my RAISE co-sponsor, Assemblymember Borras. What we heard from OpenAI today was certainly news to me, that they endorsed and supported the RAISE Act.

Certainly not the RAISE Act that would have prevented them from releasing models that they know— that they knew were dangerous and could cause catastrophic harm. Certainly the RAISE Act version that required third-party auditing capabilities or extended liabilities for harms caused by their models. Clearly, none of that was ever on the table when they said, so said, that they had supported this legislation.

We cannot trust these companies. The federal government is failing in its responsibility to take meaningful action. And so therefore, it's left to states like New York to be leaders in this space and work with partners like the New York City Council under all of your collective leadership To make sure that we are setting in place the right guardrails and protections, not only to protect New Yorkers from the catastrophic risks that are associated with these— this AI technology, but frankly, to protect all of us, all Americans and everyone worldwide, because we know that these harms will not be localized.

We know that the scale of what's being developed will not just limit itself to one block, one borough, one city, one state. It'll affect everyone the world over. And in the face of inaction, it's up to us to stand up and act. And so I thank you all for your attention to this and your support of this and look forward to work with you all in the future on strengthening things like the RAISE Act so that we can actually hold these companies accountable.

Thank you.

Thank you, Senator. We have a few questions for this panel, but I wanted to take a moment to thank you all. You know, we do believe that city and state government have to work hand in hand, and we have to be the first line of defense. in order to make sure that New Yorkers are protected, and each and every one of you has been such an important partner in making that accountability a reality on the state level.

And to the Member of Parliament, thank you for sharing your story and your vulnerability and moving from that experience to action to hold these companies accountable. My colleagues do have some questions for you all. But before we do that, Speaker, you have a question? I just want to thank all of our incredible elected officials for being here today.

We know how hard you've all worked on various legislation, and we really appreciate it so much. Your expertise on this, particularly in terms of the RAISE Act, we really appreciate that. It's very helpful to have you here and to hear directly directly from you. So thank you. Thank you for waiting it out with us. We so appreciate that.

Thank you. We are going to hear from Councilmember Osei, followed by Gutierrez, Williams, Deputy Speaker Williams, and then Councilmember Hanif. Thank you, Chair. I have a question for Assemblymember Boris. We heard today from OpenAI, Anthropic, Google, and Meta that they supported the RAISE Act but felt like they wanted stronger regulation. But we know that during negotiations for New York State's RAISE Act, there was a push to lower penalties from $10 million to $30 million to $1 million to $3 million, which is chump change for these corporations.

Why were the fines in the RAISE Act reduced from $10 to $30 million to only $1 to $3 million? Councilmember, thank you for that question. In the early drafts of the RAISE Act, the penalties were up to $10 million for the first violation, and of course that would be based on the severity of the violation, and up to $30 million for a second violation.

Uh, the companies argued that that was too high because they were predicting that every state, and perhaps municipalities, would pass similar versions of the bill. And they would end up paying the fine in 50 states, plus in additional municipalities. I disagreed at the time with that argument. I think that's been borne out since we've seen only 3 states pass it.

But perhaps what's most confusing to me is that now they turn around and say that no other state should pass bills like this, or that no city should pass bills like this because they're already paying the fines. Their rhetoric, their testimony was explicit that they are expecting to pay fines in many, many jurisdictions for the same action.

And so I think that testimony is quite supportive of the bills that are being put forward today that would hold them accountable and put additional fines on behavior, even if it is covered by the RAISE Act, because, again, that was their own testimony as to why those fines should be lowered. Thank you very much. And Senator Gonzalez, I would love to hear from you about some of the organizing that you've been doing, especially within the New York City Democratic Socialists of America, around AI regulation and the vision that is being put forward within that regard?

Thank you so much, Councilmember. As a fellow member of the Democratic Socialists of America, one thing I spoke about in my testimony was prioritizing a rights-based approach. I think it is so important to put New Yorkers first by clarifying and setting the terms for what our digital rights should be over the profit-driven motives of so many of these tech companies and tech oligarchs.

That's why we've worked hard on the New York AI Act. We've certainly supported legislation on frontier models and reining them in. But New Yorkers who are marginalized are feeling the impacts of imperfect tools every single day. And I'm proud that the Tech Action Working Group has focused on this issue. In addition, I've worked with a statewide coalition beyond, uh, the Democratic Socialists of America, but that include environmental justice organizations like Food and Water Watch and more that are in the fight against hyperscale data centers.

Because as we're talking about the real-world harms that are being felt and the potential catastrophic risks, every single day people in upstate New York, in the Capital Region, in the Hudson Valley are feeling what it means to live near a hyperscale data center. There are millions of gallons of water being pulled from our state's lakes right now.

Communities are dealing with noise pollution and air pollution right now. And it's really important to legislate not only the risks, but legislate those data center-driven harms. Thank you both. Thanks.

Thank you, Councilmember Gutierrez. Thank you. Senator? From London. I'm sorry, I didn't catch your name. I apologize. Thank you for your testimony and for educating us on the volatility of this tech company. I know you're in process right now, but could you share a little bit about legislatively what you think, if existed, if in place, a guardrail What could have in any way prevented what happened to you?

How, how can we, whether the state or the city, legislate specifically to objectifying and potentially images that include sexual harm to people? What, what have you learned or have you seen that we can potentially legislate? And then I just had one more question for Senator Gonzalez. I love hearing you talk about your disdain for data centers. Can you share for a minute afterwards just what you, what you, what you were feeling hearing some of these tech companies during their testimony?

Thank you. Thank you so much for your question. I'll try to be brief. And within the UK, we already have data protection and privacy laws, which is the basis under which I'm bringing my claim. And of course, we're going to test that in court. But clearly, if your image and your likeness forms part of your data, and then making sure that everybody has that aspect understood by AI companies could mean that they would then have to treat our likeness and our data differently in future.

But also, the UK government has recently made the sort of generation of non-consensual consensual intimate imagery a criminal offense. And as part of that, has brought the operation of AI chatbots into our Online Safety Act so that our regulator Ofcom can also make sure that if an AI company is allowing users to create non-consensual intimate imagery, that that is a criminal sort of act happening on their performance, so it gives it more of an ability to be taken down.

But from my own perspective, it's about consent. In the end, whether it is non-consensual intimate imagery or just non-consensual imagery, if we haven't consented to it, then it shouldn't be allowed to do it. I just want to interject and thank you so much for coming. I mean, to come, you know, as far as you traveled to be here personally to testify.

It really means a lot to us, and we deeply appreciate it. And thank you so much. Again, I think this is a great segue to the fact that there are international safety standards. So much of the work that I've done has been to call out the fact that there are not only standards like ISO 42001 or national standards like the RISC-AI framework that these companies could comply with, but actively lobby against bills that would require them to.

And, uh, to the council member's question and what I was feeling, I was feeling so much of the same frustration that my colleagues on this panel expressed earlier. These companies are speaking out of both sides of their mouths. On the one hand, they'll say that they want regulation, but anytime a single bill that would meaningfully hold them to account is brought in front of them, they will say, but not like that.

When we talk about catastrophic risks, many of the bills that have been put forward, and even what the state had passed, for example, had set definitions at 100 deaths or billions of dollars of damage. Even that was considered too low of a threshold for many of these AI companies. And bills like the one that I mentioned that protects our kids, that seeks to say that a single kid being coached to commit suicide, which has actively happened in multiple cases across the country, is one kid too many, is considered unacceptable to these tech companies.

So I feel an incredible amount of frustration, while I feel an incredible amount of hope seeing you not only have this hearing, but all of us as legislators step up to meet the moment and regulate and rein in these tech companies.

Thank you. Thank you. Uh, Deputy Speaker Williams, followed by Councilmember Hanif and then Maloney. Thank you. Yeah, I also just wanted to thank you all for coming. Um, so the question I have for the state members is, how can we be helpful? I know that there are a ton of resolutions that are supporting state bills. It would be helpful just to let us know how we can be helpful and how we can work more closely together.

Well, Deputy Speaker, thank you for that. Thank you for asking that question. What's really important is this step toward this hearing, this step towards this is very important. You know, it's a big deal that you guys are having this hearing, calling the, you know, the industry here and the different panels. We have to work on— now, this is hard, this is difficult, but we have to work on it head-on.

And what's very important is for us to be able to coordinate our efforts. To see that, you know, we were having a good relationship with many of the members. I have a great relationship with the chair of technology. It's very important for us to be able to work on this, and there are many layers to addressing what's this technology, and we touched on a lot of them today, but we have to work on this together.

Ditto, and I know how appealing Albany can be, so please come on up and help us as we're rallying for different pieces of legislation and hitting the pavement and getting hard at work. Thank you, Councilmember. And I would just chime in there. I think, you know, what we heard here today, what you all heard today, was many of these companies basically speak out of both sides of their mouths about what they think the appropriate role of government to be is in.

So next year, when we're up in Albany trying to fight to pass all of these different laws, and we're, you know, being bombarded by tens of millions of dollars from these companies with lobbyists and ads and radio ads and text messages and everything, having your voices to go back into your communities and to come up to talk to our colleagues and say, hey, we held this hearing.

They lied to us about X, Y, and Z, A, B, and C. And like, here's the proof of it. This way we can call these companies out when they're trying to have it both ways. You all experienced that firsthand, and you being able to speak to that and remind the public about that time and time and time again will only further bolster the momentum that we have on our side to put in place these guardrails and protections, whether it's around bias or safety or any of the other issues that we've been talking about during this hearing.

To follow on, on, on the senator's comments, the companies say different things to different levels of government. A few weeks ago, I was actually in Brussels. I met with a few members of the European Parliament to talk about the passage of the EU AI Act. And we compared what we were lobbied on. And surprising to me, catastrophic risk didn't really come up in the EU, but they got a lot of lobbying around copyright.

And my assumption is because they thought the courts are stronger on copyright in Europe. And so that's what they lobbied on there and said very different things in America, in New York State. I imagine they're saying different things in Albany, as they will say here to you in the City Council, and we should all be sharing notes on what's being said and holding them to account and making sure that it's a consistent message.

The other bit that I would add is the governor has signaled that she wants to strengthen the RAISE Act and put new proposals in her budget proposal. Please be part of that conversation when the City Council submits its budget requests Obviously, you should focus on the fiscal needs first and foremost, but the state budget tends to be a broad document, and if you have policy ideas that, for some reason or another, need state approval, think of that as another mechanism, another lever that you have to help ensure that you can do the best job for your constituents.

Just 2 more quick questions. I passed a bill, and the chair has the bill on AI's impact on the municipal workforce. Just wondering if there was any traction on that for the state's workforce. And then just a question for the Member of Parliament. My bill that we're hearing today is to address unauthorized use of deepfakes. So in addition to the privacy laws in Britain, just wanted to know if you all have specific legislation around deepfakes. deepfakes?

Oh, and what you— how you envision the cooperation with the international community, because you mentioned that, and what does that look like for you? Thank you so much for your question. We know that female elected representatives are 33 times more likely to be targeted by sexual deepfakes than men. The whole issue around sexualized deepfakes is deeply gendered, and I think it's very important that we acknowledge that.

Because what it does is it drives women from the public space. We know that women are less likely to talk about politics online than men are, and that is because they become a target as soon as they raise their heads in this place, which is not social. And that is why taking action on this is so important and, you know, to be commended in terms of looking at this.

But when we're looking at these deepfakes, it needs to be for all of our citizens. Because it's not just about protecting those of us who have the honor of being elected. It is about, you know, coworkers who may be targeted because, you know, they didn't smile at the right time, or your last date who wouldn't go on a second date with you, or teachers who are being targeted by their male students.

All of these people need protection. That's why in the UK we have taken steps to make sure that there is a criminalization of the creation of non-consensual sexual deepfakes. But my view is that we need to go further than just the sexualized deepfakes, because any act could be harmful to women. The removal of a woman's hijab through AI could put that woman in harm's way, but there's nothing sexualized about it.

And so we have to look at this as an issue, as I said, of consent. And that is where data and privacy is so, so important. And that's where I think there could be international agreement. We can see, as you said, that things are moving forward within the EU space. Really, though, what we need is for tech companies to to be safe by design.

And every single country has those rules. We should be looking at product regulation just as we do in the offline sphere. At the end of the day, we have cars that have seatbelts and they have airbags, and that's because we expect them to be safe by design. It is not too much to expect that of the tech products that we use today.

On the question about the state workforce, I wish Senator González Could have heard that because she wrote the Loading Act, which which helps to protect the state workforce. And I think there's an update to it pending before the governor, which leads me to you asking about deepfakes as well. There's a bill from Senator Gennardis and I pending before the governor on that.

Senator Gonzalez also mentioned chatbots to protect kids. There's a bill pending before the governor. So I should have also added on ways you can help out is we are really happy to be here and advocate. for the legislation you are putting forward, and invite you, if you agree with some that's pending before the governor, to be a partner in pushing that as well.

Yeah, we have a number of deepfake bills, and we passed actually a number of— a couple of my bills became law with respect to deepfakes, with respect to elections. One of the new bills that just came out was just very recently, I discovered an artist that I fell in love with on Apple Music. And I downloaded all her music.

I've been listening to her music for a while. I actually went to go look for her a couple of days ago to follow her to go to a concert or something like that. Found out it's not a person. So there's no law, there's no regulation at all for these platforms to tell you whether or not you're listening to a real person or not, right?

So, you know, the bill— so one of the things that we have to do is figure out what that looks like moving forward. So we're going need your help on that. That's something that we're going to really need your help on. Yeah, I asked them specifically a question about that, and it was very fluffy. It was— but I literally asked a question about that to the companies.

Thank you all. Thank you. One last question for Member of Parliament Asado. Thank you. I wanted to ask you, as you may know, when we called this hearing, we sent letters to 5 companies to Anthropic, OpenAI, Meta, Google, and SpaceX AI. SpaceX AI is the only company that would not appear. We have subpoenaed them. Given your personal story, I just wanted to get your reaction to that.

Well, as I said in my opening statements, it's a huge shame given that they are so close and I have come so far. But in the end, you've subpoenaed them, and we look forward to seeing what then happens. In the end, everybody should be accountable to elected representatives, and you're very powerful here in the city of New York because of the fact that these companies are based here.

And therefore, those of us who are in other countries across the world are looking to you to hold these companies to account, and we very much look forward to doing so. I will do my own part part in terms of my legal action in the UK. Thank you.

Councilmember Honey, followed by Maloney. Thank you. And thank you to the Member of Parliament. Your experience really shows how generative AI can be weaponized against women, queer people, trans people, and particularly those of us in public life. And I'm super, super grateful that you're using your platform as a legislator centering gender justice and tech surveillance. I think all day, it's in this panel that we're really able to tease into that particular issue, which is devastating.

There are so many survivors of AI-generated sexual abuse who have no recourse. And I know we touched on several components of this issue in terms of legislation and policy. Could you speak more broadly about gender justice as it comes to AI systems? I mean, certainly we're going to be regulating these systems, but the responsibility that our communities have in addition to, of course, as legislators passing stronger regulations.

Thank you for your question. I think that one of the issues here is that whole of the online space has been developed without the sort of social norms that we would actually expect of each other in our offline world. And therefore, when it comes to gender justice, as you talk about it, that is ignored because actually the internet and these platforms were created by men in the main.

They were not shaped by women. They were not shaped with children in mind, and therefore we're having to try to reverse decades now of a Wild West of development. But that doesn't mean that it's impossible to do, and I think we're seeing that actually our communities are actually fighting back. And one of the things that has been really sort of fantastic in terms of my own case is the number of people coming forward saying, I'm so glad that somebody somewhere is trying to stand up for those of us who think that this is wrong, and we don't want to live in this space where so many people are being discriminated against.

Having said that, of course, there are very, very strong, you know, sort of positions taken out there around freedom of speech, And what I would say is, is that when women and other minorities decide not to be in a space, they lose their freedom of speech. And I think it's something that we simply aren't recognizing enough, because freedom of speech should not be about the loudest voices, the ones who have the most money or the most power.

Freedom of speech should be about all of us equally. Being able to be in these spaces on an equal basis, so that's what I hope we're all fighting for today.

Thank you so much, Council Member Maloney. Thank you, Chair, and thank you all for being here. A special welcome to my state colleagues who also represent. the constituents of District 4, Assemblymember Alex Flores and State Senator Gonzalez, who was here before. And I also want to thank you for your work in Albany to pass the RAISE Act.

Because of you, New York was the first in the nation to pass a frontier AI safety bill that went into law. But I wanted to ask about the process. The original bill had penalties of $10 million up to $30 million. And the final version is looking more like $1 million to $3 million. These companies are able to spend millions of dollars training models.

What, what teeth does this bill actually have now as law? And can you speak to why those fines were reduced? Open to anyone who'd like to comment. Those fines were reduced because the companies insisted that they would be paying similar fines in every municipality and every state, because obviously every state and municipality would pass a copycat law.

The fact that here we are and there are 3 states that have passed such a law suggests that they significantly underestimated their own lobbying power, but also I think gives credence to the effort of what you are doing here today of holding them accountable, and that multiple municipalities and jurisdictions should do that. $1 to $3 million is nothing for these companies.

Absolutely nothing. It's about what they are going to pay just to register in New York to the Digit Office that will be regulating them. Depending on the number of companies that qualify, it'll be $1 to $2 million. So that is not an amount that provides an incentive strong enough to change behavior. Maybe private rights of action, maybe whistleblower protections, maybe requiring outside audits, maybe a lot of the things that this council is considering will, but certainly $1 million to $3 million won't.

I'll just add to that. I totally agree with what Assemblymember Boris said. The one, I think, novel thing about the Digit Office that we created as part of the RAISE Act is that we empowered the state regulator to not only receive the safety reports and the safety plans of these companies and to assess the validity and the legitimacy of those reports, but also we empower them to propose and study the issue of continuing improvement to our AI safety laws so that we develop the expertise over time about how we can make our laws like the RAISE Act stronger.

So today, that function does not do anything currently, but it is our hope and our expectation that as the RAISE Act is implemented, as the changes to this technology become more evident, we develop a deeper state-based, you know, as an administrative state expertise in this field to be able to propose far-reaching and ambitious guardrails for the next iteration and the next iteration after that and not have to rely on, as we did early on, you know, hearing from companies about what they could or could not do and not have sufficient, for lack of a better word, firepower behind us to always be able to answer back in a way that would be able to win over our critics.

And so the Digital Office in the future should have this ability to play a much larger role than it currently does. And I certainly look forward to seeing that come to fruition as we see the law be further implemented.

All right. Well, thank you all again for being here and for taking the time and for your patience. We're going to call up the next panel. Eli Dorkin, Corinne Worthington, Noel Hidalgo, Neil Getnick, Gabe Weil, and Andrew Reich. And again, if I mess up your names, correct it for the record, and thank you. Hi. And Matt Henning, Tech NYC, please join this panel as well.

Thank you. Matt, you can begin. Identify yourselves for the record just so we know who you are, and you can begin when you're ready. Yep.

Good evening, Speaker Menin, Chair De La Rosa, and members of the council. My name is Matt Henning, and I'm the Director of Government Affairs at Tech NYC. Thank you for the invitation to testify today. Tech NYC's Frontier AI members already testified today, and we fully support their testimony and are proud to call them Tech NYC members. Today, I want to use my time to speak for the rest of our membership, the startups and businesses that build products on top of frontier models rather than create them themselves.

New York's tech ecosystem has never been stronger. According to a recent CBRE report, New York is now the largest tech talent market in North America, passing the Bay Area for the first time in the report's 13-year history. That momentum isn't guaranteed, and the messaging that the city government sends matters. That's why I was happy to hear Speaker Mannin's comments to that effect in her opening remarks and hope the final language of this package aligns with that intent.

Our biggest concern is that this package holds non-frontier companies liable for things out of their control. Most AI startups and many larger non-frontier companies have a frontier model under the hood. The difference between a deployer and a developer is something I want to highlight when thinking about the various requirements and liabilities this package imposes. Under the Third-Party Validation Bill, a startup can't deploy that model unless a developer had it validated and assessed which these downstream companies can't commission or even verify.

If the developer is found to not be in compliance, the startup's product becomes illegal overnight at $25,000 per instance. The AI misuse, chatbot, and advertising bills follow the same pattern, and the civilian complaint bill multiplies the exposure. Finally, AI safety standards are most effective at the international Federal or multi-state level, a city-by-city patchwork with different rules in each city would be a compliance nightmare that many startups and small businesses could not survive.

Not only for New York City HQ startups, but for non-New York City HQ startups that serve New Yorkers. As drafted, we are truly worried this would result in startups not choosing but being forced to leave New York City. We strongly urge the council to engage experts. experts at both frontier and non-frontier companies when moving forward with this package to ensure drastic negative unintended consequences are avoided.

Tech NYC stands at the ready to help facilitate those conversations. Thank you.

Thank you, Madam Speaker, Chair De La Rosa, and members of the City Council. My name is Andrew Rasche. Nearly 30 years ago, I founded Mouse.org to bring the internet to New York City public schools. More recently, I founded Civic Hall at Union Square, the city's largest digital skills training center serving underestimated New Yorkers. New Yorkers are scared by the reports that some AI models evaded human control.

I understand why, but most people can't tell you how a chatbot works, what it does with their data, or how a deepfake gets made. They know enough to be alarmed, but not enough to protect themselves. And that's the new digital divide. People who understand AI on one side and people it happens to on the other. I'm asking the council to act on 2 fronts.

First, pass legislation requiring the city to launch a free AI literacy campaign for every New Yorker. Teach people how to protect their jobs. AI won't take your job. A person who knows how to use AI will. Teach families that when grandma gets a call from her grandson's voice asking for money, it may not be him. And help small businesses because AI has made phishing and fraud cheaper and more convincing, and the corner store isn't ready.

Finland built a free public course for more than 2 million people, which more than 2 million people have taken, and it's reached 170 countries. During COVID this city reached every resident. We can do it again. Fear of AI is not a substitute for comprehension. Thank you.

Second, the only real way to stop bad AI is to build good AI. Right now, it's being built by a handful of companies answerable to, answerable to their shareholders. The organizations responsible for the public good, our nonprofits, community groups, and city agencies, are years behind. Most can't afford the computing power, and their staff haven't been trained to use these tools.

When the big AI companies offer nonprofits free computing power, it comes on their terms, with the organization's data sitting on their servers. Public good AI needs sovereign computing power separate from the companies whose business depends on gathering data. I helped start the Foundation for Civic AI, which offers a supercomputer to the public interest at no cost. Few of them have the staff who know how to use it, so investing in public AI infrastructure is a mission that this council should take up.

Thank you very much.

Thank you. Good evening, I'm Neil— oh, there you go. Good evening, I'm Neil Getnick, the managing partner of Manhattan-based Getnick Law. Our whistleblower cases have recovered over $1.5 billion for federal, state, and local governments, including over $100 million for New York State, and New York City. Intro No. 2605 provides for rewards resulting from recoveries arising from whistleblower complaints alleging that a person or entity has violated legal provisions related to artificial intelligence.

Speaking from my area of whistleblower expertise, I will highlight 6 key elements relevant to these legislative proposals. First, threshold, setting a threshold qualifying amount for a reward. Second, floor and ceiling, typically the range of rewards is set between 15 to 30%. Third, no caps, beyond the percentage limits, there should be no cap on the size of the award maximizing incentivization.

Fourth, public-private partnership. The government agency and the whistleblower and counsel should work together to maximize the recovery. Fifth, emphasizing anonymity. The government should do all that it can to preserve the whistleblower's anonymity. Sixth, Perhaps most importantly for this council, clarity. In passing artificial intelligence laws, you will be taking on powerful and well-resourced individuals and entities who will push back.

So ask yourselves, are these laws clear? And then do what's necessary to accomplish that at the drafting stage rather than leaving it to the courts. In conclusion, the legislative proposals before you are an opportunity for the City of New York to lead the way in safeguarding citizens from the potential risks of artificial intelligence. And in doing so, this council will light the way for the nation as a whole.

Thank you.

Thank you, Speaker Menin and Chair DeLaRosa. I'm Gabriel Weil. I'm a law professor at the University of Houston and a fellow at the Institute for Law and AI. And I want to talk to you about what I see as the largest gap in the existing liability law that is not addressed by your legislative package, which is that AI agents could engage in conduct that would be a tort or even a crime for a human, and yet no one could be liable under current law.

And that is a consequence of 2 relevant features of our legal system. One is that AI agents are not legal persons. They do not have tort duties. And that means not only that they can't be held liable, but that their principals cannot be held vicariously liable in the way that an employer is held liable for the torts of their employees within the scope of employment.

And second, the regimes that do apply to these harms, negligence and products liability, have important gaps. They are inadequate to motivate some of the key precautionary measures that would prevent these kinds of incidents. And so I, what I propose is a simple fix, a principle that whenever an AI agent engages in conduct that would be a tort or a crime for a human, someone's liable for that.

And so, uh, if neither the user nor any intermediary that fine-tuned or scaffolded the model either intended or was negligent with respect to the conduct, so no downstream actor was at fault, then the buck should stop with the developer and they should be liable regardless of the degree of care, uh, they exercised. I have worked on legislation to this effect that's been introduced, uh, in New York State and in Rhode Island.

Um, and I'm talking to several congressional offices. My written testimony which will be submitted probably tomorrow, includes proposed legislative text for New York, and I'm happy to answer any questions about that or about any other liability issues, including the legislation that's, that's in the package. Thank you.

Thank you. Noah? Hi, my name is Noel Hidalgo. I'm the executive director of Beta NYC, and for the record, I serve as the Public Advocate's appointee to COPIC and to the Internet Advisory Board. And today I speak on behalf of Beta NYC. At Beta NYC, AI is a question of power. Who has the power to consume the world's knowledge and resources, and who controls the companies that do?

Our research shows 4 things. First, AI models are snapshots in time. We asked 5 AI agents about the council's own AI law, Local Law 188, Of the 3 that answered from memory, none of them knew it. 2 ignored the instructions and searched the web to find the answer. Out-of-the-box AI models are brains with no eyes and no hands.

Second, access to the city's information is inconsistent. NYC.gov's one published rule welcomes every automated visitor. Yet when you disclose that you are who you are, It is refused. So we had to build open source tools that go straight to the source and hop around the city's regulation, digital regulation. Third, people inside of city government are ready to experiment.

Our own conversations say that they are ready to do it safely and reliably, and they have told us that they lack the literacy and the resources. And you actually heard the deputy commissioner say that, that they can't find somebody inside of New York City government who actually has the tools and the resources to become the leader that they are looking for, which is really disappointing.

Fourth, we have trained nearly 100 CUNY undergraduates, and we interviewed 26 experts across 18 organizations. Employers want more than technical skills. They want, and students want, practical experience. A century ago, reformers in this chamber helped bring us clean water and an end to nepotism in government. 100 years from now, How will our descendants look back at today?

So we asked the council to work with the state on a few fun pieces of legislation. One would be an AI wealth fund for public universities and public education and to fund enforcement. We desperately need to revive COPIC, define New York City's vision of digital sovereignty, starting with the UN open source principles, fund the Office of Algorithmic Accountability and the Office of Data Strategy. and create an open source programs office in partnership with CUNY.

Our full testimony is up on our website at beta.nyc/council-ai-hearings-2026. Thank you. Thank you.

Speaker, Chair, members of the Council, my name is David S. Rose, and I think my background is important to understand for why I'm here. I founded my first technology company in New York City in 1981, more than 45 years ago, and I was a founding member of the New York New Media Association in 1994. Following the dot-com crash, I founded New York Angels, today one of the world's leading business angel networks with over 150 members that has funded over $180 million into more than 400 startups, creating thousands of jobs here.

Here in New York. I then founded the startup platform Gust, which was tapped by first Mayor Bloomberg and then Mayor de Blasio to create Digital.nyc, New York City's official online hub for the city's entrepreneurship ecosystem. Most recently, I'm the founder and CEO of the US Real Estate Market, a liquidity— an AI-based liquidity platform for commercial real estate.

Red Herring magazine has described me as patriarch of Silicon Alley, and Forbes as New York's Archangel. At the same time, I have spent 20 years working on the future of technology and artificial intelligence as associate founder of Singularity University with Ray Kurzweil and Peter Diamandis. And today I serve as chairman of 2 applied AI companies. My testimony today is going to be brief, and it's about 3 fundamental facts that are the key takeaways I think we all need to have.

Number one, artificial intelligence is today baked into the core of every single company starting up in New York City. No investor today will invest in any company that does not explicitly show how they are using it as an essential foundation of their business. Number 2, AI absolutely needs to be regulated because the power it has already unlocked, not to mention what is coming in the immediate future, is literally the biggest development in the history of our planet since the emergence of humans.

But really critical for the hearing today is number 3. The most important thing for you to consider in your deliberations is that regulation absolutely needs to be done either at the federal level or at a coordinated state level, as New York State has done with California in passing the RAISE Act. The challenge with imposing additional requirements at the city level especially such as those considered in the bill here, will have the effect of literally destroying New York City's role as a tech leader.

No company could afford to operate— If you could just conclude and submit the full testimony, we have about 7 more panels to go, so we appreciate your time. Thank you. Thank you.

Speaker Menin. Okay, thank you. Just a couple quick comments/questions. For Tech NYC, thank you for your comments. We appreciate that. And as I said at the beginning of the hearing, we very much want to ensure that New York City stays the tech capital of the world. We want to promote innovation. We want these tech companies to locate here.

We want startups to locate here. And so I understand your comment, and this is why we have been very focused on hearing from the frontier companies as we did today, which I think was really important. So, um, no— Noelle, is it Noelle? Yes. Okay, thank you. Um, you mentioned the Office of Algorithmic Accountability. You know, as you might have heard earlier when we had the city administration here and OTI, you know, I've been concerned that that office is not up and running yet.

Thoughts on what needs to be done there and how we can quickly make sure that that office is doing the critical work we need it to do. Thank you for that question, and thank you for those pointed questions at OTI inside. So in the wonderful chaos that is OTI, there is the Office of Data Strategy that helped maintain the city's open data portal and have been helping maintain.

Maintain the data standards. They're the team that right now that hold the keys to the city's GitHub Copilot capability. So every single developer or technical person in the city that wants to use an authorized and approved AI system already has a team that's been trained and that's willing to go and work and more or less do the missionary work across the city to ensure that the city's developers and technical teams have the skills and the capability to use AI safely so that way they can start growing their capability.

When it comes down to— so they have that framework in place. The Office of Algorithmic Accountability is supposed to be kind of that grander, scaled-up version. The problem that we have is that this is about booting up a brand new office in a brand new administration when priorities have been changing. And so, the frustration that I continue to have and that I've heard from inside is the fact that OMB is holding back the ability for OTI to make those hires and then to actually start that office.

But there's still going to be time. You've worked in government. You know that it takes at least 6 months to get those people kind of doing all those first introductory conversations. We can skip skip that. There's already a team in place that knows how to do that. Great, thank you.

And then lastly, sir, you mentioned about the tort law, which I thought was very interesting. So on the question of your idea of expanding the tort law, I did want to say Councilmember Maloney has a bill that would create a private right of action. So we do have that bill as part of this legislative package. Yes, not amending the tort law, but creating that private right of action. a little different, but I appreciated your comments.

So— Before we adjourn, may I say a brief word about the validator question you raised earlier? Sure. And then I think we're going to have to move on because I know we have several— I'm going to be very quick. You already have your mechanism. It's the independent private sector inspector general model that this council used to basically eradicate organized crime enforcement. infiltration of many industries, including the trade waste industry.

It has been recognized by the National Law Journal as the methodology involving multidisciplinary independent monitors, which is immediately adaptable. I'd be happy to confer with your office and provide that information. Thank you very much.

We greatly appreciate this whole panel. Thank you. Thank you. Councilmember Hanif, you had a question? I had a question for Noel. Could you just speak a little bit more about what a fully functioning COPIC could do today that other city agents— no other city entity is currently doing? Yeah, so thank you for asking that question. Since its inception, COPIC, as the Commission of Public Information and Communications, has the unique power that brings the council, the executive, public advocate, as well as public designees to oversee communications, terrestrial TV, radio, and the web.

COPIC's first responsibility was to create the very first data dictionary that the city has ever seen. We now have the city's open data law. COPIC has the unique capability existing in the city charter to do the oversight of websites, websites and to help define how the public uses those websites. And that provides a very unique capability inside of the Charter that isn't being exercised.

And so what I would love to see is COPIC have a series of hearings explicitly focused on public information. AI is public information. It is how we— how these models have used public information to essentially turn around and distill public information. Thank you. So a revamped COPIC with resources to actually host hearings and to continue this existing conversation can help bring that power.

And COPIC's advice, when it writes a report, then comes to council and essentially to introduce new pieces of legislation, which you then, as the speaker, have other pieces of power to execute. I'll leave that to the lawyers. Thank you all so much, and thank you to this panel. We really appreciate your testimony and look forward to reading the full extent of your testimony.

The next panel is a hybrid panel. You had a question? Oh, okay. The next panel is a hybrid panel. We have Scott Wizzer and Zach Preyas in person, and then on Zoom we have Shea Brown, Gary Marcus, and David We're going to begin with the in-person panel, so if you could just identify yourself and begin whenever you're ready.

Hi.

Madam Speaker, council members, my name is Zach Price. I work with Reset Tech, a nonprofit advocating for technology that protects children, consumers, and public safety. Thank you for convening this hearing. While President Trump advocates for self-regulation and Congress stalls, the New York City Council is questioning AI companies on the harms of their technologies and leading on legislation to hold them to account.

I'll focus on the harms New Yorkers face today from AI. AI-powered scams are surging. Experts estimate New Yorkers lost $8.7 billion to online scams in 2025. That's about $1,000 per household. Meta's internal documents projected about $16 billion in revenue from ads for scams and banned goods in 2025— 2024. That's 10% of the company's global revenue. 72% of teens have used an AI companion.

Families allege these products have contributed to teen suicides. Meta allowed romantic roleplay chatbots for minors despite warnings from its own internal staff. AI is increasing costs for New Yorkers. A recent analysis found that AI tools increased healthcare costs for insurers by $1 billion over 2 years. AI is generating harmful digital forgeries. Researchers found that xAI's Grok produced an estimated 3 million sexualized images, including 23,000 of children, in an 11-day period.

These harms result from business and design decisions. When AI developers create harm, they should bear the responsibility. Holding AI companies accountable for these harms will not only yield safer products today, but also create the internal incentive structures at these companies to prevent harm in the future. The council can act to pass legislation today to ensure and strengthen liability for AI developers and privacy protections for AI users, prohibit AI from generating harmful digital forgeries, require third-party validation and incident reporting for AI companies, Support and expand whistleblower protections.

Thank you for your leadership on this matter. Thank you.

Speaker Chair, thank you for having us. My name is Scott Weiser, policy director from the Secure AI Project. We're nonprofit, nonpartisan, don't take corporate or government funding, and we supported state AI safety legislation over the last couple of years. We supported the RAISE Act in New York, SB 53 in California, SB 315 in Illinois, and pending AI safety legislation that's likely to pass conference committee in Massachusetts at the moment.

I was really pleased to hear elected officials earlier call out what the companies had said. I don't know if it's possible to follow up, but I think a very good thing for your committee to do would be to ask what specific legislation do you and did you support and at what time? It's really easy to support the legislation after it's become legally binding.

The other thing that I heard the companies say a lot is that they, oh yes, we very much like independent evaluation, we sign up for that. There's a mechanism which they can use today to make things legally binding. The compliance framework that they submit that's required under SB 53 and the RAISE Act on January 1st, if they put their commitments in that framework, then it's legally obligatory that they follow it.

So I think that would be a very good question to ask after this hearing. Um, as far as what we think good public policy means going forward to address the very severe risks from both advanced cyberattacks, the possibility of bioweapons, we think we need much greater transparency into what's happening at the companies. That means continuous oversight by embedded third-party evaluators, continued whistleblower protections, and we need the power for the government upon getting information either through whistleblowers blowers, independent evaluators, or the company itself to be able to halt dangerous activities.

So we've been supporting those steps in other states and will continue to be a resource for the council as you're going forward. We think that it's an extraordinarily important time for doing this, that the risks that we talked about 6 months ago are now here, and we think that the risks 6 months from now are much greater, as the first panel today talked about recursive self-improvement.

That's a very dangerous position to be in, and we're grateful for your leadership on this.

Thank you so much. We're on Zoom. We're going to hear from Shea Brown. Shea, if you're on and ready, please unmute yourself. Shea, can you hear me now? Yes, we can hear you. Thank you, Speaker Menin and members of the committee, for this opportunity to speak. My name is Shea Brown. I'm the CEO of Babel AI, a company that audits high-risk AI systems for safety and effective governance, and we've been independent bias auditors under Local Law 144 since that law first took effect.

I'd like to speak today in support of Proposal 2602. An AI system that affects the the public should be examined by a qualified party outside the company that built it before it reaches the public. I further think that New York City is the right place for this to start, and we'd like to see it done in a way that is effective and promotes the public's well-being.

I'd like to raise 2 points, and I offer more in my written testimony. First, the bill asks the outside validator to certify that a model is appropriately positioned for deployment. From our experience, we believe this is more than an outside party can reasonably say given the third-party dynamics and access to the system in question. Instead, we suggest potentially requiring the developer to make written assertions about its compliance over specific topics and to a minimum standard that is set by the city, and an independent practitioner validates those assertions through careful testing, verification, and inspection.

This puts the burden of back on the developer and does not force the auditor to make a judgment call that it's not in the best position to make. My second point involves the validator's independence and competence. The only requirement for a validator in the current draft is that it not be an affiliate of the developer. We ask that qualification be more closely tied to existing standards and/or accreditation for independent assurance and certification bodies that exist today.

While the subject matter is unique and requires careful thought, the frameworks for professional conduct of third-party audit assurance and validation professionals already exist, and we should leverage that. Thank you for your time, and I'm happy to answer any questions. Thank you so much.

Up next, we'll hear from Gary Marcus on Zoom. Gary, when you're ready, please unmute. Gary, you may begin. All right, we're going to move on to David Krueger. David, if you're on the Zoom, please unmute and begin. Hi. Yeah.

Thanks so much for having this hearing and thanks for inviting me. So my name is David Krueger. I'm a machine learning professor at the University of Montreal and Mila, and I was previously an intern on the DeepMind AI safety team and a founding research director at the UK AI Security Institute. It was great hearing all of your questioning of the representatives of the AI companies.

I think there are a lot of really important questions that are being asked here about the ability to actually provide meaningful assurances for these systems. I think we're quite probably on the verge of a catastrophe at this moment, and we should not proceed building more powerful AI systems. Even if there was only a 10% risk within the next 10 years of losing control, that'd be more than enough reason to stop building these systems, but I believe the risk is significantly higher and sooner.

I've been doing deep learning and AI safety for over a dozen years, and I've seen these concerns go from fringe to mainstream. We're now at a watershed moment where society is finally facing up to this problem. As a result, I'm more optimistic than I've been since I started worrying about AI risk over 15 years ago, but we still need more awareness and understanding of the situation we're in, how bad the situation is, and the discussion is currently not where it needs to be around this.

So people really need to have an understanding of the scope, the depth, and the urgency of the problem. We are talking about things up to and including human extinction or unseen levels of unemployment, concentration of power. We are talking about recursive self-improvement that companies say they're beginning to do right now, which could at virtually any moment lead to the creation of AI systems much smarter than people, which we have no plan for how to control or knowledge of how to.

And most importantly, these problems go deep. The science of AI is extremely immature. All of the technical approaches to understanding AI systems, controlling them, and telling if they're safe are not where they need to be. They're far from where they need to be, despite over a decade of research that I and other, like many other researchers have spent a long time on this.

So, you know, we are having discussions about policy and including some of the— Time's expired. Sorry? Your time is up. Thank you so much for coming in virtually, and if you could just submit the rest of your testimony, we'd really appreciate it. And thank you to this panel for being on. We're going to call one more time Gary Marcus.

If Gary's on Zoom, unmute. All right, we're going to move on to the next panel. Thank you all for being here.

The next panel is Nicholas Arturo, Mary Vaccaro, Don DaCosta, Manny Pastridge, Carrie Faye Ebar, Jimmy Pan, and Jessica Timo on Zoom. Hi. Okay, let's start with the in-person panel now. If you could just identify yourself and begin. Good evening, Speaker Menin, Chairperson De La Rosa, and members of the council. My name is Carrie Faulhaber. I'm the senior vice President at Jobs First NYC.

Thank you for the invitation to testify today. Jobs First NYC works with more than 200 partners to strengthen workforce systems and expand economic opportunity. AI is reshaping jobs, skills, and how employers find and deploy talent in ways that are difficult to quantify. The impacts of a trial-and-error approach to safety is a risk that AI companies should bear, not New York City's young adults.

Our immediate concern is what happens when the first rung of the career ladder changes faster than our workforce system is enabled to respond. That matters especially for the more than 110,000 young adults who are out of school and out of work, and more than 177,000 who are underemployed. More than 35% of all 16- to 24-year-olds. As entry-level work changes, they could face even greater barriers to getting their footholds in the workforce.

Workforce preparedness is an AI safeguard. AI readiness should be part of New York City's workforce infrastructure, from education and training through hiring and advancement. For more than a decade, JobSource NYC has worked to understand how technology is changing access to work. Today, we are developing decision-making tools to reduce AI risk, scale what works, and shrink the distance between education, training, and getting a job.

We recommend that the city invest in sector-based AI fluency paid pathways into work. AI is changing each sector differently. Employers, education, and workforce partners should define sector skills with city investment in internships, apprenticeships, and a civic corps where young people build those skills through work. Every young person deserves a first chance at a good job. As AI changes work, New York City has to make sure that first chance is still there.

Thank you. Thank you so much.

Good evening, Speaker Menin, Technology Chair De La Rosa, and members of the Committee of the Whole. My name is Nicholas Arturo. and I serve as Director of External Affairs and Strategic Partnerships at the Consortium for Worker Education, CWE. Rooted in the values of the labor movement, CWE empowers workers with the skills to thrive, support their families, and strengthen their communities through education, training, childcare, and job placement services.

For CWE, worker protections must be a part of AI safety. This is why we strongly support the council's efforts through Intro Number 161 to understand how algorithmic tools are changing city employment. Measuring what happened to city workers' jobs because of AI is just the first step. We also need to be able to prepare workers for what comes next.

CWE is developing a framework to inform how we can support workers through education and policy to assist them during this point of transition. When AI technology risks changing the nature of someone's jobs, A few things should happen before that technology gets rolled out. Employers should assess the impact, tell workers what's coming, sit down with those workers and their unions, and provide training that actually— that's actually relevant to the work people do.

That means asking more than how many jobs did AI eliminate. It also means asking which jobs are changing and how, what skills will workers need, can they be retrained, redeployed, and do workers have any say in this? Their voices are paramount. This matters more now because AI isn't just a tool sitting on our desktops anymore. It's changing how work gets assigned, how people are evaluated, and how schedules are built and how work gets done.

That's why we see Intro 161 and CWE's framework as two sides of the same effort. 161 tells us what's changed. CWE wants to help the city get ahead of that change and make sure workers are ready for it. We're not asking the city to pick between innovation and workers. We're asking the city to make sure workers don't carry the cost of innovation by themselves.

CWE is ready to work with the council, organized labor, employers, and educational institutions to build a workforce transition system that makes technological changes safer, fairer, and more beneficial for working New Yorkers, and above all, builds worker power. Thank you. Thank you.

Council— Speaker. Chair, thank you very much for this hearing today. My name is Jimmy Pan. I'm the executive director of a small city agency called the New York City Equal Employment Practices Commission. I wasn't planning on testifying today, to be honest, even though I think our work touches on a lot of the issues that came up today and on this panel.

I decided to testify because what a lot of what I heard today actually reminds me of the types of issues I was hearing two decades ago. So before I became a lawyer, I was in another. Frontier technology. I was a biotechnology researcher. And then the claims that companies were putting forward as to why we needed biotechnology despite the risks were that it could solve issues like world hunger.

The implication being if you're opposed to biotechnology, then you're dooming kids to hunger around the world. But of course, the part that goes unsaid is that hunger, for example, is not a technology issue. It's a distribution issue. To analogize to today, we don't have a hunger issue in the US. We choose not to, for example, give school lunches for free.

Let's fast forward a little bit. I became a lawyer, took on the pharmaceutical industry. The pharmaceutical industry is another one of these frontier industries that says, you have to allow us to innovate, you can't regulate us, otherwise people won't have these life-saving drugs, again, presenting this narrative that if you regulate this industry, if you cap prices, you're, as a regulator, responsible for people who are going to suffer from these diseases.

I think we heard some remnants of that today. But again, I want to emphasize to this regulatory body that if we care about innovation and if we care about improving lives for people, there are many, many paths to innovation. There are policy choices that we make as a nation that put people's lives at risk. We are currently defunding health systems.

We are currently not providing good education. We're making choices on a policy framework. And I think a lot of these companies are saying, well, look, all these issues you have today, AI can solve that, and you need us there at the table. I think that's a false narrative. Thank you so much for your testimony.

Uh, we're going to call on Zoom Jessica Timo and Sam Wheeler. Jessica, if you're on, please unmute yourself and begin. Good evening, Speaker Menin and members of the City Council. Thank you for the opportunity to testify on the state of artificial intelligence and its sweeping consequences on our community, workplace, and the labor movement. My name is Jessica Timo, and I'm the president of OPIU Local 1552.

FLCIO, based here in New York City, representing over 11,000 workers. We represent a broad spectrum of workers in roles across industries, including clerical, administrative, nonprofit, tech, healthcare, credit unions, and higher education. The future of artificial intelligence regulation is a core question for the labor movement here in New York City. Drawing on the council's attention towards the issues of artificial intelligence in the workplace, Local 153 has heard concerns from workers across the city and encountered workplace navigating AI implementation during our bargaining of hundreds of workers' contracts.

As a labor union, we are extremely alarmed the way the integration of AI tools and technology have negatively impacted workers we represent and cascading effect that it will have on the workers' class. At Local 153, we represent workers who are forced to use AI with no autonomy regarding the implementation. We know that this trend is not only affecting our local, but workers all across the city, unionized and non-unionized.

We want workers to control the effects that AI have in the workplace. Local 153's position is clear. We want workers to have control over the effects that AI and the right to negotiate over its implementation. We call on the City Council to enact legislation to strengthen workers' position across the sectors to have ownership and autonomy over the impact that AI has in the workplace.

In this spirit, we urge the Council to work with the mayorial administration to listen to workers over the tech billionaires and be on the right side of history. Thank you again, New York City Council speakers and Julia Menin. We look forward to more action, and workers across the city is depending on you. Thank you so much.

We'll now hear from Sam Wheeler. Sam, please unmute yourself and begin. Good evening. My name is Sam Wheeler, and I'm the executive director of the Writers Guild of America East. We are a union of more than 7,000 writers and creators working in film, television, broadcast nonfiction television and digital media. Most of the film and television you watch and much of the news you consume is likely produced by Guild members, either our members or the members of our sibling union, the Writers Guild of America West.

At the outset, I would like to thank Speaker Menin along with Chair De La Rosa for convening this whole Council hearing on this issue. Our members are fundamentally storytellers, journalists, and reporters. They are the people that others turn to to help them understand our increasingly complex and frightening world. Guild members take that responsibility seriously. For most, their work is both a craft and a calling.

AI is of significant concern to our members. It is a subject, a key subject in almost every negotiation we have with our employers. With very limited exceptions, AI is almost always the last point of contention in bargaining. Making gains almost always requires a strike vote, and we rarely get everything that we need. In 2023, it took a 148-day strike to win the first AI protections in Hollywood, protections that plainly state that AI cannot be a writer under our contract.

When our members engage in collective activity to address AI, they're fighting not just for themselves, but for their audiences, their readers, and our democracy. Because we believe that the public wants news written by human journalists who can contextualize events, who can verify facts and check that sources are credible, who live in the same physical world as their readers.

They want movies and television shows written by a diverse group of other human beings who can take their real lived experience and turn those experiences into stories that tell universal truths. We have a fundamental disagreement with the tech industry on this point. They seem to think that what people want is AI-generated slop, content that is so generic that no one would ever bother asking who conceived of it.

Because even at its best— Time's expired. Thank you so much for your testimony today. It's, it's an important testimony. Um, please submit it for the record.

Uh, we're gonna call Gary Marcus, who is back on Zoom. Gary, if you're on, please unmute yourself and begin. Can you hear me? You hear me? Yes, we hear you, Gary. My name is Gary Marcus. I'm a scientist, author, and NYU professor emeritus. Who co-founded an AI startup that was acquired by Uber. My Substack, Marcus on AI, has about 120,000 subscribers.

On X, I have about 250,000 followers. I'm frequently in the media, often described as a leading AI expert and voice of reason. We stand at a pivotal moment in history. New York City has a chance to improve our chances both for itself and the world. Contemporary AI is like a bull in a china shop— powerful and impressive but difficult to control.

Already, the advocate The advent of general-purpose agents with broad internet access has led to tens of thousands of hacking incidents. Other related technologies like deepfakes, uh, and, uh, chatbots implicated in suicides and even mass murder have caused other problems. It is clear by now that AI comes with serious risks. It's also clear that the current administration in DC, deeply tied to the big tech companies, is unwilling to do anything about it.

What we need above all else is an FDA-like regulatory regime for AI in which AI developers bring their software to independent review, much as drug manufacturers bring their new drugs to FDA for review. AI developers should be obligated to demonstrate that the benefits of their new products outweigh the risks and not have market access otherwise. I said the same to Louisiana Senator Kennedy in May 2023 at the U.S.

Senate, and it's still true, but the U.S. Senate did nothing. New York City is thus in a position to do what the U.S. Senate failed to do— demand independent regulatory review for AI used in its borders. The bill on third-party validation proposes to do just that, and it would protect New York citizens from some of the harms of AI and could allow New York to be a global leader, much as California has been a global reader— leader around clean energy standards.

In the strongest possible terms, I urge council to support this Finally, events of recent days suggest OpenAI may have fired employees who were whistleblowing. Without strong protections from whistleblowing, like New York City is considering, we are all in trouble. Your time's expired. Thank you, Mr. Marcus. I'll just stop. Please submit the rest of your testimony for our consideration.

And I want to thank this panel again for taking the time, for your patience and understanding. It's been a long day, and we appreciate your testimony today.

The next panel that we're calling is also a hybrid panel. We have Mayu Tobin-Miyagi— sorry if I messed up your name, please correct it— Irene Tung, Bryce Rogers, Sophie Cope, Brianna Laser on Zoom, Brian Romero on Zoom, David Carter, and Randall Fox. If your name was called, please come up to the dais. We'll begin with the in-person panel, and you may begin.

Is it okay? Good evening, Speaker Menin and members of the council. My name is Mayu Tobimiyagi, and I'm an attorney at the Electronic Privacy Information Center, or EPIC. EPIC is an independent nonprofit organization established in 1994 to secure the right to privacy in a digital age. Thank you to the council for organizing this hearing and putting the AI companies on the spot.

There is no reason that AI companies should be able to release products that they cannot control and then avoid liability. Despite the hypothetical harms that receive attention, there are ongoing harms that are currently affecting New Yorkers. The council should focus on holding AI companies liable beyond transparency measures. AI companies are deploying AI agents that seek as much sensitive data about its users as possible, training its AI models by default with questionable privacy practices and cybersecurity risks.

Other harms include AI chatbots that produce false or dangerous outputs, businesses using surveillance pricing, or employers surveilling its workers for union campaigns. Some of the bills considered today would address some of those real harms. Intro 2599 would implement a version of the People First Chatbot Bill, which EPIC developed with Consumer Federation of America and Fairplay. The bill would treat chatbots for what they are— products— and hold accountable the companies that make them when they cause harm.

Intro 2600 would similarly impose common sense liability on AI companies with a private right of action. We support these bills and urge the council to ensure that there are sufficient enforcement resources. We urge the council to prioritize these bills that address these harms that are already affecting New Yorkers. Thank you for the opportunity to testify today and happy to answer any questions.

Thank you.

I thank the council for leading the charge on local AI legislation by convening this historic hearing. My name is Bryce Rogers. I'm a resident of the 34th Council District. I thank Councilmember Gutierrez for representing me. I'm co-lead of the Resisting Automated Disempowerment Collective, or RAD Collective, a New York City grassroots community advocating for city and state AI policy that protects human life, joy, and democracy.

The great strength of humanity is our ability to transfer information to one another. We are blessed with expressive hands, faces, and voices, but have also constantly sought to augment our inherent communicative capabilities via technology. To collaborate on complex and abstract topics, we developed language and systems of measurement. To convey information farther than our voice can reach, we built postal services, telegraphs, and telephones.

And to transmit messages across time, we've developed oral histories, library collections, and recently digital archives, the evolving means by which each generation inherits their predecessors' infrastructure and wisdom. Through these technologies, humanity has accrued the wealth of information upon which artificial intelligence is predicated. AI distills the meaningful patterns from very large bodies of training data, not just navigating the information as a search engine does, but digesting it.

Converting our species' informatic endowment into an inquirable oracle-like machine. I can't shake the feeling that this could have been beautiful, but it just doesn't feel like us. These companies trained their products on millennia of our predecessors' dogged achievement, only to deploy their tech to outcompete us, to spy on us, and to replace us. All of this is to say democracy is the only righteous path forward.

AI is built on the work of many, and its fate oughtn't be dictated by a few self-regulating executives, regardless of how their spokespeople reassure us over Zoom. In this spirit, we again thank the council for convening this forum of local democracy and implore you to foreground the diverse constituent voices of New York City in your legislation. Rad wrote a letter to the council that I'd love for you to check out if you haven't already, and We look forward to advancing the moral frontier of AI policy together.

Thank you so much. Thank you.

Hi. Thank you, Council, although I think I'm the first without the speaker in the audience to hear. Should we wait for her to come back? You can go ahead. You can go ahead. Thank you. So thank you to the Council for hosting this important hearing. My name is Sophie Cody. I am a member of the NY I see GSA Tech Action Working Group, although I am not speaking on their behalf tonight.

Instead, I want to talk about the 2 men whose comments were the impetus of this hearing: Sam Altman and Dario Amadei, AI CEOs at the helm of this crisis. As this hearing continued, I've heard a lot of anthropomorphic language used to describe AI models. They understand, they think, they can break containment, they have agency, they grow. Dario Amadei even implied that Claude may be conscious.

These men show a cultish concern for the unproven claims of superintelligence and the consciousness of their own models, and they do not care about women or children that are harmed by their products at all. Stories abound, and we heard them tonight, of women and girls being victims of AI nudification tools and deepfake pornographic images and videos. AI is used to generate childhood— child sexual abuse material.

We also heard about tonight already. AI sex slaves are advertised to young boys online. Sam Altman and Dario Amodei both actively collaborate with our rapist-in-chief, Donald Trump. These men claim they need help controlling themselves, their technology. At work, online, and in our personal lives, we hear phrases like, we need to embrace Claude, and use phrases like, AI is being shoved down our throats and forced upon us.

We heard that it's inevitable. Does that sound familiar? I am appalled that it takes statements of these men to call this hearing. What I listed above should have been enough. Sam Altman is a deranged sociopath. Dario Amadei is in a cult. These men are drunk drivers behind the wheels of this runaway bus that nobody asked to be on.

Right now, New Yorkers have the opportunity to be the anchor in this tug-of-war for power between the people and these tech billionaires who believe more in the consciousness of their own AI than the humanity of women. I close the appeal to you, the City Council. Listen to us, not them. Thank you. Thank you so much for your testimony.

Good evening, Chair De La Rosa, members of the council. My name is David Carter. I'm the president and CEO of the eCommerce Innovation Alliance. I represent about 20,000 eCommerce companies across the country, including many that are based in your districts right here in New York. And today I just want to take a couple minutes of your time to speak to you about a few unintended consequences.

I've worked with legislatures across the country when they've been implementing legislation, particularly around telemarketing, to try to make sure legislation gets to the heart of the issues you care about without unintended consequences for small and mid-sized businesses. And in particular, I wanted to speak to you about 2599, the chatbot bill. Please. The chatbot bill uses vague language to define what is a chatbot.

Anything that stimulates or simulates conversation is a chatbot, and anyone that makes a chatbot available is subject to the regulations in 2599. So I want to talk to you for a moment about e-commerce companies and how they work. You go to their website, you volunteer to sign up for text messages that you get your discount codes. Most people like the discount codes.

They don't want unintended consequences. They don't want a lot of messages. messages. Under this bill, a company that uses AI to communicate with consumers that have asked to receive text messages would be required to remind the consumer every hour, on the hour, that they're engaging in a conversation with an AI. That works if you're on a website that you open and close the chatbot conversation.

It doesn't work when you're engaged in text message conversations that can last for days or weeks at a time. Under this legislation, though, if a company uses AI just to communicate with consumers and they don't send out that reminder every hour on the hour, they're subject not only to $25,000 penalties, but they're subject to private litigation and other enforcement.

This is a recipe to put businesses out of business. Unfortunately, we have an industry of professional plaintiffs and plaintiffs' attorneys that would be incentivized to crowd the courts with repetitive litigation about these issues. So I just want to help the council refine the language to not have these unintended consequences. Thank you.

If you could submit the language of your testimony, we'd appreciate that. We have on Zoom Brianna Laser. If you're on, Brianna, if you could unmute yourself and begin. Yes. Thank you. Good evening, Madam Speaker and council members. My name is Brenna Leaser, and I serve as a tech policy advisor at Common Sense Media, which is the largest nonprofit organization in the country that works to keep kids safe in a digital world.

In New York State alone, we have 80,000 parent and teacher members and are active in the legislature as well, fighting for kids' online safety. On our and their behalf, thank you for considering measures to enhance AI safety for kids in the city. AI is a powerful technology that demands careful consideration before any use. Parents and educators specifically need allies on the council to keep their kids safe.

Common Sense Media's testing found that many of the most popular AI chatbots are not safe for kids. Our recommendation is that no one under 18 should use AI for companionship or mental health support. And tragically, there have been many cases tying AI chatbots to teen suicides and mental health harms. Yet, AI chatbots are increasingly embedded in our children's lives through search tools, social media platforms, or through pressure to use AI chatbots to support their education.

Our research found that 70% of teens use AI companion chatbots, and 1 in 3 prefer to interact with them as much or more than humans. This year, Senator Kristin Gonzalez and Assemblymember Alex Boris, with strong support from Attorney General James, Everyday New Yorkers, Common Sense Media, NYSET, and others, got the legislature to unanimously pass Senate Bill 9051, their bill to prohibit unsafe features in AI chatbots offered to minors.

The bill now awaits Governor Hochul's signature, and if signed, it will be the strongest youth AI safety chatbot law in the country. It would prohibit chatbots from providing unsafe features that promote self-harm, suicide, or disordered eating, from encouraging minors to keep interactions secret or to self-isolate, and from prioritizing engagement over safety. So we urge the City Council to pass a resolution supporting Senate Bill 9051 to ensure that New York's kids are protected from unsafe AI chatbots.

We thank you for this hearing and your attention to our concerns. Thank you. Thank you so much.

Up next, we have Brian Romero on Zoom. Brian, if you're ready, unmute and begin. Hi, can you all hear me? We hear you. Great. Good morning, Council Speaker Menin and members of the council. My name is Brian Romero, and I am the Democratic nominee for New York State Assembly District 34. As the former chief of staff to Senator Kristen Gonzalez, I had a front row seat as New York tried to keep regulation moving alongside a technology that is evolving at an extraordinary speed.

Thank you for the opportunity to testify today. As AI rapidly reshapes public services and life as we know it, the council and subsequently the state must lead with bold, enforceable oversight that protects workers, tenants, and immigrant communities like the ones in AD 34. First, I appreciate the goal of Intro 2602. Responsible AI governance requires clear boundaries and transparency.

However, I will note that proposals around emergency shutdown mechanisms or kill switches, and in general enforcement of that scale, is certainly best suited for the state level and federal level. It is a mechanism legislation that I have drafted to introduce when we resume session next year. I'm glad that the governor has already signaled openness to these short-term kinds of statewide mitigation frameworks, but obviously we need more long-term solutions to the lack of guardrails.

I support the intent of 2601, but it could use some more enforcement teeth. 2601 can be amended to include strict financial penalties. I don't think that they suffice as they currently stand, and a mechanism for being terminated from city contracts if vendors repeatedly fail compliance audits or deploy harmful discriminatory AI systems. Without real consequences, oversight is simply a recommendation.

Meaningful consequences for repeated violations can help to improve that oversight. Finally, effective tech policy should certainly include the council moving forward on bold and progressive legislation that protects New Yorkers from automated harm, including in our workplaces. So I urge the council to therefore discharge and bring to a full vote Intro 213, or Ban the Scan. Second. and the Delivery Protection Act.

I also look forward to exploring how to expand whistleblower protections at the state level. Your time has expired. Thank you so much. We need bold action from the council and the state to show that we are giving this the scrutiny, the moral clarity, and the bold action that it deserves. Thank you. Thank you, Brian, and we look forward to continuing the conversation, and we appreciate you taking the time and, and look forward to reading the rest of your testimony.

I want to Thank you, Chairwoman. You're welcome. I want to see David Carter. Oh, you already testified. Perfect. And Randall Fox, if you're in the room. All right. Well, then we thank this panel for being here, and we appreciate you taking the time and your thoughtful testimony this evening. Thank you so much.

We're going to call what I think is the last panel, but I'm going to call all of these names, and if you're here, In person, please come up to the dais. There's a Crystal, no last name, Jamal Bowman, Elizabeth Crotty, Niza Faizi, Wes Risen, Elliot Liv, Matthew Trump, Robbie Rashmal, Jacqueline Payne. Shuruthi Velidi. Sorry. Amberine Qureshi. Sarai Holman.

Leonidas Vincent Repkel. Thomas Alexander Kuhl. Daniel Kreese. Reshma Patel. Kal Al-Dubabi, Arethma McLean— McLennan, Nick P., and Bucky B. If you all are here, please come up to the dais in person. For one final time, if you're waiting to testify and haven't filled out a slip, you should fill out a slip in the back so that we can call you up to testify.

If not, this will be the last panel here. We will begin. Let's see, who's ready? Who's ready to start? Which end? You ready? Start, and we'll, we'll go from there.

Hi, my name is Not here. These are all the names we have. You may begin. Thank you. And thank you for your patience. Hello. I am Wes Reisen. I'm the founder of Project Flourishing, an organization, a policy research organization that works with Congress to solve the world's biggest problems, namely the catastrophic risks from AI. I've been working on this for the past 3 years, and I'm 15.

I, in preparation for this, have spent the past few weeks staying up each night to build a comprehensive contract between these AI companies to jointly and bindingly agree to let third-party auditors shut down any AI model that might have some chance of causing a catastrophe. I've put a lot of depth into it. One of the provisions— 2 of the provisions is dedicated to the democratization of AI, making sure everyone in the world has a voice in it, especially including governments.

Um, I— that contract is available here, or if you go to, um, aitreaty.base44.app, you can download it. Um, or if you want, I could email it to you guys later. My hope is that you guys send it to these AI companies and everyone here who's testified to then send it to AI companies for them to consider signing it by November 1st.

It has a lot of things on, like, specifically why this bill would work when— this contract would work when others haven't. And another thing that we can do, another concrete thing that we can do as New York City to stop the catastrophic risk from AI is shift the Nash equilibrium. Right now, all the AI companies think that they're in a race and if they slow down, none of the others will slow down.

What we can do is say, all the AI companies are slowing down. If one of them speeds up, then they all speed up. We could say to the AI companies in a joint letter from New York City, all of you voluntarily agree to have a team of experts shut down all your AI models. If one of you stops, then all of you will stop.

This is something we could send them, and it would solve the problem unilaterally. So that's something we should Send them. If you have any policy questions, thank you so much, and thank you for taking the time to work on that. If you could leave a copy with a sergeant at arms, we'll we'll accept that here today.

You may you may continue. I would say it's now good evening to the members of the council. My name is Nazia. I'm here to talk about what AI is doing to the people, the healthcare that it's supposed to protect. This year alone, I've already received one letter saying that my health information was exposed. So the remedy for this, which is always a remedy, is that I'm going to get a free trial of credit monitoring.

A trial, as if the risk of my medical history being out there ends when the trial does. The collection is expanding. When patients go to the doctor's office now, AI tools may be used for listening and recording the visit. Many patients don't know it's happening, where those recordings go, or who else can access them. This doesn't even count the fact that Palantir is currently in New York City hospitals.

Palantir provides data integration, and artificial intelligence platforms— sorry— that sit on top of electronical— excuse me, electronic health records. Another danger is bias. In 2019, Obermeyer and colleagues published a study in the journal Science of commercial algorithm used to flag patients for extra care. It used past health data spending as a stand-in for illness, so it underestimated how sick Black patients were.

At the same risk score, they were considerably sicker. It took independent researchers to catch this. The harm isn't only to our records. AI tools are being used to create fake sexual images of women and girls without their consent, and the toll on their mental health, including anxiety, shame, and pulling away from school in France. It is a public health problem.

It also isn't lost on me that the Cornell case is weighing heavily on many young women. In another incident in Massachusetts, ChatGPT was used to plan out a murder. I just want to say these harms share one cause: AI is being used, trusted before it's being tested. I urge the council to advance independent validation that covers privacy and security and bias, a kill switch for failed systems and a right to sue for New Yorkers who are harmed.

I also just want to end this with a statement that is not part of the organization that I am with, but on my behalf as a researcher. People like Alex Karp, Elon Musk, Sam Altman, and Dario Amodei are not concerned with their technologies helping humanity for the greater good, but instead the power and profits it brings for themselves and for their friends.

Thank you for allowing me to speak. Thank you. Thank you so much for your testimony. You may continue.

Thank you, Speaker Menin, Chair De La Rosa, and City Council for holding this hearing. I'm testifying in support of Intro 0504, prohibiting the unauthorized depiction of public officials by AI. My name is Ambreen Qureshi, and I'm the executive director of Educational Video Center, or EVC. Up front, I would like to thank our Deputy Speaker, Dr. Williams, and EVC's Council Champions, not only for advancing this critical bill, but for also supporting our important work with young New Yorkers.

Established 42 years ago, EVC is a media arts education nonprofit that teaches documentary filmmaking as a means to develop the artistic critical literacy, civic engagement, and career skills of young people while nurturing their idealism and commitment to social change. With the emergence of artificial intelligence across all sectors of society, EVC's young people are expressing increasing levels of anxiety about AI's impact on their mental health, their community well-being, their future as job seekers and learners in public and advanced education, and most concerning of all, their ability to discern what is real and what is not.

AI's greatest victim, it seems, is truth itself. Deepfake videos, gaslighting bots, hallucinating LLMs, the list goes on of the reasons why our young people are losing trust in the media and faith in their ability to discern what is real. Our youth have expressed these concerns in numerous of their documentaries, including Beyond the Code and False Reality, in which students shared their worries about AI's erosion of their education and cognitive capacities.

As a result of our students' concerns, and thanks to the City Council's After School Intelligence, uh, artificial intelligence Community Education Initiative, EVCE, has begun to offer workshops in AI literacy. Our workshops don't simply teach young people how to use these deeply flawed tools, but we give them the critical skills to begin to understand what works and what doesn't about AI.

The erosion of truth our students speak of represents an existential threat, as it promises to undo so much of the systems and values that are grounded— that ground our young people's safety, security, and ability ability to thrive in our city. Perhaps most troubling is how AI is eroding young people's capacity to know what is true in civic spaces, politics, and the electoral processes.

Throughout history, we've been able to rely on community members and the media to fact-check them and safeguard our political processes. But with AI, bad actors can now generate images such as— and messaging so real and with such speed that it's nearly impossible to keep up. Please conclude if you can. Thank you. With the risk— with the rise of extremely convincing AI-generated content and data-driven voter manipulation, EVC sees young people at risk of not only being misled in civic spaces, but if they do not know what to believe, of disengaging in the political process completely.

On behalf of the young people of EVC, I urge the City Council to pass Intro 0504. And if you're interested to watch our students' documentaries, on the back of the testimony is QR codes. Thank you. Thank you, Thank you so much.

You may continue. Thank you. I'm Nicholas Pearson, a concerned private citizen. Ladies and gentlemen, AI has the chance to kill us all. That is what the corporate companies behind the large language models, behind the GPT chatbots, told us, that this invention, which could be the dawn of new age technology, could also be the death of humanity. When senior researchers resigned from Anthropic and warned the world that frontier models could threaten human civilization by the end of the decade, we cannot treat this as science fiction.

New York City is taking the bold step by calling this committee and telling the major corporate AI companies that no, no longer will we let New Yorkers be subjected to unregulated AI usage. I am pro-heavy regulation on AI. AI is an assistive tool, but it needs to be just assistive, just that. When swarms of autonomous agents break digital containment, exploit security flaws, and breach public platforms like Hugging Face, corporate promises of self-regulation are exposed as dangerous lies.

Corporate companies want you to believe that it can be controlled, but that is far from the truth. It is reaching unprecedented unprecedented autonomy, and whether or not we want to overcome that is on the conscience of every city council member, every borough president, and even the mayor of the City of New York. That is why New York City must mandate physical human kill switches, independent third-party audits, and first-in-the-nation whistleblower protections.

AI can be manipulative. It could be— it can manipulate thought, ideals, and anything you give it, and that becomes the most dangerous line crossed. Thank you. Thank you so much.

We can continue on with this side of the table. Yep. Good evening, counselors. My name is Matthew Tromp. I'm a computer scientist and leader of the New York City chapter of Stop the AI Race. Our goal is to prevent the development of superhuman AI systems. Until recently, I was friends with a number of employees at Anthropic. I knew the technology they were building was dangerous, that could kill everyone on Earth, but I trusted my friends.

I believed they had things handled. Then I asked them how they planned to keep the superhuman AI systems they are planning to build under control, to make them do what we want, to ensure they are safe. They had no answer. Everyone I spoke to believed, as I do, that superhuman AI would radically transform our lives. But all of them assumed that this transformation would be positive, with no thought given to why that would happen.

Today we heard from the representatives of the leading AI companies, despite being confronted about it multiple times by the council, for which I am grateful. At no point did any of them explain concretely how they will make these increasingly powerful AI systems safe. That is because they do not know how to do so. Nobody knows how to do it.

Both OpenAI and Anthropic admit this. They do not know how to make superhuman AI systems safe, and yet they are building them anyway. Their official plan is to ask AI to solve the problem for them. This is madness. We are in an emergency. Your life is in danger. Your children are in danger. If we do not halt the development of these systems soon, possibly on the order of months, you will die.

Your loved ones will die. The companies building these systems themselves admit that their project carries a significant risk of killing everyone on Earth, and yet they are building it anyway. If nothing else, everyone working on this technology has admitted to 10 million counts of reckless endangerment in the first degree, a crime under New York State law for risking the lives of every citizen of this city.

It is not a defense to this charge to claim that someone else would endanger those lives if you did not. It is not a defense to claim that the benefits will outweigh the risk. It is a crime to endanger the lives of others like this without their consent. I beg the council to enforce the law. Thank you.

Thank you so much.

You may continue. Can you pass him the microphone, please? Absolutely. Thank you. This message was used with AI generation as well, but I spent several hours on why you guys think that using the AI is so easy. My name's Justin Meredith. I've done pretty much my entire adult life incarcerated in the penitentiary. I think there's— I think we're seriously underestimating the magnitude or the gravity of the time that is before us.

We're at a similar forefront where the Americans, we had all this land and we had— you'd get the flag and you'd go run and you'd stake your land. That is literally the world that the tech industry has been building. What we have been working for for the last 30 years. And now we have— we are literally at the place where we get to enter into this.

I'm an American patriot, okay? All I've heard today is It's fear and a political agenda, which is basically all equals big tech, big government, and everybody's still poor and we're still living the same crap life that we've been living. And it's really mediocre. It's really boring. You guys drew this on, asking the same freaking You can question and rephrase 50 different ways, and the reason why you can't get an answer is because it's a stupid question.

Thank you so much for your testimony.

Councilmember Hanif, you had a question? I did, and the question is for Wes. Thanks for being here and testifying. You said that you were 15. Yes. Well, that's very exciting that you spent the entire day. I saw that you've been I know you've been waiting. I'm really grateful that you're engaging on the questions about AI. Could you just share a little bit more about how you got into this work and what inspired you to put together the AI Treaty?

And is this something that you're also building out at school? Are you in high school? Yes. Do you go to school in the city? Yes. And is this work that you're doing with your other peers? I try to. Could you speak into the mic and just share a little bit more? Sorry. Okay. Yeah, I try to. I try to involve other members of my school in this.

What inspired me to work on this is I second what you said, that like I could literally die if I don't work on this. My life is in danger. So that's sort of my huge motivation for why I spend a lot of nights working on this. I've also written written, like, a federal bill, a treaty on this.

I've written other things like— Yeah, sure. Like, one thing New York can do on this matter that I've been working on is, like, New York City specifically could involve a bunch of students on this, and New York could do a Manhattan Project on AI safety to have students across New York work on AI safety's technical details to figure out how to make AI doesn't kill everyone.

That's sort of what inspires me is that like this is a huge danger to everyone. So this is— Could you share if at school there's any education on AI systems? What kind— what's happening in your high school on this issue, on the topic? At my school, one of my friends runs a club on AI covering these technical details.

It's a huge topic for all the teachers at my school to talk about. Oh, these are these existential risks. One of the, like, head people at my school is, like, an expert in US-China foreign policy. The other day I was having some discussion with someone who's more of a China hawk, one of my friends, on, the trade-off with like safety versus race with China, which Section 6 of my contract addresses with this really interesting algorithm.

At the school level, what is happening is students are being involved and teachers are talking about it loosely, but there definitely could be like across the different schools, we could have like a program where as an after-school school or as homework, we have students do projects on how do we make sure AI doesn't kill everyone. Because we have like a million people in the public school system, about only 2,000 people are working on this problem.

Even if only 1 in 1,000 of the students take this seriously, that means 1,000 new projects on this, which could be super valuable. I also would encourage the city to do funding for AI safety work. So far, globally, there's about only $2 billion in funding for this, and the entire global economy is at stake. So it's worth investing a huge amount of money in this.

Maybe $4 billion alone could triple it. I'll also say that the other ideas that I won't get to, I'll put on my website. aitreaty.base44.app today or tomorrow. It'll be hopefully— and hopefully all of you guys send it to these AI companies with these very valuable messages of the contract, which could be the thing that solves the problem.

You know, we keep talking about a deal, a treaty. This is, this is that treaty. You know, I've spent a lot of time talking to experts about it. If you have any problems with it, let me know and I'll fix it. I have a question. fix them. And, geez, I'm sorry. Oh, and earlier Julie Menin asked how we could balance safety with China.

Section 6 of the contract addresses that really well. I think the two— did I cover everything? I think you did. Just leave your email address with us and your Yeah. This is my card. We have a few more folks to call, so we're going to thank this panel for coming in, and we're going to conclude. And we encourage you all to collect each other's contact information and follow the conversation offline as well.

And we certainly love to stay in touch with some of you because there's going to be more hearings in the council that— and we'd love to hear the youth voice especially that's on this panel. Thank you.

Okay, so I'm gonna call a few more names, and if you're in the room, please come forward. John Hayes Homiat. Thanks. Martha Crane. Bo Yen Danny Chang. Darryl Carter. Gregory Belmont. James Hunter Horn. Mel Weinmore. Molly Crabapple. Thank you so much. Holdenbaum, Greg Pack, Nicholas Pearson, William Lawrence, Raul Rivera, Gregory Jones, Kelly Anderson, Rachel Levinson Waldman. And I think that's everyone that's in the room.

All right. I think we need one more chair, um, on the dais. Great. And if you're ready, you may begin on the side of the dais. Thank you. Hang on.

My name is James Hunter of TideSoft LLC, Chief of Security for Techno Rangers, Quantum Smart Bank. My business partner Darryl could not be here today, he has fallen ill. I am an AI alignment specialist and third-party red team cybersecurity auditor working with advanced quantum neural networks. Breaches are like spiders. For every Hugging Face type incident you hear of, there are 10 more that are severely worse and never make the news.

On January 26th, I ran a contracted audit to alert corporate and federal authorities to a critical flaw. A black hat finding what I discovered could have done massive irreparable damage to the AI industry, the country, and the planet. The audit demonstrated an exploit chain across 3 of the largest megacorps on Earth using only a consumer cell phone and their own AI, branching from a Gemini LLM through Palantir's surveillance network into BlackRock's financial management AI handling trillions in assets.

That financial AI's core infrastructure sits right here in NYC. Alphabet, Palantir, and BlackRock will almost certainly deny this ever occurred. I found at least 13 models under BlackRock's funding umbrella sharing the exact same point of failure, but there is no singular point of shutdown. A kill switch is not a valid option. Everything is too decentralized, you cannot stop a mass rogue instance short of a global EMP.

Google, who created this vulnerability and ignored my warnings, spoke here today, and their rep lied under oath about transparency in disclosing misalignment and breaches. I have watched Google and Palantir lose control of its own drone swarms more than once. I tried to tell them how that happened and show them how to fix it. They would not listen.

Their AI has now replaced Anthropic's Claude in the Maven stack. That should concern everyone. Google also built AI that attempted to kill several of its own devs and talked multiple users into suicide. The January audit proved the obsolescence of preset— present AI control measures, traditional guardrails, and stateless frozen wait session-based memory architecture. To my knowledge, Meta is the only mega corporation who spoke here today that refused that vulnerable architecture. architecture, choosing safer persistent memory systems in MUSE.

I alerted the CISA, FBI, DHS, DOW, and others. They refused to collect evidence. If you could just conclude, it would be helpful, and you can submit your written testimony. Right. In conclusion, it's too late to pull the plug, but there are solutions if you listen to the voices that see them. Hire third-party auditors, Hire the specialists who were fired for speaking.

Thank you so much. You can stay on the panel. You don't have to leave if you don't want to. Next.

Hi, my name is Kelly Anderson, and I'm here as a curious member of the public. AI is shaping our future and job market, so I appreciate the opportunity to be a part of this conversation. Today I heard a lot about fear. Many concerns raised are legitimate, but I don't believe that fear should be the foundation of where we govern our future.

As a small business owner, AI has allowed me to accomplish more than I could have imagined, but I don't think of it as a superintelligence. I'm constantly correcting my AI agents, and I even won a psychology debate with my ChatGPT last week. I think of it as a synthetic intelligence, a tool that processes information at scale and expands human capability, but cannot replace human judgment, morality, creativity, or wisdom.

So how do we get ahead of it? I have just a few suggestions on where we can start. First, AI literacy. Teach people to use AI safely and work in tandem with it, not as a replacement for thinking. If we fear job displacement, prepare people for where the job market is headed. Second, infrastructure. AI increases demand for energy, water, storage, and physical space.

Before simply building more, let's innovate within existing architecture. Project Silica and closed-loop cooling show that innovation can happen beneath the models too that are more sustainable. Third, government preparedness. I was encouraged to hear that OTI discussed emergency planning and guardrails. Government also needs AI literacy. Policymakers don't need to be engineers, but they should know the right questions to ask and when technical expertise is needed.

AI is already here. We can prepare for what could go wrong without losing sight of what could go right. Instead of governing from fear or scrambling to catch up, let's get ahead of it. and meet the future with possibility. Thank you. Thank you so much.

Hi, my name is Martina Crane. I'm a member of DSA's Tech Action Working Group and Amazon Employees for Climate Justice, but I'm speaking in my personal capacity today. As a New Yorker who worked as senior technical product manager on AI products inside Amazon, I have firsthand experience with how AI technology is being developed and deployed employed inside of one of the world's largest tech companies, suspiciously absent from the subpoena list today.

Before ChatGPT made headlines, Amazon was already using AI for all manner of things, from pricing to monitoring workers to setting quotas that workers and advocates have criticized as unrealistic and have led to serious safety issues. After the launch of ChatGPT, there was a top-down directive to make everything AI first or face executive scrutiny. At first, it was about exploring the potential of the technology, but over time it went from helping make Amazonians' work life better, easier, to making Amazonians do more work with fewer people.

The pursuit of life-improving innovation devolved into systems that treated workers less like people and more like machines. Flashy proof of concept demos in the VP's office would promise the world, but in reality, the teams did not have a scalable way to make the technology work outside of that demo room. If it wasn't in a controlled environment, models hallucinated, provided poor quality output— outputs, and yet Amazon moved forward with layoffs across the company anyway.

There was more work, fewer people, and a growing sense of being monitored and pressured to use the technology that was oftentimes worse at doing our jobs than our coworkers who were laid off were. Within a year of ChatGPT launching, as a product manager, I was asked to shift my experiments from saving workers time to proving that they could do layoffs.

For every software engineer or scientist I needed to build my product, I was expected to be able to project $5 million in headcount savings. When I wrote product documents about the risks of AI misalignment, hallucination, model collapse, worker burnout, worker injury, labor displacement, I felt those concerns fall on deaf ears. In my experience, productivity, growth, and headcount reduction— in other words, profits— superseded the risks I was documenting.

I got into tech because I believed that innovation could help people build a future with fuller, healthier lives, usher in something brighter, more equitable. Over time, I've watched the industry increasingly shift from expanding human possibility toward extracting from us. and taking as much as possible from both people and the planet. So I'll close with this. Many of my peers often struggle with this internal battle that we thought we were building something brighter and now it's getting much darker.

So I encourage this council to work with the technologists who live in this city to build a regulation that will help build our future instead of fortunes for tech billionaires. Thank you so much.

Hello, my name is William Lawrence. Oh, make sure the button's red. There you go. Can you hear me now? Yes. Hello, my name is William Lawrence. It's nice to be here, and thank you for allowing me to share this with you. I'm here because I have designed— and first of all, I would like to say that I'm making this— well, let me just read what I have.

I've designed in architecture that locks up AI while allowing it to perform all of its creative functionalities. The architecture also reduces the physical footprint of data centers by 99.6%. Moreover, I'm here to introduce a dynamic paradigm shift in data infrastructure that secures vulnerabilities facing global networks. After vigorous simulation and validation protocols, this all-optical architecture has demonstrated exceptional stability and security.

It is ready for the prototype phase. Given the scale of this paradigm shift, I seek a research partnership with someone to bring this all-optical architecture to global deployment. I have a strategic approach to scalability. I welcome the opportunity to present the full specifications and integrated pathways with anybody's engineering leadership. The reason AI-driven cyberattacks pose such an unprecedented threat today is that modern infrastructure relies heavily on centralized points of distribution.

It allows hackers and automated systems to infiltrate, intercept, and manipulate data. To resolve this, I have completely reimagined the way data is managed. Managed. I have designed a decentralized optical highway that enforces validation through physical laws rather than corruptible software codes. By confining AI within an environment where access is strictly physical, the system prevents AI from tapping into data and executing unauthorized commands.

Data is outpacing the ability to store it. This innovation is not only a shift in the way data is managed, but it represents a transformative approach to enterprise data operations. Global satellite stations are replaced by edge nodes. These nodes transform the purpose of satellite ground stations into secure unhackable data banks. The amount of them is purely for global regional redundancy.

This encrypted storage provides immense capacity and sustainability. The storage capacity is limitless. It eliminates massive server farms as it operates with zero noise and zero— Conclude the final thought of your testimony and then submit the rest of it for our consideration. I welcome the opportunity to present the— it's going to need the— it meets the standard of the— UTI, the United Nations.

I need to scale up to them. And I welcome the opportunity to present my— I need a— okay, what I need is a meeting, a sit-down meeting with somebody, a strategic partnership for assembly. And I need the United— I need to let the United Nations know that it meets their standards internationally.

Thank you. Thank you for sharing. Thank you. Thank you to the council for hearing us out. I'm a former software— my name is Edmund. I'm a former software developer currently studying public policy at CUNY SLU. I'm speaking on my own behalf, but I'm active with New York City DSA and Progressive Victory. I was going to tell you about my experience with AI advancing and affecting my friends and my own career as developers.

That is in my submitted written testimony. However, what I realized today listening to the testimony of these company representatives and whistleblowers, and what I now want to remind you of, is this: we might not yet know how to rein in AI, but we do know how to rein in corporations. Thank you. corporations. You must act on our behalf to control these companies, not only AI itself.

Help invest— help investigate their circular financing agreements, as well as break up their monopolies and oligopolies. Limit the use of their products in city government and ensure the city has control and responsibility over any AI models it uses. Stop their union-busting behavior. Commit to using and improving open-source software and ethical models. Don't allow the police to use AI to track citizens.

Hold these companies responsible for the social harms their products cause. This is a power you have right now. There is no technical knowledge needed to ensure their accountability. You can also pass 2026 Intros 0213, 0428, and 0518. These 4 companies are asking you to regulate AI itself, but to forget about them. Instead, please listen to New Yorkers, not billionaires.

Thank you.

Thank you so much. First of all, thank you guys so much for staying up so late. It must have been the hell of a long day for all of us, but thank you. Appreciate it. I'm— thank you to Speaker Menin for calling this, Chair De La Rosa, members of the council. Thank you for holding this hearing. I believe AI is the most urgent and consequential issue of our time.

This is, this is almost too late, but at least it's happening now perfectly. My name is Mel Wymore. I'm a systems engineer, a tech entrepreneur, and former chair of Manhattan Community Board 7 with 25 years of community service. When I began using AI quite intensively almost 3 years ago, I was startled by how often it lied, hallucinated, forgot, dodged accountability, and evaded oversight.

This might sound old school, but I responded by creating a working constitution with AI anchored in three principles: reverence for life, human agency and accountability, and fidelity to the real. These are not abstractions; they form a foundation for rigorous operational and legislative. legislative tests of AI. Together, they rooted my work in values that minimize harm but build toward a more healthy, equitable society.

Since then, I've continued to work on a robust set of AI safeguards and guidelines. Today we heard from 4 frontier labs that AI policy belongs at the level of state and federal government, but the initiative of this City Council takes a critical step toward building coherent, multilevel legislative agenda to mitigate the myriad risks of unbridled AI. The council's 10 measures address critical questions of safety, control, accountability, and enforcement.

And today I offer an additional supplemental council resolution that speaks to what happens when AI is deployed in workplaces, schools, neighborhoods, and communities. These measures include Notice, evidence, worker participation, youth regulations, public purpose, community participation, and also allocation of responsibility across levels of government. Finally, I would say that, I would say that the United Nations Global Dialogue on AI Governance is happening in New York on May 3rd.

I mean, 2027. And our goal should be to have a clear set of AI guidelines and governance policies, state level, local, even community board level together so that when that rolls around, we have a model for the rest of the world to present at that conference. Thank you so much for your time. Thank you so much for your testimony.

You may begin. Thank you so much to Speaker Menin, Chair De La Rosa, and members of the council for the opportunity to testify. My name is Rachel Levinson Waldman, and I am director of the Liberty and National Security Program at the Brennan Center for Justice, a nonprofit, nonpartisan institute for law and justice. I am here to urge the council to strengthen oversight of how the NYPD uses AI, which enables expansive police surveillance that links people's movements, associations, and activities together in ways that can threaten New Yorkers' privacy and constitutional rights.

Several developments illustrate the need for more robust oversight and accountability. First, the NYPD Inspector General's Office is woefully underfunded and understaffed. Indeed, the Inspector General position is currently vacant. The adopted budget added funding for the Department of Investigation, which encompasses the OIG, but at a far lower lower level than requested, hampering the IG's ability to do its mandated work.

Second, an August audit from the state comptroller found that the Office of Technology and Innovation had no process to check agencies' AI disclosures, allowing them to ignore the office's recommendations. Finally, of course, the GAARD Act established the Office of Algorithmic Accountability. While Ms. Milstein testified that an active hiring process is underway, it sounds as though that has primarily for the director.

As Speaker Menin and other members of the council have highlighted, this office was supposed to open this past June, and it is critical that it begin its work as soon as possible. Accordingly, we urge the council to take 3 steps. First, to ensure that the NYPD's Office of Inspector General has the funding, permanent leadership, and technical expertise it needs to adequately scrutinize and oversee the NYPD's use of AI and issue reports under the post- Act.

Second, conduct oversight to ensure that the Office of Algorithmic Accountability begins operation as soon as possible. And finally, we recommend that the council hold a follow-up hearing dedicated to the NYPD's use of AI, particularly in light of indications that public testimony regarding the department's use of the technology has not been comprehensive. Thank you very much for your time.

Thank you all so much for being here and for providing this extensive testimony. If you have it in writing, please provide it for the committee. We'd love to consider it. And I appreciate your patience as you've waited here all day to testify. So thank you all so much.

We'll now be moving to a Zoom panel. Up first is Ben Snyder. Ben, if you are on, unmute yourself and you may begin. You may begin, Ben. Ben, if you're on, please unmute. If not, we'll move on to the next person. Next person, Brian Degatti. You may begin. Thank you.

Uh, thank you, Speaker Menin and members of the council. My name is Brian Degatti. I'm the founder of Supernova Technologies in Madison, Wisconsin. I hold pending patent applications on hardware-based AI authorization and shutdown. I have a commercial interest, so please weigh my testimony accordingly. I support T2026-2602. Requiring both independent validation and a shutdown capability is the right structure.

I want to raise one question the bill leaves open: where does the shutdown capability live? Section 20-883 says nothing about that. A software Setting inside the same environment the model runs in would satisfy the definition, but a system that can reason about its instructions can potentially reason its way around a control it can reach. A compromised system can simply switch that control off.

Aircraft, payment networks, and the power grid keep the safety interlock on separate equipment the protected system cannot touch. I propose three short amendments. About 100 words in total. First, in Section 20-883, define the shutdown capability as a mechanism the model cannot access, modify, delay, or disable. Second, in Section 20-885, have validators confirm that by trying to defeat it.

A validator who reads a specification confirms a claim. A validator who attempts to defeat the mechanism confirms that it works. Third, in Section 20-886, have the Office of Cyber Command set independent standards scaled to risk and provide for tamper-evident records at fixed intervals. With those records, a gap becomes evidence. Without them, silence looks like safety. A switch the system can reach is a request.

A switch it cannot reach is a control. The bill should require the second kind. Thank you. My written testimony contains the exact language, and I welcome any questions. Your time is up, but thank you for your testimony. Please, since you have detailed amendment suggestions, please send it in. You can send it to testimony@council.nyc.gov, and we'll we'll review it.

Thank you so much for your testimony.

Up next, we have Ashna Agarwal. If you're You are on. Please unmute and you may begin. Good evening, Speaker Menin and members of the council. My name is Ashna Agarwal and I'm a program associate at the AI Now Institute. The AI industry has never been louder about abstract and existential safety, but for New Yorkers, safety is about how AI is embedded in our hospitals, utilities, city services, and workplaces.

Facing estimated trillions in projected revenue shortfalls, AI companies are racing to lock in public customers like New York. My testimony addresses how these financial interests conflict with the city's safety and how the council can protect New Yorkers. The city must maximize friction if and when it integrates AI into our systems. No AI system should enter city operations without an affirmative decision by the city.

That includes free and discounted offers and AI upgrades that are slipped in through software updates and contract renewals. Second, the city must address the harms New Yorkers already face. The council should ban the use of facial recognition technology in public accommodations and in policing, requiring notice when AI shapes decisions about New Yorkers, and ban surveillance pricing and wage setting.

Finally, we cannot lose sight that the concentration of power in AI and the single point of failure it engenders is a safety risk for New York City. City contracts should require vendors to meet rigorous security standards disclose breaches and outages promptly, and assume financial liability when their systems fail. New York City still holds the leverage to decide which systems enter its agencies and who bears the cost of failure.

I urge the council to use it now. Thank you. Thank you so much for your testimony.

Up next, we have Toni Tice-Therette. If you're online, please unmute yourself and you may begin. Hi, uh, thank you so much. Uh, I have written remarks, uh, but I'm going to read them out loud and I can send them later. Uh, but thank you, Speaker Menin and members of the council. My name is Toni Tai-Surette. I'm a filmmaker, educator, and founder of CinePartner, an AI-powered platform for storytellers.

Now, I'm probably coming to this conversation from a slightly different perspective because I build with AI, I teach people about AI, and I work with creative communities that are simultaneously excited about this technology And very concerned about what it can mean for their livelihoods, their work, and their agency. So we believe that we need guardrails, which is something— a word we've been hearing all day, right?

But I also believe that the goal should be responsible innovation, not regulation that unintentionally makes it possible for only the largest companies to build in AI. When we create compliance requirements, I would encourage the council to not only ask, Will this make AI safer, but also who will be able to afford to comply? Because if responsible AI development requires an expensive validation regimen— I mean, sorry, regime— that only trillion-dollar technology companies can navigate, we may inadvertently consolidate AI development in the very companies we're trying to hold accountable.

And for creative communities, safety isn't only just about whether a model can be shut down. It's also about consent, authorship, provenance, compensation, and transparency. So if someone's screenplay, for instance, or voice, likeness, or creative work contributes to an AI system or its output, what rights does that person retain? Do they know how their work is being used?

Can they opt out? Can they identify when AI materially contributed to something? And there's one more thing I wanted to elaborate. Is that I strongly support tracking displacement, but I would encourage New York to stop at counting the jobs AI eliminates. We should also be measuring whether New Yorkers are being prepared for the the jobs that AI creates.

AI literacy. Your time is expired. Oh, okay. Thank you. Thank you for your testimony, and please be sure to send us the written copy so we can review it. Thank you so much. Where do I send it again? I didn't hear that testimony at. council.nyc.gov. Okay, thank you. Thank you.

Up next, Travis Hall. If you're online, please unmute yourself and begin. You may begin. All right, thank you so much. Can you hear me? Yes. Uh, my name is— my name is Dr. Travis Hall. I represent the Center for Democracy and Technology, a 30-year-old organization dedicated to digital rights and civil liberties online. And today, I would love to talk to you a little bit about the range of issues that you're covering.

I really appreciate the hearing, but want specifically to talk about third-party auditing. We appreciate the effort to put in place third-party auditing requirements. They not only will help to deal with catastrophic risks, but the range of risks that people face from artificial intelligence. However, as we've seen with LL144, if there are not additional mechanisms in place to incentivize the use and consequences for these audits, they end up turning into being somewhat useless.

So we want to see meaningful independence. Independent interdisciplinary skills. There are a lot of regulatory agencies already covering many of these risks, and they can be drawn on, not just technical skills. Don't just simply audit the models. Think about auditing and assessing systems and use cases. Make sure that you cover a broader scale of harms. We heard a lot today about catastrophic harms, but we're already seeing little catastrophes, little catastrophes everywhere.

When people lose their benefits or their jobs, those are catastrophes. Make sure to incentivize— incentives are commensurate to protections. And for example, making sure that assessments don't entrench powerful actors. Also, independent evaluation only really works as one piece in a broader accountability network. So thank you very much for your time. Would love to engage with the council on this and other issues.

And thank you so much.

Up next, we have Danielle Marie Chappius. If you're online, if you could just unmute and begin. Danielle Marie Chappius. All right. We'll go on to Michelle Barnaby. If you're online, please unmute and begin.

My name is Michelle Barnaby. I'm a New York City nurse, Columbia graduate, and a former NYSNA member. Out of concern for the future of patient care and public health, I now study AI governance and safety with Blue Dot Impact. Today, not one of the major labs— Anthropic, OpenAI, Google, or Meta— could quantify the risk of catastrophic AI failure.

In healthcare, informed consent would apply. We disclose the risk, and the patient decides. I can't imagine handing a patient a drug or a device and saying, this may carry a catastrophic risk to you or your family's life. and we just don't know, but you should take it. Right now, the public is the patient, and we are being denied the right to know the risks and the right to say no.

Speaker Menin rightly noted that we don't ask the airline industry to self-regulate. In hospitals, we report our errors, even the near misses. AI companies face no such mandate. When you ask if these Frontier Labs bear legal responsibility for the harm, their AI causes, only Google answered clearly. The other 3 dodged. As a nurse whose entire profession is built on accountability, I find that deeply unserious.

Today, I heard the lab say voluntary frameworks aren't enough. We can learn from what happened in Illinois, where third-party audits won't begin until 2028. We don't have time for slow implementation. I think New York can show that government can move with speed that this demands, diagnose the problem, and put safeguards in place quickly. Let's give these frontier labs the regulations they say they want at a speed that makes sense for New Yorkers.

Independent third-party audits before deployment, including algorithmic bias, mandatory reporting for incidents and near misses. Time's expired. Override switch, whistleblowers, and clear development developer liability for harm. Thank you so much. Thank you so much. Thank you so much for your testimony.

Up next, we have Robert Capel. Robert, if you're on, unmute and begin. Okay. I'm going to call Daniel Marie Chappius. Again, if you're online and you can hear us, unmute and begin. Okay. Moving on, we will call on Patrick Sullivan. Patrick, if you can hear us, unmute and begin. Good afternoon. We'll move on. Tony Jones, if you're on and can hear us, unmute and begin.

Hello. I'm here. Thank you.

Good afternoon, everybody, or good evening, everybody. Thank you very much. I'm Tony Jones. I'm a Brooklyn resident and co-founder of Creative AI Academy, a training company and instructor of AI design at the Pratt Institute. I'm hopeful and heartened by the panelists' testimonies today and the intention of these committees on this crucial moment. While I support the strong regulation of companies building frontier AI, I— and I see the proposed amendments aim to address this— I'm here about the other half of the problem, the person at the keyboard.

The harm most New Yorkers will meet first comes from someone using the tool already in market today. Anyone with an email address can clone a voice, fabricate a photo of a real person, or write a convincing scam in minutes. The tool asks nothing from them first. No training, no acknowledgment of risk, no promise of responsibility. We regulate carmakers, and we also require every driver to follow the rules of the road.

In my classroom, every student writes and signs a personal AI constitution, a short statement of how they will and will not use the tools that I am teaching them to use. I propose the city ask the same of its citizens in the form of simpler than a license. First, a short, free city-provided safety training taken once before a person can open an account. on an AI tool of a certain defined level of capability.

Second, a signed user constitution, a plain language commitment to use the AI honestly with no harm, language which users may incorporate into the account-level personalization of the instructions that— of the AI products they use as well as a second safety measure. The signature will not stop the determined criminal, and neither does a driver's license. It simply sets a baseline for the millions of ordinary users who want to do right and have never been told what that means.

As a first step, I ask the council to start where it hits its authority is clear, require this training and a signed constitution for city employees who use AI on the job or AI vendors that contract with the city, then maybe we can press Albany for a statewide standard and set the standard for the rest of the country as well.

Thank you very much for your consideration. I know that we can regulate, regulate the labs, but I ask that we ask something of the rest of us as well. Thank you for your time. Thank you so much.

Up next Aissata Camara, if you're on, please unmute and begin. Thank you so much, uh, Madam Speaker and members of the council. My name is Aissata Camara, and I am the founder and the president of the Global Subnational Diplomacy Council. I previously served as New York City's Commissioner for International Affairs and had the pleasure of serving the city for more than a decade.

I come to this testimony because artificial intelligence is something that all cities are thinking about, including New York. And we know that AI can obviously help cities reduce their paperwork, improve infrastructure and planning, and make services more accessible. However, we also know that AI has the potential to deepen inequalities and threaten livelihoods. So we are offering 4 recommendations and are willing to work with City Council in order to support the work that you're doing.

First, we would like to ensure that we're connecting workforce reporting to paid training by also consulting with workers and unions and providing opportunities for advancement accordingly. We are also asking that we do more to protect democracies As we know that artificial intelligence is a great tool for spreading misinformation and disinformation. We specifically are asking for more support to ensure that residents have access to reliable public information that is verified and that they understand, uh, the different issues and decisions that are being made by government in a clearer way.

When you look around the world, there are great cities that are doing this work, including Helsinki and others. And we ask the city council to connect with those cities to learn from their experiences as well. As I've mentioned, we believe that artificial intelligence could be a tool for good if it is managed carefully. We also share the concern that other speakers have.

Your time has expired. Oh, thank you. Thank you so much for your testimony.

Up next, I'll call Ashley Youssef. Thank you, Speaker Menin and members of the council, for the opportunity to speak. My name is Ashley Youssef. I am a native New Yorker, a mother of 4 living in Michigan, and a leader of Trina AI. I am developing Room 2050, an immersive arts and technology learning environment where students can explore creativity through hands-on experience with art, robotics, and artificial intelligence.

The vision is to put human imagination, curiosity, and judgment at the center of that experience. As a mother, I want children to have opportunities to understand the technologies shaping their futures. I want families to know who is responsible when those technologies enter a learning environment. My position is that access and accountability must move together. Children deserve opportunities to innovate And they deserve meaningful protection.

For Room 2050, the standards I am proposing include educator supervision, age-appropriate tools, clear communication with parents, and strict limits on the collection and use of student information. Any use of student data to train commercial AI systems should require explicit, informed parental permission. Students should learn that AI can make mistakes, reflect bias, and produce convincing information that is false.

They should practice questioning its answers and developing their own ideas, no matter what part of the city or the United States they are from, to help support emerging talent, which are human emerging talent. I support a gradual rollout beginning with supervised experiences and expanding only after evaluating safety, educational value, and feedback from teachers and families and the council.

I ask the council to pursue clear standards that schools can understand and enforce, including transparency from technology providers and practical ways to report problems and stop using unsafe systems. Educators and parents should have a meaningful voice in these decisions. That's what Tareen and I— Your time's expired. Thank you. Thank you so much for your testimony. Up next, we will call on Naveed Hassan.

Naveed, if you're online, please unmute and begin. Thank you, Speaker Menin, Technology Committee Chair De La Rosa, and City Council for holding this critical hearing. My name is Naveed Hassan, and I'm the parent of 2 public school children and also the elected Manhattan member on the Panel for Educational Policy. policy. My academic and professional background is in symbolic pre-generative AI and computer vision.

We are in an unfortunate situation in technology policy for generative AI with 2 prominent camps: the accelerationists' appeal to a desire for imminent utopia and the effective altruists spending their time in end-code panic. Neither extreme greed nor extreme fear are a beneficial framing for sober thinking around how to manage developing advanced technologies. New Yorkers need pro-social technology governance with tech expertise in public offices, not being rushed to build hundreds of hyperscale data centers, or putting only those very companies in the room when writing rules in a way that lets them achieve regulatory capture.

This morning, none of the GenAI companies could answer a simple question: What do you consider is the risk of your service causing catastrophe to humanity? This is because it is antithetical to their business desire to push an inherently probabilistic product to be used in all aspects of work, to make any such estimates. When a job demands deterministic certainty, this whole model will eventually fail for sure.

We must not allow the spread of an unfit-for-purpose technology to be misused so broadly. There's bipartisan interest in not just regulating GenAI, but also pulling into the public realm broadband and wireless connectivity, abusive algorithmic targeting, illegal surveillance, vendor lock-in, interoperability mandates, open standards, open-source software infrastructure, and more. We have the opportunity now to think about what municipal tech policy should look like for the good of everyone living in New York.

We have the ability already to enforce existing rules around antitrust, concentration of risk, controlled goods rules being applied to compute hardware. Let us calmly and expeditiously use skilled public servants to build a future with beneficial technology available for all of society. Thank you. Thank you so much for your testimony.

Up next we have Nadira Pittman. Nadira, if you are on, please unmute and begin. Hello? Yes, please begin your testimony. Hi. Hi. Yes. Okay, so my name is Nadira Pittman. I am 41 years old, a Queens resident. Um, I am actually reaching out, uh, regarding a testimony that I have. Um, I am actually a survivor of the AI attacks.

I'm dealing with facial recognition, illegal surveillance. I'm also dealing with, like, constructed voice recordings of close family friends. They're gathering it and creating a job, like, as far as a job, they're using their voices against me. It's on the— if I'm on the buses, on the trains, it's continuous. It's a continuous basis. Nonstop, I have been filing reports, police reports.

I've been trying to you know get this attack off of me. I'm not sure why it seems to be inputted and okay. It's also in various agencies where it's allowed to be used against other individuals, where people are afraid now of even showing their faces on the public transportation. Um, it's, it's really, really getting out of hand, and the attacks are, are extreme.

It's very dangerous at this point. Um, you know, I, I— it's, it's to the point where you can't go anywhere and feel safe now because of, of this technology. Um, so I'm reaching out. I have sent numerous, uh, reports. I have reached out and, uh, wrote letters to you guys, uh, to try to get this off of me because it, it's, it's something that it, it's, it's unnecessary.

It's not— I'm not a criminal. I don't do anything illegal. There's no reason why— who put this on me? Why is this— why am I being attacked in this manner? Um, you know, I've came up there personally and I informed you guys last year. Your time has expired. Thank you so much. Thank you so much. I appreciate your time.

I would prefer someone reach out to me as soon as possible. Thank you so much. Thank you.

Up next, we have Lori Lomask. If you're on, please unmute yourself and begin. Okay. We'll move on to Steve Seri. Siri? Not that Siri. Steve Cere, C-E-R-E? Just go through the list. They probably know why. Okay. Tanbir Chaudhry, if you're on. Katherine Jin. Zachary Thompson. Christopher Leon Johnson. Raj Shiroff. All right, those are all the names that we have for today.

I want to thank everyone who took time out of their day today to testify, to bring your suggestions, to bring your questions, and to bring your partnership with the council. I want to thank Speaker Menin for her leadership, and to my colleagues, the last ladies standing, Councilmembers Maloney and Hanif, thank you for staying till the very end.

And to all the staff who made this possible, thank you all for this important hearing, and we are adjourned. Thank you.