cross-posted from: https://infosec.pub/post/49526716

I fought against Google’s Pentagon Al deal from the inside. Powerful people and institutions failed to keep their Al ethics promises under pressure.

---

I learned that Google sells its Cloud services to the relevant agencies within dhs. I thought that was wrong. Federal agents should not be able to kill citizens in the street. I set out to find the most effective way to push my company to stop serving these agencies. My divestment campaign quickly broadened into an attempt to prevent Google from signing an unethical military AI deal, as the Pentagon started pressuring AI providers into military AI deals with no restrictions against use for killer robots or mass surveillance.1

I wanted AI ethics commitments to hold under pressure. In particular, I wanted Google DeepMind (gdm) to maintain its existing commitment against supporting killer robots. Over several months, I asked many people to act. I asked senior people⁠—respected people⁠—people with reputations silvered by their concern about AI ethics and safety. Nearly all declined.

…

Google supports the immigration enforcement supply chain

After Alex Pretti’s death, I was determined to take effective action. To determine how to reduce harm from dhs, I researched Big Tech’s entanglement. Certainly, Microsoft and Amazon have larger involvement, but I was surprised to learn of Google’s exposure.

…

But how could I do anything about it?

The stereotypical activist action is to make a petition. But Google had already ignored a large petition on this issue. Plus, Google’s executives likely hardened their company against stereotypical organizing tactics. Sit-ins, strikes, even a mass of Google engineers quitting: I deemed all of them ineffective (if I could even pull them off).

…

Talking to Jeff Dean

I told Jeff that I respected him for speaking out, that I wanted Google to divest from the dhs supply chain. I asked if he shared these goals and, if so, how I could help.

He suggested it’d be reasonable for me to email a few guys. Their names: Sundar Pichai (ceo of Google), Demis Hassabis (ceo of Google DeepMind), and Thomas Kurian (ceo of Google Cloud). I thought “sure, Jeff. No problem. I’ll just tell them what I think.”

…

They never replied. I returned to Jeff and asked for a lunch to discuss constructive opportunities for real change within Google. I told him: “any time, any place. I’ll drive down to Mountain View to meet with you.”

At this point, I thought this was where “plan A” would fail. To my surprise, he actually accepted, for a lunch a few weeks out.

A lot would happen in that time.

The Pentagon tries to intimidate Anthropic

The Pentagon wanted the frontier AI lab Anthropic to remove red lines from its existing contract: red lines against lethal autonomous weapons systems and AI spying / profiling. The ultimatum was essentially “give us your product or we will designate you a supply chain risk.” The government wanted the AI for “all lawful use.”

There were two major problems with that kind of deal.

1. Independent legal experts had pointed out potential war crimes committed by the Pentagon (like double-tap strikes on shipwrecked survivors), though the Pentagon insisted those actions were legal. Under that kind of “legality,” “all lawful use” potentially meant “AI enabling war crimes” and “automatically profiling dissidents with AI.”
2. The Pentagon threatened a private company with economic destruction. Usually, the government would say “no thanks, we will find another supplier who will provide terms we want.” In this case, the government threatened to falsely4 designate an American company as a “supply chain risk,” which would force all military contractors to stop using Anthropic.

I’d been following the Anthropic⁠–Pentagon standoff for weeks. That morning, I read about the ultimatum. I was attending a conference in Paris held by the International Association for Safe and Ethical AI (iaseai), which is a nonprofit founded in 2024 to be “a unified voice” for safe and ethical AI. The world-famous AI scientist Stuart Russell chairs its steering committee. Its 2026 working groups include “Red Lines for Advanced AI,” focused on “autonomous weapons and escalation.” iaseai seemed built for a moment like this.

Iaseai’s venue would be full of influential AI professionals who care about ethics. I thought: we can organize a response, as a field, in support of 1) Anthropic’s right to do business without threat of destruction and 2) actual standards for whether and how to integrate AI into lethal autonomous weapons systems and surveillance apparatuses.

Anthropic had two days to comply with the administration’s demands. Perhaps other companies would agree to these “all lawful use” terms before the deadline. The main question which weighed on me: can I stop Google from caving, from accepting an “all lawful use” deal? If Anthropic says “no” and Google also says “no,” now we’re getting somewhere. That seemed hard. I wanted to make it happen anyway. Google could cave at any moment, whether before the Friday deadline or after.

…

Stuart closes out iaseai

AI company / DoD deals mattered. They marked the first public, high-profile intersection between modern generative AI and military use restrictions. Whatever compromises (or capitulations) the companies made would reverberate as precedent into the future. Iaseai attendees had a chance to take their contributions from “abstract work which might inform policymakers” to “directly influencing precedent.” Stuart was now going to mobilize them.

As promised, he spoke about the issues at closing. I recorded his remarks during the Q&A, which were originally available on iaseai’s conference schedule but were later migrated and not reuploaded to iaseai’s official YouTube channel.

Left on read by iaseai

The Pentagon set the Friday deadline for Anthropic, not for iaseai. The Anthropic autonomous weapons issue had been in the news for weeks prior. Anthropic’s lawsuit against the Pentagon ran for weeks after. Google wouldn’t sign for two more months. Iaseai had plenty of time to act and plenty of ways to act, like filing an amicus brief, putting out a statement, or contacting senior decision-makers at Google.

OpenAI did announce a deal on Friday. OpenAI claimed its deal protected the same red lines against killer robots and mass surveillance that Anthropic had insisted upon. However, some analysts concluded that OpenAI’s contract language contains wide loopholes.

Google quietly signs the deal

That weekend prior, I had heard rumblings. Along with over 600 other employees, I signed a letter asking Sundar to say “no” to classified AI contracts.15 I asked Jeff if there was anything I could do on an informal basis, crossing reporting lines and bureaucracy to help him get anything done that he wanted. I’d work through the weekend, no problem, on whatever he thought was wise. Alas.

…

I went to the field’s premier safety & ethics organization (iaseai). I asked some of the most distinguished AI scientists (Bengio and Stuart). I built a coalition and a plan for Google’s most outspoken executive (Jeff). I even cold-messaged the ceo of my company (Demis), whose lieutenants never evaluated the proposal. Besides Jeff, none took any visible action to stop the deal. And the deal contains no binding provisions, which is what I’d expect if Jeff never threatened to walk.

The deal was inked.

…

Reflections

Google DeepMind was an experiment in governance. The cofounders sold to Google on a promise never to power weapons and fought for a semi-independent governance structure. Sundar refused it. Google’s 2018 AI Principles were imposed by employee pressure but later quietly defanged by leadership.

Here’s the result of gdm’s experiment: it failed.

When profit and pressure met ethical commitment at Google DeepMind, pressure won and pledges lost. When profit and pressure met ethical commitment at Anthropic, ethics won. So the lesson is not “no one ever takes a stand.” The lesson is that society cannot rely on ethics-motivated people standing firm.

I know the other options don’t look great. Congress remains a potted plant in the corner. But we should at least stop telling ourselves that a seat at the table works.

…

Breaking free of roles

I know what broke my bounds in this instance. I have a regular reminder in my phone which shows me a picture of Alex Pretti. In one January moment, my anger flared so hot that its only outlet was to find a plan which could actually work. A mere tweet would do nothing and would count for nothing. Only a good plan would satiate. The anger burned through my bounds and broke them.

When I got scared⁠—and I did⁠—I’d think about Minneapolis. I’d think about ice shooting people in the street and dragging people from their homes.

Why I left Google DeepMind

…

In February, I realized that Google would probably sign the deal, which made me think about the door. I realized maybe I should leave and maybe I could do better AI safety work elsewhere. But I think I would have stayed a few more months if they hadn’t signed the deal. When Google signed, I just couldn’t do any more work. My brain said “no.”