AI Researcher Answers: Is AI’s Existential Threat Real?

Timnit, thank you so much for being here

and joining me on The Big Interview this week.

Thank you for having me.

A few weeks ago, a young AI researcher at Anthropic

basically stepped down and in his resignation post on X

indicated that a lot of people within the company

feel that we are facing this major existential risk with AI.

And then a few people responded

and there was this whole pile on and discourse

about the real existential threats of AI.

And when you and I spoke in the immediate aftermath of that,

you said that you felt that particular narrative

was a distraction from the real problems

that we're facing with AI.

You know, I would even go further than that

and say that it's dangerous.

It's not just a distraction, but it's harmful discourse.

And so the reason being, before I answer your question,

you know, I try to do this,

I want to give a little brief history

of how this narrative has been going on for a long time

because I wrote a Wired op-ed in 2022.

We wrote a statement in 2023 when this happened.

Elon Musk and Peter Thiel have been saying this since 2013.

Prominent people have been saying

that AI is an existential risk to humanity.

And the way it goes is usually,

and this is exactly the same rhetoric every three years.

So I can regurgitate whatever I wrote 10 years ago

to respond to these things.

For example, the Future of Life Institute,

which now is advising- And what is that?

Yeah, the Future of Life Institute

is an institute that was founded

by Max Tegmark and Jaan Tallinn,

a billionaire who led Anthropic Series A funding.

And he was a Skype co-founder, correct?

A Skype co-founder.

Jaan Tallinn also funds METR, which is an auditor,

a so-called third party, whose report went viral

about these rogue agents hacking Hugging Face.

So to the public, it might seem

that there are so many different entities coming together

saying the same thing, but just like the tobacco industry

and fossil fuel industry playbook,

there's a playbook here where the same billionaires

who founded and funded Anthropic

stand to benefit so much more from its IPO

more than anybody else,

also founded and funded the institutions

warning about existential risk, quotation marks, of AI,

also founded and funded the third parties

that are being cited right now

as if they are, like, an independent entity.

So that I want to ground the public in that fact.

And I can say more about that.

Tell us why you thought the machine got narrative.

It's a distraction.

It's even more than a distraction, it's harmful.

For example, the reason I was telling you all these facts

is that these people, the funders, the founders,

the investors who stand to benefit and profit the most

from these companies IPOing are saying this,

that they have been the ones seeding this narrative

going back multiple decades actually.

People should ask why.

If I stand to get lots of money from a particular company,

why do I seem to be the person

saying this particular company might build something

that kills us all, right?

It sounds counterintuitive.

But the first thing is that if you convince the public

that you're building something so powerful

that it's beyond even your control,

it's beyond something that we've seen before,

it's beyond something

that we don't have existing regulation for.

And this is why I have a lot of respect for Lena Khan,

because one of the things she said

is that there's no exception to the current laws

that we have for AI companies,

and that they talk as if there is an exception.

And you're saying that whatever you're building

is so powerful that it's beyond anything we've seen before.

Right.

It's self-aggrandizing.

It's saying we're the future makers here,

but the future is going to be really scary.

They don't always say, the same people,

you have to pay attention to what they say

in the news cycles.

When they say that AI can cause an existential risk,

they also say that AI can stop climate change,

bring us world peace and eradicate poverty.

AI can do all of these things because it's powerful.

It can bring us utopia, but also if the wrong people do it

and we don't have guardrails or whatever,

it can also just kill us all.

So the, the same people believe these things.

There's only a few people

who might be in one of these camps.

When you're telling people

that you have the super powerful machines,

what you're speaking to, first of all, is to your investors.

You're saying that, you know, you are,

you want to get your hands on these super powerful machines.

You're talking to government saying,

you don't want your adversaries

to get their hands on these super powerful machines,

you want to get your hands on these super powerful machines.

And you're also telling any regulating bodies

that whatever regulation they're thinking about

should be about these fictional, super powerful things.

And imagine, if I'm talking about potentially eradicating

all of humanity, things like pollution, data centers,

they sound kind of small potatoes, right?

You're talking about copyright protection for artists,

actual lawsuits that are going on right now,

government should not waste its time

thinking about this stuff.

If you waste your time regulating us on these small topics,

you risk China getting its hand

on these super powerful machines.

And you don't want that because at least us, trust us,

we can build the machine that can help us.

And we're telling you we want to be regulated,

but you don't want China to be the one.

Right. That's the first one.

The first one. The second one.

Second issue is you can always abdicate responsibility.

In our prior conversations,

we've talked about the usage of certain language or phrases

to describe this modern era of AI,

and how you take issue with some of them.

So for example, in our previous conversation,

you said something about autonomous weapons.

After that was published,

you told me you later regretted saying that

because you would've phrased it differently.

We've talked about P[doom].

We've talked about how when you say

that agents have gone rogue such as OpenAI,

you're essentially removing culpability

from the engineers who built those tools.

I can't help but wonder if maybe there

are groups that are.

We've even talked about the word alignment,

so I was just going to say I wonder if there are groups

that are actually more aligned than they realize

around safety and transparency,

but who are deploying different phrases

to describe what is essentially the same thing.

Some of the reasons that I regretted saying these words

is not that the term autonomous weapons has,

I have any issue with that.

It's a legal term that has,

even landmines are considered autonomous weapons legally.

It's just that the people

in the so-called existential risk camp

have co-opted it to mean something different.

When I said the term,

I knew, I'm like, huh, I wonder if this is going to be,

taken in that way and it was

because the people talking about existential risks of AI

took it to mean what they're saying, right?

Which is like some kind of a superhuman,

super intelligent machine doing stuff

on its own or whatever.

Just to bring it back

to the latest conversation about this,

I was at the UN General Assembly this week.

I was a part of some side panels,

so I wasn't actually in the UNGA nor the Security Council,

but there were some prominent tech folks there,

including Sam Altman from OpenAI

and Dario Amodei from Anthropic.

Dario reportedly said, If managed poorly,

I even believe AI could be a risk to humanity as a whole.

As competitors, Sam Altman said,

We could lose control of the future to AI.

So explain how it is that these folks

who once again have these,

you know, companies that are valued

at nearly a trillion dollars or more

are now here on the world stage at the UN

saying, hmm, I don't know.

There could be some risks here, right?

You know, is this part of regulatory capture?

Are they going to be looking for ways to self-govern

before there's a regulatory crackdown on this?

Absolutely.

It's part of regulatory capture.

And in the same paper where I introduced a test reel bundle,

I talk exactly about this regulatory capture

and especially what has happened at the UN.

Briefly after, you know, Sam Bankman-Fried,

as you know, was an effective altruist

and briefly after he went to prison,

the effective altruists were briefly scrutinized

by journalists and journalists were talking about,

was a political article, for example,

about the regulatory capture.

Let's talk about right now, what can be done?

What kind of governance do you think is needed for AI?

So I mean, we've been talking about,

regulatory capture has been happening for a long time.

2023, same thing.

Sam Altman said the same thing.

We need the world cooperation, et cetera, et cetera.

The EU AI Act regulated,

and then he threatened to pull out of the EU.

So you just have to see what they've actually done.

When there's actual regulation that holds them liable,

they threaten to pull out or they lobby super hard

to water down that regulation.

On the other hand, they're going around

telling these multilateral bodies

that there has to be world cooperation,

et cetera, et cetera.

And then again, what is this doing?

It is A, selling themselves as organizations

that are creating super-intelligence, right?

And so that's already marketing.

And B, they're making everybody scared

of anybody else who might be creating such things.

So they don't want open-weight models from China,

they don't want this competition.

I am not following Chinese regulation that closely,

but they're not talking about existential risk.

They're talking about deep fakes

and they're talking about, like, real things

that need to be regulated. Mm-hmm.

So the first one is marketing yourself

as creating some super powerful, unprecedented things

for which we don't have existing regulation,

which is not true, we have existing regulation.

But the second one, is what they're doing,

is when someone tries to enforce the existing regulation,

they either threaten to pull out or they lobby hard

so that they don't have this existing regulation.

And so the reason I keep on going back to 10 years ago,

five years ago, three years ago is that the same,

in my book, I say the same movie on repeat

with different heroes.

So what is the solution for governance right now

if you had to propose it?

Very, very simple things.

First of all, Lena Khan outlined five things.

One is deceptive marketing practices.

There's a law for deceptive marketing practices

and you can go after companies for that.

Two, is transparency, documentation.

Before you put something out there,

you should be able to tell us where all the data came from

and actually document.

This simple thing they don't do and they will never do.

I'm going to tell you that they will fight tooth and nail

to do the simple thing of documenting data.

And labor exploitation of data workers,

that's another one that they don't want to talk about.

We are here in the world, in the clouds,

talking about super-intelligence where you have hundreds

of millions of people around the world,

painstakingly labeling data, even pretending to be chatbots.

There are data workers called AI impersonators, right?

We now know that OpenAI hires people

to look at your conversations with their chatbots.

So people should be careful.

Don't believe that, you know, you have privacy.

Right.

404 Media just recently reported

that the new Meta Muse chatbot

is actually a person on the other end who's responding.

Yeah. Things like this.

Very simple.

Data transparency, labor exploitation,

you should not be able to steal data from people.

Even the first three things I talked about,

data transparent documentation, labor exploitation,

if they had to abide by laws like that

and they were not allowed to steal data

and not document it, the market calculation,

right now, the market calculation is not working,

but with these additional measures,

it just would not work whatsoever.

Right, so you would automatically slow them down

and have to make them accountable for something.

Back in 2021, you co-authored a paper

that was titled On the Dangers of Stochastic Parrots.

Google had approved it initially,

then it was being reviewed.

There were parts of it that were in dispute.

You said, Look, if you want me to remove my name from this,

I'm not going to be a part of this.

You ended up decamping from Google.

That paper was later presented at the 2021 ACM Conference

on Fairness, Accountability and Transparency,

and people still reference it.

Can you briefly explain

for someone who's never heard of stochastic parrots before

what it has to do with the AI we're using today,

what it means?

Stochastic parrots is a metaphor to help people understand

what large language models do.

And large language models are trained on vast amounts

of textual data on the internet,

trained to output the most likely sequences of text

given their training data.

They power most of the chatbots that we see today,

whether it's cloud, whether it is ChatGPT.

But when I wrote this paper ChatGPT hadn't come out yet,

but we saw the race to build larger

and larger language models.

And so that's the danger of building larger

and larger language models

that we were describing in this paper.

That they would essentially parrot people?

To parrot is to repeat back without understanding, right?

So there was this whole,

this whole existential risk narrative

was happening back then too, if you can believe it.

And so there was all this conversation

about how OpenAI had claimed that GPT-2,

the precursor to GPT-3 that powers ChatGPT

was too dangerous and too powerful to release.

There was this whole conversation

about whether GPTs can be ethical

or are they creative and all this stuff.

And so we really wanted to ground the conversation

in the real issues.

One of them, one of these issues

is the environmental catastrophe,

which a lot of people are now seeing,

but we discussed it back then.

And that was one of the main sections

that Google people were unhappy with,

the environmental costs.

The other one is not documenting your data

because you say you have too much data to document.

The other one is deceiving people into believing

that there is a mind behind the textual outputs

that they're interacting with.

And so there, that's where we really wanted to explain

that these systems are parroting

the patterns of their training data.

And it's very dangerous

when you're outputting text like that

because when you have a plausible sounding text

or very fluent text,

there's so many different kinds of issues that can occur

besides you believing that there's a mind behind a machine.

I gave, in that paper, we gave an example

of this Palestinian man writing good morning,

which was translated to attack them.

And because of that grammatical correctness

and there were no cues that the translation could be wrong,

and people believed the translation.

So the other issue of believing

there is a mind behind the machine

is what we call automation bias.

You over-trust automated systems, and if you believe

that this thing is an all-knowing machine,

then you're going to over trust

the errors that you get, right?

[Lauren] That's so interesting.

So we're seeing this with medical scribes

where I was just reading an article,

another article where medical scribes that were summarized

said this woman was micro-dosing on mushrooms,

since this poor woman has never heard that,

never discussed mushrooms, never done mushrooms.

She saw it on the notes.

So none of the doctors, nobody checked because again,

if you believe, this is why I believe

the super-intelligence existential risk discussion

is not just distracting, but dangerous because we,

with automation bias,

with over trusting these machines already,

if you believe that they are nearly super-intelligent

instead of error prone,

large language models parroting things,

then you're not even,

you're going to be less likely to check,

you're going to be less likely

to put checks and balances and regulation.

And you're seeing things like misdiagnosis

Medical errors. based on medical errors

and things like that, which are serious.

It's funny, I must be too much of an elder millennial

because you say that there's too much automation trust

and I'm like, I'm so distrustful.

I've got, call the bank and it's a robot,

I'm like, nope, nope,

I have to. I don't want to talk to it.

Totally. Yeah, exactly.

So one of Anthropic's co-founders,

Jack Clark, recently posted something on X.

You know what I'm going to say.

Yeah, I do.

He basically, he puts stochastic parrot in quotes

and said he,

It was a mimetically fit cognitive virus

that spread from 2021, when your paper was out, to 2025.

It temporarily blinded many gifted people

to the nature of AI progress,

burned up crucial years of research, he says.

He later says, The use of this frame causes people

to materially underestimate

what AI systems can and can't do.

When you saw Jack's post on X,

what was your initial response to that?

I was not surprised, by the way, let me just tell you,

because the effective altruists have been.

I was telling people that this is a talking point

that they're telling lawmakers now,

because every time I said something, they were like,

Oh, you should not take seriously someone

who still takes the stochastic parrots things seriously

in 2026.

So this has been a talking point of theirs for a while.

Right.

So the idea is that the research is outdated, right?

Yeah, and it's not.

And that's what I want to say.

It's so ludicrous that we're even saying this

because these are definitions

of what large language models are.

What large language models are has not changed,

will never change.

Like, large language models are large language models.

Now, you might have large language models

in a separate system that's trained in a,

you know, they have now reinforcement learning agents,

but our paper was about large language models

and that's never changed.

And these chatbots still have large language models

as a basis.

And we are seeing a.

It's so ridiculous that Jack Clark

is talking about overestimating systems

because what I'm seeing is the examples

that I just told you.

It's over-trusting these systems,

not having checks and balances

and ending up misdiagnosing someone's breast cancer

to the wrong side or - So there's a direct line

between LLMs being stochastic parrots and misdiagnosing,

giving a medical misdiagnosis using an AI tool because why?

How does that actually.

How does one lead to the other?

Because large language models are stochastic,

the stochastic parrots, they don't understand

what's inside the text.

So you cannot expect them to be factual.

Even if you see something like the AI overview,

I had another example,

where I was calling an oncologist friend of mine

to ask about a specific medication

and whether it was appropriate for a specific use,

because I read the academic paper saying that it was not,

and I wanted confirmation.

And my oncologist friend was like,

Oh, look at the Google overview.

It says that it's appropriate.

But I read that, but it turned out not be.

Why?

I encounter that all the time with Google AI overviews.

Because they are stochastic parrots

trained to give you the most likely sequences of text

based on their training data.

So this is not grounded in a world models.

Emily M. Bender has been trying to say this

in so many languages for a longtime.

Also one of the co-authors on the paper, correct?

Co-first author with me

to say that the research is outdated is ludicrous

because right now, as we are hyping up

these super-intelligence and all that,

there are news stories that are going unnoticed,

which is about the complete opposite scenario

that is actually happening in the real world.

My understanding is that some of the biggest critiques

people have had about focusing so much

on research from that era may not be incorporating

the thinking or the reasoning or even-

There's no reasoning.

Or even the recursive intelligence that-

There is no recursive intelligence.

Is there none? There is no thinking.

How is there none?

This is something, by the way,

that at the Berkeley AI Conference earlier this summer,

I heard someone from Google

talking about recursive intelligence.

It's something that- Of course they are.

The, you know, The New York Times

just did a big story about how the,

you know, the scientists and researchers right now,

that's what they're looking towards, recursive intelligence,

the AI learning from other AI.

Well- Is that a reality?

It sounds like you're saying that's not a reality.

No, because, you know,

there's one problem with AI researchers,

which is aspirational naming and aspirational stuff.

So machine learning, that's aspirational naming.

But machine learning is real.

The naming is aspirational. Okay.

The machine not necessarily.

So this happens all the time, right?

So just because there's curriculum learning in AI,

it's a field.

But let me give you a paper from my former manager,

Samy Bengio, and I ask him,

aren't you tired of your whole life being,

like, debunking the whole reasoning thing

every single paper you write?

He quit after I got fired from Google

and he's now head of machine learning research at Apple.

And if you look at almost every single paper that they have,

it's showing how if you change

the benchmarks on reasoning slightly

, the whole thing breaks down.

It's not reasoning. It's not reasoning.

0 Just because you're looking

at chain of aspirational naming.

Let me give you another example.

Just because your models are largely,

the stochastic patterns that you trained

to print out certain tokens,

you call them chain of thought reasoning.

You didn't know that they were thinking.

You don't know it's a chain.

You just know that these are tokens

that are being printed out,

but you called them chain of though reasoning.

Now you're saying that they're reasoning already.

One of the biggest crises that we have right now

is actually sound scientific research.

So if you look at my work,

if I ever have access to the data,

the code, the training data, and the evaluation data,

which none of these companies give you those things.

You don't even know whether they ingested that benchmark

during training or not.

If you ingest a benchmark during training,

it's like studying to the test.

It's like me coming to an exam,

knowing what the answers to those 10 questions are already,

studying that and writing it down, right?

So every time I have had access to these things,

I have shown how their claims are not correct.

But then with Samy Bengio and his team's paper,

what they showed is, and this is from 2025,

what they've shown is that you just change

the benchmarks a little bit, tweak it a little bit,

and it all breaks down,

showing that you were just sort of relying on the patterns.

Now, some people, when I give them this example,

they're like, well, we're talking about 2026 models.

Guess what?

Real evaluation in science takes time.

We should not be going from press releases

to lawmakers parroting those press releases

and journalists repeating those claims.

If you want to do real research and evaluation,

ask for the training data,

the evaluation data and the methodology

so we can all reproduce it.

Mm-hmm.

And so you are ascribing the thinking,

reasoning, recursive intelligence.

You, it sounds like you still see that

as something that is purely human.

It's ascribed to us humans.

It's the way our neural processes work,

but the AI doesn't work that way yet.

That's what you believe.

No, and I don't know if it'll ever.

Like, intelligence, that's aspirational naming.

And sometimes I play,

just to take people back to these hype cycles.

I tell them something, a claim that was made,

and I say, Is this 1954 or 2024?

What would you say is the biggest part

of your own thinking, your own research

that has evolved since the early days, 2020s,

of AI till now that has surprised you the most?

I have to be actively making space for the kinds of models

that I think should be built and building them,

and sometimes ignore the noise.

That's the conclusion I'm getting to over time.

That's your biggest learning, your biggest takeaway.

Yeah.

There is so much noise online right now,

it's hard to get any deep work done

if you're paying attention to whatever the AI guys say

at this point. And even if your research

is all about debunking what they're saying,

you know, it gets tiresome,

it's not fun to do that. Right.

You know, it's more fun to think

about the future you want to have,

the technological you want to have, and work on that.

Before I let you go,

what would you say gives you the most hope right now

for the future of AI?

I want to tell you that we have a list

called the AI Resist List

where we talk about people resisting

in the ways that they should resist, in many ways,

which is media capture, narrative,

data centers funding, et cetera.

And so we list the ways in which they're resisting.

And also people of creating alternative tech futures

that don't kill our environment,

that don't kill, exploit labor or steal data,

and instead are actually actively helping their communities.

And these ideologies are spreading, right?

You see one small organization somewhere doing something,

you get inspired by them, you do something different.

So I have hope that there are a lot of people

tired of what they're seeing

and they're actually in small circles

doing something different.

I believe in human agency and collective power

to imagine a better future

and stop bad things from happening

and bad things if they're bad, right?

So my belief in human agency, I think,

is what gives me hope.

Thank you so much, Timnit.

I really appreciate you joining me today

for The Big Interview and for sharing your insights

and congratulations on your upcoming book.

I look forward to reading it

when it comes out early next year,

and I'm sure we'll chat again soon.

Thank you, thank you so much.

[gentle music]