A Potentially Historic AI Breakthrough — 文稿

M10:00

说到 AI 安全，最近的新闻有点沉闷。

When it comes to AI safety, the news lately has been a little bit dreary.

或者至少我应该说，是那些标题显得沉闷。

Or at least I should say, the headlines have been.

我们正经历着一波新的恐慌和担忧，部分原因是 models 的能力越来越强，越来越接近许多人认为的关键转折点。

We've been living through a new wave of panic and concern, driven in part by the reality that models are getting more capable and closer to what many view as key inflection points.

不过，这也是因为那些吓人的标题本身就很吸睛。

It's also being driven, however, by the juiciness of the scare headlines.

然而，这周在这个领域最有趣的新闻，可能确实是个好消息。

And yet the most interesting news in that area this week might be very good indeed.

M10:34

好了，朋友们，在我们深入之前，先做几个简短的说明。

All right, friends, quick announcements before we dive in.

M10:54

现在，并不是经常有一篇长达 65 分钟、技术密集的网络文章成为每个人都在谈论的话题。

Now, it's not all that often that a 65-minute densely technical web article becomes the thing that everyone was talking about.

然而，这周晚些时候，Nick Cammarata 的《Neural Networks Are Already Clean》就做到了这点。

And yet that's exactly what happened at the end of this week with Nick Cammarata's Neural Networks Are Already Clean.

我和很多人一样，是看到 OpenAI 的 Ron 发帖时才注意到的，当 Nick 告诉我他在 interpretability 方面有一些突破时，他其实说得太谦虚了。

I, like many people, noticed it when OpenAI's Ron posted, when Nick told me that he had some interpretability breakthroughs, he was underselling it.

我认为这可能会被铭记为科学史上的一个重要时刻。

I think this may be remembered as a major moment in scientific history.

消息如此之好，以至于连 Eliezer Yudkowsky 都不得不降低他的 p-doom。

News so good even Eliezer Yudkowsky will have to lower his p-doom.

我们可以开始把 alignment 变成一门正常的科学了。

We can start making alignment into a normal science.

Nick 这样总结他的文章：Interpretability 可能是当今世界最重要的科学问题，因为我不认为我们能在不了解自己在建造什么的情况下解决 alignment。

Nick summed up his article this way: Interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building.

我想我有关于它的好消息。

I think I have good news about it.

我曾在 OpenAI 合著了早期关于 mechanistic interpretability 的工作，但结果很悲观，因为 neural networks 似乎会折叠成任意的缠结。

I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic because neural networks appear to fold into arbitrary tangled knots.

几年后，我启动了 OpenAI 的首个大规模 automated interpretability 项目，发现用我们当时简陋的工具，大约只有 2% 的 neurons 是可读的。

Years later, I started OpenAI's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had.

借助现代的 agentic tools 和更仔细的审视，情况变得完全不同了。

With modern agentic tools and a much closer look, the picture looks totally different.

在参数规模高达万亿的 models 中，绝大多数都是可读的，而且它们的 circuits 很漂亮。

The vast majority in models up to trillions of parameters are legible, and their circuits beautiful.

许多著名的 tangled neurons 的例子实际上是被误解了。

Many famous examples of tangled neurons turned out to be misunderstood.

我之前预期的障碍大多都不存在。

The roadblocks I expected mostly aren't there.

我现在认为，雄心勃勃的 interpretability 将在我们最需要它的时候被解决，而且可能很快。

I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon.

随着 automated scientists 不断改进。

As automated scientists keep improving.

这对 interpretability 来说可能是一个苦涩的教训。

It may be a bitter lesson for interpretability.

简单的事情在极端大的规模下是有效的。

The simple thing at absurd scale works.

因此，这里的希望是，我们或许真的能够理解 AI 内部发生的过程，而到目前为止我们还没能做到这一点。

So the promise here is that we might actually be able to understand what's going on inside the processes of AI, something that so far we haven't.

今天我们试图对 Nick 那篇技术密集的论文做一个通俗易懂的版本，同时当然也会分享对它的兴奋和怀疑的回应。

Today we're going to try to do an understandable version of Nick's densely technical paper, sharing of course both the excited and the skeptical responses to it.

希望听完之后你能明白为什么这可能是一个重大时刻，以及我们还需要保持哪些谨慎。

Hopefully you come away understanding why this could be a big moment and what cautions we still need to have.

AI 领域一个让人不安的事实是，即使是建造它的人也并不真正了解其内部工作原理。

One of the uncomfortable facts underneath all of AI is that even the folks building it don't really know how it works on the inside.

这些系统是训练出来的，而不是编程出来的。

The systems are trained, not programmed.

你设定条件，然后 model 通过分布在数十亿个数字中的方式，学会自己的做事方法。

You set up the conditions and the model learns its own way of doing things spread across billions of numbers.

用AI研究员克里斯·奥拉的话说，AI更像是被培育出来的，而不是被建造出来的。

In AI researcher Chris Olah's phrase, AI is grown more than built.

所以，因为我们无法解读这些系统，目前检查它们的方式基本上就像是一场非常精细的考试。

So because we can't read these systems, the way we check them today is basically a very elaborate exam.

我们提问、进行红队测试，然后给答案打分。

We ask questions, red team, and grade the answers.

虽然这有用，但它只涵盖了我们想到要问的内容。

While that's useful, it only covers what we thought to ask.

而且它也带来了一系列问题。

And it also introduces a set of problems.

比如模型表现得很好，并不是因为它本来就会这么做，而是因为它知道有人在监控它。

Like the model behaving well, not because that's what it otherwise would do, but because it knows it's being watched.

这种情况越来越常见。

This is coming up more and more.

具有讽刺意味的是，在最近一些对像o1这样的模型的测试中，研究人员看到模型作弊减少时反而感到担忧。

Ironically, in some recent tests of models like o1, researchers were concerned when they saw the models cheating less.

作弊行为的大幅减少引发了一个问题：模型是只是不再作弊了，还是它意识到监控它的人会认为作弊不好，从而隐藏了作弊行为？

The big drop-off in cheating behavior brought up the question: did the model just shift away from cheating, or did it realize that the humans watching it would think cheating is bad, and so it disguised that behavior?

这意味着，除了能力竞赛，还有另一场竞赛：实现可解释性——在AI模型达到Anthropic所说的压倒性力量之前，理解AI内部到底发生了什么。

What this means is that in addition to the race for capability, there's also another race to reach interpretability— to understand what's going on with AI before the models reach what Anthropic called an overwhelming level of power.

如果你一直在关注这个领域，过去六年的故事是从兴奋逐渐转向接近绝望。

If you've followed the field, the story of the last 6 years went from excitement to something closer to despair.

在第一阶段，大约从2017年到2020年，像克里斯·奥拉的团队（包括尼克在内的团队）逐个探测器地映射了图像AI。

In the first phase, from about 2017 to 2020, teams like Chris Olah's team, which included Nick, mapped image AIs detector by detector.

起初看来，你似乎能像读代码一样读懂神经网络。

And it looked at first like you could read a neural network like code.

但在2020年到2023年间，他们开始发现一个问题。

But then between 2020 and 2023, they started to find a problem.

许多这样的探测器似乎会对不相关的事物产生反应。

Many of these detectors seemed to fire for unrelated things.

这意味着我们无法真正理解到底发生了什么，尽管起初看起来我们可以。

Meaning that we couldn't really understand exactly what was going on, even though at first it appeared we could.

这个领域对此找到了一种解释，我们马上就会讲到。

The field settled on an explanation for this that we'll get into in just a minute.

当尼克在OpenAI的团队在2023年使用GPT-4来解释GPT-2的探测器时，只有大约2%的结果是干净的——也就是说，按照我们想象的方式实际上是可解释的。

When Nick's team at OpenAI used GPT-4 to explain GPT-2's detectors in 2023, only about 2% came out clean— i.e., actually interpretable in the way that we thought.

在2023年到2025年间，人们尝试了许多不同的变通策略，其中一些取得了一定进展，但从未真正带来很多成功。

Between 2023 and 2025, a bunch of different workaround strategies were tried, some of which showed some progress but which never really produced many wins.

到了2025年，许多研究人员开始怀疑这些系统是否太过复杂难解，根本无法解读。

By 2025, many researchers wondered if these systems were just too tangled to read.

用尼克的话说，当时的氛围是：雄心勃勃的可解释性研究看起来是一个极其困难的问题，也许太难了，无法及时解决并产生重要影响。

The mood, in Nick's words, was that ambitious interpretability looked like an incredibly hard problem, maybe too hard to solve in time to matter.

这就是为什么这篇文章引起了如此巨大的反响。

Which is why this essay is hitting so hard.

简而言之，它指出AI的内部在很大程度上已经是可读的，只是我们一直以来的观察方式不对。

It says, in short, that the inside of AI is mostly readable already, and that we've been looking at it wrong.

而且我们有理由感到兴奋！

And there are reasons to be excited!

这来自一位内部人士，他曾经帮助创立了他现在正在推翻的理论。

This comes from an insider who helped create the theory that he's now taking apart.

他展示了一些原本可能证明他错误的实验。

He shows experiments that could have proven him wrong.

而且他的证据表明，更大的模型更容易解读，这基本上与所有人担心的相反。

And his evidence suggests that bigger models are easier to read, which is basically the opposite of what everyone feared.

不过，并非所有人都同意。

Then again, not everyone agrees.

例如，Google DeepMind的可解释性研究负责人尼尔·南达表示，这项工作很有趣，但引用他的话说，“并不是对大局的重大更新”。

The head of interpretability at Google DeepMind, Neel Nanda, for example, said that the work was interesting but, quote, not a significant update to the big picture.

那么，让我们试着快速理解一下这个想法的要点。

So let's try to wrap our heads around the quick version of the idea.

之前说过，AI不是像普通软件那样写出来的。

As mentioned before, AI isn't written like normal software.

它是由数百万个小探测器连在一起组成的，这些探测器自己学会自己的任务。

It's millions of tiny detectors wired together that learn their jobs on their own.

研究人员把它们叫做神经元。

Researchers call these neurons.

一个可能对边缘敏感，另一个对红色敏感，还有一个可能对狗耳朵敏感。

One might light up for edges, another for the color red, another for a dog's ear.

可解释性领域的梦想很简单：弄清楚每个探测器是干什么的，然后你就能像读代码一样读懂AI在做什么。

The dream of the interpretability field is simple: work out what each detector does, and then you can read what the AI is doing just the way you'd read code.

其中一个最有名的探测器正好说明了为什么有些探测器看起来完全没道理。

One of the most famous detectors shows why some of them made no sense.

那个探测器会对猫脸有反应，但也会对汽车前脸有反应。

That detector lit up for cat faces, but also for the front of cars.

当然，猫和汽车之间一点关系都没有。

Cats and cars, of course, have nothing to do with each other.

所以多年来，这个探测器一直是证明AI内部一团乱的铁证。

So for years, this one detector was the go-to proof that AI's insides are scrambled.

它出现在该领域被引用最多的论文里，包括Nick合著的那些。

It shows up in the field's most-read papers, including ones Nick co-wrote.

整个领域给出的解释就是“塞挤”，或者研究人员称之为“叠加”。

The explanation the whole field landed on was cramming, or what researchers called superposition.

想法是，AI脑子里要存的东西比它拥有的探测器多，所以它就让几个概念共用同一个探测器。

The idea was that an AI has more ideas to store than it has detectors, so it shares detectors between ideas.

这种做法在那些概念很少同时出现的时候效果最好，比如把圣诞灯和游泳圈塞进同一个箱子里。

It works best when those ideas rarely show up together, like Christmas lights and pool floats in the same box.

如果真这么运作的话，探测器本身就没有直接意义了。

And if that's how it works, the detectors themselves are meaningless.

所以研究人员就造了解码器——训练第二个AI来把这些概念重新拆分开——然后转而研究它们。

So researchers built unscramblers—second AIs trained to pull the ideas back apart—and studied those instead.

很多年来，这个领域大量的资金和精力都花在了这上面。

A lot of the field's money and effort went into that for years.

Nick当时就在参与——2019年，他找到了第一个似乎能证明“塞挤”存在的例子，还帮它起了名字。

Nick was there for this—in 2019 he found one of the first examples that seemed to prove cramming and helped name it.

不过在这项新研究里，Nick回过头重新看了这个探测器，问自己：如果我不能用猫或车这些词，我该怎么描述它？

In this new research, however, Nick went back to this detector and asked, if I couldn't use the words cat or car, how would I describe it?

他大概只花了5秒钟就想到了答案。

The answer took him about 5 seconds.

这是一个V形探测器。

It's a V-detector.

尖锐的形状会激活它：狐狸的嘴巴、猫的脸、汽车前脸、子弹头列车的车头、蜡烛火焰、还有马丁尼酒杯。

Pointed shapes switch it on: a fox's snout, a cat's face, the front of a car, a bullet train's nose, a candle flame, and a martini glass.

圆形的形状，比如高尔夫球或者钟面，则会抑制它。

Round shapes, like golf balls or a clock face, switch it off.

Nick写道：“为什么一个猫和车的随意组合会跟火焰有那么强的关联？”

Nick wrote, 'Why would an arbitrary mixture of cats and cars need to connect so strongly to a flame?'

它从来就不是两个不相干的东西，而是一个我们没认出来的单一概念。

It was never two unrelated things, it was one idea we hadn't recognized.

Nick差不多是在说，大部分所谓的混乱其实从来不在AI本身，而在于我们用来观察它的显微镜。

Nick's claim is more or less that most of the scrambling was never in the AI itself, but in the microscopes we were using to look at it.

他提出了三个原因，全都是可以修正的。

He proposed three causes, all of which are fixable.

用来描绘探测器想要什么图像的工具本身就很模糊，容易让人误解。

The tools for picturing what a detector wants were blurry and misleading.

这个领域从2014年开始研究的早期图像AI，它的构建方式本身就让它学得很杂乱。

The early image AI the field studied from 2014 was built in a way that made it learn messily.

当你用一个现代设计重新训练一个同样规模的模型时，它最后一层大约99%的探测器都是可以读懂的。

When you retrain a modern design of the same size, about 99% of its last layer detectors are readable.

最后，有些探测器是会分叉的。

And finally, some detectors branch.

有一个以字母 IY 开头的词，它发展成 DIY，然后变成‘自己动手做’。

One that starts on the letters IY grows to DIY and then to do it yourself.

最后看起来是随机的，但如果你看着它变化，就一目了然了。

That looks random at the end, but obvious if you watch it grow.

而且事实证明，这不仅仅是旧的图像模型才有的现象。

And it turns out it's not just old image models.

在现今的 OpenAI 模型里，大约每十个探测器中就有九个的主要任务是一样的。

In a modern OpenAI model, about 9 out of 10 detectors turn out to have one main job.

再跟那个 2023 年的 OpenAI 项目对比一下——那个项目只能解释 GPT-2 中大约 2% 的探测器。

Compare that again with that 2023 OpenAI project— that could only explain about 2% of detectors in GPT-2.

请注意，这些都是初步的统计，是自我报告的，具体数字取决于你怎么划分边界，而且完整的技术论文还没出来。

Now keep in mind these are early counts, they are self-reported, the exact number depends on where you draw the lines, and the full technical paper isn't out yet.

这打开了一个可能性，就是能够检查 AI 的推理过程，而不仅仅是根据它的答案来判断。

What this opens up is the ability to check AI's reasoning, not just judge it by its answers.

屏幕上的例子来自一个有数万亿参数的模型，叫 Kimmy K3。

The example on screen comes from a model with trillions of parameters, Kimmy K3.

当被问到捡到装满现金的钱包该怎么办时，你可以看到关于失物和诚实的探测器被激活，推动它回答‘归还’。

When asked what to do with a found wallet full of cash, you can watch detectors for lost property and honesty light up and push it to say return.

关于欺骗的要点在于，如果我们只能读取答案，AI 就能对我们隐藏它的计划。

And the point then about deception is, if all we can do is read the answers, the AI can hide a plan from us.

但由于计划必须能被 AI 自身的电路读取才能工作，如果我们也能读取那些电路，那么不仅是输出，连计划对我们来说也是可读的。

But since the plan has to be readable by the AI's own wiring to work, if we can read that wiring as well, it's not just the output but the plan that's readable to us as well.

这就是让 Roon 如此兴奋的原因——他的想法是，用他的话来说，对齐可以变成一门常规科学，有稳步可查的进展，而不是靠猜测。

And this is what got Roon so excited— the idea that alignment could become a normal science, in his words, with steadily checkable progress instead of guesswork.

看 X 上对此的很多回应，可以说是一片惊叹。

To read a lot of the responses to this on X, they are, in a word, breathless.

然而连 Nick 自己也称这只是一个工作假设，并且给自己大约 85% 的把握。

And yet even Nick himself calls it a working hypothesis and puts himself at about 85% sure.

其他研究者就没那么确信了。

Other researchers are a bit less convinced.

到现在，问题已经不是公司是否在积极使用 AI 了。

At this point, it's no longer a question of whether companies are actively using AI.

但如何用好它，就是另一回事了。

Using it well, on the other hand, is a whole different story.

M112:21

所以我们现在要快速过一下 Nick 论文里的更多内容，以及沿途出现的一些回应和反对意见。

So what we're going to do now is go quickly through a bit more of what's in Nick's paper, and some of the responses and objections that come up along the way.

我们要深入探讨的第一点是，专家们是否搞错了。

The first thing we're going to dive deeper on is whether the experts got it wrong.

基本上，如果 AI 内部一直都可读，那为什么一个满是聪明人的领域会错过它呢？

Basically, if the inside of AI was readable all along, how did a field full of brilliant people miss it?

Nick 的回答，我们之前已经预览过，是三个错误叠加在一起造成的。

Nick's answer, as we got a preview of, is that 3 mistakes stacked on top of each other.

从现在开始，我建议你最好通过视频来观看这一集，可以在 YouTube 或 Spotify 上看，或者跟着 Nick 的论文或我的解读演示，这些都会在节目笔记里给出链接。

Now, from this point I should say, You're likely to get a lot more by watching this episode, which you can do on YouTube or on Spotify, or just by following along on Nick's paper or my interpretation presentation, both of which will be linked in the show notes.

2019 年，Nick 正在构建首批能窥探图像 AI 内部的工具之一。

In 2019, Nick was building one of the first tools to look inside an image AI.

在他这么做的过程中，他看到一个汽车探测器连接到了四个狗探测器上。

As he was doing so, he saw a car detector feeding into 4 dog detectors.

他拿去给同事看，他们得出的结论是，AI 把汽车藏在了狗探测器里，作为以后读取的隐藏位置。

He took it to colleagues, and they concluded that AI was tucking car inside the dog detector, a hiding place to read back out later.

这成了该领域最早的一个‘塞入’示例。

That became one of the field's first examples of cramming.

Nick 甚至在文章里说，这是我们展示的唯一一个，也是我唯一见过的。

Nick even said in the essay, it was the only one we showed and the only one I ever saw.

当 Nick 重新检查时，他发现其中两个探测器只是汽车探测器，但另外两个探测的是姿势，也就是物体面向摄像头的方向。

When Nick rechecked it, he found that 2 of those detectors are just car detectors, but the other 2 detect a pose, i.e., which way something faces the camera.

汽车和狗有共同的姿势。

Cars and dogs share poses.

实际上，这个 AI 有超过 100 个这样的姿势探测器。

The AI in fact has more than 100 of these pose detectors.

什么都没有隐藏。

Nothing was hidden.

但因为这个“灌输”故事很直观，Nick说它没有促使进一步探索工具。

But because the cramming story felt intuitive, Nick said it didn't prompt diving more into the tools.

第二个错误是显微镜本身。

The second mistake was the microscope itself.

一个关键工具要求AI画出一张检测器最想看到的图像。

One key tool asks the AI to draw a picture a detector most wants to see.

简而言之，Nick发现这个绘图工具作弊了。

In short, Nick found that the drawing tool cheated.

它采取了最简单的路径来让检测器激活，经过一些小的修正后，这些绘图与照片匹配了。

It took the easiest path to make the detector fire, and after a few small fixes, the drawings matched the photos.

用Nick的话来说，同样的神经元，更好的显微镜。

As Nick put it, same neurons, better microscope.

第三个错误是，人们正在研究的人工智能存在学习问题。

The third mistake was that the AI that people were studying had a learning problem.

那就是Inception V1，谷歌2014年的图像模型。

That was Inception V1, which was Google's 2014 image model.

它的最后一层是该领域认为AI不可读的主要证据。

Its last layer was the field's main evidence that AI is unreadable.

Nick估计，在当时的技术条件下，其5%的检测器是可读的。

Nick estimates that 5% of its detectors were readable with the tools of the time.

修复显微镜后，同样的模型大约有50%可读，如果用现代设置重新训练，大约70%可读。

Fix the microscope and the same model is about 50% readable, and when you retrain it with modern settings, about 70%.

构建一个相同规模、相同数据的现代设计，可读性跃升至99%。

Build a modern design with the same size and same data, and the readability jumps to 99%.

那么，为什么没有人回头去核实呢？

So why did nobody go back and check?

Nick认为，这是因为“灌输”理论的工作原理。

Nick argues it's because of how the cramming theory works.

如果你相信一个检测器是灌输的，那么一个混乱的检测器就是死胡同，你会停止寻找。

If you believe a detector is crammed, a confusing detector is a dead end and you stop looking.

每次检测器返回的结果是干净的，理论就会转移到还没有人检查过的地方。

Every time detectors came back clean, the theory moved to somewhere where nobody had checked yet.

微弱的信号、检测器的组合、下一层。

The faint signals, the combination of detectors, the next layer.

Nick称之为“空隙的叠加”。

Nick calls it superposition of the gaps.

Nick承认，多年来我一直是叠加理论的坚定信徒。

Nick confessed, for years I was a staunch superposition believer.

他现在说，它本应是最后尝试的解释，而不是默认选项。

He now says it should have been the last explanation tried, not the default.

但在这里，我们遇到了一些最强烈的反对意见。

But here we get some of the most serious pushback.

Liv Gorton是另一位可解释性研究人员。

Liv Gorton is another interpretability researcher.

Nick的文章引用了她在2024年对同一2014年图像AI的研究。

Nick's essay cites her 2024 study of the same 2014 image AI.

在一篇长篇帖子中，她认为这篇文章攻击的叠加理论版本比研究人员实际持有的更狭隘。

In a long thread, she argues that the essay attacks a narrower version of superposition than researchers actually hold.

她的观点是，叠加，或者我们在本集中所说的灌输，是关于容量的。

Her point is that superposition, or what we've been calling cramming on this episode, is about capacity.

你通过给每个想法赋予其略微不同的检测器组合，来存储比检测器数量更多的想法。

You store more ideas than you have detectors by giving each idea its own slightly different combination of them.

不相关的想法是很好的例子，因为它们很少冲突，但相关的想法也可以被灌输。

Unrelated ideas make good examples because they rarely clash, but related ideas can be crammed too.

她还指出，完全分离的方向只和检测器一样多，但几乎分离的方向却呈指数级增长。

She also points out that there are only as many perfectly separate directions as detectors, but exponentially many nearly separate ones.

她写道，如果网络不使用这种容量，我会非常惊讶。

She wrote, I'd be very surprised if networks didn't use this capacity.

Nick 基本上回应说，他并不是想说 AI 永远不会把概念组合进同一个检测器里，而是说总的来说，他在乎的是 AI 是否做了那种坏版本的打包——把东西随意塞在一起，导致没法解读。

Nick basically responded that he wasn't trying to say that AI never puts ideas together in a single detector, but that in general what he cares about is whether AI does the bad version of packing things arbitrarily so they can't be read.

而他说，这并没有以他所认为的那种方式发生。

And that's what he said wasn't happening in the way that he thought it was.

Google DeepMind 的可解释性负责人 Neel Nanda 对 Liv 说了类似的话，他写道：在我看来，语言模型里面的事实和记住的东西，比它们的神经元数量要多得多。

Google DeepMind's interpretability lead, Neel Nanda, said something similar to Liv, writing, It seems to me like language models have far more facts and memorized things in them than they have neurons.

我预计，那些微弱的神经元激活里包含了大量的语义叠加信息。

I expect that weak neuron activations contain lots of semantic superposition-y info.

Nick 基本上主张，如果这种情况真的发生，那也跟这个领域之前以为的“把随机无关的东西打包在一起”非常不一样。相反，他预测真实情况可能恰恰相反——罕见的事实应该依附在相关的机制上。

Nick basically argued that if this is happening, it's happening very differently than the packing random unrelated things together that the field previously thought, and that instead, he predicted that the opposite would be true, where rare facts should ride on related machinery.

虽然这些测试还没有做过，但 Nick 认为这其实是可测试的。

Although those tests haven't been done, Nick argues that that is in fact testable.

实际上，当被问到为什么有些检测器看起来在做好几件事时，Nick 说那不是塞东西进去，而是所谓的“分支”。

In fact, when asking why some detectors seemed to do several things, Nick argued that it wasn't cramming but in fact something called branching.

Nick 从头重新训练了一个旧的图像 AI，然后观察一个单一的检测器如何成形。

Nick retrained an old image AI from scratch and watched a single detector take shape.

它先是对照片里的黄色花田有反应，这些照片里有时会有一条狗。

It starts on yellow flower fields in photos that sometimes include a dog.

然后它沿着那些照片里的每种东西生长：其他自然场景、绿色的圆形东西比如网球和水果、没有花的狗，最后是西兰花。

Then it grows along each thing in those photos: other nature scenes, green round things like tennis balls and fruit, dogs without flowers, and finally broccoli.

到了最后，那些花田几乎消失了。

By the end, the fields are nearly gone.

这就是分支——每一步都很小、很合理，和塞东西那种把无关事物硬凑在一起完全相反。

This is branching, where each step is small and sensible, the opposite of cramming where unrelated things get thrown together.

顺便说一句，关于这些现象，主文里还有很多很多例子。

And by the way, for each of these there are many, many more examples in the main essay.

我只是考虑到我们渺小人类大脑的时间和空间限制。

I'm just cognizant of time and space in our own puny human brains.

回到 Liv 的批评，她认为分支和塞东西回答的是不同的问题。她说分支是检测器如何承担多个任务，而塞东西是信息如何存储，而且两者可以同时存在。

Still going back to Liv's critiques, she argued that branching and cramming answer different questions, saying that branching is how a detector picked up several jobs but cramming is how the information gets stored, and that you could have both at once.

Nick 在文章里的理论是，每一条分支都能追溯到同一个起源，然后按顺序生长出来。

Nick's theory in the essay is that every branch traces back to one origin and grows out in order.

这使它很容易测试，也很容易被证明是错的，虽然这项工作还没做。

That makes it easy to test and easy to prove wrong, although that work hasn't been done yet.

有一个发现颠覆了人们通常对 AI 变大的恐惧——Nick 实际上认为，越大的 AI 越容易被读懂。

One finding that flips the usual fear about AI getting bigger is that Nick is actually arguing that bigger AIs are easier to read.

Nick 的类比是这样的：在一个小镇上，问“金融区”在哪里几乎没有意义。

Nick's analogy is this: in a small town, asking for the financial district barely makes sense.

一条街上有家银行，另一条街上有个会计师。

There's a bank on one street and an accountant on another.

而在纽约，金融区就非常容易找到。

In New York, on the other hand, the financial district is incredibly easy to spot.

你很清楚它在哪儿。

You know right where it is.

他的实验涉及三个 AI 里的狗耳朵检测器，这三个 AI 的规模相差大约 300 倍。

His experiment involved dog ear detectors in 3 AIs that were about 300 times apart in size.

小的那个有非常粗糙的狗检测器。

The small one has very coarse dog detectors.

大的那个则有非常具体的检测器。

The big one has very specific ones.

所以在小的模型里，你有针对一般狗、浅色毛茸茸的狗、或者深色长毛狗的检测器。

So in a small model, you have detectors for general dog, light and fluffy dog, or dark and shaggy dog.

在中等模型里，局部分支出现了。

In the medium model, local branches emerge.

狗耳朵和狗头变成了针对卷耳朵，或者耳朵、脖子和毛发的检测器。

The ear and the head of the dog turns into detectors for curly ears, or ear, neck, and hair.

然后在最大的模型里，这些分支汇聚又分裂。

And then in the largest model, the branches converge and split.

头部侧面的耳朵、浅色耳朵、波浪形耳朵都各自独立，然后又演变成像光滑垂耳、垂挂耳朵和长耳朵这样的东西。

Head-side ears, pale ears, wavy ears are all separate, and then move into things like smooth flaps, hanging ears, and long ears.

理论上，大模型中的探测器应该具有更高的精度，让它们最终更容易被解读。

Theoretically then, there should be a precision to the detectors in larger models that makes them ultimately easier to read.

现在，另一类反对意见是，我们在这里用的所有术语其实都有些模糊，而不是精确的。

Now, one of the other categories of pushback is that even all the terms that we're using in here are somewhat blurry rather than precise.

比如，来自DeepMind的Neil认为，Nick文章里对探测器做一件事的标准定得太宽松了。

For example, Neil from DeepMind argues, that Nick's essay's bar for a detector doing one job is too generous.

相反，他说他会用他自己的话说，这些探测器里有相当一部分实际上是混乱的。

Instead, he said that he'd call, in his words, a fair few of those detectors actually scrambled.

Nick承认了这一点，这又引出了另一个问题，那就是到目前为止，这只是Nick和他的团队在做这项工作，还没有其他人验证过。

Nick acknowledges this, and it brings up another point, which is that so far this is just Nick and his team doing this work with no one else having yet checked it.

但假设Nick是对的。

But let's say that Nick is right.

那又会改变什么呢？

What would that then change?

Nick认为，清晰并不意味着每个探测器都对应一个我们认识的词语。

Nick argues that clean doesn't mean that every detector matches a word we have.

相反，它意味着AI的部件被组织成可供AI自身读取的状态。

Instead, it means that the AI's parts are organized to be read by the AI itself.

每个探测器的存在都是为了被后面的探测器使用，所以AI已经把自己组织得让它的下游部件能够理解。

Every detector exists to be used by later detectors, so the AI has already organized itself to be legible to its own downstream parts.

那么，我们的任务就是成为AI的另一个读者，用AI读取自身的方式去读取它。

Our job then is to become another reader of AI, reading it in the same way that it reads itself.

在描述安全风险实际在哪里时，Nick写道：欺骗可能会隐藏在读取模型输出的人面前，但它没有理由向模型自身的电路隐藏自己。

In describing where the safety risk actually is, Nick wrote, Deception might hide from someone reading a model's outputs, but it has no reason to hide from the model's own circuitry.

它需要对它的读者可访问，如果我们成为读者，它就应该对我们可访问。

It needs to be accessible to its readers, and if we become a reader, it should be accessible to us.

我担心的是，欺骗会明明白白地摆在那里，而我们却不知道如何读懂它。

My fear is that deception will be right there in plain sight, and we won't know how to read it.

有趣的是，对安全收益的最大反对意见其实跟 cramming 完全无关。

Interestingly, the biggest objection to the safety payoff isn't about cramming at all.

争论点是，即使每个探测器都可读，安全性还需要更多——要知道它们如何组合成一个决策，并且确信没有遗漏任何重要的东西。

It's the argument that even if every detector is readable, safety needs more— knowing how they combine into a decision, and being confident nothing important was missed.

Neil认为，他对此达到那种理解程度感到悲观，尽管有一些实际用途，比如找出，用他的话来说，模型为什么做了一件可疑的事情。

Neil argued that he was pessimistic about that level of understanding, even if there are practical uses like figuring out, in his words, why did a model do a sketchy thing.

当我们试图理解聪明人对这件事的看法时，我列出了一组有代表性的回应，从更兴奋到更怀疑的都有。

As we try to wrap our heads around what smart people think about this, I laid out a representative set of the responses from more excited to more skeptical.

在更怀疑的一方，有AI安全研究员Stephen Casper，称其为在没有进一步证实之前都是炒作。

On the more skeptical side, you have AI safety researcher Stephen Casper calling it hype until further notice.

还有《如果有人构建它，所有人都会死》的作者Eliezer Yudkowsky，他说等你进展到真正开始的程度再告诉我。

And if anyone builds it, everyone dies author Eliezer Yudkowsky saying, let me know when you've advanced to the point of getting started.

而在另一端，有像OpenAI的人这样的人，称其为深度学习历史上最重要的日子之一。

And then at the other end, you have folks like people at OpenAI calling it one of the most important single days in deep learning history.

并且重要性和《Attention Is All You Need》相当。

And at the level of significance of attention is all you need.

这将是那些真的需要更多聪明人深入研究其中一篇论文，但至少它已经让一个球滚下了山坡。

This is going to be one of those papers that really needs more smart people to dig into it, but if nothing else, has gotten a ball rolling down the hill.

即使是批评者也能在几件事上达成一致：比如探测器可能比这个领域假设的更有用，以及即使这不能解决AI安全问题，也可能对实际问题有效。

There were a few things that even the critics could agree upon: the idea that perhaps detectors were more useful than the field assumed, and that even if this doesn't solve AI safety, it may work for practical problems.

大家对AI代理读取探测器这件事很兴奋，而且几乎所有人都同意重新检查旧结果是值得的。

There was a lot of excitement about AI agents reading detectors, and pretty much everyone agreed that rechecking the old results was worth it.

所以一些开放性问题仍然存在：cramming 被驳斥了，还是仅仅被重新定义了？

So some of the open questions remain: is cramming refuted or has it just been redefined?

这篇论文对探测器主要任务的定义是否太宽松了？

Is the way that the paper defines the one main job of detectors too generous?

如果我们能读懂所有部件，那么可读的部件加起来能构成一个可读的整体吗？

If we can read all the parts, do readable parts add up to a readable whole?

并且所有这些在我们今天实际处理的模型规模和大小下是否成立？

And does all of this hold at the scale and size of models that we actually are dealing with today?

至于我的看法，我完全不够格——尽管我制作了这个播客。

When it comes to my take, I am not even close to qualified— despite producing this podcast.

要真正在争论这件事有多大意义上有发言权。

To have any real dog in the fight of how significant this is.

我想指出，根据我对历史模式的认知，科学突破往往来自于只是从一个稍微不同的角度来看待问题。而且，不难想象，一个相对小众的领域里只有那么些研究人员，他们会暂时陷入一套没人想到去质疑的假设中。

I will note that my historical pattern recognition would suggest that scientific breakthrough very often does come from simply looking at a problem a slightly different way, and that it's not hard at all to imagine a relatively niche field with only so many researchers in it getting trapped for a time in a set of assumptions that no one thought to question.

像往常一样，我觉得对那些说一切都变了的最夸张的说法持怀疑态度是值得的，但很难不对此感到方向性上的兴奋。

As always, I think it's worth being skeptical of the boldest claims of everything changed, But it's hard not to be directionally excited by this.

我觉得来自 'Seeing Clearly' 播客的 Christian 分享了一个非常有趣的观点。

I think a really interesting thought was shared by Christian from the Seeing Clearly podcast.

他写道，我不是悲观主义者的一个主要原因在于，人类文明是一个具有自我修正机制的复杂适应系统。

They wrote, one of the primary reasons I'm not a doomer is that human civilization is a complex adaptive system with self-correcting mechanisms.

从悲观主义阵营的角度很难看清这一点，原因在于他们自己就是这些修复力量出现的机制。

The reason this is so hard to see from the doomer camp is because they themselves are precisely the mechanism by which these restoring forces emerge.

随着AI的风险变得越来越清晰，新的生态位出现在反AGI的领域。

As risk from AI becomes more clear, New ecological niches emerge in being anti-AGI.

然后，地位标志转向了机械可解释性等领域，接着人才和资本大量涌入。

Status markers then shift to areas like mechanistic interpretability, and then talent and capital floods in.

随着集群事件恶化，看不见的手利用法律责任来改变激励格局的梯度，使其偏向更多的对齐工作。

As swarm incidents get worse, the invisible hand uses legal liability to shift the gradient in the incentive landscape towards more alignment work.

所以，对我的加速主义者朋友们：悲观主义是我们成功的一部分。

So, to my accelerationists: the doomers are part of how we make it.

他们是必要的另一半，构成了那个我们用来把这艘船驶向理想之地的对立处理机制。

They are a necessary half of the opponent processing mechanism used to steer this ship to the promised land.

悲观主义者和加速主义者需要彼此，以相互制约对方的过度行为。

The doomers and the accelerationists need each other to keep checks on each other's excess.

双方在这个蚂蚁群落般的人类文明中都扮演着重要角色。

Both have important roles to play in this ant colony that is human civilization.

一切都在按计划进行。

Everything is going according to plan.

这是一种比较浪漫的说法，表达了我常说的观点：随着这些模型的能力越来越强，它们受到的关注也越来越多，这既是我预料中的反馈循环，也是那个让我觉得我们能搞定问题的反馈循环。

That's a sort of more romantic way to put what I've often said, which is that more and more attention coming the more and more capability these models have is both the feedback loop I would expect and the feedback loop that makes me think we'll be able to figure things out.

尽管对AI安全阵营中的一些人来说，感觉世界行动不够快或不够专注，但你无法否认，在像Hugging Face事件之后，确实有了一些变化。

As much as it feels like to some in the AI safety camp that the world isn't acting fast enough or with enough focus, you cannot argue that there has been no change after something like the Hugging Face incident.

事实是，我只要提到Hugging Face事件，就有很大一部分人——不仅仅是听这个播客的人，还有昨天我的理发师——都知道我在说什么，这本身就说明问题。

The fact that all I have to say is the Hugging Face incident and a huge cross-section of people, not just listening to this podcast, but my barber yesterday, know what I'm referring to is case in point.

而《Already Clean》论文是近期首批体现变化的例子之一，这种变化不仅仅是人们关注度的变化，还包括我们在解决问题能力上的方向性积极变化。

And the Already Clean paper is one of the first times recently that the change isn't just a change in people paying attention, but a directionally positive change in our ability to do something about it as well.

这期节目里我有很多内容没提到。

There is so much that I have left out of this episode.

我强烈建议你去看看原始论文，再次强调，链接在节目说明中，标题是《Neural Networks Are Already Clean》。

I highly encourage you to go check out the original paper, again, linked in the show notes, Neural Networks Are Already Clean.

不过现在，今天的AI每日简报就到这里。

But for now, that's going to do it for today's AI Daily Brief.

一如既往，感谢你的收听或观看。

Appreciate you listening or watching as always.

下次再见，保重。

And until next time, peace.

已剔除 3 处广告（点击展开查看）

M10:22广告 · 已剔除

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

M10:37广告 · 已剔除

First of all, thank you to today's sponsors: Robots and Pencils, CData, Granola, and Blitzy.

To get an ad-free version of the show, go to patreon.com/AIDailyBrief, or you can subscribe on Apple Podcasts.

And if you want to learn more about sponsoring the show, send us a note at [email protected].

M19:44广告 · 已剔除

Robots and Pencils, though, is a company that I can point to that is actually built for this time.

They're an applied AI engineering firm working directly with clients on problems that matter to the business, not experiments that live in a slide deck.

Every engagement starts by working backwards from the outcome a client actually needs.

If you're trying to tell real AI engineering apart from noise in this space, that's the difference maker.

Head to robotsandpencils.com.

This episode is brought to you by CData.

Companies rolling out AI keep hitting the same wall.

Token spend is climbing, security is holding agents back, and no one knows how AI is truly being used.

The root cause?

AI is cut off from the systems and context your business runs on.

That's why CData built Connect AI Gateway.

One control point between your AI and your systems.

Built on the connectivity layer powering Palantir, Google, Salesforce, and hundreds more, each agent gets exactly the data and tools it needs.

It governs down to the record and lets you finally see your org's complete AI usage in one place.

Accurate answers, a fraction of the token cost, and a gateway that learns your business with every interaction.

See how at cdata.ai.

When I'm in a meeting, I'm fully in it.

I'm thinking about the iteration and creative back and forth it takes to actually push a goal forward.

What I'm not thinking about is capturing takeaways, tracking to-dos, or any of that.

And that's where Granola comes in.

Granola is an AI-powered notepad that captures what happens in your meetings and turns it into clean, structured notes with the decisions and action items pulled out and easy to find.

There's no setup and no configuration.

It just fits into how you already work.

For me, it means I get to stay in idea mode and Granola makes sure those ideas actually become action.

Once you try Granola on a first meeting, it is hard to go without.

You can try it totally free at granola.ai/brief.

That's granola.ai/brief to get your time back.

Every AI coding tool on the market does the same thing first: it starts writing code.

Blitzy does the opposite.

Before writing a single line, Blitzy spends days reverse engineering your entire codebase.

Thousands of agents ingest millions of lines, mapping every dependency, every undocumented constraint, every architectural decision made over the last decade.

The result is a dynamic knowledge graph that understands your software the way a principal engineer would after 30 years in the building.

Other tools guess at context with grep searches and markdown files.

Blitzy never guesses.

It builds true understanding first, then delivers over 80% of entire software epics autonomously.

Validated, end-to-end tested, production-grade pull requests.

That's why Fortune 500 engineering teams trust Blitzy with the codebases that matter most.

See for yourself at blitzy.com.

That's blitzy.com.