The AI Rankings Just Got Reshuffled — 文稿

M10:00

所以在这个节目里，你经常能听到我随口报出一堆数字，说它们能告诉你某个新模型有多厉害。

So frequently on this show, you will hear me ramble off a bunch of numbers that are supposed to tell you something significant about a new model that has just come out.

而且这些基准测试分数，大多都跟 Artificial Analysis 这家公司有关，它现在已经是大家用来评判 AI 模型的主要基准测试的提供商了。

And more often than not, those benchmark numbers are in some ways related to Artificial Analysis, the company that has become the leading provider of the overarching benchmark that people use to judge AI models.

随着 AI 模型越来越重要，那些告诉我们它们之间表现差异的基准测试，自然也就变得越来越关键了。

And as AI models become more significant, the benchmarks that tell us how they perform relative to one another consequently get more important.

今天我们要聊的是 Artificial Analysis 的智能指数的一次重大更新，以及这个基准测试的新领头羊，猜猜看，居然是来自 Google。

Today we are looking at a major update to Artificial Analysis's Intelligence Index, as well as a new leader for the benchmark, which comes, believe it or not, from Google.

AI Daily Brief 是一个每天更新的播客和视频节目，专门讲 AI 领域最重要的新闻和讨论。

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

好了朋友们，在开始之前先快速说几个公告。

All right, friends, quick announcements before we dive in.

M11:23

欢迎回到 AI Daily Brief 新闻速览版，每天用5分钟左右给你讲完所有 AI 新闻。

Welcome back to the AI Daily Brief Headlines Edition, all the daily AI news you need in around 5 minutes.

今天第一件事本来完全可以当主新闻来讲，说真的，如果更多人能用上这个模型的话，它肯定就是主新闻了。

We kick off today with a story that could have easily been the main, and honestly probably would have been if more people had had their chance to get their hands on this model.

Google 发布了他们的下一代前沿模型 Gemini 4 Carbon，距离 Gemini 4 Argon 发布才过去两周。

Google has unveiled their next frontier model, Gemini 4 Carbon, just 2 weeks after the release of Gemini 4 Argon.

自从 Argon 发布以来，我在节目里也多次提到，有了新的基础模型，Google 就可以用强化学习快速迭代，而不需要花很长时间重新做预训练来推出新模型。

As I have mentioned on this show a number of times since Argon arrived, having a new base model means Google can iterate quickly with reinforcement learning rather than needing to complete a lengthy pre-training run to put out a new model.

这个节奏确实很快——这是我们见过的超大型模型版本之间间隔最短的一次之一。

And this is definitely a quick pace— one of the shortest gaps we've seen between versions of ultra-large models.

CEO Sundar Pichai 甚至暗示这个节奏还会加快，他在周三晚上的财报电话会议上说：这种快速进步凸显了我们惊人的模型动力。

CEO Sundar Pichai even suggested the pace would increase from here, commenting on Wednesday night's earnings call: This rapid progress underscores our incredible model momentum.

我们已经到了这样一个阶段：AI 正在帮助我们提高训练和发布速度。

We are at the point where AI is helping us increase our training and launch velocity.

我预计接下来12个月会持续加速，得益于 RSI。

I would expect the next 12 months to see continued acceleration because of RSI.

所以 Google 在以创纪录的速度迭代，但我敢肯定你们坐在那儿都在问跟所有人一样的问题：这个模型到底好不好？

So Google is iterating at a record pace, but I'm sure you are sitting there asking yourself the same question that everyone is, which is, is the model any good?

尤其是，相比那个不太让人满意的 Argon，它有没有明显改进？

In particular, is it a significant improvement on the underwhelming release of Argon?

至少在基准测试方面，答案是肯定的。

Certainly when it comes to the benchmarks, the answer is a definitive yes.

在编程方面，Carbon 在 Terminalbench 4.0 上提高了8分，在 Frontier Suite 上提高了6分。

In coding, Carbon gained 8 points on Terminalbench 4.0 and 6 points on Frontier Suite.

网络安全方面，在 Exploit Gym 上提高了12分，在 CWE Bench 上提高了3分。

Cybersecurity saw a 12-point increase on Exploit Gym and a 3-point increase on CWE Bench.

在电脑使用方面，Carbon 在 OSWorld 上提高了7分，在 Agents Last Exam 上提高了8分。

For computer use, Carbon picked up 7 points on OSWorld and 8 points on Agents Last Exam.

在这8个关键基准测试中，有6个 Gemini 4 Carbon 现在都排在榜首，超过了 Astra 和 Opus 5.5。

On 6 of those 8 key benchmarks, Gemini 4 Carbon is now the top of the leaderboard, ahead of Astra and Opus 5.5.

Gemini 4 Carbon 现在是新版 Artificial Analysis Index V5 的领跑者，我们会在本期节目的主要部分详细讨论。

Gemini 4 Carbon is now the frontrunner on the new Artificial Analysis Index V5, which we will be discussing in the main part of the episode.

它得了59分，在新评分标准上提升了9分，以第一名超过了 Opus 5.5。

It scored 59 points, representing a 9-point jump on the new scale and placing it ahead of Opus 5.5 in first place.

在 AA Coding Index 上，Carbon 排在第二名，和 Sonnet 5.5 并列，比 Opus 5.5 低3分。

On the AA Coding Index, Carbon landed in 2nd place, tied with Sonnet 5.5 and 3 points behind Opus 5.5.

这对 Google 来说是非常大的进步。

Now these are huge jumps for Google.

当 Argon 两周前发布时，一些人觉得 Google 已经错过了窗口期。

When Argon was released 2 weeks ago, some felt that Google had missed their window.

这个模型如果早一两个月发布可能还很惊艳，基准测试分数超过 Fable，但 Opus 5.5 和 GPT-6.1 Sol 的发布已经把前沿拉得很高，结果 Google 只能跟 Meta 一起待在第二梯队了。

The model might have been impressive a month or two earlier, with benchmark scores topping Fable, but the release of Opus 5.5 and GPT-6.1 Sol had pushed the frontier sufficiently that it left Google in the 2nd tier alongside Meta.

这次发布，如果人们的实际体验跟基准测试一致的话，那 Google 就真的重新回到了前沿赛道上。

This release, if people's experience with it matches the benchmarks, puts Google genuinely back in the race for the frontier.

成本跟 Argon 相比基本没变，Artificial Analysis 测出的每任务成本是7.76美元。

Cost is basically unchanged compared to Argon, with Artificial Analysis measuring cost per task at $7.76.

作为参考，这比 Opus 5.5 便宜23%，比 Astra 贵20%，所以正好处于前沿模型的中等水平。

For reference, that's 23% cheaper than Opus 5.5 and 20% more expensive than Astra, so right in the middle of the road for frontier models.

现在我们到了没那么精彩的部分。

Now here we come to the less great part.

我们没有太多一手的使用反馈，因为Google是在模型还没准备全面发布时就公开了。

We don't have many firsthand testimonials, as Google unveiled the model before it's ready for general release.

他们遵循了政府发布的模型预发布测试指南，并表示他们对这次发布采取了所谓审慎的策略。

They're following the administration's pre-release model testing guidelines and said they're taking what they call a deliberate approach to the rollout.

这个模型将从本周开始提供给一部分网络防御人员，之后会扩展到他们在网络安全防御项目中信任的合作伙伴。

The model will be made available to a subset of cyber defenders beginning this week, with access extended to trusted partners in their cyber defense program.

至于我们其他人，Google写道，他们正在继续收集反馈并迭代安全护栏，以便尽快通过产品界面将Carbon开放给开发者、企业和消费者。

As for the rest of us, Google wrote, we're continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product surfaces as soon as possible.

少数几个谈论他们使用这个新模型体验的群体之一是Arena，他们获得了测试权限。

One of the few groups talking about their experience with the new model was Arena, who had access for testing purposes.

他们发布了一个视频评测，里面全是渲染和游戏设计测试，但普遍被认为不如Opus 5.5好看。

They put out a video review full of render and game design tests, that were generally slammed for not being as pretty as Opus 5.5.

以前我也说过，我不太认为3D设计是衡量大语言模型的标准，不管社交媒体用户多爱盯着这个看，但至少就这个特定应用场景来说，Carbon确实比不上它的竞争对手。

Now, as I have said before, I am not particularly convinced that 3D design is the right yardstick for LLMs, no matter how much social media users like to focus on that, but certainly at least when it comes to that particular use case, Carbon is not at the same level as its rivals.

不过Arena的Peter Gostev还是透露了一些细节，他发帖说：我在发布前用过一阵子Carbon，可以告诉你们，它比Argon好多了。

Arena's Peter Gostev did give a few more details, however, posting, I've used Carbon a bit before the release, and I can tell you it feels a lot better than Argon.

这次Google是来真的了。

This is a real deal from Google this time.

我希望他们从今以后能保持快速发布的节奏。

I hope they keep up with the rapid pace of releases from now on.

而看起来这确实是他们的计划。

And that does seem to be the plan.

在回应Tior Taxes评论说Fable 55还没出来时，DeepMind研究员Zachary Natto写道：但是Gemini删节版还没出来呢。

Responding to Tior Taxes commenting that Fable 55 isn't here yet though, DeepMind researcher Zachary Natto wrote, Gemini redacted isn't here yet though.

Chubby在X上评论说：好吧，现在事情变得疯狂了。

Chubby on X commented, okay, now things get crazy.

他们还有一个未发布的Gemini模型，比Carbon还要好，而Carbon今天已经在AA Intelligence Index上排到了第一。

They have another unreleased Gemini model that's even better than Carbon, and Carbon has already reached number 1 in the AA Intelligence Index today.

Gemini发布后训练研究员Xiao Zhang补充说，Google AI已经走出了低迷期，现在运转得像一台精密的机器。

Gemini post-training researcher Xiao Zhang added that Google AI is out of their funk and operating like a well-oiled machine.

不到一年前，他写道，发布Gemini 3 Pro是一个重大事件，整个团队都在庆祝基准测试拿到第一。

Less than a year ago, he wrote, launching Gemini 3 Pro was a big bang event where the whole team celebrated number 1 benchmark numbers.

今天，Gemini 4 Carbon在AA Index上登顶，但团队内部的感受却截然不同。

Today, Gemini 4 Carbon hits number 1 on the AA index, yet the feeling inside is distinctly different.

我们已经连续几周在日常工作中使用它，几乎已经适应了这种新的智能水平——把雄心勃勃的代码重写当成理所当然，留下过夜的循环去运行几十个实验到仓库里，根本原因分析和修复神秘bug，还用集群去检查和修复强化学习的数据。

We've been dogfooding continuously for weeks, and we've almost adapted to the new intelligence, taking ambitious code rewrites for granted, leaving loops overnight that launch dozens of experiments to repo, root cause, and fix mysterious bugs, and using swarms to inspect and fix RL data.

现在发布感觉就像是定期的检查点一样。

Launches just feel like periodic checkpoints now.

尽管在早期AI应用的讨论中消失了10个月，但可以看出Google仍然赢得了很多人的信任。

And despite taking 10 months off the early adopter AI conversation, you can see that Google still has lots of people's benefit of the doubt.

为什么Google突然像发明了transformer、坐拥半个互联网的数据、还有海量TPU一样疯狂发布模型？

Why is Google suddenly dropping models like they invented transformers, own half the internet's data, and sit on an ocean of TPUs?

哦等等……

Oh wait...

Gemini在AA Index上排第一？

Gemini #1 on AA index?

这让我很久以来第一次打开了AntiGravity。

This is making me open up AntiGravity for the first time in a long time.

如果Google真的做到了，那官方意义上这就是回归季了——Meta、Grok，现在GDM又重新酷起来了。

If Google actually pulls this off, it is officially comeback season with Meta, Grok, and now GDM being cool again.

就我个人而言，我觉得还是先别急着下结论，等我们真正用上这个模型再说，但目前看来很有希望，这本身就让人兴奋了。

For my part, I think we will suspend judgment until we get our hands on the actual model, but it looks promising and that in and of itself is exciting.

说到Google，另一个新闻来自他们和微软的财报。

Now speaking of Google, the other story was in their and Microsoft's earnings reports.

两家公司都交出了又一个非常强劲的云业务增长季度，而这当然也是他们AI收入的来源。

Both companies delivered another incredibly strong quarter in cloud growth, which is where of course they contain their AI revenue.

微软表示，他们的Azure部门在过去一年里增长了48%，收入达到了330亿美元。

Microsoft said their Azure division grew at 48% over the past year to reach $33 billion.

这是微软首次将Azure收入作为单独项列出，鉴于如此强劲的增长，这似乎是一个非常明智的决定。

This is the first time Microsoft has broken out Azure revenue as a separate line item, which seems like a very good decision with growth this strong.

Azure带动整体营收增长19%，本季度达到923亿美元。

Azure drove overall revenue growth of 19% to reach $92.3 billion for the quarter.

这些数据每一项都略高于分析师预期。

Each of those figures was a slight beat over analyst estimates.

微软还透露，OpenAI为Azure贡献了100亿美元的收入，约占总营收的三分之一。

Microsoft also disclosed that OpenAI contributed $10 billion to Azure revenue, around a third of the total business.

微软股价在盘后交易中最高上涨4.8%，创下历史新高。

Microsoft stock was up as much as 4.8% to reach a new all-time high in after-hours markets.

与此同时，谷歌云部门的营收同比增长一倍，本季度达到307亿美元，这意味着它们几乎完全缩小了与微软的差距。

Google's cloud division meanwhile doubled revenue compared to year-ago figures, reaching $30.7 billion for the quarter and meaning they've almost entirely closed the gap with Microsoft.

这远高于华尔街的高预期，然而，正如Aptus Capital Advisors的David Wagner所说，问题是其他方面。

This was well above Wall Street's high expectations, however, as David Wagner of Aptus Capital Advisors put it, the problem is everything else.

搜索营收增长率仅为12%，较上一季度的17%和此前的19%大幅放缓。

Search revenue growth came in at 12%, a massive slowdown from 17% in the previous quarter and 19% the quarter before that.

YouTube营收也连续第三个季度放缓，降至9.2%。

YouTube revenue also added a 3rd consecutive growth slowdown to reach 9.2%.

甚至TPU销售也存在一些疑问，高管们在财报电话会议上回避了关于芯片利润率的问题。

Even TPU sales was a bit of a question mark, with executives dodging a question about profit margins for the chips during the earnings call.

总体营收增长率仍为25%，与预期基本持平，但一些分析师认为，谷歌可能已经到了被迫蚕食自身搜索业务的时刻，因为搜索业务不再是可靠的增长驱动力。

Overall revenue growth still came in at 25%, roughly in line with expectations, but to some analysts, it looks like Google might have reached the moment where they're forced to cannibalize their own search business because it's no longer a reliable growth driver.

事实上，谷歌CEO Sundar Pichai在财报电话会议上承认了这一动态，他评论道，值得注意的是，AI模式查询的强劲增长产生了影响，因为我们在AI模式下的变现能力较低。

In fact, CEO Sundar Pichai acknowledged this dynamic during the earnings call, commenting, it's worth noting that the strong growth in AI mode queries had an impact because of the lower monetization we're seeing in AI mode.

此外，谷歌未能达到利润预期，主要是由于其持有的SpaceX股票进行了减值。

In addition, Google missed on bottom-line earnings largely due to a write-down on their SpaceX stock.

科技分析师Dan Ives一如既往地乐观，试图保持积极语气，他写道，这就是我们一直等待的云利润，在我们看来，这比那些熊今晚会拿来说事的较轻的搜索数字更重要。

Tech analyst Dan Ives, ever the optimist, tried to strike a positive tone, writing, this is the cloud profit we have been waiting for, and in our view, it matters more than a lighter search number the bears will lean on tonight.

而他们确实这么做了，股价在盘后交易中暴跌高达10%。

And lean on it they did, sending the stock plummeting as much as 10% in after-hours markets.

大部分报道聚焦于两家公司在云领域的竞争，它们现在并驾齐驱，但数据中心的建设速度可能更为重要。

Much of the coverage focused on the cloud race between the 2 companies, which are now neck and neck, but the pace of the data center buildout might be even more consequential.

两家公司都保持今年和下一年的资本支出预测不变，但本季度的实际资本支出低于预期。

Both companies held CapEx forecasts steady for this year and next, but actual CapEx spend for the quarter came in under expectations.

以谷歌为例，他们正准备明年将资本支出增加50%，达到3000亿美元，但目前他们的支出低于今年的预期数字。

Google, by way of example, is girding a 50% increase in CapEx next year to $300 billion, but they're currently undershooting this year's numbers.

Oracle和Meta也处于类似情况。

Oracle and Meta are in a similar position.

所以看来AI建设可能已经达到了最高速度。

So it looks like the AI buildout may have reached peak velocity.

考虑到零部件、电力、劳动力等物理限制，更不用说社区支持已经耗尽，这一点都不意外。

That's not at all surprising given physical constraints around components, power, and labor, let alone community support running on empty.

目前我们处于这一趋势的早期阶段，谷歌表示他们将在下一季度的财报中对资本支出进行更深入的讨论。

Now we are quite early in this trend, and Google said that they would have more in-depth discussions on CapEx during the next quarterly earnings.

但有趣的是，许多AI悲观人士认为，资本支出放缓将是戳破AI泡沫的催化剂。

But what makes this interesting is that many AI bears suggested that a CapEx slowdown would be the catalyst that pricks the AI bubble.

然而，我们目前看到的情况并不完全符合这一理论。

What we're seeing, however, doesn't exactly line up with that theory.

其逻辑是，需求会疲软，迫使超大规模计算公司放慢速度。

The logic was that demand would falter and force the hyperscalers to slow down.

相反，看起来这些超大规模计算公司如果能够按自己的意愿行事，会继续加速建设数据中心，但却遇到了更根本的物理限制。

Instead, it looks like the hyperscalers, if they had their druthers, would like to continue accelerating their data center buildout but are running into more fundamental and physical constraints.

科技分析师Luke Longo认为，放缓更多是财务原因，但他们遇到了更根本的物理限制。

Tech analyst Luke Longo thought the slowdown was more about finance but they are running into more fundamental and physical constraints.

最后，Nvidia面临更多反垄断审查，因为司法部正在调查他们收购Hugging Face一事。

Lastly today, more antitrust scrutiny for Nvidia as the DOJ queries their acquisition of Hugging Face.

路透社报道称，美国司法部已询问该收购将如何影响AI行业的竞争。

Reuters reports that the Department of Justice has made inquiries about how the acquisition would impact competition in the AI industry.

特别是，司法部询问，拥有Hugging Face是否会让Nvidia控制AI模型的一个关键分发点。

In particular, the DOJ has asked whether ownership of Hugging Face would allow Nvidia to control a key distribution point for AI models.

当他们在九月完成那笔130亿美元的收购时，Nvidia坚称Hugging Face将继续自由提供平台，并与广泛的模型开发者、芯片制造商和云提供商合作。

When they made the $13 billion acquisition in September, Nvidia maintained that Hugging Face would remain free to offer a platform and partner with a wide range of model developers, chip manufacturers, and cloud providers.

他们表示该交易预计在2027年上半年完成，等待监管批准。

They said the deal was expected to close in the first half of 2027, pending regulatory approval.

这些调查是现在被揭示的针对Nvidia的多年反垄断调查的一部分。

These inquiries form part of what is now revealed to be a multi-year antitrust investigation into Nvidia.

九月，《New York Times》报道称，DOJ已对Nvidia与芯片初创公司Groq的许可和人才收购交易展开调查。

In September, the New York Times reported that the DOJ had opened an investigation into Nvidia's licensing and talent acquisition deal with chipmaking startup Groq.

该交易被构造成一个acquihire交易，这是AI行业的一种常见做法，可以追溯到Google在2024年与Character AI的交易。

That deal was structured as an acquihire deal, which has been a common practice in the AI industry dating back to Google's 2024 deal with Character AI.

因为这些交易在技术上不算是收购，它们不会触发自动反垄断审查，因此理论上可以更快完成。

Because these deals are not technically acquisitions, they don't trigger automatic antitrust reviews and can therefore theoretically close much faster.

之前的假设是，DOJ正在利用对Nvidia的调查来更广泛地打击这种做法。

The assumption had been that the DOJ was using an investigation into Nvidia as a way to crack down on this practice more broadly.

然而，消息人士本周告诉Reuters，这项调查范围更广，似乎早于Groq交易。

However, sources told Reuters this week that the investigation is much broader and seems to predate the Groq deal.

对数据中心优化公司SkedgeMD和RunAI的收购也在调查范围内，因为它们对AI基础设施的竞争格局有影响。

Acquisitions of data center optimization companies SkedgeMD and RunAI are also within scope for their impact on the competitive landscape in AI infrastructure.

而且这两笔交易都是传统收购，而非acquihire交易。

And both of those deals were traditional acquisitions, not acquihire deals.

此外，消息人士称，DOJ最近的提问涵盖了Nvidia的风险投资、销售实践，以及这家芯片设计公司是否对其客户施加了压力。

Furthermore, sources said that recent questioning from the DOJ covered Nvidia's venture investments, sales practices, and whether the chip designer has exerted pressure on its customers.

换句话说，这似乎不再仅仅是一次关于规避反垄断规则的狭隘调查，而是DOJ正在对Nvidia进行一次全面的、自上而下的反垄断调查。

In other words, rather than just a narrow investigation about circumventing antitrust rules, it now seems the DOJ is embarking on a full top-to-bottom antitrust investigation of Nvidia.

显然，现阶段尚未提出任何指控，调查也可能以不采取行动告终，所以这并不暗示Nvidia有任何不当行为。

Obviously, no charges have been filed at this stage and the investigation could conclude with no action, so there's no implication that Nvidia has done anything improper.

尽管如此，与大多数人的最初看法相比，这对这家占主导地位的芯片制造商来说似乎是一次更令人担忧的调查。

Still, compared to what most people thought this was, it appears to be a much more concerning investigation for the dominant chipmaker.

这肯定是我们未来需要关注的事情，但今天就到这里，AI Daily Brief今天的节目到此结束。接下来是主节目。

Something to keep an eye on as we go forward for sure, but for now, that's gonna do it for today's AI Daily Brief. Next up, the main episode.

KPMG和University of Texas at Austin的一项新研究发现，当人们与AI合作时，相似的技能并不能保证相似的结果。

A new study from KPMG and the University of Texas at Austin found that when people work with AI, similar skills don't guarantee similar outcomes.

研究人员研究了500多名职业生涯早期的专业人士，发现表现最好的人通过引导、评估和优化AI的输出来持续放大AI的价值。

Researchers studied more than 500 early-career professionals and found that the best performers consistently amplified the value of AI by guiding, evaluating, and refining its outputs.

这些被称为AI amplifiers的顶尖表现者，并非仅由他们所知道的东西来定义，而是由他们与AI合作的方式来决定。

These top performers, called AI amplifiers, weren't defined by what they knew alone, but by how they worked with AI.

M112:39

AI与您的业务所运行的系统及环境是脱节的。

AI is cut off from the systems and context your business runs on.

M115:02

欢迎回到AI Daily Brief！

Welcome back to the AI Daily Brief!

今天我们讨论的是基准测试——具体来说，是一套新的基准测试，以及它们如何颠覆我们对AI领域的认知。

Today we're talking about benchmarks— specifically, a new set of benchmarks and how they're shaking up our sense of the AI field.

现在，听力敏锐的你们可能想知道我对基准测试到底持什么态度，因为一方面，每次新模型发布时，我都恳求大家只给予它们有限的关注，但同时我又会罗列一大堆基准测试，好像它们很重要一样。

Now, the eagle-eared among you may wonder exactly what I think about benchmarks, given that on the one hand, with every new model release, I implore you to only give them so much attention, and yet at the same time still rattle off a whole bunch of them as though they matter.

简而言之，我认为基准测试有其固有的局限性，但也很重要。

In short, I think that benchmarks are inherently limited, but also important.

没有它们，我们就缺乏好的工具来理解不同模型之间的差异，除了纯粹凭感觉，但几乎在任何情况下，我都不认为我们应该将某个模型在任何一项或甚至总体基准测试上的表现奉为金科玉律。

Without them, we don't have good tools other than strictly vibes to understand the difference between different models, and yet in almost no circumstances do I think we should take the performance of one model on any one benchmark or even in the aggregate as the gospel.

如果做得好，它们能成为有用的指南，帮助我们了解不同模型在我们整体技术栈中的位置，但正如我们今天将看到的，人们正在付出越来越多的努力来解决基准测试的一些固有局限性，并试图改进它们。

Done well, they are a useful guide to helping us understand where different models fit into our overall stack, but, as we will see today, there is more and more effort in addressing some of the inherent limitations of benchmarks and trying to make them better.

每当有新的模型出现，你都会听到我谈论它在Artificial Analysis管理的独立测试中获得的分数。

Whenever there is a new model, you'll hear me talk about the scores that it gets on the independent tests administered by Artificial Analysis.

这家公司刚刚彻底改革了他们测试模型的方式，通过其Intelligence Index的第五版为我们提供了一个全新的前沿AI衡量标准。

Well, that company has just overhauled the way that they test models, giving us a brand new yardstick for frontier AI with version 5 of their Intelligence Index.

该指数选取大约十几个关键基准测试，并将它们提炼成一个单一的分数。

The index takes around a dozen key benchmarks and distills them into a single score.

尽管该指数现在已经深深植入了我们理解AI竞赛的方式中，但如此依赖单一的分数是一个相对较新的现象。

And while the index is now entrenched in how we understand the AI race, leaning so hard on that single score is a relatively new phenomenon.

在agentic coding时代之前，当LLM在很多情况下还只是被美化的Google或写作助手时，感觉才是王道。

Prior to the era of agentic coding, when LLMs were in many cases still glorified Google or writing assistants, vibes reigned supreme.

在聊天机器人竞技场的排行榜上排名是测试新模型的最高标准，因为它告诉了开发者用户想要什么。

Ranking on the Chatbot Arena leaderboard was the pinnacle test of a new model, because it told developers what users wanted.

但随着研究和编码成为更重要的用例，我们有了更多客观的基准测试，能更清楚地显示每个模型在实际工作中的表现。

But as research and coding became more important use cases, we got more objective benchmarks giving a clearer indication of how each model would perform on real work.

这些评估方法也随着时间的推移在不断进化。

These evaluations have evolved over time.

首先，当已有基准测试变得饱和时，就会加入更难的问题。

First, by adding harder questions as benchmarks became saturated.

换句话说，当所有模型都挤到某个基准测试的最高分附近时，该基准的创造者就会加入一堆新的或更难的问题，产生一个迭代版本，希望能提供更细致的区分度，直到这个新版本也被模型饱和为止。

In other words, as all the models crowded the very top score of a benchmark, the creators of said benchmark would go add in a bunch of new or harder questions, creating an iterative version that would hopefully provide more granularity on the difference between different models until that model got saturated as well.

随着智能体式的计算机使用成为模型性能的基础部分，评估方法也经历了更彻底的重新思考。

As agentic computer use became a fundamental part of model performance, the evaluations got an even more complete rethink.

在每一次变化中，Artificial Analysis 也在进化，他们更换基准测试，并重新调整权重，因为不同模型的能力的重要性在相对变化。

Through each of these changes, Artificial Analysis also evolved, switching out benchmarks and rebalancing their weightings as different model abilities became comparatively more important.

这个过程并不总是一帆风顺的。

This hasn't always been a smooth process.

有些时候，智能指数看起来极度饱和，不再能很好地告诉我们哪些模型是最好的。

There have been moments where the Intelligence Index looked extremely saturated and stopped giving us a good indication of which models were best.

有些时候，通过刷基准测试，开发者可以操纵排名，让排名远高于实际测试中应有的水平。

There were moments when benchmark maxing allowed developers to game the system and rank much higher than real-world testing would support.

实际上，对于中国的开放权重模型来说，这种情况一直都挺常见的。

In fact, this has been a fairly consistent story when it comes to Chinese open-weight models.

九月初，Artificial Analysis 推送了 4.2 版本的早期更新，提前实施了他们为 v5 计划的一些改动。

In early September, Artificial Analysis pushed an early update with version 4.2, pulling forward many of the changes they had planned for v5.

这个决定是由 GPT-6 Astra 的发布触发的，它重点改进了计算机使用能力。

That decision was triggered by the release of GPT-6 Astra, which focused heavily on improving computer use.

问题在于，智能指数 v4.1 对智能体基准测试的权重仍然较轻，导致 GPT-6 Astra 的得分有点令人失望。

The problem was that v4.1 of the Intelligence Index still had relatively light weightings for agentic benchmarks, leading GPT-6 Astra to have a slightly disappointing score.

而且不仅仅是失望，这个分数与人们实际使用该模型的体验并不相符。

And more than just disappointing, one that didn't really comport with people's lived experience of the model.

更新之后，Astra 的排名仍然略低于 Fable-5-1，但指数已经更能反映人们实际使用新一代模型的方式了。

After the update, Astra was still ranked slightly behind Fable-5-1, but the index had become a closer reflection on how people were actually using the new crop of models.

而这就是智能指数的故事——一场持续的猫鼠游戏，不断更新基准测试的组合，以匹配快速变化的 AI 实际用例。

And this has been the story of the Intelligence Index— a constant cat-and-mouse game of updating the mix of benchmarks to match the rapidly evolving real-world use cases for AI.

在推出第 5 版指数时，Artificial Analysis 强调，这是第一次大多数基准测试都建立在私有数据集上。

Introducing version 5 of the index, Artificial Analysis highlighted that for the first time, the majority of the benchmarks are built on private datasets.

目的是让刷基准测试变得更难，因为理论上模型开发者无法针对测试来训练模型。

The idea is to make benchmark maxing more difficult, as model developers theoretically can't train their models to the test.

顺便说一下，这就是我们说的刷基准测试——当开发者把那些评估测试和答案包含在训练数据集中时，会导致人为夸大的分数，因为基准测试不再衡量模型的能力，而只是反映出对正确答案的记忆。

And that, by the way, is what we mean by benchmark maxing— when developers include those sort of evaluation tests and their answers in the training dataset, leading to an artificially inflated score, as the benchmarks cease to measure what the model can do, and instead reflect simply a memory of the correct answers.

在第 5 版中，AA 增加了两个新的基准测试。

For version 5, AA added 2 new benchmarks.

Frontier Code 2.0，这是 Cognition 针对真实世界编码任务的私有基准的全新版本。

Frontier Code 2.0, which is the brand new version of Cognition's private benchmark of real-world coding tasks.

这个测试的评分依据是仓库维护者是否会合并该代码。

This test features scoring based on whether a repository maintainer would merge the code.

他们还增加了 Terminal Bench Science，这是一个智能体式的科学研究基准。

They also added Terminal Bench Science, which is an agentic scientific research benchmark.

Humanity's Last Exam，曾被称为近乎不可能的知识搜索任务，现在升级为 HLE Diamond，难度更高。

Humanity's Last Exam, once pitched as a near-impossible knowledge search task, has been upgraded to HLE Diamond, the even more difficult version.

类似的饱和更新也应用于 AA Omniscience（衡量知识和幻觉）以及 CritPT（物理测试）。

Similar saturation updates have been made to AA Omniscience, which measures knowledge and hallucinations, as well as CritPT, a physics test.

GDPVal-AA，衡量真实环境中的智能体工作，被移除了；现在指数依赖 AA Briefcase、Automation Bench AA 和 GDP.pdf，以更细的粒度衡量白领工作。

GDPVal-AA, the measure of agentic work in real-world settings, is out, with the index now relying on AA Briefcase, Automation Bench AA, and GDP.pdf, to measure white-collar work along more granular lines.

同时移除的还有 SciCode（科学研究测试）和 AALCR（长上下文推理）。

Also removed are SciCode, a scientific research test, and AALCR, which tests long-context reasoning.

现在，与其说这些调整是一堆名称和缩写，不如说它们反映了自 1 月份发布第 4 版以来，AI 模型发生了多大的变化。

Now, rather than just a list of names and acronyms, these adjustments say a lot about how AI models have changed since V4 of the index was released in January.

当时，长上下文推理是最关键的功能之一，用于处理大型代码库。

At that time, long-context reasoning was one of the most critical features, used for working on large codebases.

很多编码模型当时的 token 上下文窗口依然有限，而且深受上下文腐烂这个致命问题的困扰。不过后来压缩和记忆方面的改进，让我们现在觉得很多长上下文表现都是理所当然的了。

Many coding models still had limited token context windows and suffered from fatal issues with context rot. Since then, improvements to compaction and memory have allowed us to take a lot of that long-context performance for granted.

智能体工作是另一个巨大变化，从指数重新设计就能明显看出来。

Agentic work is the other huge change that's evident from the redesign of the index.

现在指数里的编码基准只衡量智能体表现，因为那已经被确认为开发者使用现代 AI 的标准方式了；科学基准测的也是现实世界的智能体研究工作，而不是已有的科学知识。

The coding benchmarks included in the index now only measure agentic performance, as that has been cemented as the way developers use modern AI, and the science benchmarks are now testing real-world agentic research work rather than pre-existing scientific knowledge.

目前 Artificial Analysis 选择对指数在四个类别中进行等权加权：知识工作、编码、广义科学，后者既涵盖通用知识也涵盖幻觉率。

For the time, Artificial Analysis has chosen to equally weight the index across 4 categories: Knowledge work, coding, science in general, which encompasses both general knowledge as well as hallucination rate.

从某种意义上说，科学占 25% 的权重感觉像是一个前瞻性的赌注，因为 Anthropic 和 OpenAI 最近几周在科学和数学相关新闻头条上一直交替领跑。

In some ways, it feels like that 25% weighting for science is a forward-looking bet based on how Anthropic and OpenAI have been trading science and math headlines over recent weeks.

那么这个变化对分数产生了什么影响呢？

So what did this change in the scores?

在第五版指数里，Opus 5.5 仍然以 58 分保持领先，至少直到周三谷歌发布了 Gemini 4 Carbon。

In version 5, Opus 5.5 was still in the lead with a score of 58, at least until Google dropped Gemini 4 Carbon on Wednesday.

GPT-6 Astra 以 57 分跃升至当时的第二名。

GPT-6 Astra jumped into then-second place with a score of 57.

而 Astra 之前是第六名，说明由于指数在科学任务中更看重计算机使用能力，它出现了相当大的跃升。

And Astra had previously been in 6th place, showing there's a pretty big jump based on the index giving more weight to computer use in scientific tasks.

GPT-5.1 Sol 从第九名上升到第三名，紧随其后的是 Sonnet 5.5，从第二名跌到第四名。

GPT-5.1 Sol takes 3rd place up from 9th, followed by Sonnet 5.5 which slipped from 2nd to 4th.

GPT-6.1 Terra 从第十名上升到第五名，说明这个模型比本周早些时候很多人以为的要好一些。

GPT-6.1 Terra was in 5th up from 10th, suggesting the model is a little better than many assumed earlier in the week.

稍后我们会谈谈我认为这反映了什么。

In just a moment we'll get into what I think that reflects.

Gemini 4 Argon 上升了一个名次，和 Meta Wonderfull 并列第六名，这对 Wonder 来说是一个相当明显的下滑，因为之前是第三名，不过很多人觉得那个排名和实际体验并不相符。

Gemini 4 Argon gained a rank to tie Meta Wonderfull in 6th place, which is a pretty notable drop for Wonder, which was previously in 3rd, although many felt that that ranking didn't line up with real-world experience.

Fable 5.1 看起来有点过时了，排第八名，从第五名滑下来；第九名则是新指数里最明显的跌幅之一，Kimi K 3.1 Pro 从第四名一路跌下来。

Fable 5.1 is looking a little long in the tooth in 8th place, sliding from 5th, and in 9th place we have one of the most notable drops in the new index, with Kimi K 3.1 Pro falling all the way from 4th.

事实上，Artificial Analysis 指出，指数重新平衡之后，开放权重模型和美国专有模型之间的差距大幅扩大了。

In fact, Artificial Analysis noted that the gap between open-weight models and US proprietary models had opened up massively after the index rebalancing.

他们写道：开放权重模型的差距从 3 分扩大到 12 分。

They wrote, the open weights gap widens from 3 points to 12.

Kimi K 3.1 Pro 是开放权重模型里的第一名，得分 46。

Kimi K 3.1 Pro is the top open weights model at 46.

在已发布权重的模型中，DeepSeek V4.1 Flash 和 GLM 5.3 得分 32 分，落后第一名 26 分，也低于得分 36 分的 Claude Haiku 5.5。

Among models with released weights, DeepSeek V4.1 Flash and GLM 5.3 land at 32, 26 points behind the leader and below Claude Haiku 5.5 with 36 points.

除了美国和中国模型之间新的鸿沟之外，少数几个领先实验室和其他所有实验室之间也出现了明显的断层。

Now, in addition to the new gulf between US and Chinese models, there was also a clear drop-off between the handful of leading labs and everyone else.

Anthropic、OpenAI、谷歌和 Meta 都有得分在 50 分及以上的模型。

Anthropic, OpenAI, Google, and Meta all have models with scores of 50 and above.

Fable 5.1 和 Kimi K 3.1 Pro 刚好在这个门槛之外。

Fable 5.1 and Kimi K 3.1 Pro are just outside that cutoff.

不过，Kimi K 3.1 和 Claude Haiku 5.5 之间还有 10 分的差距，后者领先于其余所有模型。

However, there's then a 10-point gap between Kimi K 3.1 and Claude Haiku 5.5, which leads the rest of the pack.

基本上，通过在指数中加入更多智能体任务，前沿模型和其他模型之间现在有了清晰的分界。

Basically, by adding more agentic tasks to the index, there's now a clear separation between frontier models and the rest.

Artificial Analysis 指出，可以说构成前沿的前 10 名模型，成本差异非常大。

Artificial Analysis noted that costs vary wildly among the top 10 models that arguably make up the frontier.

这主要是因为前 10 名里既有大型模型、token 密集型模型，也有少数更小、更高效的模型。

That's largely because the top 10 is a mix of large models, token-intensive models, and a few smaller, more efficient models.

在光谱的两端，最便宜的是 GPT-6 1T，每个任务 0.96 美元，最贵的是 Fable 5.1T，每个任务 13.04 美元。

At each end of the spectrum, we have GPT-6 1T as the cheapest at $0.96 per task, and Fable 5.1T as the most expensive at $13.04 per task.

为了基于模型规模做更公平的比较，Opus 和 Sonnet 每个任务都要花大约 10 美元，而 GPT-6 Astra 是 6.24 美元，GPT-6 1T 只要 1.27 美元。

For a more apples-to-apples comparison based on model size, Opus and Sonnet both cost around $10 per task, compared to $6.24 for GPT-6 Astra, and only $1.27 for GPT-6 1T.

Muse、Wonder、Grok 4.7 和 Gemini 4 Carbon 都比 GPT-6 Astra 略贵，在接近 8 美元的区间。

Muse, Wonder, Grok 4.7, and Gemini 4 Carbon are all slightly more expensive than GPT-6 Astra in the high $7 range.

现在，大多数人认为这是一次受欢迎的更新。

Now, most people found this to be a welcome update.

OpenAI 战略未来负责人 Dean Ball 写道，这是 Artificial Analysis 的排名第一次大致接近我自己对模型的主观排名。

OpenAI Head of Strategic Futures Dean Ball wrote, this is the first time an Artificial Analysis ranking has even roughly approximated my own subjective ranking of models.

Hasib Qureshi 写道，Artificial Analysis 重新调整他们的智能评分，感觉上更符合大家的直觉，也清理了不少基准测试的问题。

Hasib Qureshi wrote, Artificial Analysis re-ranking their intelligence score seems to match the vibes a lot better and clear out a lot of benchmarking.

这显示出 Anthropic、OpenAI 和其他模型之间的差距更大了。

Shows a deeper gap between Anthropic, OpenAI, and everyone else.

同时，也显示了闭源模型和开源权重模型之间的差距拉大了。

Also shows a bigger gap between closed and open weights.

Ethan Mollick 写道，这个分析跟大家的普遍感觉一致，就是我们还没有看到那种传说中的顶级开源模型。

Ethan Mollick wrote that the analysis matches the general vibes that we haven't seen a fabled class open model.

还有 AI 内容创作者 Theo 进一步强调了这一点，他写道，对于"开源权重模型和 Opus 5.5 一样好"这种观点的人来说，今天可真是糟糕的一天。

And AI content creator Theo drove that point home, writing, terrible day to be an 'open weights models are just as good as Opus 5.5' guy.

不过，也有几个人为开源权重模型辩护。

Now, there were a few people who spoke up for the open weights models.

布鲁金斯学会的研究员 Kyle Chan 指出，其实在此之前，Artificial Analysis 就已经有一些私有的基准测试了。

Brookings Institution fellow Kyle Chan pointed out that even before this, Artificial Analysis had some private benchmarks.

只是份额从 45% 增加到了 60%。

It's just that the share went from 45% to 60%.

他以 Kimi 3.1 Pro 的分数为例，指出虽然头条新闻说的是与顶尖美国模型的差距从 3 分扩大到 12 分，但更重要的是要看具体情况。

Using Kimi 3.1 Pro's score as an example, he suggested that while the headline news is the gap with the top US model widening from 3 points to 12, that it's more important to look at the specifics.

他指出，科学是主要弱项，Kimi 在 TerminalBench Science 上的得分只有 34%，而 GPT-6 Astra 是 63%。

Science, he points out, is a major weakness, with Kimi scoring just 34% on TerminalBench Science versus 63% for GPT-6 Astra.

但他写道，Kimi 在编码代理方面依然非常强，指出它在 TerminalBench 4.0 上实际上领先所有模型，得分是 67%。

But, he writes, Kimi is still extremely strong at coding agents, pointing out that it actually leads all models on TerminalBench 4.0, scoring 67%.

Kyle 的结论是：这是否说明了基准测试存在作弊行为？

Kyle's conclusion: Is this evidence of benchmark gaming?

不一定。

Not necessarily.

Kimi 的一些最大短板是在非私有基准上，但它在私有的 Automation Bench AA 评估上实际超过了 Claude。

Some of Kimi's biggest deficits are on non-private benchmarks, while it actually beats Claude on the private Automation Bench AA evaluation.

他说，更重要的是：中国模型在某些代理任务上能够与美国的前沿模型匹敌甚至超越，但在高级科学推理和研究工作流程方面还有显著的差距。

The bigger story, he says: Chinese models can match or beat U.S. frontier models on certain agentic tasks, but a significant gap remains in advanced scientific reasoning and research workflows.

最后，尽管大体上人们认为这次更新是积极的，但不可能每个人都同意所有细节。

And ultimately, even if in general people think that this is a positive update, no one is going to agree on everything.

特别是，我认为最大的争论之一会是：相关的权重到底应该怎么定。

In particular, I think one of the biggest debates will become: what the relevant weighting should actually be.

没错，OpenAI 和 Anthropic 很明显在强调科学发现，但对于普通个人用户来说，科学发现在代理编码上的重要性，真的跟它在新的 V5 指数里的权重一样吗？

Yes, OpenAI and Anthropic are clearly emphasizing scientific discovery, but relative to the average individual use case, does that matter as much to agentic coding as it does in the weighting of the new V5 index?

那么，这可能意味着需要不同的权重分配，或者说与其盯着那个总体的数字，我们不如更关注我们自己关心的使用场景的各个基准测试。

Now, that might be an argument for a different weighting, or it might be an argument that instead of paying attention to the headliner number, we'd be better suited to focus on the individual benchmarks for the categories of use cases that we care about.

有一个批评我觉得值得考虑，就是排行榜都是在最高配置下运行的。

One critique that I do think is worthy of consideration is the fact that the leaderboards are run based on max settings.

现实生活中没人会用最高配置。

No one runs max in real life.

那中等、高等、超高等配置的排行榜在哪里呢？

Where's the medium/high/extra high leaderboard?

而且要说明的是，Artificial Analysis 并没有忽略这一点，尤其是在他们关于成本的分析中。

And to be clear, it's not like Artificial Analysis ignores that, especially in their analysis that involves cost.

配置水平确实是他们考量的一部分。

The effort setting does become part of the calculus.

但我确实觉得，在实际操作中，如果能让用户不仅按模型，还能按配置水平来排序，可能会更有用。

But I do think that on a practical level, people might be better served by the ability to organize the index not just by model but also by effort.

最后一点：除了新的智能指数，Artificial Analysis 还推出了一个新的进步指数。

Now one last note: alongside the new intelligence index, Artificial Analysis introduced a new progress index.

这个指数把从 2022 年 11 月以来发布的所有模型放在同一个能力标尺上，用来观察各个实验室的领先模型是怎么随着时间进步的。

The index places all models released since November 2022 on a capability scale to see how each lab's leading model improves over time.

它的目的不是关注当前这批模型，而是为了提供一个长期的模型进步视角。

It's designed less to be about the current batch of models and more to give a long-term view of model progression.

不过，这个新的衡量标准让人们可以轻松地拿到很多数据去研究。

Still, the new measure gave people easy access to a bunch of data to play with.

比如 Lisan Al-Ghaib 就看了一下转向推理模型对各大实验室的影响，他评论说："OpenAI 确实有最好的强化学习，要么就是他们的模型没有推理就根本不行。"

Lisan Al-Ghaib, for example, looked at how the switch to reasoning models impacted the major labs, commenting, "OpenAI really does have the best reinforcement learning, or their models just suck without reasoning."

通过以不同的方式绘制数据，他们估计美国和中国之间的差距现在为3.3个月，并且正在缩小。

And by charting the data in a different way, they estimated that the gap between the U.S. and China is now 3.3 months and closing.

总结一下，就像我在开头说的，我认为基准很重要，但本质上有局限性，但我喜欢的是有更多努力投入到改进它们中，而且我认为随后的辩论也极具价值创造力，有助于我们理解不同模型之间的比较。

Summing up, like I said at the beginning, I think benchmarks are important and inherently limited, but I love that more effort is going into improving them, and I think even the debates that follow are also extremely value-creative when it comes to how we understand how different models compare to one another.

不过，今天就到此为止，这是今天的AI Daily Brief。

For now, however, that is going to do it for today's AI Daily Brief.

一如既往，感谢你的收听或观看。

I appreciate you listening or watching as always.

直到下次，再见！

And until next time, peace!

已剔除 3 处广告（点击展开查看）

M10:49广告 · 已剔除

First of all, thank you to today's sponsors, KPMG, CData, Granola, and Blitzy.

To get an ad-free version of the show, go to patreon.com/AIDailyBrief, or you can subscribe on Apple Podcasts.

And if you want to learn more about sponsoring the show, send us a note at [email protected].

On aidailybrief.ai, you can also find links to everything else going on in the ecosystem— free trainings, paid trainings like our new Champions program at Superintelligent, or of course the new How We Got Here podcast, and the first episode out this week called How Tech Lost America.

Again, you can find all of those links from aidailybrief.ai.

M112:12广告 · 已剔除

Learn more about what separates AI amplifiers from everyone else at kpmg.com/us/ai-amplifiers.

/ai-amplifiers.

This episode is brought to you by CData.

Token spend is climbing, security is holding agents back, and companies rolling out AI initiatives might not know how it's truly being used.

The root cause?

AI is cut off from the systems and context your business runs on.

M112:43广告 · 已剔除

That's why CData built Connect AI Gateway, one control point between your AI and your systems.

Connect AI Gateway is not just another static model router.

Prompts get intelligently routed to the right model with exactly the tools and data it needs, govern down to the record.

The result?

Accurate answers at a fraction of the token cost, with a context engine that learns your business with every interaction.

Built on the connectivity layer powering Palantir, Google, Salesforce, and hundreds more.

Give it a try and start your free trial at cdata.ai.

That's C as in connect, data.ai.

You're in back-to-back meetings all day, you're trying to stay present, but you're also worried you'll forget the decision, the action item, the important next step.

This is a scenario that we absolutely live here at the AI Daily Brief, given how fast things move and change and how quickly we're evolving everything that we do.

That's where Granola comes in.

Granola is an AI-powered notepad for meetings.

You jot down rough notes like you always do, and, in the background, Granola transcribes and turns them into clear, useful notes when the meeting ends.

No bots joining your calls, no distractions, just a clean notepad that helps you focus.

During or after the call, you can chat with your notes, ask Granola to pull out action items, help you negotiate, write a follow-up email, or even coach you using recipes which are premade prompts.

Once you try it on a first meeting, it is very hard to go without.

Head to granola.ai/AIDaily and use code AIDAILY to get 100% off for 3 months for new users.

That's granola.ai/AIDaily and code AIDAILY to get 100% off for new users for 3 months.

Blitzy's deep codebase understanding unlocks the thing every roadmap owner cares about: shipping new features.

Here's the truth about building inside a massive enterprise codebase.

Writing code was never the bottleneck.

Context is.

Which system does this touch?

Which contracts can't break?

Which standards apply?

Blitzy already knows because it reverse-engineered your entire codebase into a dynamic knowledge graph before feature work began.

With that complete picture, Blitzy builds features end-to-end.

Architecture, APIs, UI, and tests all validated against your existing systems.

One Blitzy customer built an AI-native application from scratch with 100% autonomous completion, saving over 2,700 engineering hours.

Features that respect your codebase instead of fighting it.

Stop letting your backlog grow faster than your team.

Accelerate your roadmap at blitzy.com.

That's B-L-I-T-Z-Y dot com.