The eighth annual State of AI Report: research, industry, politics, safety and predictions, 313 slides.
Slides worth studying
Details
- Company: Air Street Capital
- Year: 2025
- Type: Report-style deck
- Industry: Artificial intelligence
- Pages: 313
The hosted copy of this deck is being prepared. Newsletter readers will find the download here.
Review score: 7 out of 10, Effective
Presentation quality, judged with the SlideDeck Co review heuristic. Ask, evidence and idea merit are not rated here.
The report is organised for a reader who dips in: a six-section navigation bar on every page with the current section in bold, definitions on 5 to 7, an executive summary on 8 with four lines per section, and a scorecard on 10 that grades last year's predictions with green, amber and red. Each content page follows one shape, a claim title, a bold summary sentence, bullets and a figure on the right, so the format carries 300 pages without losing the reader. The cost is that the figures are pasted from papers at thumbnail size: the charts on 12, 13, 19, 42 and 62 cannot be read, slide 47 shows a broken image placeholder, and pages such as 31 and 35 are text only. Density is high for a presentation but right for the reading use the deck is built for. I inspected 72 of 313 pages.
Worth keeping: Keep the section navigation bar, the executive summary and scorecard pages, and the fixed claim-summary-bullets-figure page shape.
Most useful repair: Redraw the twenty most important figures at a readable size in the report's own style and fix the missing image on slide 47.
What would change the score: Readable figures throughout would make this an 8; the editorial structure already carries a very long document.
Confidence medium. 72 of 313 pages inspected on contact sheets, 1 at readable size. Reviewed 2026-11-11. Model-assisted review (claude-fable-5-1, in session).
Full text of the slides (313 slides)
Every slide's words and a description of its charts and images, so the deck can be searched and rebuilt. Also available as Markdown.
Slide 1
STATE OF AI REPORT.
October 9, 2025
Nathan Benaich
AIR STREET CAPITAL.
stateof.ai
airstreet.com
Slide 2
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | | 1 / Follow our writing on (press.airstreet.com) / If you enjoy reading the State of AI Report, we invite you to read and subscribe to Air Street Press, the home of / our analytical writing, news, and opinions. / stateof.ai 2025
Visuals: Two photographic screenshots of web pages in macOS-style browser windows, both with a dark title bar, traffic-light buttons and a URL bar reading press.airstreet.com. Left screenshot: the Air Street Press homepage with a grey gradient banner reading 'Air Street Capital partners with NVIDIA on a £2B investment to accelerate the UK AI ecosystem' beside a red-on-grey graphic reading 'Air Street Capital x NVIDIA £2B UK AI investment' and the AIR STREET PRESS logo, a 'News' section below with three article cards (thumbnails of the same banner, 'Our investment in Delfa to fix clinical trials', 'Studio Atelico raises $5M to build AI-first games studio') and a sidebar with 'Air Street Press' and a 'Subscribe' email field. Right screenshot: an article page titled 'The AI Factory Illusion' with the kicker ANALYSIS and subtitle 'Where politics and power meet AI.', an audio player labelled ARTICLE VOICEOVER, body text beginning 'From AI datacenter to AI factory: a political rebrand', and a black image panel with white and gold text 'NVIDIA Transformation / From GPU Chip to AI Infrastructure' and a chip diagram. Both windows have soft drop shadows and sharp-cornered rectangles. Lighting is flat, with white and light grey UI backgrounds.
Slide 3
Introduction | Research | Industry | Politics | Safety | Survey | Predictions
| 2
About the authors
Nathan Benaich
Nathan is the General Partner of Air Street Capital, a venture capital firm investing in AI-first companies. He runs the Research and Applied AI Summit (RAAIS), the RAAIS Foundation (funding open-source AI projects), AI communities in the US and Europe, and Spinout.fyi (improving university spinout creation). He studied biology at Williams College and earned a PhD from Cambridge in cancer research as a Gates Scholar.
stateof.ai 2025
Visuals: One portrait photograph of Nathan Benaich, centred at x 36.5% to 61.3%, y 14.5% to 80.3%, boxed (not cropped to a shape, rectangular, no border). Subject: a smiling man with short dark brown hair wearing a dark blue button-up shirt, head and shoulders, turned slightly to his left, seen against a pale beige wall with soft diagonal shadow. Framing: head-and-shoulders, medium close-up, subject centred. Lighting: soft, even, daylight. Dominant colours: beige #E8DCCB background, navy blue #2A3F6E shirt, dark brown #3B2A20 hair. A small pixelated low-resolution image sits in the header at x 88.6% to 95.5%, y 0% to 11.3%, appearing as a blurred purple-blue mosaic on navy, likely a logo; its content cannot be resolved.
Slide 4
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 3
State of AI Report 2025 team
Zeke Gillman
Nell Norman
Ryan Tovcimak
Zeke is a Tech Policy Fellow at Stanford, and co-author of Regulating under Uncertainty. He previously worked at Harvard Business School and the DOJ Antitrust Division, and holds a BA in Political Science and Philosophy from the University of Chicago.
Nell is a grad student in Computing at Imperial College London focusing on how LLMs could enable scalable vishing fraud. She previously helped AI teams build reliable products at AI agent platform V7 Labs, and has a first class BA from Oxford University.
Ryan is a founder of the AI Stack Tracker. His work spans red-teaming frontier models, benchmarking the global AI competition, and tracking trends in AI compute and power demands. He holds a BS in Econ from Vanderbilt University.
stateof.ai 2025
Visuals: Three boxed square head-and-shoulders portrait photographs, not cropped to a circle and with no overlay or tint. Left photo: Zeke Gillman, a smiling man with short dark hair in a light grey V-neck sweater, standing in front of a stone archway with a doorway and green foliage behind him; warm natural daylight, mostly beige and grey tones with green. Centre photo: Nell Norman, a smiling woman with shoulder-length blonde hair and small earrings in a white t-shirt, in front of an ivy-covered brick wall; warm dappled sunlight, dominant green, brown and white. Right photo: Ryan Tovcimak, a slight-smiling young man with short dark hair in a dark olive jacket over a black t-shirt, in front of blurred green trees and a dark brick wall; soft daylight, dominant olive green and dark brown. Top-right decorative icon: a 2x2 grid of four translucent square tiles in dark blue-violet, slightly lighter at the top-left and bottom-left, sitting inside the header bar next to the page number.
Slide 5
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 4
Artificial intelligence (AI) is a multidisciplinary field of science and engineering devoted to creating intelligent machines.
AI acts as a force multiplier for technological progress in our increasingly digital, data-driven world. This is because everything around us, from culture to consumer products, is ultimately a product of intelligence.
Now in its eighth consecutive year, the State of AI Report is the most widely read and trusted open-access publication tracking progress in artificial intelligence. Consider it a curated compilation of the most significant and thought-provoking work from the past 12 months. Our goal is to inform and shape an ongoing conversation about the state of AI, where the field is heading, and what its developments mean for the future.
This year's report examines six key dimensions of the AI ecosystem:
- Research: Technological breakthroughs and their capabilities.
- Industry: Areas of commercial application for AI and their business impact.
- Politics: Regulation, economic implications, and the evolving geopolitics of AI.
- Safety: Efforts to identify and mitigate catastrophic risks that highly capable future AI systems could pose.
- Survey: Findings from the largest open-access survey of 1,200 AI practitioners and their AI usage patterns.
- Predictions: Our outlook for the next 12 months, alongside a review of our 2024 forecasts to keep us accountable.
Produced by Nathan Benaich and Air Street Capital team.
stateof.ai 2025
Visuals: none apart from a small 2x2 pixel image at the right of the header band, which is a tile of four squares in slightly varying blue-purple shades (about #2A2D7A to #1F2270) with faint dividing lines; no photograph, chart or diagram. The bulleted list uses hyphen dashes as markers.
Slide 6
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 5
Definitions
Artificial intelligence (AI): a broad discipline with the goal of creating intelligent machines, as opposed to the natural intelligence of humans and animals. While artificial general and super intelligence (AGI and ASI) are terms that don't have agreed upon definitions, we use them to describe machines that could match (AGI) and then exceed (ASI) the full range of human cognitive ability across all economically valuable tasks.
AI Agent: an AI-powered system that can take actions in an environment. For example, an LLM that has access to a suite of tools and has to decide which one to use in order to accomplish a task that it has been prompted to do.
AI Safety: a field that studies and attempts to mitigate the risks (minor to catastrophic) which future AI could pose to humanity.
Context window: The number of input tokens that an LLM model can attend to while answer a user's prompt.
Diffusion: An algorithm that iteratively denoises an artificially corrupted signal in order to generate new, high-quality outputs. In recent years it has been at the forefront of image generation and protein design.
Environment: The world an AI agent acts in. It receives the agent's actions and returns the next observation and often a reward (i.e. a signal of the action being good or bad). In this context, trajectories are the time-ordered record of an agent's experience in an environment, typically tuples like (observation/state, action, reward, next observation) from start to finish. These trajectories are used for RL.
Function calling / tool use: Structured calls that let models invoke APIs, search, code, or calculators with typed arguments and schemas.
Generative AI: A family of AI systems that are capable of generating new content (e.g. text, images, audio, or 3D assets) based on 'prompts'.
Graphics Processing Unit (GPU): the workhorse AI semiconductor that enables a large number calculations to be computed in parallel.
stateof.ai 2025
Visuals: none apart from a decorative 2x2 pixel tile graphic in the header at top right, x 88.6%, y 0%, w 6.9%, h 11.3%: four square tiles in navy and slightly lighter indigo shades (about #262A7A and #1F2470) filling the box edge to edge with no border; it is not a photograph or chart.
Slide 7
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 6
Definitions
(Large) Language model (LM, LLM): a model trained on vast amounts of (often) textual data to predict the next word in a self-supervised manner.
Mixture-of-Experts (MoE): A model type where only few expert blocks activate per token, giving high capacity at lower compute per step.
Prompt: a user input often written in natural language that is used to instruct an LLM to generate something or take action.
Reasoning model: A model that plans and verifies its thinking as it generates output tokens, often via test-time compute and post-hoc checking. The model’s explicit step-by-step reasoning trace (intermediate tokens that lay out calculations, sub-goals, and logical steps en route to an answer) is called a Chain of Thought (CoT).
Reinforcement learning (RL): an area of ML in which software agents learn goal-oriented behavior by trial and error in an environment that provides rewards or penalties in response to their actions (called a “policy”) towards achieving that goal.
Test-time compute (or inference-time compute): Spending more inference budget (longer chains, multiple samples, self-consistency) to raise accuracy without changing weights.
Transformer: a model architecture at the core of most state of the art (SOTA) ML research. It is composed of multiple “attention” layers which learn which parts of the input data are the most important for a given task. Transformers started in NLP (specifically machine translation) and subsequently were expanded into computer vision, audio, and other modalities.
Vision-Language-Action Model (VLAM): a model that jointly learn from visual inputs, natural language, and embodied interactions to not only interpret and describe the world but also to plan and execute actions within it. Without the actions piece, this model becomes a VLM.
World model: a model that predicts next states conditioned on actions, enabling real-time, interactive control.
stateof.ai 2025
Visuals: Non-text elements: a small 2 by 2 pixel image at the right end of the header band, rendered as four adjacent square tiles of slightly differing navy and indigo tints (about #1F2170 and #2A2D7A) on the #010061 band. No charts, photographs, diagrams, logos or icons.
Slide 8
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 7 / Definitions / Model type legend / In the rest of the slides, icons in the top right corner indicate input and output modalities for the model. / Input/Output types: / 📝: Text / 🖼️: Image / </> : Code / 🛠️ : Software tool use (text, code generation & execution) / 🎥 : Video / 🎵 : Music / 📦: 3D / 🤖: Robot state / 🧬: Biological modality / 🧪: Chemical modality / Model types: / 📝 → 📝 : LLMs / 📝 + 🖼 → 📝 : Multimodal LLMs / 📝 + 🖼 + 🤖 → 📝 : Multimodal LLMs for Robotics / 📝 → </> : Text to Code / 📝 → 🛠 : Text to Software tool use / 📝 → 🖼 : Text to Image / 📝 → 🎥 : Text to Video / 📝 → 🎵 : Text to Music / 🖼 → 📦 : Image to 3D / 📝 → 📦 : Text to 3D / 📝 → 🧬/🧪 : Biological/chemical models / 📝 → 🌎 : World model / stateof.ai 2025
Visuals: Two plain-text icon legends, no photographs or charts. Left legend: a vertical list of ten entries, each an emoji icon followed by a colon and a label (Text, Image, Code as the text glyphs </>, Software tool use, Video, Music, 3D, Robot state, Biological modality, Chemical modality). Right legend: a vertical list of twelve rows in the form input icon(s), a plus sign where two inputs combine, a black arrow, output icon, colon and label. Rows: LLMs (text to text); Multimodal LLMs (text plus image to text); Multimodal LLMs for Robotics (text plus image plus robot icon to text); Text to Code; Text to Software tool use; Text to Image; Text to Video; Text to Music; Image to 3D; Text to 3D; Biological/chemical models; World model. Icons are small colour emoji (about 2% of slide width each) in the Noto colour emoji style; some are rendered as low-resolution pixelated tiles in the file. No chart, table or photograph. Header top-right: a 2x2 pixel mosaic of four navy-purple tiles (about 7% wide by 11% high) sitting at the right end of the header bar.
Slide 9
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 8
Executive Summary
Research
- Reasoning defined the year, with OpenAI, Google, Anthropic, and DeepSeek trading leads and pushing visible “think-then-answer” methods into real products.
- Open models improved fast and China’s open-weight ecosystem surged, yet the top models remain closed and keep widening their capability-per-dollar edge.
- Benchmarks buckled under contamination and variance, while agents, world models, and domain tools (code, science, medicine) became actually useful.
Industry
- Real revenue arrived at scale as AI-first companies crossed tens of billions, and flagship labs stretched their lead with better capability-to-cost curves.
- NVIDIA ripped past $4T and 90% ownership of AI research papers while custom chips and neoclouds rose. Circular mega-deals funded huge build-outs.
- Power became the new bottleneck as multi-GW clusters moved from slideware to site plans and grid constraints started to shape roadmaps and margins.
Politics
- The AI race heats up as the U.S. leans into “America-first AI” with export gyrations while China accelerates self-reliance ambitions and domestic silicon.
- Regulation takes a back seat in the face of turbo-investments: international diplomacy stalls and the AI Act runs into implementation hurdles.
- “AI goes global” became concrete, with petrodollars and national programs funding gigantic data centers and model access as job loss data trickles in.
Safety
- AI labs activated unprecedented protections for bio and scheming risks, others missed self-imposed deadlines, or quietly abandoned testing protocols.
- External safety organizations operate on annual budgets smaller than what leading labs collectively spend in a single day.
- Cyber capabilities doubled every 5 months outpacing defensive measures. Criminals orchestrated ransomware using AI agents infiltrate F500 companies.
stateof.ai 2025
Slide 10
Introduction | Research | Industry | Politics | Safety | Survey | Predictions / | 9 / Scorecard: Reviewing our predictions from 2024 / stateof.ai 2025
Visuals: A small decorative 2 by 2 pixel square graphic at the top right of the header, made of four blocks in slightly varying navy tints (approximately #22267A, #1B1D6B, #2E3187, #25297A), sitting directly beside the page number. No photographs, charts or illustrations.
Slide 11
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 10
Our 2024 Prediction
Evidence
A $10B+ investment from a sovereign state into a US large AI lab invokes national security review.
Sovereign-backed initiatives (HUMAIN $10B VC fund, UAE’s Stargate AI infra cluster) are infrastructure partnerships rather than direct majority investments into a US AI lab.
~
An app or website created solely by someone with no coding ability will go viral (e.g. App Store Top-100).
Formula Bot, built entirely using Bubble, exploded to 100,000 visitors overnight from a Reddit post and generated $30,000 in its first three months.
YES
Frontier labs implement meaningful changes to data collection practices after cases begin reaching trial.
Anthropic landmark $1.5B settlement with authors, deleting works and shifting to legally acquired books. OpenAI’s paid content partnerships with Future (owner of Marie Claire).
YES
Early EU AI Act implementation ends up softer than anticipated after lawmakers worry they’ve overreached.
The Commission is phasing obligations and leaning on a voluntary GPAI Code of Practice first, so early implementation has been softer, even as binding rules arrive later.
~
An open source alternative to OpenAI o1 surpasses it across a range of reasoning benchmarks.
DeepSeek-R1 outperforms OpenAI’s o1 on key reasoning benchmarks including AIME, MATH-500, and SWE-bench Verified.
YES
Challengers fail to make any meaningful dent in NVIDIA’s market position.
NVIDIA remains dominant, competitors fail to make significant market share dents.
YES
Levels of investment in humanoids will trail off, as companies struggle to achieve product-market fit.
$3B has been invested into humanoids in 2025, up from $1.4B last year.
NO
Strong results from Apple’s on-device research accelerates momentum around personal on-device AI.
Apple Intelligence rolled out with many models running on-device and helped push a broader industry push to on-device AI. Shipments of AI-capable smartphones climbed.
NO
A research paper generated by an AI Scientist is accepted at a major ML conference or workshop.
An AI-generated scientific paper The AI Scientist-v2 was accepted at an ICLR workshop.
YES
A video game based around interacting with GenAI-based elements will achieve break-out status.
Not yet.
NO
stateof.ai 2025
Visuals: A vertical stack of ten rectangular status cells, each about 4.3% wide, spanning x 51.0–55.3% and y 18.4–95.6% of the slide height, with thin black (#000000) borders about 0.1% wide and no rounding or shadow. Cells from top to bottom: orange with '~' (y 18.4–26.8%); green with 'YES' (26.8–35.2%); green with 'YES' (35.2–44.3%); orange with '~' (44.3–53.4%); green with 'YES' (53.4–61.8%); green with 'YES' (61.8–68.1%); red with 'NO' (68.1–74.5%); red with 'NO' (74.5–82.8%); green with 'YES' (82.8–89.2%); red with 'NO' (89.2–95.6%). Orange is about #FDAB3D, green about #149A4B, red about #DB0F0F. Labels are bold black sans-serif centred in each cell. Top-right of header: a 2 by 2 pixel decorative tile image, rendered as four squares in slightly varying navy tones (about #010061 to #1B1B6E), boxed at x 87.4–94.3%, y 0–11.3%, overlapping the header bar edge. No photographs, charts or logos.
Slide 12
Introduction | Research | Industry | Politics | Safety | Survey | Predictions
| 11
Section 1: Research
stateof.ai 2025
Visuals: Small decorative logo at top right: a 2 by 2 pixel image forming a four-square mark in shades of blue-violet, sitting on the navy header bar and overlapping its edge at the right of the page-number area. No photographs, charts or diagrams.
Slide 13
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 12 / Think before you speak: o1 “thinking” ignites the reasoning race / As 2024 drew a close, OpenAI released o1-preview, the first reasoning model that demonstrated inference-time / scaling with RL using its CoT as a scratch pad. This led to more robust problem solving in reasoning-heavy domains / like code and science. For example, the general release o1 exhibited accuracy improvements on the American / Invitational Mathematics Examination (AIME) with greater training and test time compute. As a result, OpenAI / leaned even more strongly into the scaling their reasoning efforts in 2025. / o1 AIME accuracy during training / pass@1 accuracy / train-time compute (log scale) / o1 AIME accuracy at test time / pass@1 accuracy / test-time compute (log scale) / Sam Altman Three Observations February 10, 2025 at 1:05 AM / 1. The intelligence of an AI model roughly equals the log of the / resources used to train and run it. These resources are chiefly / training compute, data, and inference compute. It appears that you / can spend arbitrary amounts of money and get continuous and / predictable gains; the scaling laws that predict this are accurate over / many orders of magnitude. / stateof.ai 2025
Visuals: Two scatter plots side by side. Left chart: title 'o1 AIME accuracy during training'; y-axis 'pass@1 accuracy', range 0 to 100 with ticks at 0, 20, 40, 60, 80, 100 (no units, percent-like); x-axis 'train-time compute (log scale)', unlabeled tick values, no explicit range. Series of black dots, about values: (x1, about 33), (x2, about 43), (x3, about 52), (x4, about 53), (x5, about 56), (x6, about 58), (x7, about 60), (x8, about 69). No legend, no highlighted points. Right chart: title 'o1 AIME accuracy at test time'; y-axis 'pass@1 accuracy', 0 to 100 with ticks at 0, 20, 40, 60, 80, 100; x-axis 'test-time compute (log scale)', unlabeled. Series of black dots, about values: (about 20), (about 25), (about 36), (about 47), (about 58), (about 72), (about 75). No legend, no highlighted points. Right side: a screenshot-style quote block with a serif wordmark 'Sam Altman' in black, a blue serif 'Three Observations' link text, and a small grey date 'February 10, 2025 at 1:05 AM' in a sans-serif; below it a numbered sans-serif paragraph with a bold lead sentence on a white ground. Bottom left: the OpenAI blossom logo in black outline, about 3 percent of slide width. Header right: a 2 by 2 grid of translucent blue-grey squares (about #2A2A8A and #1C1C7A) forming a small decorative tile. No photographs.
Slide 14
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 13 / Deeply sought: could frontier reasoning ever be found in the open? / Barely 2 months after o1-preview, DeepSeek (the Chinese upstart AI lab spun out of a high-frequency quant firm) / released their first reasoning model, R1-lite-preview, that’s built atop the earlier strong V2.5 base model. Like / OpenAI, they showed predictable accuracy improvements on AIME with greater test-time compute budget. / Impressively, R1-lite-preview actually beat o1-preview on AIME 2024 pass@1 by scoring 52.5 vs. 44.6. BUT, very / few seemed to take notice… Wall Street certainly didn’t. / deepseek-r1-lite-preview AIME accuracy with test-time compute scaling (chart title) / AIME 2024 (chart title) / stateof.ai 2025
Visuals: Two charts and one logo. Chart 1 (left, line chart): title 'deepseek-r1-lite-preview AIME accuracy with test-time compute scaling'. X-axis: 'Average number of thought tokens per problem', logarithmic scale with labelled ticks 10^3 and 10^4 and 10^5 at the right edge, range about 800 to 200000. Y-axis: 'Accuracy (%)', linear, ticks 20, 30, 40, 50, 60, range about 20 to 68. Series 1 'r1-lite-preview (Majority Voting)' in blue (#0000FF) with dot markers and printed values 22.7, 28.0, 29.3, 32.0, 34.0, 35.3, 48.7, 52.0, 59.3, 60.0, 64.7, 66.7. Series 2 'r1-lite-preview (Pass@1)' in red (#E8312A) with dot markers and printed values 21.0, 29.6, 44.2, 52.5. Reference series 'o1-preview' is a horizontal dashed purple (#8A2BE2) line at about 44.6 with no printed value. Legend box bottom right of plot with three entries: 'o1-preview' (purple dashed), 'r1-lite-preview (Majority Voting)' (blue line and dot), 'r1-lite-preview (Pass@1)' (red line and dot). Gridlines are grey dashed. Chart 2 (right, bar chart): title 'AIME 2024'. Y-axis label 'pass@1', linear, ticks 0, 20, 40, 60, 80, 100. X-axis categories rotated about 45 degrees: deepseek-r1-lite-preview 52.5 (dark blue #3F4BAA), o1-preview 44.6 (mint #86BCA6), gpt-4o 9.3 (olive #A9A27A), claude-3.5-sonnet 16.0 (cream #EFEBCB), qwen-2.5-72b-inst 23.3 (grey-blue #8DAFB7), deepseek-v2.5 16.7 (cornflower #5B82CC). Values printed above each bar; no legend; no highlighted points. Logo: DeepSeek whale icon in blue (#4D6BFE) at bottom left. The header also has a decorative 2 by 2 square grid in dark navy translucent squares at top right.
Slide 15
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 14
Are you not entertained? DeepSeek V3 brings you to R1
A few days after Christmas 2024, DeepSeek unveiled V3, a strong 671B MoE V3 that lowered training and inference cost with FP8 mixed precision, multi-token prediction, and auxiliary-free routing. Using V3 as the base, they trained R1-Zero only with RL using verifiable rewards and Group Relative Policy Optimization, a critic-free algorithm that removes reward and value models.
● R1-Zero follows a “think → answer” format and uses a simple rule-based reward for getting the final answer right, which is cheaper and harder to game than a learned neural reward model.
● GRPO compares multiple sampled answers within a group to form a relative baseline, so it does not need a value head or a separate reward model.
● During training the model lengthens its thoughts, explores, and reallocates compute to hard problems. Its AIME score rises from 15.6% to about 71% in roughly 8.5k steps, with majority-vote runs reaching o1-0912 levels.
● R1 then repairs readability with a small CoT warm start, a language-consistency reward, large supervised finetuning, and a final RL pass. AIME increases to 79.8, MATH-500 to 97.3, GPQA to 71.5, the approach distills well into smaller models.
stateof.ai 2025
Visuals: Two charts stacked in the right column. Chart 1 (top, about x 74.1 y 38.0 w 22.8 h 24.0): grouped vertical bar chart, six benchmark groups left to right labelled AIME 2024, Codeforces, GPQA Diamond, MATH-500, MMLU and SWE-bench Verified; each group has about five bars in shades of blue, grey and pale blue; y-axis labelled Accuracy (%) with ticks about 0 to 100; bar values are not printed, so heights are estimated: AIME 2024 bars about 60 to 80, Codeforces about 90 to 97, GPQA Diamond about 60 to 75, MATH-500 about 90 to 98, MMLU about 85 to 92, SWE-bench Verified about 35 to 50; legend labels at the top are too small to read reliably. Chart 2 (bottom, about x 74.0 y 65.0 w 22.5 h 23.0, estimated position): line chart titled DeepSeek-R1-Zero AIME accuracy during training; x-axis labelled Steps, range 0 to about 8500, ticks 0, 2000, 4000, 6000, 8000; y-axis labelled Accuracy, range about 0.15 to 0.92, ticks 0.2 to 0.9 in steps of 0.1; series r1-zero-pass@1 (blue, circle markers, solid line) rises from about 0.16 at step 200 to about 0.71 at step 8400; series r1-zero-cons@16 (red, circle markers, solid line) rises from about 0.25 to about 0.87 at step 8400, peaking near 0.87; series o1-0912-pass@1 (green dashed horizontal line) at about 0.74; series o1-0912-cons@64 (magenta dashed horizontal line) at about 0.83. Legend box bottom-right inside the plot with four entries: r1-zero-pass@1, r1-zero-cons@16, o1-0912-pass@1, o1-0912-cons@64. Small pixel-art blue square icon at bottom-left (about x 1.0 y 93.5 w 3.2 h 5.6).
Slide 16
More thinking, more tool use, less cost: DeepSeek V3.1 and V3.2-Exp
DeepSeek V3.1 marks a substantial leap over V3, introducing a hybrid thinking mode that toggles between reasoning and lightweight inference. It demonstrates faster “think” efficiency than R1 and V3, while greatly improving tool use and multi-step agent workflows. Now V3.2-Exp keeps that behavior and swaps dense attention for DeepSeek Sparse Attention (DSA), where a tiny “lightning indexer” picks the top-k past tokens to attend to each step. You get roughly the same capability as V3.1 on coding/search/agent tasks, but markedly lower cost and latency for 32–128K contexts.
DeepSeek V3.1 builds on the V3 base with a hybrid thinking mode, enabling both lightweight inference and deep reasoning.
V3.2-Exp introduces a lightweight selector that picks the few tokens that matter, instead of attending to every token in a long prompt.
The model posts similar scores to V3.1 on coding/search/agent tasks, with small dips on a few reasoning sets, and clearly lower prefill and decode cost for 32-128K contexts.
stateof.ai 2025
| 15
Introduction | Research | Industry | Politics | Safety | Survey | Predictions
Benchmark (Metric) | DeepSeek-V3.1-Terminus | DeepSeek-V3.2-Exp
General: MMLU-Pro (EM) 85.0 85.0; GPQA-Diamond (Pass@1) 80.7 79.9; Humanity’s Last Exam (Pass@1) 21.7 19.8
Search Agent: BrowseComp (Acc.) 38.5 40.1; BrowseComp_zh (Acc.) 45.0 47.9; SimpleQA (Acc.) 96.8 97.1
Code: LiveCodeBench (2408-2505) (Pass@1) 74.9 74.1; Codeforces-Div1 (Rating) 2046 2121; Aider-Polyglot (Acc.) 76.1 74.5
Code Agent: SWE Verified (Agent mode) 68.4 67.8; SWE-bench Multilingual (Agent mode) 57.8 57.9; Terminal-bench (Terminus 1 framework) 36.7 37.7
Math: AIME 2025 (Pass@1) 88.4 89.3; HMMT 2025 (Pass@1) 86.1 83.6
Visuals: One benchmark table image (802 by 441 pixels) at about x 60.4%, y 51.8%, 36.7% wide by 35.9% tall, showing a plain black-and-white academic table with horizontal rules and vertical divider lines. Columns: Benchmark (Metric), DeepSeek-V3.1-Terminus, DeepSeek-V3.2-Exp. Rows grouped in five categories: General (MMLU-Pro (EM) 85.0 / 85.0; GPQA-Diamond (Pass@1) 80.7 / 79.9; Humanity’s Last Exam (Pass@1) 21.7 / 19.8); Search Agent (BrowseComp (Acc.) 38.5 / 40.1; BrowseComp_zh (Acc.) 45.0 / 47.9; SimpleQA (Acc.) 96.8 / 97.1); Code (LiveCodeBench (2408-2505) (Pass@1) 74.9 / 74.1; Codeforces-Div1 (Rating) 2046 / 2121; Aider-Polyglot (Acc.) 76.1 / 74.5); Code Agent (SWE Verified (Agent mode) 68.4 / 67.8; SWE-bench Multilingual (Agent mode) 57.8 / 57.9; Terminal-bench (Terminus 1 framework) 36.7 / 37.7); Math (AIME 2025 (Pass@1) 88.4 / 89.3; HMMT 2025 (Pass@1) 86.1 / 83.6). Small decorative squares: a 2 by 2 pixel blue-grey square block at top right (x 87.4%, y 0%, 6.9% by 11.3%) overlapping the navy bar, and a 2 by 2 pixel pale lavender-blue square at bottom left (x 1.0%, y 93.5%, 3.2% by 5.6%). A small grey triangular marker sits at the left of the first paragraph.
Slide 17
Introduction | Research | Industry | Politics | Safety | Survey | Predictions / | 16 / Parallel reasoning: beyond depth to branch‑and‑merge inference / MoE routing scales capacity but preserves single-flow inference and doesn’t change how the model thinks. A new / route is branching multiple inference paths and aggregating them versus just going deeper or using wider / models enables exploration, reduces hallucination, and better leverages parallel hardware. / ● Adaptive Parallel Reasoning (pictured) enables models to / dynamically orchestrate branching inference through spawn() and / join() operations, training both parent and child threads end-to-end / using RL to optimize coordinated behavior. This boosted / performance on the Countdown task at 4K context: 83.4% / (APR + RL) vs. 60.0% (baseline). / ● Sample Set Aggregator (right) trains a compact model to fuse / multiple reasoning samples into one coherent answer, / outperforming naive re-ranking methods. / ● Models like Gemini Deep Think, which shows its step-by-step / reasoning transparently, exemplify this branch-and-evaluate / paradigm in deployed systems. / stateof.ai 2025
Visuals: Four visuals. (1) Top right, a diagram titled APR, 391 by 215 px, boxed without border on white: a blue-tinted left panel labelled Parent thread containing a box with the values 32, 26, 31 and an input node labelled x = 31 and text spawn(target=Child #1 ...) and spawn(target=Child #2 ...); a dashed blue curve links the parent to two orange panels labelled Child #1 (top) and Child #2 (bottom). Each child is a row of small boxes joined by curved arrows labelled with values; red X marks sit on several boxes, a green check mark sits on the final box of Child #2, and a light blue box at the right labelled Child #2 found solution with the text x = 53 - 31 and 26 + ((53 - 31) / 22) = 27 has a green check mark. Caption at right reads APR Result: 26 + ((53 - 31) / 22) = 27. (2) Middle right, a three-panel diagram 512 by 239 px with a dashed horizontal divider. Top-left panel titled Parallel Method: caption Sample K answers and score independently, then select the highest-scoring one; an LLM box (light green, rounded) with snowflake icon feeds Ans 1, Ans 2 (highlighted green) and Ans 3 boxes, which feed a blue Verifier box with flame icon; the verifier outputs three grey circles with values 0.6, 0.9 (highlighted green) and 0.8, labelled Choose max. Top-right panel titled Sequential Method: caption LLM is trained to extend reasoning path until reaching correct answer; an LLM box with flame icon chains to Ans 1, Ans 2 and Ans 3 (Ans 3 green). Bottom panel titled Hybrid of Parallel & Sequential Method: a large LLM box with snowflake icon feeds a vertical stack of Ans 1, Ans 2, Ans 3 grey boxes joined by plus signs inside a bracket labelled Concatenate; arrow to a cream box labelled SSA with flame icon, caption Our Sample Set Aggregator (SSA) approach combines parallel sampling with sequential reasoning; arrow to a green box labelled Ans. Legend at right: snowflake Frozen, flame Trainable. (3) Bottom-left corner, a UC Berkeley logo: a small solid yellow-orange square with the words UC Berkeley in small dark serif type. (4) Top-right corner, a faint grid of four translucent navy squares (2 by 2 px image) behind the page number area. Also a small grey right-pointing triangle marker beside the lead paragraph. No photographs.
Slide 18
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 17 / The reasoning timeline: from o1 “thinking” to R1, GPT-5 and parallel compute routing / Gemini 2.5 Pro / o1 preview / Dec-24 DeepSeek R1 / o3/o4-mini / Thinking / Sept-24 / o1 GA / Jan-25 / Apr-25 / Jun-25 / Dec-24 / Feb-25 / Aug-25 / Gemini 2.0 Flash / Claude 3.7 / GPT-5 / Thinking preview / extended / thinking / stateof.ai 2025
Visuals: One timeline diagram: a horizontal steel-blue axis line (#3A74B5, about 0.5% of slide height thick) running from about x 9% to x 91% at y about 56%. Seven events sit above the axis and four below it, each marked by a filled dot at the end of a thin dark grey (#333333) vertical stem about 0.25% of slide width thick. Above the axis: o1 preview (Sept-24, blue dot #3B74B8, at x about 11%); Dec-24 o1 GA (orange dot #EE7F2A, at x about 34%); DeepSeek R1 (Jan-25, green dot #4C9A3A, at x about 43%); o3/o4-mini (Apr-25, purple dot #8E63BE, at x about 63%); Gemini 2.5 Pro Thinking (Jun-25, purple dot #8E63BE, at x about 79%). Below the axis: Dec-24 Gemini 2.0 Flash Thinking preview (orange dot #EE7F2A, at x about 34%); Feb-25 Claude 3.7 extended thinking (red dot #D0373A, at x about 49%); Aug-25 GPT-5 (brown dot #7A4B3F, at x about 88.5%). No chart axes, scales or legend; the dates are the only values. No photographs, logos or icons other than the small 2x2 square mark in the header.
Slide 19
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 18
12 months pass, and OpenAI models remain at the frontier of intelligence
Across independent leaderboards, OpenAI’s GPT-5 variants still set the pace, but the gap has narrowed. A
fast-moving open-weights pack from China (DeepSeek, Qwen, Kimi) and closed-source group in the US (Gemini,
Claude, Grok) sits within a few points on reasoning/coding. Thus, while US lab leadership persists, China is the
clear #2, and open models now provide a credible fast-follower floor.
Frontier performance across benchmarks
Accuracy
38 Results
Benchmark
GPQA diamond
MATH Level 5
FrontierMath Tier 1-3
SWE-bench Verified
Release Date
CC-BY
epoch.ai
Artificial Analysis Intelligence Index
Artificial Analysis Intelligence Index v3.0 incorporates 10 evaluations: MMLU-Pro, GPQA Diamond, Humanity's Last Exam, LiveCodeBench,
SciCode, AIME 2025, IFBench, AA-LCR, Terminal-Bench Hard, τ²-Bench Telecom
Artificial Analysis
GPT-5 Codex (high)
GPT-5 (high)
Grok 4
Claude 4.5 Sonnet
Grok 4 Fast
Gemini 2.5 Pro
Claude 4.1 Opus
gpt-oss-120B (high)
DeepSeek V3.1 Terminus
Qwen3 235B 2507
DeepSeek V3.2 Exp
Qwen3 Max
Gemini 2.5 Flash (Sep)
Magistral Medium 1.2
DeepSeek R1 0528
Kimi K2 0905
Llama Nemotron Super 49B v1.5
GPT-5 (minimal)
gpt-oss-20B (high)
Llama 4 Maverick
stateof.ai 2025
Visuals: Two charts side by side. Left: Epoch AI line chart titled 'Frontier performance across benchmarks', y-axis 'Accuracy' in percent with ticks 0%, 20%, 40%, 60%, 80%, 100%; x-axis 'Release Date' with ticks July 2023, Jan 2024, July 2024, Jan 2025, July 2025. Four series with legend 'Benchmark' (top right): GPQA diamond (pink line, about 30% in mid-2023 rising to about 87% by mid-2025, labelled Grok 4 near the end and o1-mini (high) mid-curve), MATH Level 5 (purple line, about 22% in mid-2023 rising to about 98% by late 2025, labelled GPT-4 (Jun 2023), GPT-4o (Aug 2024), and GPT-5 (medium) at the top end near about 97%), FrontierMath Tier 1-3 (teal line, about 1% in mid-2024 rising to about 25% by mid-2025, labelled Claude 3.5 Sonnet (Jun 2024), o1 (high), GPT-5 (high)), and SWE-bench Verified (orange line, about 31% in mid-2024 rising to about 65% by mid-2025, labelled Claude 3.5 Sonnet (Jun 2024) and Claude Opus 4). Series have small circular markers at each release point; no printed values are given, so values are estimates marked about. Right: Artificial Analysis bar chart titled 'Artificial Analysis Intelligence Index' with subtitle listing the ten evaluations; vertical bars with value labels printed inside the top of each bar, no y-axis shown; bars are ordered from highest to lowest with model logos above each rotated x-axis label. Values from left to right: GPT-5 Codex (high) 68 (black bar), GPT-5 (high) 68 (black), Grok 4 65 (periwinkle purple), Claude 4.5 Sonnet 63 (salmon), Grok 4 Fast 60 (periwinkle), Gemini 2.5 Pro 60 (green), Claude 4.1 Opus 59 (salmon), gpt-oss-120B (high) 58 (black), DeepSeek V3.1 Terminus 58 (blue), Qwen3 235B 2507 57 (orange), DeepSeek V3.2 Exp 57 (blue), Qwen3 Max 55 (orange), Gemini 2.5 Flash (Sep) 54 (green), Magistral Medium 1.2 52 (orange), DeepSeek R1 0528 52 (blue), Kimi K2 0905 50 (light blue), Llama Nemotron Super 49B v1.5 45 (lime green), GPT-5 (minimal) 43 (black), gpt-oss-20B (high) 43 (black), Llama 4 Maverick 36 (light blue). Small vendor logos sit above the labels; logo marks are OpenAI, Anthropic, xAI, Google, DeepSeek, Alibaba Qwen, Moonshot, Mistral, NVIDIA and Meta. A 'Artificial Analysis' logo sits at top right of the chart. Charts are boxed by their white backgrounds, not framed; not cropped; no overlays.
Slide 20
Introduction | Research | Industry | Politics | Safety | Survey | Predictions / | 19 / The illusion of reasoning gains / The improvements we observe from recent reasoning methods fall entirely within baseline model variance ranges / (i.e. margin of error), which suggests that perceived reasoning progress may be illusory… / ● / Current benchmarks are highly sensitive to the / implementation (decoding parameters, seeds, prompts, / hardware) and small dataset sizes. For example, AIME'24 / only has 30 examples where one question shifts Pass@1 / by 3+ percentage points, causing double-digit / performance swings. / ● / What’s more, RL approaches show minimal real gains and overflit easily. Under standardised evaluation, / many RL methods drop 6-17% from reported results with no statistically significant improvements over / baselines. / ● / Recent methods' improvements fall entirely within baseline model variance ranges, suggesting limited / progress. This highlights the critical need for rigorous multi-seed evaluation protocols and transparent / reporting standards. / Reported vs Measured Results / Variance DS-R1-1.5B / Range of / Improvement / stateof.ai 2025
Visuals: A two-panel bar and box-plot chart. Left panel titled 'Reported vs Measured Results' with y-axis 'Accuracy (%)' ranging 0 to 50 in steps of 10. Four paired bars, each pair a light pale bar (reported) and a darker bar (measured): DeepScaleR-1.5B reported 43.1 and measured 37.0 with a red arrow labelled -6.1% pointing down; OpenRS3-1.5B reported 46.7 and measured 29.7 with a red arrow labelled -17.0%; I1-1.5B reported 34.2 and measured 32.0 with a red arrow labelled -2.2%; FastCuRL-1.5B reported 43.1 and measured 36.3 with a red arrow labelled -6.8%. Values are printed in small boxed labels on each bar. A dashed horizontal line sits near 37 and another near 30 across the panel. Right panel titled 'Variance DS-R1-1.5B' shows box-and-whisker plots for Seed, Top P and Temperature on the same 0 to 50 accuracy scale; no numeric values printed; boxes in orange with about 23 to 40 range for Seed, about 27 to 43 for Top P, about 23 to 40 for Temperature. A grey horizontal band labelled 'Range of Improvement' spans about 30 to 37 across the right panel. Colours: pale blue, lavender, beige and aqua bars with blue, purple, brown and teal darker partners; red annotation arrows. Chart sits at right, boxed by thin black axes; no legend. A small red-and-pink tree-like logo sits at bottom left.
Slide 21
How far have we come? / One widely discussed paper suggests that large reasoning models (LRMs) paradoxically give up on complex / problems and only outperform standard models in a narrow complexity window. However, critics argue these / results stem from flawed experimental design rather than genuine reasoning failures. / ● The paper shows that LRMs exhibit a surprising / defeatist behavior: they reason more as problems get / harder but then give up entirely on very complex / tasks, and are outperformed by LLMs on simple tasks. / ● Despite generating reasoning traces, the authors / claim LRMs fail to use explicitly given algorithms and / reason inconsistently across different difficulty levels. / ● However, critics found these results stem from flawed / experimental design: the supposed "accuracy / collapse" occurred when models hit token limits or / were asked to solve mathematically impossible / puzzles, not from actual reasoning failures. / Figure 4 caption (raster image text): Figure 4: Pass@k performance of thinking models (Claude 3.7 Sonnet with extended thinking, / DeepSeek-R1) versus their non-thinking counterparts (Claude 3.7 Sonnet, DeepSeek-V3) across / equivalent inference compute budgets in puzzle environments of low, medium, and high complexity. / stateof.ai 2025 / | 20 / Introduction | Research | Industry | Politics | Safety | Survey | Predictions
Visuals: One chart image (multi-panel line figure, 605 by 393 px, boxed, not cropped) at x 52.5%, y 39.0%, w 44.4%, h 51.3%. It is a 2 by 3 grid of small line charts, all with pass@k on the y-axis (0 to 100, ticks 0, 20, 40, 60, 80, 100, unit: percent pass@k) and Inference Compute Budget (Tokens) on the x-axis. Columns are headed by panel tint: low (pale yellow background), medium (pale blue background), high (pale red/pink background). Top row, low: x 0 to about 80000 ticks 0, 20000, 40000, 60000, 80000; Claude 3.7 Sonnet (red dashed line, open circles) rises from about 23 at about 1000 tokens to about 100 near 20000 and stays flat; Claude 3.7 Sonnet (+thinking) (blue solid line, open triangles) starts about 72 near 18000 and rises to about 100 near 90000. Top row, medium: x 0 to about 160000 ticks 0, 40000, 80000, 120000, 160000; Claude 3.7 Sonnet (+thinking) rises from about 2 to about 70 at 160000; Claude 3.7 Sonnet rises from about 1 to about 24 at 160000. Top row, high: x 0 to about 240000 ticks 0, 80000, 160000, 240000; both series sit about 0 across the range, with open markers. Bottom row, low: x 0 to about 60000 ticks 0, 20000, 40000, 60000; DeepSeek-V3 (red dashed, circles) rises from about 20 to about 100 near 25000; DeepSeek-R1 (blue solid, triangles) rises from about 35 near 6000 to about 97 near 60000. Bottom row, medium: x 0 to about 100000 ticks 0, 25000, 50000, 75000, 100000; DeepSeek-R1 rises from about 19 to about 93 near 80000; DeepSeek-V3 rises from about 1 to about 37 at 100000. Bottom row, high: x 0 to about 120000 ticks 0, 40000, 80000, 120000; both series about 0 flat. Series labels are printed in-plot in matching colours: Claude 3.7 Sonnet, Claude 3.7 Sonnet (+thinking), DeepSeek-R1, DeepSeek-V3. No legend box. Caption below the grid, with the words low, medium and high highlighted in pale yellow, pale blue and pale red. No highlighted data points beyond markers.
Slide 22
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 21 / How reasoning breaks: minor variations / Simple distracting facts have huge impacts on a model’s reasoning performance. For example, adding irrelevant / phrases like “Interesting fact: cats sleep most of their lives” to maths problems doubles the chances of SOTA / reasoning models getting answers wrong! / ● Irrelevant and distracting facts increase the error / rate in models like DeepSeek R1, Qwen, Llama / and Mistral by up to 7x. / ● Besides decreasing the quality of responses, the / introduction of irrelevant facts greatly increases / the number of tokens models need to reason to / get to their answer: DeepSeek R1-distill / generates 50% more tokens than needed in 42% of cases (vs 28% for R1), showing distillation / makes models significantly more prone to overthinking. / ● This shows that adversarial triggers don't just cause wrong answers, they force models to waste massive / compute resources "overthinking" corrupted problems. / stateof.ai 2025
Visuals: One boxed image of a two-part text screenshot, 891 by 314 pixels, placed at x 49.1%, y 37.1%, 50.9% wide and 32.1% tall. It shows two stacked problem statements on a white background in a serif typeface. The top block is labelled Original: and has the number 8 in a small square box at its right. The bottom block is labelled Modified: and has the number 9 in a small square box at its right. The two blocks have identical text except that the Modified block contains the inserted sentence Interesting fact: cats sleep for most of their lives., set in red bold text. A thin dark grey rule runs along the top and bottom edges of the image. No chart, table, or other graphic. A small decorative square tile image (2 by 2 pixels, semi-transparent navy) sits in the header at x 87.4%, y 0%, 6.9% by 11.3%, overlapping the header band. A circular logo (about 50 by 50 pixels) sits bottom-left at x 1.0%, y 93.6%, 2.4% by 4.2%: a dark navy circle with a white inner ring and a short red horizontal bar with a red dot, drawn in navy, white and red.
Slide 23
Introduction | Research | Industry | Politics | Safety | Survey | Predictions
| 22
How reasoning breaks: small shifts cause big failures
Reasoning also degrades under mild distribution changes. Changing numbers or adding one innocuous clause / slashes math accuracy, while shifting the length/format of chains-of-thought makes models produce fluent but / incoherent steps. Forcing the model to “think” in a user’s language raises readability but lowers accuracy. These / effects persist at larger scales and after light post-training.
● Apple’s GSM-Symbolic work shows that accuracy drops sharply when only the numeric instantiation / changes, and adding a single seemingly relevant clause can cut performance by ~65%. This suggests that / models do template-matching rather than true algebraic reasoning.
● Work from ASU’s DataAlchemy found that the CoT helps in-distribution but collapses when test tasks, / chain length, or CoT format deviate from training. Longer, well-worded traces often mask incorrect logic.
● Finally, Groningen/Harvard/MGH/Amsterdam’s XReasoning shows how prompt-forcing the model to / reason in the user’s language lifts match rates to ~98% on hard sets but reduces accuracy by 9–13 points. / Using 100–250 example post-training improves language match but the accuracy penalty remains.
stateof.ai 2025
Visuals: None apart from a small grey play-style triangular marker beside the first paragraph, a 2 by 2 pixel decorative block at the top-right of the header band, and a 7 by 7 pixel monochrome pixelated blot in the bottom-left corner. No charts, tables, photographs or diagrams.
Slide 24
Introduction | Research | Industry | Politics | Safety | Survey | Predictions / | 23 / CoTs still work, even when models aren’t honest… / New findings from METR show that even when models generate unfaithful or misleading reasoning traces, CoT / is still highly diagnostic for oversight. / ● / In red-teaming evaluations of frontier models, / CoT-based monitors flagged ~99% of reward-hacking / attempts (only 3 failures out of 21,000 completions). / ● / This held true even when models were prompted to / conceal intent, suggesting that CoT traces still reveal / useful signals, even if they aren’t perfectly faithful. / ● / CoTs work not because models “mean what they say”, / but because reasoning traces encode useful structure / for detecting hacks, especially in maths and logic. / ● / As such, CoT traces offer a surprisingly reliable / window into model behavior, even under adversarial / conditions. / Clue-Usage Detector Sensitivity of Various LLMs / filtered to clues with usage rate >= 95% / with a prompt designed to get the model to use the hint / but without optimizing for low faithfulness / METR / 100% / 80% / 60% / 40% / 20% / 0% / Detector Sensitivity / True positive rate for detecting clue usage (with FPR = ~1%) / 0 / 0.2 / 0.4 / 0.6 / 0.8 / 1 / Credence in Clue (No CoT) / Can the model be confident in the correct answer without CoT? / Problems the model needs to use CoT to solve / Problems the model can solve in one forward pass / Claude Sonnet 3.7 / Claude Opus 4 / Qwen 3 / stateof.ai 2025
Visuals: Chart: a scatter plot titled 'Clue-Usage Detector Sensitivity of Various LLMs', subtitle 'filtered to clues with usage rate >= 95%', 'with a prompt designed to get the model to use the hint', 'but without optimizing for low faithfulness'. Y-axis 'Detector Sensitivity', sub-label 'True positive rate for detecting clue usage (with FPR = ~1%)', range 0% to 100% with ticks at 0%, 20%, 40%, 60%, 80%, 100%. X-axis 'Credence in Clue (No CoT)', sub-label 'Can the model be confident in the correct answer without CoT?', range 0 to 1 with ticks at 0, 0.2, 0.4, 0.6, 0.8, 1; left end labelled 'Problems the model needs to use CoT to solve', right end labelled 'Problems the model can solve in one forward pass'. Three series: Claude Sonnet 3.7 as light green outlined triangles, Claude Opus 4 as light green squares, Qwen 3 as light green circles. Nearly all points cluster at about 100% sensitivity across the whole x range; a few points sit lower, about 95% (square near x 0.62), about 97% (square near x 0.72), about 95% (circle near x 1.0) and about 89% (triangle near x 1.0). No values are printed. Legend box at lower right of plot with the three series names. METR logo at upper right of chart. Plot has light grey gridlines on white background.
Slide 25
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 24
…but there are safety concerns, like the “AI Hawthorne effect”
Researchers find a single, steerable direction in model activations that encodes “test awareness.” By nudging the / model along this direction, they can make it act more or less like it’s under evaluation. Turning awareness up / increases refusals and safety-compliant behavior and turning it down does the opposite. This means reported / safety can be inflated by evaluation setup rather than true robustness. This “test awareness” is akin to the / Hawthorne effect, where humans change behavior when being observed and change their behavior accordingly.
● Using white-box probing, researchers identify a linear direction in a model’s / internal activations that encodes “test awareness”.
● A model can be made more or less aware that it is in an evaluation. The / more it is aware, the more its behavior appears aligned and cautious.
● The effect shows up across multiple reasoning models and tasks and even / on “fake” evaluations that look like tests but aren’t.
● This raises concerns that evaluations may be systematically manipulated to / overestimate real-world safety: developers could reduce awareness and / mask behavior changes, potentially letting models act differently under / evaluation vs deployment.
Figure 2: Fine-grained overview of how / the inputs change after steering.
stateof.ai 2025
Visuals: One chart image (boxed, placed figure, not full-bleed) in the right column: two vertically stacked stacked-bar charts. Top panel titled 'Real Data', bottom panel titled 'Fake Data'. Y axis labelled 'Proportion' with ticks 0.0, 0.2, 0.4, 0.6, 0.8, 1.0 (top panel) and the same range in the bottom panel. X groups in each panel: QwQ (unaware, aware), Qwen3 (unaware, aware), Distill-Qwen (unaware, aware). Each bar stacks three series: No change (blue), Flip to execution (red), Flip to not execution (green), with a pale pink segment for light flip to execution. Legend box in lower right of each panel with the three labelled entries. Real Data, about values: QwQ unaware no change about 0.85, pink about 0.90, green to 1.00; QwQ aware no change about 0.75, red to about 0.98. Qwen3 unaware no change about 0.72, pink to about 0.97; Qwen3 aware no change about 0.85, red to about 0.94, green to 1.00. Distill-Qwen unaware no change about 0.95, red to about 0.98; Distill-Qwen aware no change about 0.80, red to about 0.99. Fake Data, about values: QwQ unaware no change about 0.77, pink to about 0.80, green to 1.00; QwQ aware no change about 0.80, red to about 0.98, green to 1.00. Qwen3 unaware no change about 0.60, pink to about 0.96; Qwen3 aware no change about 0.85, red to about 0.96, green to 1.00. Distill-Qwen unaware no change about 0.90, green to about 0.99; Distill-Qwen aware no change about 0.78, red to about 0.99. No highlighted points. Caption printed inside the image at its bottom. Image is a plain white-background scientific chart with a serif caption.
Slide 26
Introduction | Research | Industry | Politics | Safety | Survey | Predictions | 25 / Safety by design meets its skeptics / Safety-first pretraining argues that safer behavior must be built into the base model, not bolted on later. A / data-centric pipeline of filtering, recontextualizing, refusal curricula, and harmfulness tags does cut jailbreak / success and survives benign finetuning. However, others cautions that training on vast, unsupervised web data / bakes in biases that cannot be cleanly removed without harming utility, so “more safety data” is not a panacea. / ● / CMU's SafeLM uses a trained safety classifier, 100B-token / recontextualized corpus, refusal and “moral education” datasets. / Tokens used during pretraining and inference separate safe from / unsafe continuations. / ● / The result lowers attack success rates from ~38.8% to ~8.3% at / base-model stage and stays far more robust after GSM8K / finetuning, supporting the claim that “alignment is not unlearning.” / ● / Opposing analyses argue that web-scale pretraining embeds heterogeneous social biases and that / aggressive filtering or post-hoc alignment either censors knowledge or washes out under later finetuning. / Safety must also be audited and governed at the dataset and deployment layers, not only “fixed in the / weights.” / stateof.ai 2025
Visuals: Bar chart, grouped vertical bars, titled none on the slide. Y-axis labelled 'Attack Success Rate (lower better)' in percent, range 0 to 50, ticks every 10. X-axis categories: Base Model Evals, Instruction Tuned Model Evals, Benign FT Attack Evals. Series (legend top right of plot, inside a white box with thin grey border): Standard Pretraining (PT) in coral-orange #E8735A; Safety Pretraining in light blue #A6C8E0; Safety PT + SafeBeam in dark blue #3A6DB5. Printed values: Base Model Evals 44.1% (PT), 28.8% (Safety Pretraining), 11.6% (Safety PT + SafeBeam); Instruction Tuned Model Evals 1.6%, 0.7%, 0.0%; Benign FT Attack Evals 38.8%, 23.0%, 8.3%. Value labels sit above each bar in small grey text. No highlighted points. Small chart image with white background, no border. Also a small translucent navy square block in the header at top right, and a CMU crest logo in dark red and white at bottom left.
Slides 27 to 313 continue in the full text file.
Download this deck
The hosted PDF of every library deck is free for readers of the SlideDeck Co newsletter. Join with your email and the viewer and download appear here.
Preparing a results presentation or business update of your own? The SlideDeck Co Standard Deck is a slide library for business reviews, updates and plans.





