slidedeck.co

Anthropic course deck, 2024

Cover slide of the Anthropic 2024 course deck
Cover of Anthropic's 2024 course deck.

The deck opens with a title slide and a short history of Anthropic, marking founding, model releases and research milestones on a timeline. It then sets an agenda covering why measurement matters, the company's approach to AI development and the themes of agents, safety and evaluation. The middle section covers the Responsible Scaling Policy: what it is, its goals, the AI Safety Levels and the stated preparation for ASL-3. A section on measuring capabilities follows, showing benchmark saturation, task completion time relative to humans, and comparisons of Claude 3.5 Sonnet with OpenAI o1 on SWE-bench Verified and Aider benchmarks, with cost per token noted. A case study on computer use includes demo videos, then practical safety measures and lessons from the first year. The deck closes with future directions in scaling governance, capability measurement and academic collaboration, followed by a question slide and a thank-you slide. Several slides use section dividers and agenda lists to orient the audience, and benchmark slides pair a headline claim with a short definition of each test.

Slides worth studying

Slide 18 of the Anthropic course deck
Slide 18: A chart of AI test scores against a human baseline set to zero makes the point that benchmarks saturate, with the argument carried by the visual.
Slide 19 of the Anthropic course deck
Slide 19: Plotting task completion time against human developer time turns an abstract capability claim into one clear, memorable number.
Slide 22 of the Anthropic course deck
Slide 22: The headline states the claim with a specific figure and a one-line definition of the benchmark beneath it, a clean pattern for evidence slides.

Questions about this deck

How many slides does the Anthropic measuring agent capabilities deck have?

The deck has 30 slides, with slide 1 as the title slide and slide 30 as the thank-you slide.

What does the Anthropic deck say about the Responsible Scaling Policy?

Slide 12 describes the Responsible Scaling Policy as Anthropic's public commitment to ensuring that model capability does not outstrip its ability to create effective guardrails for that capability and mitigate harm. Slide 15 states that the company is preparing for ASL-3.

What pricing does the Anthropic deck give for Claude 3.5 Sonnet compared with OpenAI o1 on SWE-bench Verified?

On slide 20, the deck says Claude 3.5 Sonnet is $3/Mtok Input and $15/Mtok Output, compared to OpenAI o1 at $15/Mtok Input and $60/Mtok output. The same slide says Claude 3.5 Sonnet beats o1-preview on key agentic benchmarks like SWE-bench Verified.

How is the Anthropic deck structured?

Slide 4 is an overview listing why measurement matters, the company's approach to AI development, and themes of agents, safety, and evaluation. Slide 17 introduces the measuring capabilities section, covering challenges, Anthropic's methods, and a case study on computer use.

Details

More from Anthropic

All Anthropic decks in the library

Review score: 6 out of 10, Needs substantial work

Presentation quality, judged with the SlideDeck Co review heuristic. Ask, evidence and idea merit are not rated here.

The deck has a consistent visual identity and several clear chart slides (19, 20 to 22), where a headline claim sits beside a highlighted bar and a short definition of the benchmark. The route is weakened by an out-of-place QR slide at 2, a historical timeline (3) with tiny illegible labels, and an opening trio of slides (5 to 7) that arrive before the agenda is explained and are not tied to it. The core RSP section is thin: slide 12 is a lone paragraph, slide 14 uses unrelated stock portraits, and slide 16 is a dense process diagram with unreadable text, so the central ASL idea is hard to take in. Several slides are bare agenda or bullet lists (11, 17, 24, 27, 28) and slides 8, 9, 25 and 26 are video placeholders, so the deck depends on a live speaker to carry much of its meaning.

Worth keeping: Keep the chart-plus-claim layout on slides 19 to 22 with the highlighted Sonnet bar, the benchmark definitions and the cost footnote, and the consistent warm palette and section dividers.

Most useful repair: Rebuild slide 16 into larger readable steps, replace the stock portraits on slide 14 with a meaningful ASL illustration, move the QR slide to the end only, and enlarge the labels on slides 3, 5 and 18.

What would change the score: I would add a takeaway headline to each chart and diagram slide (3, 5, 6, 7, 18) and give the RSP section more than one paragraph and a list of goals to build from.

Confidence medium. 30 of 30 pages inspected on contact sheets, 4 at readable size. Reviewed 2026-11-08. Model-assisted review (claude-sonnet-5-5).

Full text of the slides (30 slides)

Every slide's words and a description of its charts and images, so the deck can be searched and rebuilt. Also available as Markdown.

Slide 1

ANTHROP\C

Measuring Agent

Capabilities and

Anthropic’s RSP

Berkeley CS294/194-196 Large Language Model Agents

Ben Mann | 2024.11

CONFIDENTIAL

Visuals: One visual element: the white Anthropic wordmark 'ANTHROP\C' in a bold geometric sans-serif, top left, with the backslash rendered as a stylised slash before the C. No photographs, charts, or diagrams.

Slide 2

Add your

questions!

ANTHROP\C

2

CONFIDENTIAL

Visuals: One QR code, a square black-and-white matrix (500 by 500 pixels) placed at x 44.5%, y 20.0%, w 38.4%, h 68.2%. It has three finder squares, at top-left, top-right and bottom-left, each a black outer square, white ring and black centre. The modules are black on white with no rounded corners, no border, no frame and no tint. It is not cropped and sits boxed only by its own quiet zone. No photographs, icons or charts.

Slide 3

Anthropic’s history

1 year

Anthropic Founded

as a PBC

Claude 1

Completed

Claude 1

Released

Claude 2

Claude

2.1

Claude 3

2021

2022

2023

2024

2025

Mathematical Framework for

Transformer Circuits

Constitutional AI

Sleeper Agents:

Deceptive LLMs

Toy Models of Superposition

(Interpretability)

Dictionary Learning

(Interpretability)

Visuals: Timeline diagram with a terracotta horizontal axis running left to right across the slide, ending in a filled arrowhead at the right. Five year markers 2021, 2022, 2023, 2024 and 2025 are filled terracotta dots on the axis, spaced evenly, each with its bold year label below. Above the axis: an Anthropic logo in a tan circle at 2021 labelled 'Anthropic Founded as a PBC'; 'Claude 1 Completed' with a plain leader line and no icon; 'Claude 1 Released' with a small grey-square icon of a network of nodes and a line-drawn hand and head; 'Claude 2' with a terracotta rounded-square icon of arcs and nodes; 'Claude 2.1' with a tan rounded-square icon of a line-drawn head with a geodesic network, partly overlapped by the label 'Claude'; and 'Claude 3' in bold with a larger terracotta rounded-square icon of a white head profile with a black radial burst. Below the axis: italic descriptions 'Mathematical Framework for Transformer Circuits' (leader from 2022 marker), 'Constitutional AI' (leader from 2023), 'Toy Models of Superposition (Interpretability)' (long leader from between 2023 and 2024), 'Dictionary Learning (Interpretability)' (leader from 2024 marker), and 'Sleeper Agents: Deceptive LLMs' (diagonal leader from the 2024 marker). Above the timeline a dark maroon double-headed arrow spans the interval from about Claude 1 Released to Claude 3, labelled '1 year' in bold maroon type.

Slide 4

Overview

Why measurement matters

Our approach to AI development

Themes: agents, safety, evaluation

ANTHROP\C

4

CONFIDENTIAL

Visuals: none. The slide has no photographs, charts, diagrams or icons. The only graphic marks are three filled terracotta bullet dots (#CC785C) and the ANTHROP\C wordmark in the footer.

Slide 5

AI model release and capabilities timeline / BENCHMARK SCORES / Averaged / 80 / 70 / 60 / 50 / GPT-4 / Claude 2 / GPT-4 / Turbo¹ / Gemini / 1.5 Pro¹ / Claude 3 / Opus / Gemini / 1.0 Ultra / GPT-4 / Turbo² / GPT-4o / Gemini / 1.5 Pro² / Claude 3.5 / Sonnet / Claude 1 / Feb. '23 / Apr. '23 / July '23 / Oct. '23 / Jan. '24 / Apr. '24 / July '24 / RELEASE DATE / Averaged benchmarks are highest reported without best-of-N: / MMLU, GPQA, MATH, MGSM, DROP F1, HumanEval pass@1, / MMMU, AI2D, ChartQA, DocQA, Mathvista / Source: Publicly available data; evaluation scores are / the average of representative scores found online. / 1 = Initial release; 2 = Second release / Claude / GPT / Gemini / ANTHROP\C

Visuals: Line chart titled 'AI model release and capabilities timeline'. Y-axis: 'BENCHMARK SCORES' (bold caps) with 'Averaged' beside it in grey; range about 50 to 85 with labelled ticks at 50, 60, 70, 80, no unit. X-axis: 'RELEASE DATE' from Feb. '23 to July '24, labelled ticks Feb. '23, Apr. '23, July '23, Oct. '23, Jan. '24, Apr. '24, July '24. Three series drawn as smoothed stepped lines with markers: Claude (terracotta/orange-red #E0765A, circle markers), GPT (khaki #BAB39A, square markers), Gemini (pale blue #A9BCD0, diamond markers). Labelled points with values where printed: none print numeric values; estimates: GPT-4 about 68 (about Mar '23); Claude 1 about 54 (Apr '23); Claude 2 about 61.5 (July '23); GPT-4 Turbo¹ about 74 (about Nov '23); Gemini 1.0 Ultra about 70 (about Dec '23); Gemini 1.5 Pro¹ about 71 (about Feb '24); Claude 3 Opus about 75 (about Mar '24); GPT-4 Turbo² about 76 (about Apr '24); Gemini 1.5 Pro² about 77 (about May '24); GPT-4o about 80 (about May '24); Claude 3.5 Sonnet about 81.5 (about June '24). Labels for points: 'GPT-4', 'Claude 2', 'GPT-4 Turbo¹', 'Gemini 1.5 Pro¹', 'Gemini 1.0 Ultra', 'Claude 3 Opus' (Opus in italic), 'GPT-4 Turbo²', 'GPT-4o', 'Gemini 1.5 Pro²', 'Claude 3.5 Sonnet' (Sonnet in italic), 'Claude 1'. Legend box at lower right of plot: light warm grey rounded rectangle #EFEEE8 with three entries: orange-red filled circle 'Claude', khaki square 'GPT', pale blue diamond 'Gemini'. Vertical and horizontal light grid lines in #E4E1D6 at each x tick and y tick; plot baseline and y-axis line dark #1A1A1A. Footnote text is small grey. ANTHROP\C is a bold geometric sans wordmark in grey. A #FAFAF8 rectangle sits at bottom-right of the slide and covers a hidden text box containing CONFIDENTIAL; that text is not visible.

Slide 6

Training compute of notable models / EPOCH AI / Training compute (FLOP) / 333 models / 10^26 / 10^24 / 10^22 / 10^20 / 10^18 / 10^16 / 10^14 / Gemini Ultra / GPT-4 / PaLM / AlphaGo Zero / GPT-3 / AlphaGo Master / AlexNet / 0.6 OOM/year / 2010 / 2012 / 2014 / 2016 / 2018 / 2020 / 2022 / 2024 / Publication date / CONFIDENTIAL

Visuals: Chart: scatter plot titled Training compute of notable models, with a y-axis labelled Training compute (FLOP) on a logarithmic scale from about 10^14 to above 10^26 with labelled ticks at 10^14, 10^16, 10^18, 10^20, 10^22, 10^24 and 10^26, and an x-axis labelled Publication date from 2010 to 2024 with ticks every two years. About 333 teal circular markers (#5CC3C3 fill, about 60 percent opacity, thin dark teal outline) show individual models, rising from about 10^14 in 2010 to about 10^25 by 2024, with the densest cluster between 2021 and 2024 at about 10^21 to 10^24. Seven markers are highlighted as larger dark-outlined points with black leader lines and labels: AlexNet at about 10^17.7 in late 2012, AlphaGo Master at about 10^22.5 in 2017, AlphaGo Zero at about 10^23 in late 2017, GPT-3 at about 10^23.5 in mid 2020, PaLM at about 10^24.3 in early 2022, GPT-4 at about 10^25.2 in early 2023, and Gemini Ultra at about 10^25.5 in late 2023. A straight pink (#E0408F) trend line runs from about 10^14 at 2010 to about 10^24.5 at 2024.5. A pink curved arrow points from the label 0.6 OOM/year to the trend line near 2014 at about 10^16.5, stating the trend rate of 0.6 orders of magnitude per year. No legend is shown. Gridlines are light grey-blue vertical and horizontal lines at each tick. The chart has a light grey frame around the plot area.

Slide 7

Base Scaleup of Effective Compute / 10^8 / 10^7 / 10^6 / 10^5 / 10^4 / 10^3 / 10^2 / 10^1 / 10^0 / 10^-1 / 10^-2 / 10^-3 / 10^-4 / 10^-5 / 10^-6 / 10^-7 / Effective Compute (Normalized to GPT-4) / 2018 / 2019 / 2020 / 2021 / 2022 / 2023 / 2024 / 2025 / 2026 / 2027 / 2028 / Based on public estimates / Automated AI / Researcher/Engineer? / GPT-4: / Smart High Schooler / GPT-3: / Elementary Schooler / GPT-2: / Preschooler / SITUATIONAL AWARENESS | Leopold Aschenbrenner / ANTHROP\C (partly hidden behind the chart image) / CONFIDENTIAL

Visuals: Chart: line chart with dashed projection, shaded bands, and horizontal reference lines, drawn on a logarithmic y-axis. Title: 'Base Scaleup of Effective Compute'. Y-axis: logarithmic, labelled 'Effective Compute (Normalized to GPT-4)', ticks from 10^-7 at the bottom to 10^8 at the top, one tick per decade. X-axis: years 2018 to 2028, one tick per year, no unit label; a note beneath reads 'Based on public estimates'. Series 1, solid dark-blue line (#1E3A8A, about 2.5 px): starts near 10^-7 in 2018, rises to about 10^-3 in 2020, and reaches 10^0 at 2023 (the crossing point with the reference line). Series 2, dashed navy projection from 2023 at 10^0 up to about 10^6 at 2028, labelled 'Automated AI Researcher/Engineer?'. Shaded blue-grey band around the projection, fanning out from a point at 2023 to about 10^7.5 at the upper edge and about 10^4 at the lower edge in 2028 (estimated values, not printed). Reference elements: a horizontal dotted grey line at 10^0 labelled 'GPT-4: Smart High Schooler'; a vertical dashed grey line at 2023; horizontal grey gradient bands around 10^-3 ('GPT-3: Elementary Schooler') and around 10^-5 ('GPT-2: Preschooler'), and a broad grey gradient band spanning roughly 10^4 to 10^8. Legend: none. Highlighted point: the 2023, 10^0 crossing. No values are printed on the data points.

Slide 8

Claude 3.5 Sonnet / Agentic coding

Agentic coding with

Claude 3.5 Sonnet

Claude

Visuals: One photograph-style image, a 320 by 180 pixel 16:9 frame placed centred inside a beige rounded rectangle. The image shows a dark slate-navy title card reading 'Agentic coding with Claude 3.5 Sonnet' in large off-white serif capitals and lowercase, with a small coral-orange sunburst asterisk followed by the word 'Claude' in smaller off-white serif beneath the title. The image is not cropped, sits boxed inside the beige panel with beige bands visible on the left and right, and has no overlay or tint. Dominant colours are dark slate #2E313C, off-white #E8E6E1 and coral #D97757. The image is low resolution and slightly soft. A small black Anthropic-style wordmark mark sits in the bottom-right corner of the slide.

Slide 9

Claude 3.5 Sonnet / Game development

Visuals: One screenshot-style photograph of a chat interface shown inside a beige rounded panel, centred and boxed (not full-bleed). Subject: a chat application window with a left conversation column and a right preview pane. The left column contains a short assistant message beginning 'Certainly! I'll create a simple 8-bit style crab for you using SVG.' followed by a numbered list of four items, a generated artifact card titled '8-bit style crab SVG', and a paragraph describing the crab as an 8-bit style sprite with geometric shapes and a limited color palette of red and black. The right pane shows a red pixel-art crab on a cream background: a rectangular body, two small black square eyes, two wide side claws, and four rectangular legs at the bottom. Composition: the screenshot is centred horizontally with the window in the middle of a warm tan backdrop; framing is wide and slightly soft-focus. Lighting: even, soft, no hard shadows, with a subtle drop shadow beneath the window. Dominant colours: tan #D4A584, cream #F5F1E6, red #FF0000, black #000000. Not cropped on the window; the outer tan area is the photo edge. The image is boxed inside the beige panel with the beige visible on left and right sides.

Slide 10

Responsible Scaling Policy

How is Anthropic developing safe models?

ANTHROP\C

10

Slide 11

Safety frameworks

Historical context

Existing parallels

Anthropic’s RSP

Updates

ANTHROP\C

11

Visuals: none. The only non-text marks are four small filled terracotta bullet dots (#CC785C), one beside each list item, each about 1.7% wide and 4.9% tall, sitting left of the item text.

Slide 12

What is the RSP?

The Responsible Scaling Policy represents Anthropic’s public

commitment to ensuring that model capability does not

outstrip our ability to create effective guardrails for that

capability and mitigate harm.

ANTHROP\C

12

Slide 13

Our Responsible Scaling Policy outlines how we will measure for potential catastrophic risks and then mitigate them / Our goals are to: / ● Provide structure to help us make hard decisions about safety / ● Hold ourselves publicly accountable to developing models safely / ● Learn how to make and iterate on safe decisions / ● Provide a template for policymakers and others in industry / ANTHROP\C / 13

Slide 14

AI Safety Levels

High Level Overview of AI Safety Levels (ASLs)

ASL-1

(smaller models)

ASL-2

(present large models)

ASL-3

(significantly higher risk)

ASL-4

(speculative)

Increasing Effective Compute

ANTHROP\C

14

CONFIDENTIAL (hidden under a background-coloured fill, not visible)

Visuals: One diagram image (1600 by 338 pixels) with the heading High Level Overview of AI Safety Levels (ASLs) in bold black sans-serif at top-left. Below it are four horizontal bands in ascending colour intensity, left to right: ASL-1 (smaller models) on the off-white ground, ASL-2 (present large models) in pale tan, ASL-3 (significantly higher risk) in medium terracotta-tan, and ASL-4 (speculative) in deep rust. Thick black vertical divider lines separate ASL-1 from ASL-2 and ASL-2 from ASL-3 and ASL-3 from ASL-4; thin grey dashed lines sit inside the ASL-2 and ASL-3 bands. Beneath the bands a black horizontal arrow with an open arrowhead runs left to right across the width, labelled Increasing Effective Compute in grey sans-serif. Below, four photographs: (1) a small white rounded robot with two large black eyes and a grey vent, standing on a pale reflective surface against a soft grey-blue background, lit evenly from the front, full-frame and unboxed. (2) A black-and-white head-and-shoulders portrait of a man with swept-back grey hair, dark suit and tie, looking to his right with a stern expression, against a blurred vertical-structure background, unboxed and tightly cropped. (3) A landscape photograph of a man in a red shirt seated at a wooden desk with a white monitor showing a red screen, books and papers beside him, a night city skyline with lit windows through the window behind him; warm and dark tones, unboxed. (4) A close-up of a pale female android face in three-quarter view, with a mechanical grid panel on the back of the head and exposed metallic neck, neutral expression, soft light background, unboxed.

Slide 15

How do we implement our RSP?

When Anthropic approaches a new level of model capability, the RSP mandates

that Anthropic prepare necessary safety measures for it.

We are preparing for ASL-3.

The RSP highlights AI Safety Levels:

ASL-1

Smaller Models

ASL-2

Present Large

Models

ASL-3

Significantly

Higher Risk

ASL-4

Speculative

ANTHROP\C

15

CONFIDENTIAL

Visuals: A four-cell horizontal band, not a chart. Cell 1 (left) is white #FFFFFF with label ASL-1 / Smaller Models. Cell 2 is pale sand #EADBBC with label ASL-2 / Present Large / Models. Cell 3 is muted tan #D4A27F with label ASL-3 / Significantly / Higher Risk. Cell 4 (right) is terracotta #CC785C with label ASL-4 / Speculative. The cells increase in saturation from left to right in a continuous progression, each separated by a thin grey vertical rule. Above the cells is a thin grey outlined rectangle containing the heading The RSP highlights AI Safety Levels:. No photographs, icons or other images.

Slide 16

How do we implement our RSP?

Step 1: Group capability triggers and corresponding safeguards by AI Safety Level (ASL)

Step 2: Evaluate current models for ASL-3 capabilities

Current regime

Testing for

Defining

ASL-1

No catastrophic risk

ASL-2

Early signs, but no catastrophic risk

ASL-3

Mitigates catastrophic risk from non-state actors

ASL-4

Mitigates catastrophic misuse risk by state actors and/or autonomy risks

...

Model is ASL-2

Approaching ASL-3 capabilities

ASL-3 capabilities imminent

Trigger Capabilities

1. Biological weapons development

2. Cyberattack assistance

3. Early autonomous replication/AI R&D capabilities*

*AI R&D capability only triggers ASL-4 definition, not ASL-3 safeguards

Claude 3.5 Sonnet is here

Step 3: Trigger ASL-3, implement predefined safeguards, publish ASL-4

ASL-3 Standard

Security Standards

Prevent theft by most non-state actors

Deployment Standards

Prevent misuse despite hundreds of hours of jailbreaking attempts

Define ASL-4 Triggers

Publish a list of Trigger Capabilities requiring ASL-4 safeguards, and test ASL-3 models for these

ANTHROP\C

CONFIDENTIAL

Visuals: Diagram, not a photo. Top: a tan rounded container holding a row of four column headers in italics (Current regime, Testing for, Defining) and boxes. ASL-1 is a cream box with the text No catastrophic risk. ASL-2 is a pale cream box spanning two columns, with Early signs, but no catastrophic risk above three sub-cells: a green Model is ASL-2 cell, an amber Approaching ASL-3 capabilities cell, and a red ASL-3 capabilities imminent cell. ASL-3 is a tan box above a Trigger Capabilities tan box that lists three numbered items. A dashed line separates the ASL-3 and Trigger Capabilities boxes. ASL-4 is a brick-red box. A row of three dots sits to the right of ASL-4. A yellow five-point star sits on the bottom edge of the green cell with the caption Claude 3.5 Sonnet is here below it. A footnote in italics sits in the lower right of the container. A thin black arrow runs from the bottom of the Trigger Capabilities box down to the ASL-3 Standard container. The ASL-3 Standard container holds three rounded white cards: Security Standards, Deployment Standards and Define ASL-4 Triggers. No photographs or icons. The rendered image shows no CONFIDENTIAL mark because a #FAFAF8 patch covers that corner, but the text layer is present at the measured position.

Slide 17

Measuring capabilities

Challenges

Anthropic’s methods

Case study: computer use

ANTHROP\C

17

Slide 18

Benchmarks don’t last

Test scores of AI systems on various capabilities relative to human performance

Within each domain, the initial performance of the AI is set to –100. Human performance is used as a baseline, set to zero.

When the AI’s performance crosses the zero line, it scored more points than humans.

Our World in Data

20

0

-20

-40

-60

-80

-100

1998

2005

2010

2015

2020

2023

Reading comprehension with unanswerable questions

Reading comprehension

Image recognition

Language understanding

Nuanced language interpretation

Handwriting recognition

Speech recognition

Predictive reasoning

General knowledge tests

Math problem-solving

Code generation

Complex reasoning

Human-performance as the benchmark, set to zero

The capability of each AI system is normalized to an initial performance of –100

Data source: Kiela et al. (2023)

OurWorldInData.org/artificial-intelligence | CC BY

Note: For each capability, the first year always shows a baseline of –100, even if better performance was recorded later that year.

Test scores of AI systems on / various capabilities relative to / human performance. Human / performance is used as a / baseline, set to zero.

Kiela et al. (2023)

CONFIDENTIAL

Visuals: Chart (a multi-series line chart embedded as a picture, boxed, not cropped, sharp edges, white background). Title: 'Test scores of AI systems on various capabilities relative to human performance'. Y-axis: test score relative to human performance, ticks at 20, 0, -20, -40, -60, -80, -100 (no units; zero is the human baseline, -100 is the normalised start). X-axis: years, ticks labelled 1998, 2005, 2010, 2015, 2020, 2023. Values are not printed; estimates: every series starts at -100 in its first year. Reading comprehension with unanswerable questions (dark green) starts about 1998 at -100 and rises to about 19 by about 2022. Reading comprehension (purple) starts about 2016 at -100 and reaches about 18 by about 2020. Image recognition (orange) starts about 2009 at -100 and reaches about 16 by about 2021. Language understanding (magenta) starts about 2018 at -100 and reaches about 15 by about 2023. Nuanced language interpretation (dark red-brown) starts about 2019 at -100 and reaches about 8 by about 2023. Handwriting recognition (pink-red) starts about 1998 at -100 and reaches about 2 by about 2018. Speech recognition (teal green) starts about 1998 at -100 and reaches about -1 by about 2016. Predictive reasoning (blue) starts about 2019 at -100 and reaches about 0 by about 2023. General knowledge tests (tan) starts about 2020 at -100 and reaches about -5 by about 2023. Math problem-solving (rust orange) starts about 2021 at -100 and reaches about -12 by about 2023. Code generation (coral) starts about 2021 at -100 and reaches about -48 by about 2023. Complex reasoning (navy) starts about 2021 at -100 and reaches about -43 by about 2023. Legend: a right-hand list of twelve labelled series in their line colours, with thin leader lines to each line end. A dashed grey horizontal line at 0 is labelled Human-performance as the benchmark, set to zero. Highlighted points: the start point of each series at -100 and the end points at the right of each series. Embedded logo: Our World in Data, navy box with white text at top right of the chart. Footer inside the chart: Data source and Note lines at bottom left; OurWorldInData.org/artificial-intelligence | CC BY at bottom right. Source: Kiela et al. (2023). Only the chart is a visual; no photograph.

Slide 19

Measuring task completion time relative to humans / Claude 3.5 Sonnet can handle tasks / that would take human developers / around 30 minutes, in seconds. / August 2024 Eval from METR (Model Evaluation and Threat Research) / ANTHROP\C

Visuals: One bar chart inside a rounded white panel, on the right. Chart type: vertical bar chart with error bars. Title printed inside the chart: 'Agent Performance vs Humans with Time Limits (95% CI)'. Y-axis: label 'Fraction of Tasks Completed (Weighted)', linear, range 0.0 to 1.0, ticks at 0.0, 0.2, 0.4, 0.6, 0.8, 1.0. X-axis: categories, left to right: GPT-4o mini, GPT-4 Turbo, GPT-4o, Claude 3 Sonnet, Claude 3 Opus, Claude 3.5 Sonnet. No values are printed on the bars; estimated values: GPT-4o mini about 0.18 (95% CI about 0.05 to 0.31); GPT-4 Turbo about 0.26 (about 0.14 to 0.38); GPT-4o about 0.34 (about 0.19 to 0.49); Claude 3 Sonnet about 0.15 (about 0.04 to 0.26); Claude 3 Opus about 0.28 (about 0.15 to 0.42); Claude 3.5 Sonnet about 0.42 (about 0.28 to 0.57). Three OpenAI-model bars are blue (#2EA3D5 outlines) and three Claude bars are orange (#E8822A outlines), each with a different hatch pattern: diagonal stripes, cross-hatch lattice, and dense crosshatch. Error bars are black with caps. Reference dashed grey lines labelled at left above each line: 'Human (no time limit)' at about 0.8, 'Human (8 hrs)' at about 0.67, 'Human (2 hrs)' at about 0.55, 'Human (30 mins)' at about 0.36, 'Human (10 mins)' at about 0.24. The Claude 3.5 Sonnet bar and its error bar are highlighted by a rounded orange-red (#D9704F) rectangle outline around the full category. Legend: none. Source line below the panel in italic serif.

Slide 20

3.5 Sonnet / outperforms / OpenAI o1

SWE-bench Verified

Assesses a model’s ability to complete real-world software engineering

tasks and understand, modify, and test code with tools

Despite being significantly faster

and cheaper than o1*, Claude 3.5

Sonnet beats o1-preview on key

agentic benchmarks like

SWE-bench Verified – making it

the best choice across the

industry for production usage

*Claude 3.5 Sonnet is $3/Mtok Input and $15/Mtok

Output, compared to OpenAI o1 at $15/Mtok Input and

$60/Mtok output (billing is also non-deterministic)

ANTHROP\C

Sources: OpenAI o1 model system card, Anthropic model card (October 2024 3.5 model addendum)

20

CONFIDENTIAL

Visuals: Bar chart (image, 600 by 371 px) titled SWE-bench Verified, with a subtitle describing the benchmark. Y axis: percentage, 0% to 50%, gridlines every 10% (0%, 10%, 20%, 30%, 40%, 50%). X axis: model names, no axis title. Series with labelled values (single series, all values printed): OpenAI o1-preview (pre-mitigation) 38%; OpenAI o1-mini (pre-mitigation) 32%; OpenAI GPT-4o 33%; Claude 3.5 Sonnet (new) 49%; Claude 3.5 Sonnet 33%; Claude 3.5 Haiku 41%; Claude 3 Opus 22%. Bars for the three OpenAI models are black (#000000); Claude bars are terracotta (about #C97757). The Claude 3.5 Sonnet (new) bar is highlighted by a terracotta outline rectangle (about 2 px stroke) enclosing the bar and its label. Value labels sit inside the top of each bar: white on black bars, dark on terracotta bars. Gridlines are thin light grey (about #C8C8C8). No legend. Chart sits on a white rectangle with no border, inside the image box. Image is not cropped and is boxed (not full-bleed).

Slide 21

3.5 Sonnet

outperforms

OpenAI o1

Despite being significantly faster

and cheaper than o1*, Claude 3.5

Sonnet beats o1-preview on key

agentic benchmarks such the

aider code editing benchmark –

making it the practical choice

*Claude 3.5 Sonnet is $3/Mtok Input and $15/Mtok

Output, compared to OpenAI o1 at $15/Mtok Input and

$60/Mtok output (billing is also non-deterministic)

ANTHROP\C

Aider’s code editing benchmark

Measures the LLM’s coding ability, ability to write new code that

integrates into existing code, and apply changes to the source file

without human intervention

Sources: Aider LLM Leaderboards

21

CONFIDENTIAL

Visuals: One bar chart, presented as a raster screenshot (826 by 434 pixels) placed in a white box at x 41.1%, y 32.1%, w 53.3%, h 49.8%, not cropped, with a square corner. Chart type: vertical bar chart with four bars. No chart title inside the image; the heading above it is the slide's text. Y axis: unlabelled numeric scale from 0 to 90, tick labels every 10 (0, 10, 20, 30, 40, 50, 60, 70, 80, 90), light grey horizontal gridlines at each tick and faint vertical gridlines between bars. X axis category labels: Sonnet 10/22, o1-preview, Sonnet 06/20, GPT-4o 08/06. No legend. Values (none printed, estimates): Sonnet 10/22 about 84; o1-preview about 79.5; Sonnet 06/20 about 77; GPT-4o 08/06 about 71. Highlighted point: the Sonnet 10/22 bar is filled pale pink (about #FFE0E6) with a pink-red outline (about #F08A9A), and a rectangular box outlined in burnt orange (about #C8704F, about 2 px stroke) surrounds that bar and its label. The other three bars are filled pale blue (about #D6EAF8) with a mid-blue outline (about #4A9FD8). Also a small ANTHROP\C wordmark is visible in grey at bottom-left (see elements).

Slide 22

3.5 Sonnet outperforms OpenAI o1

Despite being significantly faster and cheaper than o1*, Claude 3.5 Sonnet beats o1-preview on key agentic benchmarks such as an 18% lead in the aider refactoring benchmark – making it the practical choice

*Claude 3.5 Sonnet is $3/Mtok Input and $15/Mtok Output, compared to OpenAI o1 at $15/Mtok Input and $60/Mtok output (billing is also non-deterministic)

ANTHROP\C

Aider’s refactoring benchmark

A more challenging benchmark which tests the model’s ability to output long chunks of code without skipping sections or making mistakes

Chart: 100 90 80 70 60 50 40 30 20 10 0 | Sonnet 10/22 o1-preview Opus Sonnet 06/20 GPT-4o 08/06 o1-mini

Sources: Aider LLM Leaderboards

22

CONFIDENTIAL

Visuals: One bar chart, placed as a raster image of 680 by 366 pixels in a white box at x 40.7%, y 29.6%, w 54.3%, h 52.0%, with no visible border. Chart type: vertical bar chart, single series, no chart title inside the image, no legend. Y axis: unitless numeric scale from 0 to 100 in steps of 10, labelled 0 to 100 at left, light grey horizontal gridlines at each 10. X axis: six categories labelled left to right Sonnet 10/22, o1-preview, Opus, Sonnet 06/20, GPT-4o 08/06, o1-mini. Values (about, as no values are printed): Sonnet 10/22 about 92; o1-preview about 75; Opus about 72; Sonnet 06/20 about 64; GPT-4o 08/06 about 49; o1-mini about 45. Highlighted point: the first bar, Sonnet 10/22, is filled pale pink and enclosed in a rounded red-orange outline box that extends from the top of the plot down through the category label. All other bars are pale light-blue fill with thin medium-blue outlines. Lighting is flat, no shadows.

Slide 23

Case study / Computer use / ANTHROP\C / 23

Slide 24

Computer / use

Demos!

Technical implementation

Safety considerations

Future implications

ANTHROP\C

24

CONFIDENTIAL

Visuals: none (the only non-text marks are the terracotta bullet dots, described under the bullet list element, and the ANTHROP\C wordmark, listed as a logo element)

Slide 25

Claude 3.5 Sonnet

(upgraded)

Computer

use demo:

Coding

ANTHROP\C

https://www.youtube.com/watch?v=jqx18KgIzAE

25

CONFIDENTIAL

Visuals: One photograph, a still frame from a YouTube video, placed as a boxed rectangle. Subject: a smiling woman with dark hair and clear-framed glasses, seated at a desk behind an open dark laptop, with a red pixel-style robot figurine on a green table in the foreground. Overlaid white serif text reads 'Computer use for orchestrating tasks' with a small orange asterisk-burst logo followed by 'Claude' beneath it. The composition is a medium shot with the woman on the right third and the text on the left; the image is cropped at the right slide edge. Lighting is soft and diffuse, with a muted grey-olive wall, a dark green table and dark clothing. Dominant colours: grey-olive #6F6F66, dark green #2E5B4A, red #B0212B, cream text #F2EFE6. A slight dark tint overlays the whole frame. The image is low resolution (320 by 180 pixels).

Slide 26

Claude 3.5 Sonnet

(upgraded)

Computer

use demo:

Operations

ANTHROP\C

https://youtu.be/ODaHJzOyVCQ

26

CONFIDENTIAL

Visuals: Photograph (video still), boxed and cropped at the right slide edge, at x 26.1%, y 13.0%, w 73.9%, h 73.9%. Subject: a young man with curly blond hair, wearing a black hoodie with a small white 'Claude' logo on the chest, seated at a desk and gesturing with his right hand toward the camera. Beside him at right is an open dark laptop, and in front of him a white ceramic mug, a small red pixel-art figurine and a flat notepad with a dark pen. The wall behind him is warm grey with faint pinkish texture and a shelf with small objects at upper right. Overlaid on the wall in cream serif type is the text 'Computer use for automating operations', with a terracotta asterisk-star mark and the word 'Claude' below it. Composition: medium close-up, subject on the right third, text overlay on the left two-thirds. Lighting: soft, diffuse, low-contrast indoor light. Dominant colours: warm grey #6E6A62, deep green desk #1F5C4A, black hoodie #1A1A1A, cream text #EDE8DC, terracotta accent #CC785C. The image is low resolution and slightly blurred. No tint is applied beyond the video's own colour grade. Logos and icons: the Claude asterisk mark on the wall overlay and the small Claude logo on the hoodie.

Slide 27

Practical safety measures

ASL standard implementation

Security measures

Deployment safeguards

Lessons learned in year 1

ANTHROP\C

27

Slide 28

Future

Scaling governance

Capability measurement

Academic collaboration

ANTHROP\C

28

Visuals: Three small filled terracotta circles (#CC785C) act as bullet markers, one beside each bullet line. The Anthropic wordmark 'ANTHROP\C' in bold grey sans-serif capitals functions as a logo. No photographs, charts, tables or diagrams.

Slide 29

Add your questions! ANTHROP\C 29

Visuals: One image: a black-and-white QR code, square, 38.4% of slide width by 68.2% of slide height, at x 44.5%, y 20.0%. Black square modules on white, roughly a 25 by 25 module grid, with three finder patterns (solid black 7x7 squares framed by a white ring and black outer ring) in the top-left, top-right and bottom-left corners. Pure black #000000 and white #FFFFFF, no border, no rounded corners, no shadow, sharp edges, unboxed, not cropped, not overlaid, flat lighting, no tint. No charts, tables, icons or other logos besides the wordmark.

Slide 30

Thank you

CONFIDENTIAL

30

Visuals: One logo: a bold two-part mark in off-white #FAFAF8, a stylised 'A' (left, a solid triangular letter with a horizontal crossbar cut-out) and a slanted vertical bar to its right, reading as 'AI'. The mark is flat, with no shadow, gradient or outline. No photographs, charts or other graphics.

Download this deck

The hosted PDF of every library deck is free for readers of the SlideDeck Co newsletter. Join with your email and the viewer and download appear here.

Building a deck of your own? The SlideDeck Co Standard Deck is a slide library for business presentations: reviews, updates and plans.

More Education and training decks

Deck reviews in your inbox.

See what works in real company decks, what we’d change, and get the latest SDC releases.

Expanded slide preview