The Claude Opus 5 API: Build and Benchmark a Coding Agent in Python

Sending code to a model is the easy part. The work is everything around it: deciding which files it may read, executing the tool calls it asks for, knowing when it should stop, and choosing how much reasoning to pay for.

Claude Opus 5 makes that last decision explicit. The effort parameter has five settings, and Anthropic says spending more of it pays off on exactly the work a coding agent does. That claim is easy to repeat and harder to check, so I wanted a narrow answer. On one real bug, with tests that either pass or fail, does turning effort up produce a better repair?

To find out, we'll build a small bug-fixing agent and run the same task at every effort level. Along the way we'll cover:

• Calling claude-opus-5 and parsing a response that leads with a thinking block

• Controlling reasoning depth with effort and sizing max_tokens around it

• Running a multi-turn tool loop that survives thinking blocks and tool errors

• Returning a validated Pydantic report instead of prose

• Measuring success, tokens, latency, and cost across all five effort levels

• Cutting repeated input cost with prompt caching inside a single run

I'll give away the ending: on this task, the cheapest setting fixed the bug on every run, and nothing above it did better. That result is specific to this repository, and the more useful part is the rig that produced it.

Setting Up Our Claude Opus 5 Project

The agent takes a bug report and controlled access to a small Python repository, then works the problem: list files, read the relevant ones, run the suite, patch, rerun, and report.

Claude never touches the filesystem. It emits a tool request with a name and JSON arguments, and our Python code decides whether to run it and what to send back. That boundary is the whole security model, worth remembering whenever the agent looks like it is acting on its own.

Bug report flows through controlled tools. Image by Author.

You need Python 3.10 or newer, an API key with access to claude-opus-5, Git for resetting the sample repository, and enough pytest familiarity to read a failure. The finished project is on GitHub if you would rather read it whole; the sections below build it a piece at a time.

git clone https://github.com/KhalidAbdelaty/claude-opus-5-api-tutorial.git
cd claude-opus-5-api-tutorial
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Copy .env.example to .env and put your key in it. The key belongs on the server side of anything you build from this, never in a request body and never in a log line.

# agent/config.py
MODEL = "claude-opus-5"

PRICE_INPUT = 5.00
PRICE_OUTPUT = 25.00
PRICE_CACHE_WRITE_5M = 6.25
PRICE_CACHE_READ = 0.50

MAX_TOKENS = 32_000
MAX_ITERATIONS = 10

Those four prices matter later. Charging everything at the base input rate understates cache writes by a quarter and overstates cache reads tenfold, the kind of error that makes a benchmark look better than it was.

Creating a Reproducible Bug-Fixing Task

A fair comparison needs a bug that fails the same way every time, so I wrote the fixture before the agent. The sample repository is a billing service with a shared date helper, a subscriptions module, and an invoices module. The helper documents one behavior and implements another.

# billing/timeutils.py
def days_between(start: datetime, end: datetime) -> int:
    """Return the number of days from start to end, counting a partial day as a full day.

    Billing treats any remaining fraction of a day as a whole day, so a customer
    with four hours left has one day remaining, not zero.
    """
    delta = end - start
    return delta.days

timedelta.days truncates, so ten hours of remaining trial reports as zero days. Two modules call this helper, so one defect surfaces as two unrelated failures: test_trial_active_on_final_day locks customers out early, and test_prorated_credit_counts_partial_day shorts a refund by a day.

The bug report mentions only the first symptom, as a support ticket would. There is also a trap: patching is_trial_active to accept zero makes the reported test pass and leaves the invoice test broken, so a shallow fix is detectable. The baseline is two failures and four passes, committed to Git so git checkout -- . restores it between runs.

Making Your First Claude Opus 5 API Call

Before any tools, confirm authentication and response parsing work. The Messages API needs a model string, a required max_tokens, and a messages array.

from anthropic import Anthropic
from dotenv import load_dotenv

load_dotenv()
client = Anthropic()

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "A pytest assertion fails with: assert 0 == 1. What should I check first?"}],
)

print([block.type for block in response.content])
text = next((b.text for b in response.content if b.type == "text"), "")

That first print is the point of the exercise. It returns ['thinking', 'text'], because thinking is on by default and the thinking block arrives first.

First call returns thinking then text. Image by Author.

Any code reaching for response.content[0].text will raise an AttributeError on Opus 5. Filter by block type instead. While you are at it, drop temperature, top_p, and top_k from any request you are porting, because non-default values return a 400 on this model.

How Thinking and the Effort Parameter Work in Claude Opus 5

Thinking and effort are two controls that get conflated. Thinking is whether the model reasons before answering, which the first call already showed it doing. Effort is how much reasoning it does, across every output token, tool calls included. The subsections below cover the ladder, the token ceiling it hits, and the one combination the API rejects.

Using the five effort levels

Effort is set inside output_config, and no beta header is needed.

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=2048,
    output_config={"effort": "xhigh"},  # low | medium | high | xhigh | max
    messages=[{"role": "user", "content": bug_report}],
)

The API default is high, and passing "high" explicitly is the same as omitting it. Anthropic suggests starting at xhigh for agentic coding and notes that low and medium are stronger on Opus 5 than on earlier Opus models. Their guidance also warns that max can overthink simpler tasks, which turned out to be the most interesting thing I measured.

Choosing a max_tokens value

As I mentioned earlier, max_tokens covers the reasoning as well as the answer, so a value tuned for a model without thinking will truncate Opus 5 mid-task. There is also a constraint that catches people out.

client.messages.create(model="claude-opus-5", max_tokens=32000, messages=[...])
# ValueError: Streaming is required for operations that may take longer than 10 minutes.

The SDKs refuse a non-streaming request above 21,333 max_tokens. That is client-side validation rather than an API rule, so the fix is to stream: the loop calls client.messages.stream() and reads get_final_message(), and the same 32,000 goes through. I settled there because it leaves room for the higher effort levels.

When thinking can be disabled

You can switch thinking off, but only at high effort or below. Combining it with xhigh or max is rejected outright.

effort=high  accepted, blocks=['text']
effort=xhigh HTTP 400: output_config.effort 'xhigh' is not supported when thinking
             is disabled on this model. Use effort 'high' or below, or enable thinking.

Disabled thinking rejected above high effort. Image by Author.

Notice the block list on the accepted call: ['text'] alone, thinking block gone. Lowering effort is the safer cost control anyway, so I left thinking on throughout.

Defining Safe Repository Tools

Tools turn a model into an agent, and they are also where an agent stops being safe. Each definition is a name, a description, and a JSON input schema, and every one costs input tokens on every request. I kept the set to five, grouped below by how much damage they can do.

Reading and searching the repository

list_files and search_code let the model find the relevant file without pasting the repository into the prompt. read_file pulls one file by a repository-relative path, and that is where the safety checks live.

def _resolve(relative_path: str) -> Path:
    """Resolve a repository-relative path, refusing anything outside the repo."""
    candidate = Path(relative_path)
    if candidate.is_absolute():
        raise ToolError("Absolute paths are not allowed.")

    resolved = (REPO_ROOT / candidate).resolve()
    if resolved != REPO_ROOT.resolve() and REPO_ROOT.resolve() not in resolved.parents:
        raise ToolError("Path escapes the repository.")
    return resolved

Resolve before you compare, or ../../.ssh/id_rsa walks straight out of your sandbox. Returned files are capped at 8,000 characters, because every tool result is resent on every later request and you pay for it each time.

Running tests

run_tests takes no arguments. The command is fixed in configuration, so the model has no way to ask for an arbitrary shell command.

def run_tests() -> str:
    completed = subprocess.run(
        [sys.executable, "-m", "pytest", "tests", "-q", "--no-header", "--tb=short"],
        cwd=REPO_ROOT, capture_output=True, text=True, timeout=120,
    )
    output = (completed.stdout + completed.stderr).strip()
    return f"exit_code: {completed.returncode}\n{output[:3000]}"

An allowlist of one is the strongest allowlist available. If you need more commands later, add them as separate named tools, not as a command argument.

Applying a patch

For writes I used exact-string replacement rather than a unified diff, because diffs fail on whitespace in ways that waste a turn. The tool rejects non-Python files, caps the patch size, and refuses ambiguity.

occurrences = source.count(old_text)
if occurrences == 0:
    raise ToolError("old_text was not found in the file. Read the file and retry.")
if occurrences > 1:
    raise ToolError(f"old_text appears {occurrences} times. Include more context.")

Those errors go back to Claude as tool results rather than crashing the loop, and the model reads them and retries with more context.

Building the Tool-Use Agent Loop

The loop is short once the pieces are in place. Send the task with the tool definitions, inspect the returned blocks, execute what was requested, send the results back, and repeat until the model stops asking.

for iteration in range(1, max_iterations + 1):
    with client.messages.stream(
        model=MODEL,
        max_tokens=32_000,
        system=SYSTEM_PROMPT,
        tools=TOOLS,
        output_config={"effort": effort},
        output_format=RepairReport,
        cache_control={"type": "ephemeral"},
        messages=messages,
    ) as stream:
        response = stream.get_final_message()

    usage.add(response.usage)
    messages.append({"role": "assistant", "content": response.content})

    if response.stop_reason != "tool_use":
        break

    tool_results = []
    for block in response.content:
        if block.type == "tool_use":
            output, is_error = execute(block.name, dict(block.input or {}))
            tool_results.append({
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": output,
                "is_error": is_error,
            })
    messages.append({"role": "user", "content": tool_results})

Three lines there are load-bearing. Appending response.content whole, rather than just the blocks you care about, keeps thinking and redacted_thinking intact; filter them out and the model restarts its reasoning after every tool result. tool_result blocks must come first in the user message, because text before them returns a 400. And is_error tells Claude a tool failed, which beats a silent empty string.

The system prompt stays fixed across every run so the comparison means something. It tells the model to investigate before editing, find the root cause rather than the reported symptom, make the smallest change that works, and treat running the suite as part of the task. What it omits is "double-check your work." Opus 5 verifies itself, and Anthropic's guidance is to remove those instructions when migrating, because they add latency and tokens without improving the patch.

Agent inspects, patches, then reruns tests. Image by Author.

Returning a Structured Bug-Fixing Report

Prose is fine for one run. For a benchmark that has to compare many, I want a typed object, so the loop passes a Pydantic model as output_format and the SDK turns it into a JSON schema with constrained decoding.

class RepairReport(BaseModel):
    status: Literal["fixed", "partial", "not_fixed"]
    root_cause: str
    files_inspected: list[str]
    files_changed: list[str]
    fix_summary: str
    tests_before: str
    tests_after: str
    remaining_risks: list[str]
    confidence: Literal["low", "medium", "high"]

    @field_validator("status", "confidence", mode="before")
    @classmethod
    def _normalize_case(cls, value):
        """Enum capitalization is not guaranteed, so fold it before validating."""
        return value.strip().lower() if isinstance(value, str) else value

The validator earns its place. Structured outputs do not guarantee enum capitalization, so a schema expecting "high" can receive "High" with no error and no special stop reason, and strict Literal validation would reject a response that was basically correct.

Keep latency and cost out of this schema. The model reports what it concluded; the runner measures what happened by rerunning pytest and reading git diff. I was glad of that separation later, for reasons the benchmark makes obvious.

Benchmarking Claude Opus 5 Effort Levels

This is the experiment the tutorial exists for: the same bug, the same prompt, the same tools, at low, medium, high, xhigh, and max. Output is not deterministic, and sampling parameters are rejected on this model, so repeating each level is the only control I have.

Keeping the experiment fair

Each run starts from that clean baseline commit, then measures the suite before and after. The bug report, system prompt, tool definitions, iteration cap, and max_tokens are identical everywhere, and only the effort string changes. I ran each level three times.

A budget guard wraps the whole thing, summing the real four-rate cost after every request and stopping at a ceiling. Agent loops are an easy way to overspend, and a stop condition you control beats discovering the number later.

What the numbers showed

Here are the averages across three runs per level, measured on July 26, 2026.

Every level that engaged with the repository found the same root cause in the shared date helper and fixed it with about four lines in one file. Not one took the bait and patched the trial check instead. The whole fifteen-run matrix cost $0.92.

Cost climbs, the outcome does not. Image by Author.

Interpreting the results

The honest reading is that this task was not hard enough to separate the top of the ladder from the bottom. Going from low to xhigh doubled the cost and added eleven seconds for an identical outcome. Shipping this, I would run low and spend the difference on test coverage.

That max row needs an explanation. One of its three runs returned a schema-valid report on the first turn without calling a single tool, with root_cause set to "b0" and an empty fix_summary. Pydantic accepted it because the shape was right. The tests were untouched. Excluding it, the other two max attempts fixed the bug at 12.5 tool calls and $0.1112 each, roughly three times the cost of low.

I find that failure more instructive than the table. A schema guarantees structure, not substance, and an application trusting report.status would have logged a "partial" success on a run that did nothing. The runner caught it because it ran a pytest instead of believing the model. Whatever you take from this article, take that. And since your task is not mine, treat all of this as a method rather than a result.

Reducing Repeated Costs With Prompt Caching

Caching is where the loop gets cheap, and it works within a run rather than across effort levels. Changing the effort value invalidates cached prefixes, so each level pays its own cache-creation cost and no warm cache is carried over. Inside one run it is different, because the system prompt and tool definitions never change while the history grows.

A top-level cache_control field turns on automatic caching, which puts the breakpoint at the last cacheable block and moves it forward as the conversation grows.

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=2048,
    system=SYSTEM_PROMPT,
    tools=TOOLS,
    cache_control={"type": "ephemeral"},
    messages=messages,
)

Two consecutive turns show the effect clearly:

turn 1: uncached_input=2  cache_write=1231  cache_read=0     output=26
turn 2: uncached_input=2  cache_write=111   cache_read=1231  output=108

Cached prefix read back next turn. Image by Author.

The stable prefix is 1,231 tokens, comfortably over the cache minimum I listed earlier. On turn two it is read back at a tenth of the input rate while only the new tool result is written. Across a full xhigh run, roughly 32,600 of about 36,200 input tokens came from cache. Caching does nothing for output tokens.

Adding Tools Mid-Conversation

Here is the pattern I found most useful. Start the agent with read-only tools, then grant write access only after it has diagnosed the problem. Done naively, adding a tool means editing the tools array, which sits at the front of the cached prefix and invalidates the whole conversation's cache.

Mid-conversation tool changes solve that. Declare every tool up front so the array never changes, then use tool_addition and tool_removal blocks to offer or withdraw individual tools from a point onward. The blocks travel inside a role: "system" message, and a system message cannot be the first entry in messages. Neither constraint is where you would look for it, and I found both the way you are imagining.

withhold_patch = {
    "role": "system",
    "content": [{"type": "tool_removal",
                 "tool": {"type": "tool_reference", "name": "apply_patch"}}],
}
messages = [{"role": "user", "content": task_prompt}, withhold_patch]

Swap tool_removal for tool_addition later and the tool comes back. In my run the model spent five turns on read-only investigation, stopped with its diagnosis, and only then received apply_patch, which it used on turns six and seven before the suite passed. The cache held throughout, with 25,557 tokens read back. Write access arrived after the evidence was in, which is the real argument for the pattern. Get the fixed-tool loop working first, then add this.

Calculating Token Usage and API Cost Across an Agent Loop

An agentic task is many requests, so cost lives in the sum rather than the final response. Four numbers come back on every response, and each bills at a different rate.

def cost_usd(self) -> float:
    return (
        self.input_tokens * 5.00
        + self.output_tokens * 25.00
        + self.cache_creation_input_tokens * 6.25
        + self.cache_read_input_tokens * 0.50
    ) / 1_000_000

Total input is the sum of all three input fields, because input_tokens alone counts only what falls after the last cache breakpoint, which is a common way to underreport. Output tokens include thinking and tool calls as well as visible text, and at five times the input rate they dominate every run above.

One thing to watch as you scale up: prior turns' thinking blocks stay in context and get billed again on every subsequent turn, so long loops cost more than the tool results suggest. For offline evaluation, the Batch API halves both rates. Fast mode goes the other way at $10 and $50 per million tokens, as an access-gated research preview.

There is also a way to cap the whole loop rather than one request. Task budgets hand the model an advisory allowance and a countdown to pace against, in beta behind task-budgets-2026-03-13, with a floor of 20,000 tokens. I had a section written about them and cut it, because I measured no difference here and a non-result does not need three paragraphs. Expect the same whenever the work never comes near the budget.

Migrating From Claude Opus 4.8 to Opus 5

Most existing Claude code runs once you change the model string. The failures cluster in a few places, and we hit every one of them while building the agent, so treat this as the checklist rather than a fresh explanation. Anthropic's migration guide carries the full version.

• response.content[0] is usually a thinking block now, not text

• A loop that rebuilds the assistant message from selected blocks drops thinking and redacted_thinking

• max_tokens tuned for a non-thinking model will truncate, and anything above 21,333 has to stream

• Disabling thinking returns a 400 at xhigh or max effort

• Non-default temperature, top_p, and top_k are rejected

• Verification instructions carried over from older prompts make Opus 5 over-verify

• Priority Tier and the web fetch tool are not available on Opus 5

• Effort deserves a fresh sweep rather than the value you tuned for the last model

That last one is the point I would argue hardest for, and this article is the argument. My tuned setting turned out to be the cheapest on the ladder.

Production Considerations

Two things change before this ships. Agent-generated code is untrusted code, so the tool layer belongs in an isolated environment with CPU, memory, and time limits, and a generated patch belongs in front of a human before it merges.

The second is less obvious. Opus 5 runs safety classifiers, and an agent reading security-adjacent code sits near the "cyber" boundary, but a refusal arrives as an HTTP 200 with stop_reason: "refusal", so monitoring built on 5xx rates never sees one. Instrument refusals separately, branch on stop_reason rather than the stop_details fields underneath it, which can be null, and switch fallbacks off during benchmarks so you know which model answered.

When Should You Use Claude Opus 5?

Use Opus 5 when a wrong answer costs more than the request does. Multi-file debugging, root-cause analysis, repository-wide review, long tool loops, and migration planning all fit, and the effort ladder lets you run routine cases cheaply and escalate the hard ones.

Skip it for formatting, classification, and high-volume low-complexity work, where Claude Sonnet 5 is the better trade. Skip it too for anything you cannot validate, since all the value here came from a test suite that answers honestly. Paying double for Fable 5 only earns its keep on the work I flagged at the top.

The routing advice I would actually give is to measure task difficulty and let that pick the effort level, rather than sending everything to the top of the ladder because it sounds safer. It is not safer. As the benchmark showed, it was the expensive way to get the same patch, and once it produced a report about nothing.

The Streamlit app in the repo lets you run this yourself: pick an effort level, watch the reasoning and the tool calls arrive live, and read the tokens, cost, and latency off the same run. It resets the repository the same way the benchmark did, so the table it builds is your measurement rather than mine. The screen recording below shows the workflow.