library(ellmer)
library(jsonlite)
source("tools.R")
task <- "Find the CSV file in this folder, make a plot from it and save the plot as a PNG file."42 Experiments with your agent
Having built the agent yourself, you now have a loop you can play with. run_agent() returns steps, tool_calls and completed. You can also run controlled experiments like “which model is better at this?” and “does a more specific prompt help?”. This is what we’ll do here
The method is the same one you’d use for any experiment: change one thing at a time, reset the state between runs, and record the result.
This chapter carries on from the previous one — you’ll need tools.R in your project directory, plus system_prompt built and run_agent() defined as they were there.
42.1 Compare LLMs
Let’s compare results from a few different LLMs. Make sure you move the PNGs to another folder first so each run is a fair test — if the plot is already there, a clever agent may notice and skip the work.
Now run the same task with one model. Watch the steps as they print: that is where you see the differences between models, not just in the summary row.
# Check https://openrouter.ai/models for current model names
chat <- chat_openrouter(model = "anthropic/claude-sonnet-5.5", system_prompt = system_prompt)
run_claude <- run_agent(chat, task, max_steps = 15)Then the same thing again with a different model. We’ll use Qwen 3.8, an open weight model (make sure you move the other png first):
chat <- chat_openrouter(model = "qwen/qwen3.8-27b", system_prompt = system_prompt)
run_qwen <- run_agent(chat, task, max_steps = 15)rbind(claude = run_claude, qwen = run_qwen)Did both models complete the task? Now compare the images.
The summary row only tells you whether and how expensively. The interesting differences are usually in the transcript: a weaker model is more likely to forget the tool-call format, re-read the same file twice, or declare task_complete before checking that the PNG was actually written. Those are the same types of failures you’ll see with a commercial agent.
Smaller models often fail at the format rather than the task. They understand what to do but write the <tool_call> block wrong. If a cheap model keeps stalling, look at whether it’s the reasoning or the JSON that broke before you write it off.
42.2 Specific vs vague prompts
Same idea, but now the model stays fixed and the task wording changes. Run the vague one first:
vague <- "Make a plot of distance to logging and branching coral cover from the data and save it."
chat <- chat_openrouter(model = model, system_prompt = system_prompt)
run_vague <- run_agent(chat, vague, max_steps = 15)Then the specific one:
specific <- paste(
"Read data/fish-coral-cover-sites.csv.",
"Use ggplot2 to make a scatter plot with dist_to_logging_km on the x axis and cb_cover on the y axis.",
"Save it as coral-plot.png."
)
chat <- chat_openrouter(model = model, system_prompt = system_prompt)
run_specific <- run_agent(chat, specific, max_steps = 15)rbind(vague = run_vague, specific = run_specific)The vague prompt usually costs more steps, because the agent has to go looking for the data and decide what a plot of it should even be. The specific prompt spends your words instead of the agent’s steps.
This is the prompting advice from Section 2 with a number attached to it. It’s also a reminder that the steps an agent spends exploring aren’t wasted when you genuinely don’t know what’s in the data — they’re only wasted when you did know and didn’t say.
42.3 Registering tools with ellmer instead
Everything so far has been hand-rolled on purpose: we wrote the tool descriptions into the system prompt as text, asked the model to wrap calls in <tool_call> tags, and parsed them back out with a regex. That’s the clearest way to see what an agent is, but it isn’t how you’d write one for real.
Modern LLM APIs accept tool definitions as a structured field in the request, separate from the system prompt, and return tool calls as structured data rather than text you have to parse. ellmer exposes this through tool() and register_tool(). Here’s list_files done that way:
list_files_tool <- tool(
list_files,
"List all files in a directory (searches sub-directories too).",
path = type_string("Directory to list. Use '.' for the working directory.")
)
chat <- chat_openrouter(
model = model,
system_prompt = "You are an agent working inside an R session."
)
chat$register_tool(list_files_tool)
chat$chat("What CSV files are in this folder?")Compare that to what it replaces. The tool() call carries the same three things our definition did — the function, its description, and a typed description of each argument — but written once, next to the function, instead of duplicated between tool_definitions and call_tool(). There’s no <tool_call> syntax to explain in the system prompt, because the format is handled by the API.
More importantly, chat$chat() is the loop. Internally ellmer sends your prompt, checks whether the response is a tool call, executes it, sends the result back as part of the conversation, and repeats — until the model replies with plain text instead of a tool call. The while loop, the regex, the switch() dispatch and the is.na(json_text) check all disappear into that one line.
So why build it by hand at all? Because the loop is where the control lives. Our version had an approval prompt, a max_steps ceiling, and a results row for experiments — the three things this chapter depends on. When you hand the loop to ellmer, you get correctness and brevity, and you give up the seams you’d hook those things into. That’s the right trade for production code and the wrong one for understanding, which is why we did it the slow way first.
Structured tool calling is also more reliable. Our regex breaks if the model emits two <tool_call> blocks, writes malformed JSON, or wraps the block in a markdown fence — failure modes you’ll have seen if you ran the model comparison above with a smaller model. The API-level version can’t make those mistakes.
ImportantChallenge
Register a second tool with ellmer — read_file from tools.R is a good candidate — and ask the agent to find and summarise the CSV. Then run the same task through the hand-rolled run_agent(). The outcome should be the same; note which version you’d rather debug, and what you lost in the switch.