2 Introduction
Slides from the workshop presentation (on a google drive)
LLMs were created to write text. But it soon became apparent that they excel at writing programming code in many different languages.
Since then AI companies have been optimising their training and development for coding and logic.
There are a series of standardised tests that are used to compare quality of LLMs. Common evaluation tests are the SWE benchmark which looks at the ability of LLMs to autonomously create bug fixes. Frontier models can solve >70% of tasks on this benchmark.
Their progress on math and logic is a bit more controversial. It seems like some of the math benchmarks (like AIME annual tests for top 5% highschool students) are saturated as LLMs are scoring close to 100% on these tests.. So newer tests of unsolved maths problems are being developed.
However, others are finding that the ability of LLMs on math and logic are overstated, perhaps because the LLMs have been trained on the questions and the answers. It’s also clear that AI companies have a strong financial incentive to find ways (real and otherwise) of improving on the benchmarks. At the moment there is tough competition to be ‘industry leaders’ and grab market share with impressive results on benchmarks.
Either way, it does seem that the current areas of progress are programming, math and logic.
Others have written about the broader promise of these tools for coding, teaching and inclusion in ecology and evolution (Cooper et al. 2024). But evaluations on statistics and the R software specifically are less common. So my collaborators and I ran our own. Two of those studies are now published, and the advice in this book is built on what we found.
2.1 What we found when we tested LLMs on ecological statistics
The first study asked how much the prompt matters: Brown & Spillias (2026) “Prompting large language models for quality ecological statistics” (Brown and Spillias 2026) in Methods in Ecology and Evolution (see also the summary blog post). We ran repeatable evaluations in R, replicating each prompt 10 times across multiple LLMs. The short version: LLMs can help you do better statistics, but only if you ask well.
A few results stood out.
Specificity matters enormously for test selection. We compared four prompts for choosing a statistical test for an ecological dataset, ranging from “How do I test the relationship between two continuous variables?” to a detailed prompt specifying variable types, sample size, and study design. The generic prompt never suggested count models (the appropriate family for fish abundance data). The detailed prompt guaranteed them, regardless of which LLM we used.
Detailed prompts make agent-generated code more consistent. We asked the GitHub Copilot agent to write an entire analysis workflow from two different prompts — one brief, one detailed. We then ran multivariate ordination on the resulting code to measure how similar the 10 replicates were to each other. The detailed prompt produced much tighter clusters — the agent kept using the same functions, variable names, and structure. Inconsistent code is harder to review, which matters if you’re trying to catch statistical errors.
Being specific overcomes weaker models. When we tasked LLMs with writing R code to calculate a distance matrix, the detailed prompt got the right answer 90–100% of the time across all models. The brief prompt mostly failed — except for GPT-5 Codex, which guessed correctly 9/10 times. Good prompts effectively compensated for using a smaller, cheaper model.
These findings match the earlier published evaluations. An evaluation of several models, including GPT-4 as the most up-to-date model (Zhu et al. 2024), found accuracy at suggesting the correct statistical test of between 8% and 90%. In general LLMs were good at choosing descriptive statistics (accuracy of up to 90% for GPT-4). Whereas when choosing inferential tests accuracy was much less impressive - GPT-4 scored between 20% and 43% accuracy on questions for which a contingency table was the correct answer. The results also indicate the improvements that can be gained through better prompts (i.e. doubling in accuracy for GPT 4).
2.2 What happens when you let the agent run the whole analysis
Our second study went further and handed the entire modelling job to an agent: Brown et al. (2026) “Automating Ecological and Fisheries Modelling With Agentic AI” (Brown et al. 2026) in Fish and Fisheries, which is open access (and there’s a summary blog post too).
Agentic AI tools can write and run code, fix their own errors, and produce a formatted report with figures. We wanted to know whether that translates into reliable ecological modelling, so we ran a test: three fisheries tasks, four AI models, ten independent runs each, scored against a rubric covering accuracy, code quality and report quality.
We found agents can be genuinely useful, but only if you know how to use them well and only if you know enough about the analysis to catch what they miss.
Two findings shaped how this book is written. First, agents default to the most common method in their training data, which may not be appropriate for your question — and even when you tell them otherwise they may not comply.
Second, agents don’t check the things you didn’t ask them to check. None of our agents checked for collinearity between predictors in the GLM, even though it’s standard practice. We deliberately left it out of the specification to see if they’d do it anyway. The GLMs ran fine and the results looked coherent, but there was in fact strong collinearity between the predictors.
The error type that concerns me most is professionally formatted output containing logical errors. A coding syntax error is immediately obvious. A methodological shortcut embedded in otherwise clean output may be invisible unless you already know what the answer should look like. Inexperienced researchers will use these tools to produce analyses they cannot evaluate, and experienced researchers may get overconfident and not check results thoroughly enough.
2.3 What this means for how you work
The lesson is two-fold. Just because LLMs excel at some tasks doesn’t mean they will excel at others. Second, good prompting strategies pay off.
For us in the niche R world there is also another lesson. The LLMs should be good at helping us implement analyses (ie write the R code). However, they are less reliable as statisticians who can guide us on the scientific question of what type of analysis to do.
I have had many experiences where the most powerful models of the day make trivial logic mistakes (e.g. Claude Fable, the current most powerful at writing). They can scaffold a complex workflow of multiple interacting steps, that handles data wrangling, analysis and plotting.
But more often than not they get some core detail wrong. Such as the equation in the heart of your generalized linear model.
What this tells me is that LLMs are great at programming, but not so crash hot at the scientific logic and reasoning.
As a scientist and competent programmer, this is great news. I can offload a lot of the tedious and time- consuming set-up work required in scientific computing and focus on the parts that require real scientific judgement.
Statistical expertise is still required. You need to know enough to evaluate whether the LLM’s suggestions are appropriate, check that code is scientifically valid (not just syntactically correct), and understand what the results mean. Novices who lack that background are more likely to write poor prompts and less likely to catch bad advice.
In this book we’ll walk through many of the ways you can use LLMs to help you do statistics, data analysis and R coding (and a bit of python too), working through short self-paced modules built around ecological datasets. The workflow we recommend in the prompting paper breaks LLM-assisted analysis into three stages, with separate prompts for each — choosing the statistical approach, planning the implementation, then writing the code — and that structure runs through Section 3 of this book.
One of the biggest first hurdles is choosing among the multitude of AI assistant software and R package options. There are so many it is bewildering. In the next module I’ll give you a few of the easiest set-up options available.