12  Should I use AI to help with my modelling

The most common question I get asked these days is ‘how are you using AI?’.

These people usually mean generative AI tools like Claude Code that write text. I rarely use other types of AI, like image classification (defintions are important after all, Chapter 3)

I’ve written another guide on the topic of how to use AI: AI assistants for Scientific Coding.

Here I’ll look at the high level of how and why.

Let’s break the question of ‘how to use AI’ down into two parts: planning the model (the science bit), and writing the code (the implementation bit).

In fact, we wrote a whole paper on how best to use AI in modelling workflows. If you want to use AI that is. If you don’t want to that is fine.

12.1 Should I use AI for planning the model?

NO

At the most you can use an AI chatbot as a kind of tutor. Examples would be check your understanding of terms, find new ways to solve equations you are uncomfortable with, cross search for similar concepts in different disciplines.

In my experience that answer will be sufficient for more than half of my students. These students value their intellectual development and are insulted to think that a learning opportunity could be outsourced to an AI (rightly so). If this is you, you can skip to the next question.

The remaining students either (A) feel compelled to use AI because they are concerned about being left behind in the race to ‘AI’ everything or (B) see themselves at the bleeding edge of a new generation of AI-enabled humanity.

(A third category (C) exists, these are students who are trying to find the easiest pathway to an end-product like good grades or a high impact paper or whatever. I don’t take on these students for research projects).

If you fall into the ‘compelled to use AI’ or ‘generation AI’ category, then here’s my explanation for why I’m saying you should not be using AI to plan your model.

You are a student, you are here to learn. In fact, as scientists we are all students of nature. If you use AI to create your model you won’t learn anything. If it isn’t hard, you aren’t learning. That’s why I say its ok to use AI as a tutor to learn new ways to do your work, but not ok to use it to create your whole model.

One of the challenges of appropriate AI use is keeping honest to your commitment to learn. These AI chatbots have been described as ‘slot machines’, you put a relatively small amount of effort in (a prompt) and sometimes it pays off with a big gain. But oftentimes it does not. Those pointless payouts are AI slop. They are not useful science and, even worse, pulling the lever on the slot machine hasn’t contributed to developing your intellect one iota.

Trust me, I’ve tried and I’ve learned how much time needs to be spent wading through slop to find gold. I now realize that its easier to find gold by just putting the hard work in yourself at the start.

The other hard part is ignoring the hype on AI. You need to go against the flow to some extent to not overuse AI these days. Be wary of hypers of AI, do they have real quantitative evidence that they are producing anything of value? Is their success context dependent?

AI to automate work works well in contexts where we have a clear goal and can evaluate the outcomes. For instance, software development, where we want a software that does a particular thing. But science is discovery, the whole point of it is that we don’t know the right answer when we start. In fact, many research outcomes take decades of work to verify! Therefore, the cases where AI can do the discovery for us are much more limited, and are usually about outcomes that involve building things.

12.2 Should I use AI to write the code?

Good news for AI lovers here: If you’ve done the hard work making a solid plan that is backed up by prior research and your own understanding, then your AI tools can make huge strides in writing the code.

AI is great at writing code, especially when you give it a very clear specification for what to do. By a ‘specification’ I mean a really excellent plan.

12.2.1 If you are learning to code

I advocate you make judicious use of AI to write the code. (now I’m borrowing text from that blog here).

Code completion tools that complete the line of code or write code based on comments are a good place to start. These tools let you build the model and your understanding step-by-step.

Plugging your IDE (integrated development environment, like VScode or Rstudio) into a chatbot (like Claude Code UI or Posit Assistant) is great to get help on your project. The chatbot will be able to navigate your project (just be aware of data privacy issues!) to orientate itself and provide more relevant advice.

12.2.2 If you are a coding expert

You can use an AI agent like Claude Code to write your code, start to finish.

Then, there are some important principles to follow when using coding agents. This has been called agentic engineering. I intend to write more about agentic engineering for researchers at some point. But for now a few quick tips.

1. Clear plan with examples.

Write down a very clear specification for the agent. I studied ‘how clear’ in a multitude of examples (ecology and fisheries) and clear specification was key.

Note the specification can be so clear and so long that many coding experts find it easier to just write the code themselves with only minimal AI tools like line completion. Also expect the agent to need multiple iterations and possibly fresh starts with revised instructions.

The plan (specification) can be human language, code or both. For instance, I get the best results when I ask the agent to emulate an existing workflow, then I give it the code for that workflow. Then it can do spectacular things to wrangle new datastructures to fit an existing modelling workflow.

2. Tests of code logic

Ask your agent to write unit tests to make sure the analyses are rational. One way to do this is ask for ‘red/green test driven development’. This means each test needs to fail before the agent fixes the code so it passes. Think about the types of tests you regularly use to check your own work, like: did the join replicate rows? did the transform change the range of values as expected, does x==y when we fix parameter a. Then get the agent to write these tests into the code.

3. Evaluate outcomes

This is the tricky part. We need to find a way to test the agent’s work so we know its done the right analysis.

The problem is, if you are doing novel science then you don’t know the answer in advance. Lots of plots of all the data and intermediate steps are essential, so you can check the work. If you can devise quantitative ways to check answers that is even better. For instance, check the model’s prediction against a known analytical solution, feed the agent’s workflow simulated data with a known truth, or put bounds on the possible range of outcomes.

12.3 Summary: Two key questions to ask yourself if you’re using AI

If you plan on using AI to help with your modelling, then there’s two questions you should check-in with yourself about frequently.

1. Am I learning about the topic I’m studying?

It’s easy to use AI and not learn anything about the topic you are studying. So check in with yourself regularly to make sure you are learning more than just how to prompt an AI.

One of my recent students eschewed AI coding tools as he sought to teach himself deep learning (he did use a chatbot, just for help and advice like a book or tutorial). Progress was slow at first and it took weeks before he got the algorithms to fit the data. But when he did we rejoiced. The sense of satisifcation for student and supervisor was that much deeper for knowing that he did it on his own.

2. Am I sure the modelling is correct?

It can be hard to know if the AI has done the modelling correctly without checking its work.

Mistakes happen, for humans and for AI. The best models today can take on larger and larger tasks, but in my experience they still consistently make small but logically fatal mistakes. One recent example, Claude Code perfectly replicated a workflow for a new datastructure, but made a silly mistake and double corrected a fraction to a percentage, multiplying all the results by a factor of 100.

So if you are using agentic tools, then you need to be very good at finding ways to test the AI’s solutions. Even then, I often find it would have been faster to have just built the model the old fashioned way.