9 The modelling workflow
Building a model never goes smoothly. It is not a tidy linear pipeline of write code → debug → run → make plots → done. It’s a set of feedback loops. Running your simulations is often how you discover the bugs in your code: you look at a result and try to interpret it, then realise “that can’t be right,” and go back.
Papers don’t usually capture this. A paper reports the data, the equations, the assumptions and their basis, and then the simulations and results, but not the messy sequence of steps that actually got you there.
Recently, the Bayesian statistics community has started writing down workflows: sensible ordered sequences of steps (Gelman et al. 2020). These aren’t derived from experiments comparing workflows; they’ve accumulated through experience and culture, as people learned which checks catch which logical traps and encoded that. Other workflows could work equally well, but a good one captures a lot of the mistakes the literature has already made so that you can avoid them.
A generic workflow, stripped down so it applies to almost any kind of modelling, looks roughly like this:
1. Build the model. Start simple, as above.
2. Run simulations to explore how it responds. Here the mindset is important: try to break your own model. You are looking for two kinds of problem:
- Coding bugs: outright errors. Syntax errors announce themselves in red text in your terminal. But far more dangerous are the logical bugs where the code runs fine but does the wrong thing inside. AI-assisted coding is especially good at producing this second kind, oftentimes because you weren’t clear on what you wanted in the model.
- Logical / structural problems in the model itself: the code is correct but the model has an error that means you shouldn’t trust the results.
3. Do a prior predictive check. Borrowed from Bayesian practice: run the model with no data and minimal prior assumptions for parameters, purely from your prior knowledge and structure, and see whether the outcomes are even sensible. This surfaces problems like non-identifiability, or a model that makes such extreme predictions that it needs further constraints before it produces plausible output.
4. Compare results to your expectations. Once the model runs, ask whether the output looks like what you’d have predicted before you built it. Where it doesn’t, work out why: sometimes that reveals a bug, and sometimes it reveals that your intuition was wrong, which is itself a finding (compare the consequence-modelling discussion in the previous chapter). Either way, don’t skip this step just because the model ran without errors — running is not the same as being right.
5. Compare results to other studies. Where similar models or data exist elsewhere, check whether your results are broadly consistent with them. Large, unexplained differences are worth chasing down before you trust your own output; they can point to a coding bug, a difference in study system that’s worth explaining, or an exciting new finding.
6. Compare different model structures. This is distinct from varying a parameter: here you change the model’s structure itself — a different functional form, a different set of processes included or excluded — and see whether your conclusions hold up. This is exactly the structural uncertainty discussed earlier, and it’s addressed by comparing models, not by collecting more data under one structure. It’s easy to get comfortable with the one structure you’ve built; deliberately trying a rival structure is often the fastest way to find out whether your conclusions depend on a modelling choice you’d stopped noticing you’d made.
7. Do a sensitivity analysis, and treat it as more than a box to tick.
9.1 Sensitivity analysis is a source of insight, not a checkbox
Strictly, “sensitivity analysis” once meant something specific: the derivative of an outcome with respect to a parameter. Nowadays it usually just means varying one thing and seeing how another responds. However you define it, the key point (which I owe to a reviewer, years ago) is that sensitivity analyses are a source of insight in their own right:
- They can represent different contexts: vary an input to stand in for different ecological settings, and see how the response changes.
- They tell you what matters: if changing a parameter doesn’t change the outcome, you may not need to know that parameter well. This is exactly how modelling guides empirical work.
- It’s your chance to break your own conclusions. Once you start drawing conclusions, the temptation is to hold them tightly. Instead, push on them: use sensitivity analysis to find where a conclusion stops being robust. Great papers are built on discoveries of the form “X is true, but only if Y, therefore we need to know if Y is true”
- In management and decision settings, you often don’t care about biological sensitivity at all; the sensitivity of the decision is what’s critical.
Where do the parameters come from? Sometimes you estimate them from data (back to the estimation constraints discussed earlier). Sometimes you don’t know them, and that’s fine if your goal is to build a story or run scenarios. When your purpose is to find what matters for empiricists or decision makers, sensitivity analysis is the whole point: run the what-if scenarios and ask whether the outcome is sensitive to the thing you’re uncertain about.
9.2 On the reality of debugging
Building a model is iterative because you frequently can’t know the outcome until you see it.
That’s also, incidentally, why AI can’t just do the whole thing for you: you have to run it, look, and interpret before you even know what you were aiming at. When AI-written code goes wrong it’s usually one of two things. Either you weren’t specific enough about what you wanted, or (more commonly) you were specific about some parts and the AI filled the gaps with assumptions that turned out to be wrong. It’s less “hallucination” than under-specification. You only catch these by running the model and interpreting the results, which takes real work. Use AI to translate your modelling ideas into code, but go slowly and in small pieces: have it do a bit, have it explain that bit back to you, and make sure you understand it, because AI is very good at producing complex, plausible code that is quietly, logically wrong (see my blog post on judicious AI use for how this looks in practice). If the error turns out to be not in the code but in the discipline’s accepted logic, then you may have a good paper. Understanding is the thing that lets you tell the difference.