testthat::expect_true(all(df$column >= 0))34 Tests and data processing checks
34.1 Red Green TDD
You can add ‘use red green TDD’ (‘test driven development’) to your prompt. This will encourage the agent to use a particular programming pattern. It will write tests that fail first, then fix them. This ensures the test works before correcting it.
As an example a test in R to ensure there are no negative values in a column of a data frame would look like this:
The agent can program these itself.
You should pay particular attention to make sure it tests weak points in the analysis. These can include in particular data joins and conversions among units.
Instruct your agent to check all data processing steps and fail loudly if inconsistencies are found. That’s probably sufficient these days, but if you want to be thorough you can get it to create a ‘source of truth’ document that is then verified against at critical checkpoints.
34.2 Using a source of truth table
In a complex data pre-processing workflow that ends in model fitting your source of truth document might include: the expected number of samples, the expected number of unique sites (or other category), the expected range of different variables as suggested by the units they should have.
Then the data frame is verified at a checkpoint right before model fitting.
The model output can also be verified against this source of truth, such as checking that the number of residuals also matches the number of samples.
34.3 Multi-agent workflows
If an idea needs testing you can ask your agent to fire up multiple sub-agents to test the idea in different ways. Instruct the lead agent to initialize multiple independent sub-agents, each with a slightly different idea to test.
The key then is having a way to evaluate the results of each sub-agent. Ideally this is a quantitative evaluation. For example, it could be the number of species that the workflow runs for without failure.
Just remember that you should not use the scientific result as the evaluation for choosing which sub-agents workflow you use (e.g. using a p-value, effect size). If you do this you are gaming the system to find a stronger result (also called ‘over-hyping’ or ‘p-hacking’). A more ethical way to approach it would be report each individual workflow as a robustness test on the main result.