Section 6: Agentic engineering for data analysis

LLM agent are getting so good at data analysis that we are increasingly tempted not to directly review the code they write. In fact, I’m seeing Claude Code find more mistakes in my code than I find in its code.

This tendency towards not reviewing code directly means we need ways to check the validity of the code an agent writes.

Part of the problem is that once you start using an agent it will write very complex code. Claude Code has this obsession with making absolutely generalizable scripts. For instance, it commonly writes functions just to identify the root file path for a key data file.

The other part of the complexity agents code in is adding numerous print statements to the data analysis scripts, so it can see intermediate results in the terminal output. I normally do these interactively so they stay out of my scripts.

All this extra code means it becomes even more tedious to review its work. Its like a double edged sword, the agent is powerful, but it makes more review work.

Here are some strategies for verifying the validatity of a data analysis. I can’t guarantee they will find all the mistakes, however these principles are habits that experienced data analyses have developed to prevent mistakes in their own code. They translate well to agents as well.

But first, we should consider the different classes of errors that can occur in an analysis workflow.

Connolly, R. M., D. V. Fairclough, E. L. Jinks, E. M. Ditria, G. Jackson, S. Lopez-Marcano, A. D. Olds, and K. I. Jinks. 2021. “Improved Accuracy for Automated Counting of a Fish in Baited Underwater Videos for Stock Assessment.” Frontiers in Marine Science 8: 658135. https://doi.org/10.3389/fmars.2021.658135.