Randy Au on the manual process of exploratory data analysis that is still irreplaceable:
While I’m fine offloading specific tasks to an LLM, I find they’re just really terrible when given any leeway with decision-making. Since there’s no absolute positive/negative feedback in an analysis, typical coding agent harnesses have no way to tell if their analysis has completely flown off the rails. While toying with the idea, I’ve seen Claude get a peculiar idea that obviously takes assumptions that doesn’t apply to the data and attempt to build a giant testing harness around that analysis idea. I watched it run to completion and asked some pointed questions about why it was doing that and it did the “oh! my bad!” routine. You can jump through all sorts of hoops and harnesses to try to get a handle on things, but the only creativity I’ve encountered with this process is how it comes up with new and unimaginable ways to do the wrong thing.
When you know exactly what you want, generative AI is useful for joining that dataset or making that one chart. Exploratory data analysis relies on not having an exact answer and relying on clues during the process to guide you.
This will evolve as the LLMs grow more extensive, but for now, analysis output from the chatbots converge towards generic results without outside guidance.
Tags: automation, Large Language Model, Randy Au