Troubleshooting Time Series
Building a tool for data deep-dives has been a long-standing interest for me. What I would like, ideally, is a way to investigate any kind of time series through a series of semi-automated steps.
I am used to looking at time series plots for correlations and understanding why apps, backends, or APIs are not performing as expected.
Before AI arrived, this was a slow process of figuring out which dimension had changed and what change correlated with what other change. It meant looking at several different datasets, with occasional SQL detours to do joins. Slow, deliberate, and noisy. This process turns up a lot of details you would rather not know — mostly implementation idiosyncrasies or logging deficiencies that have little to no bearing on what you are investigating.
Is building a generic time series analysis tool possible in principle, and how much of it is domain-specific?
When you do enough troubleshooting, you realize how little the tooling, the data, or the UX have to do with how effective you are. The key is always in generating the questions to ask. The better and more specific the question, the more likely you are to land on the answer or move toward one. The investigation stalls when you run out of good questions to ask.
Get Evgeny Arbatov’s stories in your inbox
Join Medium for free to get updates from this writer.
There are also some things that are non-obvious at a smaller scale, such as: 1) looking at graphs is slow and time-consuming, 2) the data is incomplete and sampled because keeping every single data point is not technically possible, 3) it takes time to find out where data lives and what the exact definition of it is, 4) without understanding of the whole flow, you will see patterns in data that are misleading.
With all this in mind, I think the best use of AI is not building better workflows for analyzing data; it is simply enabling you to ask better questions. The model should not be asked to make sense of the numbers. The meaning of numbers is often beyond the models you can run locally, unless you are open to feeding your data to cloud models.
Troubleshooting is clearly an iterative process with frequent backtracking and going back to the original premise and wondering if it is valid. That is why a misdirected question is so hard to answer — because you usually start on the assumption that the premise is valid until the data proves you wrong. In a lot of cases, the data is ambiguous or simply missing, so it cannot validate the hypothesis either way. This is a big time sink.
In my mind, a lot of the basic steps of generating hypotheses and either validating or disproving them can be encoded in a simple state machine, with the LLM as the only open-ended reasoning part that generates narrowing or widening hypotheses to test and calls tools. With occasional stops to ask a human when the investigation has made no progress, or to share status updates.
How much of this is specific to time series, and should it be strictly limited to time series? I think a general troubleshooting agent is less valuable and mostly already available with Grok or Claude. Time series is already structured data that can be queried and reasoned about, so you do not have to interpret the wide variety of inputs.
Given that the amount of possible things you can build is increasing and the services are often verbose in their logs, something that predictably works for any time series would be really useful and save a non-trivial amount of time. The interesting bugs, useful properties, and unexpected issues appear from the interaction of multiple high-level components and rarely from the static version you can unit-test.
It is curious how hard it is to build a general troubleshooting agent. You may want to start general, but as you make simplifying assumptions, your design starts to become clear — until you have a tool that is good at one specific task. Then you start again when you have a new problem. The question is how to simplify without being overwhelmed by the details of the specific implementation in front of you?
