What OpenTelemetry still can’t tell about the LLM calls

Abhinav Gupta

3 min read

·

Sep 26, 2026

--

1

Listen

Share

When my code calls any LLM model, how do I actually know what happened inside the call?

Press enter or click to view image in full size

The answer is tracing. OTel is the open standard for writing it, so any tool can read it the same way.

For LLM calls specifically, OTel defines real fields for those tags like gen_ai.request.model (which model answered), gen_ai.token.type (whether a token count is input or output).

But problem is, every one of those fields is labeled as Development, not Stable. Development means it is not promised that a rename comes with a compatibility path, the next release can rename gen_ai.token.type to something else entirely, and every dashboard filter or alert you built around that exact name just stops matching. There will be no error, no warning, it silently returns nothing.

So the basics like model, tokens, timing will work today, but they are built on a name that is not guaranteed to still be the name next month.

Knowing what happened is one problem. Knowing whether the answer was actually correct is a different, harder one, and there is no field for it. Not because nobody has gotten it yet, but because of how a span works.

A span opens when the request starts and closes the moment it ends — that’s the instant it gets shipped off to the tracing backend. Checking the answer means sending it to a second, separate model to grade it, and that grading call almost never finishes before the original span has already closed and shipped. By the time you have an answer, there is no open record left to write it into.

So we are left with 2 options:

1. Make the users request wait for the grader too, then write the score onto the same span. This works well for end to end trace, but slower for the end user.
2. Let the response return immediately, and send the grade later as its own span, correlated back to the original by trace ID. This is faster for the user, but the cost doesn’t disappear — it just shifts from the user’s wait time to the tracing backend. Correlation is the technique that makes this possible, but it doesn’t make it free (you still have to manually carry the original trace ID into the later grading call, since the request’s own context is already gone by the time that call runs, and the tracing backend has to accept a span that shows up after the trace already looked finished. From then on, every time anyone wants to read the result, they are joining two separate records by trace ID instead of reading one attribute off one span. This is not a one time setup cost, but every time)

If the user already has their answer by the time the grade comes back, what is the point of grading it at all? Not for that user, that chance is gone the moment the response returns. It is for the system which aggregates the grades across thousands of requests and you can watch a pass rate over time, catch it drop after a bad prompt change or model change, pull the failed answers out for a human review or feed them into fixing the next version. (Inline grading protects one answer, async grading protects the next thousand)

This is not a hypothetical situation, it is one every real tool that traces LLM calls today has already had to solve on its own, each picking a name ahead of any standard, so whichever way OTel eventually settles this, some of them will have to migrate.