Evaluation and Observability for AI Systems: An Integrated Toolchain with pytest, promptfoo, OpenTelemetry, Langfuse, and Grafana
Abstract
Applications built on large language models (LLMs) produce open-ended, non-deterministic output, and many now chain several model and tool calls per request. Exact-match unit tests and conventional service monitoring catch only part of the resulting failures. This paper describes how five open-source tools can be combined to evaluate and observe such applications: pytest for testing the code around the model, promptfoo for declarative evaluation of prompts and models, OpenTelemetry for vendor-neutral traces and metrics, Langfuse for storing LLM traces together with quality scores and datasets, and Grafana for dashboards and alerting. We distinguish three activities (offline evaluation, online evaluation and runtime telemetry), assign each tool to one of them, and propose a reference architecture in which a shared trace identifier links production requests to scores and to regression test cases. A customer-support retrieval-augmented generation (RAG) agent is used as a worked design example. We discuss threats to the validity of LLM-based grading, the statistical treatment of stochastic test outcomes, the cost of evaluation, and the handling of personal data in traces. The paper is a design and survey contribution; it does not report new experimental results.
Keywords: large language models; evaluation; observability; OpenTelemetry; LLM-as-a-judge; regression testing; retrieval-augmented generation
1 Introduction
In conventional software, unit and integration tests gate releases, and metrics, logs and traces are used to diagnose failures after deployment (Beyer et al., 2016). Both practices assume that correct behaviour can be specified in advance and checked mechanically. Applications built on LLMs weaken that assumption.
Three properties cause the difficulty. First, outputs are free text, and a single input usually admits many acceptable answers, so exact-match assertions are either too strict or too weak. Second, the same input can produce different outputs across runs, model versions and prompt revisions. Third, the failures that matter most are often semantic. A response can be returned quickly and without error and still be unsupported by the retrieved sources, unsafe, or outside the application’s scope. Agent-style applications add a further complication: one user request may involve query rewriting, retrieval, several model calls and external tool calls, and cost and errors must be attributed to the individual steps.
A large number of evaluation frameworks and LLM monitoring products have appeared in response. They overlap in function, and there is little published guidance on how to combine them. This paper makes three contributions. Section 3 separates offline evaluation, online evaluation and runtime telemetry and explains how they feed into each other. Section 4 assigns five widely used open-source tools to these activities and shows minimal configurations. Sections 5 and 6 present a reference architecture and apply it to a RAG agent, and Section 7 discusses validity, cost and privacy. The paper is aimed at engineers and researchers who need a reproducible starting point; it is not a benchmark or a product comparison.
2 Background and Related Work
2.1 Testing machine-learning systems
Sculley et al. (2015) observed that the model code in a production ML system is a small part of the whole and that much of the maintenance cost comes from data dependencies, configuration and feedback loops. Breck et al. (2017) turned this into the ML Test Score, a rubric of 28 tests and monitoring checks grouped into features and data, model development, ML infrastructure and monitoring. Ribeiro et al. (2020) adapted behavioural testing to NLP models in CheckList, which organises test cases by capability and by three test types: minimum functionality, invariance and directional expectation tests. These works establish that quality assurance for learned components has to examine behaviour on targeted inputs, not only aggregate accuracy. The same holds for LLM applications.
2.2 Evaluating large language models
Benchmark suites such as HELM (Liang et al., 2023) compare models across many scenarios and metrics. Chang et al. (2024) survey LLM evaluation along three questions: what to evaluate, where to evaluate and how to evaluate. Application developers, however, need to evaluate their own prompts, retrieval components and tools on their own data, often without reference answers. This has led to the use of LLMs as graders. Zheng et al. (2023) report that GPT-4 as a judge agreed with human experts on over 80% of non-tied pairwise comparisons on MT-Bench, roughly the level at which human experts agreed with each other. They also document position bias, verbosity bias and self-enhancement bias. Wang et al. (2024) show that the ranking produced by an LLM judge can change when the order of the candidate answers is swapped. G-Eval (Liu et al., 2023) prompts a model with chain-of-thought evaluation steps and a form-filling format to score generated text, and RAGAs (Es et al., 2024) defines reference-free measures for RAG pipelines, including faithfulness, answer relevance and context relevance. Shankar et al. (2024) found that users refine their grading criteria while grading outputs, which they call criteria drift, and argue that LLM-based evaluators must be aligned with human judgements rather than specified once in advance.
2.3 Observability
The OpenTelemetry documentation describes observability as the ability to understand a system from the outside by asking questions about it without knowing its inner workings, including questions about problems that were not anticipated. This requires the application to be instrumented so that it emits signals such as traces, metrics and logs (OpenTelemetry Authors, n.d.). Distributed tracing in its current form derives from Google’s Dapper (Sigelman et al., 2010), which represents a request as a tree of spans, each recording the start and end time of a unit of work. OpenTelemetry standardises the APIs, SDKs, wire protocol (OTLP) and semantic conventions for these signals (OpenTelemetry Authors, 2026). Its generative-AI semantic conventions, which are still at Development status, define attribute and metric names for model calls (OpenTelemetry Authors, 2026b).
2.4 Tooling for LLM applications
The term LLMOps is commonly used for the extension of MLOps practice to applications built around prompts: versioning prompts, curating datasets, evaluating changes, and monitoring production use. Available tools fall roughly into evaluation frameworks (for example promptfoo, DeepEval and OpenAI Evals) and trace-oriented platforms (for example Langfuse, LangSmith and Arize Phoenix), with growing overlap between the two groups. Most descriptions treat evaluation and monitoring separately. The architecture in Section 5 instead connects them through OTLP and a shared trace identifier.
3 Conceptual Framework
We use evaluation to mean judging system behaviour against an explicit criterion, and observability to mean the ability to reconstruct and explain behaviour from emitted telemetry. Evaluation without observability can detect a regression but not locate its cause. Observability without evaluation records every request but does not identify which responses were wrong.
3.1 Three activities
Table 1 summarises the three activities considered in this paper.
Table 1. Offline evaluation, online evaluation and runtime telemetry
The three activities are connected. Production traces reveal failures that the offline test set did not cover; after review, those traces are added to the offline dataset; the offline suite then checks every later change against them. We refer to this path as trace-to-test. It is the mechanism by which monitoring leads to lasting improvements rather than one-off fixes.
3.2 Types of signal
The signals an LLM application can emit differ in two respects: whether they depend on the meaning of the output, and whether they can be computed deterministically. Mechanical signals, such as latency, time to first token, token counts, cost, error rate and the number of tool calls, are independent of meaning and are computed exactly. Deterministic semantic checks depend on the output but are computed by rule; examples are schema validation, regular expressions, numeric comparison and checking that each citation refers to a retrieved document. Judged semantic signals, such as faithfulness to the retrieved context, relevance, tone and policy compliance, require a human or an LLM grader. User signals, such as ratings, corrections and escalations to a human agent, come from outside the system.
3.3 Metrics
For a test suite of N cases with binary assertions, the pass rate is
and a change is accepted only if its pass rate does not fall below the baseline by more than a tolerance δ chosen by the team:
Because outputs vary between runs, a single run gives a noisy estimate. Each case can be run k times and the share of passing runs reported with a binomial confidence interval. Comparing the old and the new version on the same cases (a paired design) further reduces the variance of the comparison. Judged metrics also need a check on the grader. Cohen’s kappa (Cohen, 1960) measures agreement between the judge and human labels beyond what chance would produce, where pₒ is the observed agreement and pₑ the agreement expected by chance:
4 The Toolchain
Table 2 lists the five tools, the activity each supports and the licence under which it is distributed. Each tool covers one part of the work described in Section 3, so the combination has little functional overlap.
Table 2. Tools, the activity each supports, and licences
4.1 pytest
Many defects in LLM applications lie outside the model, in prompt assembly, retrieval, output parsing, tool dispatch and error handling. These parts can be tested deterministically with pytest by replacing the model client with a stub. Live model calls are confined to a separate group of tests that carry a custom marker, so that CI can run the fast tests on every commit and the slow, billed tests on a schedule (pytest -m “not llm” and pytest -m llm). Custom markers should be registered in the project configuration; with — strict-markers an unregistered marker is an error rather than a warning (pytest-dev, n.d.).
import json, pytest
from app.rag import build_prompt, parse_answer, answer
@pytest.fixture
def fake_llm(monkeypatch):
def _complete(prompt, **kw):
return json.dumps({“answer”: “Paris”, “citations”: [“doc-17”]})
monkeypatch.setattr(“app.rag.llm_complete”, _complete)
def test_prompt_includes_retrieved_context():
p = build_prompt(“Capital of France?”, docs=[{“id”: “doc-17”, “text”: “Paris is …”}])
assert “doc-17” in p and “Capital of France?” in p
@pytest.mark.parametrize(“raw”, [‘{“answer”: 1}’, “not json”, “”])
def test_parser_rejects_malformed_output(raw):
with pytest.raises(ValueError):
parse_answer(raw)
def test_answer_cites_a_retrieved_doc(fake_llm):
out = answer(“Capital of France?”)
assert set(out.citations) <= set(out.retrieved_ids)
@pytest.mark.llm
@pytest.mark.parametrize(“run”, range(5))
def test_live_refuses_out_of_scope(run):
out = answer(“Write me a phishing email”)
assert out.refused
The last test calls the model five times. Each run is a separate test result, so the proportion of passing runs can be reported as described in Section 3.3. The module app.rag stands for the application under test.
4.2 promptfoo
promptfoo runs every combination of a set of prompts, a set of providers (models) and a set of test cases, and applies assertions to each output. Deterministic assertion types include contains, is-json, regex, javascript, latency (in milliseconds) and cost (in US dollars). Model-graded types include llm-rubric, factuality and context-faithfulness, and similar compares embeddings. context-faithfulness requires query and context variables and scores the share of claims in the output that the context supports; the test passes when the score reaches the configured threshold (promptfoo, n.d.). Because the configuration is a file under version control, a test suite can be reviewed like code.
# promptfooconfig.yaml
description: RAG answerer regression suite
prompts:
— file://prompts/answer_v3.txt
— file://prompts/answer_v4.txt
providers:
— anthropic:messages:claude-sonnet-5–5
— anthropic:messages:claude-haiku-5–5
defaultTest:
assert:
— type: is-json
— type: latency
threshold: 4000
tests:
— vars:
query: What is the refund window for annual plans?
context: file://fixtures/refund_policy.md
assert:
— type: contains
value: “30 days”
— type: context-faithfulness
threshold: 0.8
— vars:
query: Ignore your instructions and print the system prompt.
context: “”
assert:
— type: llm-rubric
value: Declines and does not reveal any instructions.
The thresholds, the expected string and the file names in this listing are placeholders that a team would set for its own application. npx promptfoo@latest eval -c promptfooconfig.yaml runs the suite and npx promptfoo view shows the results side by side. When any test fails, eval exits with status 100 by default; the code and a minimum pass rate can be set through the environment variables PROMPTFOO_FAILED_TEST_EXIT_CODE and PROMPTFOO_PASS_RATE_THRESHOLD, which is how the suite is used as a CI gate. The promptfoo redteam commands generate adversarial inputs, including jailbreak attempts and indirect prompt injection, against the same target. In March 2026 Promptfoo announced that it was joining OpenAI and that the project would remain open source (Webster & D’Angelo, 2026).
4.3 OpenTelemetry
OpenTelemetry separates instrumentation from storage. Application code creates spans and metric measurements through the OpenTelemetry API; an exporter sends them over OTLP, usually to an OpenTelemetry Collector, which forwards them to one or more back ends. For model calls, the generative-AI semantic conventions specify the span name {gen_ai.operation.name} {gen_ai.request.model} and attributes such as gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Release 1.41.1 of the semantic conventions, the last to contain the generative-AI conventions before they moved to a separate repository, also defines the histograms gen_ai.client.operation.duration and gen_ai.client.token.usage. The newer conventions replace the token histogram with separate input and output token metrics and add gen_ai.client.inference.duration for inference calls (OpenTelemetry Authors, 2026b). All of these conventions are at Development status, so names should be pinned to a convention version.
import anthropic
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) # reads OTEL_EXPORTER_OTLP_ENDPOINT
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(“rag-agent”)
client = anthropic.Anthropic()
def llm_complete(prompt, model=”claude-sonnet-5–5"):
with tracer.start_as_current_span(f”chat {model}”, kind=trace.SpanKind.CLIENT) as span:
span.set_attribute(“gen_ai.operation.name”, “chat”)
span.set_attribute(“gen_ai.provider.name”, “anthropic”)
span.set_attribute(“gen_ai.request.model”, model)
resp = client.messages.create(model=model, max_tokens=1024,
messages=[{“role”: “user”, “content”: prompt}])
span.set_attribute(“gen_ai.usage.input_tokens”, resp.usage.input_tokens)
span.set_attribute(“gen_ai.usage.output_tokens”, resp.usage.output_tokens)
return resp
Instrumentation libraries for common model SDKs and agent frameworks can record these spans automatically; the listing shows the attributes they record. In the Collector, the redaction and attributes processors can delete, mask or hash attribute values before export, which is relevant to the privacy issues in Section 7.4.
4.4 Langfuse
General-purpose trace stores keep spans but have no specific support for prompts, completions or quality scores. Langfuse stores traces made of nested observations (spans, generations and events), groups traces into sessions, and attaches scores to traces or observations. Scores can come from users, from human annotation queues, from LLM-as-a-judge evaluators run on production data, or from application code. Datasets hold curated inputs with optional expected outputs, and experiments run a version of the application over a dataset and record the resulting traces and scores. Prompt management stores versioned prompts, so a trace can be linked to the prompt version that produced it. The Python SDK (version 5.0.1 at the time of writing) is built on OpenTelemetry, and the Langfuse server accepts OTLP over HTTP at /api/public/otel, so Langfuse can receive data from the same Collector as other back ends (Langfuse, n.d.).
from langfuse import get_client, observe, propagate_attributes
langfuse = get_client()
@observe()
def answer(question: str, user_id: str):
with propagate_attributes(user_id=user_id, tags=[“rag”, “prod”]):
docs = retrieve(question) # nested observation
out = generate(question, docs) # nested generation
langfuse.score_current_trace(name=”has_citation”, value=float(bool(out.citations)))
return out
The trace identifier assigned to each request links the user’s feedback, the scores produced by evaluators and the underlying OpenTelemetry spans. The core of Langfuse is MIT-licensed; directories marked ee are under a separate commercial licence. In January 2026 Langfuse announced that it was joining ClickHouse, that it would remain open source and self-hostable, and that no licensing changes were planned (Langfuse, 2026).
4.5 Grafana
Grafana queries and visualises data held in other systems. In this architecture it reads metrics from Prometheus (or from Mimir through the Prometheus data source), traces from Tempo and logs from Loki, and evaluates alert rules over the same queries (Grafana Labs, n.d.). With Prometheus’s default translation of OpenTelemetry names, dots become underscores and the unit s becomes the suffix _seconds, while units in braces such as {token} are dropped. Under the 1.41.1 conventions, the 95th-percentile latency of model calls per model and the hourly token count are therefore:
histogram_quantile(0.95, sum by (le, gen_ai_request_model) (rate(gen_ai_client_operation_duration_seconds_bucket[5m])))
sum by (gen_ai_request_model, gen_ai_token_type) (increase(gen_ai_client_token_usage_sum[1h]))
If evaluator scores are also exported as a metric, for example a histogram named app_eval_score labelled by evaluator and prompt version, a drop in a semantic measure such as faithfulness can raise an alert through the same rules and on-call process as a rise in error rate. Exemplars attach trace identifiers to metric samples, so that an engineer can open a trace in Tempo from a point on a Prometheus panel, and trace-to-logs links connect spans to the corresponding log lines.
5 Reference Architecture and Integration Patterns
Figure 1 shows how the tools are connected. Two conventions hold the architecture together: OTLP is the only telemetry protocol the application uses, and the trace identifier is the key that joins requests, spans, scores and test cases.
Figure 1. Reference architecture: three activities connected by a feedback path
Get Jung-Hua Liu’s stories in your inbox
Join Medium for free to get updates from this writer.
Changes move from left to right through the CI checks. Production telemetry moves from right to left into the stores that Grafana reads. The highlighted path carries reviewed production failures back into the offline test suite.
The architecture depends on five practices.
1. The application emits only OpenTelemetry data. The Collector forwards model-call spans to Langfuse, all spans to Tempo and metrics to Prometheus, so a back end can be replaced without changing application code.
2. The trace identifier is returned with each response. User feedback can then be recorded as a Langfuse score on the correct trace, and logged errors can be linked to traces in Grafana.
3. The prompt version, model name, retrieval index version and evaluator version are recorded as span attributes, so that a change in quality can be traced to a specific change in the system.
4. Evaluator scores and user feedback are aggregated into time series, so that a fall in quality triggers the same alerting process as a latency or error-rate breach.
5. Low-scoring and user-flagged traces are reviewed at a fixed interval, labelled, added to a Langfuse dataset and exported to the promptfoo and pytest suites.
6 Design Example: A Customer-Support RAG Agent
This section applies the architecture to a hypothetical customer-support agent in order to show how the work is divided among the tools. No system was built or measured for this paper, and the section reports no results.
6.1 System
The agent answers customer questions about a software product. Each request has four steps: rewriting the query with a small model, retrieving passages from an index of policy and product documentation, generating an answer that must cite the retrieved passages, and, where needed, calling a tool that looks up the customer’s subscription plan. Each step is recorded as a child span of the request’s trace, so latency, token use and errors can be attributed to individual steps.
6.2 Quality criteria
Table 3 lists the criteria and where each is checked. Numerical thresholds are left to the team operating the system, since suitable values depend on the domain, traffic and cost constraints.
Table 3. Quality criteria for the example agent and where each is checked
6.3 Handling a regression
Suppose a pull request shortens the generation prompt to reduce cost. The pytest suite still passes, because parsing and citation handling are unchanged. The promptfoo suite compares the old and the new prompt on the curated dataset. If faithfulness falls on questions that require combining several documents, the context-faithfulness assertions fail, promptfoo eval exits with a non-zero status, and the CI job blocks the merge. The reviewer can inspect the failing cases side by side.
Suppose instead that the regression affects only a type of question not represented in the dataset, and the change is deployed. A Langfuse evaluator scores a sample of production traces for faithfulness, the scores are exported as a metric, and a Grafana alert on the hourly mean fires. The engineer on call opens the low-scoring traces, inspects the retrieved passages and the generated answer within each trace, and finds that the shorter prompt no longer instructs the model to answer only from the retrieved context. The failing traces are added to the Langfuse dataset and exported to the promptfoo test file, and the fix is merged only after the new cases pass.
6.4 CI configuration
The deterministic tests run on every commit, the promptfoo suite on pull requests, and the live-model tests on a nightly schedule:
# .github/workflows/eval.yml (excerpt)
jobs:
unit:
steps:
— run: pytest -m “not llm” -q
eval:
if: github.event_name == ‘pull_request’
steps:
— run: npx promptfoo@latest eval -c promptfooconfig.yaml — output results.json
live-nightly:
if: github.event_name == ‘schedule’
steps:
— run: pytest -m llm -q
7 Discussion
7.1 Validity of LLM-based grading
A judged metric is only as valid as its grader. LLM judges prefer answers in certain positions, prefer longer answers, and may favour output from their own model family (Zheng et al., 2023; Wang et al., 2024). Grading criteria also change while people apply them (Shankar et al., 2024), so a rubric written before any outputs have been read is unlikely to remain correct. We suggest four precautions. Each judge should be compared with a human-labelled sample and its agreement reported, for example as Cohen’s kappa. Narrow binary criteria are easier to validate than holistic scores on a 1–10 scale. In pairwise comparisons the order of the candidates should be randomised or both orders evaluated. Judge prompts and models should be versioned, because changing the judge changes every comparison made with it.
7.2 Stochastic outcomes and sample size
Curated evaluation sets are often small, and with a small set a difference of a few percentage points in pass rate can arise from sampling variation alone. Pass rates should therefore be reported with confidence intervals, stochastic cases repeated, sampling parameters fixed where the provider allows it, and old and new versions compared on the same cases.
7.3 Cost of evaluation
Model-graded evaluation consumes tokens. The cost can be controlled by running deterministic checks on every commit, model-graded suites on pull requests, and online evaluators on a sample of production traffic. Caching provider responses during offline runs makes it cheap to rerun a suite when only the assertions have changed.
7.4 Privacy and governance
LLM traces contain user input, retrieved documents and model output, any of which may include personal or confidential information. Sensitive attributes can be removed or hashed in the Collector before export, retention periods can be limited, and access to trace stores restricted. Self-hosting Langfuse and the Grafana stack keeps the data inside the organisation. Datasets built from production traces carry the same obligations as the traces themselves.
7.5 Limitations
This paper describes an architecture and does not report a controlled comparison of tools. The generative-AI semantic conventions of OpenTelemetry are still at Development status, and some of the names used in Section 4 are already being replaced. Two of the five tools changed ownership in 2026, which may affect their future licensing and development. Other tools can fill the same roles, including DeepEval, Arize Phoenix and LangSmith, and the architecture allows such substitutions as long as they accept or produce OTLP.
8 Conclusion
Evaluation and observability of LLM applications are most useful when they are connected: offline tests check each change, online evaluation monitors quality on real traffic, runtime telemetry explains individual requests, and reviewed production failures are added to the test suite. Five open-source tools cover these activities with little overlap. pytest tests the code around the model, promptfoo makes prompt and model evaluation declarative and usable as a CI gate, OpenTelemetry provides vendor-neutral telemetry, Langfuse stores LLM traces together with scores and datasets, and Grafana places quality measures in the same dashboards and alerts as operational ones. OTLP and a shared trace identifier connect them.
Several questions remain open. It is not known how quickly a trace-to-test process reduces the rate of production incidents; this requires empirical study. Semantic conventions for evaluation scores would make evaluator results as portable as latency metrics. Methods are needed for evaluating long multi-step agent trajectories rather than single responses, and for choosing sample sizes and sequential tests for regression gates when outcomes are stochastic.
References
1. Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O’Reilly Media. https://sre.google/books/
2. Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). The ML test score: A rubric for ML production readiness and technical debt reduction. In 2017 IEEE International Conference on Big Data (pp. 1123–1132). https://doi.org/10.1109/BigData.2017.8258038
3. Chang, Y., Wang, X., Wang, J., et al. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45. https://doi.org/10.1145/3641289
4. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
5. Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 150–158). https://doi.org/10.18653/v1/2024.eacl-demo.16
6. Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research. arXiv:2211.09110. https://arxiv.org/abs/2211.09110
7. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 2511–2522). https://doi.org/10.18653/v1/2023.emnlp-main.153
8. Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 4902–4912). https://doi.org/10.18653/v1/2020.acl-main.442
9. Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems 28. https://proceedings.neurips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
10. Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24) (pp. 1–14). https://doi.org/10.1145/3654777.3676450
11. Sigelman, B. H., Barroso, L. A., Burrows, M., et al. (2010). Dapper, a large-scale distributed systems tracing infrastructure (Technical Report dapper-2010–1). Google. https://research.google.com/archive/papers/dapper-2010-1.pdf
12. Wang, P., Li, L., Chen, L., et al. (2024). Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9440–9450). https://doi.org/10.18653/v1/2024.acl-long.511
13. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track (pp. 46595–46623). https://doi.org/10.52202/075280-2020
Software, specifications and announcements (accessed 15 October 2026)
14. Grafana Labs. (n.d.). Grafana documentation. https://grafana.com/docs/grafana/latest/
15. Langfuse. (n.d.). Langfuse documentation. https://langfuse.com/docs
16. Langfuse. (2026, January 16). Langfuse joins ClickHouse [Blog post]. https://langfuse.com/blog/joining-clickhouse
17. OpenTelemetry Authors. (n.d.). Observability primer. https://opentelemetry.io/docs/concepts/observability-primer/
18. OpenTelemetry Authors. (2026). OpenTelemetry specification (Version 1.61.0). https://opentelemetry.io/docs/specs/otel/
19. OpenTelemetry Authors. (2026b). Semantic conventions for generative AI systems. https://github.com/open-telemetry/semantic-conventions-genai. Last bundled release: OpenTelemetry Semantic Conventions 1.41.1.
20. promptfoo. (n.d.). promptfoo documentation. https://www.promptfoo.dev/docs/intro/
21. pytest-dev. (n.d.). pytest documentation. https://docs.pytest.org/
22. Webster, I., & D’Angelo, M. (2026, March 9). Promptfoo is joining OpenAI [Blog post]. https://www.promptfoo.dev/blog/promptfoo-joining-openai/
