storage

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 25, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package storage is the mAPI-ng data plane: the ClickHouse-backed writer and query layer for Summary rows. Percentiles are computed at query time from the merged DDSketch bucket maps, matching the frozen gamma/offset contract shared with the client (ADR-0001, ADR-0003).

Index

Constants

View Source
const (
	// SketchGamma is the DDSketch relative-error base (gamma).
	SketchGamma = 1.01
	// SketchGammaPlusOne is gamma+1, precomputed for value(i).
	SketchGammaPlusOne = SketchGamma + 1.0
)

Sketch mapping constants. These are the FROZEN DDSketch contract shared with the client and encoded in query.go's percentile SQL. gamma = 1.01; a bucket index i represents latency value(i) = 2*pow(gamma, i)/(gamma+1) seconds. Changing these is a breaking migration.

View Source
const (
	// DefaultFlushInterval bounds how long a row waits before being inserted.
	DefaultFlushInterval = 2 * time.Second
	// DefaultFlushRows triggers a flush once this many rows are buffered.
	DefaultFlushRows = 100_000
	// DefaultInsertTimeout bounds a single batch insert so a ClickHouse
	// connection that stalls mid-insert cannot block the sole batcher goroutine
	// indefinitely (and, through it, wedge all ingest). It is a hang guard, not
	// an SLA: it must exceed a healthy insert's latency by a wide margin. On
	// expiry the steady-state flush drops the batch and the final drain retries.
	DefaultInsertTimeout = 30 * time.Second
)

Batcher flush thresholds: app-side batcher, no ClickHouse Buffer engine.

Variables

View Source
var ErrEmptyTenant = errors.New("storage: row has empty tenant")

ErrEmptyTenant is returned by the enqueue methods when a row carries the zero-value tenant.ID. Rows are normally built from a resolved tenant (see ingest.summaryToRow), so the type already keeps an unvalidated tenant out of a query; this guards the one hole the type leaves open — a row struct-literal'd with the zero ID — so an un-tenanted write fails closed instead of silently persisting under an empty tenant. It is the write-path half of ADR-0010.

View Source
var ErrWriterClosed = errors.New("storage: writer closed")

ErrWriterClosed is returned by Enqueue after Close has been called.

Functions

func ApplyMigrations

func ApplyMigrations(ctx context.Context, conn driver.Conn, log *slog.Logger) error

ApplyMigrations runs every embedded ClickHouse migration in lexical order. Every statement is idempotent (CREATE ... IF NOT EXISTS, ALTER ... MODIFY TTL), so applying them on every startup is safe — this mirrors the control plane's applyMigrations and keeps mAPI-ng zero-config: a fresh ClickHouse gets its schema at boot rather than needing an out-of-band migrate step.

func HasAnySummary

func HasAnySummary(ctx context.Context, conn driver.Conn, tenantID tenant.ID) (bool, error)

HasAnySummary reports whether tenant has at least one summary row in the raw tier. It drives the dashboard's onboarding gate (step 4: first data received) and the "show dashboard vs. onboarding panel" decision — cheap existence check, not a full aggregate. Uses the raw tier because onboarding cares about "any data ever ingested", and the raw tier is where fresh summaries land.

func LastIngestByTenant added in v0.11.0

func LastIngestByTenant(ctx context.Context, conn driver.Conn) (map[string]time.Time, error)

LastIngestByTenant returns the most recent ingest time for every tenant that has ever ingested, read in one scan of the 1d tier. It is a deliberate CROSS-TENANT read: unlike the tenant-scoped dashboard API (reached only through Tenant), this powers the operator console's account list and is exposed through the dedicated Operator handle, so cross-tenant reads stay explicit and greppable rather than smuggled into the scoped path. It returns only timestamps, never tenant metrics.

func LatencyByStatusClass

func LatencyByStatusClass(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
) (map[string]ClassLatency, error)

LatencyByStatusClass returns the per-status-class latency split for one endpoint for tenant over [from, to). The returned map is keyed on the status_class Enum8 values (STATUS_CLASS_2XX, _3XX, _4XX, _5XX, _NO_STATUS) and always contains every class: classes with no traffic in the window map to a zero-valued ClassLatency, so the caller sees a stable, complete shape. It answers whether a latency rise is on failures or also on successes. method and route may be empty to aggregate across all methods / all route templates.

func QuantileFromBuckets

func QuantileFromBuckets(buckets map[int32]uint64, q float64) float64

QuantileFromBuckets computes a quantile from a merged DDSketch bucket map using the FROZEN convention that MUST match both the client and query.go's SQL:

total = sum of counts
rank  = clamp(ceil(q * total), 1, total)
walk buckets in ASCENDING index order accumulating counts; the first bucket
whose cumulative count >= rank gives the answer; return value(index).

It returns 0 for an empty sketch. This is the Go oracle the integration test asserts the SQL against, and it keeps the storage package meaningfully covered without a live ClickHouse.

Types

type ClassLatency

type ClassLatency struct {
	Count uint64
	P50   float64
	P95   float64
	P99   float64
}

ClassLatency is the per-status-class latency split for an endpoint: the merged request count in the class and its p50/p95/p99 over that class's own sketch.

type Config

type Config struct {
	// DSN is the ClickHouse native-protocol connection string.
	DSN string
	// FlushInterval bounds batcher latency.
	FlushInterval time.Duration
	// FlushRows triggers a flush once buffered rows reach this count.
	FlushRows int
	// InsertTimeout bounds a single batch insert so a stalled connection cannot
	// block the batcher goroutine indefinitely. Defaults to DefaultInsertTimeout.
	InsertTimeout time.Duration
}

Config holds the ClickHouse connection and batcher settings.

func ConfigFromEnv

func ConfigFromEnv() Config

ConfigFromEnv builds a Config from MAPING_CLICKHOUSE_DSN, falling back to the local dev default.

type DownstreamStat

type DownstreamStat struct {
	Count           uint64
	SumDurationNs   uint64
	SumDownstreamNs uint64
}

DownstreamStat is the self-vs-downstream time split for an endpoint over the window: the request count, the total request time, and the total time spent waiting on downstream calls. Self time is SumDurationNs - SumDownstreamNs. A zero SumDownstreamNs means no downstream timing was reported (the RoundTripper is unwired), which the UI uses to feature-gate the panel.

func DownstreamForEndpoint

func DownstreamForEndpoint(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
) (DownstreamStat, error)

DownstreamForEndpoint returns the self-vs-downstream time split for the given endpoint for tenant over [from, to). method and route may be empty to aggregate across all methods / all route templates of the service. It answers "how much of this endpoint's latency is its own work vs waiting on a dependency?".

type EndpointDetail

type EndpointDetail struct {
	Count         uint64
	ErrorRate     float64
	P50           float64
	P95           float64
	P99           float64
	Histogram     []HistogramBar
	StatusClasses []StatusClassCount
	StatusCodes   map[uint32]uint64
}

EndpointDetail is the endpoint-detail level's aggregate: headline RED numbers, the DDSketch-derived latency histogram, the per-class breakdown, and the exact status-code map. All computed over the whole [from, to) window.

func QueryEndpointDetail

func QueryEndpointDetail(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
) (EndpointDetail, error)

QueryEndpointDetail returns the full aggregate for one endpoint over [from, to): the merged sketch as a latency histogram, the per-class breakdown, the exact status-code map, and p50/p95/p99 computed in Go from the same merged sketch via QuantileFromBuckets — so the percentiles and the histogram can never disagree. Returns a zero-count EndpointDetail (no error) when the endpoint has no data in the window. Named QueryEndpointDetail (not EndpointDetail) because the return type already owns that identifier; the QueryService method is EndpointDetail, since methods have their own namespace.

type EndpointStat

type EndpointStat struct {
	Method        string
	Route         string
	Count         uint64
	ErrorRate     float64
	Client4xxRate float64
	P50           float64
	P95           float64
	P99           float64
	ReqBytesAvg   float64
	RespBytesAvg  float64
}

EndpointStat is one row of the endpoint-table level: aggregate RED metrics for one (method, route_template) of a service over the query window. The ReqBytesAvg / RespBytesAvg fields are the per-request average payload sizes (sum(req_bytes)/sum(count), sum(resp_bytes)/sum(count)) for the bytes-symmetry view; they are appended so existing scanners and callers stay valid.

func Endpoints

func Endpoints(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service string,
	from, to time.Time,
) ([]EndpointStat, error)

Endpoints returns one aggregate EndpointStat per (method, route_template) of service for tenant over [from, to), ordered by Count descending.

type ErrorClass

type ErrorClass struct {
	Class string
	Count uint64
}

ErrorClass is one normalized error-label count for the error_classes map column. Class is an uppercase [A-Z0-9_] token bounded by the client.

type ErrorClassStat

type ErrorClassStat struct {
	Class string
	Count uint64
}

ErrorClassStat is one row of the error-class breakdown for an endpoint: a normalized error label and how many requests carried it in the window. It answers "5xx up because of what?" ("DB_POOL_EXHAUSTED" vs "UPSTREAM_TIMEOUT").

func ErrorClassesForEndpoint

func ErrorClassesForEndpoint(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
) ([]ErrorClassStat, error)

ErrorClassesForEndpoint returns the top error-class labels attached to the given endpoint for tenant over [from, to), ordered by request count descending and capped at maxErrorClassesPerEndpoint. method and route may be empty to aggregate across all methods / all route templates of the service.

type Exemplar

type Exemplar struct {
	At         time.Time
	DurationNs uint64
	StatusCode uint32
	TraceID    string
	SpanID     string
	RequestID  string
}

Exemplar is one real-request breadcrumb attached to a Row. It is stored only in the raw summaries tier (rollups drop it) and lets a user pivot from a p99 / error spike to an actual request. TraceID/SpanID/RequestID are best-effort and may be empty. The tuple column order in ClickHouse is (at, duration_ns, status_code, trace_id, span_id, request_id).

type ExemplarRow

type ExemplarRow struct {
	At         time.Time
	DurationNs uint64
	StatusCode uint32
	TraceID    string
	SpanID     string
	RequestID  string
}

ExemplarRow is one real-request breadcrumb read back for the endpoint-detail view: the flattened exemplar plus its status class (derived at query time from the status code) so the caller can label errors without a second lookup.

func ExemplarsForEndpoint

func ExemplarsForEndpoint(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
) ([]ExemplarRow, error)

ExemplarsForEndpoint returns up to maxExemplarsPerEndpoint real-request breadcrumbs for the given endpoint for tenant over [from, to), ordered by latency descending, so a user can pivot from a p99 / error spike to an actual request to open in a tracing tool or logs. method and route may be empty to aggregate across all methods / all route templates of the service.

It reads the RAW summaries tier directly (NOT via selectTier): exemplars live only in the finest tier under its short TTL, so this query is only meaningful for recent windows within the raw retention. Older windows return no exemplars even though the RED aggregates remain available on the rollups.

type HistogramBar

type HistogramBar struct {
	LatencySeconds float64
	Count          uint64
}

HistogramBar is one bar of the latency histogram: the bucket's latency value in seconds (from the frozen value(i) mapping) and the merged count in it.

type InstanceResourceStat

type InstanceResourceStat struct {
	Instance        string
	CPUNs           uint64
	RSSBytes        uint64
	HeapAllocBytes  uint64
	GCPauseNs       uint64
	Goroutines      uint64
	NumGC           uint64  // completed GC cycles over the window (delta sum)
	TotalAllocBytes uint64  // heap bytes allocated over the window (delta sum)
	Mallocs         uint64  // heap objects allocated over the window (delta sum)
	GCCPUFraction   float64 // average GC CPU fraction over the window (gauge avg)
	HeapInuseBytes  uint64  // peak in-use heap bytes over the window (gauge max)
	GOMAXPROCS      uint32  // peak GOMAXPROCS over the window (gauge max)
	PostGCHeapBytes uint64  // peak post-GC heap baseline over the window (gauge max)
	RSSTrueBytes    uint64  // peak true resident set size over the window (gauge max)
	OpenFDs         uint64  // peak open file-descriptor count over the window (gauge max)
	FDLimit         uint64  // peak soft RLIMIT_NOFILE ceiling over the window (gauge max)
	InFlight        uint64  // peak in-flight request concurrency over the window (gauge max)
}

InstanceResourceStat is the per-instance resource summary over the query window: CPU and GC-pause time summed (they are per-window deltas), and the peak memory / goroutine gauges (max over the window). It answers "which replica is saturated, and how?".

func InstanceResourcesForService

func InstanceResourcesForService(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service string,
	from, to time.Time,
) ([]InstanceResourceStat, error)

InstanceResourcesForService returns one InstanceResourceStat per instance of the service for tenant over [from, to), ordered by instance. It is the USE side of the debug view: saturation gauges to sit next to the RED metrics so a latency rise can be attributed to GC or goroutine growth without a release.

type InstanceStat

type InstanceStat struct {
	Instance     string
	Count        uint64
	ErrorRate    float64
	P50          float64
	P95          float64
	P99          float64
	ReqBytesAvg  float64
	RespBytesAvg float64
}

InstanceStat is one row of the instance-outlier breakdown: aggregate RED metrics plus the average payload sizes for a single instance (replica) of an endpoint over the query window. ErrorRate is a fraction in [0,1]; the ...Avg fields are per-request byte averages (sum(bytes)/count).

func InstancesForEndpoint

func InstancesForEndpoint(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
) ([]InstanceStat, error)

InstancesForEndpoint returns one InstanceStat per instance (replica) serving the given endpoint for tenant over [from, to), ordered by instance. method and route may be empty to aggregate across all methods / all route templates of the service. This is the flagship outlier query: it answers whether a degradation is confined to one replica or is fleet-wide. Percentiles use the frozen DDSketch convention; the error rate is (4xx+5xx+no_status)/total.

type InstanceWindowRow

type InstanceWindowRow struct {
	Tenant          tenant.ID
	Service         string
	Instance        string
	WindowStart     time.Time
	WindowEnd       time.Time
	CPUNs           uint64
	RSSBytes        uint64
	HeapAllocBytes  uint64
	GCPauseNs       uint64
	Goroutines      uint64
	NumGC           uint64  // completed GC cycles during the window (delta)
	TotalAllocBytes uint64  // heap bytes allocated during the window (delta)
	Mallocs         uint64  // heap objects allocated during the window (delta)
	GCCPUFraction   float64 // fraction of CPU time in GC since start (gauge, 0..1)
	HeapInuseBytes  uint64  // in-use heap bytes at sample time (gauge)
	GOMAXPROCS      uint32  // GOMAXPROCS at sample time (gauge)
	PostGCHeapBytes uint64  // live heap as of the last GC mark, post-GC baseline (gauge)
	RSSTrueBytes    uint64  // true resident set size from the OS, 0 if unavailable (gauge)
	OpenFDs         uint64  // open file-descriptor count, 0 if unavailable (gauge)
	FDLimit         uint64  // soft RLIMIT_NOFILE ceiling, 0 if unavailable (gauge)
	InFlight        uint64  // peak in-flight request concurrency during the window (gauge)
}

InstanceWindowRow is one process resource snapshot (USE gauges) ready for insertion into the instance_windows table: the server-resolved tenant plus service/instance and the per-window gauges. cpu_ns and gc_pause_ns are per-window deltas; the rest are point-in-time reads. It is a separate stream from Row (per-endpoint summaries).

type MemoryTrendPoint

type MemoryTrendPoint struct {
	TS              time.Time
	PostGCHeapBytes uint64
	HeapInuseBytes  uint64
}

MemoryTrendPoint is one time-bucket of the service's fleet memory: the peak post-GC live heap (the leak-vs-burst signal) and peak in-use heap over the bucket, across every instance/sample. Memory is a per-process property of the instances serving the service (instance_windows has no endpoint dimension), so this trend is service-scoped, not per-endpoint.

func MemoryTrendForService

func MemoryTrendForService(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service string,
	from, to time.Time,
	step time.Duration,
) ([]MemoryTrendPoint, error)

MemoryTrendForService returns the per-bucket fleet memory trend for a service over [from, to), bucketed by step, ordered by time. It reads the same [from, to, step] window the detail-page timeline uses so heap and latency line up. step is floored to one second; the caller passes the timeline step.

type NoStatusReason

type NoStatusReason struct {
	Reason uint8
	Count  uint64
}

NoStatusReason is one reason count for the no_status_reasons map column. Reason is the proto NoStatusReason enum value, stored as UInt8 (the domain is a tiny fixed enum).

type NoStatusReasonStat

type NoStatusReasonStat struct {
	Reason uint8
	Count  uint64
}

NoStatusReasonStat is one row of the NO_STATUS reason breakdown: the proto NoStatusReason enum value (as UInt8) and how many aborted requests it explains, telling apart timing-out vs canceling vs crashing.

func NoStatusReasonsForEndpoint

func NoStatusReasonsForEndpoint(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
) ([]NoStatusReasonStat, error)

NoStatusReasonsForEndpoint returns the NO_STATUS reason breakdown for the given endpoint for tenant over [from, to), ordered by reason. method and route may be empty to aggregate across all methods / all route templates of the service.

type OperatorQuery added in v0.11.0

type OperatorQuery struct {
	// contains filtered or unexported fields
}

OperatorQuery is the cross-tenant read surface for the operator console. It is the deliberate counterpart to TenantQuery: where Tenant(id) scopes every read to one validated tenant, Operator() reaches reads that span all tenants. Keeping it a distinct handle (rather than adding cross-tenant methods to the scoped path) makes every cross-tenant read explicit at the call site and greppable in review.

func (OperatorQuery) LastIngestByTenant added in v0.11.0

func (o OperatorQuery) LastIngestByTenant(ctx context.Context) (map[string]time.Time, error)

LastIngestByTenant forwards to the package-level cross-tenant last-ingest scan.

type PerformanceStat

type PerformanceStat struct {
	Requests         uint64
	Summaries        uint64
	SummaryDiskBytes uint64
}

PerformanceStat is the real, tenant-scoped basis for the performance page: how many requests the tenant's stored summaries represent, how many summary rows were actually shipped to carry them, and an estimate of the on-disk bytes those summaries occupy. Requests/Summaries are measured exactly; SummaryDiskBytes multiplies the tenant's summary-row count by the measured average on-disk size of a summary row (falling back to a constant when system.parts is unavailable).

func PerformanceStats

func PerformanceStats(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	from, to time.Time,
) (PerformanceStat, error)

PerformanceStats measures, for tenant over [from, to), the total requests the stored summaries represent (sum of the per-summary count) and the number of shipped summary rows, both from the raw summaries tier. It then estimates the on-disk size of those summaries from the measured average summary-row size (system.parts, best-effort — a failure falls back to a constant rather than failing the page). Their ratio is the compression the page reports; the caller projects the raw-event-pipeline size from Requests and a documented per-event assumption (mAPI-ng stores no raw events, so that side cannot be measured).

type QueryService

type QueryService struct {
	// contains filtered or unexported fields
}

QueryService wraps a ClickHouse connection to expose the read API as methods, so callers (the web layer) can depend on a small interface rather than a free function plus a raw driver.Conn.

func NewQueryService

func NewQueryService(conn driver.Conn) *QueryService

NewQueryService builds a QueryService over an open connection.

func (*QueryService) Operator added in v0.11.0

func (s *QueryService) Operator() OperatorQuery

Operator returns the cross-tenant operator read handle. Reads issued through it are not tenant-scoped and are intended only for the allowlisted operator console.

func (*QueryService) Tenant

func (s *QueryService) Tenant(id tenant.ID) TenantQuery

Tenant binds the service to a tenant, returning the scoped query handle through which every data-plane read is issued. The tenant is a validated tenant.ID, so a query can never be scoped to an unvalidated identity.

type Row

type Row struct {
	Tenant          tenant.ID
	Service         string
	Instance        string
	Method          string
	RouteTemplate   string
	StatusClass     string // Enum8 string value, e.g. "STATUS_CLASS_2XX".
	WindowStart     time.Time
	WindowEnd       time.Time
	Count           uint64
	SumDurationNs   uint64
	ReqBytes        uint64
	RespBytes       uint64
	Sketch          []SketchBucket   // sorted ascending by Index.
	StatusCodes     []StatusCode     // sorted ascending by Code.
	MaxDurationNs   uint64           // exact slowest request in the window; merges via max.
	Exemplars       []Exemplar       // bounded breadcrumbs; raw tier only.
	ErrorClasses    []ErrorClass     // sorted ascending by Class; merges via sumMap.
	NoStatusReasons []NoStatusReason // sorted ascending by Reason; merges via sumMap.
	SumDownstreamNs uint64           // summed downstream wait time; merges via sum.

	// Deploy identity: stored low-cardinality dimensions carried from the
	// Envelope. DeployVersion is a sort-key dimension (rows from different
	// versions never collapse); the rest are non-key stored columns.
	// InstanceStart is the process boot wall-clock (zero when the client did not
	// report one).
	DeployVersion string
	DeployID      string
	Environment   string
	Region        string
	InstanceStart time.Time
}

Row is one Summary ready for insertion: the server-resolved tenant plus the full series key, RED counters, and the merged-at-query DDSketch buckets. The sketch keys are pre-sorted (see NewRow) for deterministic map ordering.

func NewRow

func NewRow(
	tenantID tenant.ID,
	service, instance, method, routeTemplate, statusClass string,
	windowStart, windowEnd time.Time,
	count, sumDurationNs, reqBytes, respBytes uint64,
	sketch map[int32]uint64,
	statusCodes map[uint32]uint64,
	deployVersion, deployID, environment, region string,
	instanceStart time.Time,
	maxDurationNs uint64,
	exemplars []Exemplar,
	errorClasses map[string]uint64,
	noStatusReasons map[uint32]uint64,
	sumDownstreamNs uint64,
) Row

NewRow builds a Row from an already-mapped sketch and status-code maps, sorting both by key so the ClickHouse map column ordering is deterministic across inserts.

type ServiceStat

type ServiceStat struct {
	Service       string
	Count         uint64
	ErrorRate     float64
	Client4xxRate float64
	P50           float64
	P95           float64
	P99           float64
}

ServiceStat is one row of the service-overview level: aggregate RED metrics for a whole service over the query window. ErrorRate is a fraction in [0,1].

func Services

func Services(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	from, to time.Time,
) ([]ServiceStat, error)

Services returns one aggregate ServiceStat per service for tenant over [from, to), ordered by Count descending. Percentiles use the frozen DDSketch convention; the error rate is the CONTEXT (4xx+5xx+no_status)/total.

type SketchBucket

type SketchBucket struct {
	Index int32
	Count uint64
}

SketchBucket is one DDSketch bucket: an index and its count. Rows carry the sketch as a sorted slice (not a map) so the map column insert order into ClickHouse is deterministic.

type StatusClassCount

type StatusClassCount struct {
	Class string
	Count uint64
}

StatusClassCount is the merged request count for one status class label (2xx/3xx/4xx/5xx/no_status), used for the breakdown beside the error rate.

type StatusCode

type StatusCode struct {
	Code  uint32
	Count uint64
}

StatusCode is one exact-code count for the status_codes map column.

type TenantQuery

type TenantQuery struct {
	// contains filtered or unexported fields
}

TenantQuery is a QueryService bound to a single tenant. It is the ONLY way to reach the data-plane read API: QueryService exposes no tenant-taking methods, so a cross-tenant read is unrepresentable — callers must go through Tenant(tenant) and the bound tenant threads into every query.

func (TenantQuery) DownstreamForEndpoint

func (q TenantQuery) DownstreamForEndpoint(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
) (DownstreamStat, error)

DownstreamForEndpoint forwards to the package-level self-vs-downstream time split for the bound tenant.

func (TenantQuery) EndpointDetail

func (q TenantQuery) EndpointDetail(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
) (EndpointDetail, error)

EndpointDetail forwards to the package-level EndpointDetail aggregate for the bound tenant.

func (TenantQuery) Endpoints

func (q TenantQuery) Endpoints(
	ctx context.Context,
	service string,
	from, to time.Time,
) ([]EndpointStat, error)

Endpoints forwards to the package-level Endpoints aggregate for the bound tenant.

func (TenantQuery) ErrorClassesForEndpoint

func (q TenantQuery) ErrorClassesForEndpoint(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
) ([]ErrorClassStat, error)

ErrorClassesForEndpoint forwards to the package-level error-class breakdown for the bound tenant.

func (TenantQuery) ExemplarsForEndpoint

func (q TenantQuery) ExemplarsForEndpoint(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
) ([]ExemplarRow, error)

ExemplarsForEndpoint forwards to the package-level raw-tier exemplar breadcrumb query for the bound tenant.

func (TenantQuery) HasAnySummary

func (q TenantQuery) HasAnySummary(ctx context.Context) (bool, error)

HasAnySummary forwards to the package-level existence check for the bound tenant.

func (TenantQuery) InstanceResourcesForService

func (q TenantQuery) InstanceResourcesForService(
	ctx context.Context,
	service string,
	from, to time.Time,
) ([]InstanceResourceStat, error)

InstanceResourcesForService forwards to the package-level per-instance USE gauge breakdown for the bound tenant.

func (TenantQuery) InstancesForEndpoint

func (q TenantQuery) InstancesForEndpoint(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
) ([]InstanceStat, error)

InstancesForEndpoint forwards to the package-level instance-outlier breakdown for the bound tenant.

func (TenantQuery) LatencyByStatusClass

func (q TenantQuery) LatencyByStatusClass(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
) (map[string]ClassLatency, error)

LatencyByStatusClass forwards to the package-level per-class latency split for the bound tenant.

func (TenantQuery) MemoryTrendForService

func (q TenantQuery) MemoryTrendForService(
	ctx context.Context,
	service string,
	from, to time.Time,
	step time.Duration,
) ([]MemoryTrendPoint, error)

MemoryTrendForService forwards to the package-level per-window memory trend for the bound tenant.

func (TenantQuery) NoStatusReasonsForEndpoint

func (q TenantQuery) NoStatusReasonsForEndpoint(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
) ([]NoStatusReasonStat, error)

NoStatusReasonsForEndpoint forwards to the package-level NO_STATUS reason breakdown for the bound tenant.

func (TenantQuery) PerformanceStats

func (q TenantQuery) PerformanceStats(
	ctx context.Context,
	from, to time.Time,
) (PerformanceStat, error)

PerformanceStats forwards to the package-level performance basis (represented requests, shipped summaries, estimated summary disk) for the bound tenant.

func (TenantQuery) SeriesOverTime

func (q TenantQuery) SeriesOverTime(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
	step time.Duration,
) ([]TimePoint, error)

SeriesOverTime forwards to the package-level query using the wrapped connection and the bound tenant.

func (TenantQuery) Services

func (q TenantQuery) Services(
	ctx context.Context,
	from, to time.Time,
) ([]ServiceStat, error)

Services forwards to the package-level Services aggregate using the wrapped connection and the bound tenant.

func (TenantQuery) Usage added in v0.11.0

func (q TenantQuery) Usage(ctx context.Context, now time.Time) (TenantUsage, error)

Usage forwards to the package-level operator volumetry for the bound tenant.

func (TenantQuery) VersionsForEndpoint

func (q TenantQuery) VersionsForEndpoint(
	ctx context.Context,
	service, method, route string,
	from, to time.Time,
) ([]VersionStat, error)

VersionsForEndpoint forwards to the package-level per-deploy-version breakdown for the bound tenant.

type TenantUsage added in v0.11.0

type TenantUsage struct {
	FirstIngest time.Time
	LastIngest  time.Time
	Endpoints   uint64
	Series      uint64
	Services    uint64
	Instances   uint64
	Requests30d uint64
	DiskBytes   uint64
}

TenantUsage is the operator-facing volumetry for one tenant: liveness (First/LastIngest), current cardinality (Endpoints/Series/Services/Instances), recent activity (Requests over the last 30 days), and an estimate of the on-disk bytes the tenant's raw summaries occupy. Series counts distinct (method, route_template, status_class) — the exact key the cardinality guardrail meters (guardrail.SeriesKey), so a caller can honestly render "series vs cap". A never-ingested tenant yields the zero value (zero times, zero counts).

func Usage added in v0.11.0

func Usage(ctx context.Context, conn driver.Conn, tenantID tenant.ID, now time.Time) (TenantUsage, error)

Usage assembles the operator volumetry for tenantID as of now. It runs three cheap aggregates: liveness from the 1d tier (longest retention, so a churned tenant still shows its last-ingest date), cardinality and 30-day requests from the 1m tier, and the disk estimate from the raw tier via PerformanceStats. now is a parameter (not time.Now()) so the windows are deterministic in tests.

type TimePoint

type TimePoint struct {
	TS        time.Time
	Count     uint64
	ErrorRate float64
	P50       float64
	P95       float64
	P99       float64
}

TimePoint is one time-bucket of the RED distribution: request rate inputs (Count over the step), the derived error rate, and p50/p95/p99 latency in seconds computed from the merged DDSketch buckets.

func SeriesOverTime

func SeriesOverTime(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
	step time.Duration,
) ([]TimePoint, error)

SeriesOverTime returns the per-time-bucket RED distribution for a series over [from, to), bucketed by step. method and route may be empty to aggregate across all methods / all route templates of the service. Percentiles use the frozen DDSketch convention above; the same math runs client-side, which is the whole correctness point.

type VersionStat

type VersionStat struct {
	Version   string
	Count     uint64
	ErrorRate float64
	P50       float64
	P95       float64
	P99       float64
}

VersionStat is one row of the per-deploy-version breakdown: aggregate RED metrics for a single deploy_version of an endpoint over the query window. ErrorRate is a fraction in [0,1]. It answers "did release X regress this endpoint?" by comparing metrics across versions.

func VersionsForEndpoint

func VersionsForEndpoint(
	ctx context.Context,
	conn driver.Conn,
	tenantID tenant.ID,
	service, method, route string,
	from, to time.Time,
) ([]VersionStat, error)

VersionsForEndpoint returns one VersionStat per deploy_version serving the given endpoint for tenant over [from, to), ordered by deploy_version. method and route may be empty to aggregate across all methods / all route templates of the service. It answers whether a release regressed an endpoint by putting each version's RED metrics side by side. Percentiles use the frozen DDSketch convention; the error rate is (4xx+5xx+no_status)/total.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer batches Row inserts into ClickHouse. A single background goroutine drains an unbounded-intent buffered channel and flushes on either the flush interval or the row threshold, whichever comes first. Close drains and flushes remaining rows within the caller's context deadline.

func NewWriter

func NewWriter(ctx context.Context, cfg Config, log *slog.Logger) (*Writer, error)

NewWriter opens a ClickHouse connection from cfg and starts the batcher goroutine. The caller owns the returned Writer and must call Close.

func (*Writer) Close

func (w *Writer) Close(ctx context.Context) error

Close signals the batcher to drain the buffered rows and stop, then waits for the run loop to exit or ctx to expire.

func (*Writer) Enqueue

func (w *Writer) Enqueue(row Row) error

Enqueue hands a row to the batcher. It returns ErrEmptyTenant if the row has no tenant (fail-closed, never persisted) and ErrWriterClosed once Close has been called. It never blocks indefinitely: if the buffer is full it blocks only until the batcher drains, which is bounded by the flush cadence.

func (*Writer) EnqueueInstanceWindow

func (w *Writer) EnqueueInstanceWindow(row InstanceWindowRow) error

EnqueueInstanceWindow hands one USE-gauge row to the batcher's instance-window stream. Like Enqueue it returns ErrWriterClosed after Close and never blocks indefinitely (bounded by the flush cadence).

func (*Writer) Migrate

func (w *Writer) Migrate(ctx context.Context, log *slog.Logger) error

Migrate applies the embedded ClickHouse schema on the writer's connection. Keeping the raw connection inside the storage package means callers cannot run un-scoped SQL against the data plane: reads go through QueryService.Tenant and writes through Enqueue.

func (*Writer) QueryService

func (w *Writer) QueryService() *QueryService

QueryService returns a read handle over the writer's connection. Reads remain tenant-scoped — callers reach a query only via QueryService.Tenant(tenant) — so the raw connection never leaves the storage package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL