Documentation
¶
Overview ¶
Package loge is an embedded log-storage engine: it ingests log payloads into rotating SQLite segments, compresses them with zstd, optionally tiers them to S3, and serves LogQL-style queries over the local and remote segments.
Index ¶
- func MarshalLabels(tags map[string]string) string
- func ParseSelector(input string) ([]managers.Matcher, string, error)
- func RemoveWAL(dir string) error
- func ReplaySegments(segments []string, fn func(*Payload)) (int, error)
- func ReplayWAL(dir string, fn func(*Payload)) (int, error)
- type BucketOption
- func WithDurableReport(fn func(filename string, seqs []uint64)) BucketOption
- func WithFlushCompression(level zstd.EncoderLevel) BucketOption
- func WithFlushCompressionName(name string) (BucketOption, error)
- func WithFlushInterval(interval time.Duration) BucketOption
- func WithFlushQueue(n int) BucketOption
- func WithFlushWorkers(n int) BucketOption
- func WithWAL(wal *WAL) BucketOption
- type Buckets
- type CLI
- type Checkpointer
- type Compactor
- type CompactorOption
- type DefaultJSONSerializer
- type Entry
- type LabelResponse
- func (z *LabelResponse) DecodeMsg(dc *msgp.Reader) (err error)
- func (z *LabelResponse) EncodeMsg(en *msgp.Writer) (err error)
- func (z *LabelResponse) MarshalMsg(b []byte) (o []byte, err error)
- func (z *LabelResponse) Msgsize() (s int)
- func (z *LabelResponse) UnmarshalMsg(bts []byte) (o []byte, err error)
- type ObjectStore
- type Payload
- func (z *Payload) DecodeMsg(dc *msgp.Reader) (err error)
- func (z *Payload) EncodeMsg(en *msgp.Writer) (err error)
- func (z *Payload) MarshalMsg(b []byte) (o []byte, err error)
- func (z *Payload) Msgsize() (s int)
- func (z *Payload) UnmarshalMsg(bts []byte) (o []byte, err error)
- func (p *Payload) Valid() (string, bool)
- type PlanResponse
- type PlanSegment
- type QueryResponse
- type S3Config
- type S3Store
- func (s *S3Store) List(ctx context.Context, prefix string) ([]string, error)
- func (s *S3Store) Presign(ctx context.Context, key string, expiry time.Duration) (string, error)
- func (s *S3Store) Put(ctx context.Context, key, localPath string) (string, error)
- func (s *S3Store) ReadURL(key string) string
- func (s *S3Store) Size(ctx context.Context, key string) (int64, error)
- type SearchCmd
- type ServeCmd
- type StatsResponse
- type Stream
- type Streams
- type Uploader
- type UploaderOption
- type Value
- func (z *Value) DecodeMsg(dc *msgp.Reader) (err error)
- func (z *Value) EncodeMsg(en *msgp.Writer) (err error)
- func (z *Value) MarshalMsg(b []byte) (o []byte, err error)
- func (z *Value) Msgsize() (s int)
- func (v *Value) Timestamp() (int64, error)
- func (z *Value) UnmarshalMsg(bts []byte) (o []byte, err error)
- type Values
- type WAL
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func MarshalLabels ¶
MarshalLabels renders a label set as a JSON object string, escaping keys and values. Key order follows Go's map iteration order and is not sorted.
func ParseSelector ¶
ParseSelector parses a LogQL-style selector into stream-label matchers and an optional substring line filter. The accepted grammar is:
{name op "value", ...} |= "keyword"
where op is one of =, !=, =~, !~ (the four stream-label operators), and the optional trailing |= "keyword" is the only supported line filter — a substring match (a LIKE scan, with a per-segment binary-fuse trigram filter pruning segments that cannot contain it). The brace block may be empty ({}) or omitted entirely when only a |= filter is given. The keyword must be at least minTrigramLen characters so the trigram filter applies.
func RemoveWAL ¶
RemoveWAL deletes all log segments in dir. It is safe to call only after the ingest pipeline has durably flushed every replayed and appended payload (i.e. after a clean shutdown).
func ReplaySegments ¶
ReplaySegments reads the given segments in order and invokes fn for each intact record. A torn tail (short read or CRC mismatch) ends a segment's replay without erroring, since it represents an interrupted write. It returns the number of records replayed.
Types ¶
type BucketOption ¶
type BucketOption func(*Buckets)
BucketOption configures optional Buckets behaviour.
func WithDurableReport ¶
func WithDurableReport(fn func(filename string, seqs []uint64)) BucketOption
WithDurableReport registers a callback invoked after a file has been flushed and compressed, reporting the WAL sequence numbers it contains so they can be checkpointed.
func WithFlushCompression ¶
func WithFlushCompression(level zstd.EncoderLevel) BucketOption
WithFlushCompression overrides the zstd level used to compress short-lived flush files on the latency-sensitive ingest path. Lower levels (SpeedFastest/SpeedDefault) cut compressor-worker CPU under sustained load; the durable on-disk/S3 size is governed by compaction (SpeedBestCompression), which recompresses surviving segments, so the flush-tier level trades hot-path CPU for the size of transient, soon-recompacted files.
func WithFlushCompressionName ¶
func WithFlushCompressionName(name string) (BucketOption, error)
WithFlushCompressionName returns a flush-compression BucketOption selected by CLI level name (fastest/default/better/best). An unknown name is an error.
func WithFlushInterval ¶
func WithFlushInterval(interval time.Duration) BucketOption
WithFlushInterval overrides how often a bucket worker flushes a non-empty batch. A non-positive duration is ignored.
func WithFlushQueue ¶
func WithFlushQueue(n int) BucketOption
WithFlushQueue overrides the depth of the flush and compress job queues, which default to the bucket count. A deeper queue absorbs ingest bursts before backpressure engages, at the cost of memory; a non-positive value keeps the default.
func WithFlushWorkers ¶
func WithFlushWorkers(n int) BucketOption
WithFlushWorkers overrides the number of flush and compress worker goroutines. They default to max(buckets/2, 2); a non-positive value keeps that default. This is the main lever for flush throughput on a machine with more CPUs than buckets/2, decoupling flush parallelism from the (memory-heavy) bucket count.
func WithWAL ¶
func WithWAL(wal *WAL) BucketOption
WithWAL makes Append durably log each payload (and assign it a sequence number) before queueing it.
type Buckets ¶
type Buckets struct {
// contains filtered or unexported fields
}
Buckets is the ingest front end: it shards incoming payloads across a fixed set of buckets, each batching and flushing to its own SQLite segment files.
func NewBuckets ¶
func NewBuckets( ctx context.Context, size int, payloadSize int, outputPath string, dropOnBackpressure bool, opts ...BucketOption, ) (*Buckets, error)
NewBuckets creates size buckets that flush under outputPath, batching up to payloadSize rows per segment. When dropOnBackpressure is set, Append drops payloads instead of blocking once the bucket queues are full.
type CLI ¶
type CLI struct {
Serve ServeCmd `cmd:"" default:"withargs" help:"run the loge ingest+query HTTP server (default)"`
Search SearchCmd `cmd:"" help:"query a running loge server with a LogQL-style selector"`
}
CLI is the top-level command. The server is the default command (so bare flags like `loge --port 6500` still start it), and `loge search` queries a running server. CLI itself has no Run() method on purpose: kong invokes the Run() of every selected node's ancestors, so a CLI.Run() would also fire for the search subcommand.
type Checkpointer ¶
type Checkpointer struct {
// contains filtered or unexported fields
}
Checkpointer bounds the write-ahead log. It receives reports that segments have been written, fsyncs those segments (so the data is durable on its own), then advances the WAL's durable watermark and lets it delete the now-redundant log segments. Reports for files that were already compacted away are treated as durable, since the compactor fsyncs its merged segment before deleting the sources.
func NewCheckpointer ¶
func NewCheckpointer(wal *WAL, interval time.Duration) *Checkpointer
NewCheckpointer creates a checkpointer for wal. A non-positive interval uses the default.
func (*Checkpointer) Report ¶
func (c *Checkpointer) Report(filename string, seqs []uint64)
Report notes that filename is now a durable segment containing seqs. It is the callback passed to buckets via WithDurableReport and is safe to call after Stop (the report is simply dropped).
func (*Checkpointer) Stop ¶
func (c *Checkpointer) Stop()
Stop ends the checkpointer. It must be called only after the flush/compress pipeline has drained (so no further reports arrive).
type Compactor ¶
type Compactor struct {
// contains filtered or unexported fields
}
Compactor merges the many small per-flush SQLite files into fewer, larger time-local "segment" files, building the expensive trigram line index and the timestamp/label indexes once per segment instead of once per flush.
func NewCompactor ¶
func NewCompactor(dir string, minFiles, maxFiles int, interval time.Duration, opts ...CompactorOption) *Compactor
NewCompactor builds a Compactor for dir. Zero values select defaults.
type CompactorOption ¶
type CompactorOption func(*Compactor)
CompactorOption configures optional Compactor behaviour.
func WithCompactorCatalog ¶
func WithCompactorCatalog(catalog *managers.Catalog) CompactorOption
WithCompactorCatalog records each new segment in the catalog so the query path can prune by time without opening files.
type DefaultJSONSerializer ¶
type DefaultJSONSerializer struct{}
DefaultJSONSerializer implements JSON encoding using github.com/goccy/go-json.
func (DefaultJSONSerializer) Deserialize ¶
func (d DefaultJSONSerializer) Deserialize(c *echo.Context, i interface{}) error
Deserialize reads a JSON from a request body and converts it into an interface.
type Entry ¶
type Entry struct {
Stream Stream `json:"stream" msg:"stream"`
Values Values `json:"values" msg:"values"`
}
Entry pairs a stream's labels with its log values.
func (*Entry) MarshalMsg ¶
MarshalMsg implements msgp.Marshaler
type LabelResponse ¶
type LabelResponse struct {
Status string `json:"status" msg:"status"`
Data []string `json:"data" msg:"data"`
}
LabelResponse is the JSON body returned by the label-listing endpoints.
func (*LabelResponse) DecodeMsg ¶
func (z *LabelResponse) DecodeMsg(dc *msgp.Reader) (err error)
DecodeMsg implements msgp.Decodable
func (*LabelResponse) EncodeMsg ¶
func (z *LabelResponse) EncodeMsg(en *msgp.Writer) (err error)
EncodeMsg implements msgp.Encodable
func (*LabelResponse) MarshalMsg ¶
func (z *LabelResponse) MarshalMsg(b []byte) (o []byte, err error)
MarshalMsg implements msgp.Marshaler
func (*LabelResponse) Msgsize ¶
func (z *LabelResponse) Msgsize() (s int)
Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (*LabelResponse) UnmarshalMsg ¶
func (z *LabelResponse) UnmarshalMsg(bts []byte) (o []byte, err error)
UnmarshalMsg implements msgp.Unmarshaler
type ObjectStore ¶
type ObjectStore interface {
// Put uploads the file at localPath under key and returns the public URL it
// can be read back from over HTTP.
Put(ctx context.Context, key, localPath string) (readURL string, err error)
// Size returns the size of the stored object, for upload verification.
Size(ctx context.Context, key string) (int64, error)
// List returns the object keys stored under prefix.
List(ctx context.Context, prefix string) ([]string, error)
// ReadURL returns the public HTTP URL a key is read back from.
ReadURL(key string) string
}
ObjectStore is the remote storage the uploader rotates segments to. It is an interface so rotation can be tested without real S3.
type Payload ¶
type Payload struct {
Streams Streams `json:"streams" msg:"streams"`
}
Payload is the wire form of a push request: one or more labeled streams.
func (*Payload) MarshalMsg ¶
MarshalMsg implements msgp.Marshaler
func (*Payload) Msgsize ¶
Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
func (*Payload) UnmarshalMsg ¶
UnmarshalMsg implements msgp.Unmarshaler
type PlanResponse ¶
type PlanResponse struct {
Status string `json:"status"`
Hot []managers.QueryEntry `json:"hot"`
Segments []PlanSegment `json:"segments"`
ExpiresAt int64 `json:"expires_at"`
}
PlanResponse answers a client-side search plan request. The server scans the hot tier itself (Hot) and hands the client the pruned, presigned cold segments (Segments) to scan on its own machine; ExpiresAt is when the presigned URLs stop working.
type PlanSegment ¶
PlanSegment is one cold-tier (S3) segment in a client-side search plan: its catalog ID and a short-lived presigned URL the client reads it from directly.
type QueryResponse ¶
type QueryResponse struct {
Status string `json:"status"`
Data []managers.QueryEntry `json:"data"`
}
QueryResponse is the JSON shape returned by the query endpoint.
type S3Config ¶
type S3Config struct {
Bucket string
Prefix string
Endpoint string // custom endpoint for S3-compatible stores (MinIO); empty = AWS
Region string
ForcePathStyle bool
ReadURLBase string // public/CDN base for reads; empty = derived
ACL string // optional canned ACL, e.g. "public-read"
}
S3Config configures an S3Store. Credentials come from the standard AWS chain (environment, shared config, IAM role) — never from flags.
type S3Store ¶
type S3Store struct {
// contains filtered or unexported fields
}
S3Store is an ObjectStore backed by S3 (or any S3-compatible endpoint) via aws-sdk-go-v2. Reads happen out-of-band as plain public HTTP GETs (see ReadURLBase), so the store is only used for the write/verify path.
func NewS3Store ¶
NewS3Store builds an S3Store from cfg.
func (*S3Store) List ¶
List returns every object key stored under prefix, paginating through the bucket listing.
func (*S3Store) Presign ¶
Presign returns a short-lived presigned GET URL for key, valid for expiry. The URL carries the SigV4 signature in its query string, so a client can read a private object directly from S3 without AWS credentials. It is signed against the real S3 endpoint/bucket (not ReadURLBase/CDN), so it works for private buckets, and because Range is not a signed header the same URL serves every range read of a segment.
type SearchCmd ¶
type SearchCmd struct {
Selector string `arg:"" help:"LogQL-style selector, e.g. '{app=\"web\"} |= \"timeout\"'"`
Addr string `default:"http://localhost:3000" help:"base URL of the loge server"`
Since time.Duration `help:"relative window start, e.g. 1h (from now); ignored if --start is set"`
Until time.Duration `help:"relative window end, e.g. 5m ago (from now); ignored if --end is set"`
Start string `help:"absolute window start: RFC3339 or unix nanoseconds; overrides --since"`
End string `help:"absolute window end: RFC3339 or unix nanoseconds; overrides --until"`
Limit int `default:"100" help:"max results (server caps at 5000)"`
Output string `` /* 138-byte string literal not displayed */
// Client-side ("--local") search: instead of having the server scan
// everything, fetch a plan and scan the cold (S3) segments on this machine,
// moving the heavy historical scan off the server.
Local bool `help:"scan the cold (S3) segments on this machine; the server only scans its hot tier"`
APIKey string `` /* 171-byte string literal not displayed */
Concurrency int `` /* 175-byte string literal not displayed */
FrameCacheSize int `` /* 180-byte string literal not displayed */
// Out is where results are written; nil defaults to os.Stdout. Excluded from
// kong flag parsing so tests can capture output.
Out io.Writer `kong:"-"`
}
SearchCmd queries a running loge server over its HTTP API using a LogQL-style selector. It is a thin client: it builds the same managers.QueryRequest the server's POST /api/v1/query handler consumes, so CLI results match the server.
type ServeCmd ¶
type ServeCmd struct {
Port int `` /* 172-byte string literal not displayed */
Buckets int `` /* 172-byte string literal not displayed */
PayloadSize int `` /* 172-byte string literal not displayed */
OutputPath string `` /* 172-byte string literal not displayed */
DropOnBackpressure bool `default:"false" help:"drop data instead of blocking when backpressure occurs"`
FlushInterval time.Duration `default:"1s" help:"how often a bucket flushes a non-empty batch"`
FlushCompression string `` /* 334-byte string literal not displayed */
FlushWorkers int `` /* 153-byte string literal not displayed */
FlushQueue int `` /* 160-byte string literal not displayed */
CompactInterval time.Duration `default:"30s" help:"how often to compact small files into segments (0 disables)"`
CompactMinFiles int `default:"8" help:"minimum number of flush files before a compaction pass runs"`
Durable bool `default:"true" help:"write-ahead log each payload (fsync) before acknowledging; disable for faster, lossy-on-crash ingest"`
CheckpointInterval time.Duration `default:"2s" help:"how often to fsync new segments and prune the write-ahead log"`
QueryConcurrency int `default:"8" help:"max segments a query opens in parallel"`
// Profiling (opt-in). The pprof listener binds to loopback only because
// heap profiles can expose log contents.
PprofPort int `default:"0" help:"serve net/http/pprof on 127.0.0.1:<port> (0 disables)" name:"pprof-port"`
PprofBlockRate int `default:"0" help:"runtime.SetBlockProfileRate sampling rate in ns (0 disables block profiling)" name:"pprof-block-rate"`
PprofMutexFraction int `` /* 127-byte string literal not displayed */
// S3 tiered storage (rotate old segments to S3, read them back over HTTP).
// Leave --s3-bucket empty to keep everything local.
S3Bucket string `` /* 197-byte string literal not displayed */
S3Prefix string `default:"loge/" help:"key prefix for uploaded segments" name:"s3-prefix"`
S3Endpoint string `` /* 199-byte string literal not displayed */
S3Region string `` /* 197-byte string literal not displayed */
S3ForcePathStyle bool `` /* 134-byte string literal not displayed */
S3ReadURLBase string `` /* 131-byte string literal not displayed */
S3ACL string `default:"" help:"canned ACL for uploaded objects (e.g. public-read); empty relies on bucket policy" name:"s3-acl"`
S3RotateAge time.Duration `` /* 128-byte string literal not displayed */
S3RotateInterval time.Duration `` /* 133-byte string literal not displayed */
S3RotateGrace time.Duration `` /* 130-byte string literal not displayed */
S3FrameCacheSize int `` /* 134-byte string literal not displayed */
S3PresignExpiry time.Duration `` /* 132-byte string literal not displayed */
// APIKey gates the read + search endpoints (/query, /labels, /stats,
// /search/plan) and the web UI's data calls. When set, callers must present
// it as a bearer token; when empty everything is unauthenticated (matching
// the trusted-network posture). Ingest (/push) and the static UI assets stay
// open regardless, so the login page can load.
APIKey string `` /* 159-byte string literal not displayed */
}
ServeCmd runs the ingest+query HTTP server. Its fields are every server flag.
type StatsResponse ¶
type StatsResponse struct {
Status string `json:"status"`
SegmentsTotal int `json:"segments_total"`
SegmentsLocal int `json:"segments_local"`
SegmentsRemote int `json:"segments_remote"`
RowsTotal int64 `json:"rows_total"`
MinTimestamp int64 `json:"min_timestamp"`
MaxTimestamp int64 `json:"max_timestamp"`
}
StatsResponse summarizes the catalog (segment tiering, row count, time span) for benchmarking and observability.
type Stream ¶
Stream is the set of label key/value pairs identifying a log stream.
func (Stream) MarshalMsg ¶
MarshalMsg implements msgp.Marshaler
type Streams ¶
type Streams []Entry
Streams is a list of label-keyed entries, the body of a Payload.
func (Streams) MarshalMsg ¶
MarshalMsg implements msgp.Marshaler
type Uploader ¶
type Uploader struct {
// contains filtered or unexported fields
}
Uploader rotates compacted segments older than a threshold to an ObjectStore (cold tier), flips the catalog to point at the remote copy, and deletes the local copy after a grace window. Recent segments stay local (hot tier).
func NewUploader ¶
func NewUploader(dir string, catalog *managers.Catalog, store ObjectStore, opts ...UploaderOption) *Uploader
NewUploader builds an Uploader for dir.
func (*Uploader) ReconcileRemote ¶
ReconcileRemote rebuilds catalog rows for remote (S3) segments from a bucket listing alone, so a fresh server with an empty catalog rediscovers cold-tier segments without opening any file. Keys whose bounds are encoded in the filename are cataloged from the name; legacy-named keys fall back to opening the segment over HTTP (the expensive path). It returns how many remote rows it added. Run it once at startup before the rotation loop starts.
func (*Uploader) Rotate ¶
Rotate uploads every local segment older than the rotate age and flips the catalog to remote. A failed upload is logged and retried next cycle (the catalog is only flipped after the upload is verified), so an S3 outage never loses data — the segment just stays local.
type UploaderOption ¶
type UploaderOption func(*Uploader)
UploaderOption configures an Uploader.
func WithRotateAge ¶
func WithRotateAge(d time.Duration) UploaderOption
WithRotateAge sets how old (since sealing) a local segment must be before it rotates to remote storage.
func WithRotateGrace ¶
func WithRotateGrace(d time.Duration) UploaderOption
WithRotateGrace sets how long a rotated segment's local copy is kept (so in-flight queries that resolved it locally finish) before deletion.
func WithRotateInterval ¶
func WithRotateInterval(d time.Duration) UploaderOption
WithRotateInterval sets how often the rotation loop runs.
func WithUploadPrefix ¶
func WithUploadPrefix(prefix string) UploaderOption
WithUploadPrefix sets the key prefix for uploaded objects.
func WithUploaderVFS ¶
func WithUploaderVFS(name string) UploaderOption
WithUploaderVFS sets the sqlite VFS name used to open remote segments when ReconcileRemote must fall back to reading a legacy-named segment over HTTP.
type Value ¶
type Value [2]string
Value is a single log entry: a [timestamp, line] pair, both as strings.
func (*Value) MarshalMsg ¶
MarshalMsg implements msgp.Marshaler
func (*Value) Msgsize ¶
Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
type Values ¶
type Values []Value
Values is an ordered list of log entries within a stream.
func (Values) MarshalMsg ¶
MarshalMsg implements msgp.Marshaler
type WAL ¶
type WAL struct {
// contains filtered or unexported fields
}
WAL is an append-only, group-committing write-ahead log. A single writer goroutine batches concurrent appends and fsyncs them together, assigning each a sequence number.
func OpenWAL ¶
OpenWAL starts a WAL writer for dir, creating the directory if needed. New records are written to fresh segments; any pre-existing segments are recorded as "recovered" for ReplaySegments and are only removed by RemoveWAL after a clean shutdown.
func (*WAL) Append ¶
Append durably writes payload to the log and returns its sequence number once it (and any batched concurrent appends) have been fsynced.
func (*WAL) MarkDurable ¶
MarkDurable records that the given sequence numbers have reached a durable segment, advances the contiguous watermark, and deletes any sealed WAL segments that lie entirely below it. Sequence 0 (untracked/replayed payloads) is ignored.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
loge-loadgen
command
Command loge-loadgen generates realistic web-access and structured JSON app logs and pushes them to a loge server's /api/v1/push endpoint.
|
Command loge-loadgen generates realistic web-access and structured JSON app logs and pushes them to a loge server's /api/v1/push endpoint. |
|
Package filewatcher tracks the set of files in a directory that match a pattern, keeping it current as files are created, removed, or renamed, and lets callers iterate over the live set.
|
Package filewatcher tracks the set of files in a directory that match a pattern, keeping it current as files are created, removed, or renamed, and lets callers iterate over the live set. |
|
Command loge is the CLI entry point for the loge log-storage engine: it parses the command line and dispatches to the selected subcommand.
|
Command loge is the CLI entry point for the loge log-storage engine: it parses the command line and dispatches to the selected subcommand. |
|
Package managers indexes the engine's SQLite segments in a catalog and answers queries by fanning out across the local and S3-tiered segments.
|
Package managers indexes the engine's SQLite segments in a catalog and answers queries by fanning out across the local and S3-tiered segments. |
|
Package slogloge provides a log/slog Handler that ships structured logs to a loge server's /api/v1/push endpoint.
|
Package slogloge provides a log/slog Handler that ships structured logs to a loge server's /api/v1/push endpoint. |