Documentation
¶
Overview ¶
Package patterns contains the declarative state-machine matcher used by the multiline aggregator, together with the bundled stack-trace definitions for common languages (see All).
A matcher is described as one or more StateSet values and compiled with Compile into an immutable StateMachine, which implements the multiline.Matcher interface. Custom formats are declared exactly like the bundled sets in this package:
set := patterns.StateSet{Name: "tx", States: []patterns.State{
{Name: patterns.StartState, Transitions: []patterns.Transition{
{Pattern: `^BEGIN TX`, Next: "body"},
}},
{Name: "body", Transitions: []patterns.Transition{
{Pattern: `^\s`, Next: "body"},
{Pattern: `^(COMMIT|ROLLBACK)`, Next: "body"},
}},
}}
m, err := patterns.Compile(append(patterns.All, set)...)
Index ¶
Examples ¶
Constants ¶
const StartState = "start_state"
StartState is the name of the entry state where every group begins. Each set's transitions declared on StartState are merged into the single, shared start state; all other state names are private to their set.
Variables ¶
All lists the bundled state sets; the default matcher of multiline.New is MustCompile(All...). Compile a subset to aggregate only some formats, or append your own sets to extend it. Like the individual set variables, All is exported data: treat it as read-only and build new slices instead of mutating it.
var DotNet = StateSet{Name: "dotnet", States: []State{ { Name: StartState, Transitions: []Transition{ {Pattern: `^Unhandled exception\. .+Exception`, Next: "message"}, {Pattern: `^Unhandled exception\. .+Exception`, Next: "cont"}, }, }, { Name: "message", NonTerminal: true, Transitions: []Transition{ {Pattern: `.+`, Next: "cont"}, }, }, {Name: "cont", NonTerminal: true, Transitions: dotnetBody}, {Name: "frame", Transitions: dotnetBody}, { Name: "mark", Transitions: []Transition{ {Pattern: `^ at `, Next: "frame"}, }, }, }}
DotNet matches .NET unhandled-exception stack traces. The exception message may span one extra line before the frames start ("cont"); the double transition out of the start state keeps both readings alive until a frame line decides.
var Go = StateSet{Name: "go", States: []State{ { Name: StartState, Transitions: []Transition{ {Pattern: `\bpanic: `, Next: "after_panic"}, {Pattern: `http: panic serving`, Next: "goroutine"}, }, }, { Name: "after_panic", Transitions: append([]Transition{ {Pattern: `^\[signal `, Next: "after_signal"}, }, goBlank...), }, {Name: "after_signal", Transitions: goBlank}, { Name: "goroutine", Transitions: []Transition{ {Pattern: `^goroutine \d+ \[[^\]]+\]:$`, Next: "function"}, }, }, { Name: "function", Transitions: append([]Transition{ {Pattern: `^(?:[^\s.:]+\.)*[^\s.():]+\(|^created by `, Next: "location"}, }, goBlank...), }, { Name: "location", Transitions: []Transition{ {Pattern: `^\s`, Next: "function"}, }, }, }}
Go matches Go runtime panics and goroutine dumps.
var Java = StateSet{Name: "java", States: []State{ {Name: StartState, Transitions: javaHeader}, { Name: "after_exception", Transitions: append([]Transition{ {Pattern: `^[\t ]*nested exception is:[\t ]*$`, Next: "nested"}, }, javaFrames...), }, {Name: "nested", NonTerminal: true, Transitions: javaHeader}, {Name: "frames", Transitions: javaFrames}, }}
Java matches JVM exception stack traces (and, incidentally, Node.js ones). Derived from fluent-bit's java multiline parser, with fixes and enhancements.
var NodeJS = StateSet{Name: "nodejs", States: []State{ { Name: StartState, Transitions: []Transition{ {Pattern: `^Error: `, Next: "frames"}, {Pattern: `V8 errors stack trace:`, Next: "frames"}, }, }, { Name: "frames", Transitions: []Transition{ {Pattern: `^\s+(eval )?at `, Next: "frames"}, }, }, }}
NodeJS matches Node.js / V8 error stack traces whose headline the java set does not claim: a bare "Error:" at the start of the line (the most common Node headline), and the V8 "errors stack trace" marker. Headlines with an error-class prefix ("TypeError: ...") share their frame shape with the java set and are matched — and reported — as "java"; the two formats cannot be told apart reliably by line shape alone.
var PHP = StateSet{Name: "php", States: []State{ { Name: StartState, Transitions: []Transition{ {Pattern: `(PHP )?Fatal error:\s+Uncaught `, Next: "message"}, }, }, { Name: "message", NonTerminal: true, Transitions: []Transition{ {Pattern: `^Stack trace:$`, Next: "trace"}, }, }, { Name: "trace", NonTerminal: true, Transitions: []Transition{ {Pattern: `^#\d+ `, Next: "frames"}, }, }, { Name: "frames", Transitions: []Transition{ {Pattern: `^#\d+ `, Next: "frames"}, {Pattern: `^\s+thrown in .+ on line \d+`, Next: "thrown"}, }, }, {Name: "thrown"}, }}
PHP matches uncaught-exception reports, e.g.
PHP Fatal error: Uncaught Exception: boom in /app/index.php:3
Stack trace:
#0 /app/index.php(7): foo()
#1 {main}
thrown in /app/index.php on line 3
var Python = StateSet{Name: "python", States: []State{ {Name: StartState, Transitions: pythonTraceback}, { Name: "frames", Transitions: []Transition{ {Pattern: `^ File `, Next: "frames"}, {Pattern: `^ `, Next: "frames"}, {Pattern: `^([a-zA-Z0-9]+\.)*[a-zA-Z0-9]+Error:`, Next: "error"}, }, }, { Name: "error", Transitions: []Transition{ {Pattern: `^$`, Next: "after_error"}, }, }, { Name: "after_error", Transitions: []Transition{ {Pattern: `^The above exception was the direct cause of the following exception:`, Next: "chained"}, }, }, { Name: "chained", Transitions: []Transition{ {Pattern: `^$`, Next: "restart"}, }, }, {Name: "restart", Transitions: pythonTraceback}, }}
Python matches CPython tracebacks, including chained exceptions ("The above exception was the direct cause ...").
var Ruby = StateSet{Name: "ruby", States: []State{ { Name: StartState, Transitions: []Transition{ {Pattern: `^\S+:\d+:in .+: .+ \([A-Z][A-Za-z0-9_:]*(Error|Exception)\)$`, Next: "error"}, }, }, {Name: "error", NonTerminal: true, Transitions: rubyFrame}, {Name: "frames", Transitions: rubyFrame}, }}
Ruby matches uncaught-exception reports from the Ruby CLI, e.g.
main.rb:4:in `foo': undefined method `bar' for nil:NilClass (NoMethodError) from main.rb:8:in `baz' from main.rb:12:in `<main>'
var Rust = StateSet{Name: "rust", States: []State{ { Name: StartState, Transitions: []Transition{ {Pattern: `^thread '[^']*' panicked at .*:$`, Next: "panicked"}, }, }, { Name: "panicked", NonTerminal: true, Transitions: []Transition{ {Pattern: `.+`, Next: "message"}, }, }, { Name: "message", Transitions: []Transition{ {Pattern: "^note: run with `RUST_BACKTRACE=", Next: "note"}, {Pattern: `^stack backtrace:$`, Next: "backtrace"}, }, }, {Name: "note"}, {Name: "backtrace", NonTerminal: true, Transitions: rustFrame}, {Name: "frames", Transitions: rustFrame}, }}
Rust matches Rust panics (the Rust >= 1.73 two-line format), with or without a backtrace:
thread 'main' panicked at src/main.rs:5:5: index out of bounds: the len is 1 but the index is 2 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Functions ¶
This section is empty.
Types ¶
type State ¶
type State struct {
Name string
NonTerminal bool
Transitions []Transition
}
State is one node of a StateSet. A state is accepting unless NonTerminal is set: a group may only complete on a line that lands in an accepting state. Use NonTerminal for intermediate states that are not a valid stopping point. Several State entries may share a Name; their transitions are merged, but their NonTerminal flags must agree. The state named StartState is always non-terminal.
type StateMachine ¶
type StateMachine struct {
// contains filtered or unexported fields
}
StateMachine is the compiled form of one or more StateSet values, produced by Compile. It is immutable and safe to share between several aggregators. It implements the multiline.Matcher interface.
func Compile ¶
func Compile(sets ...StateSet) (*StateMachine, error)
Compile builds a StateMachine from the given sets. Each set contributes its StartState transitions to the shared start state (index 0); all other state names are scoped to their set, so sets cannot collide. Compile reports an error for an empty or duplicate set name, a transition that references an unknown state, an invalid pattern, or State entries that share a name but disagree on NonTerminal.
Example ¶
package main
import (
"context"
"fmt"
"github.com/JohanLindvall/multiline"
"github.com/JohanLindvall/multiline/patterns"
)
func main() {
// A custom format, declared exactly like the bundled sets: "BEGIN" opens
// a group, indented lines and "COMMIT" extend it. Compiling it alongside
// patterns.All keeps the bundled stack-trace formats working too.
set := patterns.StateSet{Name: "tx", States: []patterns.State{
{Name: patterns.StartState, Transitions: []patterns.Transition{
{Pattern: `^BEGIN`, Next: "body"},
}},
{Name: "body", Transitions: []patterns.Transition{
{Pattern: `^\s`, Next: "body"},
{Pattern: `^COMMIT`, Next: "body"},
}},
}}
matcher, err := patterns.Compile(append(patterns.All, set)...)
if err != nil {
panic(err)
}
ml := multiline.New(func(_ context.Context, e multiline.Entry[struct{}]) error {
match := e.Match
if match == "" {
match = "plain"
}
fmt.Printf("%s: %d line(s)\n", match, e.Lines)
return nil
}, multiline.WithMatcher(matcher))
ctx := context.Background()
for _, line := range []string{
"BEGIN 42",
" UPDATE accounts",
"COMMIT 42",
"idle",
} {
if err := ml.Add(ctx, "session-1", line, struct{}{}); err != nil {
panic(err)
}
}
if err := ml.Stop(ctx); err != nil {
panic(err)
}
}
Output: tx: 3 line(s) plain: 1 line(s)
func MustCompile ¶
func MustCompile(sets ...StateSet) *StateMachine
MustCompile is like Compile but panics on error. Use it for state sets that are known valid, such as the bundled ones.
func (*StateMachine) Format ¶
func (s *StateMachine) Format(index int) string
Format implements the multiline.Matcher interface. It returns the name of the StateSet that owns the state at index; this is what a completed group reports as its Match.
func (*StateMachine) StartLiterals ¶ added in v0.0.3
func (s *StateMachine) StartLiterals() []string
StartLiterals returns the literal prefilter derived from the start-state patterns at Compile time: every line that matches any start transition contains at least one of the returned substrings, so lines containing none skip the start regexes entirely (and a hit runs only the transitions the literal implies). It returns nil when no such set could be proven for every start pattern and the prefilter is disabled — worth checking in a test when start-pattern matching is on your hot path.
func (*StateMachine) Step ¶
func (s *StateMachine) Step(line string, active []int) (next []int, accepted int)
Step implements the multiline.Matcher interface. It applies line to the transitions of the active states and returns the new active set, plus the index of an accepting state the line landed in (-1 if none). An empty next means line does not continue any active state. The returned slice may be shared across calls; callers must not modify it.
When only the start state is active — the steady state of a log stream — lines that cannot possibly begin a group are rejected by the literal prefilter without running any regex, and lines that hit a probe literal run only the start transitions that literal implies (see StateMachine.StartLiterals).
type StateSet ¶
StateSet is a named group of states describing one multi-line format. The set's Name is reported as the Match of entries it aggregated (for the bundled sets: "go", "java", "nodejs", "python", "dotnet", "ruby", "rust", "php").
type Transition ¶
Transition is a single edge in a State: when Pattern (a regular expression) matches the current line, the machine moves to the state named Next. Next must name a state in the same StateSet (or StartState).
Source Files