Documentation
¶
Overview ¶
Package timefault provides deterministic time-fault injectors and a clock-skew detector for testing distributed-clock robustness in HelixCluster.
Model ¶
A shared base clock advances in discrete ticks (seconds). Each node derives its own logical clock from the base tick through a per-node Fault. The base clock represents the host / quorum reference time and is never mutated by a node-level fault.
Three fault injectors are provided:
- SkewFault: offsets a node's clock by a fixed delta so it reads base+delta.
- FreezeFault: stops a node's clock at the instant it was frozen; the node falls progressively behind as the base advances.
- MonotonicViolationFault: makes a node's clock jump backward exactly once at a given base instant, violating monotonicity, then resumes tracking the base from the (lower) jumped value.
A Detector compares each node's reported time against the base reference and flags a node INCONSISTENT when |reported-base| exceeds a tolerance. It also detects monotonicity violations distinctly from skew by observing a backward step in a node's successive reported values. A lease/heartbeat decision uses the detector to deny leadership to a clock-faulty node (split-brain avoidance). Resetting a node's fault re-syncs it to the base.
The package is pure Go, deterministic and self-contained: time is modeled as int64 seconds driven by explicit base ticks, never by the wall clock.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Cluster ¶
type Cluster struct {
Detector *Detector
// contains filtered or unexported fields
}
Cluster is an ordered collection of nodes sharing a base clock and detector.
func NewCluster ¶
NewCluster builds a cluster from node IDs with the given tolerance.
func (*Cluster) EvaluateAll ¶
EvaluateAll evaluates every node at the base tick, returning verdicts keyed by node ID in sorted order.
func (*Cluster) Inconsistent ¶
Inconsistent returns the sorted IDs of nodes the detector flags inconsistent at the base tick.
type Detector ¶
type Detector struct {
Tol Tolerance
}
Detector evaluates node reports against the base reference.
func NewDetector ¶
NewDetector builds a Detector with the given tolerance in seconds.
func (*Detector) EvaluateAt ¶
EvaluateAt reads the node at base and judges it against the base reference.
A node is INCONSISTENT when its absolute skew exceeds the tolerance OR when a backward step (monotonicity violation) is observed relative to its previous read. The monotonicity check fires only on a strict backward step between two reads taken at non-decreasing base values, so it distinguishes a backward jump from a steady offset (skew).
func (*Detector) GrantLease ¶
func (d *Detector) GrantLease(n *Node, base int64) LeaseDecision
GrantLease evaluates the node and grants leadership only if the detector finds it consistent. A skewed, frozen-behind or monotonicity-violating node is denied to avoid split-brain.
type Fault ¶
type Fault interface {
Kind() FaultKind
// ReportedTime maps a base time (seconds) to the node's reported time.
ReportedTime(base int64) int64
}
Fault is the per-node clock transform applied to a base tick (seconds). Given the current base time, ReportedTime returns the time the node would report. Implementations must be deterministic and pure.
type FaultKind ¶
type FaultKind int
FaultKind enumerates the supported time faults.
const ( // FaultNone means the node tracks the base clock exactly. FaultNone FaultKind = iota // FaultSkew offsets the node clock by a fixed delta. FaultSkew // FaultFreeze stops the node clock at the freeze instant. FaultFreeze // FaultMonotonicViolation makes the node jump backward once. FaultMonotonicViolation )
type FreezeFault ¶
type FreezeFault struct {
At int64
}
FreezeFault stops the node clock at At. For base < At the node still tracks the base (the freeze has not occurred yet); from At onward the node is stuck at At and falls progressively behind as the base advances.
func (FreezeFault) ReportedTime ¶
func (f FreezeFault) ReportedTime(base int64) int64
ReportedTime returns min(base, At): frozen at At once the base passes it.
type LeaseDecision ¶
LeaseDecision records whether a node may be granted leadership at a base tick.
type MonotonicViolationFault ¶
type MonotonicViolationFault struct {
At int64
Jump int64 // positive magnitude of the backward jump
}
MonotonicViolationFault makes the node jump backward by Jump seconds once the base reaches or passes At. Before At the node tracks the base; from At onward it reports base-Jump, so the reported value steps backward across the At boundary (a monotonicity violation) and then advances normally from there.
func (MonotonicViolationFault) Kind ¶
func (MonotonicViolationFault) Kind() FaultKind
Kind reports the fault kind.
func (MonotonicViolationFault) ReportedTime ¶
func (f MonotonicViolationFault) ReportedTime(base int64) int64
ReportedTime returns base before At and base-Jump at/after At.
type Node ¶
type Node struct {
ID string
// contains filtered or unexported fields
}
Node is a cluster member with a per-node fault transform and a record of the last reported time (used to detect backward steps / monotonicity violations).
func (*Node) Inject ¶
Inject installs a fault on the node and clears any monotonicity history so a freshly injected fault is evaluated from a clean state.
type SkewFault ¶
type SkewFault struct {
Delta int64
}
SkewFault offsets the node clock by Delta seconds (negative = behind).
func (SkewFault) ReportedTime ¶
ReportedTime returns base+Delta.
Source Files