bstates

package module
v0.9.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 4, 2026 License: MIT Imports: 20 Imported by: 4

Documentation

Overview

Package bstates implements a parser for Idefix event blobs.

Overview

Each binary blob is an event queue which is composed of one or more state encoded using the same schema. States represent a system state in a point of time. Schemas are json files which define States binary formats. Here there is an example schema which defines a 102bit State composed of 4 fields: 

	{
		"version": "2.0",
		"encoderPipeline": "t:z",

		"decodedFields":
		{
			{
				"name": "MESSAGE"
				"from": "MESSAGE_BUFFER",
				"decoder": "BufferToString"

			},
			{
				"name": "STATE",
				"decoder": "IntMap",
				"params": {
					"from": "STATE_CODE",
					"mapId": "STATE_MAP"
			}
		},

		"decoderIntMaps":
			{
				"STATE_MAP": {
					"0" : "IDLE",
					"1" : "STOPPED",
					"2" : "RUNNING"
				}
			},

	    "fields": [
	        {
	            "name": "3BITS_INT",
	            "type": "int",
	            "size": 3
	        },
	        {
	            "name": "STATE_CODE",
	            "type": "uint",
	            "size": 2
	        },
	        {
	            "name": "BOOL",
	            "type": "bool"
	        },
	        {
	            "name": "MESSAGE_BUFFER",
	            "type": "buffer",
	            "size": 96
	        }
	    ]
}

Decoders allow for complex codification of data saving space. In the above example there is a decodedField MESAGE which allows us to read MESSAGE_BUFFER as a string. Decoders work by executing the "decoder" parameter on the "from" field. Currently there are three types of Decoder.

States are the objects which hold data. They are composed by a frame.Frame which holds the data and a StateSchema that specifies a codification. StateField and DecodedStateField both can be retrieved using the State.Get function.

All objects in this library, can be serialized as JSON or as MSI map[string]interface{}

Index

Constants

View Source 
const (
	MOD_GZIP     = "z"    // run gzip compression
	MOD_ZSTD     = "zstd" // run zstd compression
	MOD_BITTRANS = "t"    // transpose the event matrix, for better compression
)

encoderPipeline options

View Source 
const (
	SCHEMA_VERSION_1_0 = "1.0"
	SCHEMA_VERSION_2_0 = "2.0"
)

Variables

View Source 
var ErrInvalidType = errors.New("invalid type")

ErrInvalidType represents a type validation error

View Source 
var ErrOutOfRange = errors.New("out of range")

ErrOutOfRange represents a range validation error

Functions

func GetDeltaMsiState 

func GetDeltaMsiState(from *State, to *State) (map[string]any, error)

GetDeltaMsiState compares two State objects and returns a map containing the values that have changed between them. Includes aliases for backward compatibility, similar to State.ToMsi().

func GetDeltaMsiStates 

func GetDeltaMsiStates(states []*State) ([]map[string]any, error)

GetDeltaMsiStates returns a slice of maps that represent the changes between each successive pair of State objects in the provided slice.

The first State is converted to its MSI representation, and subsequent States are compared to the last seen State using GetDeltaMsiState().

func GzipDec 

func GzipDec(b *buffer.Buffer) (*buffer.Buffer, error)

GzipDec decompresses the provided buffer which is expected to be in Gzip format and returns a new buffer containing the decompressed data.

func GzipEnc 

func GzipEnc(b *buffer.Buffer) (*buffer.Buffer, error)

GzipEnc compresses the provided buffer using Gzip compression and returns a new buffer containing the compressed data.

func StatesToMsiStates 

func StatesToMsiStates(states []*State) (out []map[string]any, err error)

StatesToMsiStates converts a slice of State pointers into a slice of maps containing the corresponding MSI states.

Each State is transformed into a map using the ToMsi() method.

func ZstdDec 

func ZstdDec(b *buffer.Buffer) (*buffer.Buffer, error)

ZstdDec decompresses the provided buffer which is expected to be in Zstandard (Zstd) format and returns a new buffer containing the decompressed data.

func ZstdEnc 

func ZstdEnc(b *buffer.Buffer) (*buffer.Buffer, error)

ZstdEnc compresses the provided buffer using Zstandard (Zstd) compression and returns a new buffer containing the compressed data.

Types

type BufferToStringDecoder 

type BufferToStringDecoder struct {
	From string // "from" parameter: name of the encoded field as defined in StateSchema.Fields
}

BufferToString implements a Decoder which returns a string from a buffer. This means that the original buffer will be returned as a [string] object stopping at the first null character.

func NewBufferToStringDecoder 

func NewBufferToStringDecoder(params map[string]any) (d *BufferToStringDecoder, err error)

func (*BufferToStringDecoder) Decode 

func (d *BufferToStringDecoder) Decode(s *State) (any, error)

func (*BufferToStringDecoder) Encode added in v0.8.0 

func (d *BufferToStringDecoder) Encode(s *State, v any) error

func (*BufferToStringDecoder) GetParams 

func (d *BufferToStringDecoder) GetParams() map[string]any

func (*BufferToStringDecoder) Name 

func (d *BufferToStringDecoder) Name() FieldDecoderType

type DecodedStateField 

type DecodedStateField struct {
	Name    string   // Name used to access this field
	Aliases []string // Alternative names for accessing this decoded field (used for backward compatibility with deprecated field names)
	Decoder Decoder  // Decoder used to access the original [StateField].
}

DecodedStateField represents a view of a StateField decoded by a Decoder.

The Name is used to access the decoded view. The raw encoded StateField is provided on the parameter "from" to the Decoder.

func (*DecodedStateField) FromMsi 

func (df *DecodedStateField) FromMsi(m map[string]any) error

FromMsi populates the DecodedStateField from a map representation.

func (*DecodedStateField) MarshalJSON 

func (df *DecodedStateField) MarshalJSON() (res []byte, err error)

MarshalJSON serializes the DecodedStateField to JSON format.

func (*DecodedStateField) ToMsi 

func (df *DecodedStateField) ToMsi() (map[string]any, error)

ToMsi converts the DecodedStateField to a map representation.

func (*DecodedStateField) UnmarshalJSON 

func (df *DecodedStateField) UnmarshalJSON(b []byte) error

UnmarshalJSON deserializes the JSON data into a DecodedStateField.

type Decoder 

type Decoder interface {
	Name() FieldDecoderType       // Decoder type
	Decode(s *State) (any, error) // function called to get the decoded value from the state
	Encode(s *State, v any) error // function called to encode the value into the state
	GetParams() map[string]any    // returns a MSI
}

Decoder is an interface that defines how to decode or transform state information. They provide a "virtual" field.

While decoders usually use the "from" parameter to specify the name of a StateField to decode, it's not mandatory.

Decoders can:

  • Use multiple input fields (similar to having multiple "from" parameters), allowing for operations with several operands.
  • Access the entire state, enabling more complex decoding logic that isn't limited to specific fields.
  • Define constant fields that do not depend on any state field, although this might have limited practical use.

func NewDecoder 

func NewDecoder(dtype string, params map[string]any) (d Decoder, err error)

NewDecoder creates a new Decoder instance based on the provided decoder type and parameters.

dtype: Should be one of FieldDecoderType.

type FieldDecoderType 

type FieldDecoderType string

FieldDecoderType defines the type for different field decoder names. 

const (
	BufferToStringDecoderType   FieldDecoderType = "BufferToString"
	NumberToUnixTsMsDecoderType FieldDecoderType = "NumberToUnixTsMs"
	IntMapDecoderType           FieldDecoderType = "IntMap"
	FlagsDecoderType            FieldDecoderType = "Flags"
)

Implemented decoders are: BufferToStringDecoder, NumberToUnixTsMsDecoder, IntMapDecoder and FlagsDecoder.

type FlagsDecoder added in v0.9.0 

type FlagsDecoder struct {
	From  string           // "from" parameter: name of the encoded field as defined in StateSchema.Fields
	Flags map[string]uint8 // "flags" parameter: map of flag name to bit position
}

func NewFlagsDecoder added in v0.9.0 

func NewFlagsDecoder(params map[string]any) (d *FlagsDecoder, err error)

func (*FlagsDecoder) Decode added in v0.9.0 

func (d *FlagsDecoder) Decode(s *State) (any, error)

func (*FlagsDecoder) Encode added in v0.9.0 

func (d *FlagsDecoder) Encode(s *State, v any) error

func (*FlagsDecoder) GetParams added in v0.9.0 

func (d *FlagsDecoder) GetParams() map[string]any

func (*FlagsDecoder) Name added in v0.9.0 

func (d *FlagsDecoder) Name() FieldDecoderType

type IntMapDecoder 

type IntMapDecoder struct {
	From  string // "from" parameter: name of the encoded field as defined in StateSchema.Fields
	MapId string // "mapId" parameter: name of the map as defined in the StateSchema.DecoderIntMaps
}

IntMapDecoder implements a Decoder which decodes an integer value into a string based on a mapping defined in the State object.

func NewIntMapDecoder 

func NewIntMapDecoder(params map[string]any) (d *IntMapDecoder, err error)

func (*IntMapDecoder) Decode 

func (d *IntMapDecoder) Decode(s *State) (any, error)

func (*IntMapDecoder) Encode added in v0.8.0 

func (d *IntMapDecoder) Encode(s *State, v any) error

func (*IntMapDecoder) GetParams 

func (d *IntMapDecoder) GetParams() map[string]any

func (*IntMapDecoder) Name 

func (d *IntMapDecoder) Name() FieldDecoderType

type NumberToUnixTsMsDecoder 

type NumberToUnixTsMsDecoder struct {
	From   string  // "from" parameter: name of the encoded field as defined in StateSchema.Fields
	Year   uint    // "year"
	Factor float64 // "factor"
}

NumberToUnixTsMsDecoder implements a Decoder which decodes a numeric value using the following formula:

decodedValue = UnixMillis(year) + valueToDecode*factor

func NewNumberToUnixTsMsDecoder 

func NewNumberToUnixTsMsDecoder(params map[string]any) (d *NumberToUnixTsMsDecoder, err error)

func (*NumberToUnixTsMsDecoder) Decode 

func (d *NumberToUnixTsMsDecoder) Decode(s *State) (any, error)

func (*NumberToUnixTsMsDecoder) Encode added in v0.8.0 

func (d *NumberToUnixTsMsDecoder) Encode(s *State, v any) error

func (*NumberToUnixTsMsDecoder) GetParams 

func (d *NumberToUnixTsMsDecoder) GetParams() map[string]any

func (*NumberToUnixTsMsDecoder) Name 

func (d *NumberToUnixTsMsDecoder) Name() FieldDecoderType

type State 

type State struct {
	*frame.Frame // Underlying binary data of the state
	// contains filtered or unexported fields
}

State represents a system state in a point of time. It holds data in the frame.Frame as defined by the provided StateSchema.

func CreateState 

func CreateState(schema *StateSchema) (*State, error)

CreateState initializes a new empty State based on the provided StateSchema.

func (*State) Get 

func (f *State) Get(fieldName string) (value any, err error)

Get retrieves the value of the specified field from the State.

It first tries to retrieve the raw value from the frame.Frame and, if unsuccessful, attempts to decode the field using the schema's decoding logic.

func (*State) GetCopy 

func (e *State) GetCopy() *State

GetCopy returns a deep copy of the current State. This includes a copy of the underlying [Frame] and retains the original StateSchema.

func (*State) GetSchema 

func (e *State) GetSchema() *StateSchema

GetSchema returns the StateSchema associated with the current State.

func (*State) Same added in v0.8.0 

func (f *State) Same(fieldName string, newValue any) (same bool, err error)

func (*State) Set added in v0.8.0 

func (f *State) Set(fieldName string, newValue any) error

Set updates the value of the specified field in the State.

It first checks if the field is a decoded field and, if so, uses the schema's encoding logic. Otherwise, it validates the value and updates the field using default encoding logic for the type of the field.

For T_BUFFER fields with range errors (oversized data), the value is still written to allow truncation during encode/decode, but an error is returned to notify about potential data loss.

func (*State) ToMsi 

func (e *State) ToMsi() (map[string]interface{}, error)

ToMsi converts the State into a map[string]interface{} representation, where each field's name is a key, and its corresponding value is the field's value. It includes both regular fields, decoded fields, and field aliases for backward compatibility.

type StateField 

type StateField struct {
	Name         string   // Name of the field, used for retrieval
	Aliases      []string // Alternative names for accessing this field (used for backward compatibility with deprecated field names)
	Size         int      // size in bits
	DefaultValue any
	Type         StateFieldType
	Decimals     uint // Number of decimal places for fixed-point fields, ignored for non-fixed types
	// contains filtered or unexported fields
}

StateField defines a field in a StateSchema.

func (*StateField) FromMsi 

func (e *StateField) FromMsi(rawField map[string]any) (err error)

FromMsi initializes a StateField from a map[string]interface{}.

func (*StateField) GetRange added in v0.9.0 

func (e *StateField) GetRange() (min, max any, err error)

GetRange returns the valid range for this field type and size.

func (*StateField) MarshalJSON 

func (e *StateField) MarshalJSON() (res []byte, err error)

MarshalJSON serializes the StateField to JSON format.

func (*StateField) ToMsi 

func (e *StateField) ToMsi() (msiData map[string]any, err error)

ToMsi converts a StateField to a map[string]interface{} for further processing.

func (*StateField) UnmarshalJSON 

func (e *StateField) UnmarshalJSON(b []byte) error

UnmarshalJSON deserializes the JSON data into a StateField.

func (*StateField) Validate added in v0.9.2 

func (e *StateField) Validate(value any) error

Validate validates that a value has the correct type and is within the valid range for this field type and size.

type StateFieldType 

type StateFieldType int

StateFieldType represents the type of a field in the StateSchema. 

const (
	T_INT StateFieldType = iota
	T_UINT
	T_FLOAT32
	T_FLOAT64
	T_BOOL
	T_BUFFER
	T_FIXED
	T_UFIXED
)

type StateQueue 

type StateQueue struct {
	StateSchema *StateSchema // Schemas used to encode each of the states in the queue.
	// contains filtered or unexported fields
}

StateQueue is a queue of states stored in a buffer one after another.

All elements in the same queue must use the StateSchema of the queue.

func CreateStateQueue 

func CreateStateQueue(schema *StateSchema) *StateQueue

Creates a StateQueue of [States] encoded with the StateSchema provided.

func (*StateQueue) Clear 

func (s *StateQueue) Clear()

Clear deletes all elements in the queue by creating a new internal buffer.

func (*StateQueue) Decode 

func (s *StateQueue) Decode(data []byte) (err error)

Decode [Clear] the queue and then runs the decoderPipeline of the schema populating the queue.

func (*StateQueue) Encode 

func (s *StateQueue) Encode() (dataOut []byte, err error)

Encode runs the encoderPipeline of the schema and outputs a binary blob with the queue compressed.

func (*StateQueue) FromMsi 

func (s *StateQueue) FromMsi(msg map[string]interface{}) error

FromMsi populates the StateQueue from a map representation.

func (*StateQueue) GetBitSize 

func (s *StateQueue) GetBitSize() int

GetBitSize return the number of bits used by the internal buffer.

func (*StateQueue) GetByteSize 

func (s *StateQueue) GetByteSize() int

GetByteSize return the number of bytes used by the internal buffer.

func (*StateQueue) GetNumStates 

func (s *StateQueue) GetNumStates() (num int)

GetNumStates returns the number of states in the queue.

func (*StateQueue) GetStateAt 

func (s *StateQueue) GetStateAt(index int) (*State, error)

GetStateAt returns the state at an index.

func (*StateQueue) GetStates 

func (s *StateQueue) GetStates() ([]*State, error)

GetStates returns a slice with all the events on the queue.

func (*StateQueue) Pop 

func (s *StateQueue) Pop() (*State, error)

Pop the first State in the queue removing it from the queue.

func (*StateQueue) Push 

func (s *StateQueue) Push(state *State) error

Push the State object provided into the end of the queue.

func (*StateQueue) PushAll 

func (s *StateQueue) PushAll(states []*State) error

PushAll pushes all the State objects provided at the end of the queue.

func (*StateQueue) StateBufferIter 

func (s *StateQueue) StateBufferIter(iterFunc func(stateBuffer []byte) (end bool))

StateBufferIter iterates trough the queue executing the callback for each State. If the callback returns true the iterations ends early.

func (*StateQueue) StateBufferIterFrom 

func (s *StateQueue) StateBufferIterFrom(from int, iterFunc func(stateBuffer []byte) (end bool))

StateBufferIterFrom iterates the queue as [StateBufferIter] but starting from the state index provided.

func (*StateQueue) ToMsi 

func (s *StateQueue) ToMsi() (msg map[string]interface{}, err error)

ToMsi converts the StateQueue into a map[string]interface{} representation.

type StateSchema 

type StateSchema struct {
	// contains filtered or unexported fields
}

StateSchema represents the schema used for encoding/decoding states. Fields within an schema can be plain or encoded.

func CreateStateSchema 

func CreateStateSchema(params *StateSchemaParams) (e *StateSchema, err error)

CreateStateSchema initializes a StateSchema from the provided parameters.

func (*StateSchema) CreateState 

func (s *StateSchema) CreateState() (*State, error)

CreateState initializes a new State object using the StateSchema.

func (*StateSchema) GetBitSize 

func (s *StateSchema) GetBitSize() int

GetBitSize returns the total bit size of the fields in the StateSchema.

func (*StateSchema) GetByteSize 

func (s *StateSchema) GetByteSize() int

GetByteSize returns the total byte size of the fields in the StateSchema.

func (*StateSchema) GetDecodedFields 

func (s *StateSchema) GetDecodedFields() []*DecodedStateField

GetDecodedFields returns a copy of the list of DecodedStateField in the schema.

func (*StateSchema) GetDecoderPipeline 

func (s *StateSchema) GetDecoderPipeline() []string

GetDecoderPipeline returns the decoder pipeline steps as a list of strings.

func (*StateSchema) GetEncoderPipeline 

func (s *StateSchema) GetEncoderPipeline() []string

GetEncoderPipeline returns the encoder pipeline steps as a list of strings.

func (*StateSchema) GetFields 

func (s *StateSchema) GetFields() []*StateField

GetFields returns a copy of the list of StateField in the schema.

func (*StateSchema) GetHashString 

func (s *StateSchema) GetHashString() string

GetHashString returns the SHA256 hash of the JSON representation of the StateSchema as a base64 encoded string.

func (*StateSchema) GetMeta 

func (s *StateSchema) GetMeta() map[string]any

GetMeta returns the meta data associated with the StateSchema.

func (*StateSchema) GetSHA256 

func (s *StateSchema) GetSHA256() [32]byte

GetSHA256 generates and returns the SHA256 hash of the JSON representation of the StateSchema.

func (*StateSchema) MarshalJSON 

func (s *StateSchema) MarshalJSON() (res []byte, err error)

MarshalJSON serializes the StateSchema into JSON format.

func (*StateSchema) ToMsi 

func (s *StateSchema) ToMsi() map[string]any

ToMsi converts the StateSchema into a map[string]interface{} for serialization.

func (*StateSchema) UnmarshalJSON 

func (s *StateSchema) UnmarshalJSON(b []byte) error

UnmarshalJSON deserializes the JSON into a StateSchema.

type StateSchemaParams 

type StateSchemaParams struct {
	Meta            map[string]any           // Meta data to associate with the schema
	Fields          []StateField             // List of fields to define in the schema
	DecodedFields   []DecodedStateField      // List of decoded views to define in the schema
	EncoderPipeline string                   // Encoder pipeline to use to package and unpackage a [StateQueue]
	DecoderIntMaps  map[string]map[int64]any // Integer mappings used for decoding encoded integer fields
}

StateSchemaParams represents the parameters for constructing a StateSchema.

Source Files

View all Source files

Directories

Path Synopsis
examples
basic module
decodeIdefixEvent module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.