whispertool

package module
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 7, 2023 License: MIT Imports: 11 Imported by: 0

README

whisprtool PkgGoDev

This project is open source but closed development.

I developed this command line tool for my own purpose.

Feel free to fork it and modify it yourself for your needs.

License

MIT

Documentation

Overview

Package whispertool implements Graphite's Whisper database format.

This package is created for the CLI tool in the cmd/whispertool subpackage.

Note this package's API is incompatible to github.com/go-graphite/go-whisper.

Index

Constants

View Source 
const (
	Second Duration = 1
	Minute          = 60 * Second
	Hour            = 60 * Minute
	Day             = 24 * Hour
	Week            = 7 * Day
	Year            = 365 * Day
)

Duration constants

View Source 
const ArchiveIDBest = -1

ArchiveIDBest is used to find the best archive for time range in FetchFromArchive.

View Source 
const UTCTimeLayout = "2006-01-02T15:04:05Z"

Timestamp format layout used in this package. Note time zone is fixed to "Z".

Variables

View Source 
var ErrArchiveIDOutOfRange = errors.New("archive ID out of range")

ErrArchiveIDOutOfRange is the error when an archive ID if out of range.

View Source 
var Now = time.Now

Now is a function which returns the current time. You can mock this function in tests.

Functions

This section is empty.

Types

type AggregationMethod 

type AggregationMethod int

AggregationMethod is the type of aggregation used in a whisper database. Note: 4 bytes long in Whisper Header, 1 byte long in Archive Header 

const (
	Average AggregationMethod = iota + 1
	Sum
	Last
	Max
	Min
	First

	Mix        // only used in whisper header
	Percentile // only used in archive header
)

AggregationMethod constants

func AggregationMethodString 

func AggregationMethodString(s string) (AggregationMethod, error)

AggregationMethodString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func AggregationMethodValues 

func AggregationMethodValues() []AggregationMethod

AggregationMethodValues returns all values of the enum

func (AggregationMethod) IsAAggregationMethod 

func (i AggregationMethod) IsAAggregationMethod() bool

IsAAggregationMethod returns "true" if the value is listed in the enum definition. "false" otherwise

func (AggregationMethod) String 

func (i AggregationMethod) String() string

type AppendTaker 

type AppendTaker interface {
	AppenderTo
	TakerFrom
}

AppendTaker is the interface that groups AppendTo and TakeFrom methods.

type AppenderTo 

type AppenderTo interface {
	AppendTo(dst []byte) []byte
}

AppenderTo is the interface for serializing an object and appends the encoded bytes to dst.

type ArchiveInfo 

type ArchiveInfo struct {
	// contains filtered or unexported fields
}

ArchiveInfo is a retention level. ArchiveInfo levels describe a given archive in the database. How detailed it is and how far back it records.

func NewArchiveInfo 

func NewArchiveInfo(secondsPerPoint Duration, numberOfPoints uint32) ArchiveInfo

NewArchiveInfo creats a retention.

func ParseArchiveInfo 

func ParseArchiveInfo(s string) (ArchiveInfo, error)

ParseArchiveInfo parses a single retention definition as you would find in the storage-schemas.conf of a Carbon install. Note that this only parses a single retention definition. An example input is "10s:2h". If you would like to parse multiple retention definitions like "10s:2h,1m:1d", use ParseRetentions instead.

func (*ArchiveInfo) AppendTo 

func (a *ArchiveInfo) AppendTo(dst []byte) []byte

AppendTo appends encoded bytes of a to dst and returns the extended buffer.

AppendTo method implements the AppenderTo interface.

func (ArchiveInfo) Equal 

func (a ArchiveInfo) Equal(b ArchiveInfo) bool

Equal returns whether or not a equals to b.

func (*ArchiveInfo) MaxRetention 

func (a *ArchiveInfo) MaxRetention() Duration

MaxRetention returns the whole duration of a.

func (*ArchiveInfo) NumberOfPoints 

func (a *ArchiveInfo) NumberOfPoints() uint32

NumberOfPoints returns the number of points in a.

func (*ArchiveInfo) SecondsPerPoint 

func (a *ArchiveInfo) SecondsPerPoint() Duration

SecondsPerPoint returns the duration of the step of a.

func (ArchiveInfo) String 

func (a ArchiveInfo) String() string

String returns the spring representation of a.

func (*ArchiveInfo) TakeFrom 

func (a *ArchiveInfo) TakeFrom(src []byte) ([]byte, error)

TakeFrom updates a from encoded bytes in src and returns the rest of src.

TakeFrom method implements the TakerFrom interface. If there is an error, it may be of type *WantLargerBufferError.

type ArchiveInfoList 

type ArchiveInfoList []ArchiveInfo

ArchiveInfoList is a slice of Retention.

func ParseArchiveInfoList 

func ParseArchiveInfoList(s string) (ArchiveInfoList, error)

ParseArchiveInfoList parses multiple retention definitions as you would find in the storage-schemas.conf of a Carbon install. Note that this parses multiple retention definitions. An example input is "10s:2h,1m:1d".

See: http://graphite.readthedocs.org/en/1.0/config-carbon.html#storage-schemas-conf

func (ArchiveInfoList) Equal 

func (aa ArchiveInfoList) Equal(bb ArchiveInfoList) bool

Equal returns whether or not aa equals to bb.

func (ArchiveInfoList) String 

func (aa ArchiveInfoList) String() string

String returns the spring representation of rr.

type Duration 

type Duration int32

Duration is seconds between two Timestamps.

func ParseDuration 

func ParseDuration(s string) (Duration, error)

ParseDuration parses a Duration string.

func (*Duration) AppendTo 

func (d *Duration) AppendTo(dst []byte) []byte

AppendTo appends encoded bytes of d to dst and returns the extended buffer.

func (Duration) String 

func (d Duration) String() string

String returns the string representation of d.

func (*Duration) TakeFrom 

func (d *Duration) TakeFrom(src []byte) ([]byte, error)

TakeFrom updates d from encoded bytes in src and returns the rest of src.

TakeFrom method implements the TakerFrom interface. If there is an error, it may be of type *WantLargerBufferError. 

type Header struct {
	// contains filtered or unexported fields
}

Header respresents a whisper file header.

func NewHeader 

func NewHeader(aggregationMethod AggregationMethod, xFilesFactor float32, archiveInfoList ArchiveInfoList) (*Header, error)

NewHeader creates and retruns a whisper file header. It returns an error if validations of passed arguments failed.

func (*Header) AggregationMethod 

func (h *Header) AggregationMethod() AggregationMethod

AggregationMethod returns the aggregation method of the whisper file.

func (*Header) AppendTo 

func (h *Header) AppendTo(dst []byte) []byte

AppendTo appends encoded bytes of h to dst and returns the extended buffer.

AppendTo method implements the AppenderTo interface.

func (*Header) ArchiveInfoList 

func (h *Header) ArchiveInfoList() ArchiveInfoList

ArchiveInfoList returns the archive info list of the whisper file.

func (*Header) ExpectedFileSize 

func (h *Header) ExpectedFileSize() int64

ExpectedFileSize returns the expected file size from h.

func (*Header) MaxRetention 

func (h *Header) MaxRetention() Duration

MaxRetention returns the max retention of the whisper file.

func (*Header) Size 

func (h *Header) Size() int64

ByteSize returns the size in bytes of h in the whisper file.

func (*Header) String 

func (h *Header) String() string

String returns the string representation of h in LTSV format [1].

[1] Labeled Tab-separated Values http://ltsv.org/

func (*Header) TakeFrom 

func (h *Header) TakeFrom(src []byte) ([]byte, error)

TakeFrom updates ts from encoded bytes in src and returns the rest of src.

TakeFrom method implements the TakerFrom interface. If there is an error, it may be of type *WantLargerBufferError.

func (*Header) XFilesFactor 

func (h *Header) XFilesFactor() float32

XFilesFactor returns the xFilesFactor of the whisper file.

type Option 

type Option func(*Whisper)

Option is the type for options for creating or opening a whisper file.

func WithOpenFileFlag 

func WithOpenFileFlag(flag int) Option

WithOpenFileFlag sets the flag for opening the file. This option is useful only when no WithInMemory is passed. Without this option, the default value is os.O_RDWR | os.O_CREATE | os.O_EXCL for Create and os.O_RDWR for Open.

func WithPerm 

func WithPerm(perm os.FileMode) Option

WithPerm sets the permission for the file. This option is useful only when no WithInMemory is passed. Without this option, the default value is 0644.

func WithoutFlock 

func WithoutFlock() Option

WithoutFlock disables flock for the file.

type Point 

type Point struct {
	Time  Timestamp
	Value Value
}

Point represent a data point in whisper databases.

func (*Point) AppendTo 

func (p *Point) AppendTo(dst []byte) []byte

AppendTo appends encoded bytes of p to dst and returns the extended buffer.

AppendTo method implements the AppenderTo interface.

func (Point) Equal 

func (p Point) Equal(q Point) bool

Equals returns whether or not p equals to q. It returns true if time and value of p equals to q. For comparison of value, Value's Equals method is used.

func (Point) String 

func (p Point) String() string

String returns a string representation of p for debugging.

func (*Point) TakeFrom 

func (p *Point) TakeFrom(src []byte) ([]byte, error)

TakeFrom updates p from encoded bytes in src and returns the rest of src.

TakeFrom method implements the TakerFrom interface. If there is an error, it may be of type *WantLargerBufferError.

type Points 

type Points []Point

Points represents a slice of Point.

func (*Points) AppendTo 

func (pp *Points) AppendTo(dst []byte) []byte

AppendTo appends encoded bytes of pp to dst and returns the extended buffer.

AppendTo method implements the AppenderTo interface.

func (Points) Diff 

func (pp Points) Diff(qq Points) (Points, Points)

Diff returns the different points in comparison of pp and qq.

func (Points) Equal 

func (pp Points) Equal(qq Points) bool

Equals returns whether or not pp equals to qq.

func (Points) Len 

func (pp Points) Len() int

Len is the number of elements in the collection. Implements sort.Interface.

func (Points) Less 

func (pp Points) Less(i, j int) bool

Less reports whether the element with index i should sort before the element with index j. Implements sort.Interface.

func (Points) String 

func (pp Points) String() string

String returns a string representation of pp for debugging.

func (Points) Swap 

func (pp Points) Swap(i, j int)

Swap swaps the elements with indexes i and j. Implements sort.Interface.

func (*Points) TakeFrom 

func (pp *Points) TakeFrom(src []byte) ([]byte, error)

TakeFrom updates pp from encoded bytes in src and returns the rest of src.

TakeFrom method implements the TakerFrom interface. If there is an error, it may be of type *WantLargerBufferError.

func (Points) Values 

func (pp Points) Values() []Value

Values returns values of pp.

type TakerFrom 

type TakerFrom interface {
	TakeFrom(src []byte) (rest []byte, err error)
}

TakerFrom is the interface for deserializing an object and returns the rest of src.

If there is an error, it may be of type *WantLargerBufferError.

type TimeSeries 

type TimeSeries struct {
	// contains filtered or unexported fields
}

TimeSeries is a result from Fetch.

func NewTimeSeries 

func NewTimeSeries(fromTime, untilTime Timestamp, step Duration, values []Value) *TimeSeries

NewTimeSeries returns a new TimeSeries.

func (*TimeSeries) AppendTo 

func (ts *TimeSeries) AppendTo(dst []byte) []byte

AppendTo appends encoded bytes of ts to dst and returns the extended buffer.

AppendTo method implements the AppenderTo interface.

func (*TimeSeries) DiffPoints 

func (ts *TimeSeries) DiffPoints(ts2 *TimeSeries) (Points, Points)

DiffPoints returns the different points between ts and us.

func (*TimeSeries) DiffPointsExcludeSrcNaN 

func (ts *TimeSeries) DiffPointsExcludeSrcNaN(ts2 *TimeSeries) (Points, Points)

DiffPointsExcludeSrcNaN returns the different points between ts and us excluding points of source NaN values.

func (*TimeSeries) Equal 

func (ts *TimeSeries) Equal(ts2 *TimeSeries) bool

Equal returns whether or not ts equals to us.

func (*TimeSeries) EqualTimeRangeAndStep 

func (ts *TimeSeries) EqualTimeRangeAndStep(ts2 *TimeSeries) bool

EqualTimeRangeAndStep returns whether or not all of FromTime(), UntilTime() and Step() are the same between ts and us.

func (*TimeSeries) FromTime 

func (ts *TimeSeries) FromTime() Timestamp

FromTime returns the start time of ts.

func (*TimeSeries) Points 

func (ts *TimeSeries) Points() Points

Points converts ts to points.

func (*TimeSeries) Step 

func (ts *TimeSeries) Step() Duration

Step returns the duration between points in ts.

func (*TimeSeries) String 

func (ts *TimeSeries) String() string

String returns the string representation of ts.

func (*TimeSeries) TakeFrom 

func (ts *TimeSeries) TakeFrom(src []byte) ([]byte, error)

TakeFrom updates ts from encoded bytes in src and returns the rest of src.

TakeFrom method implements the TakerFrom interface. If there is an error, it may be of type *WantLargerBufferError.

func (*TimeSeries) UntilTime 

func (ts *TimeSeries) UntilTime() Timestamp

UntilTime returns the end time of ts.

func (*TimeSeries) Values 

func (ts *TimeSeries) Values() []Value

Values returns the values in ts.

type Timestamp 

type Timestamp uint32

Timestamp is the Unix timestamp, the number of seconds elapsed since January 1, 1970 UTC.

func ParseTimestamp 

func ParseTimestamp(s string) (Timestamp, error)

ParseTimestamp parses timestamp in "2006-01-02T15:04:05Z" format. Note only "Z" is allowed for timezone.

func TimestampFromStdTime 

func TimestampFromStdTime(t time.Time) Timestamp

StdTimeToTimestamp returns t as a Timestamp.

func (Timestamp) Add 

func (t Timestamp) Add(d Duration) Timestamp

Add returns t+d.

func (*Timestamp) AppendTo 

func (t *Timestamp) AppendTo(dst []byte) []byte

AppendTo appends encoded bytes of t to dst and returns the extended buffer.

AppendTo method implements the AppenderTo interface.

func (Timestamp) String 

func (t Timestamp) String() string

String returns the string representation of t with UTCTimeLayout format.

func (Timestamp) Sub 

func (t Timestamp) Sub(u Timestamp) Duration

Sub returns the Duration t-u. To compute t-d for Duration, use t.Add(-d).

func (*Timestamp) TakeFrom 

func (t *Timestamp) TakeFrom(src []byte) ([]byte, error)

TakeFrom updates t from encoded bytes in src and returns the rest of src.

TakeFrom method implements the TakerFrom interface. If there is an error, it may be of type *WantLargerBufferError.

func (Timestamp) ToStdTime 

func (t Timestamp) ToStdTime() time.Time

ToStdTime returns t as a time.Time.

func (Timestamp) Truncate 

func (t Timestamp) Truncate(d Duration) Timestamp

Truncate returns the result of rounding t down to a multiple of d (since the Unix epoch time). If d <= 0, Truncate returns t.

type Value 

type Value float64

Value represents a value of Point.

func (Value) Add 

func (v Value) Add(u Value) Value

Add returns the sum (v + u) if both v and u is not NaN. It returns u if v is NaN, v if u is NaN.

func (*Value) AppendTo 

func (v *Value) AppendTo(dst []byte) []byte

AppendTo appends encoded bytes of v to dst and returns the extended buffer.

AppendTo method implements the AppenderTo interface.

func (Value) Diff 

func (v Value) Diff(u Value) Value

Diff returns the difference (v - u) if both v and u is not NaN. It returns NaN if either v or u is NaN.

func (Value) Equal 

func (v Value) Equal(u Value) bool

Equal returns whether or not v equals to u. It returns true if both of v and u is NaN, or both of v and u is not NaN and v == u.

func (Value) IsNaN 

func (v Value) IsNaN() bool

IsNaN returns whether or not v is NaN.

func (*Value) SetNaN 

func (v *Value) SetNaN()

SetNaN sets the value to NaN.

func (Value) String 

func (v Value) String() string

String returns the string representation of v.

func (*Value) TakeFrom 

func (v *Value) TakeFrom(src []byte) ([]byte, error)

TakeFrom updates v from encoded bytes in src and returns the rest of src.

TakeFrom method implements the TakerFrom interface. If there is an error, it may be of type *WantLargerBufferError.

type WantLargerBufferError 

type WantLargerBufferError struct {
	WantedBufSize int
}

WantLargerBufferError records the length of bytes wanted to deserialize.

If WantedByteLen is zero, it means the length wanted is unkown.

func (*WantLargerBufferError) Error 

func (e *WantLargerBufferError) Error() string

Error returns the error message for e.

type Whisper 

type Whisper struct {
	// contains filtered or unexported fields
}

Whisper represents a Whisper database file.

func Create 

func Create(filename string, archiveInfoList []ArchiveInfo, aggregationMethod AggregationMethod, xFilesFactor float32, opts ...Option) (*Whisper, error)

Create creates a whisper database file.

func Open 

func Open(filename string, opts ...Option) (*Whisper, error)

Open opens an existing whisper database file. In the current implementation, the whole content is read after opening the file. This behavior may be changed in the future.

func (*Whisper) AggregationMethod 

func (w *Whisper) AggregationMethod() AggregationMethod

AggregationMethod returns the aggregation method of the whisper file.

func (*Whisper) ArchiveInfoList 

func (w *Whisper) ArchiveInfoList() ArchiveInfoList

ArchiveInfoList returns the archive info list of the whisper file.

func (*Whisper) Close 

func (w *Whisper) Close() error

Close closes the file. For the file created with WithInMemory, this is a no-op.

func (*Whisper) Fetch 

func (w *Whisper) Fetch(from, until Timestamp) (*TimeSeries, error)

Fetch fetches points from the best archive for the specified time range.

It fetches points in range between `from` (exclusive) and `until` (inclusive).

func (*Whisper) FetchFromArchive 

func (w *Whisper) FetchFromArchive(arhiveID int, from, until, now Timestamp) (*TimeSeries, error)

FetchFromArchive fetches points in the specified archive and the time range.

FetchFromArchive fetches points from archive specified with `arhiveID`. It fetches points in range between `from` (exclusive) and `until` (inclusive). If `now` is zero, the current time is used.

func (*Whisper) GetAllRawUnsortedPoints 

func (w *Whisper) GetAllRawUnsortedPoints(archiveID int) (Points, error)

GetAllRawUnsortedPoints returns the raw unsorted points. This is provided for the debugging or investination purpose.

func (*Whisper) Header 

func (w *Whisper) Header() *Header

Header returns the header of the whisper file.

func (*Whisper) MaxRetention 

func (w *Whisper) MaxRetention() Duration

MaxRetention returns the max retention of the whisper file.

func (*Whisper) Sync 

func (w *Whisper) Sync() error

Sync flushes modifications on the memory buffer to the file and sync commits the content to the storage by calling os.File.Sync(). Note it is caller's responsibility to call Sync and the modification will be lost without calling Sync. For the file created with WithInMemory, this is a no-op.

func (*Whisper) Update 

func (w *Whisper) Update(t Timestamp, v Value) error

Update a value in the database.

If the timestamp is in the future or outside of the maximum retention it will fail immediately.

func (*Whisper) UpdateMany 

func (w *Whisper) UpdateMany(points []Point) (err error)

UpdateMany updates points in the best matching archives.

func (*Whisper) UpdatePointForArchive 

func (w *Whisper) UpdatePointForArchive(archiveID int, t Timestamp, v Value, now Timestamp) error

UpdatePointForArchive updates one point in the specified archive.

func (*Whisper) UpdatePointsForArchive 

func (w *Whisper) UpdatePointsForArchive(points []Point, archiveID int, now Timestamp) error

UpdatePointsForArchive updates points in the specified archive.

If archiveID is ArchiveIDBest, points are first splitted for the best matching archives and update the corresponding archives.

Note this does NOT update points whose timestamp is out of the timestamp range of the specified archive. This behavior is not compatible to Whisper.UpdateMany in github.com/go-graphite/go-whisper.

func (*Whisper) XFilesFactor 

func (w *Whisper) XFilesFactor() float32

XFilesFactor returns the xFilesFactor of the whisper file.

Source Files

View all Source files

Directories

Path Synopsis
cmd
whispertool command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.