README
¶
diffdb
Differential state tracking for Go
DiffDB is a library for tracking changes to Go objects by hashing the contents and comparing it to a previous version. It makes uses of BoltDB to persist state history to disk.
DiffDB was created to store ETL client state and only process changes to a remote datasource such as MySQL. This allows longer running or more computationally expensive operations to run outside of the database query context.
Documentation
¶
Index ¶
- Variables
- func HashOf(x interface{}) ([]byte, error)
- type ApplyFunc
- type DB
- type Decoder
- type Differential
- func (diff *Differential) Add(obj Object) (updated bool, err error)
- func (diff *Differential) AddChan(ctx context.Context, stream <-chan Object) error
- func (diff *Differential) AddTx(tx *bolt.Tx, obj Object) (bool, error)
- func (diff *Differential) Changed(id []byte, x interface{}) (changed bool, err error)
- func (diff *Differential) CountChanges() (pending int)
- func (diff *Differential) CountTracking() (count int)
- func (diff *Differential) Each(ctx context.Context, f ApplyFunc) error
- func (diff *Differential) EachN(ctx context.Context, f ApplyFunc, n int) error
- func (diff *Differential) MustNotConflict() error
- func (diff *Differential) Name() string
- func (diff *Differential) UpdateUserData(f func(b *bolt.Bucket) error) error
- func (diff *Differential) ViewUserData(f func(b *bolt.Bucket) error) error
- type Object
Constants ¶
This section is empty.
Variables ¶
var ( // ErrConflictingKey indicates that MustNotConflict() was enabled and a conflicting ID was entered into the state database. ErrConflictingKey = errors.New("diffdb: multiple objects with the same ID were added in the same change version") )
Functions ¶
Types ¶
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
A DB is a wrapper around a BoltDB to open multiple differential buckets
type Decoder ¶
type Decoder interface {
Decode(interface{}) error
}
A Decoder decodes serialised byte data of a diff entry into a native object. The object passed to Decode should be the same type added to the diff.
type Differential ¶
type Differential struct {
// contains filtered or unexported fields
}
A Differential tracks changes between serialised Go objects.
func (*Differential) Add ¶
func (diff *Differential) Add(obj Object) (updated bool, err error)
Add as a new object x to the list of pending changes. Changes to x are tracked through its given ID which uniquely identifies x across changes. For example, if x was an SQL row then ID would be the primary key of that row.
If Add is called multiple times same ID before applying changes then only the latest change will be taken to be applied.
func (*Differential) AddChan ¶
func (diff *Differential) AddChan(ctx context.Context, stream <-chan Object) error
AddChan adds objects sent from a channel until the channel is closed, the object is nil, or the context is cancelled. AddChan may stop processing the stream if an error occurs in which case no more messages will be consumed and that error will be returned.
func (*Differential) AddTx ¶
AddTx adds an object to start tracking by using an existing BoltDB transaction.
func (*Differential) Changed ¶
func (diff *Differential) Changed(id []byte, x interface{}) (changed bool, err error)
Changed returns true if the hash of x has changed for its ID.
func (*Differential) CountChanges ¶
func (diff *Differential) CountChanges() (pending int)
CountChanges returns the number of items in the change pending bucket.
func (*Differential) CountTracking ¶
func (diff *Differential) CountTracking() (count int)
CountTracking counts the number of entries in the hash tracking table. In other words, this is the amount of all items tracked by the differential db.
func (*Differential) Each ¶
func (diff *Differential) Each(ctx context.Context, f ApplyFunc) error
Each scans through each change and attempts to apply f() to each item waiting to be changed
func (*Differential) EachN ¶
EachN scans through each change until N items have been processed. If n is <= 0 then all pending changes will be applied.
func (*Differential) MustNotConflict ¶
func (diff *Differential) MustNotConflict() error
MustNotConflict sets a flag to track duplicate IDs given to subsequent calls to Add. This can be used as a debugging tool to check if additions in the same version have conflicting IDs. Calling MustNotConflict will delete any existing conflict information.
func (*Differential) Name ¶
func (diff *Differential) Name() string
func (*Differential) UpdateUserData ¶
func (diff *Differential) UpdateUserData(f func(b *bolt.Bucket) error) error
UpdateUserData wraps a BoltDB update transaction to allow custom user data to viewed or updated in the differential database.
func (*Differential) ViewUserData ¶
func (diff *Differential) ViewUserData(f func(b *bolt.Bucket) error) error
ViewUserData wraps a BoltDB view transaction to allow custom user data to be viewed in the differential database. This could include information such as run times, last exported differential, etc.
Source Files