README
¶
laozi
archiver of events.
stores events to s3 partitioned however you want. imagine AWS firehose service but with a configurable partition method.
usage
package main
import (
"fmt"
"time"
laozi "github.com/seedboxtech/laozi"
)
func main() {
l := laozi.NewLaozi(&laozi.Config{
LoggerFactory: laozi.S3LoggerFactory{
Bucket: "laozi-test",
Region: "us-east-1",
Prefix: "events/", // optional
FlushInterval: time.Second * 30, // optional
Compression: "gzip", // optional
IsDupeFunc: func(event []byte, line []byte) bool {
// implement some method of checking for duplicates
return string(event) == string(line)
}
},
EventChannelSize: 10000000,
LoggerTimeout: time.Minute,
PartitionKeyFunc: func(e []byte) (string, error) {
return "event-file.csv.gz", nil
},
})
quit := time.After(2 * time.Minute)
i := 0
loop:
for {
select {
case <-quit:
break loop
default:
i++
l.Log([]byte(fmt.Sprintf("%d\n", i)))
time.Sleep(time.Millisecond)
}
}
fmt.Println("Done logging!")
fmt.Println(" - logged:", i)
// give some time for the final flush to happenx
<-time.After(2 * time.Minute)
}
testing
currently this package uses s3 directly in tests. this does mean tests will cost a very small amount of money to run and requires a connection to the internet. tests can be run...
go test ./...
Documentation
¶
Overview ¶
Package laozi is an archiver of events. stores events to s3 partitioned however you want. imagine AWS firehose service but with a configurable partition method.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
LoggerFactory LoggerFactory
LoggerTimeout time.Duration
PartitionKeyFunc func([]byte) (string, error)
EventChannelSize int
RoutingWorkers int
}
Config is a struct used to configure Laozi to your implementation.
type Laozi ¶
type Laozi interface {
Log([]byte)
Close()
}
Laozi is an archiver responsible for receiving events and archiving them to a safe, reliable storage. Currently it achieves this by implementing s3 storage. It is named after one of the most famous archivists in the world, https://en.wikipedia.org/wiki/Laozi
type Logger ¶
type Logger interface {
// send event data
Log([]byte)
// Close method called before it removed from internal map
Close() error
// LastActive is used to get the time a logger last logged. Used for deleting stale
// loggers from internal map.
LastActive() time.Time
}
Logger defines the behaviour of all loggers.
type LoggerFactory ¶
LoggerFactory is an interface that defines how to make a new logger. This Logger will be responsible for logging all events to it that match the same partition key.
type S3LoggerFactory ¶
type S3LoggerFactory struct {
Prefix string
Bucket string
Region string
FlushInterval time.Duration
Compression string
IsDupeFunc func(event []byte, line []byte) bool
LogChannelSize int
// contains filtered or unexported fields
}
S3LoggerFactory is a logger factory for creating loggers that log received events to S3.
func (*S3LoggerFactory) New ¶
func (lf *S3LoggerFactory) New(key string) Logger
NewLogger return a new instance of an S3 Logger for a corresponding partition key.
func (*S3LoggerFactory) Release ¶
func (lf *S3LoggerFactory) Release(logger Logger)
NewLogger return a new instance of an S3 Logger for a corresponding partition key.
Source Files
Directories