README
¶
On The Wire
Intro
When writing data between services, whether across the network or inter-process communication, it's common to encode and compress data or perform other byte manipulations when transmitting. This library provides a builder pattern for constructing those Read and Write operations. Operations include:
- Gob encoding using Golang's native object encoding
- JSON encoding
- Compression using
compress/zlib - Encryption/decryption using
crypto/rsa - Signing/verifying using
crypto/rsa - Applying and validating nonces
- Handling timeouts
- Custom []byte -> []byte transforms during reading and writing
Installation and Import
go get "github.com/EddisonKing/on-the-wire"
import (
otw "github.com/EddisonKing/on-the-wire"
)
The Pipeline
The core concept is a Pipeline (actually a wrapper of a paired ReadPipeline and WritePipeline) which is in effect just a list of intended actions to perform on data during a write or read. A pipeline does not perform these actions, the Build() function of the pipeline produces a function capable of running these instructions.
The simplest Pipeline can be defined as:
read, write := otw.New[T].Build()
The New() function will produce a Pipeline[T] where T is any. It can be any Golang type, whether it be a base type like a string or int, or more complicated types like slices, maps and structs. The Build() function produces two functions for reading and writing that implement the desired effects, in the non-functional example above, would just perform Gob encoding by default to serialise and deserialise the data.
If the type T is not the same for both reading and writing for whatever reason, you can construct two separate pipelines using equivalent methods below but on ReadPipeline[R] and WritePipeline[W].
Encoding/Decoding
Currently, there are two supported encodings, but others are planned. The default is encoding/gob, but can be requested explicitly:
read, write := otw.New[T].UseGobEncoding().Build()
If JSON is preferred, it can be requested:
read, write := otw.New[T].UseJSONEncoding().Build()
Compression
To enable compression in the pipeline:
read, write := otw.New[T].UseCompression().Build()
Compression use the compress/zlib library, but future plans will allow this to be switched out if this isn't suitable.
Encryption/Decryption
Encryption and decryption is performed using crypto/rsa and currently only supports RSA for asymmetric encryption/decryption. AES for symmetric encryption is on the roadmap.
pubKeyFn := func() *rsa.PublicKey { ... }
privKeyFn := func() *rsa.PrivateKey { ... }
read, write := otw.New[T].UseAsymmetricEncryption(pubKeyFn, privKeyFn).Build()
The UseAsymmetricEncryption() function takes in functions that are used to callback to during read and write operations to get the private and public keys respectively. This is more ergonomic since the keys won't get baked into the read and write functions the pipelines create and the keys are free to change over time.
Signing/Verification
Like encryption and decryption, the crypto/rsa library is used. The function to add signing behaves similar to the encryption and decryption as well since it gets the keys during each read and write operation and the keys aren't baked into the functions at Build() time.
pubKeyFn := func() *rsa.PublicKey { ... }
privKeyFn := func() *rsa.PrivateKey { ... }
read, write := otw.New[T].UseSigning(pubKeyFn, privKeyFn).Build()
Timeouts
read, write := otw.New[T].UseTimeout(time.Duration).Build()
Since the underlying io.Reader and io.Writer that the read and write functions could act upon may be delayed for whatever reason, it's sometimes necessary to add a timeout.
Nonces
Nonces are simple int that are written and read from the data. This is often used as an additional security measure to prevent replays of messages. UseNonce() is provided as a mechanism for the user to hook in a custom nonce check if this additional security is required.
set := func() int { ... }
check := func(int) bool { ... }
read, write := otw.New[T].UseNonce(set, check).Build()
In one scenario, the set function would do something like adding the generated nonce to a map, then the check function would verify the nonce it read exists in the map. Any other nonce could be suspicious.
In a different scenario, where the sender is not the same as the receiver, the check function might just test that it hasn't seen the same nonce twice, thereby protecting against replays.
The usage and relevance of a nonce is dependant on the context obviously, so it's up to the library consumer to determine whether this feature should be used.
Custom Operations
writeTransformer := func([]byte) ([]byte, error) { ... }
readTransformer := func([]byte) ([]byte, error) { ... }
read, write := otw.New[T].UseCustomOperation(readTransformer, writeTransformer).Build()
Custom operations are provided as an additional way to add functionality to the pipeline that doesn't already exist.
Logging
otw.SetLogger(*slog.Logger)
Most of the logging is very verbose in this library so only setting the Level to Debug will produce significant output and shouldn't be necessary unless attempting to debug a custom operation.
Putting It All Together!
type Test struct {
someMessage string
someInt int
someBool bool
}
pubKeyFn := func() *rsa.PublicKey { ... }
privKeyFn := func() *rsa.PrivateKey { ... }
set := func() int { ... }
check := func(int) bool { ... }
read, write := otw.New[Test].
UseJSONEncoding().
UseNonce(set, check).
UseAsymmetricEncryption(pubKeyFn, privKeyFn).
UseSigning(pubKeyFn, privKeyFn).
UseCompression().
UseTimeout(time.Second * 15).
Build()
In this example, write would be a function that takes a Test struct and some io.Writer, and does the following to it:
- Converts
Testto JSON - Prepends a nonce to the data, created from the
setfunction - Encrypts all of those bytes with the public key produced by
pubKeyFn - Compresses the bytes using
compress/zlib - (optional) Times out if write takes too long
And for read it does the inverse:
- Decompresses using
compress/zlib - Decrypts the bytes using the private key produced by
privKeyFn - Strips the nonce off the bytes and uses the
checkfunction to verify the nonce - Converts the bytes, in JSON format, back into the
Testtype - (optional) Times out if read takes too long
Documentation
¶
Index ¶
- Variables
- func SetLogger(l *slog.Logger)
- type Pipeline
- func (p *Pipeline[T]) Build() (func(io.Reader) (T, error), func(T, io.Writer) error)
- func (p *Pipeline[T]) UseAsymmetricEncryption(publicKeyFn func() *rsa.PublicKey, privateKeyFn func() *rsa.PrivateKey) *Pipeline[T]
- func (p *Pipeline[T]) UseCompression() *Pipeline[T]
- func (p *Pipeline[T]) UseCustomOperation(readFn, writeFn func([]byte) ([]byte, error)) *Pipeline[T]
- func (p *Pipeline[T]) UseGobEncoding() *Pipeline[T]
- func (p *Pipeline[T]) UseJSONEncoding() *Pipeline[T]
- func (p *Pipeline[T]) UseNonce(set func() int, check func(int) bool) *Pipeline[T]
- func (p *Pipeline[T]) UseSigning(publicKeyFn func() *rsa.PublicKey, privateKeyFn func() *rsa.PrivateKey) *Pipeline[T]
- func (p *Pipeline[T]) UseTimeout(t time.Duration) *Pipeline[T]
- type ReadPipeline
- func (p *ReadPipeline[R]) Build() func(io.Reader) (R, error)
- func (p *ReadPipeline[R]) UseAsymmetricEncryption(privateKeyFn func() *rsa.PrivateKey) *ReadPipeline[R]
- func (p *ReadPipeline[R]) UseCompression() *ReadPipeline[R]
- func (p *ReadPipeline[R]) UseCustomOperation(readFn func([]byte) ([]byte, error)) *ReadPipeline[R]
- func (p *ReadPipeline[R]) UseGobEncoding() *ReadPipeline[R]
- func (p *ReadPipeline[R]) UseJSONEncoding() *ReadPipeline[R]
- func (p *ReadPipeline[R]) UseNonce(check func(int) bool) *ReadPipeline[R]
- func (p *ReadPipeline[R]) UseSigning(publicKeyFn func() *rsa.PublicKey) *ReadPipeline[R]
- func (p *ReadPipeline[R]) UseTimeout(t time.Duration) *ReadPipeline[R]
- type WritePipeline
- func (p *WritePipeline[W]) Build() func(W, io.Writer) error
- func (p *WritePipeline[W]) UseAsymmetricEncryption(publicKeyFn func() *rsa.PublicKey) *WritePipeline[W]
- func (p *WritePipeline[W]) UseCompression() *WritePipeline[W]
- func (p *WritePipeline[W]) UseCustomOperation(writeFn func([]byte) ([]byte, error)) *WritePipeline[W]
- func (p *WritePipeline[W]) UseGobEncoding() *WritePipeline[W]
- func (p *WritePipeline[W]) UseJSONEncoding() *WritePipeline[W]
- func (p *WritePipeline[W]) UseNonce(set func() int) *WritePipeline[W]
- func (p *WritePipeline[W]) UseSigning(privateKeyFn func() *rsa.PrivateKey) *WritePipeline[W]
- func (p *WritePipeline[W]) UseTimeout(t time.Duration) *WritePipeline[W]
Constants ¶
This section is empty.
Variables ¶
var ErrNonceInvalid = fmt.Errorf("nonce is invalid")
var ErrTimedOut = fmt.Errorf("operation timed out")
Functions ¶
Types ¶
type Pipeline ¶
type Pipeline[T any] struct { // contains filtered or unexported fields }
Represents a pipeline of read/write operations for any type T. The underlying operations act upon byte slices and return byte slices and error if one occured.
func New ¶
Creates a new empty pipeline with read and write pipelines underpinning it's operation.
This pipeline can be build and used immediately. By default, if no encoder has been selected, the Gob encoder will be used at build time.
func (*Pipeline[T]) Build ¶
Compiles the pipline into a read func and write func following the specification of the pipeline operations and selected encoders.
func (*Pipeline[T]) UseAsymmetricEncryption ¶
func (p *Pipeline[T]) UseAsymmetricEncryption(publicKeyFn func() *rsa.PublicKey, privateKeyFn func() *rsa.PrivateKey) *Pipeline[T]
Use RSA asymmetric encryption for encrypting and decrypting data.
It is up to the consumer of the library to provide callback functions that return the public and private keys. The functions will only be used during read and write operations, not during the building of the pipeline.
func (*Pipeline[T]) UseCompression ¶
Appends a compression step to the write operations and a decompression strep to the read operations.//
Compression is done with the `compress/zlib` library.
func (*Pipeline[T]) UseCustomOperation ¶
Custom Operations allow the consumer to define their own write and read operations to append to the pipeline.
Both functions take in a []byte and output a modified []byte or error. It is ok to return a new []byte or modify the existing one.
func (*Pipeline[T]) UseGobEncoding ¶
Enables on-boarding and off-boarding to the pipeline using Go's native Go Object Encoding.
Gob Encoding requires that structs export their fields to be transmitted. No exported fields will result in an error on write.
func (*Pipeline[T]) UseJSONEncoding ¶
Enables on-boarding and off-boarding to the pipeline using JSON Encoding.
func (*Pipeline[T]) UseNonce ¶
Enables the use of integer nonces in read and write operations. The provided nonce set and check functions are callbacks for how to generate the nonce for writing and how to check them upon reading.
func (*Pipeline[T]) UseSigning ¶
func (p *Pipeline[T]) UseSigning(publicKeyFn func() *rsa.PublicKey, privateKeyFn func() *rsa.PrivateKey) *Pipeline[T]
Use RSA asymmetric encryption for signing and verifying data being sent. The write operation to the pipeline appends a []byte containing the signature. The read operation will cut the signature from the pipeline, verify it, and either continue processing or error if the signature fails to validate.
It is up to the consumer of the library to provide callback functions that return the public and private keys. The functions will only be used during read and write operations, not during the building of the pipeline.
type ReadPipeline ¶
type ReadPipeline[R any] struct { // contains filtered or unexported fields }
Represents a pipeline of read operations for any type R. The underlying operations act upon byte slices and return byte slices and error if one occured.
func NewReadPipeline ¶
func NewReadPipeline[R any]() *ReadPipeline[R]
Creates a new empty pipeline supporting read operations.
This pipeline can be build and used immediately. By default, if no encoder has been selected, the Gob encoder will be used at build time.
func (*ReadPipeline[R]) Build ¶
func (p *ReadPipeline[R]) Build() func(io.Reader) (R, error)
Compiles the pipline into a read func following the specification of the pipeline operations and selected encoders.
func (*ReadPipeline[R]) UseAsymmetricEncryption ¶
func (p *ReadPipeline[R]) UseAsymmetricEncryption(privateKeyFn func() *rsa.PrivateKey) *ReadPipeline[R]
Use RSA asymmetric encryption for decrypting data.
It is up to the consumer of the library to provide a callback function to return the private key. The functions will only be used during read and write operations, not during the building of the pipeline.
func (*ReadPipeline[R]) UseCompression ¶
func (p *ReadPipeline[R]) UseCompression() *ReadPipeline[R]
Appends a decompression step to the read operations.
Compression is done with the `compress/zlib` library.
func (*ReadPipeline[R]) UseCustomOperation ¶
func (p *ReadPipeline[R]) UseCustomOperation(readFn func([]byte) ([]byte, error)) *ReadPipeline[R]
Custom Operations allow the consumer to define their own read operations to append to the pipeline.
Functions take in a []byte and output a modified []byte or error. It is ok to return a new []byte or modify the existing one.
func (*ReadPipeline[R]) UseGobEncoding ¶
func (p *ReadPipeline[R]) UseGobEncoding() *ReadPipeline[R]
Enables off-boarding from the pipeline using Go's native Go Object Encoding.
Gob Encoding requires that structs export their fields to be transmitted. No exported fields will result in an error on write.
func (*ReadPipeline[R]) UseJSONEncoding ¶
func (p *ReadPipeline[R]) UseJSONEncoding() *ReadPipeline[R]
Enables off-boarding from the pipeline using JSON Encoding.
func (*ReadPipeline[R]) UseNonce ¶
func (p *ReadPipeline[R]) UseNonce(check func(int) bool) *ReadPipeline[R]
Enables nonces during read operations. An integer nonce is checked for validity using the check callback during reading
func (*ReadPipeline[R]) UseSigning ¶
func (p *ReadPipeline[R]) UseSigning(publicKeyFn func() *rsa.PublicKey) *ReadPipeline[R]
Use RSA asymmetric encryption for verifying data being sent. The read operation will cut the signature from the pipeline, verify it, and either continue processing or error if the signature fails to validate.
It is up to the consumer of the library to provide callback functions that return the public key. The functions will only be used during read and write operations, not during the building of the pipeline.
func (*ReadPipeline[R]) UseTimeout ¶
func (p *ReadPipeline[R]) UseTimeout(t time.Duration) *ReadPipeline[R]
Enables a timeout on all operations. This timeout is used for each operation, as well as decoding final payloads.
type WritePipeline ¶
type WritePipeline[W any] struct { // contains filtered or unexported fields }
Represents a pipeline of write operations for any type W. The underlying operations act upon byte slices and return byte slices and error if one occured.
func NewWritePipeline ¶
func NewWritePipeline[W any]() *WritePipeline[W]
Creates a new empty pipeline supporting write operations.
This pipeline can be build and used immediately. By default, if no encoder has been selected, the Gob encoder will be used at build time.
func (*WritePipeline[W]) Build ¶
func (p *WritePipeline[W]) Build() func(W, io.Writer) error
Compiles the pipline into a read func and write func following the specification of the pipeline operations and selected encoders.
func (*WritePipeline[W]) UseAsymmetricEncryption ¶
func (p *WritePipeline[W]) UseAsymmetricEncryption(publicKeyFn func() *rsa.PublicKey) *WritePipeline[W]
Use RSA asymmetric encryption for encrypting data.
It is up to the consumer of the library to provide a callback function to return the public key. The function will only be used during write operations, not during the building of the pipeline.
func (*WritePipeline[W]) UseCompression ¶
func (p *WritePipeline[W]) UseCompression() *WritePipeline[W]
Appends a compression step to the write operations.
Compression is done with the `compress/zlib` library.
func (*WritePipeline[W]) UseCustomOperation ¶
func (p *WritePipeline[W]) UseCustomOperation(writeFn func([]byte) ([]byte, error)) *WritePipeline[W]
Custom Operations allow the consumer to define their own write operations to append to the pipeline.
Functions take in a []byte and output a modified []byte or error. It is ok to return a new []byte or modify the existing one.
func (*WritePipeline[W]) UseGobEncoding ¶
func (p *WritePipeline[W]) UseGobEncoding() *WritePipeline[W]
Enables on-boarding to the pipeline using Go's native Go Object Encoding.
Gob Encoding requires that structs export their fields to be transmitted. No exported fields will result in an error on write.
func (*WritePipeline[W]) UseJSONEncoding ¶
func (p *WritePipeline[W]) UseJSONEncoding() *WritePipeline[W]
Enables on-boarding to the pipeline using JSON Encoding.
func (*WritePipeline[W]) UseNonce ¶
func (p *WritePipeline[W]) UseNonce(set func() int) *WritePipeline[W]
Enables nonces during read operations. An integer nonce is checked for validity using the check callback during reading
func (*WritePipeline[W]) UseSigning ¶
func (p *WritePipeline[W]) UseSigning(privateKeyFn func() *rsa.PrivateKey) *WritePipeline[W]
Use RSA asymmetric encryption for signing the data being sent. The write operation to the pipeline appends a []byte containing the signature.
It is up to the consumer of the library to provide the callback function to return the private key. The function will only be used during write operations, not during the building of the pipeline.
func (*WritePipeline[W]) UseTimeout ¶
func (p *WritePipeline[W]) UseTimeout(t time.Duration) *WritePipeline[W]
Enables a timeout on all operations. This timeout is used for each operation, as well as encoding the initial payload.
Source Files