lfsb

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 5, 2024 License: MIT Imports: 7 Imported by: 0

README

📦 litefs-backup (lfsb)

A drop-in, self-hosted replacement for LiteFS Cloud, which was deprecated in July 2024.

Litefs-backup is a standalone backup process that streams backups of your LiteFS cluster to any s3-compatible storage provider, with 5 minute point-in-time restore granularity. It supports both static and dynamic LiteFS lease modes.

This project is sponsored by Tender - a private, productive personal finance tool.

This codebase is a simplified fork of LiteFS Cloud, originally authored and graciously donated by @benbjohnson and co at fly.io.

Limitations & differences with LiteFS Cloud

  • Lfsb does not do any authentication and assumes access across a private network.
  • Lfsb is API-compatible with LiteFS, but its control plane API slightly differs from LiteFS Cloud. Calls from packages like lfsc-go may not work or have expected results.
  • Lfsb expects to be run as a singleton service. Running multiple instances could cause corruption or inconsistencies.

Quickstart: switching from LiteFS Cloud

Lfsb is intended to be a mostly drop-in replacement for LiteFS Cloud. For system requirements, lfsb needs a single-node machine with a persistent disk and s3-compatible object storage.

Deployment on fly.io

Prerequisite: set up a fly.io account and install flyctl.

  1. Launch the app from this repository.
mkdir litefs-backup
cd litefs-backup
fly launch --from git@github.com:stephen/litefs-backup.git --build-only --generate-name --copy-config

When prompted to tweak settings, choose no.

  1. Add tigris object storage and sentry (optional)

If you want to use S3 or another S3-compatible provider, skip this step and use fly secrets set to set the necessary environment variables.

fly ext storage create
fly ext sentry create # optional

These commands should automatically configure the necessary environment variables.

  1. Deploy privately with a single instance
fly deploy --ha=false --no-public-ips --flycast
  1. Finish
fly logs

# Expected output:
# INFO running litefs-backup
# INFO server listening addr=:2200
# INFO waiting for signal or subprocess to exit
# INFO monitoring compaction level=1 interval=10s retention=1h0m0s
# INFO monitoring compaction level=2 interval=5m0s retention=720h0m0s
# INFO monitoring compaction level=3 interval=1h0m0s retention=720h0m0s

This configuration gives the same 5-minute granularity point-in-time restore as LiteFS Cloud.

Configure LiteFS to use lfsb

Use litefs from commit 2e78724973211ba426c224e89d99c256fd6722be or later. If you pull litefs from docker, this looks like:

COPY --from=flyio/litefs:sha-2e78724 /usr/local/bin/litefs /usr/local/bin/litefs

Configure your service running litefs with two environment variables:

  • Set LITEFS_CLOUD_ENDPOINT to the location of your newly deployed lfsb. On fly, this might look like http://someones-litefs-backup.internal:2200.
  • Set LITEFS_CLOUD_TOKEN to cluster [cluster name], e.g. cluster canary. Clusters do not need to be pre-registered.

Using the control plane

Install and configure the cli

Install the release for your system and place it in your $PATH.

curl "https://github.com/stephen/litefs-backup/releases/latest/download/litefs-backup_$(uname -s)_$(uname -m).tar.gz" 2>/dev/null | tar -xz run 2>/dev/null

You can optionally set LFSB_CLUSTER and LFSB_ENDPOINT to your expected cluster and endpoint url, e.g.

export LFSB_CLUSTER="prod" # or --cluster
export LFSB_ENDPOINT="http://someones-litefs-backup.internal:2200" # or --endpoint

If you are using fly.io, setup a wireguard vpn to connect into your private network:

fly wireguard create
Basic operation

List known databases:

lfsb list

List restore points for a database:

lfsb info data.db

Download the current snapshot of a database:

lfsb export data.db

Restore a database to a timestamp or txid:

# first, do a dry run to check that the restore point is available:
lfsb restore data.db --check --txid=[txid] # or --timestamp=[timestamp, e.g. 2024-06-08T12:22:39]

# now, actually run the restore
lfsb restore data.db --txid=[txid]

System overview

Lfsb is organized into clusters. A cluster can contain multiple sqlite databases. For instance, you might have a prod cluster with users.db and datapoints.db and another cluster beta with separate users.db and jobs.db.

Authentication

Lfsb does not support any authentication scheme. To keep API-compatibility with LiteFS, the Authorization header is still used to identify which cluster LiteFS is connecting to.

The authorization header is in the format cluster [cluster name], instead of the LiteFS Cloud format FlyV1 [token].

Storage subsystem

Lfsb stores data in two places:

  • a sqlite metadata db on disk
  • s3-compatible remote storage

Both are expected to be available and durable for the system to function.

Development

Setup

go install github.com/amonks/run/cmd/run@latest
run install

Test

run test
run test-integration

Migrations

dbmate [up, down, new]

Configuration

Configuration is done through environment variables.

LFSB_DATA_PATH (required)

The directory where lfsb will keep its local data store. This should be a persistent, durable location.

LFSB_BIND (optional)

The address to bind to. The default is :2200.

Note that the .envrc sets this to 127.0.0.1:2200 to help with macOS security prompting.

BUCKET_NAME (required)

The AWS S3 bucket name to use.

AWS_ENDPOINT_URL_S3 (optional)

The AWS S3 endpoint to use. The default is AWS S3.

This option is useful for using S3-compatible storage providers.

AWS_REGION (required)

The AWS S3 region to use.

AWS_ACCESS_KEY (required)

The AWS S3 access key to use.

AWS_SECRET_KEY (required)

The AWS S3 secret key to use.

SENTRY_DSN (optional)

The Sentry DSN to use. Sentry reporting will be disabled if unset.

Documentation

Index

Constants

View Source 
const (
	EPOSMISMATCH       = "EPOSMISMATCH"
	ENOCLUSTER         = "ENOCLUSTER"
	ENOCOMPACTION      = "ENOCOMPACTION"
	EPARTIALCOMPACTION = "EPARTIALCOMPACTION"
)

Common error codes. Used for compile-time checks.

View Source 
const (
	// CompactionLevelMax is the highest compaction level allowed.
	CompactionLevelMax = 8

	// CompactionLevelSnapshot is a reserved level for full snapshots only.
	CompactionLevelSnapshot = 9
)
View Source 
const (
	MaxClusterLen  = 32
	MaxDatabaseLen = 256
)

Maximum identifier lengths.

View Source 
const LTXFilenameLen = 16 + 1 + 16 + 4 // min,dash,max,ext

LTXFilenameLen is the length of LTX base filenames.

Variables

View Source 
var (
	ErrInvalidLTXFilename = Errorf(ErrorTypeValidation, "EBADPATH", "invalid ltx filename")

	ErrClusterRequired = Errorf(ErrorTypeValidation, "EBADCLUSTER", "cluster required")
	ErrClusterInvalid  = Errorf(ErrorTypeValidation, "EBADCLUSTER", "cluster invalid")

	ErrDatabaseRequired = Errorf(ErrorTypeValidation, "EBADDB", "database required")
	ErrDatabaseNotFound = Errorf(ErrorTypeNotFound, "ENODB", "database not found")

	ErrMinTXIDRequired = Errorf(ErrorTypeValidation, "EBADTXID", "minimum transaction id required")
	ErrMaxTXIDRequired = Errorf(ErrorTypeValidation, "EBADTXID", "maximum transaction id required")

	ErrCannotCompactToLevelZero = Errorf(ErrorTypeValidation, "EBADLEVEL", "cannot compact to level zero")
	ErrCompactionLevelTooHigh   = Errorf(ErrorTypeValidation, "EBADLEVEL", "compaction level too high")

	ErrTxNotAvailable = Errorf(ErrorTypeNotFound, "ENOTXID", "tx not available")
	// ErrPgnoOutOfBounds = Errorf(ErrorTypeNotFound, "EPGNOOOB", "page number out of bounds")
	ErrPageNotFound = Errorf(ErrorTypeNotFound, "ENOPAGE", "page not found")

	ErrTimestampNotAvailable = Errorf(ErrorTypeNotFound, "ENOTIMESTAMP", "timestamp not available")

	ErrPageSizeMismatch = Errorf(ErrorTypeUnprocessable, "EBADHEADER", "page size mismatch")
)

Functions

func ErrorCode 

func ErrorCode(err error) string

ErrorCode returns the error code from an error. Returns blank if err is nil. Returns EINTERNAL if no lfsc.Error is found.

func FormatLTXFilename 

func FormatLTXFilename(min, max ltx.TXID) string

FormatLTXFilename returns a filename based on the min & max TXID.

func IsApplicationError 

func IsApplicationError(err error) bool

IsApplicationError returns true if err is an lfsc.Error or ltx.PositionMismatchError.

func ParseLTXFilename 

func ParseLTXFilename(name string) (min, max ltx.TXID, err error)

ParseLTXFilename parses a base filename into min & max TXID. Returns ErrInvalidLTXFilename if it is invalid.

func ValidateClusterName 

func ValidateClusterName(s string) error

ValidateClusterName returns nil if s is a valid cluster.

func ValidateDatabase 

func ValidateDatabase(s string) error

ValidateDatabase returns nil if s is a valid database name.

Types

type Config 

type Config struct {
	// Path is the directory where litefs-backup should store its
	// local data.
	Path string

	// Address is the address to bind to.
	Address string

	// S3Bucket is the bucket to use for long term storage.
	S3Bucket string

	// S3Endpoint is the S3 api endpoint to use. Useful for using S3-compatible alternatives.
	S3Endpoint string

	// SentryDSN is the sentry DSN to use.
	// See: https://docs.sentry.io/concepts/key-terms/dsn-explainer/
	SentryDSN string
}

func ConfigFromEnv 

func ConfigFromEnv() (*Config, error)

type Error 

type Error struct {
	Type              ErrorType
	Code              string
	Message           string
	TXID              ltx.TXID
	PostApplyChecksum ltx.Pos
}

func Errorf 

func Errorf(typ ErrorType, code, format string, a ...any) *Error

Errorf is a helper function for returning Error values.

func (*Error) Error 

func (e *Error) Error() string

Error implements Error interace

func (*Error) Is 

func (e *Error) Is(err error) bool

type ErrorType 

type ErrorType string
const (
	ErrorTypeAuth          ErrorType = "auth"
	ErrorTypeConflict      ErrorType = "conflict"
	ErrorTypeNotFound      ErrorType = "notfound"
	ErrorTypeValidation    ErrorType = "validation"
	ErrorTypeUnprocessable ErrorType = "unprocessable"

	ErrorTypeUnknown = "unknown"
)

Source Files

View all Source files

Directories

Path Synopsis
cmd
lfsb command
lfsb-server command
lfsb/cmd
* From https://github.com/superfly/flyctl/blob/19481dfa110c8a6274500321b3214c5d2abbfc71/internal/render/render.go * Copyright 2023 https://github.com/superfly * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * From https://github.com/superfly/flyctl/blob/19481dfa110c8a6274500321b3214c5d2abbfc71/internal/render/render.go * Copyright 2023 https://github.com/superfly * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
lfsb/cmd/client
* Copyright 2023 https://github.com/superfly * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
 * Copyright 2023 https://github.com/superfly * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
db
migrations
sqliteutil
testingutil
s3

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.