persist

package module
v1.9.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 14, 2025 License: MIT Imports: 13 Imported by: 0

README

go-persist

A high-performance, type-safe, persisted key-value store for Go that combines the speed of in-memory maps with the durability of persistent storage.

License Go Reference

🤔 Why Another Key-Value Store?

At first glance, creating yet another embedded key-value database in Go seems redundant. Indeed, popular solutions like Bolt, BuntDB, Badger, or Pebble already exist, each having niche strengths.

However, existing solutions fall short for common Go application patterns:

  • Manual Serialization Overhead: Most existing databases represent values and keys as raw []byte or string. To deal with structured data (e.g., Go structs), you repetitively marshal/unmarshal data (typically via JSON), incurring CPU and GC overhead and complicating your code.

  • Performance vs Persistence Dilemma: High-performance concurrent solutions (sync.Map or mutex-protected map) lack persistence. Persistent databases trade convenience for speed and require complex caching logic to accelerate operations.

  • Complexity of Custom Caching Layers:

    • You often have two sources of truth (cache/in-memory and disk), making synchronization error-prone.
    • Application logic becomes polluted by persistence management.
    • State debugging and consistency across shutdowns/restarts requires significant effort.

Go-persist aims to deliver the exact sweet spot:

  • 🚀 Performance: Near-native sync.Map performance for structured data.
  • 🛡 Safety: Native Go types with transparent persistence.
  • 🔋 Convenience: Simple, intuitive map-like APIs requiring no manual serialization.
  • ⚙️ Flexibility: Explicit control over persistence guarantees (async, immediate, fsync).

📖 More details on design considerations and trade-offs

🧪 Performance Benchmarks Summary

(Intel N100 Quad-Core @3.4GHz, Linux environment)

1M struct operations (150 goroutines), 100K items dataset

Solution Operations/sec ns/op File Size
go-persist Async 7,117,079 140 6.07 MB
sync.Map 5,509,706 181 N/A
map+RWMutex 2,532,314 394 N/A
go-persist Sync 1,463,708 683 6.07 MB
buntdb 251,218 3980 11.15 MB
bolt NoSync 181,481 5510 24.00 MB

Check detailed benchmarks and comparisons in the separate benchmark docs:

📦 Installation

go get github.com/Jipok/go-persist

🚀 Quick Start Example

package main

import (
	"fmt"
	"log"

	"github.com/Jipok/go-persist"
)

type User struct {
    Name  string
    Email string
    Age   int
}

func main() {
    // Open a single persistent map in one call
    users, err := persist.OpenSingleMap[User]("users.db")
    if err != nil {
        log.Fatal(err)
    }
    defer users.Store.Close()

    // Store a user with balanced performance/durability
    users.Set("alice", User{
        Name: "Alice Smith",
        Email: "alice@example.com",
        Age: 28,
    })

    // Retrieve a user
    john, ok := users.Get("alice")
    if !ok {
        log.Fatal("User not found")
    }
    fmt.Printf("User: %+v\n", john)

    // Atomically update a user's age
    users.Update("alice", func(upd *persist.Update[User]) {
        if !upd.Exists {
            upd.Cancel() // Don't do anything if user doesn't exist
            return
        }
        // Modify the value directly
        upd.Value.Age++
    })

    // Count users
    fmt.Printf("Total users: %d\n", users.Size())

    // Iterate through all users
    users.Range(func(key string, value User) bool {
        fmt.Printf("Key: %s, User: %+v\n", key, value)
        return true // continue iteration
    })

    // Delete a user
    users.Delete("alice")
}

🛠️ API Examples

Using Multiple Typed Maps Together 
package main

import (
    "log"
    "time"
    "github.com/Jipok/go-persist"
)

type User struct {
    Name  string
    Age   int
}

type Product struct {
    Name  string
    Price float64
}

type Session struct {
    UserID     string
    Expiration int64
}

func main() {
    store := persist.New()
    defer store.Close()

    // Create typed maps for different entity types
    users, _ := persist.Map[User](store, "users")
    products, _ := persist.Map[Product](store, "products")
    sessions, _ := persist.Map[Session](store, "sessions")

    // Create or load store file
    err := store.Open("app.db")
    if err != nil {
        log.Fatal(err)
    }

    // Set up automatic compaction
    store.StartAutoShrink(time.Minute, 1.8)

    // Use each map independently
    users.Set("u1", User{Name: "Admin", Age: 35})
    products.Set("p1", Product{Name: "Widget", Price: 19.99})
    sessions.SetAsync("sess123", Session{UserID: "u1", Expiration: 1718557123})
}
PersistMap Methods 
// Retrieve data
value, exists := myMap.Get("key")

// Store data with different durability options
myMap.SetAsync("key", value)         // High performance, background persistence
myMap.Set("key", value)              // Balanced performance and durability
err := myMap.SetFSync("key", value)  // Maximum durability with fsync

// Delete data
myMap.DeleteAsync("key")             // Background delete
myMap.Delete("key")                  // Immediate WAL write
err := myMap.DeleteFSync("key")      // With fsync for maximum durability

// Atomic updates with different durability levels
newVal, existed := myMap.UpdateAsync("key", func(upd *persist.Update[T]) {
    // Modify upd.Value directly (default action is "set")
    // Or explicitly call:
    // upd.Set(newValue)    // to update the value
    // upd.Delete()         // to delete the key
    // upd.Cancel()         // to keep original value unchanged
})

newVal, existed := myMap.Update("key", func(upd *persist.Update[T]) {
    // Same options as above
})

newVal, existed, err := myMap.UpdateFSync("key", func(upd *persist.Update[T]) {
    // Same options as above
})

// Get number of items
count := myMap.Size()

// Iterate through all items
myMap.Range(func(key string, value ValueType) bool {
    // Process each item
    return true // return true to continue, false to stop
})

// Memory-only operations (for non-exported or derived fields)
myMap.SetInMemory("key", value)
myMap.UpdateInMemory("key", func(upd *persist.Update[T]) {
    // Only set action allowed
})
Using the Basic Store API 
type Config struct {
    Debug          bool
    MaxConnections int
}

func main() {
    store := persist.New()
    err := store.Open("app.db")
    if err != nil {
        log.Fatal(err)
    }
    defer store.Close()

    // Store configuration directly
    err = store.Set("system_config", Config{
        Debug:          true,
        MaxConnections: 100,
    })
    if err != nil {
        log.Fatal(err)
    }

    config, err := persist.Get[Config](store, "system_config")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Config: Debug=%v, MaxConnections=%d\n", config.Debug, config.MaxConnections)

    // When you need to update the config
    config.MaxConnections = 200
    err = store.Set("system_config", config)
    if err != nil {
        log.Fatal(err)
    }
}
WAL Management and Monitoring 
// Force immediate durability of all data
if err := store.FSyncAll(); err != nil {
    log.Fatal("Failed to sync data to disk:", err)
}

// Get statistics about the store
activeKeys, walRecords := store.Stats()
fmt.Printf("Active keys: %d, WAL records: %d, Ratio: %.2f\n", 
    activeKeys, walRecords, float64(walRecords)/float64(activeKeys))

// Manually compact the WAL file to reclaim space
if err := store.Shrink(); err != nil {
    log.Fatal(err)
}

// Or set up automatic compaction when record count exceeds 2x the active keys
store.StartAutoShrink(1*time.Minute, 2.0) // Check ratio every minute
🚩 Performance Control

go-persist lets you choose your performance and durability trade-off explicitly:

Method Performance Persistence Guarantee
.SetAsync() Highest 🚀 Background persistence (potential loss on app crash)
.Set() Balanced Immediate WAL (application failure safe, OS failure risk)
.SetFSync() Lowest 🔐 Immediate WAL+fsync (fully durable, failure-proof)
📚 Details
Durability Levels

  1. Async Methods (SetAsync, DeleteAsync, UpdateAsync): Highest performance with deferred persistence.

    • Updates are applied in-memory immediately
    • Changes are flushed to disk by a background process
    • Best for high-throughput scenarios where occasional data loss on crashes is acceptable

  2. Immediate Methods (Set, Delete, Update): Balanced performance with immediate WAL updates.

    • Updates are applied in-memory and written to WAL immediately
    • Safe against application crashes, but susceptible to system crashes
    • Good for most typical use cases

  3. FSync Methods (SetFSync, DeleteFSync, UpdateFSync): Maximum durability with fsync guarantee.

    • Updates are written to WAL and flushed to physical disk with fsync
    • Safe against both application and system crashes
    • Use when data integrity is critical
    • See Design Trade-offs
Configuring Sync Interval

The sync interval controls:

  • When batched Async operations are written to the WAL file
  • When regular Set operations are synced from OS page cache to physical disk
// Get the current sync interval
interval := store.GetSyncInterval()

// Set a custom sync interval
store.SetSyncInterval(500 * time.Millisecond) // More frequent syncing
// or
store.SetSyncInterval(1 * time.Second)  // Default
// or
store.SetSyncInterval(10 * time.Minute) // Minimal disk activity

Adjusting the sync interval lets you fine-tune the trade-off between performance and durability:

  • Short intervals (milliseconds to second): Reduce potential data loss window but cause more frequent disk activity
  • Medium intervals (seconds): Good balance for most applications
  • Long intervals (minutes to hours): Minimize disk activity and extend SSD/HDD lifespan, but with larger potential data loss windows in case of crashes

With very long intervals, Async operations will cause practically no disk writes during normal operation, making this option excellent for conserving storage device lifespan when persistence is mainly needed for planned shutdowns rather than crash recovery.

For the Set method, even with a very long sync interval, changes are initially written to the OS page cache. The system itself will eventually flush these changes to disk (i.e., perform an fsync) according to its own caching policies. On Linux, by default:

  • The parameter /proc/sys/vm/dirty_writeback_centisecs is typically set to 500 (≈5 seconds), meaning the kernel scans for dirty pages and may flush them every ~5 seconds.
  • The parameter /proc/sys/vm/dirty_expire_centisecs is usually around 3000 (≈30 seconds), so pages older than ~30 seconds are forced to be written to disk.

📋 Human-readable and writable WAL format

go-persist 1                                          # Version header
S key1                                                # Set operation for key1
{"Name":"Alice","Age":30,"Email":"alice@example.com"} # Value
S key2                                                # Set operation for key2
"some plain string"                                   # Value
D key1                                                # Delete operation for key1
                                                      # Empty string for delete op
S key2                                                # New version of key2
"some another plain string"                           # Updated value
  • S: Set an operation with a valid JSON payload
  • D: Delete the key
  • Easy to inspect and debug without special tools

📌 Intended Use Cases

  • Development and rapid prototyping: simple setup without boilerplate.
  • Configuration storage: typed data, effortlessly saved to disk.
  • Local persistent caching layer: structured data requiring fast, concurrent access.
  • Applications with moderate data volume (up to a few GB of RAM).

🚧 Design Trade-offs

Human-Readable Format vs Checksums

  • The WAL format prioritizes human readability and debuggability
  • No built-in checksums or CRCs to validate file integrity
  • If you need stronger corruption detection, consider using this on a filesystem with checksumming (like ZFS, btrfs) and RAID
  • Can detect incomplete format entries from crashes but can't detect bitrot or partial corruption within a syntactically valid entry

Memory-First Approach

  • All data is kept in memory for maximum performance
  • Not suitable for datasets larger than available RAM
  • Provides map-like access patterns rather than database query capabilities
Performance and Scale Considerations
  • Tested with datasets up to ~1GB of real data
  • WAL file grows unbounded until Shrink() is called
  • No complex recovery mechanisms, distributed capabilities, or transaction isolation
  • Memory usage scales linearly with dataset size, with reasonable overhead compared to raw data

For applications requiring complex queries, distributed access, full ACID compliance, strong data integrity guarantees, protection against hardware failures, or datasets larger than available memory, a traditional database system would be more appropriate.

Documentation

Index

Constants

View Source 
const DefaultSyncInterval = time.Second

Default value for store.syncInterval

View Source 
const WalHeader = "go-persist 1"

Identification string written at the beginning of each WAL file to validate file format and version compatibility

Variables

View Source 
var (
	ErrMapAlreadyExists = errors.New("persist map with the given name already exists in store")
	ErrTooLate          = errors.New("cannot register new map after store has been loaded")
)
View Source 
var (
	ErrKeyNotFound      = errors.New("key not found")
	ErrNotLoaded        = errors.New("store is not loaded")
	ErrShrinkInProgress = errors.New("shrink operation is already in progress")
)

Functions

func Get added in v1.7.0 

func Get[T any](s *Store, key string) (T, error)

Get retrieves a typed value from orphaned records. Returns ErrKeyNotFound if the key doesn't exist or was deleted in the most recent operation.

func ValidateKey added in v1.5.0 

func ValidateKey(key string) error

ValidateKey validates the provided key ensuring it is not empty and that it does not include forbidden characters: ASCII (0x00–0x1F, 0x7F) and additional ones in the extended control range (0x80–0x9F).

Types

type PersistMap 

type PersistMap[T any] struct {
	Store *Store // underlying WAL store
	// contains filtered or unexported fields
}

func Map 

func Map[T any](store *Store, mapName string) (*PersistMap[T], error)

Map creates or loads PersistMap from store.

PersistMap represents a thread-safe persistent key-value store with type-safe values. It maintains an in-memory map for fast access while ensuring durability through the WAL.

The mapName parameter is used as a namespace: keys will be stored as "mapName:key" in the WAL.

func OpenSingleMap added in v1.5.0 

func OpenSingleMap[T any](path string) (*PersistMap[T], error)

OpenSingleMap is the simplest way to get started with a persistent map when you need just one map per file. It opens the store, compacts the WAL, and initializes the map in a single operation.

It returns a PersistMap that represents a thread-safe persistent key-value store with type-safe values of type T. This map maintains an in-memory representation for fast access while ensuring durability through the WAL.

func (*PersistMap[T]) Delete 

func (pm *PersistMap[T]) Delete(key string) (existed bool)

Delete immediately deletes the key from both WAL and in-memory map Returns true if the key existed and was deleted

func (*PersistMap[T]) DeleteAsync 

func (pm *PersistMap[T]) DeleteAsync(key string) (existed bool)

DeleteAsync removes the key from the in-memory map and marks it as dirty for background flush Returns true if the key existed and was deleted

func (*PersistMap[T]) DeleteFSync 

func (pm *PersistMap[T]) DeleteFSync(key string) error

DeleteFSync writes a delete record to WAL immediately, flushes to disk (fsync), and updates the in-memory map.

func (*PersistMap[T]) Get 

func (pm *PersistMap[T]) Get(key string) (T, bool)

Get retrieves the value associated with the key from the in-memory map.

Returns the value and true if the key exists, or a zero value and false otherwise.

func (*PersistMap[T]) Range added in v1.2.0 

func (pm *PersistMap[T]) Range(f func(key string, value T) bool)

Range calls f sequentially for each key and value present in the map. If f returns false, range stops the iteration.

Range does not necessarily correspond to any consistent snapshot of the Map's contents: no key will be visited more than once, but if the value for any key is stored or deleted concurrently, Range may reflect any mapping for that key from any point during the Range call.

It is safe to modify the map while iterating it, including entry creation, modification and deletion. However, the concurrent modification rule apply, i.e. the changes may be not reflected in the subsequently iterated entries.

func (*PersistMap[T]) Set 

func (pm *PersistMap[T]) Set(key string, value T)

Set updates both in-memory data and WAL file immediately, but without fsync.

Safe for application crashes, as WAL ensures recovery, but may lose updates during system crashes if data remains in OS cache.

func (*PersistMap[T]) SetAsync 

func (pm *PersistMap[T]) SetAsync(key string, value T)

SetAsync updates the in-memory map and marks the key as dirty.

Its actual persistence is deferred to a background flush, providing higher performance at the cost of delayed durability.

func (*PersistMap[T]) SetFSync 

func (pm *PersistMap[T]) SetFSync(key string, value T) error

SetFSync updates in-memory data, WAL file, and forces physical disk write with fsync.

Most durable option that protects against both application and system crashes, but with highest performance cost.

func (*PersistMap[T]) SetInMemory added in v1.9.0 

func (pm *PersistMap[T]) SetInMemory(key string, value T)

SetInMemory updates the value in memory only without explicitly writing to WAL or marking the key as dirty. This change won't trigger immediate persistence, but it may be persisted if:

- This key is later modified with Set/Update/etc.

- A Shrink operation occurs

Useful for non-exported, derived, or cached fields.

func (*PersistMap[T]) Size added in v1.2.0 

func (pm *PersistMap[T]) Size() int

Size returns current size of the map

func (*PersistMap[T]) Sync added in v1.3.0 

func (pm *PersistMap[T]) Sync()

Sync writes all pending changes made by Async methods to the WAL file. It processes all keys marked as dirty, persisting their current values or deletion status, then clears their dirty flags upon successful write.

Sync is automatically called periodically from the store's background synchronization process. This method only ensures consistency between memory and the WAL file, but doesn't guarantee data is physically written to disk - that step is handled by the Store.Flush method.

func (*PersistMap[T]) Update added in v1.3.0 

func (pm *PersistMap[T]) Update(key string, updater func(upd *Update[T])) (newValue T, exists bool)

Update atomically updates the value for the given key using the updater function, and immediately writes the change to the WAL (without forcing disk fsync).

The updater receives an UpdateAction containing the current value and existence state. It can:

- Simply modify upd.Value to update the value (default action is "set")

- Call upd.Set() to explicitly set a new value

- Call upd.Delete() to remove the key

- Call upd.Cancel() to keep the original value unchanged

This method locks the relevant hash table bucket during execution, so avoid long-running operations in the updater function to prevent blocking other bucket operations.

func (*PersistMap[T]) UpdateAsync added in v1.3.0 

func (pm *PersistMap[T]) UpdateAsync(key string, updater func(upd *Update[T])) (newValue T, exists bool)

UpdateAsync atomically updates a key using the updater function.

It only updates the in-memory value and marks the key as dirty so that background FSyncAll will eventually persist the change.

The updater receives an UpdateAction containing the current value and existence state. It can:

- Simply modify upd.Value to update the value (default action is "set")

- Call upd.Set() to explicitly set a new value

- Call upd.Delete() to remove the key

- Call upd.Cancel() to keep the original value unchanged

This method locks the relevant hash table bucket during execution, so avoid long-running operations in the updater function to prevent blocking other bucket operations.

func (*PersistMap[T]) UpdateFSync added in v1.3.0 

func (pm *PersistMap[T]) UpdateFSync(key string, updater func(upd *Update[T])) (newValue T, exists bool, err error)

UpdateFSync atomically updates a key using the updater function, writes to the WAL, and forces a physical disk flush (fsync).

Offers maximum durability against both application and system crashes, but with the highest performance cost.

The updater receives an UpdateAction containing the current value and existence state. It can:

- Simply modify upd.Value to update the value (default action is "set")

- Call upd.Set() to explicitly set a new value

- Call upd.Delete() to remove the key

- Call upd.Cancel() to keep the original value unchanged

This method locks the relevant hash table bucket during execution, so avoid long-running operations in the updater function to prevent blocking other bucket operations.

func (*PersistMap[T]) UpdateInMemory added in v1.9.0 

func (pm *PersistMap[T]) UpdateInMemory(key string, updater func(upd *Update[T])) T

UpdateInMemory atomically updates a value in memory only without writing to WAL or marking the key as dirty. This change won't trigger immediate persistence, but will be persisted if:

- This key is later modified with Set/Update/etc.

- A Shrink operation occurs

The method uses the same Update struct as regular Update methods for API consistency, but only supports the Set action. Attempts to use Delete or Cancel will cause panic.

Useful for non-exported, derived, or cached fields.

type Store 

type Store struct {

	//
	ErrorHandler     func(err error)
	ReaderBufferSize int
	// contains filtered or unexported fields
}

Store represents the WAL(write-ahead log) storage

func New added in v1.7.0 

func New() *Store

New creates and initializes a new Store instance.

The returned Store is not yet connected to any file - you must call Open() with a file path to load existing data or create a new persistence file.

By default, the Store is configured with:

- DefaultSyncInterval (1 second) for background synchronization

- A default error handler that calls log.Fatal

- Empty maps for tracking PersistMap instances and orphaned records

Example usage: 

store := persist.New()
err := store.Open("mydata.db")
if err != nil {
    log.Fatal(err)
}
defer store.Close()

func (*Store) Close 

func (s *Store) Close() error

Saves all pending changes and stops the background sync goroutine Then closes the underlying file.

The Store should not be used after calling Close.

func (*Store) Delete 

func (s *Store) Delete(key string) error

Delete marks a key as deleted by writing a "delete" record to the log. The record format consists of two lines:

  1. D <key>
  2. <Empty value line>

The newline after the empty value line serves as a marker that the delete record was successfully written and can be safely processed during recovery.

func (*Store) FSyncAll added in v1.3.0 

func (s *Store) FSyncAll() error

FSyncAll ensures complete data durability by:

  1. Synchronizing all dirty map entries to the WAL file
  2. Performing an fsync operation to guarantee data is physically written to disk

This operation provides the strongest durability guarantee, protecting against both application crashes and system failures. It's automatically called periodically based on the configured syncInterval, but can also be called manually when immediate durability is required.

func (*Store) GetSyncInterval 

func (s *Store) GetSyncInterval() time.Duration

SetSyncInterval configures how frequently the background goroutine will call FSyncAll() to ensure all changes are durably committed to disk

func (*Store) Open added in v1.7.0 

func (s *Store) Open(path string) error

Open opens the persistent storage file, validates/writes the WAL header, starts the background sync goroutine and immediately loads all WAL records into the registered maps.

func (*Store) Set 

func (s *Store) Set(key string, value interface{}) error

Set persists a key-value pair by writing a "set" record to the WAL log and updates the corresponding entry in orphanRecords.

This is a synchronous operation that writes to the WAL file immediately, but without fsync.

func (*Store) SetSyncInterval 

func (s *Store) SetSyncInterval(interval time.Duration)

SetSyncInterval sets a new sync interval

func (*Store) Shrink 

func (s *Store) Shrink() error

Shrink compacts the WAL file by discarding deleted records and redundant updates, retaining only the latest state for each key. The operation is designed to be minimally blocking:

  1. Most compaction happens without locks, allowing concurrent operations
  2. Operations performed during shrinking are captured and preserved
  3. Only brief locks are used to swap files and finalize pending operations

The function creates a temporary file with current state only, then atomically replaces the original WAL file.

func (*Store) StartAutoShrink added in v1.8.0 

func (s *Store) StartAutoShrink(checkInterval time.Duration, shrinkRatio float64) error

StartAutoShrink initiates a background goroutine that automatically compacts the WAL file at regular intervals when certain conditions are met.

Parameters:

  • checkInterval: How frequently to check if compaction is needed
  • shrinkRatio: The threshold ratio of (WAL records)/(active keys) that triggers shrinking

func (*Store) Stats added in v1.8.0 

func (s *Store) Stats() (activeKeys int32, walRecords int32)

Stats returns statistics about the store state.

Returns:

  • activeKeys: The total number of currently active (non-deleted) keys across all registered maps and orphan records. This represents the actual data items currently stored.
  • walRecords: The total number of records written to the WAL file since opening, including both set and delete operations. This can be significantly higher than activeKeys due to updates and deletions.

This method can be useful for monitoring storage growth and determining when a Shrink() operation might be beneficial (when walRecords is much larger than activeKeys).

type Update added in v1.6.1 

type Update[T any] struct {
	Value  T    // Current value retrieved from the map
	Exists bool // Whether the key exists
	// contains filtered or unexported fields
}

Update encapsulates the state for updating a key in the map. It holds the current value and its existence flag, and allows the updater to specify the intended action: update (set), deletion, or cancellation. By default, the action is set to update, so modifying the Value field directly implies a "set" operation.

func (*Update[T]) Cancel added in v1.6.1 

func (ua *Update[T]) Cancel()

Cancel indicates that no changes should be applied

func (*Update[T]) Delete added in v1.6.1 

func (ua *Update[T]) Delete()

Delete indicates that the key should be deleted

func (*Update[T]) Set added in v1.6.1 

func (ua *Update[T]) Set(newVal T)

Set updates the internal value and marks the action as "set". Note that simply modifying the Value field directly also works since "set" is the default action.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.