table

package
v0.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 2, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package table provides SST file reading and writing.

TableBuilder creates SST files in the block-based table format. The format is compatible with RocksDB v10.7.5.

Reference: RocksDB v10.7.5

  • table/block_based/block_based_table_builder.h
  • table/block_based/block_based_table_builder.cc
  • table/table_builder.h

Whitebox Testing Hooks

This file contains kill points for crash testing (requires -tags crashtest). In production builds, these compile to no-ops with zero overhead. See docs/testing/README.md for usage.

Package table provides SST file reading and writing functionality. This implements the RocksDB block-based table format (format_version 0-7).

SST File Layout: 

[data block 1]
[data block 2]
...
[data block N]
[meta block 1: filter block]     (optional)
[meta block 2: index block]
[meta block 3: compression dict] (optional)
[meta block 4: range del block]  (optional)
[meta block 5: properties block]
[metaindex block]
[Footer]                         (fixed size, at end of file)

Reference: RocksDB v10.7.5

  • table/block_based/block_based_table_reader.h
  • table/format.h
  • table/format.cc

Index

Constants

View Source 
const (
	PropDBID                           = "rocksdb.creating.db.identity"
	PropDBSessionID                    = "rocksdb.creating.session.identity"
	PropDBHostID                       = "rocksdb.creating.host.identity"
	PropOriginalFileNumber             = "rocksdb.original.file.number"
	PropDataSize                       = "rocksdb.data.size"
	PropIndexSize                      = "rocksdb.index.size"
	PropIndexPartitions                = "rocksdb.index.partitions"
	PropTopLevelIndexSize              = "rocksdb.top-level.index.size"
	PropIndexKeyIsUserKey              = "rocksdb.index.key.is.user.key"
	PropIndexValueIsDeltaEncoded       = "rocksdb.index.value.is.delta.encoded"
	PropFilterSize                     = "rocksdb.filter.size"
	PropRawKeySize                     = "rocksdb.raw.key.size"
	PropRawValueSize                   = "rocksdb.raw.value.size"
	PropNumDataBlocks                  = "rocksdb.num.data.blocks"
	PropNumEntries                     = "rocksdb.num.entries"
	PropNumFilterEntries               = "rocksdb.num.filter.entries"
	PropDeletedKeys                    = "rocksdb.deleted.keys"
	PropMergeOperands                  = "rocksdb.merge.operands"
	PropNumRangeDeletions              = "rocksdb.num.range-deletions"
	PropFormatVersion                  = "rocksdb.format.version"
	PropFixedKeyLen                    = "rocksdb.fixed.key.length"
	PropFilterPolicy                   = "rocksdb.filter.policy"
	PropColumnFamilyName               = "rocksdb.column.family.name"
	PropColumnFamilyID                 = "rocksdb.column.family.id"
	PropComparator                     = "rocksdb.comparator"
	PropMergeOperator                  = "rocksdb.merge.operator"
	PropPrefixExtractorName            = "rocksdb.prefix.extractor.name"
	PropPropertyCollectors             = "rocksdb.property.collectors"
	PropCompression                    = "rocksdb.compression"
	PropCompressionOptions             = "rocksdb.compression_options"
	PropCreationTime                   = "rocksdb.creation.time"
	PropOldestKeyTime                  = "rocksdb.oldest.key.time"
	PropNewestKeyTime                  = "rocksdb.newest.key.time"
	PropFileCreationTime               = "rocksdb.file.creation.time"
	PropSlowCompressionEstimatedSize   = "rocksdb.sample_for_compression"
	PropFastCompressionEstimatedSize   = "rocksdb.sample_for_compression.2"
	PropTailStartOffset                = "rocksdb.tail.start.offset"
	PropUserDefinedTimestampsPersisted = "rocksdb.user.defined.timestamps.persisted"
	PropKeyLargestSeqno                = "rocksdb.key.largest.seqno"
	PropKeySmallestSeqno               = "rocksdb.key.smallest.seqno"
)

Property name constants from RocksDB. Reference: include/rocksdb/table_properties.h

Variables

View Source 
var (
	// ErrInvalidSST indicates the file is not a valid SST file.
	ErrInvalidSST = errors.New("table: invalid SST file")

	// ErrUnsupportedVersion indicates the format version is not supported.
	ErrUnsupportedVersion = errors.New("table: unsupported format version")

	// ErrChecksumMismatch indicates a block checksum verification failed.
	ErrChecksumMismatch = errors.New("table: checksum mismatch")

	// ErrBlockNotFound indicates a requested block was not found.
	ErrBlockNotFound = errors.New("table: block not found")

	// ErrUnsupportedPartitionedIndex indicates the SST uses partitioned index which is not supported.
	// Partitioned index splits the index across multiple blocks; our reader treats the index
	// as a single block and would produce incorrect results.
	ErrUnsupportedPartitionedIndex = errors.New("table: partitioned index not supported")
)

Functions

This section is empty.

Types

type BuilderOptions 

type BuilderOptions struct {
	// BlockSize is the target size for data blocks (default: 4KB).
	BlockSize int

	// BlockRestartInterval is the number of keys between restart points (default: 16).
	BlockRestartInterval int

	// FormatVersion is the SST format version (default: 6 for v10.7.5 compatibility).
	FormatVersion uint32

	// ChecksumType is the checksum algorithm (default: XXH3).
	ChecksumType checksum.Type

	// ComparatorName is the name of the key comparator.
	ComparatorName string

	// ColumnFamilyID is the column family ID.
	ColumnFamilyID uint32

	// ColumnFamilyName is the column family name.
	ColumnFamilyName string

	// FilterBitsPerKey controls Bloom filter accuracy (default: 10 = ~1% FP rate).
	// Set to 0 to disable filter.
	FilterBitsPerKey int

	// FilterPolicy is the name of the filter policy (e.g., "rocksdb.BuiltinBloomFilter").
	FilterPolicy string

	// Compression is the compression type for data blocks.
	Compression compression.Type
}

BuilderOptions configures the TableBuilder.

func DefaultBuilderOptions 

func DefaultBuilderOptions() BuilderOptions

DefaultBuilderOptions returns default options for TableBuilder.

type IndexBlockIterator 

type IndexBlockIterator struct {
	// contains filtered or unexported fields
}

IndexBlockIterator is a specialized iterator for index blocks. Index blocks use value_delta_encoding (format_version >= 4), where: - Entries have format: <shared:byte><non_shared:byte><key_delta><value> - No value_size is stored; value extends to next entry or end of data

func NewIndexBlockIterator 

func NewIndexBlockIterator(data []byte, dataEnd int) *IndexBlockIterator

NewIndexBlockIterator creates an iterator for an index block.

func (*IndexBlockIterator) Key 

func (it *IndexBlockIterator) Key() []byte

func (*IndexBlockIterator) Next 

func (it *IndexBlockIterator) Next()

func (*IndexBlockIterator) Prev 

func (it *IndexBlockIterator) Prev()

Prev moves to the previous entry.

func (*IndexBlockIterator) Seek 

func (it *IndexBlockIterator) Seek(target []byte)

func (*IndexBlockIterator) SeekToFirst 

func (it *IndexBlockIterator) SeekToFirst()

func (*IndexBlockIterator) SeekToLast 

func (it *IndexBlockIterator) SeekToLast()

func (*IndexBlockIterator) Valid 

func (it *IndexBlockIterator) Valid() bool

func (*IndexBlockIterator) Value 

func (it *IndexBlockIterator) Value() []byte

type ReadableFile 

type ReadableFile interface {
	io.Closer

	// ReadAt reads len(p) bytes from the file starting at offset.
	ReadAt(p []byte, off int64) (n int, err error)

	// Size returns the total size of the file.
	Size() int64
}

ReadableFile is an interface for reading from an SST file.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads an SST file in the block-based table format.

func Open 

func Open(file ReadableFile, opts ReaderOptions) (*Reader, error)

Open opens an SST file for reading.

func (*Reader) Close 

func (r *Reader) Close() error

Close releases resources associated with the reader.

func (*Reader) Footer 

func (r *Reader) Footer() *block.Footer

Footer returns the parsed footer.

func (*Reader) GetRangeTombstoneList 

func (r *Reader) GetRangeTombstoneList() (*rangedel.TombstoneList, error)

GetRangeTombstoneList returns the raw (non-fragmented) tombstone list.

func (*Reader) GetRangeTombstones 

func (r *Reader) GetRangeTombstones() (*rangedel.FragmentedRangeTombstoneList, error)

GetRangeTombstones reads and parses range tombstones from the SST file. Returns a FragmentedRangeTombstoneList for efficient lookup. Returns an empty list if there are no range tombstones.

func (*Reader) HasFilter 

func (r *Reader) HasFilter() bool

HasFilter returns true if this table has a Bloom filter.

func (*Reader) HasRangeTombstones 

func (r *Reader) HasRangeTombstones() bool

HasRangeTombstones returns true if the SST file contains range tombstones.

func (*Reader) KeyMayMatch 

func (r *Reader) KeyMayMatch(key []byte) bool

KeyMayMatch returns true if the key may be in this SST file. Returns true (may match) if: - No filter is present - The filter indicates the key might be present Returns false (definitely not present) if the filter says the key is not present.

func (*Reader) NewIterator 

func (r *Reader) NewIterator() *TableIterator

NewIterator returns an iterator over the table contents. The iterator is initially invalid; call SeekToFirst or Seek before use.

func (*Reader) Properties 

func (r *Reader) Properties() (*TableProperties, error)

Properties returns the table properties, loading them if necessary.

type ReaderOptions 

type ReaderOptions struct {
	// VerifyChecksums enables checksum verification for all blocks.
	VerifyChecksums bool

	// CacheBlocks enables caching of data blocks.
	// (Not implemented yet - for future block cache integration)
	CacheBlocks bool
}

ReaderOptions controls the behavior of the table reader.

type TableBuilder 

type TableBuilder struct {
	// contains filtered or unexported fields
}

TableBuilder builds SST files in the block-based table format.

func NewTableBuilder 

func NewTableBuilder(w io.Writer, opts BuilderOptions) *TableBuilder

NewTableBuilder creates a new TableBuilder that writes to w.

func (*TableBuilder) Abandon 

func (tb *TableBuilder) Abandon()

Abandon abandons the table being built. After calling Abandon, the TableBuilder should not be used.

func (*TableBuilder) Add 

func (tb *TableBuilder) Add(key, value []byte) error

Add adds a key-value pair to the table. Keys must be added in sorted order.

func (*TableBuilder) AddFragmentedRangeTombstones 

func (tb *TableBuilder) AddFragmentedRangeTombstones(list *rangedel.FragmentedRangeTombstoneList) error

AddFragmentedRangeTombstones adds tombstones from a fragmented list.

func (*TableBuilder) AddRangeTombstone 

func (tb *TableBuilder) AddRangeTombstone(startKey, endKey []byte, seqNum dbformat.SequenceNumber) error

AddRangeTombstone adds a range deletion to the table. The range [startKey, endKey) will be marked as deleted at the given sequence number. Range tombstones are stored in a separate block from regular data.

func (*TableBuilder) AddRangeTombstones 

func (tb *TableBuilder) AddRangeTombstones(list *rangedel.TombstoneList) error

AddRangeTombstones adds multiple range tombstones from a tombstone list.

func (*TableBuilder) FileSize 

func (tb *TableBuilder) FileSize() uint64

FileSize returns the size of the file generated so far.

func (*TableBuilder) Finish 

func (tb *TableBuilder) Finish() error

Finish finalizes the table and writes the footer. After calling Finish, the TableBuilder should not be used.

func (*TableBuilder) HasRangeTombstones 

func (tb *TableBuilder) HasRangeTombstones() bool

HasRangeTombstones returns true if range tombstones have been added.

func (*TableBuilder) NumEntries 

func (tb *TableBuilder) NumEntries() uint64

NumEntries returns the number of entries added so far.

func (*TableBuilder) Status 

func (tb *TableBuilder) Status() error

Status returns any error encountered during building.

type TableCache 

type TableCache struct {
	// contains filtered or unexported fields
}

TableCache caches open SST file readers to avoid repeatedly opening files. It uses an LRU-style eviction policy when the cache is full.

func NewTableCache 

func NewTableCache(fs vfs.FS, opts TableCacheOptions) *TableCache

NewTableCache creates a new TableCache.

func (*TableCache) Close 

func (tc *TableCache) Close() error

Close closes all cached readers and clears the cache.

func (*TableCache) Evict 

func (tc *TableCache) Evict(fileNum uint64)

Evict removes a specific file from the cache.

func (*TableCache) Get 

func (tc *TableCache) Get(fileNum uint64, path string) (*Reader, error)

Get returns a Reader for the given file. The caller must call Release() when done with the reader.

func (*TableCache) NewIterator 

func (tc *TableCache) NewIterator(fileNum uint64, path string) (*TableIterator, error)

NewIterator creates an iterator over an SST file.

func (*TableCache) Release 

func (tc *TableCache) Release(fileNum uint64)

Release decrements the reference count for a reader. The reader may be evicted from the cache if it has no more references.

func (*TableCache) Size 

func (tc *TableCache) Size() int

Size returns the current number of cached readers.

type TableCacheOptions 

type TableCacheOptions struct {
	// MaxOpenFiles is the maximum number of SST files to keep open.
	MaxOpenFiles int

	// VerifyChecksums enables checksum verification when reading blocks.
	VerifyChecksums bool
}

TableCacheOptions configures the TableCache.

func DefaultTableCacheOptions 

func DefaultTableCacheOptions() TableCacheOptions

DefaultTableCacheOptions returns default options.

type TableIterator 

type TableIterator struct {
	// contains filtered or unexported fields
}

TableIterator iterates over key-value pairs in an SST file.

func (*TableIterator) Error 

func (it *TableIterator) Error() error

Error returns any error encountered during iteration.

func (*TableIterator) Key 

func (it *TableIterator) Key() []byte

Key returns the current key.

func (*TableIterator) Next 

func (it *TableIterator) Next()

Next moves to the next entry.

func (*TableIterator) Prev 

func (it *TableIterator) Prev()

Prev moves to the previous entry.

func (*TableIterator) Seek 

func (it *TableIterator) Seek(target []byte)

Seek positions the iterator at the first entry with key >= target.

func (*TableIterator) SeekToFirst 

func (it *TableIterator) SeekToFirst()

SeekToFirst positions the iterator at the first entry.

func (*TableIterator) SeekToLast 

func (it *TableIterator) SeekToLast()

SeekToLast positions the iterator at the last entry.

func (*TableIterator) Valid 

func (it *TableIterator) Valid() bool

Valid returns true if the iterator is positioned at a valid entry.

func (*TableIterator) Value 

func (it *TableIterator) Value() []byte

Value returns the current value.

type TableProperties 

type TableProperties struct {
	// Basic statistics
	DataSize          uint64
	IndexSize         uint64
	IndexPartitions   uint64
	TopLevelIndexSize uint64
	FilterSize        uint64
	RawKeySize        uint64
	RawValueSize      uint64
	NumDataBlocks     uint64
	NumEntries        uint64
	NumFilterEntries  uint64
	NumDeletions      uint64
	NumMergeOperands  uint64
	NumRangeDeletions uint64
	FormatVersion     uint64
	FixedKeyLen       uint64
	ColumnFamilyID    uint64
	CreationTime      uint64
	OldestKeyTime     uint64
	NewestKeyTime     uint64
	FileCreationTime  uint64
	OrigFileNumber    uint64
	TailStartOffset   uint64
	KeyLargestSeqno   uint64
	KeySmallestSeqno  uint64

	// Boolean-like properties (stored as uint64)
	IndexKeyIsUserKey              uint64
	IndexValueIsDeltaEncoded       uint64
	UserDefinedTimestampsPersisted uint64
	SlowCompressionEstimatedSize   uint64
	FastCompressionEstimatedSize   uint64

	// String properties
	DBID                    string
	DBSessionID             string
	DBHostID                string
	FilterPolicyName        string
	ColumnFamilyName        string
	ComparatorName          string
	MergeOperatorName       string
	PrefixExtractorName     string
	PropertyCollectorsNames string
	CompressionName         string
	CompressionOptions      string

	// User-collected properties
	UserCollectedProperties map[string]string
}

TableProperties contains metadata about an SST file.

func ParsePropertiesBlock 

func ParsePropertiesBlock(data []byte) (*TableProperties, error)

ParsePropertiesBlock parses a properties block into TableProperties.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.