Supertable Layer

Relevant source files

• AGENTS.md
• crate-docs.md
• docs/architecture/overview.md
• docs/architecture/supertable.md
• examples/demo.rs
• src/supertable/compaction/mod.rs
• src/supertable/mod.rs
• src/supertable/tombstones/cache.rs
• src/supertable/wal/persistence.rs
• src/supertable/wal/pipeline.rs
• tests/supertable/writer_mutations.rs

The supertable layer is the multi-file table abstraction in Infino. It provides a logical view over a set of immutable superfiles similar to how Apache Iceberg or Delta Lake manage Parquet files. It is responsible for manifest management, snapshot isolation, ACID-compliant writes to object storage, and distributed query execution across files src/supertable/mod.rs4-10

A supertable is defined by its SupertableOptions, which include the Arrow schema, FTS column configurations, and vector index parameters src/supertable/options.rs8-17

Core Architecture

The supertable maintains a two-tier hierarchical manifest to scale to millions of files while keeping memory overhead low. It uses Optimistic Concurrency Control (OCC) for commits, ensuring that multiple writers can attempt updates without locking, with a "pointer-file" serving as the atomic visibility barrier src/supertable/mod.rs4-10

System Entity Mapping

The following diagram maps high-level supertable concepts to their corresponding Rust entities.

Supertable Code Entity Map

Sources: src/supertable/mod.rs4-24 src/supertable/handle.rs61-141 src/supertable/options.rs41-62

---

Manifest & Snapshot Management

The manifest tracks the set of superfiles that constitute a table at a specific point in time. It uses a two-tier hierarchy: a ManifestList (JSON) pointing to multiple ManifestPart (Avro) files. This allows for lazy loading of metadata and efficient pruning during queries src/supertable/manifest/mod.rs48-51

• Snapshot Isolation: Readers pin a ManifestSnapshot src/supertable/handle.rs8-12 ensuring a consistent view even if a concurrent writer publishes new data.
• Atomic Commits: Commits are finalized by updating a single pointer file in object storage using put_if_match (OCC) src/supertable/wal/persistence.rs7-10

For details, see Manifest & Snapshot Management.

Write Path & Commit Pipeline

The SupertableWriter manages the lifecycle of data ingestion. It buffers incoming RecordBatch data, handles the "vector split" (separating vector blobs from scalar Parquet data), and coordinates the parallel building of superfiles across a thread pool src/supertable/mod.rs15-17 src/supertable/writer.rs56

• Auto-Flush: The writer automatically triggers a commit when the in-memory buffer exceeds a configurable threshold src/supertable/options.rs174-175
• Atomic Publishing: New superfiles are uploaded, followed by an OCC-based manifest update that makes the new data visible to readers src/supertable/wal/pipeline.rs35-37

For details, see Write Path & Commit Pipeline.

Write-Ahead Log (WAL) & Mutations

Infino uses a WAL to ensure durability for updates and deletes. Mutations go through a state machine (Intent → Appended → Complete) to ensure recovery safety across crashes src/supertable/wal/pipeline.rs19-43

• Tombstones: Deletions are managed via bitmapped tombstones stored as sidecar files. A per-process SidecarCache ensures that readers can filter out deleted rows with microsecond latency src/supertable/tombstones/cache.rs4-16
• Idempotency: The WAL pipeline is designed for recovery-safe replay, using preallocated superfile IDs and content-hashing src/supertable/wal/pipeline.rs44-55

For details, see Write-Ahead Log (WAL) & Mutations.

Query Engine & Fan-Out

The query engine integrates with Apache DataFusion to provide SQL support, while also offering native FTS (BM25) and vector (kNN) search src/supertable/mod.rs4-10

• Pruning: The engine uses manifest summaries (Bloom filters, stats) to skip files that cannot contain matching rows src/supertable/mod.rs20-22
• Table-Valued Functions: Search modalities are exposed as SQL functions (e.g., vector_search, bm25_search), allowing hybrid queries to be expressed in standard SQL docs/architecture/supertable.md94-110

For details, see Query Engine & Fan-Out.

Compaction & Garbage Collection

The compaction subsystem merges small or tombstone-heavy superfiles into larger, target-sized files (typically 1 GiB) to maintain query performance src/supertable/compaction/mod.rs8-9

• Selection: Compaction jobs are selected based on tombstone density (reclaiming space) and file size (reducing fan-out) src/supertable/compaction/mod.rs128-135
• GC: A garbage collection sweep identifies and removes "orphaned" superfiles—files no longer referenced by any active manifest snapshot—from object storage src/supertable/mod.rs29

For details, see Compaction & Garbage Collection.

Disk Cache & Reader Cache

Infino implements a tiered caching strategy to accelerate reads from high-latency object storage docs/architecture/overview.md114-118

• In-Memory Cache: Frequently accessed SuperfileReader instances are cached to avoid repeated metadata parsing src/supertable/reader_cache/mod.rs54
• Disk Cache: The DiskCacheStore manages a local NVMe/SSD cache of superfile segments, using memory-mapped files for RAM-speed access once data is "warm" src/supertable/reader_cache/mod.rs21-22

For details, see Disk Cache & Reader Cache.

---

Data Flow: Write to Read

The following diagram illustrates how data flows from an ingestion call through the manifest and into the query engine.

Supertable Data Pipeline

Sources: src/supertable/writer.rs41-43 src/supertable/options.rs8-17 src/supertable/handle.rs82-118 src/supertable/compaction/mod.rs33-44

Supertable Options & Configuration

The behavior of a supertable is governed by SupertableOptions. Key defaults include:

• Consistency: Staleness windows for BoundedStaleness allow for high-performance reads with a configurable window of staleness src/supertable/options.rs158-159
• Thread Pools: Separated reader_pool (all cores) and writer_pool (half cores) prevent ingestion from starving queries src/supertable/options.rs91-99
• ID Column: Automatically injects a 128-bit Snowflake-style _id column (Decimal128) if not provided src/supertable/options.rs134-154

Sources: src/supertable/mod.rs1-76 src/supertable/options.rs1-177 src/supertable/handle.rs1-150 src/supertable/compaction/mod.rs1-112