Write Path & Commit Pipeline

Relevant source files

• src/supertable/manifest/commit.rs
• src/supertable/utils/idgen.rs
• src/supertable/utils/vector_split.rs
• src/supertable/writer.rs
• tests/supertable/commit/id_uniqueness.rs
• tests/supertable/commit/open_refresh.rs
• tests/supertable/commit/persistence.rs
• tests/supertable/commit/pointer_atomic.rs
• tests/supertable/commit/stats.rs
• tests/supertable/compact_gc.rs
• tests/supertable/disk_cache/basic.rs
• tests/supertable/disk_cache/hybrid.rs
• tests/supertable_commit_crash_localfs.rs

The write path in Infino is a single-writer, multi-phase pipeline managed by the SupertableWriter. It coordinates the transition of data from in-memory buffers to immutable, content-addressed superfiles and finally publishes them via an atomic visibility barrier in the manifest.

SupertableWriter Lifecycle

The SupertableWriter is a long-lived handle acquired via Supertable::writer(). Infino enforces a single-writer model per supertable using an internal writer_outstanding flag src/supertable/writer.rs14-17

The lifecycle follows an append*N -> commit pattern:

1. Buffering: Data is appended to an in-memory buffer as BufferedBatch payloads src/supertable/writer.rs132-137
2. Sharding: Upon commit(), the buffer is partitioned into row-balanced shards src/supertable/writer.rs31-33
3. Building: Each shard is processed in parallel by a SuperfileBuilder on a Rayon worker pool src/supertable/writer.rs11-13
4. Publishing: The resulting superfiles are committed to the manifest list, and the pointer file is updated atomically src/supertable/manifest/commit.rs23-37

Write Path Flow: Buffer to Manifest

This diagram illustrates the association between high-level write stages and the code entities responsible for them.

Sources: src/supertable/writer.rs6-37 src/supertable/manifest/commit.rs23-43 tests/supertable/commit/persistence.rs9-13

Buffering and Sharding

When append() is called, Infino performs schema and null validation via split_vectors src/supertable/writer.rs27-30 It clones the underlying Float32Array payloads (using Arc to avoid copying bytes) into a BufferedBatch to keep buffer ownership clean across append calls src/supertable/writer.rs40-48

The writer tracks an estimated byte cost (buffer_bytes). If this exceeds a configured threshold, an internal commit() is triggered automatically src/supertable/writer.rs138-140 During commit(), the fanout_shards utility partitions the data across the writer pool's rayon workers src/supertable/writer.rs19-21

ID Generation

During the write path, Infino auto-injects a 128-bit Snowflake-style ID into the _id column using IdGenerator src/supertable/utils/idgen.rs4-21

• Layout: 64-bit millisecond timestamp, 40-bit machine/worker ID, and 24-bit sequence counter src/supertable/utils/idgen.rs10-14
• Uniqueness: The 40-bit random worker ID minimizes birthday collisions across concurrent handles src/supertable/utils/idgen.rs67-73
• Ordering: The high bit stays 0 for current-era timestamps, ensuring signed i128 comparison matches time order for efficient manifest pruning src/supertable/utils/idgen.rs17-21

Sources: src/supertable/writer.rs131-140 src/supertable/utils/vector_split.rs48-58 src/supertable/utils/idgen.rs4-73

Superfile Building and Upload

Each shard is consumed by a SuperfileBuilder, which produces a single Parquet-compatible superfile containing columnar data, FTS blobs, and vector regions src/supertable/writer.rs20-22

Upload Strategies

Infino selects an upload strategy based on the file size:

• Atomic PUT: For manifest components and small superfiles, the writer uses put_atomic (or put_if_match) to write the entire file in one RTT src/supertable/manifest/commit.rs12-14
• Multipart Upload: Larger superfiles are streamed via put_multipart src/supertable/writer.rs67 This is verified via tests that force every upload through the multipart path by setting a 1-byte threshold tests/supertable/commit/persistence.rs157-164

Sources: src/supertable/writer.rs20-22 src/supertable/manifest/commit.rs12-19 tests/supertable/commit/persistence.rs157-164

The Commit Pipeline (OCC)

Infino uses Optimistic Concurrency Control (OCC) to manage manifest updates. The commit process is orchestrated by commit_manifest src/supertable/manifest/commit.rs23-37

Parallel Execution and Visibility Barrier

The pipeline issues multiple I/O requests in parallel to minimize RTTs:

1. Encode: Manifest parts (Avro+zstd) and the Manifest List (JSON) are encoded src/supertable/manifest/commit.rs24-25
2. Parallel PUT: New manifest parts and the new manifest list are uploaded to storage simultaneously. Parts are content-addressed by their Blake3 hash, allowing for deduplication and reuse src/supertable/manifest/commit.rs27-30
3. The Barrier: Once all parts and the list are durable, the writer attempts to update the Pointer File (_supertable/current) src/supertable/manifest/commit.rs32-36

Jittered Backoff

If the pointer update fails due to a version mismatch (another writer committed first), the storage layer returns a PreconditionFailed error. The commit path captures this as CommitError::WriteContentionExhausted, triggering a jittered backoff before retrying the commit against the new manifest head tests/supertable/commit/pointer_atomic.rs13-15

Commit Pipeline Sequence

The following diagram maps the sequence of function calls and storage interactions during a commit.

Sources: src/supertable/manifest/commit.rs23-43 src/supertable/writer.rs19-37 tests/supertable/commit/pointer_atomic.rs13-25

Mutation Path (Updates & Deletes)

Updates and deletes follow the same "buffer then commit" pattern but involve the Write-Ahead Log (WAL) for durability src/supertable/writer.rs141-150

1. Resolution: update() or delete() resolves the predicate against the current snapshot to find matching target_ids src/supertable/writer.rs157-172
2. WAL Pipeline: The commit() call drives these entries through the WAL pipeline stages (Append Phase, Tombstone Phase) src/supertable/writer.rs86-93
3. Tombstone Phase: Tombstone bits are written to sidecar files to mark rows as deleted src/supertable/writer.rs80-82

Sources: src/supertable/writer.rs141-172 src/supertable/wal/state_doc.rs89-92

Pointer-File Visibility Barrier

The pointer file located at _supertable/current is the single source of truth for the supertable's state src/supertable/manifest/commit.rs68 It contains:

• manifest_id: Monotonically increasing version number.
• manifest_uri: Path to the JSON manifest list.
• content_hash: Blake3 hash of the manifest list src/supertable/manifest/commit.rs97-101

Because the pointer update is the final step and is performed using an atomic rename or conditional PUT, any crash before this point results in an "orphan" superfile or manifest part that is ignored by readers and eventually cleaned up by Garbage Collection tests/supertable_commit_crash_localfs.rs19-30

Sources: src/supertable/manifest/commit.rs68-101 tests/supertable_commit_crash_localfs.rs4-36