Supertable integration tests validate the end-to-end correctness of the multi-file table abstraction, focusing on atomic commits, durability under crash scenarios, metadata consistency, and concurrent access. These tests ensure that the interaction between the SupertableWriter, ManifestSnapshot, and various StorageProvider implementations remains robust.
The suite verifies that the Supertable follows the "visibility barrier" design, where no data is visible to readers until the pointer file (_supertable/current) is atomically updated src/supertable/manifest/commit.rs9-11
Tests in pointer_atomic.rs cover the persistence primitives:
PointerFile text format (containing manifest_id, manifest_uri, and content_hash) correctly serializes and parses tests/supertable/commit/pointer_atomic.rs119-136CommitError::WriteContentionExhausted, triggering the writer's retry logic tests/supertable/commit/pointer_atomic.rs13-15The persistence.rs suite validates the physical layout on disk/object storage after a SupertableWriter::commit() tests/supertable/commit/persistence.rs4-13:
data/seg-<uuid>.sf.parquet.manifest-parts/part-<hash>.avro.zst.manifest/manifest-NNNNNN.json._supertable/current src/supertable/manifest/commit.rs68The id_uniqueness.rs suite validates that the 128-bit Snowflake-shaped _id values minted by IdGenerator are globally unique even under concurrent writers tests/supertable/commit/id_uniqueness.rs4-11
worker_id space tests/supertable/commit/id_uniqueness.rs58-87Supertable handles sharing the same LocalFsStorageProvider to verify that concurrent append() and commit() calls result in a globally unique ID corpus tests/supertable/commit/id_uniqueness.rs117-162This diagram illustrates the relationship between the SupertableWriter and the physical storage entities.
Sources: src/supertable/writer.rs11-23 src/supertable/manifest/commit.rs23-36 tests/supertable/commit/persistence.rs42-94
The integration suite includes property-based crash tests to verify the "all-or-nothing" property of commits. These tests use a CrashStorage wrapper that intercepts put_atomic or put_if_match calls and invokes std::process::abort() to simulate a hard process failure (e.g., kill -9) tests/supertable_commit_crash_localfs.rs9-14
Tests exercise crashes at specific stages of the commit pipeline tests/supertable_commit_crash_localfs.rs40-47:
| Test Function | Crash Point | Expected Recovery State |
|---|---|---|
crash_post_superfile | After Parquet data lands | Prior manifest_id (orphaned data file) |
crash_post_list | After JSON manifest lands | Prior manifest_id (orphaned metadata) |
crash_post_pointer | After pointer write succeeds | New manifest_id (durable commit) |
The CrashStorage tracks matches_seen for specific URI prefixes (e.g., manifest/) and triggers an abort only after the N-th successful storage operation tests/supertable_commit_crash_localfs.rs95-134 This ensures that the system handles "orphaned" files—objects that exist in storage but are not referenced by the current PointerFile tests/supertable_commit_crash_localfs.rs20-26
Sources: tests/supertable_commit_crash_localfs.rs7-36 tests/supertable_commit_crash_localfs.rs148-164
The compact_gc.rs test verifies the full lifecycle of data merging and orphan removal tests/supertable/compact_gc.rs6-12
CompactionSettings tests/supertable/compact_gc.rs119-126st.gc(Duration::ZERO) to delete stale objects (old manifests and superfiles) that are no longer reachable from the current pointer tests/supertable/compact_gc.rs140-155Sources: tests/supertable/compact_gc.rs53-168
The supertable_concurrent_processes.rs test simulates multiple independent writers and readers accessing the same storage root. It validates that:
Tests in open_refresh.rs validate the Consistency settings:
Supertable handle must re-check the storage pointer and refresh the manifest if a newer version exists tests/supertable/commit/open_refresh.rs106-110SupertableReader acquired before a new commit remains pinned to its original version, even if the parent Supertable handle refreshes to a newer version tests/supertable/commit/open_refresh.rs154-161Sources: tests/supertable/commit/open_refresh.rs47-76 tests/supertable/commit/open_refresh.rs106-161
The stats.rs suite ensures that Supertable::stats() provides an accurate view of the table's internal state tests/supertable/commit/stats.rs4-24:
manifest_id must increment exactly once per successful writer.commit() tests/supertable/commit/stats.rs10-11n_manifest_parts (total) vs n_manifest_parts_loaded (in-memory). Verifies that Supertable::open eager-fetches all parts, while the writer seeds the cache with newly written parts tests/supertable/commit/stats.rs12-18process_rss_bytes correctly reports physical memory usage by comparing it against independent readings from the memory-stats crate tests/supertable/commit/stats.rs137-152Sources: tests/supertable/commit/stats.rs47-58 tests/supertable/commit/stats.rs110-120
The integration suite extends to Python and Node.js bindings, ensuring the core Rust logic behaves correctly when invoked through high-level APIs over real cloud storage (Azure Blob).
Both test_azure_e2e.py and azure.e2e.test.mjs mirror the Rust integration logic:
createTable, append, bm25Search, and vectorSearch over the wire infino-node/__test__/azure.e2e.test.mjs57-72 infino-python/tests/test_azure_e2e.py86-100update and delete operations correctly land in the WAL and sidecars infino-node/__test__/azure.e2e.test.mjs85-100 infino-python/tests/test_azure_e2e.py110-123connect() to an existing URI infino-node/__test__/azure.e2e.test.mjs74-83 infino-python/tests/test_azure_e2e.py101-108Bridging Natural Language Space to Code Entity Space for Azure E2E tests.
Sources: infino-node/__test__/azure.e2e.test.mjs47-55 infino-python/tests/test_azure_e2e.py77-85
Refresh this wiki