Feat/s3 gateway additions - #138
Merged
Merged
Conversation
…ateway
Three additions for the 0G S3 gateway, kept self-contained in new files
so the existing surface is untouched:
Downloader streaming (transfer/downloader_writer.go):
DownloadToWriter(ctx, root, w, withProof) — full file
DownloadRangeToWriter(ctx, root, offset, length, w) — byte range
Avoids the temp-file disk hop in (*Downloader).Download. Sequential
segment fetch via DownloadSegmentByTxSeq; per-segment slicing for
range reads. Encryption is rejected at the entry point — encrypted
files require buffering for header parsing, defeating streaming;
callers should use Download(filename) for those.
Indexer wrappers (indexer/client_writer.go):
Same two methods on (*indexer.Client), discovering nodes via
NewDownloaderFromIndexerNodes and forwarding.
Cheap fee preflight (indexer/client_estimate.go):
EstimateFee(ctx, w3, marketAddr, data, tags) returns the on-chain
protocol fee a submission would charge — pricePerSector * numSectors.
Caches pricePerSector for ~30s (configurable via SetEstimateCacheTTL)
so a hot-path preflight on a high-QPS gateway doesn't pin the chain
RPC. Cheaper than Uploader.EstimateFee, which requires
NewUploaderFromIndexerNodes (storage-node selection round-trip).
The 5 chunk-math unit tests for the writer methods carry over from the
v1.2.2 fork. No upstream callers changed.
pricePerSector is governance-gated (FixedPrice.setPricePerSector in 0g-storage-contracts/contracts/market/FixedPrice.sol) and in practice never changes — a 30s TTL re-reads it ~120×/hour for no reason. The actual hot-path consumer (0g-storage-s3-server) already wraps its own cache around Market.PricePerSector, so the SDK cache was redundant double-caching. Strip the four cache fields (estimateMu, estimateTTL, estimatePrice, estimateLoaded) and SetEstimateCacheTTL / cachedPricePerSector. Every EstimateFee now does one Market.PricePerSector eth_call — cheap and not on any hot path. Callers that need caching can layer their own. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 1 of 2 toward eliminating the tryFetchSegment dup in downloader_writer.go (#139). This change: - Adds transfer/segment_fetch.go with fetchSegmentFromNode(...) — the single source of truth for the shard-skip + fetch + proof-validation per-node download path. - Replaces tryFetchSegment in downloader_writer.go with a call to the new helper. Behavior tightened: the helper returns an error if the client has no ShardConfig (download_parallel.go was already this strict; the writer path was loose). Loose nil-shard behavior was a latent bug — we'd silently try to fetch from a node that may not have the segment. Step 2 will refactor download_parallel.go's downloadSegment to also use this helper, deleting its inline shard-check + downloadWithProof method. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 2 of 2 closing #139. download_parallel.go now calls the shared fetchSegmentFromNode helper for shard-skip + RPC + proof-validation, deleting its own segmentDownloader.downloadWithProof method. Net effect: one implementation of the per-node fetch path serves both file-based parallel download and io.Writer streaming download. Preserved invariants: - Per-attempt logging in download_parallel.go (with the same fields: node index, segment range, chunk range) - The (endChunk-startChunk)*ChunkSize length-check from the old downloadWithProof — moved into the helper so both callers get it - Padding trim for the last segment stays at the call site (only download_parallel.go needs it; downloader_writer.go has its own range-trim logic) Closes #139. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The helper was a 10-arg function — readable enough at the def site, hostile at the call site. Group the parameters into a request struct: - file-level: TxSeq, Root, FileSize - segment-level: SegIdx, GlobalSegIdx, StartChunk, EndChunk - mode: WithProof No behavior change; both call sites now construct the struct literally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r_writer (endByte + chunkSize - 1) / chunkSize is exactly what core.NumSplits already does. Call the SDK helper instead of rolling our own. Closes #140. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two pre-existing inline reinventions of core.SegmentRange now call it:
- download_parallel.go:36-37 was computing
startSegmentIndex := info.Tx.StartEntryIndex / DefaultSegmentMaxChunks
endSegmentIndex := (info.Tx.StartEntryIndex + NumSplits(...) - 1) / DefaultSegmentMaxChunks
which is exactly what SegmentRange returns. Replaced with a
single call.
- downloader_writer.go:70 was computing globalStartSeg = startEntryIndex
/ segChunks inline. Now reads SegmentRange's startSegmentIndex
(endSegmentIndex discarded — the byte-range case computes its own
file-relative endSeg from the requested offset/length).
The uploader (uploader.go:853, uploader_segments.go:104) already uses
SegmentRange; downloaders were stragglers.
Closes #141.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of the e2e "indexer port not available within 60s" flake.
cmd/indexer.go:59 → InitDefaultNodeManager → AddTrustedNodes calls
defaultIPLocationManager.Query(parseIP(url)) for every trusted node
at startup. For loopback URLs ("http://127.0.0.1:11260") this fires
http.Get("http://ipinfo.io/127.0.0.1/json") using the default
http.Client — which has no timeout. On a machine where ipinfo.io
is slow/blocked (WARP/VPN, restricted DNS, captive portal, etc.)
the indexer can't bind its RPC port until that GET returns.
Fix:
- isNonRoutableIP — short-circuit Query for loopback / RFC1918 /
link-local / unspecified addresses. ipinfo has nothing useful
to say about them anyway.
- ipLocationHTTPClient with 5s Timeout — for real public IPs that
DO go through, bound the call so a sick upstream can't wedge
indexer startup. The Query result is already non-fatal at the
call site (node_manager.go:158 logs Warn and continues), so
failing fast is strictly better.
- unit test for isNonRoutableIP covering loopback, RFC1918, link-
local, public, and unparseable cases.
Closes #143.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SDK changes needed by the 0G S3 gateway: stream-to-
io.Writerdownloads, range downloads, and cheap fee estimation. Plus a string of refactor / bug fixes the gateway work surfaced.Commits
fetchSegmentFromNodeshared helperdownload_parallel.gothrough the shared helpersegmentFetchRequeststructcore.NumSplitsfor ceil-divide indownloader_writer.gocore.SegmentRangeCloses
Test plan
go build ./...cleango test ./transfer/..../indexer/...green🤖 Generated with Claude Code