* feat(gateway): IPIP-523 format query param takes precedence over Accept header

this change simplifies precedence rules by making the ?format= URL query
parameter always take priority over the Accept HTTP header when both are
present.

in practice, this is largely compatible with existing browser use cases
since browsers send Accept headers with wildcards which were already
treated as non-specific. prioritizing ?format= also ensures deterministic
HTTP caching behavior, protecting against CDNs that comingle different
response types under the same cache key.

the only breaking change is for edge cases where a client sends both a
specific Accept header and a different ?format= value. previously Accept
would win, now ?format= wins. this scenario is rare and arguably
represents client misconfiguration. when detected, gateway returns HTTP
400 to signal the ambiguity.

specs: ipfs/specs#523
tests: ipfs/gateway-conformance#252

* docs(changelog): add IPIP-523 to unreleased

* fix(gateway): IPIP-523 ?format always wins over Accept header

remove HTTP 400 error for conflicting ?format and Accept values.
instead, ?format silently takes precedence, which is simpler and
less breaking for browser clients that send Accept headers automatically.

* ci: use gateway-conformance with IPIP-523 tests

temporary switch to ipfs/gateway-conformance#252

* chore(ci): switch to gateway-conformance@v0.9

* feat(unixfs): add SizeEstimationMode for HAMT threshold decisions

add configurable size estimation modes for determining when to switch
between BasicDirectory and HAMTDirectory:

- SizeEstimationLinks: legacy mode using len(name) + len(CID), default
- SizeEstimationBlock: full serialized dag-pb block size (accurate)
- SizeEstimationDisabled: link-count only via MaxLinks, ignores size

includes:
- HAMTSizeEstimation global for default mode
- WithSizeEstimationMode option for per-directory override
- helper functions for accurate protobuf size calculation

part of IPIP-499 UnixFS CID Profiles implementation.

* feat(unixfs): add UnixFSProfile for IPIP-499 CID determinism

introduces UnixFSProfile struct with predefined profiles:
- UnixFS_v0_2015: legacy CIDv0 settings (256 KiB chunks, 174 links/node)
- UnixFS_v1_2025: modern CIDv1 settings (1 MiB chunks, 1024 links/node)

profiles control file chunking, DAG width, and HAMT sharding parameters.
ApplyGlobals() sets all relevant global variables at once.

part of IPIP-499 implementation.

* feat(files): add DereferenceSymlinks option for IPIP-499

add SerialFileOptions and NewSerialFileWithOptions to control whether
symlinks are preserved as UnixFS symlink nodes (Data.Type=4) or
dereferenced and replaced with their target content during file
traversal.

* chore(unixfs): remove unnecessary uint64 conversions

Link.Size is already uint64, so the explicit conversions are redundant
and flagged by golangci-lint unconvert check.

* fix(unixfs): align HAMT sharding threshold with JS implementation

HAMT sharding threshold comparison was historically implemented as
`>` in JS and `>=` in Go:

- JS: https://github.com/ipfs/helia/blob/005c2a7/packages/unixfs/src/commands/utils/is-over-shard-threshold.ts#L31
- Go: https://github.com/ipfs/boxo/blob/319662c/ipld/unixfs/io/directory.go#L438

This inconsistency meant a directory exactly at the 256 KiB threshold
would stay basic in JS but convert to HAMT in Go, producing different
CIDs for the same input.

This commit changes Go to use `>` (matching JS), so a directory exactly
at the threshold now stays as a basic (flat) directory. This aligns
cross-implementation behavior for CID determinism per IPIP-499.

Also adds SizeEstimationMode to MkdirOpts so MFS directories respect
the configured estimation mode instead of always using the global default.

* chore: gofmt and changelog PR references

- fix trailing newline in directory_test.go
- add #1088 PR references to changelog entries

* fix: nil filter check and thread-safety docs

- files: fix nil filter check in serialFile.Size()
- unixfs/io: document thread-safety for global vars and ApplyGlobals
- changelog: move DefaultBlockSize to Changed section with breaking marker

* fix: correct go-ipfs-chunker URL in comment

* Add circular symlink test

* feat(unixfs): optimize SizeEstimationBlock and add mode/mtime tests

IPIP-499 block-bytes estimation improvements:

- add fast path optimization in needsToSwitchByBlockSize to skip
  expensive exact calculation when clearly above threshold (+256 margin)
- clarify documentation for linkSerializedSize, calculateBlockSize,
  cachedBlockSize, and SetStat methods with IPIP-499 context
- extract saveAndRestoreGlobals as package-level test helper

tests for mode/mtime block size overhead:
- verify exact protobuf overhead: mode (3 bytes), mtime seconds (8 bytes),
  nanoseconds (5 bytes), combined (16 bytes)
- verify cachedBlockSize accuracy after add/remove/replace operations
- verify linkSerializedSize matches actual link contribution
- verify HAMT threshold accounts for metadata overhead
- test fast path and near-boundary exact calculation behavior

* refactor(unixfs): unify size tracking and make SizeEstimationMode immutable

consolidate fragmented size tracking into a single method and field:
- merge `cachedBlockSize` into `estimatedSize` (single field for all modes)
- replace `addToEstimatedSize`, `removeFromEstimatedSize`, and
  `updateCachedBlockSize` with unified `updateEstimatedSize(name, oldLink, newLink)`
- remove `SetSizeEstimationMode` from Directory interface; mode is now
  set only at creation time via `WithSizeEstimationMode` option

this prevents mode changes after directory creation which could cause
size tracking inconsistencies, and simplifies the calling code from
two method calls per operation to one.

test coverage:
- TestHAMTToBasicDowngrade: new test for HAMT->Basic threshold boundaries
  covering both SizeEstimationLinks and SizeEstimationBlock modes
- TestEstimatedSizeAccuracy: verifies size tracking after add/remove/replace
- TestProfileHAMTThresholdBehavior: upgrade threshold boundaries
- TestDynamicDirectorySwitch: Basic<->HAMT conversions

* refactor(unixfs): use arithmetic for exact block size calculation

removes the need for protobuf serialization when checking HAMT threshold.
the block size is now computed arithmetically from protobuf field definitions:
- dataFieldSerializedSize(): UnixFS Data field (Type + optional mode/mtime)
- linkSerializedSize(): PBLink fields (Hash, Name, Tsize) + wrapper

this replaces the previous approach that serialized a temporary node copy
when near the threshold boundary. the arithmetic calculation is exact and
verified against actual serialization in TestDataFieldSerializedSizeMatchesActual.

calculateBlockSize() moved to test-only code in profile_test.go.

* docs(unixfs): clarify protobuf tag encoding comments

replace bit shift notation with plain arithmetic for readability

* docs(unixfs): clarify varintLen and negative timestamp encoding

add inline comments explaining:
- varintLen formula derivation (ceil(N/7) without branching)
- why negative int64 always uses 10 bytes in protobuf encoding

* fix(mfs): produce raw leaves for single-block files when RawLeaves=true

add maybeCollapseToRawLeaf() to DagModifier that collapses a ProtoNode
with a single RawNode child (and no metadata) to just the RawNode.
this ensures CID compatibility with `ipfs add` for single-block files
when RawLeaves is enabled.

the collapse happens in GetNode() rather than during Sync() to avoid
issues with intermediate operations like seeks and modifications.

* feat(mfs): add RootOption for chunker, maxLinks, and sizeEstimationMode

enables MFS to respect kubo's Import.* config options by accepting
tree-wide settings via functional options on NewRoot():
- WithChunker: configures chunker for files (was hardcoded to default)
- WithMaxLinks: configures directory link threshold for HAMT sharding
- WithSizeEstimationMode: configures size estimation for HAMT decisions

changes:
- mfs/root.go: add RootOption pattern and apply settings to root directory
- mfs/dir.go: inherit chunker from parent, propagate SizeEstimationMode
  to child directories when loaded from disk via cacheNode()
- mfs/file.go: use configured chunker instead of hardcoded default
- mfs/ops.go: add Chunker field to MkdirOpts
- ipld/unixfs/io/directory.go: add SetSizeEstimationMode() to interface,
  fix addLinkChild() to correctly handle entry replacements (was double-
  decrementing totalLinks)
- ipld/unixfs/mod/dagmodifier.go: add identitySafeDAGService wrapper to
  handle identity CID overflow gracefully during append operations,
  preserving identity for small files while switching to sha2-256 when
  data exceeds the 128-byte limit

* test(mfs): add tests for RootOption propagation

add tests to verify that WithChunker, WithMaxLinks, and
WithSizeEstimationMode correctly propagate settings to files
and directories created in MFS:

- TestRootOptionChunker: verifies custom chunker produces
  expected block count (512-byte chunks vs default 256KB)
- TestRootOptionMaxLinks: verifies custom MaxLinks triggers
  HAMT sharding (3 links vs default ~174)
- TestRootOptionSizeEstimationMode: verifies mode propagates
  to directories after reload from DAG
- TestChunkerInheritance: verifies chunker propagates through
  nested subdirectories (/a/b/c)

all tests use non-default values and include assertions to
confirm the custom settings are actually being used rather
than falling back to defaults.

* feat(mfs): add WithMaxHAMTFanout and WithHAMTShardingSize RootOptions

add per-directory HAMT sharding size threshold support:
- add hamtShardingSize field to BasicDirectory and HAMTDirectory
- add Get/SetHAMTShardingSize() methods to Directory interface
- add getEffectiveShardingSize() helper for per-directory or global fallback
- propagate HAMTShardingSize to child directories in cacheNode/setNodeData
- add HAMTShardingSize to MkdirOpts with parent inheritance

add RootOptions:
- WithMaxHAMTFanout(n) sets HAMT bucket width
- WithHAMTShardingSize(size) sets per-directory size threshold

add tests:
- TestRootOptionMaxHAMTFanout
- TestRootOptionHAMTShardingSize
- TestHAMTShardingSizeInheritance

* fix(unixfs/mod): update curNode after sparse expansion

expandSparse creates zero-padding when writing past end of file,
but wasn't updating dm.curNode to point to the new node.
subsequent writes would use the old unexpanded node, losing data.

* chore: modernize for loops and enhance package docs

- use Go 1.22+ range-over-int syntax in IPIP-499 tests
- expand ipld/unixfs/io package documentation with overview of
  directory types, HAMT sharding config, and IPIP-499 profiles

* fix(unixfs): preserve mode/mtime during HAMT conversions

directories with mode/mtime metadata (set via WithStat) would lose this
optional metadata when:
- converting from BasicDirectory to HAMTDirectory (during sharding)
- converting from HAMTDirectory to BasicDirectory (when shrinking)
- reloading a HAMT directory from disk

root cause: HAMT shards did not support mode/mtime in their UnixFS data,
and the conversion functions did not propagate these fields.

changes:
- add HAMTShardDataWithStat() to include mode/mtime in HAMT shard nodes
- add SetStat() to hamt.Shard to store metadata for serialization
- propagate mode/mtime and SizeEstimationMode during Basic<->HAMT
  conversions in DynamicDirectory
- extract mode/mtime from fsNode when loading HAMT via NewDirectoryFromNode

also adds tests for: negative mtime encoding, SizeEstimationDisabled with
maxLinks=0, unicode filenames in size estimation, concurrent HAMT
conversion, mode/mtime preservation after reload, and exact HAMT threshold
boundary behavior.

* fix(mfs): propagate Chunker to parent directories in Mkdir

when calling Mkdir with Mkparents=true and a custom Chunker, intermediate
directories would inherit the chunker from root instead of using the one
specified in MkdirOpts. now parentsOpts includes opts.Chunker so all
directories created in the path use the same chunker.

* fix(unixfs): fix CI failures in directory_test.go

- fix gofumpt formatting: comment alignment with unicode chars, octal literal
- rename TestConcurrentHAMTConversion to TestSequentialHAMTConversion and
  serialize operations to avoid race condition (Directory is not thread-safe
  for concurrent reads and writes)

* refactor(unixfs): simplify maxLinks check in addLinkChild

move the maxLinks check into the error handling block to eliminate
the intermediate `existed` boolean variable. the logic is equivalent
but more idiomatic: check maxLinks only when RemoveChild returns
ErrNotExist (new entry), skip it when removal succeeds (replacement).

adds test for replacement behavior at maxLinks capacity.

suggested by @gammazero in #1088 review:
#1088 (comment)

* fix(unixfs/mod): check Mode in maybeCollapseToRawLeaf

previously maybeCollapseToRawLeaf only checked ModTime when deciding
whether to keep a ProtoNode wrapper. files with Mode metadata (unix
permissions) but no ModTime would incorrectly collapse to RawNode,
losing the permission information.

now checks both ModTime and Mode before collapsing:
  if !fsn.ModTime().IsZero() || fsn.Mode() != 0 {

also refactored metadata preservation tests into a table-driven test
covering ModTime-only, Mode-only, and both metadata fields.

suggested-by: @gammazero
ref: #1088 (comment)

* docs(unixfs/mod): add doc.go with package documentation

moved package docs from dagmodifier.go to dedicated doc.go file.
expanded documentation to cover:

- MFS semantics for metadata handling
- clarification that Mode and ModTime are optional (most use cases
  do not set them)
- mtime update behavior on content modification (matches Unix fs)
- identity CID handling
- RawNode growth conversion
- raw leaf collapsing behavior

also added inline comments at mtime update sites explaining the
behavior and how to preserve specific mtime values if needed.

* fix(unixfs/io): preserve all options during Basic<->HAMT conversion

AddChild HAMT->Basic was missing WithSizeEstimationMode, and
RemoveChild HAMT->Basic was missing WithMaxHAMTFanout. This caused
settings to be lost when directories converted between types.

adds test verifying all settings (MaxLinks, MaxHAMTFanout,
SizeEstimationMode, HAMTShardingSize, CidBuilder) are preserved
in both conversion directions.

* fix(unixfs/io): validate MaxHAMTFanout and return error for invalid values

add upfront validation for WithMaxHAMTFanout option instead of silently
falling back to default. NewBasicDirectory and NewHAMTDirectory now
return ErrInvalidHAMTFanout when an invalid value is provided.

valid values must be a positive power of 2 AND multiple of 8
(e.g., 8, 16, 32, 64, 128, 256). use 0 to explicitly request default.

this is a cosmetic improvement: previously invalid values like 2, 4, or 7
would silently fall back to DefaultShardWidth with a warning log. now
the error is returned explicitly, making misconfiguration easier to detect.

---------

Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>

* feat(gateway): add AllowCodecConversion config option

Add AllowCodecConversion to gateway.Config to control codec conversion
behavior per IPIP-0524. When false (default), the gateway returns
406 Not Acceptable if the requested format doesn't match the block's
codec. When true, conversions between codecs are performed for backward
compatibility.

Codec conversion tests moved here from gateway-conformance since
conversions are now an optional implementation feature, not a spec
requirement. Gateway-conformance now tests for 406 responses.

Ref: ipfs/specs#524
Ref: ipfs/gateway-conformance#254

* ci: update gateway-conformance to e17586f4

* fix(gateway): 406 error now tells you how to get the data you need

when codec conversion is disabled (IPIP-0524) and you request
?format=dag-json for a dag-pb block, the 406 response now suggests
fetching the raw block with ?format=raw and converting client-side.

covers dag-pb directories, dag-pb files, and raw blocks requested
with ?format=dag-json or ?format=dag-cbor. plain ?format=json and
Accept: application/json continue to serve the default response,
so existing HTTP clients are not affected.

* chore: update changelog with PR link for IPIP-0524

* fix(gateway): show only native codec download link in HTML preview

only show the matching format download link (dag-json or dag-cbor) in
the HTML preview page when AllowCodecConversion is disabled, preventing
users from clicking links that would return 406.

when AllowCodecConversion is enabled, both links are shown as before.

* ci: switch gateway-conformance to v0.10

cover AllowCodecConversion with explicit dag-pb to dag-json
scenarios and verify response body matches IPLD Logical Format

* fix: raise block size limits from 1MiB to 2MiB

align chunker and importer block size limits with the bitswap spec
(https://specs.ipfs.tech/bitswap-protocol/#block-sizes) which mandates
2MiB as the maximum block size.

the previous 1MiB limit broke `dag import` of 1MiB-chunked non-raw-leaf
data where protobuf wrapping pushes blocks slightly over 1MiB.

max chunker size is set to 2MiB - 256 bytes to leave room for protobuf
framing overhead when chunks are wrapped in non-raw leaves. IPIP-499
profiles use lower chunk sizes (256KiB and 1MiB) and are not affected.

* test: add guard tests for block size limits and transport fit

- chunker: verify ChunkSizeLimit + ChunkOverheadBudget == BlockSizeLimit
- bitswap/message: verify BlockSizeLimit block (CIDv1+raw+SHA2-256)
  serializes within libp2p network.MessageSizeMax and round-trips
- use explicit byte values instead of bit-shift notation

* update go-multiaddr-dns to v0.5.0

* update opentelemetry

* remove opentelemetry zipkin exporter

- `tracing`: opentelemetry zipkin exporter (`go.opentelemetry.io/otel/exporters/zipkin`) is deprecated and has been removed. It is recommended to switch to OTLP. Configure your application to send traces using OTLP and enable [Zipkin’s OTLP ingestion support](https://github.com/openzipkin-contrib/zipkin-otel).

update dependencies

* Ensure http response body is closed
  - When appropriate, consume all body data before close to allow connection reuse.

- apply go fix modernizers from Go 1.26
- require go1.25 or later
- go1.26 is required to run sharness tests

…as been removed

Release v0.37.0