Skip to content

Proposal: tool pack.go — Add func PackProject

xushiwei edited this page Apr 12, 2026 · 3 revisions

1. Background

xgo pack was designed as a CLI command that walks a real filesystem directory tree, discovers index.* configuration files, merges them into a single index_pack.* artifact, and writes that artifact to disk.

XBuilder stores spx projects as .xbp files — ZIP archives containing the same assets/ directory tree that xgo pack understands. When XBuilder needs a packed configuration view of a project, it cannot invoke the CLI command directly:

  • There is no on-disk directory to point xgo pack at; the project lives inside a ZIP.
  • Writing a temporary index_pack.* file to disk and then discarding it is fragile, slow, and incompatible with environments that have a read-only or sandboxed filesystem (e.g., WASM, mobile, cloud runners).

The solution is to expose the core merge logic as a pure, in-process library functionPackProject — that operates entirely on an fs.ReadDirFS and returns the packed content as a byte slice, with no filesystem side effects.


2. Goals

  • Accept any fs.ReadDirFS implementation, including ZIP-backed filesystems (e.g., archive/zip wrapped via io/fs).
  • Perform the same discovery and merge semantics as xgo pack (§4.2–4.4 of the design spec), rooted at a caller-supplied dir/indexFile pair rather than auto-discovered.
  • Return the merged content as []byte so the caller decides what to do with it (cache, send over the wire, write to disk, etc.).
  • Leave all existing CLI behaviour unchanged.

3. Out of Scope

  • Writing index_pack.* to any filesystem — callers own that step.
  • Discovering multiple pack roots within the tree; PackProject targets exactly one root (the dir/indexFile pair supplied by the caller).
  • Supporting formats other than those already recognised by xgo pack (index.json, index.yml, index.yaml).

4. API Design

4.1 Signature

// PackProject merges all index.* configuration files found under dir into a
// single packed document and returns its serialised content.
//
// fs is the filesystem to read from (may be a ZIP-backed fs.ReadDirFS).
// dir is the root directory within fs that contains the root configuration file.
// indexFile is the filename of the root configuration file (e.g. "index.json").
//
// The returned []byte is the fully-merged configuration in the same format as
// indexFile (JSON, YAML, or YAML with .yml extension). The caller is responsible
// for writing or caching the result; PackProject never writes to any filesystem.
//
// Errors are returned for all fatal conditions listed in the design spec (§4.6):
// multiple index.* files in one directory, unparseable files, key collisions, etc.
func PackProject(
    fs        fs.ReadDirFS,
    dir       string,
    indexFile string,
) (indexPackContent []byte, err error)

4.2 Parameters

Parameter Type Description
fs fs.ReadDirFS The filesystem to read from. Any implementation is accepted, including os.DirFS, a ZIP-backed fs, or an in-memory fs for testing.
dir string Path within fs to the root directory that contains indexFile. Use "." for the filesystem root.
indexFile string Filename of the root configuration file ("index.json", "index.yml", or "index.yaml"). The format (JSON or YAML) is inferred from this extension and applied consistently to all child files and the output. The caller is responsible for supplying the correct filename; PackProject reads it as given.

4.3 Return Values

Return value Description
indexPackContent The serialised, merged configuration document. For JSON sources the output is indented JSON; for YAML sources the output is YAML. nil is returned on error.
err Non-nil on any fatal condition (see §4.4 below). Errors are descriptive and include the relevant file path and, where applicable, the conflicting key name.

4.4 Error Conditions

PackProject propagates all fatal conditions from the existing design spec (§4.6), adapted for library use:

Condition Error behaviour
indexFile not found under dir Returns a descriptive error; no panic.
Any configuration file is unparseable Fatal error with file path and underlying parse error.
A merge would overwrite an existing key Fatal error with the colliding key and the source file path.

A directory containing no index.* files beneath the root is silently skipped (the directory may contain only assets). If dir itself contains no indexFile, an error is returned immediately.


5. Design Details

5.1 Relationship to Existing Code

PackProject becomes the single implementation of the merge logic. The existing xgo pack CLI code in tool/pack.go is refactored to call PackProject rather than owning the merge logic itself:

  1. Extract the current discovery + merge core into PackProject (exported, fs.ReadDirFS-based).
  2. Refactor the CLI handler to construct an os.DirFS-backed fs.ReadDirFS, auto-discover the pack root(s), and delegate each root to PackProject.
  3. The CLI handler remains responsible for writing the returned []byte to index_pack.* on disk and for the -t test-mode comparison — concerns that belong at the CLI layer, not in the library function.

This makes PackProject the authoritative implementation; any future fix or format addition automatically benefits both the CLI and library callers.

5.2 Format Inference

The output format is determined entirely by the extension of indexFile:

indexFile value Output format MIME type
index.json Indented JSON application/json
index.yml YAML application/yaml
index.yaml YAML application/yaml

All child index.* files under dir must use the same extension as the root. If a child uses a different extension, PackProject returns a fatal error identifying the mismatched file. This preserves the existing invariant that a single directory tree uses exactly one configuration format.

5.3 Determinism

The merge walk visits subdirectories in lexicographic order (the same ordering guarantee provided by fs.ReadDir). This ensures that, given the same filesystem contents, PackProject always returns identical bytes — a requirement for reliable use in CI and content-addressable caching.


6. Usage Example

// Open a .xbp file (ZIP archive) and pack its configuration.
func packXBP(xbpPath string) ([]byte, error) {
    r, err := zip.OpenReader(xbpPath)
    if err != nil {
        return nil, fmt.Errorf("open xbp: %w", err)
    }
    defer r.Close()
 
    // r.File implements fs.ReadDirFS via archive/zip.
    return pack.PackProject(r, "assets", "index.json")
}

The caller receives the merged JSON bytes and can write them to an index_pack.json entry inside the ZIP, serve them over HTTP, or store them in a cache — all without touching the real filesystem.


7. Testing Strategy

Test case Method
Standard spx layout (sprites + sounds) fstest.MapFS with known content; assert output bytes match expected JSON.
YAML variant (index.yaml) Same structure with YAML files; assert YAML output.
Empty subtree (no children) Only dir/index.json exists; output equals root object.
Key collision Child introduces key already present in root; assert error.
Unparseable child file Assert error with correct file path.
ZIP-backed fs.ReadDirFS Use archive/zip in-memory writer; assert round-trip correctness.
Determinism Call PackProject twice on the same fstest.MapFS; assert bytes.Equal.

All tests use fstest.MapFS or an in-memory ZIP — no real filesystem I/O required.

Clone this wiki locally