ufs

package module
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 17, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

README

ufs

Unified File System (UFS) is a Go library that allows apps to access multiple storage backends through a single fs.FS-based API. It provides a factory constructor (ufs.New) that dispatches to the appropriate implementation based on URI scheme, and a layering helper (ufs.CreateURI) for composing nested file systems.

Features

  • Unified API — All backends implement the same FS / ReadFS / File interfaces, so you can swap storage without changing application code.
  • Factory constructor — ufs.New(ctx, uri) opens any supported backend; no per-backend import needed at call site.
  • Layering — ufs.CreateURI() mounts additional virtual file systems at specific paths inside a base FS (caching, scratch space, etc.).
  • Cross-platform — Tested on Linux, macOS, and Windows; includes platform-specific build tags where needed.

File Systems

Storage URI Prefix Implementation Description
Null null:// nullfs.go Acts as /dev/null. Writes discarded, reads return empty.
Memory memory: memfs.go In-memory storage; lost when the process exits.
Local file:///path localfs.go Local disk, mounted at a root path via os.OpenRoot.
Google Cloud gs://bucket gcsfs.go Google Cloud Storage bucket as a read-only file system.
Git git://<url> gitfs.go Reads files from a git repository (clones on first open).
Archive archive:// archivefs.go Reads archives (zip, tar, 7z) as read-only FSs.
Nested via CreateURI nestfs.go Layers one or more virtual FSs at specific mount paths
inside a base FS.

Public API

// Open any supported file system from a URI.
func New(ctx context.Context, name string) (FS, error)

// Compose nested mounts: base FS with additional FSs at specific paths.
func CreateURI(baseName string, nested map[string]string) (string, error)
Interfaces
Interface Content
ReadFile Read-only file; wraps fs.File
File Read-write file; extends ReadFile with ReaderAt, Seek
ReadFS Read-only FS; adds Close, ListFilenames, ForEach iterators
FS Read-write FS; extends ReadFS with Create, MkdirAll
FileInfo Name, Size, Mode, ModTime, IsDir, Type, Sys

Commands

# Build
go build ./...

# Test (CGO disabled)
make test

# Test with race detector
CGO_ENABLED=1 go test -race ./...

# Run a single test
go test -run TestName ./...

# Lint (requires golangci-lint)
golangci-lint run

# Presubmit (lint + check)
make presubmit

# Deflake flaky tests (runs race tests 10 times)
make test-deflake

# Cross-compile all binaries
make build

Use ollama with Claude Code

ANTHROPIC_AUTH_TOKEN="ollama" ANTHROPIC_API_KEY="" ANTHROPIC_BASE_URL="http://mega:11434" claude --model qwen3.6:35b

License

Apache 2.0 — see LICENSE.

Documentation

Overview

Package ufs provides a unified virtual file system abstraction for Go. It allows applications to treat diverse storage backends — local disk, memory, archives, Google Cloud Storage, remote Git repositories, and more — through a single consistent interface.

Creating a file system

Use New with a URI to open a file system:

fsys, err := ufs.New("memory://")       // in-memory, volatile
fsys, err := ufs.New("null://")         // /dev/null semantics
fsys, err := ufs.New("/path/to/dir")    // local directory
fsys, err := ufs.New("file:///abs/dir") // same, explicit scheme
fsys, err := ufs.New("gs://bucket/prefix") // Google Cloud Storage
fsys, err := ufs.New("https://host/file.zip") // remote archive (downloaded to temp dir)

Every FS must be closed when no longer needed.

Nested mounts

CreateURI builds a URI that layers multiple file systems at different paths. Pass the resulting URI to New:

uri, _ := ufs.CreateURI("file:///data", map[string]string{
    "cache": "memory://",
})
fsys, _ := ufs.New(uri)

Path conventions

All path arguments follow fs.ValidPath: forward-slash separated, no leading slash, no "." or ".." components. The root directory is always ".".

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Copy

func Copy(srcFS fs.FS, srcFilename string, destFS FS, destFilename string) error

Copy copies the single file at srcFilename in srcFS to destFilename in destFS. The parent directory of destFilename must already exist. The destination file is created (or truncated) via [FS.Create].

Example

ExampleCopy shows copying a single file between two file systems.

ctx := context.Background()
src, _ := New(ctx, "memory://")
dst, _ := New(ctx, "memory://")
defer src.Close()
defer dst.Close()

f, _ := src.Create("hello.txt")
f.WriteString("hello")
f.Close()

if err := Copy(src, "hello.txt", dst, "copy.txt"); err != nil {
	log.Fatal(err)
}

data, _ := dst.ReadFile("copy.txt")
fmt.Println(string(data))
Output:
hello

func CreateURI

func CreateURI(name string, nested map[string]string) (string, error)

CreateURI constructs a URI understood by New that layers additional file systems at specific mount paths inside the base file system. The nested map maps mount-point paths (e.g. "cache", "data/scratch") to the URI of the file system to mount there. Paths follow fs.ValidPath conventions.

Pass the returned URI directly to New:

 ctx := context.Background()
	uri, _ := ufs.CreateURI("file:///srv/data", map[string]string{
	    "tmp": "memory://",
	})
	fsys, _ := ufs.New(ctx, uri)

Returns an error if name or any nested URI cannot be parsed.

Example

ExampleCreateURI shows building a URI for a file system with a nested mount.

ctx := context.Background()
// A memory FS with no nested mounts.
uri, err := CreateURI("memory://", nil)
if err != nil {
	log.Fatal(err)
}
fmt.Println(uri)

// Open it — New accepts URIs produced by CreateURI.
fsys, err := New(ctx, uri)
if err != nil {
	log.Fatal(err)
}
defer fsys.Close()
fmt.Println(fsys.String())
Output:
memory:
nestFS(memory://)

func ForEachFileInfo

func ForEachFileInfo(fsys fs.FS, dir string, f func(fs.FileInfo) error) error

ForEachFileInfo calls f for each file (not directory) under dir, providing its fs.FileInfo. It is the typed companion to ForEachFilename and prefers a native ForEachFileInfoIter implementation when available, falling back to fs.WalkDir. The walk stops and returns the first non-nil error from f.

func ForEachFilename

func ForEachFilename(fsys fs.FS, dir string, f func(string) error) error

ForEachFilename calls f for each file path (not directory) under dir, streaming results without building an intermediate slice. If fsys implements ForEachFilenameIter, its native implementation is used directly; otherwise the paths are collected via fs.WalkDir and iterated. f receives paths relative to dir. The walk stops and returns the first non-nil error from f.

Example

ExampleForEachFilename shows streaming file names without building a slice, which saves memory for large trees.

ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()

for _, name := range []string{"a.txt", "b.txt"} {
	f, _ := fsys.Create(name)
	f.Close()
}

ForEachFilename(fsys, ".", func(name string) error {
	fmt.Println(name)
	return nil
})
Output:
a.txt
b.txt

func List

func List(fsys fs.FS, dir string) ([]string, error)

List returns all paths (both files and directories) under dir in lexical order. The root directory "." is never included in the result. For files-only, prefer ListFiles.

Example

ExampleList shows listing all entries including directories.

ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()

fsys.MkdirAll("subdir", fs.ModePerm)
f, _ := fsys.Create("subdir/c.txt")
f.Close()

entries, _ := List(fsys, ".")
for _, p := range entries {
	fmt.Println(p)
}
Output:
subdir
subdir/c.txt

func ListFiles

func ListFiles(fsys fs.FS, dir string) ([]string, error)

ListFiles returns the paths of all files (excluding directories) under dir in lexical order. If fsys implements ListFilenames, its native implementation is used to avoid building intermediate fs.FileInfo values.

This method may take a long time since it may traverse a large file system and build a large slice of paths in memory.

Example

ExampleListFiles shows listing only files (no directories) under a path.

ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()

fsys.MkdirAll("subdir", fs.ModePerm)
for _, name := range []string{"a.txt", "b.txt", "subdir/c.txt"} {
	f, _ := fsys.Create(name)
	f.Close()
}

files, _ := ListFiles(fsys, ".")
for _, p := range files {
	fmt.Println(p)
}
Output:
a.txt
b.txt
subdir/c.txt

func Remove added in v0.6.0

func Remove(fsys fs.FS, name string) error

Remove removes the file or empty directory at name in fsys. If fsys implements Remover, its Remove method is used directly. Otherwise Remove returns fs.ErrPermission wrapped in an fs.PathError.

func RemoveAll added in v0.6.0

func RemoveAll(fsys fs.FS, name string) error

RemoveAll removes name and everything beneath it in fsys. If fsys implements Remover, its RemoveAll method is used directly. Otherwise RemoveAll returns fs.ErrPermission wrapped in an fs.PathError.

func Rsync

func Rsync(srcFS fs.FS, destFS FS, dir string) error

Rsync copies all files under dir from srcFS into destFS, preserving the relative path structure. Parent directories in destFS are created with fs.ModePerm as needed. Existing files in destFS are overwritten. The copy is not atomic: if an error occurs mid-walk, destFS may be partially written.

dir must satisfy fs.ValidPath; use "." to copy the entire file system.

Example

ExampleRsync shows recursively mirroring all files from one FS into another.

ctx := context.Background()
src, _ := New(ctx, "memory://")
dst, _ := New(ctx, "memory://")
defer src.Close()
defer dst.Close()

src.MkdirAll("subdir", fs.ModePerm)
for _, name := range []string{"a.txt", "subdir/b.txt"} {
	f, _ := src.Create(name)
	f.WriteString("content")
	f.Close()
}

if err := Rsync(src, dst, "."); err != nil {
	log.Fatal(err)
}

files, _ := ListFiles(dst, ".")
for _, p := range files {
	fmt.Println(p)
}
Output:
a.txt
subdir/b.txt

func Walk added in v0.6.0

func Walk(fsys fs.FS, dir string, args WalkArgs, f func(string) error) error

Walk walks dir in fsys, calling f for each file whose ancestor directories pass the filters in args. It differs from ForEachFilename in two ways: virtual archive-mount directories (e.g. "data.zip.d") are skipped by default (set [WalkArgs.IncludeMountedArchive] to descend into them), and directories whose base names match any [WalkArgs.ExcludeDirectory] glob are skipped entirely. The walk stops and returns the first non-nil error from f.

Types

type FS

type FS interface {
	ReadFS
	Remover

	// Create opens a new writable file at name, replacing any existing file at
	// that path. Parent directories are not created automatically; call
	// MkdirAll first if needed.
	Create(name string) (File, error)

	// MkdirAll creates the directory at name and any missing parent directories,
	// using perm for newly created nodes. It is a no-op if the directory already
	// exists. Backends that do not have a real directory concept (e.g. GCS) treat
	// this as a no-op.
	MkdirAll(name string, perm fs.FileMode) error

	// String returns a human-readable description of the file system, typically
	// the URI or absolute path that was used to open it.
	String() string
}

FS is a read-write file system. It extends ReadFS with file creation, directory creation, and deletion.

func New

func New(ctx context.Context, name string) (FS, error)

New opens a file system identified by name. The returned FS wraps the backend in a nestFS layer that automatically mounts archives found inside the tree (see below). Always call Close on the returned FS when done.

URI schemes

  • memory:// — volatile in-memory file system; all data is lost when the FS is closed or the process exits. Safe for concurrent use.
  • null:// — /dev/null semantics: Create and MkdirAll always succeed, writes are accepted but discarded, reads return empty content, Stat reports everything as a directory. Useful in tests.
  • angry:// — always returns fs.ErrInvalid; used to exercise error-handling paths in tests.
  • file://path or a bare path — local directory, mounted read-write via os.OpenRoot (Go 1.24+). Access outside the mount root is rejected by the OS. On Windows, directory Stat always reports size 0 (unlike the raw os package which may report 4096).
  • gs://bucket/prefix — Google Cloud Storage bucket, optionally scoped to a prefix. Credentials are resolved via ADC; unauthenticated access is tried as a fallback.
  • https:// or http:// URL ending in a recognized archive extension — the archive is downloaded to a temporary directory, mounted read-only, and the temporary directory is removed when Close is called.
  • A path ending in .git — the repository is shallow-cloned into a temporary directory (not available on AIX).
  • A local path pointing to a recognized archive (.zip, .tar, .tar.gz, etc.) is mounted read-only through the archive's contents.

Nested mounts and archive auto-mounting

The returned FS wraps all backends in a nestFS layer. When a directory entry named foo.zip (or any recognized archive extension) exists, the virtual path foo.zip.d is automatically exposed as a read-only mount of that archive's contents. No explicit configuration is required.

Use CreateURI to pre-configure additional mount points before calling New.

Example (Memory)

ExampleNew_memory demonstrates a volatile in-memory file system. All data is lost when the FS is closed or the process exits.

ctx := context.Background()
fsys, err := New(ctx, "memory://")
if err != nil {
	log.Fatal(err)
}
defer fsys.Close()

f, err := fsys.Create("hello.txt")
if err != nil {
	log.Fatal(err)
}
f.WriteString("hello, world")
f.Close()

data, err := fsys.ReadFile("hello.txt")
if err != nil {
	log.Fatal(err)
}
fmt.Println(string(data))
Output:
hello, world
Example (Null)

ExampleNew_null demonstrates the null file system. It accepts all writes and Create calls without error, but data is immediately discarded. Reads always return empty content. Useful as a write sink in tests.

ctx := context.Background()
fsys, err := New(ctx, "null://")
if err != nil {
	log.Fatal(err)
}
defer fsys.Close()

f, err := fsys.Create("discard.txt")
if err != nil {
	log.Fatal(err)
}
n, writeErr := f.WriteString("this data is discarded")
f.Close()
fmt.Printf("wrote %d bytes, err=%v\n", n, writeErr)

// ReadFile always returns an empty byte slice, not an error.
data, _ := fsys.ReadFile("discard.txt")
fmt.Printf("read %d bytes\n", len(data))
Output:
wrote 22 bytes, err=<nil>
read 0 bytes

func NewEmbedFS added in v0.6.0

func NewEmbedFS(name string, fsys embed.FS) FS

NewEmbedFS wraps a Go embed.FS as a read-only FS. name is used as the label returned by [FS.String]; it is typically the mount path or a description of the embedded content. Read operations delegate directly to the embed.FS; all write operations return fs.ErrPermission.

type FSBuilder added in v0.6.0

type FSBuilder struct {
	// contains filtered or unexported fields
}

FSBuilder composes an FS from a root URI and a set of mounts that may be specified as URI strings or as pre-built FS instances (e.g. NewEmbedFS). Call NewFSBuilder to create one, chain FSBuilder.Mount / FSBuilder.MountFS to add mounts, then call FSBuilder.Build or FSBuilder.BuildURI.

func NewFSBuilder added in v0.6.0

func NewFSBuilder(name string) *FSBuilder

NewFSBuilder creates a builder rooted at the given URI string. An empty string is treated as "null://" (a FS that discards all writes and returns empty content on reads). To use a pre-parsed *url.URL, pass u.String().

func (*FSBuilder) Build added in v0.6.0

func (b *FSBuilder) Build(ctx context.Context) (FS, error)

Build constructs the FS, opening the root URI and applying all configured mounts. The caller must Close the returned FS when done.

func (*FSBuilder) BuildURI added in v0.6.0

func (b *FSBuilder) BuildURI() (string, error)

BuildURI serialises the builder to a URI string accepted by New. It returns an error if any mount was added via FSBuilder.MountFS, because pre-built FS instances have no URI representation.

func (*FSBuilder) Mount added in v0.6.0

func (b *FSBuilder) Mount(path, uri string) *FSBuilder

Mount adds a URI-based mount at path. It returns the builder for chaining.

func (*FSBuilder) MountFS added in v0.6.0

func (b *FSBuilder) MountFS(path string, fsys FS) *FSBuilder

MountFS adds a pre-built FS as a mount at path. It returns the builder for chaining. Pre-built mounts cannot be serialized by FSBuilder.BuildURI.

type File

File is a read-write file handle. It extends ReadFile with write, seek, and random-read operations. All methods are safe to call at any time after the file is opened; unlike bare fs.File, no method returns a "not supported" error.

type FileInfo

type FileInfo interface {
	fs.FileInfo
}

FileInfo provides file metadata. It currently mirrors fs.FileInfo and is defined as a separate interface to allow future extensions without breaking callers.

type ForEachFileInfoIter

type ForEachFileInfoIter interface {
	// ForEachFileInfo calls f for each file (not directory) under dir. If f
	// returns a non-nil error the walk stops and that error is returned.
	ForEachFileInfo(dir string, f func(fs.FileInfo) error) error
}

ForEachFileInfoIter is an optional interface for streaming fs.FileInfo values without building a full slice. ForEachFileInfo uses this when the file system implements it.

type ForEachFilenameIter

type ForEachFilenameIter interface {
	// ForEachFilename calls f for each file path (not directory) under dir. If f
	// returns a non-nil error the walk stops and that error is returned.
	ForEachFilename(dir string, f func(string) error) error
}

ForEachFilenameIter is an optional interface for streaming file names without building a full slice. ForEachFilename uses this when the file system implements it.

type ListFilenames

type ListFilenames interface {
	// ListFilenames returns the paths of all files (not directories) under dir,
	// in unspecified order, with reduced allocations.
	ListFilenames(string) ([]string, error)
}

ListFilenames is an optional interface that a file system may implement to return all file paths under a directory without building an intermediate fs.FileInfo slice, reducing memory usage for large trees. ListFiles will use this interface when available.

type NotifyHook added in v0.6.0

type NotifyHook func(op NotifyOp, path string)

NotifyHook is invoked for each change observed by a Watcher. The path argument is root-relative, forward-slash separated, and satisfies fs.ValidPath — it is never an OS-native or absolute path.

type NotifyOp added in v0.6.0

type NotifyOp int

NotifyOp describes the kind of change observed on a path.

const (
	// NotifyCreate indicates a file or directory was created.
	NotifyCreate NotifyOp = iota
	// NotifyWrite indicates a file was written to.
	NotifyWrite
	// NotifyRemove indicates a file or directory was removed.
	NotifyRemove
	// NotifyRename indicates a file or directory was renamed.
	NotifyRename
	// NotifyChmod indicates permissions or attributes changed.
	NotifyChmod
)

type ReadFS

type ReadFS interface {
	fs.FS
	io.Closer
	fs.ReadDirFS
	fs.ReadFileFS
	fs.ReadLinkFS
	fs.StatFS
}

ReadFS is a read-only file system. In addition to the standard fs.FS interface it requires io.Closer for lifecycle management and the four extended read interfaces from the standard library. Callers should always Close a ReadFS when they are done with it.

func FromFS added in v0.6.0

func FromFS(fsys fs.FS) ReadFS

FromFS wraps a standard library fs.FS as a read-only ReadFS.

type ReadFile

type ReadFile interface {
	fs.File
}

ReadFile is a read-only file handle. It satisfies fs.File.

type Remover added in v0.6.0

type Remover interface {
	// Remove deletes the file or empty directory at name. It returns an error
	// wrapping [fs.ErrNotExist] if name does not exist, or an error if name is
	// a non-empty directory. Removing the root (".") returns [fs.ErrPermission].
	Remove(name string) error

	// RemoveAll removes name and all contents beneath it. It is a no-op (returns
	// nil) if name does not exist.
	RemoveAll(name string) error
}

Remover is an optional interface that a file system may implement to support file and directory deletion. It is embedded in FS, so every writable backend must implement it.

type WalkArgs added in v0.6.0

type WalkArgs struct {
	// IncludeMountedArchive controls whether virtual archive-mount directories
	// (e.g. "data.zip.d") are descended into during the walk. When false
	// (the default), such directories are skipped entirely.
	IncludeMountedArchive bool

	// ExcludeDirectory is a list of glob patterns matched against each
	// directory's base name using [path.Match]. Directories whose names match
	// any pattern are skipped along with all their contents. A nil or empty
	// slice applies no filter.
	ExcludeDirectory []string
}

WalkArgs configures traversal behavior for Walk.

type Watcher added in v0.6.0

type Watcher interface {
	// Watch begins watching name (a directory) and all nested directories,
	// invoking hook for each observed change. Watching stops when ctx is
	// canceled or the returned [io.Closer] is closed, whichever comes first.
	// Closing is idempotent and must terminate all background goroutines.
	//
	// The hook is called serially from a single background goroutine; it
	// should not block for long.
	Watch(ctx context.Context, name string, hook NotifyHook) (io.Closer, error)
}

Watcher is an optional interface implemented by file systems that can deliver recursive change notifications for a directory subtree.

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL