Documentation
¶
Overview ¶
Package ufs provides a unified virtual file system abstraction for Go. It allows applications to treat diverse storage backends — local disk, memory, archives, Google Cloud Storage, remote Git repositories, and more — through a single consistent interface.
Creating a file system ¶
Use New with a URI to open a file system:
fsys, err := ufs.New("memory://") // in-memory, volatile
fsys, err := ufs.New("null://") // /dev/null semantics
fsys, err := ufs.New("/path/to/dir") // local directory
fsys, err := ufs.New("file:///abs/dir") // same, explicit scheme
fsys, err := ufs.New("gs://bucket/prefix") // Google Cloud Storage
fsys, err := ufs.New("https://host/file.zip") // remote archive (downloaded to temp dir)
Every FS must be closed when no longer needed.
Nested mounts ¶
CreateURI builds a URI that layers multiple file systems at different paths. Pass the resulting URI to New:
uri, _ := ufs.CreateURI("file:///data", map[string]string{
"cache": "memory://",
})
fsys, _ := ufs.New(uri)
Path conventions ¶
All path arguments follow fs.ValidPath: forward-slash separated, no leading slash, no "." or ".." components. The root directory is always ".".
Index ¶
- func Copy(srcFS fs.FS, srcFilename string, destFS FS, destFilename string) error
- func CreateURI(name string, nested map[string]string) (string, error)
- func ForEachFileInfo(fsys fs.FS, dir string, f func(fs.FileInfo) error) error
- func ForEachFilename(fsys fs.FS, dir string, f func(string) error) error
- func List(fsys fs.FS, dir string) ([]string, error)
- func ListFiles(fsys fs.FS, dir string) ([]string, error)
- func Remove(fsys fs.FS, name string) error
- func RemoveAll(fsys fs.FS, name string) error
- func Rsync(srcFS fs.FS, destFS FS, dir string) error
- func Walk(fsys fs.FS, dir string, args WalkArgs, f func(string) error) error
- type FS
- type FSBuilder
- type File
- type FileInfo
- type ForEachFileInfoIter
- type ForEachFilenameIter
- type ListFilenames
- type NotifyHook
- type NotifyOp
- type ReadFS
- type ReadFile
- type Remover
- type WalkArgs
- type Watcher
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Copy ¶
Copy copies the single file at srcFilename in srcFS to destFilename in destFS. The parent directory of destFilename must already exist. The destination file is created (or truncated) via [FS.Create].
Example ¶
ExampleCopy shows copying a single file between two file systems.
ctx := context.Background()
src, _ := New(ctx, "memory://")
dst, _ := New(ctx, "memory://")
defer src.Close()
defer dst.Close()
f, _ := src.Create("hello.txt")
f.WriteString("hello")
f.Close()
if err := Copy(src, "hello.txt", dst, "copy.txt"); err != nil {
log.Fatal(err)
}
data, _ := dst.ReadFile("copy.txt")
fmt.Println(string(data))
Output: hello
func CreateURI ¶
CreateURI constructs a URI understood by New that layers additional file systems at specific mount paths inside the base file system. The nested map maps mount-point paths (e.g. "cache", "data/scratch") to the URI of the file system to mount there. Paths follow fs.ValidPath conventions.
Pass the returned URI directly to New:
ctx := context.Background()
uri, _ := ufs.CreateURI("file:///srv/data", map[string]string{
"tmp": "memory://",
})
fsys, _ := ufs.New(ctx, uri)
Returns an error if name or any nested URI cannot be parsed.
Example ¶
ExampleCreateURI shows building a URI for a file system with a nested mount.
ctx := context.Background()
// A memory FS with no nested mounts.
uri, err := CreateURI("memory://", nil)
if err != nil {
log.Fatal(err)
}
fmt.Println(uri)
// Open it — New accepts URIs produced by CreateURI.
fsys, err := New(ctx, uri)
if err != nil {
log.Fatal(err)
}
defer fsys.Close()
fmt.Println(fsys.String())
Output: memory: nestFS(memory://)
func ForEachFileInfo ¶
ForEachFileInfo calls f for each file (not directory) under dir, providing its fs.FileInfo. It is the typed companion to ForEachFilename and prefers a native ForEachFileInfoIter implementation when available, falling back to fs.WalkDir. The walk stops and returns the first non-nil error from f.
func ForEachFilename ¶
ForEachFilename calls f for each file path (not directory) under dir, streaming results without building an intermediate slice. If fsys implements ForEachFilenameIter, its native implementation is used directly; otherwise the paths are collected via fs.WalkDir and iterated. f receives paths relative to dir. The walk stops and returns the first non-nil error from f.
Example ¶
ExampleForEachFilename shows streaming file names without building a slice, which saves memory for large trees.
ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()
for _, name := range []string{"a.txt", "b.txt"} {
f, _ := fsys.Create(name)
f.Close()
}
ForEachFilename(fsys, ".", func(name string) error {
fmt.Println(name)
return nil
})
Output: a.txt b.txt
func List ¶
List returns all paths (both files and directories) under dir in lexical order. The root directory "." is never included in the result. For files-only, prefer ListFiles.
Example ¶
ExampleList shows listing all entries including directories.
ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()
fsys.MkdirAll("subdir", fs.ModePerm)
f, _ := fsys.Create("subdir/c.txt")
f.Close()
entries, _ := List(fsys, ".")
for _, p := range entries {
fmt.Println(p)
}
Output: subdir subdir/c.txt
func ListFiles ¶
ListFiles returns the paths of all files (excluding directories) under dir in lexical order. If fsys implements ListFilenames, its native implementation is used to avoid building intermediate fs.FileInfo values.
This method may take a long time since it may traverse a large file system and build a large slice of paths in memory.
Example ¶
ExampleListFiles shows listing only files (no directories) under a path.
ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()
fsys.MkdirAll("subdir", fs.ModePerm)
for _, name := range []string{"a.txt", "b.txt", "subdir/c.txt"} {
f, _ := fsys.Create(name)
f.Close()
}
files, _ := ListFiles(fsys, ".")
for _, p := range files {
fmt.Println(p)
}
Output: a.txt b.txt subdir/c.txt
func Remove ¶ added in v0.6.0
Remove removes the file or empty directory at name in fsys. If fsys implements Remover, its Remove method is used directly. Otherwise Remove returns fs.ErrPermission wrapped in an fs.PathError.
func RemoveAll ¶ added in v0.6.0
RemoveAll removes name and everything beneath it in fsys. If fsys implements Remover, its RemoveAll method is used directly. Otherwise RemoveAll returns fs.ErrPermission wrapped in an fs.PathError.
func Rsync ¶
Rsync copies all files under dir from srcFS into destFS, preserving the relative path structure. Parent directories in destFS are created with fs.ModePerm as needed. Existing files in destFS are overwritten. The copy is not atomic: if an error occurs mid-walk, destFS may be partially written.
dir must satisfy fs.ValidPath; use "." to copy the entire file system.
Example ¶
ExampleRsync shows recursively mirroring all files from one FS into another.
ctx := context.Background()
src, _ := New(ctx, "memory://")
dst, _ := New(ctx, "memory://")
defer src.Close()
defer dst.Close()
src.MkdirAll("subdir", fs.ModePerm)
for _, name := range []string{"a.txt", "subdir/b.txt"} {
f, _ := src.Create(name)
f.WriteString("content")
f.Close()
}
if err := Rsync(src, dst, "."); err != nil {
log.Fatal(err)
}
files, _ := ListFiles(dst, ".")
for _, p := range files {
fmt.Println(p)
}
Output: a.txt subdir/b.txt
func Walk ¶ added in v0.6.0
Walk walks dir in fsys, calling f for each file whose ancestor directories pass the filters in args. It differs from ForEachFilename in two ways: virtual archive-mount directories (e.g. "data.zip.d") are skipped by default (set [WalkArgs.IncludeMountedArchive] to descend into them), and directories whose base names match any [WalkArgs.ExcludeDirectory] glob are skipped entirely. The walk stops and returns the first non-nil error from f.
Types ¶
type FS ¶
type FS interface {
ReadFS
Remover
// Create opens a new writable file at name, replacing any existing file at
// that path. Parent directories are not created automatically; call
// MkdirAll first if needed.
Create(name string) (File, error)
// MkdirAll creates the directory at name and any missing parent directories,
// using perm for newly created nodes. It is a no-op if the directory already
// exists. Backends that do not have a real directory concept (e.g. GCS) treat
// this as a no-op.
MkdirAll(name string, perm fs.FileMode) error
// String returns a human-readable description of the file system, typically
// the URI or absolute path that was used to open it.
String() string
}
FS is a read-write file system. It extends ReadFS with file creation, directory creation, and deletion.
func New ¶
New opens a file system identified by name. The returned FS wraps the backend in a nestFS layer that automatically mounts archives found inside the tree (see below). Always call Close on the returned FS when done.
URI schemes ¶
- memory:// — volatile in-memory file system; all data is lost when the FS is closed or the process exits. Safe for concurrent use.
- null:// — /dev/null semantics: Create and MkdirAll always succeed, writes are accepted but discarded, reads return empty content, Stat reports everything as a directory. Useful in tests.
- angry:// — always returns fs.ErrInvalid; used to exercise error-handling paths in tests.
- file://path or a bare path — local directory, mounted read-write via os.OpenRoot (Go 1.24+). Access outside the mount root is rejected by the OS. On Windows, directory Stat always reports size 0 (unlike the raw os package which may report 4096).
- gs://bucket/prefix — Google Cloud Storage bucket, optionally scoped to a prefix. Credentials are resolved via ADC; unauthenticated access is tried as a fallback.
- https:// or http:// URL ending in a recognized archive extension — the archive is downloaded to a temporary directory, mounted read-only, and the temporary directory is removed when Close is called.
- A path ending in .git — the repository is shallow-cloned into a temporary directory (not available on AIX).
- A local path pointing to a recognized archive (.zip, .tar, .tar.gz, etc.) is mounted read-only through the archive's contents.
Nested mounts and archive auto-mounting ¶
The returned FS wraps all backends in a nestFS layer. When a directory entry named foo.zip (or any recognized archive extension) exists, the virtual path foo.zip.d is automatically exposed as a read-only mount of that archive's contents. No explicit configuration is required.
Use CreateURI to pre-configure additional mount points before calling New.
Example (Memory) ¶
ExampleNew_memory demonstrates a volatile in-memory file system. All data is lost when the FS is closed or the process exits.
ctx := context.Background()
fsys, err := New(ctx, "memory://")
if err != nil {
log.Fatal(err)
}
defer fsys.Close()
f, err := fsys.Create("hello.txt")
if err != nil {
log.Fatal(err)
}
f.WriteString("hello, world")
f.Close()
data, err := fsys.ReadFile("hello.txt")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))
Output: hello, world
Example (Null) ¶
ExampleNew_null demonstrates the null file system. It accepts all writes and Create calls without error, but data is immediately discarded. Reads always return empty content. Useful as a write sink in tests.
ctx := context.Background()
fsys, err := New(ctx, "null://")
if err != nil {
log.Fatal(err)
}
defer fsys.Close()
f, err := fsys.Create("discard.txt")
if err != nil {
log.Fatal(err)
}
n, writeErr := f.WriteString("this data is discarded")
f.Close()
fmt.Printf("wrote %d bytes, err=%v\n", n, writeErr)
// ReadFile always returns an empty byte slice, not an error.
data, _ := fsys.ReadFile("discard.txt")
fmt.Printf("read %d bytes\n", len(data))
Output: wrote 22 bytes, err=<nil> read 0 bytes
func NewEmbedFS ¶ added in v0.6.0
NewEmbedFS wraps a Go embed.FS as a read-only FS. name is used as the label returned by [FS.String]; it is typically the mount path or a description of the embedded content. Read operations delegate directly to the embed.FS; all write operations return fs.ErrPermission.
type FSBuilder ¶ added in v0.6.0
type FSBuilder struct {
// contains filtered or unexported fields
}
FSBuilder composes an FS from a root URI and a set of mounts that may be specified as URI strings or as pre-built FS instances (e.g. NewEmbedFS). Call NewFSBuilder to create one, chain FSBuilder.Mount / FSBuilder.MountFS to add mounts, then call FSBuilder.Build or FSBuilder.BuildURI.
func NewFSBuilder ¶ added in v0.6.0
NewFSBuilder creates a builder rooted at the given URI string. An empty string is treated as "null://" (a FS that discards all writes and returns empty content on reads). To use a pre-parsed *url.URL, pass u.String().
func (*FSBuilder) Build ¶ added in v0.6.0
Build constructs the FS, opening the root URI and applying all configured mounts. The caller must Close the returned FS when done.
func (*FSBuilder) BuildURI ¶ added in v0.6.0
BuildURI serialises the builder to a URI string accepted by New. It returns an error if any mount was added via FSBuilder.MountFS, because pre-built FS instances have no URI representation.
type File ¶
type File interface {
ReadFile
io.ReadWriteSeeker
io.ReaderAt
io.StringWriter
}
File is a read-write file handle. It extends ReadFile with write, seek, and random-read operations. All methods are safe to call at any time after the file is opened; unlike bare fs.File, no method returns a "not supported" error.
type FileInfo ¶
FileInfo provides file metadata. It currently mirrors fs.FileInfo and is defined as a separate interface to allow future extensions without breaking callers.
type ForEachFileInfoIter ¶
type ForEachFileInfoIter interface {
// ForEachFileInfo calls f for each file (not directory) under dir. If f
// returns a non-nil error the walk stops and that error is returned.
ForEachFileInfo(dir string, f func(fs.FileInfo) error) error
}
ForEachFileInfoIter is an optional interface for streaming fs.FileInfo values without building a full slice. ForEachFileInfo uses this when the file system implements it.
type ForEachFilenameIter ¶
type ForEachFilenameIter interface {
// ForEachFilename calls f for each file path (not directory) under dir. If f
// returns a non-nil error the walk stops and that error is returned.
ForEachFilename(dir string, f func(string) error) error
}
ForEachFilenameIter is an optional interface for streaming file names without building a full slice. ForEachFilename uses this when the file system implements it.
type ListFilenames ¶
type ListFilenames interface {
// ListFilenames returns the paths of all files (not directories) under dir,
// in unspecified order, with reduced allocations.
ListFilenames(string) ([]string, error)
}
ListFilenames is an optional interface that a file system may implement to return all file paths under a directory without building an intermediate fs.FileInfo slice, reducing memory usage for large trees. ListFiles will use this interface when available.
type NotifyHook ¶ added in v0.6.0
NotifyHook is invoked for each change observed by a Watcher. The path argument is root-relative, forward-slash separated, and satisfies fs.ValidPath — it is never an OS-native or absolute path.
type NotifyOp ¶ added in v0.6.0
type NotifyOp int
NotifyOp describes the kind of change observed on a path.
const ( // NotifyCreate indicates a file or directory was created. NotifyCreate NotifyOp = iota // NotifyWrite indicates a file was written to. NotifyWrite // NotifyRemove indicates a file or directory was removed. NotifyRemove // NotifyRename indicates a file or directory was renamed. NotifyRename // NotifyChmod indicates permissions or attributes changed. NotifyChmod )
type ReadFS ¶
ReadFS is a read-only file system. In addition to the standard fs.FS interface it requires io.Closer for lifecycle management and the four extended read interfaces from the standard library. Callers should always Close a ReadFS when they are done with it.
type Remover ¶ added in v0.6.0
type Remover interface {
// Remove deletes the file or empty directory at name. It returns an error
// wrapping [fs.ErrNotExist] if name does not exist, or an error if name is
// a non-empty directory. Removing the root (".") returns [fs.ErrPermission].
Remove(name string) error
// RemoveAll removes name and all contents beneath it. It is a no-op (returns
// nil) if name does not exist.
RemoveAll(name string) error
}
Remover is an optional interface that a file system may implement to support file and directory deletion. It is embedded in FS, so every writable backend must implement it.
type WalkArgs ¶ added in v0.6.0
type WalkArgs struct {
// IncludeMountedArchive controls whether virtual archive-mount directories
// (e.g. "data.zip.d") are descended into during the walk. When false
// (the default), such directories are skipped entirely.
IncludeMountedArchive bool
// ExcludeDirectory is a list of glob patterns matched against each
// directory's base name using [path.Match]. Directories whose names match
// any pattern are skipped along with all their contents. A nil or empty
// slice applies no filter.
ExcludeDirectory []string
}
WalkArgs configures traversal behavior for Walk.
type Watcher ¶ added in v0.6.0
type Watcher interface {
// Watch begins watching name (a directory) and all nested directories,
// invoking hook for each observed change. Watching stops when ctx is
// canceled or the returned [io.Closer] is closed, whichever comes first.
// Closing is idempotent and must terminate all background goroutines.
//
// The hook is called serially from a single background goroutine; it
// should not block for long.
Watch(ctx context.Context, name string, hook NotifyHook) (io.Closer, error)
}
Watcher is an optional interface implemented by file systems that can deliver recursive change notifications for a directory subtree.