fstools

package module
v0.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 12, 2025 License: MIT Imports: 12 Imported by: 0

README

FsTools - Tools for file systems.

Go Reference Go Report Card CI License

Utilities for traversing and manipulating abstract file systems that implement the absfs.Filer interface.

Implemented Features

  • Walk - Traverses an absfs.Filer similar to filepath.Walk from the standard library, using PreOrder (depth-first) traversal by default.

  • WalkWithOptions - Configurable traversal with options for sorting, custom comparison functions, and traversal order selection.

  • PreOrder - Depth-first traversal visiting parent directories before their children (standard traversal order).

  • PostOrder - Depth-first traversal visiting children before their parent directories, useful for operations like recursive deletion.

  • BreadthOrder - Level-by-level traversal (breadth-first search), visiting all entries at each depth before descending.

  • KeyOrder - Traversal that visits only files, skipping directories entirely.

  • Exists - Simple utility to check if a path exists in a filesystem.

  • Size - Calculate total size of a file or directory tree in bytes.

  • Count - Count the number of files and directories under a path.

  • Equal - Compare two files or directory trees for identical content.

  • Copy - Copy files and directory trees between filesystems with options for:

    • Parallel file copying (for thread-safe filesystems)
    • Metadata preservation (permissions, timestamps)
    • Filtering via callbacks (skip files/directories)
    • Error handling callbacks for partial copy recovery

Installation

go get github.com/absfs/fstools

Usage

import (
    "fmt"
    "os"
    "path/filepath"

    "github.com/absfs/fstools"
    "github.com/absfs/memfs"
)

func main() {
    fs, _ := memfs.NewFS()

    // Walk with default PreOrder traversal
    fstools.Walk(fs, "/", func(path string, info os.FileInfo, err error) error {
        fmt.Println(path)
        return nil
    })

    // Walk with custom options
    opts := &fstools.Options{
        Less: func(a, b os.FileInfo) bool {
            return a.Name() < b.Name()
        },
        Traversal: fstools.BreadthTraversal,
    }
    fstools.WalkWithOptions(fs, opts, "/", func(path string, info os.FileInfo, err error) error {
        fmt.Println(path)
        return nil
    })
}

Documentation

Index

Constants

This section is empty.

Variables

View Source
var TraverseLink = errors.New("traverse symlink, assuming target is a directory")

TraverseLink is a sentinel error that can be returned from FastWalkFunc to indicate that a symlink should be traversed as a directory. The caller is responsible for cycle detection.

Functions

func BreadthOrder

func BreadthOrder(fs absfs.Filer, options *Options, path string, fn filepath.WalkFunc) error

func Copy

func Copy(src, dst absfs.FileSystem, srcPath, dstPath string, opts *CopyOptions) error

Copy copies files and directories from src filesystem to dst filesystem. It recursively copies the entire directory tree starting at srcPath to dstPath. If opts.Parallel is true, uses concurrent copying for better performance on thread-safe filesystems.

func CopyToFS

func CopyToFS(src, dst absfs.FileSystem, path string) error

CopyToFS is a convenience wrapper that copies from src to dst filesystem using the same path for both source and destination.

func Equal

func Equal(fs1 absfs.FileSystem, path1 string, fs2 absfs.FileSystem, path2 string) (bool, error)

Equal compares two files or directory trees for identical content. For files, it compares the content byte-by-byte. For directories, it recursively compares all files and subdirectories.

func Exists

func Exists(fs absfs.Filer, path string) bool

Exists checks if a path exists in the filesystem.

func FastWalk

func FastWalk(filer absfs.Filer, root string, fn FastWalkFunc) error

FastWalk walks the file tree rooted at root concurrently, calling fn for each file. It is significantly faster than Walk because:

  • Runs multiple goroutines concurrently to traverse directories in parallel
  • Avoids unnecessary stat calls by using DirEntry type information

When the underlying filesystem is osfs, the platform-optimized ReadDir implementation is used automatically (e.g., getdents with d_type on Linux).

The fn must be safe for concurrent use as it may be called from multiple goroutines. If fn returns filepath.SkipDir, the directory is skipped. If fn returns TraverseLink for a symlink, it will be traversed as a directory.

func FastWalkWithConfig

func FastWalkWithConfig(filer absfs.Filer, root string, config *FastWalkConfig, fn FastWalkFunc) error

FastWalkWithConfig is like FastWalk but accepts a configuration.

func KeyOrder

func KeyOrder(fs absfs.Filer, options *Options, path string, fn filepath.WalkFunc) error

func PostOrder

func PostOrder(fs absfs.Filer, options *Options, path string, fn filepath.WalkFunc) error

func PreOrder

func PreOrder(fs absfs.Filer, options *Options, path string, fn filepath.WalkFunc) error

func Size

func Size(fs absfs.Filer, path string) (int64, error)

Size calculates the total size in bytes of a file or directory tree. For directories, it recursively sums the size of all contained files.

func Walk

func Walk(filer absfs.Filer, path string, fn filepath.WalkFunc) error

func WalkWithOptions

func WalkWithOptions(filer absfs.Filer, options *Options, path string, fn filepath.WalkFunc) error

Types

type CopyOptions

type CopyOptions struct {
	// Parallel enables concurrent copying for thread-safe filesystems
	Parallel bool

	// PreserveMode preserves file permissions
	PreserveMode bool

	// PreserveTimes preserves modification times
	PreserveTimes bool

	// Filter is called for each file/directory to determine if it should be copied
	// Return false to skip the item
	Filter func(path string, info os.FileInfo) bool

	// OnError is called when an error occurs during copy
	// Return nil to continue, or an error to stop
	OnError func(path string, err error) error
}

CopyOptions configures the behavior of copy operations.

type CountResult

type CountResult struct {
	Files int64
	Dirs  int64
}

CountResult contains the results of a Count operation.

func Count

func Count(fs absfs.Filer, path string) (CountResult, error)

Count counts the number of files and directories under a path. The path itself is included in the count.

func (CountResult) Total

func (c CountResult) Total() int64

Total returns the total number of files and directories.

type FastWalkConfig

type FastWalkConfig struct {
	// NumWorkers sets the number of concurrent workers.
	// If 0 or negative, defaults to max(4, runtime.NumCPU()).
	NumWorkers int

	// FollowSymlinks determines whether symlinks should be followed.
	// When true, symlinks to directories will be traversed.
	// The caller must handle potential cycles.
	FollowSymlinks bool
}

FastWalkConfig holds configuration options for FastWalk.

type FastWalkFunc

type FastWalkFunc func(path string, d fs.DirEntry, err error) error

FastWalkFunc is the callback function type for FastWalk. It receives fs.DirEntry instead of os.FileInfo for better performance (avoids stat calls when only type information is needed).

type Options

type Options struct {
	Sort      bool
	Less      SortFunc
	Fast      bool
	Traversal Traversal
}

type ParallelCopyOptions

type ParallelCopyOptions struct {
	// NumWorkers sets the number of concurrent copy workers.
	// If 0 or negative, defaults to max(4, runtime.NumCPU()).
	NumWorkers int

	// BufferSize sets the size of the copy buffer per worker.
	// If 0, defaults to 32KB.
	BufferSize int

	// PreserveMode preserves file permissions.
	PreserveMode bool

	// PreserveTimes preserves modification times.
	PreserveTimes bool

	// Filter is called for each file/directory to determine if it should be copied.
	// Return false to skip the item. Must be safe for concurrent use.
	Filter func(path string, info os.FileInfo) bool

	// OnError is called when an error occurs during copy.
	// Return nil to continue, or an error to stop. Must be safe for concurrent use.
	OnError func(path string, err error) error

	// OnProgress is called after each file is copied with the number of bytes copied.
	// Must be safe for concurrent use.
	OnProgress func(path string, bytesWritten int64)
}

ParallelCopyOptions configures the behavior of parallel copy operations.

type ParallelCopyStats

type ParallelCopyStats struct {
	FilesCopied int64
	DirsCopied  int64
	BytesCopied int64
	Errors      int64
}

ParallelCopyStats contains statistics from a parallel copy operation.

func ParallelCopy

func ParallelCopy(src, dst absfs.FileSystem, srcPath, dstPath string, opts *ParallelCopyOptions) (*ParallelCopyStats, error)

ParallelCopy copies files and directories from src to dst using concurrent workers. It is significantly faster than Copy for large directory trees with many files.

type SortFunc

type SortFunc func(i os.FileInfo, j os.FileInfo) bool

type Traversal

type Traversal int
const (
	BreadthTraversal Traversal = iota
	DepthTraversal
	PreOrderTraversal
	PostOrderTraversal
	KeyTraversal
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL