zstash

package module
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 16, 2025 License: MIT Imports: 18 Imported by: 0

README

zstash

Go cache library providing cache save/restore and API functionality for the Buildkite Agent.

Cache Key Pattern Matching

zstash supports full glob pattern matching for cache keys using the zzglob library.

S3 Self-Managed Bucket

When using S3 as the storage backend (local_s3 store type), configure the bucket URL with query parameters to customize behavior.

URL Format

s3://bucket-name[/prefix][?options]

Options

Parameter Description Default Valid Range
region AWS region for the bucket us-east-1 Any valid AWS region
endpoint Custom S3 endpoint (for S3-compatible storage or local testing) AWS default Any valid URL
use_path_style Use path-style addressing instead of virtual-hosted-style false true or false
concurrency Number of parallel upload/download parts 5 0-100 (0 = default)
part_size_mb Size of each part in MB for multipart transfers 5 0, or 5-5120 (0 = default)

Examples

Basic S3 bucket:

s3://my-cache-bucket

S3 bucket with prefix and region:

s3://my-cache-bucket/buildkite/cache?region=us-west-2

S3-compatible storage (e.g., MinIO) with path-style access:

s3://my-bucket?endpoint=http://localhost:9000&use_path_style=true

High-performance configuration for large files:

s3://my-cache-bucket?concurrency=20&part_size_mb=100

All options combined:

s3://my-cache-bucket/prefix?region=eu-west-1&concurrency=10&part_size_mb=50

Notes

  • Part size: AWS S3 requires a minimum part size of 5 MB and maximum of 5 GB (5120 MB) for multipart uploads.
  • Concurrency: Higher concurrency can improve throughput for large files but uses more memory and network connections.
  • Endpoint: Use for S3-compatible storage like MinIO, LocalStack, or custom endpoints.

API Documentation

See the API documentation on pkg.go.dev for details.

License

MIT © Buildkite

SPDX-License-Identifier: MIT

Documentation

Overview

Package zstash provides a library for saving and restoring cache archives to/from cloud storage with the Buildkite cache API.

The main entry point is NewCache, which creates a Cache client for managing cache operations. The Cache client is safe for concurrent use by multiple goroutines.

Basic usage:

client := api.NewClient(ctx, version, endpoint, token)
cacheClient, err := zstash.NewCache(zstash.Config{
    Client:       client,
    BucketURL:    "s3://my-bucket",
    Branch:       "main",
    Pipeline:     "my-pipeline",
    Organization: "my-org",
    Caches: []cache.Cache{
        {ID: "node_modules", Key: "v1-{{ checksum \"package-lock.json\" }}", Paths: []string{"node_modules"}},
    },
})
if err != nil {
    log.Fatal(err)
}

// Save a cache
result, err := cacheClient.Save(ctx, "node_modules")

// Restore a cache
result, err := cacheClient.Restore(ctx, "node_modules")

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrCacheNotFound is returned when a requested cache ID doesn't exist
	// in the cache client's configuration.
	ErrCacheNotFound = errors.New("cache not found")

	// ErrInvalidConfiguration is returned when configuration validation fails
	// during cache client creation.
	ErrInvalidConfiguration = errors.New("invalid configuration")
)

Sentinel errors for common scenarios

Functions

This section is empty.

Types

type ArchiveMetrics

type ArchiveMetrics struct {
	// Size is the total size of the archive file in bytes (compressed).
	Size int64

	// WrittenBytes is the uncompressed size of all files in bytes.
	WrittenBytes int64

	// WrittenEntries is the number of files/directories in the archive.
	WrittenEntries int64

	// CompressionRatio is WrittenBytes / Size.
	// Higher values indicate better compression (e.g., 3.0 means 3:1 compression).
	CompressionRatio float64

	// Sha256Sum is the SHA-256 hash of the archive file.
	// Only populated for save operations, empty for restore.
	Sha256Sum string

	// Duration is how long the archive build or extraction took.
	Duration time.Duration

	// Paths are the filesystem paths that were archived or extracted.
	Paths []string
}

ArchiveMetrics contains metrics about archive build and extraction operations.

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache provides cache save and restore operations with the Buildkite cache API.

A Cache client is created once with configuration and can be used for multiple operations. The client is safe for concurrent use by multiple goroutines.

All cache operations respect context cancellation and will clean up resources when the context is cancelled.

func NewCache

func NewCache(cfg Config) (*Cache, error)

NewCache creates and validates a new cache client.

The function performs the following steps:

  1. Validates the configuration (Client must be provided)
  2. Sets defaults for Format (zip) and Platform (runtime.GOOS/runtime.GOARCH)
  3. Expands cache templates using cfg.Env if provided, otherwise uses OS environment
  4. Validates all expanded cache configurations
  5. Returns a ready-to-use cache client

The returned Cache client is safe for concurrent use by multiple goroutines.

Returns ErrInvalidConfiguration (wrapped) if:

  • Template expansion fails
  • Cache validation fails (invalid paths, missing required fields, etc.)

Example:

client := api.NewClient(ctx, version, endpoint, token)
cacheClient, err := zstash.NewCache(zstash.Config{
    Client:    client,
    BucketURL: "s3://my-bucket",
    Branch:    "main",
    Pipeline:  "my-pipeline",
    Caches: []cache.Cache{
        {ID: "deps", Key: "v1-{{ checksum \"go.mod\" }}", Paths: []string{"vendor"}},
    },
})
if err != nil {
    log.Fatal(err)
}

func (*Cache) GetCache

func (c *Cache) GetCache(id string) (cache.Cache, error)

GetCache returns a specific cache configuration by ID.

Returns ErrCacheNotFound if the cache ID is not found in the client's configuration.

func (*Cache) ListCaches

func (c *Cache) ListCaches() []cache.Cache

ListCaches returns all cache configurations managed by this cache client.

The returned caches have already been expanded (templates resolved) and validated during NewCache construction.

func (*Cache) Restore

func (c *Cache) Restore(ctx context.Context, cacheID string) (RestoreResult, error)

Restore restores a cache from storage by ID.

The function performs the following workflow:

  1. Validates the cache configuration
  2. Checks if the cache exists (tries exact key, then fallback keys)
  3. Downloads the cache archive from cloud storage
  4. Extracts files to their original paths
  5. Cleans up temporary files

If no matching cache is found (including fallback keys), the function returns early with CacheRestored=false. This is not an error condition.

The operation respects context cancellation and will stop immediately when ctx is cancelled, cleaning up any temporary resources (downloaded archives).

Progress callbacks (if configured) are invoked at each stage with the following stages: "validating", "checking_exists", "downloading", "extracting", "complete".

Returns RestoreResult with detailed metrics, or an error if the operation failed.

Use RestoreResult.CacheHit to check if the exact key matched, and RestoreResult.FallbackUsed to check if a fallback key was used.

Example:

result, err := cacheClient.Restore(ctx, "node_modules")
if err != nil {
    log.Fatalf("Cache restore failed: %v", err)
}
if !result.CacheRestored {
    log.Printf("Cache miss for key: %s", result.Key)
} else if result.FallbackUsed {
    log.Printf("Restored from fallback key: %s", result.Key)
} else {
    log.Printf("Cache hit: %s (%.2f MB)", result.Key, float64(result.Archive.Size)/(1024*1024))
}

func (*Cache) Save

func (c *Cache) Save(ctx context.Context, cacheID string) (SaveResult, error)

Save saves a cache to storage by ID.

The function performs the following workflow:

  1. Validates the cache configuration and paths exist
  2. Checks if the cache already exists (early return if yes)
  3. Builds an archive of the cache paths
  4. Creates a cache entry in the Buildkite API
  5. Uploads the archive to cloud storage
  6. Commits the cache entry

If the cache already exists, no upload is performed and the function returns early with CacheCreated=false and Transfer=nil.

The operation respects context cancellation and will stop immediately when ctx is cancelled, cleaning up any temporary resources.

Progress callbacks (if configured) are invoked at each stage with the following stages: "validating", "checking_exists", "fetching_registry", "building_archive", "creating_entry", "uploading", "committing", "complete".

Returns SaveResult with detailed metrics, or an error if the operation failed.

Example:

result, err := cacheClient.Save(ctx, "node_modules")
if err != nil {
    log.Fatalf("Cache save failed: %v", err)
}
if !result.CacheCreated {
    log.Printf("Cache already exists for key: %s", result.Key)
} else {
    log.Printf("Cache saved: %s (%.2f MB)", result.Key, float64(result.Archive.Size)/(1024*1024))
}

type Config

type Config struct {
	// Client is the Buildkite API client (required).
	// Create with api.NewClient(ctx, version, endpoint, token).
	Client api.CacheClient

	// BucketURL is the storage backend URL (required for most store types).
	// Examples: "s3://bucket-name", "gs://bucket-name", "file:///path/to/dir"
	BucketURL string

	// Format is the archive format. Defaults to "zip" if not specified.
	Format string

	// Branch is the git branch name, used for cache scoping in the Buildkite API.
	Branch string

	// Pipeline is the pipeline slug, used for cache scoping in the Buildkite API.
	Pipeline string

	// Organization is the organization slug, used for cache scoping in the Buildkite API.
	Organization string

	// Platform is the OS/arch string (e.g., "linux/amd64", "darwin/arm64").
	// If empty, defaults to runtime.GOOS/runtime.GOARCH.
	Platform string

	// Registry is the default cache registry to use for all cache operations.
	// If empty, defaults to "~" (the default registry).
	// Individual cache configurations can override this by setting their own Registry field.
	Registry string

	// Env is an optional environment variable map used for cache template expansion.
	// If nil, OS environment variables are used instead via os.Getenv.
	// Cache keys and paths can use templates like "{{ env \"NODE_VERSION\" }}".
	Env map[string]string

	// Caches is the list of cache configurations to manage.
	// Cache keys and paths will be expanded using template variables.
	Caches []cache.Cache

	// OnProgress is an optional callback for progress updates during operations.
	// If nil, no progress callbacks are made. The callback must be thread-safe
	// as it may be called from multiple goroutines.
	OnProgress ProgressCallback
}

Config holds all configuration for creating a Cache client.

The only required field is Client. All other fields have sensible defaults or are optional depending on your use case.

type ProgressCallback

type ProgressCallback func(cacheID string, stage string, message string, current int, total int)

ProgressCallback is called during long-running operations to report progress.

The callback is invoked at various stages during Save and Restore operations to provide visibility into the operation's progress. Implementations must be thread-safe as the callback may be called from multiple goroutines.

Parameters:

  • cacheID: The ID of the cache being operated on.
  • stage: The current operation stage. See below for possible values.
  • message: A human-readable description of the current action.
  • current: Current progress value (bytes transferred, files processed, etc.).
  • total: Total expected value (0 if unknown).

Save operation stages:

  • "validating": Validating cache configuration
  • "checking_exists": Checking if cache already exists
  • "fetching_registry": Looking up cache registry
  • "building_archive": Building archive (current=files processed, total=total files)
  • "creating_entry": Creating cache entry in API
  • "uploading": Uploading cache (current=bytes sent, total=total bytes)
  • "committing": Committing cache entry
  • "complete": Operation finished successfully

Restore operation stages:

  • "validating": Validating cache configuration
  • "checking_exists": Checking if cache exists
  • "downloading": Downloading cache (current=bytes received, total=total bytes)
  • "extracting": Extracting files (current=files extracted, total=total files)
  • "complete": Operation finished successfully

type RestoreResult

type RestoreResult struct {
	// CacheHit indicates whether the exact cache key was found.
	// false means either no cache found, or a fallback key was used.
	// When false, check CacheRestored to distinguish between cache miss
	// and fallback hit.
	CacheHit bool

	// CacheRestored indicates whether any cache was restored (including fallbacks).
	// false means complete cache miss (no matching key or fallback keys).
	CacheRestored bool

	// Key is the actual cache key that was restored.
	// May differ from the requested key if FallbackUsed is true.
	Key string

	// FallbackUsed indicates whether a fallback key was used.
	// true means the exact key wasn't found, but a fallback key matched.
	FallbackUsed bool

	// Archive contains information about the archive that was extracted,
	// including size, compression ratio, and file counts.
	Archive ArchiveMetrics

	// Transfer contains information about the download operation,
	// including bytes transferred and transfer speed.
	Transfer TransferMetrics

	// ExpiresAt indicates when this cache entry will expire.
	ExpiresAt time.Time

	// TotalDuration is the end-to-end duration of the restore operation,
	// from validation through extraction.
	TotalDuration time.Duration
}

RestoreResult contains detailed information about a cache restore operation.

Check CacheRestored to see if a cache was found. Use CacheHit and FallbackUsed to determine if the exact key matched.

type SaveResult

type SaveResult struct {
	// CacheCreated indicates whether a new cache entry was created.
	// false means the cache already existed and no upload occurred.
	// When false, Transfer will be nil since no upload was performed.
	CacheCreated bool

	// Key is the actual cache key that was used (after template expansion).
	Key string

	// UploadID is the unique identifier for this upload (if created).
	// Empty if CacheCreated is false.
	UploadID string

	// Archive contains information about the archive that was built,
	// including size, compression ratio, and file counts.
	Archive ArchiveMetrics

	// Transfer contains information about the upload (if performed).
	// Nil if CacheCreated is false (cache already existed).
	Transfer *TransferMetrics

	// TotalDuration is the end-to-end duration of the save operation,
	// from validation through commit (if created) or early exit (if exists).
	TotalDuration time.Duration
}

SaveResult contains detailed information about a cache save operation.

Check CacheCreated to see if a new cache was uploaded or if the cache already existed.

type TransferMetrics

type TransferMetrics struct {
	// BytesTransferred is the number of bytes uploaded or downloaded.
	BytesTransferred int64

	// TransferSpeed is the transfer rate in MB/s.
	TransferSpeed float64

	// Duration is how long the transfer took.
	Duration time.Duration

	// RequestID is the provider-specific request identifier for debugging.
	// Format depends on the storage backend (e.g., S3 request ID).
	RequestID string

	// PartCount is the number of parts used in multipart transfer (0 if not multipart).
	PartCount int

	// Concurrency is the number of concurrent uploads/downloads used.
	Concurrency int
}

TransferMetrics contains metrics about upload and download operations.

Directories

Path Synopsis
internal
key

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL