| Benchmark | go-toml v1 | BurntSushi/toml |
|---|---|---|
| Marshal/HugoFrontMatter-2 | 1.9x | 2.2x |
| Marshal/ReferenceFile/map-2 | 1.7x | 2.1x |
| Marshal/ReferenceFile/struct-2 | 2.2x | 3.0x |
| Unmarshal/HugoFrontMatter-2 | 2.9x | 2.7x |
| Unmarshal/ReferenceFile/map-2 | 2.6x | 2.7x |
| Unmarshal/ReferenceFile/struct-2 | 4.6x | 5.1x |
See more
+The table above has the results of the most common use-cases. The table below +contains the results of all benchmarks, including unrealistic ones. It is +provided for completeness.
+ +| Benchmark | go-toml v1 | BurntSushi/toml |
|---|---|---|
| Marshal/SimpleDocument/map-2 | 1.8x | 2.7x |
| Marshal/SimpleDocument/struct-2 | 2.7x | 3.8x |
| Unmarshal/SimpleDocument/map-2 | 3.8x | 3.0x |
| Unmarshal/SimpleDocument/struct-2 | 5.6x | 4.1x |
| UnmarshalDataset/example-2 | 3.0x | 3.2x |
| UnmarshalDataset/code-2 | 2.3x | 2.9x |
| UnmarshalDataset/twitter-2 | 2.6x | 2.7x |
| UnmarshalDataset/citm_catalog-2 | 2.2x | 2.3x |
| UnmarshalDataset/canada-2 | 1.8x | 1.5x |
| UnmarshalDataset/config-2 | 4.1x | 2.9x |
| geomean | 2.7x | 2.8x |
This table can be generated with ./ci.sh benchmark -a -html.
| Benchmark | go-toml v1 | BurntSushi/toml |
|---|---|---|
| {} | {} | {} |
See more
") +print("""The table above has the results of the most common use-cases. The table below +contains the results of all benchmarks, including unrealistic ones. It is +provided for completeness.
""") +printtable(below) +print('This table can be generated with ./ci.sh benchmark -a -html.
stdlib |
+conc |
+
|---|---|
| + +```go +type caughtPanicError struct { + val any + stack []byte +} + +func (e *caughtPanicError) Error() string { + return fmt.Sprintf( + "panic: %q\n%s", + e.val, + string(e.stack) + ) +} + +func main() { + done := make(chan error) + go func() { + defer func() { + if v := recover(); v != nil { + done <- &caughtPanicError{ + val: v, + stack: debug.Stack() + } + } else { + done <- nil + } + }() + doSomethingThatMightPanic() + }() + err := <-done + if err != nil { + panic(err) + } +} +``` + | ++ +```go +func main() { + var wg conc.WaitGroup + wg.Go(doSomethingThatMightPanic) + // panics with a nice stacktrace + wg.Wait() +} +``` + | +
stdlib |
+conc |
+
|---|---|
| + +```go +func main() { + var wg sync.WaitGroup + for i := 0; i < 10; i++ { + wg.Add(1) + go func() { + defer wg.Done() + // crashes on panic! + doSomething() + }() + } + wg.Wait() +} +``` + | ++ +```go +func main() { + var wg conc.WaitGroup + for i := 0; i < 10; i++ { + wg.Go(doSomething) + } + wg.Wait() +} +``` + | +
stdlib |
+conc |
+
|---|---|
| + +```go +func process(stream chan int) { + var wg sync.WaitGroup + for i := 0; i < 10; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for elem := range stream { + handle(elem) + } + }() + } + wg.Wait() +} +``` + | ++ +```go +func process(stream chan int) { + p := pool.New().WithMaxGoroutines(10) + for elem := range stream { + elem := elem + p.Go(func() { + handle(elem) + }) + } + p.Wait() +} +``` + | +
stdlib |
+conc |
+
|---|---|
| + +```go +func process(values []int) { + feeder := make(chan int, 8) + + var wg sync.WaitGroup + for i := 0; i < 10; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for elem := range feeder { + handle(elem) + } + }() + } + + for _, value := range values { + feeder <- value + } + close(feeder) + wg.Wait() +} +``` + | ++ +```go +func process(values []int) { + iter.ForEach(values, handle) +} +``` + | +
stdlib |
+conc |
+
|---|---|
| + +```go +func concMap( + input []int, + f func(int) int, +) []int { + res := make([]int, len(input)) + var idx atomic.Int64 + + var wg sync.WaitGroup + for i := 0; i < 10; i++ { + wg.Add(1) + go func() { + defer wg.Done() + + for { + i := int(idx.Add(1) - 1) + if i >= len(input) { + return + } + + res[i] = f(input[i]) + } + }() + } + wg.Wait() + return res +} +``` + | ++ +```go +func concMap( + input []int, + f func(*int) int, +) []int { + return iter.Map(input, f) +} +``` + | +
stdlib |
+conc |
+
|---|---|
| + +```go +func mapStream( + in chan int, + out chan int, + f func(int) int, +) { + tasks := make(chan func()) + taskResults := make(chan chan int) + + // Worker goroutines + var workerWg sync.WaitGroup + for i := 0; i < 10; i++ { + workerWg.Add(1) + go func() { + defer workerWg.Done() + for task := range tasks { + task() + } + }() + } + + // Ordered reader goroutines + var readerWg sync.WaitGroup + readerWg.Add(1) + go func() { + defer readerWg.Done() + for result := range taskResults { + item := <-result + out <- item + } + }() + + // Feed the workers with tasks + for elem := range in { + resultCh := make(chan int, 1) + taskResults <- resultCh + tasks <- func() { + resultCh <- f(elem) + } + } + + // We've exhausted input. + // Wait for everything to finish + close(tasks) + workerWg.Wait() + close(taskResults) + readerWg.Wait() +} +``` + | ++ +```go +func mapStream( + in chan int, + out chan int, + f func(int) int, +) { + s := stream.New().WithMaxGoroutines(10) + for elem := range in { + elem := elem + s.Go(func() stream.Callback { + res := f(elem) + return func() { out <- res } + }) + } + s.Wait() +} +``` + | +
-A FileSystem Abstraction System for Go
-[](https://travis-ci.org/spf13/afero) [](https://ci.appveyor.com/project/spf13/afero) [](https://godoc.org/github.com/spf13/afero) [](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[](https://github.com/spf13/afero/actions?query=workflow%3ACI)
+[](https://pkg.go.dev/mod/github.com/spf13/afero)
+[](https://goreportcard.com/report/github.com/spf13/afero)
+![Go Version](https://img.shields.io/badge/go%20version-%3E=1.23-61CFDD.svg?style=flat-square")
-# Overview
-Afero is a filesystem framework providing a simple, uniform and universal API
-interacting with any filesystem, as an abstraction layer providing interfaces,
-types and methods. Afero has an exceptionally clean interface and simple design
-without needless constructors or initialization methods.
+# Afero: The Universal Filesystem Abstraction for Go
-Afero is also a library providing a base set of interoperable backend
-filesystems that make it easy to work with afero while retaining all the power
-and benefit of the os and ioutil packages.
+Afero is a powerful and extensible filesystem abstraction system for Go. It provides a single, unified API for interacting with diverse filesystems—including the local disk, memory, archives, and network storage.
-Afero provides significant improvements over using the os package alone, most
-notably the ability to create mock and testing filesystems without relying on the disk.
+Afero acts as a drop-in replacement for the standard `os` package, enabling you to write modular code that is agnostic to the underlying storage, dramatically simplifies testing, and allows for sophisticated architectural patterns through filesystem composition.
-It is suitable for use in any situation where you would consider using the OS
-package as it provides an additional abstraction that makes it easy to use a
-memory backed file system during testing. It also adds support for the http
-filesystem for full interoperability.
+## Why Afero?
+Afero elevates filesystem interaction beyond simple file reading and writing, offering solutions for testability, flexibility, and advanced architecture.
-## Afero Features
+🔑 **Key Features:**
-* A single consistent API for accessing a variety of filesystems
-* Interoperation between a variety of file system types
-* A set of interfaces to encourage and enforce interoperability between backends
-* An atomic cross platform memory backed file system
-* Support for compositional (union) file systems by combining multiple file systems acting as one
-* Specialized backends which modify existing filesystems (Read Only, Regexp filtered)
-* A set of utility functions ported from io, ioutil & hugo to be afero aware
-* Wrapper for go 1.16 filesystem abstraction `io/fs.FS`
+* **Universal API:** Write your code once. Run it against the local OS, in-memory storage, ZIP/TAR archives, or remote systems (SFTP, GCS).
+* **Ultimate Testability:** Utilize `MemMapFs`, a fully concurrent-safe, read/write in-memory filesystem. Write fast, isolated, and reliable unit tests without touching the physical disk or worrying about cleanup.
+* **Powerful Composition:** Afero's hidden superpower. Layer filesystems on top of each other to create sophisticated behaviors:
+ * **Sandboxing:** Use `CopyOnWriteFs` to create temporary scratch spaces that isolate changes from the base filesystem.
+ * **Caching:** Use `CacheOnReadFs` to automatically layer a fast cache (like memory) over a slow backend (like a network drive).
+ * **Security Jails:** Use `BasePathFs` to restrict application access to a specific subdirectory (chroot).
+* **`os` Package Compatibility:** Afero mirrors the functions in the standard `os` package, making adoption and refactoring seamless.
+* **`io/fs` Compatibility:** Fully compatible with the Go standard library's `io/fs` interfaces.
-# Using Afero
+## Installation
-Afero is easy to use and easier to adopt.
+```bash
+go get github.com/spf13/afero
+```
-A few different ways you could use Afero:
+```go
+import "github.com/spf13/afero"
+```
-* Use the interfaces alone to define your own file system.
-* Wrapper for the OS packages.
-* Define different filesystems for different parts of your application.
-* Use Afero for mock filesystems while testing
+## Quick Start: The Power of Abstraction
-## Step 1: Install Afero
+The core of Afero is the `afero.Fs` interface. By designing your functions to accept this interface rather than calling `os.*` functions directly, your code instantly becomes more flexible and testable.
-First use go get to install the latest version of the library.
+### 1. Refactor Your Code
- $ go get github.com/spf13/afero
+Change functions that rely on the `os` package to accept `afero.Fs`.
-Next include Afero in your application.
```go
+// Before: Coupled to the OS and difficult to test
+// func ProcessConfiguration(path string) error {
+// data, err := os.ReadFile(path)
+// ...
+// }
+
import "github.com/spf13/afero"
+
+// After: Decoupled, flexible, and testable
+func ProcessConfiguration(fs afero.Fs, path string) error {
+ // Use Afero utility functions which mirror os/ioutil
+ data, err := afero.ReadFile(fs, path)
+ // ... process the data
+ return err
+}
```
-## Step 2: Declare a backend
+### 2. Usage in Production
-First define a package variable and set it to a pointer to a filesystem.
-```go
-var AppFs = afero.NewMemMapFs()
+In your production environment, inject the `OsFs` backend, which wraps the standard operating system calls.
-or
-
-var AppFs = afero.NewOsFs()
+```go
+func main() {
+ // Use the real OS filesystem
+ AppFs := afero.NewOsFs()
+ ProcessConfiguration(AppFs, "/etc/myapp.conf")
+}
```
-It is important to note that if you repeat the composite literal you
-will be using a completely new and isolated filesystem. In the case of
-OsFs it will still use the same underlying filesystem but will reduce
-the ability to drop in other filesystems as desired.
-## Step 3: Use it like you would the OS package
+### 3. Usage in Testing
-Throughout your application use any function and method like you normally
-would.
+In your tests, inject `MemMapFs`. This provides a blazing-fast, isolated, in-memory filesystem that requires no disk I/O and no cleanup.
-So if my application before had:
```go
-os.Open('/tmp/foo')
-```
-We would replace it with:
-```go
-AppFs.Open('/tmp/foo')
+func TestProcessConfiguration(t *testing.T) {
+ // Use the in-memory filesystem
+ AppFs := afero.NewMemMapFs()
+
+ // Pre-populate the memory filesystem for the test
+ configPath := "/test/config.json"
+ afero.WriteFile(AppFs, configPath, []byte(`{"feature": true}`), 0644)
+
+ // Run the test entirely in memory
+ err := ProcessConfiguration(AppFs, configPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+}
```
-`AppFs` being the variable we defined above.
+## Afero's Superpower: Composition
+Afero's most unique feature is its ability to combine filesystems. This allows you to build complex behaviors out of simple components, keeping your application logic clean.
-## List of all available functions
+### Example 1: Sandboxing with Copy-on-Write
-File System Methods Available:
-```go
-Chmod(name string, mode os.FileMode) : error
-Chown(name string, uid, gid int) : error
-Chtimes(name string, atime time.Time, mtime time.Time) : error
-Create(name string) : File, error
-Mkdir(name string, perm os.FileMode) : error
-MkdirAll(path string, perm os.FileMode) : error
-Name() : string
-Open(name string) : File, error
-OpenFile(name string, flag int, perm os.FileMode) : File, error
-Remove(name string) : error
-RemoveAll(path string) : error
-Rename(oldname, newname string) : error
-Stat(name string) : os.FileInfo, error
-```
-File Interfaces and Methods Available:
-```go
-io.Closer
-io.Reader
-io.ReaderAt
-io.Seeker
-io.Writer
-io.WriterAt
-
-Name() : string
-Readdir(count int) : []os.FileInfo, error
-Readdirnames(n int) : []string, error
-Stat() : os.FileInfo, error
-Sync() : error
-Truncate(size int64) : error
-WriteString(s string) : ret int, err error
-```
-In some applications it may make sense to define a new package that
-simply exports the file system variable for easy access from anywhere.
+Create a temporary environment where an application can "modify" system files without affecting the actual disk.
-## Using Afero's utility functions
+```go
+// 1. The base layer is the real OS, made read-only for safety.
+baseFs := afero.NewReadOnlyFs(afero.NewOsFs())
-Afero provides a set of functions to make it easier to use the underlying file systems.
-These functions have been primarily ported from io & ioutil with some developed for Hugo.
+// 2. The overlay layer is a temporary in-memory filesystem for changes.
+overlayFs := afero.NewMemMapFs()
-The afero utilities support all afero compatible backends.
+// 3. Combine them. Reads fall through to the base; writes only hit the overlay.
+sandboxFs := afero.NewCopyOnWriteFs(baseFs, overlayFs)
-The list of utilities includes:
+// The application can now "modify" /etc/hosts, but the changes are isolated in memory.
+afero.WriteFile(sandboxFs, "/etc/hosts", []byte("127.0.0.1 sandboxed-app"), 0644)
-```go
-DirExists(path string) (bool, error)
-Exists(path string) (bool, error)
-FileContainsBytes(filename string, subslice []byte) (bool, error)
-GetTempDir(subPath string) string
-IsDir(path string) (bool, error)
-IsEmpty(path string) (bool, error)
-ReadDir(dirname string) ([]os.FileInfo, error)
-ReadFile(filename string) ([]byte, error)
-SafeWriteReader(path string, r io.Reader) (err error)
-TempDir(dir, prefix string) (name string, err error)
-TempFile(dir, prefix string) (f File, err error)
-Walk(root string, walkFn filepath.WalkFunc) error
-WriteFile(filename string, data []byte, perm os.FileMode) error
-WriteReader(path string, r io.Reader) (err error)
+// The real /etc/hosts on disk is untouched.
```
-For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero)
-They are available under two different approaches to use. You can either call
-them directly where the first parameter of each function will be the file
-system, or you can declare a new `Afero`, a custom type used to bind these
-functions as methods to a given filesystem.
+### Example 2: Caching a Slow Filesystem
-### Calling utilities directly
+Improve performance by layering a fast cache (like memory) over a slow backend (like a network drive or cloud storage).
```go
-fs := new(afero.MemMapFs)
-f, err := afero.TempFile(fs,"", "ioutil-test")
+import "time"
-```
+// Assume 'remoteFs' is a slow backend (e.g., SFTP or GCS)
+var remoteFs afero.Fs
-### Calling via Afero
+// 'cacheFs' is a fast in-memory backend
+cacheFs := afero.NewMemMapFs()
-```go
-fs := afero.NewMemMapFs()
-afs := &afero.Afero{Fs: fs}
-f, err := afs.TempFile("", "ioutil-test")
+// Create the caching layer. Cache items for 5 minutes upon first read.
+cachedFs := afero.NewCacheOnReadFs(remoteFs, cacheFs, 5*time.Minute)
+
+// The first read is slow (fetches from remote, then caches)
+data1, _ := afero.ReadFile(cachedFs, "data.json")
+
+// The second read is instant (serves from memory cache)
+data2, _ := afero.ReadFile(cachedFs, "data.json")
```
-## Using Afero for Testing
+### Example 3: Security Jails (chroot)
-There is a large benefit to using a mock filesystem for testing. It has a
-completely blank state every time it is initialized and can be easily
-reproducible regardless of OS. You could create files to your heart’s content
-and the file access would be fast while also saving you from all the annoying
-issues with deleting temporary files, Windows file locking, etc. The MemMapFs
-backend is perfect for testing.
+Restrict an application component's access to a specific subdirectory.
-* Much faster than performing I/O operations on disk
-* Avoid security issues and permissions
-* Far more control. 'rm -rf /' with confidence
-* Test setup is far more easier to do
-* No test cleanup needed
+```go
+osFs := afero.NewOsFs()
-One way to accomplish this is to define a variable as mentioned above.
-In your application this will be set to afero.NewOsFs() during testing you
-can set it to afero.NewMemMapFs().
+// Create a filesystem rooted at /home/user/public
+// The application cannot access anything above this directory.
+jailedFs := afero.NewBasePathFs(osFs, "/home/user/public")
-It wouldn't be uncommon to have each test initialize a blank slate memory
-backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere
-appropriate in my application code. This approach ensures that Tests are order
-independent, with no test relying on the state left by an earlier test.
+// To the application, this is reading "/"
+// In reality, it's reading "/home/user/public/"
+dirInfo, err := afero.ReadDir(jailedFs, "/")
-Then in my tests I would initialize a new MemMapFs for each test:
-```go
-func TestExist(t *testing.T) {
- appFS := afero.NewMemMapFs()
- // create test files and directories
- appFS.MkdirAll("src/a", 0755)
- afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644)
- afero.WriteFile(appFS, "src/c", []byte("file c"), 0644)
- name := "src/c"
- _, err := appFS.Stat(name)
- if os.IsNotExist(err) {
- t.Errorf("file \"%s\" does not exist.\n", name)
- }
-}
+// Attempts to access parent directories fail
+_, err = jailedFs.Open("../secrets.txt") // Returns an error
```
-# Available Backends
-
-## Operating System Native
+## Real-World Use Cases
-### OsFs
+### Build Cloud-Agnostic Applications
-The first is simply a wrapper around the native OS calls. This makes it
-very easy to use as all of the calls are the same as the existing OS
-calls. It also makes it trivial to have your code use the OS during
-operation and a mock filesystem during testing or as needed.
+Write applications that seamlessly work with different storage backends:
```go
-appfs := afero.NewOsFs()
-appfs.MkdirAll("src/a", 0755)
-```
+type DocumentProcessor struct {
+ fs afero.Fs
+}
-## Memory Backed Storage
+func NewDocumentProcessor(fs afero.Fs) *DocumentProcessor {
+ return &DocumentProcessor{fs: fs}
+}
-### MemMapFs
+func (p *DocumentProcessor) Process(inputPath, outputPath string) error {
+ // This code works whether fs is local disk, cloud storage, or memory
+ content, err := afero.ReadFile(p.fs, inputPath)
+ if err != nil {
+ return err
+ }
+
+ processed := processContent(content)
+ return afero.WriteFile(p.fs, outputPath, processed, 0644)
+}
-Afero also provides a fully atomic memory backed filesystem perfect for use in
-mocking and to speed up unnecessary disk io when persistence isn’t
-necessary. It is fully concurrent and will work within go routines
-safely.
+// Use with local filesystem
+processor := NewDocumentProcessor(afero.NewOsFs())
-```go
-mm := afero.NewMemMapFs()
-mm.MkdirAll("src/a", 0755)
+// Use with Google Cloud Storage
+processor := NewDocumentProcessor(gcsFS)
+
+// Use with in-memory filesystem for testing
+processor := NewDocumentProcessor(afero.NewMemMapFs())
```
-#### InMemoryFile
+### Treating Archives as Filesystems
-As part of MemMapFs, Afero also provides an atomic, fully concurrent memory
-backed file implementation. This can be used in other memory backed file
-systems with ease. Plans are to add a radix tree memory stored file
-system using InMemoryFile.
+Read files directly from `.zip` or `.tar` archives without unpacking them to disk first.
-## Network Interfaces
+```go
+import (
+ "archive/zip"
+ "github.com/spf13/afero/zipfs"
+)
-### SftpFs
+// Assume 'zipReader' is a *zip.Reader initialized from a file or memory
+var zipReader *zip.Reader
-Afero has experimental support for secure file transfer protocol (sftp). Which can
-be used to perform file operations over a encrypted channel.
+// Create a read-only ZipFs
+archiveFS := zipfs.New(zipReader)
-## Filtering Backends
+// Read a file from within the archive using the standard Afero API
+content, err := afero.ReadFile(archiveFS, "/docs/readme.md")
+```
-### BasePathFs
+### Serving Any Filesystem over HTTP
-The BasePathFs restricts all operations to a given path within an Fs.
-The given file name to the operations on this Fs will be prepended with
-the base path before calling the source Fs.
+Use `HttpFs` to expose any Afero filesystem—even one created dynamically in memory—through a standard Go web server.
```go
-bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path")
-```
+import (
+ "net/http"
+ "github.com/spf13/afero"
+)
-### ReadOnlyFs
+func main() {
+ memFS := afero.NewMemMapFs()
+ afero.WriteFile(memFS, "index.html", []byte("