Working with files is something I do nearly every day in Go. It starts simply enough—reading a configuration, writing a log—but filesystems are full of surprises. Over time, I’ve learned that robust file handling isn’t about clever tricks; it’s about using clear, reliable patterns. Let me share some essential approaches that make file operations more predictable and less error-prone.
A fundamental task is copying a file. It seems straightforward, but doing it efficiently and safely matters. In Go, the io.Copy function is your best friend for this. It handles the memory management and the loop of reading and writing for you. Here’s how I typically write a copy function.
package main
import (
"io"
"os"
)
func copyFile(src, dst string) error {
sourceFile, err := os.Open(src)
if err != nil {
return err
}
defer sourceFile.Close()
destinationFile, err := os.Create(dst)
if err != nil {
return err
}
defer destinationFile.Close()
_, err = io.Copy(destinationFile, sourceFile)
return err
}
The defer statements are crucial. They guarantee the files will be closed when the function finishes, whether it succeeds or returns an error. This prevents resource leaks, which can cause your program to run out of file descriptors, especially under heavy load.
Sometimes, you need to create or replace a file. If your application crashes or loses power in the middle of a write, you could end up with a corrupted file. To prevent this, I use atomic writes. The idea is to write the complete data to a temporary file first. Only when the write is fully successful do you rename it to the final target filename. On most systems, the rename operation is atomic.
func writeFileAtomic(filePath string, data []byte) error {
// Create a temporary file in the same directory
tempPath := filePath + ".tmp"
// Write the data to the temporary file
err := os.WriteFile(tempPath, data, 0644)
if err != nil {
// Clean up the temporary file if the write failed
os.Remove(tempPath)
return err
}
// Atomically replace the target file with the temporary file
return os.Rename(tempPath, filePath)
}
This pattern is vital for configuration files, state files, or any data where consistency is more important than raw speed. A reader will either see the old, complete file or the new, complete file—never a mix.
What about very large files? Loading a multi-gigabyte log file into memory isn’t just slow; it’s often impossible. The solution is to read and process it in chunks. A buffered reader is perfect for this. You control the size of each chunk, keeping memory usage stable.
func processLargeFile(path string) error {
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
// Create a buffered reader with a 64KB buffer
reader := bufio.NewReaderSize(file, 65536)
for {
// Read up to the next newline
line, err := reader.ReadString('\n')
if err == io.EOF {
// Process the last line if it exists
if len(line) > 0 {
processLine(line)
}
break // End of file
}
if err != nil {
return err // Some other error
}
// Process the line (strip the newline if needed)
processLine(strings.TrimSuffix(line, "\n"))
}
return nil
}
This method allows you to process files of any size. I use it for parsing logs, transforming datasets, or reading line-oriented data streams.
Applications often need to work with entire directory trees. Go provides filepath.WalkDir, which is efficient and simple to use. It walks every file and directory, calling your function for each one.
import "path/filepath"
func collectGoFiles(rootDir string) ([]string, error) {
var goFiles []string
err := filepath.WalkDir(rootDir, func(path string, d os.DirEntry, err error) error {
if err != nil {
// Report the error but continue walking
fmt.Printf("Error accessing %s: %v\n", path, err)
return nil
}
// Check if it's a regular file and has a .go extension
if !d.IsDir() && filepath.Ext(path) == ".go" {
goFiles = append(goFiles, path)
}
return nil
})
return goFiles, err
}
You can modify the logic to filter by size, date, or any other attribute. This pattern is the backbone of build tools, backup scripts, and file synchronizers.
In a world with multiple processes, you might need to prevent two programs from writing to the same file at once. File locks provide a coordination mechanism. While Go’s standard library doesn’t have a cross-platform lock, packages like github.com/gofrs/flock work well.
import "github.com/gofrs/flock"
func updateWithLock(lockFilePath string, updateFunc func() error) error {
fileLock := flock.New(lockFilePath)
// Try to get an exclusive lock
locked, err := fileLock.TryLock()
if err != nil {
return fmt.Errorf("failed to acquire lock: %w", err)
}
if !locked {
return fmt.Errorf("file is already locked by another process")
}
// Ensure the lock is released when we're done
defer fileLock.Unlock()
// Now it's safe to perform the update
return updateFunc()
}
I use this for cron jobs or service scripts where only one instance should run at a time. The lock file itself becomes a signal.
Temporary files are useful for intermediate processing steps. It’s important to create them securely and clean them up reliably. os.CreateTemp handles the creation in the system’s secure temp directory.
func processWithTempFile(data []byte) error {
// Create a temporary file. The "*" is replaced with random characters.
tempFile, err := os.CreateTemp("", "myapp-process-*.tmp")
if err != nil {
return err
}
tempPath := tempFile.Name()
// Always try to clean up, even if later steps fail.
defer os.Remove(tempPath)
// Write data to the temp file
if _, err := tempFile.Write(data); err != nil {
tempFile.Close()
return err
}
// Close the file so the next step can read it
if err := tempFile.Close(); err != nil {
return err
}
// Now do something with the temp file, like compress it or parse it.
return doComplexWork(tempPath)
}
The defer os.Remove is a safety net. Even if your function panics, the cleanup will be attempted when the program continues.
Waiting for files to change is a common need. Polling a directory with a loop and time.Sleep is inefficient. Instead, I use the fsnotify library to get notified by the operating system when files are created, written, or deleted.
import "github.com/fsnotify/fsnotify"
func watchForChanges(dirToWatch string) {
watcher, err := fsnotify.NewWatcher()
if err != nil {
panic(err)
}
defer watcher.Close()
// Add the directory to the watch list
err = watcher.Add(dirToWatch)
if err != nil {
panic(err)
}
for {
select {
case event, ok := <-watcher.Events:
if !ok {
return // channel closed
}
// Often, a write creates a CREATE event followed by a WRITE.
if event.Has(fsnotify.Write) {
fmt.Printf("File was written: %s\n", event.Name)
}
case err, ok := <-watcher.Errors:
if !ok {
return
}
fmt.Printf("Watcher error: %v\n", err)
}
}
}
This is how development servers automatically reload or how log aggregators start processing new files instantly. You should add logic to handle rapid sequences of events, so you don’t process the same file five times in one second.
Files are more than just their content. You often need to check their size, permissions, or when they were last modified. The os.Stat function returns a FileInfo object with these details.
func checkFileDetails(path string) error {
info, err := os.Stat(path)
if err != nil {
return err // File might not exist
}
fmt.Printf("Size: %d bytes\n", info.Size())
fmt.Printf("Permissions: %s\n", info.Mode())
fmt.Printf("Last modified: %s\n", info.ModTime())
// Check if it's a directory
if info.IsDir() {
fmt.Println("This is a directory")
}
// You can also change permissions
if info.Mode().Perm() != 0644 {
err := os.Chmod(path, 0644)
if err != nil {
return fmt.Errorf("failed to change permissions: %w", err)
}
}
return nil
}
Getting permissions right is critical for security. You don’t want a configuration file with secrets to be world-readable, nor do you want a log file your application can’t write to.
Building file paths by joining strings with slashes is a recipe for bugs. Windows uses backslashes. Go’s filepath package builds paths correctly for the operating system your code is running on.
func getCachePath(userID string, cacheKey string) string {
// This is safe and cross-platform
return filepath.Join("var", "cache", "myapp", userID, cacheKey + ".dat")
}
Always use filepath.Join for concatenating path components. Use filepath.Clean to remove unnecessary clutter like ./ or ../ from a path string. This makes your code portable and less prone to subtle errors.
Errors when working with files are not all the same. A file not found is expected in many cases, while a permission error requires user intervention. Go provides helper functions to check error types.
func readImportantFile(path string) ([]byte, error) {
data, err := os.ReadFile(path)
if err != nil {
// Check what kind of error it is
if os.IsNotExist(err) {
// File doesn't exist. Maybe return a default?
return []byte("{}"), nil
}
if os.IsPermission(err) {
// User can't read this. Need to tell them.
return nil, fmt.Errorf("cannot read %s: check file permissions", path)
}
// Some other error (disk full, hardware failure)
return nil, fmt.Errorf("failed to read file: %w", err)
}
return data, nil
}
Distinguishing errors allows you to build more resilient software. It can retry on temporary errors, use defaults for missing files, and give clear instructions for permission problems.
Finally, let’s consider concurrency within a single Go program. If multiple goroutines need to write to the same file handle, you must coordinate their access. A simple sync.Mutex can serialize the writes.
import "sync"
type ConcurrentWriter struct {
file *os.File
mu sync.Mutex
}
func (cw *ConcurrentWriter) WriteLine(text string) error {
cw.mu.Lock()
defer cw.mu.Unlock()
_, err := cw.file.WriteString(text + "\n")
return err
}
Alternatively, design your program so that one dedicated goroutine handles all file writes, receiving data on a channel. This is a cleaner model for high-throughput logging.
func startLogWriter(filePath string) (chan<- string, error) {
file, err := os.OpenFile(filePath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return nil, err
}
messages := make(chan string, 100)
go func() {
for msg := range messages {
file.WriteString(msg + "\n")
}
file.Close()
}()
return messages, nil
}
Other parts of your program send log lines to the channel, and the single writer goroutine handles them sequentially. This is efficient and thread-safe.
These patterns are tools. You won’t need every one in every project, but knowing they exist helps you solve problems with confidence. File system code is often at the foundation of an application. Getting it right means your program will be stable, efficient, and easy to maintain. Start with the simple copy, think about atomicity for important data, handle errors specifically, and use the right concurrency model for your needs. The rest follows from these principles.
