Skip to content

Digging deeper

Storage

Local, in-memory and S3 disks, a file handler, signed temporary URLs, presigned URLs and safe uploads.

Last updated:

Go
disk, err := storage.NewLocal("storage/app/public", "/storage") // os.Root: no "..", no escaping symlinks
r.Handle("/storage/", http.StripPrefix("/storage", storage.Handler(disk, nil)))

path, err := storage.PutUploaded(ctx, disk, fileHeader, "avatars", storage.Images(2<<20))
url := disk.URL(path)

Disks#

storage.Disk has Put (atomic rename on local disks), Open, Exists, Delete, List and URL; storage.Get and storage.PutBytes are helpers. storage.NewMemory(baseURL) serves tests, and S3 lives in its own module. Custom disks pass storage/storagetest (conformance and traversal suites).

Serving files#

storage.Handler(disk, signer) never serves dotfiles or directories and sends X-Content-Type-Options: nosniff. With a signed.Signer it answers only signed, unexpired URLs:

Go
private, _ := storage.NewLocal("storage/app/private", "")
r.Handle("/files/", http.StripPrefix("/files", storage.Handler(private, signer)))

link, err := storage.TemporaryURL(signer, "/files", "invoices/42.pdf", 15*time.Minute)

Public files are served by the handler instead of a storage:link symlink (which needs extra privileges on Windows).

Uploads#

PutUploaded never uses the client's file name: it stores a random name with an extension from the type detected in the content (http.DetectContentType), checks the size from the header and again while copying, and answers 422 for disallowed types or sizes.

Go
storage.Images(2 << 20)                                               // JPEG, PNG, GIF, WebP up to 2 MiB
storage.UploadRules{MaxSize: 10 << 20, Types: []string{"application/pdf"}}

For streaming, resumable (tus), scanned and direct-to-bucket uploads, see File uploads.

S3#

Terminal
go get github.com/AlexGalhardo/vukem/drivers/s3

A storage.Disk on Amazon S3 or any S3-compatible service (Cloudflare R2, DigitalOcean Spaces, Backblaze B2, RustFS, SeaweedFS, Garage). There is no AWS SDK: requests are signed by a standard-library Signature Version 4 implementation verified against the AWS test suite.

Go
cfg, err := config.Load[s3.Config](app.Source)
disk, err := s3.New(cfg) // validates the config, no I/O

err = disk.Put(ctx, "videos/intro.mp4", file)                   // one PUT up to S3_PART_SIZE, multipart above
info, err := disk.Stat(ctx, "videos/intro.mp4")                 // Size, ETag, ContentType, LastModified
up, err := disk.PresignPut("uploads/avatar.png", 5*time.Minute) // browser uploads straight to the bucket
up, err := disk.PresignPutWith("uploads/avatar.png", 5*time.Minute, s3.PresignPutOptions{
	ContentType: "image/png", ContentLength: size, // signed: the upload must send exactly these
})
link, err := disk.PresignGet("invoices/42.pdf", time.Hour)
VariableDefaultMeaning
S3_BUCKET—bucket name (3–63 of a-z 0-9 . -)
S3_REGIONus-east-1signing region (auto for R2)
S3_ENDPOINThttps://s3.<region>.amazonaws.comorigin of an S3-compatible service
S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY—required; S3_SESSION_TOKEN for temporary credentials
S3_USE_PATH_STYLEfalseendpoint/bucket/key instead of bucket.endpoint/key (most self-hosted servers)
S3_INSECURE_HTTPfalseallow an http:// endpoint (local development only)
S3_PART_SIZE8 MiBmultipart part size, 5 MiB to 5 GiB
S3_PUBLIC_URL—prefix for disk.URL (CDN or public bucket); empty means private
  • Put buffers at most one part. A multipart upload is aborted on any failure (part error, reader error, cancelled context, a 200 carrying an <Error>), so no orphaned parts are billed. The content type comes from the extension, else from the first bytes.
  • Open streams the object; List follows continuation tokens and skips folder placeholders. Missing objects match fs.ErrNotExist; other failures are *s3.Error (StatusCode, Code, Message, RequestID).
  • disk.CreateBucket(ctx) creates the bucket in S3_REGION, for dev setup and tests against a fresh local S3 (RustFS, MinIO). A bucket you already own is not an error; a name taken by another account is an *s3.Error with Code BucketAlreadyExists.
  • Keys go through storage.Clean: no empty key, no leading /, no .., backslash or :.
  • Presigned URLs expire after 1 second to 7 days. PresignGet and PresignPut sign only the host header, so a PresignPut URL accepts any type and size. PresignPutWith also signs Content-Type and/or Content-Length (s3.PresignPutOptions): the uploader must send exactly those values or the bucket answers 403. Browsers set Content-Length from the body, but the Content-Type header must be sent explicitly: fetch(url, {method: "PUT", headers: {"Content-Type": type}, body: file}). A signed Content-Length is an exact size, not a ceiling: pass the real file size the client declared (after checking it against your maximum), never the maximum itself, or every smaller file gets 403. Credentials are config.Secret, so errors and %+v never show them.
  • Integration tests run against RustFS (task services:up); the driver passes storagetest.Run and storagetest.RunTraversal.