Skip to content
Vukem

Digging deeper

File uploads

Streaming multipart uploads with per-route limits, content sniffing and virus scanning, resumable tus uploads and direct-to-S3 presigned uploads.

Last updated:

Package upload (stdlib only) goes beyond storage.PutUploaded: it never buffers a whole file.

Go
rules := upload.Images(5 << 20) // MaxFileSize and Types are required: zero Rules reject everything
rules.MaxFiles, rules.Dir = 3, "photos"
rules.Scanner = upload.ClamAV("127.0.0.1:3310") // optional

form, err := upload.Receive(c, disk, rules) // form.Values, form.Files[i].Path/Name/Type/Size/SHA256

Streaming multipart#

  • Receive reads r.MultipartReader() part by part and streams each file to the storage.Disk: a 64 MiB upload allocates about 100 KiB.
  • Rules bound the file size, the whole request (MaxSize, default MaxFiles*MaxFileSize + 1 MiB, which replaces the router-wide body limit for that route), the file count, and field count and size. Sizes fail with 413; types, extensions and count with 422. On any error every file already written is deleted.
  • Type checks: http.DetectContentType on the first 512 bytes must be in Types, and the client's extension must be one known for that type (upload.Extensions("image/jpeg") → .jpg, .jpeg). The client's Content-Type is ignored. Stored names are random with the canonical extension; the sanitized client name (upload.SanitizeName) is metadata only.
  • upload.Save(ctx, disk, rules, field, filename, r) applies the same checks to any reader (finished tus uploads, imports).
  • Image dimensions: a 50 KB file can declare a 100 000 × 100 000 canvas that takes gigabytes to decode (a decompression bomb). upload.Images sets MaxPixels to 25 megapixels: after storing a JPEG, PNG, GIF or WebP, only its header is read, and the file is deleted with 422 when the declared width × height is larger, or when the header cannot be read. Raise it for photo apps (rules.MaxPixels = 50_000_000); 0 turns the check off. Direct uploads get the same check in Verify.

Virus scanning#

With Rules.Scanner, files are written to QuarantineDir/.<random> (dot-named, so storage.Handler never serves them), scanned, then copied to Dir. ErrInfected deletes the file and answers 422; a scanner error deletes it and answers 503 (fail closed). upload.ClamAV(addr) speaks clamd's INSTREAM over TCP with per-operation timeouts.

Resumable uploads (tus)#

upload.Tus implements tus 1.0 core with the creation, expiration and termination extensions:

Go
tus, err := upload.Tus(upload.TusConfig{
	Dir: "storage/app/tus", BasePath: "/uploads/tus/", MaxSize: 2 << 30,
	Authorize: func(r *http.Request) error { /* session or token check */ return nil },
	OnComplete: func(ctx context.Context, info upload.TusInfo, open func() (io.ReadCloser, error)) error {
		rc, err := open()
		if err != nil {
			return err
		}
		defer rc.Close()
		_, err = upload.Save(ctx, disk, rules, "file", info.Filename(), rc)
		return err
	},
})
r.Handle("/uploads/tus/", tus)
app.OnShutdown(func(context.Context) error { return tus.Close() })
  • POST checks Upload-Length and Upload-Metadata and answers 201 with a random Location; HEAD returns the offset; PATCH needs Tus-Resumable: 1.0.0, Content-Type: application/offset+octet-stream and the current Upload-Offset, and appends the body as it streams. An interrupted request keeps what arrived. Concurrent PATCHes on one upload get 423.
  • The upload id is the capability, so Authorize is required (boot error when nil; return nil explicitly to allow everyone).
  • Abandoned uploads expire after Expiry (24h); schedule tus.Cleanup(ctx) to delete them. When the last byte arrives OnComplete runs; its error goes to the client (422 for validation) and the partial data is deleted either way. Validation means *errs.ValidationError, z.FieldError or z.Issues (the first message is the response text; a plain Refine error reads "The value field is invalid."); an Authorize error keeps its 4xx status (422 for z errors, anything else 403) but never its text.

Direct-to-S3 uploads#

Browsers can upload straight to the bucket. upload.Direct takes any upload.Presigner:

Go
type Presigner interface {
	PresignPut(ctx context.Context, key, contentType string, maxSize int64, ttl time.Duration) (url string, headers http.Header, err error)
}

direct, err := upload.Direct(upload.DirectConfig{Presigner: presigner, Rules: rules, Authorize: authorize})
r.Handle("/uploads/presign", direct)

// later, when the client reports the upload done:
file, err := upload.Verify(ctx, disk, key, rules)

Direct answers POST {"filename","content_type","size"} with {"key","url","method":"PUT","headers","expires_at"} after checking the declared type, extension and size, under a random key in Rules.Dir (TTL 15 minutes, at most 1 hour). The declared type cannot be trusted, so upload.Verify sniffs the stored object and deletes it when it breaks the rules.

Thumbnails (contrib/imaging)#

Product photos, avatars and galleries need smaller copies. The optional module contrib/imaging makes them:

Terminal
go get github.com/AlexGalhardo/vukem/contrib/imaging
Go
form, err := upload.Receive(c, disk, upload.Images(10<<20))
if err != nil {
	return err
}

photo := form.Files[0]

// One size:
thumb, err := imaging.Thumbnail(c.Context(), disk, photo.Path, "products/"+id+"/thumb",
	imaging.Options{MaxWidth: 400, MaxHeight: 400})
// thumb.Path = "products/<id>/thumb.jpg", thumb.Width, thumb.Height

// Several sizes from one decode:
sizes, err := imaging.Thumbnails(c.Context(), disk, photo.Path, "products/"+id, map[string]imaging.Options{
	"sm": {MaxWidth: 320, MaxHeight: 320},
	"lg": {MaxWidth: 1200, MaxHeight: 1200, Quality: 88},
})
// sizes["sm"].Path = "products/<id>-sm.jpg"

What it does, step by step:

  1. Reads the file from the disk (at most MaxBytes, 25 MiB by default) and picks the decoder from the content (JPEG, PNG, GIF, WebP), never the file name.
  2. Reads only the header and refuses images that declare more than MaxPixels (upload.DefaultMaxPixels, 25 megapixels) before decoding: a decompression bomb never gets decoded.
  3. Scales down to fit MaxWidth × MaxHeight, keeping the aspect ratio (CatmullRom filter). Small images are never upscaled.
  4. Writes a JPEG (Quality, default 82), or a PNG when the image has transparency, replacing a file already at that path. Re-encoding removes all metadata (EXIF, GPS location). EXIF orientation is not applied: photos taken sideways stay sideways. Animated GIFs become a still of their first frame.
ErrorWhen
imaging.ErrTooLargethe file is bigger than MaxBytes or declares more than MaxPixels
imaging.ErrUnsupportednot a readable JPEG, PNG, GIF or WebP
imaging.ErrInvalidOptionsMaxWidth/MaxHeight missing or above 16384, Quality outside 0–100, a Thumbnails size name that is not letters, digits, - or _

On failure the output is removed. Thumbnails applies the strictest MaxPixels/MaxBytes of all sizes (defaults included) to its single decode.

Memory: one call holds the source file plus its decoded pixels, up to about 8 bytes per pixel (16-bit PNG), so a 25-megapixel image can take ~200 MB. Resizing also takes CPU time. For big uploads, store the original in the request and make thumbnails in a queue job with limited concurrency.

HTMX progress component#

The HTMX starter ships resources/views/components/upload.html (a data-upload form with a <progress>) and a /uploads demo page. resources/js/app.ts swaps htmx's fetch for XMLHttpRequest on those forms to report progress, without inline scripts, so the CSP holds. The demo stores files under UPLOADS_DIR (default storage/app/uploads); UPLOADS_DISK=memory swaps in storage.NewMemory for tests (refused in production), so the generated TestUploads never writes to the filesystem: on Windows, antivirus filters briefly recreate just-deleted files, which can make t.TempDir cleanup fail with "directory is not empty". Use a memory disk the same way in your own upload tests. Validate files received through c.Input() with z.File() (validation).