This is how I've been constructing HTTP services for a while now, or some variation thereof. Finally decided to make it into a reusable, shared repo.
And now, the elevator pitch (to myself):
A lightweight Go toolkit for HTTP, shipped as two independent modules:
client: HTTP client wrapper with functional options, streaming file downloads, async batch downloads, and token-bucket rate limiting.web: Mux, middleware, server lifecycle, request/response helpers, and structured errors built onnet/http.
Each module is versioned and imported independently. Use one, the other, or both. The mux and server packages are also independent of each other: you can use the mux with your own server, or wrap any http.Handler with the server for signal-driven lifecycle management.
More often than not, HTTP clients and servers are highly use-case specific; I could never find myself to like other highly engineered solutions. This is meant to be lightweight, a mere wrapper around the standard library.
# Client module
go get github.com/adamwoolhether/httper/client
# Web module
go get github.com/adamwoolhether/httper/webpackage main
import (
"context"
"fmt"
"log"
"net/http"
"time"
"github.com/adamwoolhether/httper/client"
)
func main() {
// Defaults to http defaults out of the box. Optionally supply your own
// http.Client or http.RoundTripper for full control.
c, err := client.Build(
client.WithTimeout(10 * time.Second),
client.WithUserAgent("my-app/1.0"),
// client.WithClient(customHTTPClient),
// client.WithTransport(customRoundTripper),
)
if err != nil {
log.Fatal(err)
}
u := client.URL("https", "httpbin.org", "/get")
req, err := client.Request(context.Background(), u, http.MethodGet)
if err != nil {
log.Fatal(err)
}
var dest map[string]any
if err := c.Do(req, http.StatusOK, client.WithDestination(&dest)); err != nil {
log.Fatal(err)
}
fmt.Println(dest)
}See Client Options.
Build a request with a JSON payload and decode the response into a struct. See Request Options, Do Options, and URL Options.
type Payload struct {
Name string `json:"name"`
}
type Response struct {
ID int `json:"id"`
Name string `json:"name"`
}
u := client.URL("https", "api.example.com", "/users")
req, err := client.Request(ctx, u, http.MethodPost,
client.WithPayload(Payload{Name: "alice"}),
)
if err != nil {
return err
}
var resp Response
err = c.Do(req, http.StatusCreated, client.WithDestination(&resp))Stream a file to disk with optional checksum verification and progress logging. See Download Options.
import (
"crypto/sha256"
"github.com/adamwoolhether/httper/client/download"
)
u := client.URL("https", "example.com", "/archive.tar.gz")
req, _ := client.Request(ctx, u, http.MethodGet)
err := c.Download(req, http.StatusOK, "/tmp/archive.tar.gz",
download.WithChecksum(sha256.New(), "e3b0c44298fc1c14..."),
download.WithProgress(),
download.WithSkipExisting(),
)Download multiple files concurrently with a bounded worker pool. See Download Options.
import "github.com/adamwoolhether/httper/client/download"
u1 := client.URL("https", "example.com", "/file1.zip")
req1, _ := client.Request(ctx, u1, http.MethodGet)
result, err := c.DownloadAsync(req1, http.StatusOK, "/tmp/file1.zip",
download.WithBatch(4), // max 4 concurrent downloads
)
if err != nil {
return err
}
// Add more downloads to the same batch.
u2 := client.URL("https", "example.com", "/file2.zip")
req2, _ := client.Request(ctx, u2, http.MethodGet)
result.Add(req2, http.StatusOK, "/tmp/file2.zip")
// Wait for all downloads to finish.
if err := result.Wait(); err != nil {
return err
}Wrap the transport with a token-bucket limiter. See Client Options.
c, err := client.Build(
client.WithThrottle(10, 5), // 10 req/s, burst of 5
)For a rate that whole requests per second cannot express, such as a per-minute limit, set the interval between requests instead:
c, err := client.Build(
client.WithThrottleEvery(time.Minute/3500, 1), // 3500 req/min, burst of 1
)Send a request again after a 429 or 5xx response. The retry honors Retry-After; without it, the retry backs off exponentially with jitter.
The retries stop and return the last response when the next wait would pass the deadline or exceed the backoff cap.
WithTimeout sets one deadline for all attempts, and each attempt waits on the throttle.
- A transport error with no response is never retried.
- A request with a body but no
GetBodyis sent once. Requests built byclient.RequesthaveGetBody, so a POST retries too. - When the retries give up,
UnexpectedStatusError.RetryAfterholds the wait the server asked for.
See Client Options and Retry Options.
c, err := client.Build(
client.WithTimeout(30 * time.Second),
client.WithRetry(3), // up to 3 retries after the first attempt
)
err = c.Do(req, http.StatusOK)
if client.IsRetryable(err) {
// A 429 or 5xx the retries did not clear, a timeout, or a transport failure with no response.
}Passed to client.Build(...).
client.WithClient(hc) // Start from a copy of hc instead of the default http.Client
client.WithTransport(rt) // Set a custom http.RoundTripper
client.WithTimeout(d) // Set the overall request timeout
client.WithUserAgent(s) // Add a persistent User-Agent header
client.WithThrottle(rps, burst) // Enable token-bucket rate limiting
client.WithThrottleEvery(d, b) // Enable token-bucket rate limiting at one request per d, burst b
client.WithRetry(n, opts...) // Retry 429 and 5xx responses up to n times
client.WithNoFollowRedirects() // Prevent following HTTP redirects
client.WithLogger(l) // Inject a custom slog.LoggerPassed to client.WithRetry(...).
client.WithBackoff(base, max) // Set the backoff's first wait and cap (default 1s, 30s); a longer Retry-After ends the retriesPassed to client.Request(...).
client.WithPayload(body) // Set the JSON-encoded request body
client.WithContentType(ct) // Override the default "application/json" Content-Type
client.WithHeaders(h) // Add custom headers to the request
client.WithCookies(c...) // Attach cookies to the requestPassed to client.Do(...).
client.WithDestination(&v) // Decode the response body into v
client.WithJSONNumb() // Preserve number precision as json.NumberPassed to client.URL(...).
client.WithQueryStrings(kv) // Append query parameters
client.WithPort(p) // Set the port number on the hostPassed to client.Download(...) / client.DownloadAsync(...).
download.WithBatch(n) // Enable batch mode with bounded concurrency
download.WithChecksum(h, expected) // Verify file checksum after download
download.WithProgress() // Enable periodic progress logging
download.WithSkipExisting() // Skip download if the file already existspackage main
import (
"context"
"log/slog"
"net/http"
"github.com/adamwoolhether/httper/web"
"github.com/adamwoolhether/httper/web/middleware"
"github.com/adamwoolhether/httper/web/mux"
"github.com/adamwoolhether/httper/web/server"
)
func main() {
log := slog.Default()
app := mux.New(mux.WithMiddleware(
middleware.Logger(log),
middleware.Errors(log),
middleware.Panics(),
))
app.Get("/hello/{name}", func(ctx context.Context, w http.ResponseWriter, r *http.Request) error {
name, err := web.Param(r, "name")
if err != nil {
return err
}
return web.RespondJSON(ctx, w, http.StatusOK, map[string]string{"hello": name})
})
srv := server.New(app, server.WithHost(":3000"))
if err := srv.Run(); err != nil {
log.Error("server", "error", err)
}
}The mux package wraps net/http.ServeMux with error-returning handlers, middleware chaining, and OpenTelemetry tracing.
// Handler is an http.Handler that returns an error.
type Handler func(ctx context.Context, w http.ResponseWriter, r *http.Request) error
// Middleware wraps a Handler.
type Middleware func(handler Handler) HandlerRegister routes with convenience methods:
app.Get(path, handler, mw...)
app.Post(path, handler, mw...)
app.Put(path, handler, mw...)
app.Patch(path, handler, mw...)
app.Delete(path, handler, mw...)Or use the lower-level methods directly:
app.Handle(method, group, path, handler, mw...) // full control
app.HandleRaw(method, group, path, httpHandler, mw...)// adapt a std http.Handler
app.HandleNoMiddleware(method, group, path, handler) // skip all route middlewareGroup() shares the same ServeMux, prefix, and virtual host but gets an independent middleware stack.
Mount(prefix) scopes all routes under a URL prefix. A nested Mount adds to the current prefix, so app.Mount("api").Mount("v1") serves routes under /api/v1.
Prefixes and route paths work with or without a leading slash.
VirtualHost(host) scopes routes to requests whose Host header is host. Matching ignores ASCII case, and route handlers see the lowercased Host. Routes without a virtual host match every host. A virtual host route matches only when the request names that host, so a request to localhost or an IP address does not reach it.
For its host, a virtual host route takes precedence over a route without a host. A virtual host catch-all such as Get("/") therefore also hides the host-less GET routes and WithStaticFS files on that host.
api := app.Mount("api/v1")
api.Use(authMiddleware)
api.Get("/users", listUsers)
api.Post("/users", createUser)
// Public group: no auth middleware
pub := app.Group()
pub.Get("/health", healthCheck)
// Routes for one host only
admin := app.VirtualHost("admin.example.com")
admin.Get("/dashboard", dashboard)server.New wraps net/http.Server with signal-driven graceful shutdown.
Defaults: host :8080, read timeout 5s, write timeout 10s, idle timeout 120s, shutdown timeout 20s.
srv := server.New(app,
server.WithHost(":3000"),
server.WithReadTimeout(10 * time.Second),
server.WithShutdownFunc(func(ctx context.Context) error {
log.Info("closing database")
return db.Close()
}),
)
// Run blocks until SIGINT/SIGTERM, then gracefully shuts down.
if err := srv.Run(); err != nil {
log.Error("server", "error", err)
}Shutdown(ctx) can also be called directly; the caller's context controls the deadline.
Pass middleware to mux.WithMiddleware(...) and they are automatically sorted by priority:
| Priority | Middleware | Scope | Description |
|---|---|---|---|
| 1 | CORS |
Global | Cross-origin resource sharing |
| 2 | CSRF |
Global | Cross-site request forgery protection |
| 3 | Logger |
Route | Request start/completion logging |
| 4 | Errors |
Route | Structured error responses |
| 5 | custom | Route | Any user-supplied middleware |
| 100 | Panics |
Route | Panic recovery |
Global middleware runs on every request (via ServeHTTP). Route middleware runs per matched route.
A lower priority wraps a higher one, so Logger is the outermost route middleware and Panics is the innermost.
middleware.CORS(origins, headers...) // []string origins, optional custom headers
middleware.CSRF(origins...) // exact scheme://host[:port] trusted origins, no wildcards; panics on an invalid one
middleware.Logger(log) // *slog.Logger
middleware.Errors(log, onUnknown...) // *slog.Logger; catches *errs.Error and FieldErrors; optional hooks run for unknown errors
middleware.Panics(onPanic...) // recovers from panics; optional hooks run for each panicErrors and Panics accept optional hooks of type func(ctx context.Context, err error). A hook is a closure, so it can capture any client it needs, such as an error tracker:
report := func(ctx context.Context, err error) {
sentryHub.CaptureException(err)
}
app := mux.New(mux.WithMiddleware(
middleware.Logger(log),
middleware.Errors(log, report), // errors that are not *errs.Error or FieldErrors
middleware.Panics(report), // every panic except http.ErrAbortHandler
))Panics reports a panic and then returns it as an internal *errs.Error, so Errors does not report it again.
CORS([]string{"*"}) allows every origin with Access-Control-Allow-Origin: * and no credentials.
An explicit allowlist reflects the request origin and allows credentials. In an allowlist entry, * matches characters other than ., :, and /, so it stays inside one hostname label or the port: https://*.example.com matches https://api.example.com but not https://a.b.example.com, and http://localhost:* matches any localhost port.
Per-route middleware can also be added inline:
app.Get("/admin", adminHandler, authMiddleware)Path parameters:
web.Param(r, "id") // string
web.ParamInt(r, "id") // int
web.ParamInt64(r, "id") // int64Query parameters:
web.QueryString(r, "q") // string
web.QueryBool(r, "flag") // bool
web.QueryInt(r, "page") // int
web.QueryInt64(r, "ts") // int64
// Plural forms read repeated keys and comma-separated values: ?id=1,2&id=3
web.QueryStrings(r, "tag") // []string
web.QueryBools(r, "flag") // []bool
web.QueryInts(r, "id") // []int
web.QueryInt64s(r, "id") // []int64Decode & Respond:
web.Decode(r, &input) // JSON decode; validates a struct or slice of structs; no body limit
web.DecodeLimit(w, r, &input, maxBytes) // Decode with the body limited to maxBytes
web.RespondJSON(ctx, w, statusCode, data) // JSON response
web.RespondError(ctx, w, errsErr) // structured error response
web.Redirect(w, r, url, code) // HTTP redirect (3xx)The errs package provides typed errors that map to HTTP status codes.
errs.New(http.StatusNotFound, err) // app-level error with status code
errs.NewInternal(err) // 500: message hidden from clients
errs.NewFieldsError("email", err) // field validation errorFieldErrors is a slice of FieldError{Field, Err}. The Errors middleware automatically responds with 422 when it encounters one.
Passed to mux.New(...).
mux.WithMiddleware(mw...) // Register middleware (auto-prioritized)
mux.WithTracer(tracer) // Inject an OpenTelemetry tracer
mux.WithLogger(log) // Set the logger for internal errors
mux.WithStaticFS(fsys, pathPrefix) // Serve static files from an fs.FSPassed to server.New(handler, ...).
server.WithServer(srv) // Inject an existing *http.Server as base
server.WithHost(addr) // Listen address (default ":8080")
server.WithReadTimeout(d) // Read timeout (default 5s)
server.WithWriteTimeout(d) // Write timeout (default 10s)
server.WithIdleTimeout(d) // Idle timeout (default 120s)
server.WithShutdownTimeout(d) // Shutdown timeout for Run (default 20s)
server.WithLogger(log) // Lifecycle logger
server.WithShutdownFunc(fn) // Register a shutdown hook
server.WithTLS(certFile, keyFile) // Enable TLSThe repository releases client and web as independent Go modules. Git tags point to a complete commit. The client/ or web/ prefix selects the module from that commit.
Release from an up-to-date main branch after the change merges:
git switch main
git pull --ff-only origin main
git status --short
git tag --list 'client/v*' --sort=-v:refname | head -1
git tag --list 'web/v*' --sort=-v:refname | head -1Choose the next semantic version for each changed module. Replace vX.Y.Z and vPREVIOUS before running these commands:
git tag -a client/vX.Y.Z HEAD -m "client vX.Y.Z"
git tag -a web/vX.Y.Z HEAD -m "web vX.Y.Z"
git diff client/vPREVIOUS..client/vX.Y.Z -- client
git diff web/vPREVIOUS..web/vX.Y.Z -- web
git push origin client/vX.Y.Z web/vX.Y.ZTag only modules that changed. Each pushed tag starts a separate Release workflow and creates a GitHub release.
This grew over the years, originally based on knowledge obtained from Powerful Command-Line Applications in Go by Ricardo Gerardi. And of course a shoutout to Ardan Labs, who has inspired many decisions here.