Go for Cloud
Over the last decade, Go became the default language for a large share of cloud infrastructure: containers, orchestrators, service meshes, proxies, and serverless backends. A lot of that adoption has little to do with syntax and everything to do with how Go binaries behave once they leave your laptop. Small static executables, fast startup, predictable memory behavior, and built-in diagnostics map directly onto the realities of running ephemeral workloads in the cloud.
Still, taking a Go service from go run to a production cloud environment
comes with a handful of operational habits and subtle gotchas that are easy
to miss at first. Below are the practices I keep coming back to when building,
deploying, and operating Go services in the cloud.
Build small binaries. Go builds small binaries, which makes it well suited
for containerized and serverless environments where image pull time directly
affects cold starts and rollout speed. A typical Go service binary with all its
runtime dependencies can be as small as 20-25 MB. If image size is tight, you
can trim binaries further with -ldflags="-s -w" to strip symbol and DWARF
tables, though keep an unstripped copy in your build artifacts if you rely on
symbolized profiles or postmortem debugging.
Runtime initialization is fast. Go’s runtime initialization is fast. If you are writing autoscaling servers in Go, cold start is rarely bottlenecked by Go’s runtime initialization. Go libraries and frameworks also tend to avoid heavy reflection or classpath scanning at startup compared to ecosystems like the JVM, so the entire process reaches ready state quickly.
Build static binaries. Go programs compile into a single static binary,
which simplifies delivery: CI/CD pipelines can produce one self-contained
artifact and copy it straight into a minimal container or remote host. Watch
out for CGO, though. When CGO is enabled, the binary dynamically links against
the host’s libc, which can fail at startup in scratch or Alpine containers.
Disable CGO explicitly when you want a truly static Linux binary:
$ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o server .
Cross compile to 64-bit Linux. The Go compiler supports cross compilation out of the box. As long as you do not have CGO dependencies, you can build for your production OS and architecture from any developer machine:
$ GOOS=linux GOARCH=amd64 go build
Don’t ship your toolchain. You do not need the Go toolchain installed in
production to run a Go binary. Avoid shipping production containers based on
the full golang image. Instead, use a multi-stage container build to compile
the binary and copy only the final executable (and CA certificates if you make
outbound TLS calls) into a minimal base image such as distroless or scratch:
FROM golang:1.13 AS builder
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /out/server .
FROM gcr.io/distroless/static-debian10
COPY --from=builder /out/server /server
ENTRYPOINT ["/server"]
Rebuild and redeploy with Go releases. Go only supports the last two major versions. Just because Go runtime is compiled in the body, with each Go release, rebuild and redeploy your production services.
At Google, we use the release candidates to build production services as soon as there is an RC version. You can use the RC version for production services, or at least push to canary with the RC version. If you see an unexpected behavior, immediately file an issue.
The go tool can print the Go version used to build a binary:
$ go version <binary>
<binary>: go1.13.5
You can additionally use tools like gops to list and report the Go versions of the binaries currently running on your system.
Embed commit versions into binaries. Embed the revision numbers
when you are building a Go binary.
You can also embed the build constraint and other
options used when building.
debug.BuildInfo
also provides information about the module as well as the dependencies.
Alternatively, go command can report module information and the dependencies:
$ go version -m dlv
dlv: go1.13.5
path github.com/go-delve/delve/cmd/dlv
mod github.com/go-delve/delve v1.3.2 h1:K8VjV+Q2YnBYlPq0ctjrvc9h7h03wXszlszzfGW5Tog=
dep github.com/cosiner/argv v0.0.0-20170225145430-13bacc38a0a5 h1:rIXlvz2IWiupMFlC45cZCXZFvKX/ExBcSLrDy2G0Lp8=
dep github.com/mattn/go-isatty v0.0.3 h1:ns/ykhmWi7G9O+8a448SecJU3nSMBXJfqQkl0upE1jI=
dep github.com/peterh/liner v0.0.0-20170317030525-88609521dc4b h1:8uaXtUkxiy+T/zdLWuxa/PG4so0TPZDZfafFNNSaptE=
dep github.com/sirupsen/logrus v0.0.0-20180523074243-ea8897e79973 h1:3AJZYTzw3gm3TNTt30x0CCKD7GOn2sdd50Hn35fQkGY=
dep github.com/spf13/cobra v0.0.0-20170417170307-b6cb39589372 h1:eRfW1vRS4th8IX2iQeyqQ8cOUNOySvAYJ0IUvTXGoYA=
dep github.com/spf13/pflag v0.0.0-20170417173400-9e4c21054fa1 h1:7bozMfSdo41n2NOc0GsVTTVUiA+Ncaj6pXNpm4UHKys=
dep go.starlark.net v0.0.0-20190702223751-32f345186213 h1:lkYv5AKwvvduv5XWP6szk/bvvgO6aDeUujhZQXIFTes=
dep golang.org/x/arch v0.0.0-20171004143515-077ac972c2e4 h1:TP7YcWHbnFq4v8/3wM2JwgM0SRRtsYJ7Z6Oj0arz2bs=
dep golang.org/x/crypto v0.0.0-20180614174826-fd5f17ee7299 h1:zxP+xTjjk4kD+M5IFPweL7/4851FUhYkzbDqbzkN1JE=
dep golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k=
dep gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE=
FaaS is Go binary as a service. Function-as-a-service products such as Google Cloud Functions or AWS Lambda serves Go functions. But in fact, they are building a user function into a binary and serve the binary. This means you have to organize and build packages acknowledging this fact. Because the final binary is not forked for every incoming request but is being reused:
- You may have data races if you access to common resources from multiple functions.
- You may need to use
sync.Oncein the function to initialize some of the resources if you need the incoming request to initialize. - Background goroutines may need to keep working even after the function is finished and binary is about to be terminated. You may need to flush data manually or gradually shutdown background routines.
- Providers are not consistent about signaling the Go process before a shutdown. Expect hard terminations as soon as your function exits.
- You may want to use the incoming request’s context for calls initiated in the function. In such cases, being able to reuse resources are getting harder.
Gracefully reject incoming requests. When an orchestrator scales down
replicas or rolls out a new version, it sends SIGTERM before forcefully
killing the process. Listen for termination signals and call http.Server’s
Shutdown with a bounded
context so your server stops accepting new connections while giving in-flight
requests time to finish:
srv := &http.Server{Addr: ":8080", Handler: mux}
go srv.ListenAndServe()
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Printf("graceful shutdown failed: %v", err)
}
Report the essential metrics. The Go runtime exposes a set of diagnostic
metrics that are cheap to collect and catch common production issues early.
Export runtime.NumGoroutine (to spot goroutine leaks), runtime.NumThreads,
runtime.NumCGOCalls, and runtime.ReadMemStats
(for live heap size, allocation rates, and GC pause times) to your monitoring
system. See instrumentation guides such as Prometheus’ Go library
as a reference on what to export.
Print scheduling and GC events. Go can print scheduler and garbage
collector diagnostics directly to standard error via the GODEBUG environment
variable, which is handy when debugging latency spikes or CPU starvation in
containers.
The following command prints each GC cycle (heap size before/after collection and wall-clock pause times) along with scheduler snapshots (idle processors and runnable queue lengths) every 5000 ms:
$ GODEBUG=gctrace=1,schedtrace=5000 <binary>
Propagate the incoming context. Go allows propagating the context in the process via context.Context. You can also signal cancellation or timeout decisions to other goroutines using context. You can use context to propagate values such as trace/request IDs or other metadata relevant in the critical path. You can log with context key/values where it applies. If you have an incoming request context, keep propagating it. For example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// use r.Context() for the calls made here.
})
Continuously profile in production. Go uses pprof which is a lightweight profiling collection mechanism. It only adds a single-digit percentage overhead to the execution when enabled. You can utilize this strength by collecting profiles from production systems and understanding the fleet-wide hotspots to optimize. See Continuous Profiling of Go programs for more insights and a reference implementation of a continuous profiling product.
pprof can symbolize the profiling
data by incorporating the binary. If you are collecting
profiles from production, you’d like to store profiling data with symbols.
Even though there is no good standard library function for this task,
there is an existing reference that
can be adopted.
Dump debuggable postmortems. Go allows post-mortem debugging. When running Go in production, core dumps allow you to retrospectively investigate why binaries crash. If you have Go programs constantly crashing, you can retrieve their core dumps and understand why they crashed and which state they were in. You can also utilize core dumps to debug in production by taking a snapshot (a core dump) and using your debugger. See core dumps for more.