https://go.dev/doc/diagnostics

Diagnostics is the umbrella term for tools/techniques that help you understand why a Go program is slow, misbehaving, or using resources the way it does. The Go ecosystem groups this into four categories: profiling, tracing, debugging, and runtime statistics/events.
- Note: some diagnostics tools interfere with each other — e.g. precise memory profiling skews CPU profiles, and goroutine blocking profiling affects scheduler traces. Prefer running one diagnostic tool at a time for accurate results.

Profiling

Profiling identifies the expensive or frequently called parts of your code — where CPU time goes, how memory is allocated, how many goroutines/threads exist, etc. It answers "what is my program spending resources on?"
Go's runtime exposes profiling data via the runtime/pprof package, which you then inspect using the go tool pprof visualizer. Data can be collected two main ways:
- During tests, via go test -cpuprofile / memprofile.
- At runtime, via HTTP endpoints exposed by the net/http/pprof package.
Built-in profile types:
- cpu: where the program spends time while actively burning CPU cycles (not while sleeping/blocked on I/O).
- heap: memory allocation samples — used to track memory usage over time and catch leaks.
- threadcreate: which parts of the code trigger new OS thread creation.
- goroutine: stack traces of all currently running goroutines.
- block: where goroutines block on synchronization primitives (channels, mutexes, timers). Disabled by default — enable with runtime.SetBlockProfileRate.
- mutex: reports lock contention. Useful when CPU utilization looks low but something's still slow. Disabled by default — enable with runtime.SetMutexProfileFraction.

How to enable profiling in a live HTTP server — just blank-import net/http/pprof and it registers diagnostic endpoints on the default mux under /debug/pprof/:

import (
    "log"
    "net/http"
    _ "net/http/pprof" // registers /debug/pprof/* handlers
)

func main() {
    log.Println(http.ListenAndServe("localhost:6060", nil))
}

You can then pull a 30-second CPU profile straight from the running process:
```
  go tool pprof <http://localhost:6060/debug/pprof/profile?seconds=30>
```

Want the pprof handlers on a custom path/port instead of the default mux? Register them manually:

package main

import (
    "log"
    "net/http"
    "net/http/pprof"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/custom_debug_path/profile", pprof.Profile)
    log.Fatal(http.ListenAndServe(":7777", mux))
}

Is it safe to profile production? Yes — but some profiles (especially CPU) add overhead, so expect a small performance hit. A common pattern: pick a random replica out of many, profile it for X seconds every Y seconds, save the results, and repeat — rather than profiling every instance continuously.
Visualizing profiles — go tool pprof supports several output modes once you have a profile file:
```
go tool pprof -top myprofile.prof      # text listing of most expensive calls
go tool pprof -web myprofile.prof      # graph visualization
go tool pprof -weblist=funcName myprofile.prof  # line-by-line source cost
```
- There's also flame graphs, which let you visually zoom in/out of a specific call ancestry to spot the most expensive code paths.
You're not limited to the built-in profiles — you can define custom ones with pprof.Profile and inspect them with the same tooling.

Tracing

Tracing instruments your code to measure latency across the lifecycle of a call or request — it tells you "where did the time go, step by step?" as opposed to profiling's "what consumed the most resources overall?"
Within a single process, Go provides the golang.org/x/net/trace package as a lightweight tracing backend with a simple dashboard.
Go also ships a runtime execution tracer that records low-level scheduler, GC, syscall, and goroutine events over a time window — analyzed via go tool trace.
- Capture a trace like this:
```
import (
    "os"
    "runtime/trace"
)

func main() {
    f, _ := os.Create("trace.out")
    defer f.Close()

    trace.Start(f)
    defer trace.Stop()

    // ... your program logic ...
}
```
```
go tool trace trace.out
```
  - The execution tracer is great for spotting poorly parallelized execution or lock contention (e.g. seeing goroutines execute fine, then suddenly serialize), but it's not the right tool for finding raw CPU/memory hot spots — use profiling for that instead.
Distributed tracing extends this idea across process boundaries. Once a system is no longer a single monolith — a request hops from a front-end service to several back-ends — following that request's latency requires trace data that spans multiple Go processes, not just one.
- Distributed tracing helps you:
  - Instrument and analyze latency across an entire request's lifecycle, not just one process.
  - See integration issues between services that only show up in production.
  - Find bottlenecks that aren't obvious until you have the tracing data in front of you.
- Go doesn't automatically intercept every function call to create spans — you have to manually instrument code to start, annotate, and end trace spans.
- Trace context (IDs, tags) is typically propagated through context.Context, since there's no single standard trace-header format — each tracing backend (Jaeger, Zipkin, OpenTelemetry, etc.) provides its own propagation helpers.
```
func handler(ctx context.Context) {
    ctx, span := tracer.Start(ctx, "handler")
    defer span.End()

    doWork(ctx) // pass ctx down so the trace continues
}
```
  - net/http/httptrace.ClientTrace is a standard-library hook that lets you observe low-level events (DNS lookup, connection reuse, etc.) in the lifecycle of an outgoing HTTP request — useful for feeding into your own traces.

Debugging

Debugging is the process of pausing a running program and inspecting its state (variables, stack, goroutines) to figure out why it's misbehaving — as opposed to profiling/tracing, which tell you about performance and latency.