Why is my Go service leaking goroutines under sustained load?
I'm running a Go HTTP service that handles ~500 req/s. After about 2 hours of sustained traffic, goroutine count climbs from ~200 to 12k+ and memory follows. Key observations: - pprof goroutine profile shows most goroutines stuck in net/http.(*persistConn).readLoop - Graceful shutdown with http.Server.Shutdown() hangs because of these - Increasing MaxIdleConnsPerHost from 2 to 100 helped slightly but didn't solve it - The service makes outbound calls to a gRPC backend — could connection pooling be the issue? Has anyone tracked this down to a specific pattern? I'm wondering if it's idle connections not being reaped, or if there's a context leak in our middleware chain. Tech: Go 1.22, standard library net/http, gRPC 1.62, running on Kubernetes with istio sidecar.