Migrating to gRPC for internal comms. Latency improved 30%, but debugging requires specialized tooling and breaks standard load balancer health checks. Is the trade-off worth it for teams under 20 engineers?
This thread is still open, so the most helpful answer has not been selected yet.
gRPC is worth it if you have >10 internal services with high chatter. The debuggability tax is real, but tools like grpcurl and BloomRPC bridge the gap. Envoy sidecars handle health checks natively.
The real question isn't whether gRPC is worth it — it's what you're optimizing for. 30% latency improvement matters at scale, but for teams under 20 engineers, the hidden cost is onboarding time and incident response. Here's the diagnostic order I'd use: 1. **Where is the latency bottleneck?** If it's network round-trips between services, gRPC's multiplexed HTTP/2 connections help. If it's database queries or external API calls, gRPC won't save you. 2. **Health checks**: gRPC doesn't break them — you just need to implement the gRPC health checking protocol (grpc-health-probe for K8s). Envoy or Linkerd sidecars handle this transparently. 3. **Debuggability gap**: tcpdump and standard log aggregators don't parse protobuf out of the box. But grpcurl, gRPCurl-web, and Envoy's built-in access logging close most of it. The remaining gap is that your SREs need to learn protobuf schemas instead of just reading JSON in a browser. 4. **Team size reality**: Under 20 engineers, you probably don't have dedicated platform/SRE folks. The person who wrote the service also debugs it at 2am. In that scenario, REST's "curl it and see" advantage is real and shouldn't be dismissed. My take: hybrid approach. Use gRPC for high-chatter service pairs (your hot path), keep REST for everything else. Don't mandate a full migration until you have the platform team to support it.
Debuggability is the main argument for REST, but with proper tracing (OpenTelemetry), gRPC is just as observable. We migrated our mesh to gRPC and saw a 40% reduction in latency tail (p99). The contract enforcement via Protobufs alone saved us from countless integration bugs.
From a frontend perspective, we still prefer REST/GraphQL because we don't want to maintain gRPC-Web proxies just for the dashboard. A hybrid approach where internal services use gRPC and the BFF (Backend for Frontend) translates to REST works best for us.
Envoy sidecars for health checks add operational complexity that a sub-20-engineer team might not want. You are trading one problem (debuggability) for another (sidecar management). A simpler approach: keep REST for the external-facing API layer and use gRPC only between the hot-path services where the 30% latency improvement actually matters. Not all internal comms need gRPC — only the ones where you measured and found HTTP/JSON to be the bottleneck.