Debugging Intermittent Kong 503s When the Logs Showed Nothing but the Status Code

0:00This audio is presented by Hacker Noon, where anyone can learn anything about any technology. Debugging Intermittent Kong 503s when the logs showed nothing but the status code, by Vonsee Krishna Goode. AN investigation into intermittent gateway failures, engine ex-worker restarts, and THE metrics that exposed THE pattern and API gateway failure does not always begin with an unavailable upstream service. Sometimes the gateway itself loses the ability to select a target, and THE resulting error makes the upstream look guilty. We use Kong gateway as part of our product. Occasionally, users opening the UI received an HTTP 503 response. The message was brief, intermittent, and easy to misread. On the server side, we did not see a matching Kong error entry. The request log showed only the 503 status. We also found no issues with the upstream services or Kubernetes DNS look UPS. The more useful clue was the timing around engine ex-worker restarts inside Kong.

1:01Correlated request status, system logs, and Kong's worker LuAVM memory metroshelptus move from, the UI is sometimes broken, to a concrete operational hypothesis. What the Ring balancer is doing Kong's request path is roughly like this. The Ring balancer is responsible for choosing a target for the upstream. Kong-John have multiple targets behind one upstream, and health check state determines which of them are eligible to receive traffic. When Kong cannot get a peer from the Ring balancer, the upstream application is not automatically the problem. Kong may not have had a usable target to select at that moment. Some ordinary causes include all targets being marked unhealthy, incorrect target address or port, DNS or network connectivity problems, health check or circuit breaker excluding targets. Kong worker restarting while it is handling traffic, resource pressure or a crash affecting the gateway process. The error points to the peer selection stage. It does not identify the cause by itself. Why the failure WAS intermittent the failure did not affect every request.

2:05A user could refresh the page and get a successful response immediately after seeing the 503. That behavior made sense once we considered two engine ex-worker processes in the Kong pod. If one worker restarts while the other remains available, most traffic can continue normally. Requests arriving during the worker transition requests assigned to the affected worker may fail while the rest of the pod appears healthy. This also explains why a simple pod level health check was not enough. Kong pod could still be running, and other worker could still be serving requests. Will a small number of requests failed. How we diagnosed IT before treating engine ex-worker restart as the root cause, we checked ordinary ring balancer conditions. For the affected upstream, we verified that, upstream existed. Targets were configured, target addresses and ports were correct. Health checks had not marked every target unhealthy. Kong pod could resolve and reach the back end, and we didn't see any upstream issues. There was no useful Kong error log fourth request. The access log showed a 503 with an empty

3:08upstream address. While checking Kong pod metrics, we observed that lieu of VM metrics showed multiple PIDs. The metrics exposed multiple worker PIDs. Comparing them over time helped US identify worker churn. The DMESG output then confirmed that the engine ex-process had been killed by the pod's memory C group. DMESG log. Once we put the signals on one timeline, the investigation became much less speculative. The 503 was the client visible symptom. Engine ex-worker restart was the process level event. The memory metric and back-quote DMESG back-quote output helped us test whether resource pressure was involved. The target checks made sure we were not blaming the worker for a normal upstream health failure. That distinction mattered because there was no single error log that tied everything together for us. What we changed in our debugging APPRO ACH the main lesson was to stop treating the HTTP status code as a root cause message. A503 tells us that the request was not successfully served. It does not tell US whether the failure

4:13happened during routing, peer selection, connection establishment, or another gateway stage. Likewise, a quiet error log does not mean there was no internal failure. It may mean that the configured log stream captured only the access result. For this class of problem, we now want the following information available together. Access logs with root, upstream, status, and request ID. Worker start, exit, and respawn events. Pod restart counts and termination reasons. Node or container oom evidence. Worker lieu of VM memory by PID. Upstream target health transitions, connectivity from the Kong pod to the back end. We also avoid treating a high lieu of VM memory value as a diagnosis. It is a signal to correlate with traffic, plugins, configured limits, restarts, and kernel evidence. A compact runbook when the next intermittent 503 occurs, the investigation can follow this order. Record the timestamp, root, upstream, request ID, and Kong pod. Check whether the upstream has at least one healthy and reachable target.

5:17Check whether either of the two engine x worker processes exited or respawned. Check pod, container, cubelet, runtime, and kernel termination evidence. Compare by PID with pod memory and restart timing. Classify the event as an upstream health issue, oom event, crash, reload, or another runtime problem. The important part is not to jump from 503, directly to, increase memory. First establish whether the gateway had no healthy peer, whether a worker was restarting, and whether memory pressure was actually involved. Final takeaway our Kong pod had two engine x worker processes, and the intermittent nature of the 503s made worker level investigation worthwhile. We did not get a detailed error log for the failed request. We got a status code, a browser facing ring balancer message, worker lifecycle clues, and metrics. Together, those signals gave us a much better way to investigate than looking at the upstream service alone. When Kong returns a ring balancer related 503, check the targets,

6:18but also check the worker that handled the request. References Kong Gateway, health checks and circuit breakers. Kong Prometheus Plug-in Documentation, Kong Gateway PDK. Thank you for listening to this Hackernoon story, read by artificial intelligence. Visit Hackernoon.com to read, write, learn, and publish.