502 errors after multi requests

I’ve been observing this problem for a week or two…

My service is returning 502 errors. Initially, the service starts and everything is fine. If I have a single browser tab open on the service, everything is fine. If I duplicate the browser tab a couple of times, requests for assets start returning 502. After a few tries, the service just returns the stock 502 “I give up” page. After a minute, the service returns to normal, but quickly falls over again if I hit it with multiple browser requests.

From the Koyeb console, the service remains healthy all the time. The service logs don’t show anything unusual. The service logs each request, and I see logs for the successful requests, but the request that return 502 to the browser are not logged by the service. I believe the service is not seeing those requests at all and something upstream is returning 502.

To troubleshoot, I tried running the same service on GCP and it works perfectly there…no 502 ever. I tried running the docker image locally, and everything works fine, no 502s.

My service serves up static web pages and websockets.

Hello,

Thanks for your report, we are investigating why you are seeing these errors, which seems to be an anomaly.
We cannot reproduce this behavior in an isolated setup, but we have been able to reproduce it on your service and we will be digging a bit more.

Bastien

After spending some time with the team we believe there might be some complex issues here.

We will contact you by email to get more information, on how we could reproduce it, but we don’t have a clear conclusion yet.

One common root cause of similar issue would be resource exhaustion, memory, CPU, file descriptor, …

After much investigation with @Scott_Feldman we managed to find the root cause of the issue with the team.

It was related to some websocket disconnections that were considered as a critical failure of the backend, resulting in removing the backend from the pool for a while.

This behavior should now be fixed.

Once again, thanks you for the report.

Hi Bastien! Thank you for the prompt fix. I did some brief testing and the problem seems fixed…no more 502s! Go Koyeb!

1 Like

Same problem for us! Many 502 errors after successful deploy and no errors into log…
Can you please contact me? The support chatbox doesn’t let me send any message