Service not responding (error 502) despite being healthy

Hey team, I’ve been using Koyeb for about three months now. My services have been completely normal. When I deploy them, they respond, at least when I go to the URL, I see “Hello World” or something.

However, on my recent deployment, when I go to the URL, there’s no response at all. And I haven’t been receiving any of the webhooks delivered to my server. When I go to app.heymojo.club, which is supposed to be the subdomain that my service is on… It keeps loading for about a minute and then gives me a 502 error (Service unavailable).

What is this, how can I fix it, and how can I make sure that it doesn’t happen again?

For reference, this is what I see in the logs:
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Instance is healthy. All health checks are passing.

Hey!

I’m looking into it.

hey @nicolas , appreciate it :slight_smile: Do let me know if there’s anything that I can do from my side.

I’ve already:

  1. Tested locally
  2. Made sure the binding is happening with the correct port
  3. Made sure that public traffic is allowed

Anything else?

I think it would be helpful if you could enable debug logs, or maybe even print something whenever a request is handled on port 8000.

From what I was able to see, requests flow well through the public load balancing stack.

When the debug logs are activated, could try to run the following on the Console: curl -vvv 127.0.0.1:8000?

I just added --log-level debug to the run command. Becomes:

uvicorn main:app --log-level debug --host 0.0.0.0

I’m adding logging right now to when an endpoint is accessed.

I’ll test as soon as the build is done.

Hey @nicolas ,

enabled logging, and added logging to my endpoint.

Went again to app.heymojo.club/, nothing in the logs on Koyeb.

Ran curl -vvv 127.0.0.1:8000, the request was sent but no response was received…

What next?

@nicolas just got 2 GET requests to my app, and it responded with 200.

I’m assuming you’re troubleshooting something. Still, I’m trying to access app.heymojo.club, and it’s not responding.

Yes, I was sending a few queries :slightly_smiling_face:.

The issue might come from FastApi tasks, that you mention here: Health check failing for application startup. They could be blocking the event loop. Could you be running into something like this?

This could be a good chance.

It’s a mistake from me that I have all tasks async to be honest. They don’t need to be async.

Let me try to make them synchronous and testing if this issue would be fixed.

I’m curious though… why have I never encountered this issue in the past? I had the same number of schedulers running for the past 3 months without any issues.

hey @nicolas, this issue is resolved.

So I removed the asynchronous schedulers and made them synchronous and that solved the issue.

This made the response time of the service WAY faster. Was that expected?

Also, does this mean I can go a tier lower and use a Medium size instance? I don’t even think my load would need the Large size instance.

@Ahmad_Elsaeed feel free to experiment with the instance types. Usually folks try with the smallest types and if necessary they go up.