Deployment failing without changing anything

Troubleshooting and help
1

Hey, I had posted here in the past when some of my deployments were not working. I did fix the issue by making my schedulers run after that application startup.

Since then, nothing has happened - my deployment failed. But today I’m trying to deploy the same app that I have running right now on a large instance while changing only one environment variable, and I’m failing to deploy it again.

Can someone from the team please look into this with me? My stack is Python FastAPI.

2

Can someone from the team please look into my account and inform me with the issue? I’ve changed nothing in any of the project’s configuration (or code), and I’m already on the large instance. I want to deploy a change ASAP.

For you reference I am getting this in my deployment logs:

Instance created. Preparing to start...
Instance is starting... Waiting for health checks to pass.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started parent process [1]
TCP health check failed on port 8000.
INFO:     Received SIGTERM, exiting.
INFO:     Terminated child process [17]
INFO:     Terminated child process [18]

Also, I tried to do my last deployment in Frankfurt (not my usual Singapore) because the latency was extremely high in Singapore, so I thought this could be the issue. But it wasn’t the issue. : )

3

Hi @Ahmad_Elsaeed,

I’ve checked your app. I see that it logs INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) but it actually doesn’t listen on the port yet.
From what I see, during the start, it uses 100% CPU and almost 100% memory.

Could you share your code so we could take a look?

4

Yes, sure. Can you give me a GitHub account to add to the repo?

5

my gh accountid: lukaszo

6

I just added you to my repo! Lmk if I can help on anything.

7

Hey @Lukasz_Oles , any updates?

8

I don’t see anything obvious. Let me play with it a little. I will come back to you tomorrow morning.

9

I remember that I initially had something wrong with the schedulers. They were trying to run before the application startup completed, but I changed that and then it worked.

10

I’m afraid that you were just lucky last time.
I’ve set the max values in settings for port health checks settings and after 30m it finally started.

I was able to reproduce the issue with simple code:

@app.on_event("startup")
def handle_reminders():
    time.sleep(600)

Basically, something running in a function decorated with @app.on_event("startup") takes really long time to finish. According to docs the code is run “before” the app starts to serve the requests.

11

Yes, I understand that is happening. However, it’s weird to me since none of the functions in the startup take this long to run.