Lost connection to MySQL server during query

Hello!

I want to launch a python fastapi application. But I get an error: Lost connection to MySQL server during query.

This error is only on the Koyeb. Previously, redeploy helped solve this problem. I simply redeployed 3-5 times in a row and my API finally started. But now the redeploy has stopped helping.

This error appears only during redeploy. When I release a new version of the API.

And the error is not in my code. On other servers the code starts and works. Maybe I need to change the version of Python? Or something else?

This is logs:

Instance allocated in nomad
Instance is unhealthy
INFO:     Started server process [12]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 657, in _read_bytes
    data = await self._reader.readexactly(num_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/asyncio/streams.py", line 747, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 4 expected bytes

The above exception was the direct cause of the following exception:


Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.12/site-packages/starlette/routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/app/.heroku/python/lib/python3.12/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/app/.heroku/python/lib/python3.12/site-packages/starlette/routing.py", line 654, in startup
    await handler()
  File "/workspace/utils/database.py", line 15, in startup_event
    await database.connect()
  File "/app/.heroku/python/lib/python3.12/site-packages/databases/core.py", line 112, in connect
    await self._backend.connect()
  File "/app/.heroku/python/lib/python3.12/site-packages/databases/backends/mysql.py", line 69, in connect
    self._pool = await aiomysql.create_pool(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/pool.py", line 29, in _create_pool
    await pool._fill_free_pool(False)
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/pool.py", line 182, in _fill_free_pool
    conn = await connect(echo=self._echo, loop=self._loop,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 75, in _connect
    await conn._connect()
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 539, in _connect
    await self._get_server_information()
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 1048, in _get_server_information
    packet = await self._read_packet()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 609, in _read_packet
    packet_header = await self._read_bytes(4)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 661, in _read_bytes
    raise OperationalError(CR.CR_SERVER_LOST, msg) from e
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

ERROR:    Application startup failed. Exiting.
Instance is unhealthy
Instance is unhealthy
INFO:     Started server process [12]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 657, in _read_bytes
    data = await self._reader.readexactly(num_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/asyncio/streams.py", line 747, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 4 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.12/site-packages/starlette/routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/app/.heroku/python/lib/python3.12/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/app/.heroku/python/lib/python3.12/site-packages/starlette/routing.py", line 654, in startup
    await handler()
  File "/workspace/utils/database.py", line 15, in startup_event
    await database.connect()
  File "/app/.heroku/python/lib/python3.12/site-packages/databases/core.py", line 112, in connect
    await self._backend.connect()
  File "/app/.heroku/python/lib/python3.12/site-packages/databases/backends/mysql.py", line 69, in connect
    self._pool = await aiomysql.create_pool(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/pool.py", line 29, in _create_pool
    await pool._fill_free_pool(False)
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/pool.py", line 182, in _fill_free_pool
    conn = await connect(echo=self._echo, loop=self._loop,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 75, in _connect
    await conn._connect()
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 539, in _connect
    await self._get_server_information()
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 1048, in _get_server_information
    packet = await self._read_packet()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 609, in _read_packet
    packet_header = await self._read_bytes(4)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/aiomysql/connection.py", line 661, in _read_bytes
    raise OperationalError(CR.CR_SERVER_LOST, msg) from e
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

ERROR:    Application startup failed. Exiting.
Instance is unhealthy
Instance terminated in nomad

Thanks for the report! To make sure I understand correctly, the connection to a remote MySQL server fails and this only happens at startup?

Yes. Only at Health check. And there are no errors in the code. Previously, several redeploys in a row helped, but now even this has stopped helping.