Telegram bot loses connection

Troubleshooting and help
,
1

I deployed a Telegram bot on Koyeb. Since it doesn’t listen to any port, regular health-checks fail, resulting in stopping my application. As adviced here, I modified code so that simple web-server runs when the application starts working.

The problem is that every 40-240 minutes an exeption emerges:

2022-12-13 00:17:54,367 (__init__.py:960 MainThread) ERROR - TeleBot: "Infinity polling exception: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))"
2022-12-13 00:17:54,368 (__init__.py:962 MainThread) ERROR - TeleBot: "Exception traceback:
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
chunked=chunked,
File "/app/.heroku/python/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/app/.heroku/python/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/app/.heroku/python/lib/python3.7/site-packages/urllib3/connection.py", line 424, in connect
tls_in_tls=tls_in_tls,
File "/app/.heroku/python/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
sock, context, tls_in_tls, server_hostname=server_hostname
File "/app/.heroku/python/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/app/.heroku/python/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/app/.heroku/python/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/app/.heroku/python/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

When I run this app on my local computer, the problem does not arise. (Python 3.7.9 is used on both local computer and in my application on Koyeb). On the other serverless platform for deployment I also didn’t have this problem. That platform doesn’t perform health-checks, hence I didn’t have to run a dummy server within my application. So may be my problem is somehow related to running a server? I can’t check this assumption, since if I run my app without running a server, it won’t work longer than 15 minutes due to health-check fails.

This is the code I added to run a server (my app is in Python):

# web_server.py
from os import environ

from http.server import BaseHTTPRequestHandler, HTTPServer

APP_HOST = ''
APP_PORT = int(environ.get('PORT'))


class GetHandler(BaseHTTPRequestHandler):
    def _set_headers(self):
        self.send_response(204)
        self.end_headers()

    def do_GET(self):
        self._set_headers()


def run_server(handler_class=GetHandler):
    server_address = (APP_HOST, APP_PORT)
    httpd = HTTPServer(server_address, handler_class)
    httpd.serve_forever()

I run my bot and server together like that:

# main.py
if __name__ == '__main__':
    threading.Thread(target=run_server).start()
    bot.infinity_polling()

Do you have any ideas on what’s the reason of loss of connection and how can I avoid it? Thank you in advance!

2

oi how did you fix this?

3

Hi @dsaperov Good news! The platform now natively supports deploying workers for those on the Starter plan, so you can remove that web-server. Let me know if you want any help getting started deploying workers.

Hi @franesco_ottev :wave: Is everything ok or are you facing an issue deploying a worker? Let me know, I am happy to help!

1 Like
4

Thanks for being here @alisdair !

Since I deployed my workers few weeks ago, I am continously getting the same error @dsaperov was facing, I keep getting “ConnectionResetError: [Errno 104] Connection reset by peer” on my telegram bots. I also deployed the same apps to other serverless platforms and this has never happened, at least not so frequently. I used to get this error about once a month, now it is showing at least 3 times every single day…

5

Hey @Franesco_Ottev,
A connection reset error is most likely related to a TCP idle connection timeout which seems to be lower on our platform compared to others. It has no impact on your application, which should handle it by creating a new connection. We are investigating internally if this timeout is configured too low and if we can increase it.

1 Like
6

Thank you so much @Sebastian, I deeply appreciate the help. How can I follow updates about this? Will you update us on this thread?

7

Yes, we will notify you here.

1 Like
8

Hey @Sebastian any updates? It’s been quite some time since my workers have been experiencing this problem and I need them to work for important projects, 1/4 of the workers is not working properly due to this issue.

Thank you for understanding :pray:

9

Hi @Franesco_Ottev

I’m sorry for the inconvenience. We are investigating the issue and will be coming back to you when we have more details to share.

11

Hi @Franesco_Ottev
I’m was trying to reproduce this but didn’t succeeded, do you have any specific options related to timeout ?

Can you share a bit more about your setup, library and settings you are using ?

12

Hello @bastien, thanks for being here. I don’t have any specific options related to timeout, I don’t think my telegram bot library has either honestly.

the telegram bot library I am using is pyTelegramBotAPI (GitHub - eternnoir/pyTelegramBotAPI: Python Telegram bot api.) and one of the bots which is experiencing this issue the most is an amazon-telegram bot.

Here is how it works: it uses python Amazon SDK to get the products list, and every 30 minutes (time.sleep(1800)), it sends one post to a telegram channel using the telegram library I just mentioned.

This works perfectly fine locally and in other serverless platforms, but on koyeb, often when the 30min wait hits, it returns the connection error: “ConnectionResetError: [Errno 104] Connection reset by peer”

If needed, I am more than glad to help you getting this figured out troubleshooting together here or in any fast messaging app, like Discord or Telegram.

13

Thanks !

After taking a quick look at pyTelegramBotAPI I see that they have a paragraph on how to handle connection reset errors from the telegram API.

Did you already try to configure this specific variable ?

How can I handle reocurring ConnectionResetErrors?

Bot instances that were idle for a long time might be rejected by the server when sending a message due to a timeout of the last used session. Add apihelper.SESSION_TIME_TO_LIVE = 5 * 60 to your initialisation to force recreation after 5 minutes without any activity.

1 Like
14

Sorry for the late reply, I didnt notice the notification. I will try this right now. I will keep you updated here

15

@bastien update: this does not work. it still returns the same error. This makes me think even more that the issue is coming from koyeb itself, as on other serverless platform it works seamlessly.

16

Thank you for testing it!

Connection reset errors are usually emitted by the remote server, this is pretty common and should be handled by the app.

I tried to reproduce using a golang telegram bot, but didn’t succeeded, I might also try to reproduce this later using the python project you linked.

This issue seems to be be common in pyTelegramBotAPI, it seems to be related to how the message are sent after a long timeout.

Maybe checking theses tickets could help.

It seems that the proposed solution is to retry based on a read timeout failure.

telebot.apihelper.READ_TIMEOUT = 5

def send_msg(text, id):
    try:
        bot.send_message(id, text)
    except Exception as e:
        logging.info(e)
        send_msg(text, id)

I hope this helps

17

Thanks @bastien . I tried setting threaded=False, I will see if this fixes it in the coming days.