Websockets disconnecting after a minute or so

Troubleshooting and help
1

Love Koyeb! Except for one issue I’m having with websocket connections dropped after a minute or so, when the websocket is idle.

App is Go web server that serves http but also serves websockets. An http client connects to the server and the client opens a websocket (using js) back to the server. All is fine for a minute or two, when the websocket is rudely disconnected. The client js will reconnect the websocket, but it’s quite disruptive to be disconnected.

I tried on other platforms (Linode, GCP) and I don’t get this behaviour.

Is there a way to disable the disconnection of idle websocket connections?

2

Hi @Scott_Feldman,

This is expected behaviour because the idle timeout is low. Application is required to handle it by restarting connection or ping-pong/keepalive strategy.

We are considering adding an option allowing users to customize the HTTP timeout:

3

Hey @Scott_Feldman :wave:

To add more context, we cannot completely disable the disconnection of idle WebSocket connections as this would put a lot of pressure on the load-balancing layer with ghost connections consuming resources.

I would be surprised if Linode and GCP have no idle timeout on their load-balancing layer, are you referring to VMs?

4

Thanks for the reply. I did work around the issue by adding a ping-pong strategy to keep the websocket connection active. Is it documented somewhere that the default timeout is 120s?

5

Oh, yes, was referring to VMs when comparing to GCP and linode. I see that’s not a fair comparison. Sorry about that.

6

I fail to get websocket working at all. I deployed a simple websocket echo server at

https://websocket-echo-intustack.koyeb.app

But I can only access it over https. Trying websocket fails. While if I run the same container locally, client can open websocket properly.

UPDATE: Actually ws://websocket-echo-intustack.koyeb.app works but not wss://…
Perhaps that can be fixed?

7

Hi Aleksaandar, I’m using wss:// with Koyeb. Wonder why it’s not working for you? My app is Go app using golang.org/x/net/websocket package.

8 
# websocat -v ws://websocket-echo-intustack.koyeb.app
[INFO  websocat::lints] Auto-inserting the line mode
[INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
[INFO  websocat::ws_client_peer] get_ws_client_peer
[INFO  websocat::ws_client_peer] Connected to ws
WebSocket-Echo-IO server powered by Iodine 0.7.55.
[INFO  websocat::ws_peer] Received WebSocket ping
[INFO  websocat::ws_peer] Received WebSocket ping
asdasdadfsfd
asdasdadfsfd
[INFO  websocat::ws_peer] Received WebSocket ping
Goodbye!
Goodbye!
[INFO  websocat::ws_peer] Received WebSocket close message
See you in a bit.
[INFO  websocat::sessionserve] Reverse finished

With wss:

# websocat -v wss://websocket-echo-intustack.koyeb.app
[INFO  websocat::lints] Auto-inserting the line mode
[INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
[INFO  websocat::ws_client_peer] get_ws_client_peer
Segmentation fault (core dumped)

I’m not sure what’s going on.

9

Ok, please ignore, tried another websocat build and it works both with ws:// and wss://

Sorry for the noise, apparently I had faulty client build somehow.

10

I’m having trouble again with websockets disconnecting even with the client pinging the websocket every 1 second. The websocket connections are dropped anywhere from 10secs to 5mins. I’ve tried different clients over different network paths, all showing the same result. I believe the disconnect is happening at the (Koyeb) server side as independent clients connecting over LTE, 5G, wifi (starlink), etc. The disconnects are coordinated…all clients see the same disconnect event at the same time.

I thought this was working for me back in July once I added a ping on each websocket. But something seems to have changed on the Koyeb side?

11

same here, websockets are disconnecting after ~30 seconds, even if in use

12

Hi guys,

We’re noticing the same issue and are looking into it.

Sorry for the inconvenience

13

We shipped a fix, it should not timeout that often again.

Please let us know if you encounter other issues!

14

@nicolas Thanks for looking into this. I tried it on my service and it seems OK now.

Is there an external visible bug report/fix for this issue? Is there a production test in-place to test for this (persistent connections) in future builds? What is the default idle timeout? Is that value documented somewhere? I’d like to set my ping time period to be just under the default time-out to minimize traffic.

15

Hey @Scott_Feldman

Glad to know that this fixes it for you.

Is there an external visible bug report/fix for this issue?

This is a regression that has been introduced a few weeks ago. I think that this is the right place to publicly discuss the topic.
If you are interested in the technical details of what causes this issue, let me know, I’ll gladly explain, it’s pretty interesting :grin:

Is there a production test in-place to test for this (persistent connections) in future builds?

Now we do :slightly_smiling_face:. Hopefully we will be more reactive in case it ever happens again

What is the default idle timeout? Is that value documented somewhere?

It’s 100 seconds. We have planned on documenting a more complete overview of the various timeouts in place in our reference docs, but we’re not there yet