Description
A device on air can become permanently unreachable while the process keeps running: the relay no longer knows the device (its URL returns 404), but the socket.io client still reports connected == True, so the reconnect timer in Air.__init__ (timer(5, self.connect)) never acts — connect() returns early on if self.relay.connected: return. The only remedy is restarting the process.
Observed twice within days on the same machine (Linux, air-link 0.5.0, NiceGUI 2.24.2, python-socketio 5.16.3, python-engineio 4.13.3, websocket transport), both times right after a network change (NetworkManager event, Wi-Fi/DHCP). The logic is unchanged on current main, so 3.x should be affected as well.
Two log signatures of the stuck state:
connected. arrives but the ready event never does — no NiceGUI is on air at ... line, no further log output, and the relay answers 404 from then on:
DEBUG:nicegui.air:Going to connect...
DEBUG:nicegui.air:Connecting...
DEBUG:nicegui.air:connected.
(nothing ever again)
- A reconnect loop where the transport dies ~15 s after every handshake, which eventually goes silent — again with
relay.connected stuck at True and no open TCP connection to the relay (verified via the process's socket list):
DEBUG:nicegui.air:connected.
NiceGUI is on air at https://europe.on-air.io/dennis/tp/
ERROR:engineio.client:packet queue is empty, aborting
DEBUG:nicegui.air:disconnected.
DEBUG:nicegui.air:Going to connect...
... (repeats every ~20 s, then stops with the client believing it is connected)
Root cause
air.py trusts self.relay.connected as the sole signal for "the tunnel works". That flag is maintained by python-socketio and can stay True when the engineio transport dies without a clean disconnect event (the packet queue is empty, aborting path). There is no application-level liveness check, so the 5-second keep-alive timer becomes a no-op exactly when it is needed.
Suggested solution
Two complementary pieces, both in Air:
- Cheap consistency check every timer tick:
self.relay.connected can disagree with the actual engineio state. Reconciling them makes the existing timer effective again in the crashed-transport case:
async def connect(self) -> None:
if self.connecting:
return
if self.relay.connected:
if self.relay.eio.state == 'connected':
return
self.log.warning('socket.io claims connected but engine.io is not; resetting connection')
await self.disconnect()
...
- End-to-end check as the robust guarantee (also covers the "connected but
ready never arrived / relay forgot the device" case, where the socket may look healthy): every Nth timer tick, fetch self.remote_url over plain HTTPS; if the relay answers 404 while the client claims to be connected (twice in a row, to avoid acting on transients), call self.disconnect() so the timer reconnects. Deliberately do nothing when the relay is unreachable — that is an internet outage, and reconnecting would not help.
Item 2 is implemented downstream in air-link as a workaround: zauberzeug/air-link#32 (air_link/relay_check.py) — feel free to lift it into air.py.
It may also be worth reporting the stale connected flag to python-socketio, since a transport death without a disconnect event looks like a bug there too; but NiceGUI should be defensive regardless, because for on-air devices "believes it is connected forever" means permanently losing remote access.
Description
A device on air can become permanently unreachable while the process keeps running: the relay no longer knows the device (its URL returns 404), but the socket.io client still reports
connected == True, so the reconnect timer inAir.__init__(timer(5, self.connect)) never acts —connect()returns early onif self.relay.connected: return. The only remedy is restarting the process.Observed twice within days on the same machine (Linux, air-link 0.5.0, NiceGUI 2.24.2, python-socketio 5.16.3, python-engineio 4.13.3, websocket transport), both times right after a network change (NetworkManager event, Wi-Fi/DHCP). The logic is unchanged on current
main, so 3.x should be affected as well.Two log signatures of the stuck state:
connected.arrives but thereadyevent never does — noNiceGUI is on air at ...line, no further log output, and the relay answers 404 from then on:relay.connectedstuck atTrueand no open TCP connection to the relay (verified via the process's socket list):Root cause
air.pytrustsself.relay.connectedas the sole signal for "the tunnel works". That flag is maintained by python-socketio and can stayTruewhen the engineio transport dies without a clean disconnect event (thepacket queue is empty, abortingpath). There is no application-level liveness check, so the 5-second keep-alive timer becomes a no-op exactly when it is needed.Suggested solution
Two complementary pieces, both in
Air:self.relay.connectedcan disagree with the actual engineio state. Reconciling them makes the existing timer effective again in the crashed-transport case:readynever arrived / relay forgot the device" case, where the socket may look healthy): every Nth timer tick, fetchself.remote_urlover plain HTTPS; if the relay answers 404 while the client claims to be connected (twice in a row, to avoid acting on transients), callself.disconnect()so the timer reconnects. Deliberately do nothing when the relay is unreachable — that is an internet outage, and reconnecting would not help.Item 2 is implemented downstream in air-link as a workaround: zauberzeug/air-link#32 (
air_link/relay_check.py) — feel free to lift it intoair.py.It may also be worth reporting the stale
connectedflag to python-socketio, since a transport death without a disconnect event looks like a bug there too; but NiceGUI should be defensive regardless, because for on-air devices "believes it is connected forever" means permanently losing remote access.