Skip to content

fix: bound connection acquisition with queryTimeout to prevent handshake hangs#33

Draft
raprav wants to merge 1 commit into
runnerty:mainfrom
raprav:fix/connection-acquire-timeout
Draft

fix: bound connection acquisition with queryTimeout to prevent handshake hangs#33
raprav wants to merge 1 commit into
runnerty:mainfrom
raprav:fix/connection-acquire-timeout

Conversation

@raprav

@raprav raprav commented Jul 3, 2026

Copy link
Copy Markdown
Member

🇬🇧 English

Problem

queryTimeout (#30) turned out to be necessary but not sufficient. In production (AWS RDS MySQL 8.0, Runnerty chains on cron) we hit two more permanent hangs with the 3.3.0 fix deployed and verified in the running image, and the timeout never fired.

The reason is that a MySQL operation has two phases, and #30 only covers the second one:

  1. Connection acquisition (connect + handshake). mysql2 disarms connectTimeout as soon as the first byte arrives from the server (lib/base/connection.js, stream.on('data')). If the server stalls halfway through the handshake — in our case it emitted an out-of-sequence auth packet ~12–24 s into the connection — the client logs Warning: got packets out of order. Expected 2 but received 1, keeps waiting for an auth response that never comes, and pool.getConnection() never calls back. No timer covers this window.
  2. Query execution. mysql2 arms the query timeout inside Query.start(), which only runs once a fully-handshaken connection executes the command. If phase 1 never completes, this timer never arms.

Observable evidence from 14 days of production logs: 2 occurrences of the packets out of order warning, each one immediately followed by a permanent chain hang (until host restart), and zero occurrences of PROTOCOL_SEQUENCE_TIMEOUT — the phase-2 timer never got the chance to arm.

Fix

Reuse the same queryTimeout deadline to bound phase 1: before dispatching the query, acquireConnection() proves the pool can deliver a healthy connection within the deadline (then releases it back to the per-execution pool, where pool.query() reuses it — measured overhead: negligible). On expiry it destroys every connection held by the pool (a connection stuck mid-handshake is checked out, so pool.end() alone would never close it) and rejects, so the process always ends and the chain is never blocked.

No new configuration: queryTimeout now bounds each phase independently. Fully backwards compatible — without queryTimeout nothing changes.

Reproduction & validation

test/acquire-timeout-repro.js (manual script, not wired to npm test) reproduces the production failure byte-for-byte: real MySQL 8.0 greeting → server ignores the client HandshakeResponse → desynced caching_sha2 full-auth packet (seq 1 while the client expects 2) → silence.

Scenario Before (3.3.0) After
Handshake stall (repro script) ❌ hang forever ✅ settles with error at queryTimeout (3003 ms)
SELECT 1 + queryTimeout vs real MySQL 8.0 ✅ 24 ms ✅ 24 ms
SELECT SLEEP(60) + queryTimeout: 3000 vs real MySQL 8.0 ✅ error at ~3 s ✅ error at ~3 s (native query timer intact)
SELECT 1 without queryTimeout ✅ (gate disabled)

🇪🇸 Español

Problema

El queryTimeout (#30) resultó necesario pero no suficiente. En producción (AWS RDS MySQL 8.0, cadenas Runnerty bajo cron) sufrimos dos cuelgues permanentes más con el fix de la 3.3.0 desplegado y verificado en la imagen corriendo, y el timeout nunca disparó.

La razón es que una operación MySQL tiene dos fases, y #30 solo cubre la segunda:

  1. Adquisición de conexión (connect + handshake). mysql2 desarma el connectTimeout en cuanto llega el primer byte del servidor (lib/base/connection.js, stream.on('data')). Si el servidor se atasca a mitad de handshake — en nuestro caso emitió un paquete de auth fuera de secuencia a los ~12–24 s — el cliente loguea Warning: got packets out of order. Expected 2 but received 1, se queda esperando una respuesta de auth que nunca llega, y pool.getConnection() no retorna jamás. Ningún timer cubre esta ventana.
  2. Ejecución de la query. mysql2 arma el timeout de query dentro de Query.start(), que solo corre cuando una conexión con handshake completo ejecuta el comando. Si la fase 1 no termina, este timer nunca se arma.

Evidencia observable en 14 días de logs de producción: 2 apariciones del warning packets out of order, cada una seguida inmediatamente de un cuelgue permanente de la cadena (hasta reiniciar el host), y cero apariciones de PROTOCOL_SEQUENCE_TIMEOUT — el timer de la fase 2 nunca llegó a armarse.

Fix

Reutilizamos el mismo deadline de queryTimeout para acotar la fase 1: antes de despachar la query, acquireConnection() comprueba que el pool puede entregar una conexión sana dentro del plazo (y la devuelve al pool por-ejecución, donde pool.query() la reutiliza — overhead medido: despreciable). Si expira, destruye todas las conexiones del pool (una conexión a medio handshake está checked-out, así que pool.end() por sí solo nunca la cerraría) y rechaza, de modo que el proceso siempre termina y la cadena nunca queda bloqueada.

Sin configuración nueva: queryTimeout ahora acota cada fase de forma independiente. Totalmente retrocompatible — sin queryTimeout no cambia nada.

Reproducción y validación

test/acquire-timeout-repro.js (script manual, no cableado a npm test) reproduce el fallo de producción byte a byte: greeting real de MySQL 8.0 → el servidor ignora la HandshakeResponse del cliente → paquete full-auth de caching_sha2 desincronizado (seq 1 cuando el cliente espera 2) → silencio.

Escenario Antes (3.3.0) Después
Stall de handshake (script de repro) ❌ cuelgue eterno ✅ settlea con error al queryTimeout (3003 ms)
SELECT 1 + queryTimeout contra MySQL 8.0 real ✅ 24 ms ✅ 24 ms
SELECT SLEEP(60) + queryTimeout: 3000 contra MySQL 8.0 real ✅ error a ~3 s ✅ error a ~3 s (timer nativo de query intacto)
SELECT 1 sin queryTimeout ✅ (gate desactivado)

@raprav raprav marked this pull request as draft July 3, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant