While benchmarking the socket backend I found that the GSO send path never executes, and that making it execute breaks delivery. Two separate things, reporting both. No patch proposed — the second half depends on the receive path.
Environment: Linux 6.8, OTP 28.5.0.1, socket_backend => socket on client and listener, quic_socket:detect_capabilities/0 reports #{backend => socket, gso => true, gro => true}.
1. GSO never fires
flush/1 only takes the GSO path when gso_batch_uniform(Buffer, GSO) holds, which requires every packet except the last to be exactly gso_size. gso_size defaults to ?DEFAULT_GSO_SEGMENT_SIZE (1200), but 1-RTT packets are sized from the current max datagram size and come out at 1398 bytes.
Instrumented gso_batch_uniform/2 over a 10 MB single-stream transfer, logging {Result, GSO, FirstPacketSize}:
{false, 1200, 1398} => 108
Never true, so every flush falls through to flush_individual/1. Counting flushes by batch size over the same transfer:
individual_flushes=3939 packets=6214 batch_size_1=3898
So ~1.6 packets per syscall, and flush_gso/1 is dead code in practice. The batching machinery itself works — add_to_batch/2 accumulates correctly up to max_batch_packets, and a minority of flushes do carry ~56 packets — it is only the uniformity comparison against the fixed 1200 that fails.
Related: the preemptive flush before an ACK-only packet in quic_connection (the comment referencing gso_batch_uniform/2) is protecting a path that never runs today.
2. Enabling it breaks the connection
I tried the obvious change locally: derive the segment size from the batch (all packets except the last must share a size, use that as UDP_SEGMENT) instead of comparing to gso_size, and split writes so one sendmsg stays under the 64 KB GSO payload cap.
It compiles clean and the GSO path then runs — but the connection no longer completes a transfer. Even 1 MB fails:
and larger transfers die mid-flight with {error, {invalid_state, draining}}. Reverting restores ~26 MB/s on loopback. I did not chase this further because the likely interaction is on the receive side (whether the UDP_GRO cmsg is delivered and split_gro_packets/2 applied on the active-mode paths, and how loopback segmentation behaves), which you know far better than I do.
Why it matters
Per-packet sends put a hard ceiling on throughput on any real path. For calibration, a raw gen_udp sender through a 20 ms netem qdisc on loopback delivers ~6,300 pps (~8.1 MB/s) end-to-end at 1398-byte datagrams; erlang_quic reaches ~4 MB/s there, i.e. it is close to the per-packet ceiling of the path rather than to what coalescing would allow. A C stack using GSO on the same path is not bound by that ceiling because the kernel sees a few large writes instead of thousands of small ones.
Happy to retest any change you make, or to hand over the instrumentation patches (batch-size histogram, uniformity probe, per-gate blocked counters) if useful — they were about 20 lines and I threw them away to keep the branch clean.
While benchmarking the socket backend I found that the GSO send path never executes, and that making it execute breaks delivery. Two separate things, reporting both. No patch proposed — the second half depends on the receive path.
Environment: Linux 6.8, OTP 28.5.0.1,
socket_backend => socketon client and listener,quic_socket:detect_capabilities/0reports#{backend => socket, gso => true, gro => true}.1. GSO never fires
flush/1only takes the GSO path whengso_batch_uniform(Buffer, GSO)holds, which requires every packet except the last to be exactlygso_size.gso_sizedefaults to?DEFAULT_GSO_SEGMENT_SIZE(1200), but 1-RTT packets are sized from the current max datagram size and come out at 1398 bytes.Instrumented
gso_batch_uniform/2over a 10 MB single-stream transfer, logging{Result, GSO, FirstPacketSize}:Never true, so every flush falls through to
flush_individual/1. Counting flushes by batch size over the same transfer:So ~1.6 packets per syscall, and
flush_gso/1is dead code in practice. The batching machinery itself works —add_to_batch/2accumulates correctly up tomax_batch_packets, and a minority of flushes do carry ~56 packets — it is only the uniformity comparison against the fixed 1200 that fails.Related: the preemptive flush before an ACK-only packet in
quic_connection(the comment referencinggso_batch_uniform/2) is protecting a path that never runs today.2. Enabling it breaks the connection
I tried the obvious change locally: derive the segment size from the batch (all packets except the last must share a size, use that as
UDP_SEGMENT) instead of comparing togso_size, and split writes so onesendmsgstays under the 64 KB GSO payload cap.It compiles clean and the GSO path then runs — but the connection no longer completes a transfer. Even 1 MB fails:
and larger transfers die mid-flight with
{error, {invalid_state, draining}}. Reverting restores ~26 MB/s on loopback. I did not chase this further because the likely interaction is on the receive side (whether theUDP_GROcmsg is delivered andsplit_gro_packets/2applied on the active-mode paths, and how loopback segmentation behaves), which you know far better than I do.Why it matters
Per-packet sends put a hard ceiling on throughput on any real path. For calibration, a raw
gen_udpsender through a 20 msnetemqdisc on loopback delivers ~6,300 pps (~8.1 MB/s) end-to-end at 1398-byte datagrams; erlang_quic reaches ~4 MB/s there, i.e. it is close to the per-packet ceiling of the path rather than to what coalescing would allow. A C stack using GSO on the same path is not bound by that ceiling because the kernel sees a few large writes instead of thousands of small ones.Happy to retest any change you make, or to hand over the instrumentation patches (batch-size histogram, uniformity probe, per-gate blocked counters) if useful — they were about 20 lines and I threw them away to keep the branch clean.