HTTP/3 support via OpenSSL 3.5

[bneradt]

Overview

This now contains two commits based directly on master. The prerequisite
HTTP/2 fix from #13363 and H3 quiche/BoringSSL fixes and tests from #13213
have landed and are no longer part of this PR's commit stack.

First Commit: HTTP/3 via OpenSSL 3.5 + quiche

Fedora now ships OpenSSL 3.5 with the third-party QUIC TLS callback API, but
quiche still links against the older quictls/BoringSSL symbols. ATS therefore
could not use the system OpenSSL library for downstream HTTP/3 without
bringing in a different TLS stack.

This adds CMake detection for the OpenSSL callback API and provides a private
compatibility layer that maps quiche's legacy hooks onto
SSL_set_quic_tls_cbs. This requires static quiche in that mode so ATS
resolves the shim symbols locally and links the final binaries against the
system OpenSSL libraries.

This also relaxes verifier-only HTTP/3 AuTest gates that do not execute curl,
so those tests can run when ATS has QUIC support but the installed curl lacks
HTTP/3.

Second Commit: HTTP/3 via OpenSSL 3.5 QUIC

OpenSSL 3.5 can terminate QUIC connections directly, but ATS only had a
quiche-backed HTTP/3 listener. Operators who want to use the system OpenSSL
QUIC stack needed a separate downstream backend without changing the existing
quiche path or origin HTTP/3 scope.

This adds an optional ENABLE_OPENSSL_QUIC backend that uses OpenSSL's native
QUIC listener and stream APIs for downstream HTTP/3. This keeps the backend
mutually exclusive with quiche, exposes TS_HAS_OPENSSL_QUIC, and shares
ATS's existing HTTP/3 stream handling above the transport.

This also installs native-QUIC TLS callbacks for ALPN and SNI certificate
selection before ATS has a QUIC NetVC to bind. OpenSSL native QUIC does not
make a selected SSL_CTX certificate active via SSL_set_SSL_CTX alone, so
this applies the selected certificate, key, and chain to the connection SSL.

This also broadens client-side HTTP/3 tests to run with either backend and
hardens transaction cleanup when OpenSSL closes stream state before ATS
finishes teardown. This caches stream identifiers, declines listener-time
QUIC tickets until a NetVC is bound, and adds focused H3 lifecycle and
session-ticket coverage.

Testing

Validated in the Fedora 44 asfats3 container:

• Generic Fedora CI build with QUIC disabled.
• System OpenSSL 3.5 + static quiche build and HTTP/3 unit tests.
• Native OpenSSL 3.5 QUIC build and HTTP/3 unit tests.
• All ten h3* AuTests with both OpenSSL 3.5 backends.
• The complete 10.2.x backport stack with BoringSSL/quiche and native OpenSSL
  3.5. The BoringSSL run passes all nine applicable tests; the OpenSSL-only
  session-ticket test is skipped as designed.

Production Testing

This is running in docs for
https://docs.trafficserver.apache.org/ targets via Alt-Svc header rewrite
rules to negotiate h3 when supported.

[Copilot]

Pull request overview

This PR updates Apache Traffic Server’s HTTP/3 / QUIC support to work with upstream OpenSSL 3.5 by (1) adding a quiche-to-OpenSSL 3.5 QUIC TLS callback compatibility shim and (2) introducing an optional OpenSSL-native QUIC backend that is mutually exclusive with quiche. It also broadens and adds gold tests so core HTTP/3 scenarios can run across either QUIC backend where appropriate.

Changes:

• Add CMake feature detection + build options for OpenSSL 3.5 QUIC (native QUIC + QUIC TLS callbacks compatibility for quiche).
• Introduce an OpenSSL-native QUIC NetProcessor/NetVC/PacketHandler path and wire it into existing QUIC/HTTP3 plumbing.
• Expand/add gold tests and ATSReplay support for HTTP/3, plus minor HTTP/3 transaction / stream lifecycle fixes.

Reviewed changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/gold_tests/timeout/quic_no_activity_timeout.test.py	Generalize QUIC gating to TS_USE_QUIC and keep quiche-specific assertions conditional.
tests/gold_tests/timeout/active_timeout.test.py	Run HTTP/3 active-timeout curl path under TS_USE_QUIC (still requires curl http3).
tests/gold_tests/headers/via.test.py	Run HTTP/3 portion of Via header test under TS_USE_QUIC.
tests/gold_tests/h3/replays/h3_stream_lifetime.replay.yaml	New Proxy Verifier replay covering concurrent stream lifetime behavior.
tests/gold_tests/h3/replays/h3_sni.replay.yaml	Adjust SNI/authority values used by the HTTP/3 SNI replay.
tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml	New replay covering multi-connection HTTP/3 proxy verifier interop.
tests/gold_tests/h3/replays/h3_active_timeout.replay.yaml	New replay exercising HTTP/3 active-timeout cleanup behavior.
tests/gold_tests/h3/h3_stream_lifetime.test.py	New ATSReplay-based test for stream lifetime handling.
tests/gold_tests/h3/h3_sni_check.test.py	Update QUIC gating + cert material for the SNI behavior test.
tests/gold_tests/h3/h3_session_ticket.test.py	New OpenSSL QUIC s_client-based session ticket test (OpenSSL 3.5+).
tests/gold_tests/h3/h3_proxy_verifier.test.py	New ATSReplay-based test for proxy verifier client interop.
tests/gold_tests/h3/h3_curl.test.py	New curl-based HTTP/3 interop test (requires curl http3 + --http3-only).
tests/gold_tests/h3/h3_active_timeout.test.py	New verifier-based HTTP/3 active-timeout cleanup test.
tests/gold_tests/autest-site/ats_replay.test.ext	Extend ATSReplay to pass http3_ports when QUIC is enabled in replay config.
src/traffic_server/traffic_server.cc	Switch QUIC include gate to TS_USE_QUIC.
src/traffic_layout/info.cc	Expose TS_HAS_OPENSSL_QUIC in feature reporting.
src/proxy/http3/test/test_Http3FrameDispatcher.cc	Adjust dispatcher test expectations for incremental feed behavior.
src/proxy/http3/Http3Transaction.cc	Improve stream-closure, timeout, and deferred read/write event handling; add safe self-delete gating.
src/proxy/http3/Http3StreamDataVIOAdaptor.cc	Fix reader lifecycle and make finalize() idempotent.
src/proxy/http3/Http3Session.cc	Avoid iterator invalidation while iterating transactions on session close.
src/proxy/http3/Http3HeaderVIOAdaptor.cc	Notify transaction when header decode completes (to schedule pending reads).
src/proxy/http3/Http3Frame.cc	Add DATA frame readiness parsing based on available payload bytes.
src/proxy/http3/Http3App.cc	Inform transaction on QUIC stream close for cleanup.
src/iocore/net/SSLSessionTicket.cc	Avoid hard-failing ticket callback when OpenSSL QUIC listener parses tickets before NetVC binding.
src/iocore/net/QUICNetVConnection.cc	Refactor stream send/recv to use QUICStreamIO interface and add stream I/O wrappers.
src/iocore/net/QUICMultiCertConfigLoader.cc	Add OpenSSL QUIC-specific SNI/cert selection + ALPN/curves hooks and TLS1.3-only enforcement for QUIC.
src/iocore/net/quic/QUICStreamVCAdapter.cc	Split read vs consume semantics and improve write reenable behavior.
src/iocore/net/quic/QUICStreamManager.cc	Implement local uni/bidi stream id allocation.
src/iocore/net/quic/QUICStreamAdapter.cc	Add consume() API and move on_read() notification there.
src/iocore/net/quic/QUICStream.cc	Abstract stream I/O via QUICStreamIO, support partial sends with pending blocks, and add on-write notification.
src/iocore/net/quic/QUICGlobals.cc	Adjust session handling compilation gates for new QUIC backends.
src/iocore/net/quic/QUICConfig.cc	Add QUIC server SSL_CTX creation via OpenSSL native QUIC method; gate quiche-only config APIs.
src/iocore/net/quic/OpenSSLQuicCompat.cc	New compatibility shim exporting quiche-expected legacy QUIC TLS callback symbols via OpenSSL 3.5 QUIC TLS callbacks API.
src/iocore/net/quic/CMakeLists.txt	Make quiche linkage conditional and add static compat shim library when required.
src/iocore/net/P_QUICPacketHandler.h	Add OpenSSL QUIC includes and OpenSSL-specific handler members/methods under feature guards.
src/iocore/net/P_QUICNetVConnection.h	Add OpenSSL QUIC init/stream plumbing, introduce QUICStreamIO, and add feature guards for quiche-only fields.
src/iocore/net/P_QUICNetProcessor.h	Make quiche-only members conditional and adjust signatures for new handler creation paths.
src/iocore/net/OpenSSLQUICPacketHandler.cc	New OpenSSL QUIC listener-based inbound packet handler implementation.
src/iocore/net/OpenSSLQUICNetVConnection.cc	New OpenSSL QUIC-based NetVC implementing stream accept/read/write via OpenSSL QUIC stream APIs.
src/iocore/net/OpenSSLQUICNetProcessor.cc	New OpenSSL QUIC NetProcessor wiring for inbound QUIC accept.
src/iocore/net/CMakeLists.txt	Select OpenSSL QUIC vs quiche QUIC sources and link dependencies based on feature flags.
include/tscore/ink_config.h.cmake.in	Add TS_HAS_OPENSSL_QUIC feature macro.
include/proxy/http3/Http3Transaction.h	Add stream close notification, new read scheduling helper, close-state tracking, and safe delete gating.
include/proxy/http3/Http3StreamDataVIOAdaptor.h	Track reader and finalized state for safe finalize/destruction.
include/proxy/http3/Http3HeaderVIOAdaptor.h	Minor formatting cleanup.
include/proxy/http3/Http3Frame.h	Allow DATA frame to override parsing readiness behavior.
include/iocore/net/QUICMultiCertConfigLoader.h	Add OpenSSL QUIC hook overrides behind TS_HAS_OPENSSL_QUIC.
include/iocore/net/quic/QUICStreamVCAdapter.h	Add _consume() override hook.
include/iocore/net/quic/QUICStreamManager.h	Make stream-creation methods virtual with out-params; track next local stream ids.
include/iocore/net/quic/QUICStreamAdapter.h	Add consume() API and require _consume() implementations.
include/iocore/net/quic/QUICStream.h	Introduce QUICStreamIO interface and pending-send tracking for partial writes.
include/iocore/net/quic/QUICConnection.h	Add optional on_stream_updated() hook.
include/iocore/net/quic/QUICConfig.h	Gate quiche-only APIs and declare quic_new_server_ssl_ctx().
include/iocore/net/quic/Mock.h	Update mock stream adapter behavior to align with read/consume split.
CMakePresets.json	Add CI preset wiring for OpenSSL QUIC enablement and adjust Fedora preset inheritance.
CMakeLists.txt	Add ENABLE_OPENSSL_QUIC, OpenSSL QUIC feature checks, mutual exclusivity with quiche, and quiche static selection under compat mode.
cmake/Findquiche.cmake	Support selecting static quiche when required by compat shim linkage needs.
cmake/CheckOpenSSLHasQuicTlsCbs.cmake	New compile-check for OpenSSL QUIC TLS callback API availability.
cmake/CheckOpenSSLHasNativeQuic.cmake	New compile-check for OpenSSL native QUIC listener/stream APIs availability.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/iocore/net/OpenSSLQUICPacketHandler.cc:553

• If configuring the listener BIO fails, _start_listener() returns false without freeing this->_listener. Since the listener now owns the BIO via SSL_set_bio(), this can leak both objects and keep the acceptor in a broken state.

  SSL_set_bio(this->_listener, bio, bio);
  if (SSL_get_rbio(this->_listener) == nullptr || SSL_get_wbio(this->_listener) == nullptr) {
    Error("failed to configure OpenSSL QUIC listener socket");
    return false;
  }

src/iocore/net/OpenSSLQUICPacketHandler.cc:565

• If SSL_listen() fails, _start_listener() returns false but leaves this->_listener allocated. Freeing and nulling the listener on this path avoids leaking and prevents later code from assuming the listener is valid.

  if (SSL_listen(this->_listener) != 1) {
    Error("failed to start OpenSSL QUIC listener");
    return false;
  }

src/proxy/http3/Http3HeaderFramer.cc:106

• Http3HeaderFramer::reset() clears most framing state but does not reset _is_final_header_sent. Once a final header has been sent, this flag can remain true across subsequent writes, causing Http3Transaction::is_response_header_sent() to return true prematurely (and can trip the ink_assert(!is_response_header_sent()) in generate_frame()).
  src/iocore/net/OpenSSLQUICPacketHandler.cc:548
• If new_peer_capture_bio() fails, _start_listener() returns false after creating this->_listener but does not free it or reset the member. This leaks the listener SSL object and leaves the acceptor in a partially-initialized state.

This issue also appears in the following locations of the same file:

• line 549
• line 562

  BIO *bio = new_peer_capture_bio(this->server.sock.get_fd());
  if (bio == nullptr) {
    Error("failed to create OpenSSL QUIC peer capture BIO");
    return false;
  }

[maskit]

I don't get why we need another callback function.

[bneradt]

The existing ssl_alpn_select_callback retrieves ALPNSupport from the SSL object’s ex-data, which is bound when the QUICNetVConnection is initialized. With OpenSSL’s native QUIC listener, ALPN negotiation happens before SSL_accept_connection() returns, while ATS allocates and initializes the QUICNetVConnection only after that call succeeds, so the existing callback has no ALPNSupport available. This context-level callback therefore performs only the early HTTP/3 ALPN selection needed by the listener; after acceptance, the SSL object is attached to the NetVC and normal ATS protocol handling takes over.

[maskit]

I already dislike the OpenSSL API (the inconsistency between TCP and QUIC)... but now I understand the necessity. This is fine for now but we should revisit this later.

[maskit]

Why a new function? The code surrounded by ifdef could be placed in quic_new_ssl_ctx. And the version settings are done there.

[bneradt]

quic_new_ssl_ctx() cannot unconditionally use OSSL_QUIC_server_method() because it is also called by quic_init_client_ssl_ctx(). Native OpenSSL requires the server-specific method for the listener, while the client and non-native quiche paths must retain the existing TLS_method() context, so moving the #ifdef into the shared function would make that function return a server-only context to client callers in native builds. Keeping quic_new_server_ssl_ctx() makes the role distinction explicit; its non-native path delegates to quic_new_ssl_ctx(), and the duplicated TLS-version settings noted separately have now been removed.

[maskit]

I forgot we already have quic_init_client_ssl_ctx. I still feel like these functions could be reorganized but this is not a blocker.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[maskit]

Overall looks good. No blockers.

We probably should reorganize functions so both implementations sit together nicely, but I may want to add QMux (QUIC over TCP) support before doing it to have a better picture.

[bneradt]

10.2.x version:
#13463