LLM inference callback support

[SteveSandersonMS]

Summary

This PR adds SDK support for intercepting LLM inference requests and handling them in user code across all six SDK languages: Node.js, .NET, Python, Go, Rust, and Java.

Consumers register one client-global LlmRequestHandler (constructed once, no args). The runtime invokes it over JSON-RPC (llmInference.*) whenever it would otherwise issue a model-layer HTTP or WebSocket request — for both BYOK and CAPI — fully replacing the outbound call. A handler that overrides nothing is a transparent pass-through.

It includes the full feature work on this branch:

• wire up LLM inference provider registration and generated RPC types
• add the raw chunked callback protocol for outbound inference requests and responses
• cover plain HTTP, streaming responses, cancellation (runtime- and consumer-initiated), error mapping, session-id threading, and WebSocket transport
• port the feature to every SDK language with idiomatic HTTP types
• keep the public callback surface to a single LlmRequestHandler model with forwarding helpers

What changed

Shared protocol and plumbing

• add generated RPC / session event types needed for LLM inference callbacks across all SDK surfaces
• add SDK-side registration for a process-global LLM inference provider (llmInference.setProvider)
• route outbound inference requests through the callback bridge instead of requiring provider-specific hooks

Per-language ports

Each language exposes the same LlmRequestHandler model, mapped onto the most canonical HTTP representation available in that ecosystem:

Language	HTTP request/response type	WebSocket type
Node.js	Request / Response (Fetch)	per-connection handler
.NET	HttpRequestMessage / HttpResponseMessage	ClientWebSocket
Python	httpx request/response	per-connection handler
Go	*http.Request / *http.Response	per-connection handler
Rust	http::Request / http::Response	per-connection handler
Java	java.net.http HttpRequest / HttpResponse	java.net.http.WebSocket

All ports thread cancellation and session id through the request context, and provide a ForwardingWebSocketHandler (or equivalent) for the common mutate-and-forward case.

API shape

• collapse HTTP interception to a single send hook (SendRequestAsync / sendRequest / sendHttp / language equivalents)
• expose WebSocket interception through OpenWebSocketAsync / openWebSocket, returning a per-connection handler object
• allow consumers to mutate, drop, duplicate, or fully replace request/response messages while keeping the common forwarding case straightforward

Usage examples

C#

using GitHub.Copilot;
using System.Net.Http;

sealed class MyHandler : LlmRequestHandler
{
    protected override async Task<HttpResponseMessage> SendRequestAsync(
        HttpRequestMessage request,
        LlmRequestContext ctx)
    {
        request.Headers.Add("X-Debug-Session", ctx.SessionId ?? "none");
        return await base.SendRequestAsync(request, ctx);
    }

    protected override Task<CopilotWebSocketHandler> OpenWebSocketAsync(LlmRequestContext ctx)
        => Task.FromResult<CopilotWebSocketHandler>(new MyForwardingSocket(ctx));
}

sealed class MyForwardingSocket : ForwardingWebSocketHandler
{
    public MyForwardingSocket(LlmRequestContext ctx)
        : base(ctx)
    {
    }

    public override Task SendRequestMessageAsync(LlmWebSocketMessage message)
    {
        var text = message.GetText().Replace("model-A", "model-B");
        return base.SendRequestMessageAsync(LlmWebSocketMessage.Text(text));
    }
}

Node.js

import {
    ForwardingWebSocketHandler,
    LlmRequestContext,
    LlmRequestHandler,
} from "@github/copilot";

class MyHandler extends LlmRequestHandler {
    protected override async sendRequest(request: Request, ctx: LlmRequestContext): Promise<Response> {
        const headers = new Headers(request.headers);
        headers.set("x-debug-session", ctx.sessionId ?? "none");

        return super.sendRequest(new Request(request, { headers }), ctx);
    }

    protected override async openWebSocket(ctx: LlmRequestContext) {
        return new MyForwardingSocket(ctx);
    }
}

class MyForwardingSocket extends ForwardingWebSocketHandler {
    override sendRequestMessage(data: string | Uint8Array) {
        if (typeof data === "string") {
            return super.sendRequestMessage(data.replace("model-A", "model-B"));
        }
        return super.sendRequestMessage(data);
    }
}

Java

import com.github.copilot.ForwardingWebSocketHandler;
import com.github.copilot.LlmRequestContext;
import com.github.copilot.LlmRequestHandler;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.io.InputStream;

final class MyHandler extends LlmRequestHandler {
    @Override
    protected HttpResponse<InputStream> sendHttp(HttpRequest request, LlmRequestContext ctx) throws Exception {
        HttpRequest mutated = HttpRequest.newBuilder(request, (n, v) -> true)
                .header("X-Debug-Session", ctx.sessionId() == null ? "none" : ctx.sessionId())
                .build();
        return super.sendHttp(mutated, ctx);
    }

    @Override
    protected CopilotWebSocketHandler openWebSocket(LlmRequestContext ctx) {
        return new ForwardingWebSocketHandler(ctx.url(), ctx.headers()) {
            @Override
            protected byte[] onSendRequestMessage(byte[] data, boolean binary) {
                String text = new String(data, java.nio.charset.StandardCharsets.UTF_8).replace("model-A", "model-B");
                return text.getBytes(java.nio.charset.StandardCharsets.UTF_8);
            }
        };
    }
}

Register the handler when constructing the client, e.g. in Java:

CopilotClient client = new CopilotClient(
    new CopilotClientOptions().setLlmInference(new LlmInferenceConfig().setHandler(new MyHandler())));

Tests

Each language adds e2e coverage (mirroring a shared reference suite) for:

• callback provider registration
• HTTP inference interception
• streaming inference interception
• error mapping
• runtime-initiated and consumer-initiated cancellation
• session-id threading
• WebSocket callback handling
• an idiomatic handler test exercising mutate-and-forward over both HTTP and WebSocket

Plus per-language unit coverage where applicable (e.g. .NET handler/adapter behavior, Node.js unit flows).

Resolves github/copilot-sdk-internal#88

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[SteveSandersonMS]

Why do we have this wrapper type and not just put LlmRequestHandler directly onto the create/resume config?

[SteveSandersonMS]

Is this IVT really needed?

[SteveSandersonMS]

Why is this public when everything else in here is internal? Maybe it could move to the file with all the public API.

[SteveSandersonMS]

Why are you doing that? What's wrong with holding the RPC reply until the consumer's I/O completes? Isn't tat necessary to achieve natural backpressure and error propagation?

Altogether I'm surprised by how much complexity there is in this file and wonder if you're trying to make it too clever. Why isn't there a significantly simpler way of adapting the RPC protocol to the public base class, for example by eliminating any separate queueing/buffering/etc?

[SteveSandersonMS]

Seems really odd to have an adapter like this when the source data is already in the right format. What stops us from simplifying all this?

[github-actions]

Generated by SDK Consistency Review Agent for issue #1689 · sonnet46 3.4M

[github-actions]

Cross-SDK naming note: This class is named CopilotRequestHandler (along with CopilotRequestContext and ForwardingCopilotWebSocketHandler), following the Node.js convention. However, the three other SDKs added in this same PR (Go, Java) and the two pre-existing ones (Python, Rust) all use LlmRequestHandler / LlmRequestContext / ForwardingWebSocketHandler.

After this PR the naming is 4-vs-2: LlmRequestHandler (Python, Rust, Go, Java) vs CopilotRequestHandler (Node.js, .NET). If this split is intentional (grouping .NET with Node.js), that's fine, but it would be worth noting somewhere—otherwise it may be worth aligning .NET with the majority Llm* naming.

[github-actions]

Cross-SDK config API note: The handler is set here as a flat RequestHandler CopilotRequestHandler? property (same pattern as Node.js's requestHandler). The other four SDKs (Python, Rust, Go, Java) all wrap it in an LlmInferenceConfig config object, e.g.:

// Java
options.setLlmInference(new LlmInferenceConfig().setHandler(new MyHandler()));

// Go
options.LlmInference = &LlmInferenceConfig{Handler: &MyHandler{}}

# Python
options.llm_inference = LlmInferenceConfig(handler=MyHandler())

After this PR the pattern is 4-vs-2: config wrapper (Python, Rust, Go, Java) vs direct property (Node.js, .NET). Developers targeting multiple SDKs will encounter a different configuration shape for the same feature. Worth a deliberate decision here.

[github-actions]

Minor cross-SDK naming note: This method is named sendHttp (like Rust's send_http), but Node.js and Python — both also in the LlmRequestHandler family — name the equivalent hook sendRequest / send_request. All three ultimately receive an "HTTP request" object, so the naming is semantically equivalent, but within the LlmRequestHandler group it differs between languages:

SDK	HTTP hook name
Python	send_request(httpx.Request, ctx)
Rust	send_http(LlmHttpRequest, ctx)
Java	sendHttp(HttpRequest, ctx)

Suggest clarifying whether sendHttp or sendRequest should be the canonical hook name for the LlmRequestHandler family (it won't matter for users of a single SDK, but affects multi-language teams reading docs side-by-side).

[Copilot]

Pull request overview

This PR introduces SDK-wide support for intercepting the runtime’s outbound model-layer LLM inference traffic (HTTP/SSE + WebSocket) by registering a single client-global handler over JSON-RPC (llmInference.*). It extends the shared codegen pipeline to generate client-global reverse-RPC handlers and wires language-specific handler abstractions plus E2E coverage across Node.js, .NET, Python, Go, Rust, and Java.

Changes:

• Extend RPC schema/codegen to support a new clientGlobal API surface and generate registration helpers for llmInference.*.
• Add per-language “request handler” APIs/adapters to forward/replace runtime model HTTP + WebSocket traffic, including cancellation and session-id threading.
• Add cross-language E2E tests validating session-id propagation, cancellation/error behavior, and forwarding behavior for both HTTP and WebSocket paths.

Show a summary per file

File	Description
scripts/codegen/utils.ts	Adds clientGlobal support to the shared API schema shape.
scripts/codegen/typescript.ts	Generates TS client-global handler interfaces + registration function.
scripts/codegen/python.ts	Generates Python client-global handler Protocols/dataclass + registration function.
scripts/codegen/go.ts	Generates Go client-global handler interfaces + registration function.
scripts/codegen/csharp.ts	Generates C# client-global handler interfaces + registration plumbing.
rust/tests/e2e/support.rs	Adds no-snapshot E2E context and helper to start an LLM handler-wired client.
rust/tests/e2e.rs	Registers new Rust E2E module for request-handler coverage.
rust/src/types.rs	Re-exports request-handler types for the Rust public surface.
rust/src/router.rs	Routes llmInference.* requests via a client-global dispatcher rather than per-session routing.
rust/src/lib.rs	Adds request-handler option wiring, early router start for startup model calls, and internal handle reconstruction.
rust/Cargo.toml	Adds HTTP/WebSocket dependencies needed for the Rust forwarding/transport implementation.
python/pyproject.toml	Promotes httpx to a runtime dependency and adds websockets to dev deps for tests.
python/e2e/test_copilot_request_session_id_e2e.py	Verifies session-id threading into handler for CAPI and BYOK sessions.
python/e2e/test_copilot_request_handler_e2e.py	Validates mutate-and-forward behavior across HTTP + WebSocket via a real upstream.
python/e2e/test_copilot_request_cancel_error_e2e.py	Covers thrown-error mapping and runtime cancellation behavior via cancel_event.
python/e2e/_copilot_request_helpers.py	Shared synthetic model endpoint builders and isolated-client fixtures for request-handler tests.
python/copilot/generated/rpc.py	Adds generated client-global handler types and registration for llmInference.*.
python/copilot/client.py	Wires request-handler registration and llmInference.setProvider into client startup.
python/copilot/init.py	Re-exports request-handler types/helpers as part of the public Python API surface.
nodejs/test/e2e/copilot_request_session_id.e2e.test.ts	Node E2E for session-id threading through the request handler.
nodejs/test/e2e/copilot_request_cancel_error.e2e.test.ts	Node E2E for handler error propagation and runtime-driven cancellation.
nodejs/src/types.ts	Adds requestHandler option + exports request-handler types from the Node public surface.
nodejs/src/index.ts	Re-exports request-handler APIs from the Node entrypoint.
nodejs/src/generated/rpc.ts	Adds generated client-global handler interfaces + registerClientGlobalApiHandlers.
nodejs/src/client.ts	Registers client-global handlers on the connection and calls llmInference.setProvider when configured.
nodejs/package.json	Adds ws and @types/ws for Node E2E WebSocket upstream tests.
nodejs/package-lock.json	Locks new ws / @types/ws additions.
java/src/test/java/com/github/copilot/FakeUpstreamServer.java	Adds an in-process raw HTTP/WebSocket upstream for Java forwarding E2E.
java/src/test/java/com/github/copilot/CopilotRequestSessionIdE2ETest.java	Java E2E verifying session-id threading for CAPI + BYOK.
java/src/test/java/com/github/copilot/CopilotRequestHandlerE2ETest.java	Java E2E validating synthetic HTTP servicing and forwarding over HTTP + WebSocket.
java/src/test/java/com/github/copilot/CopilotRequestCancelErrorE2ETest.java	Java E2E for error propagation and runtime cancellation behavior.
java/src/main/java/module-info.java	Makes java.net.http a required module dependency (runtime usage).
java/src/main/java/com/github/copilot/rpc/CopilotClientOptions.java	Adds requestHandler option to configure the connection-level interception handler.
java/src/main/java/com/github/copilot/LlmWebSocketResponseBridge.java	Ensures WebSocket upgrade head is emitted before any body/terminal frames.
java/src/main/java/com/github/copilot/LlmInferenceExchange.java	Implements per-request body ingestion + strict response emission state machine.
java/src/main/java/com/github/copilot/LlmInferenceAdapter.java	Bridges llmInference.* reverse-RPC frames to CopilotRequestHandler execution.
java/src/main/java/com/github/copilot/ForwardingCopilotWebSocketHandler.java	Default Java WS forwarding implementation using java.net.http.WebSocket.
java/src/main/java/com/github/copilot/CopilotWebSocketMessage.java	Represents WS text/binary messages exchanged via the handler seam.
java/src/main/java/com/github/copilot/CopilotWebSocketHandler.java	Base per-connection WS handler contract for runtime↔upstream message forwarding.
java/src/main/java/com/github/copilot/CopilotWebSocketCloseStatus.java	Carries terminal close/error state for callback-owned WS connections.
java/src/main/java/com/github/copilot/CopilotRequestTransport.java	Exposes the runtime’s intended transport selection (HTTP vs WebSocket).
java/src/main/java/com/github/copilot/CopilotRequestHandler.java	Adds Java handler base class with default HTTP forwarding + WS forwarding hooks.
java/src/main/java/com/github/copilot/CopilotRequestContext.java	Provides request-id/session-id/headers/url/transport/cancellation context to handlers.
java/src/main/java/com/github/copilot/CopilotClient.java	Registers the adapter + calls llmInference.setProvider when handler is configured.
go/types.go	Adds RequestHandler to Go client options for connection-level interception.
go/rpc/zrpc.go	Adds generated client-global handler types and registration for llmInference.*.
go/internal/e2e/copilot_request_session_id_e2e_test.go	Go E2E for session-id threading into intercepted inference calls.
go/internal/e2e/copilot_request_helpers_test.go	Shared synthetic upstream builders for Go request-handler E2Es.
go/internal/e2e/copilot_request_handler_e2e_test.go	Go E2E validating mutate-and-forward over HTTP + WebSocket.
go/internal/e2e/copilot_request_cancel_error_e2e_test.go	Go E2E covering error propagation and runtime cancellation behavior.
go/go.sum	Adds checksum entries for new Go WebSocket dependency.
go/go.mod	Adds github.com/coder/websocket for Go WebSocket upstream tests.
go/client.go	Wires client-global handler registration and LlmInference.SetProvider when configured.
dotnet/test/E2E/CopilotRequestSessionIdE2ETests.cs	.NET E2E verifying session-id threading for CAPI + BYOK sessions.
dotnet/test/E2E/CopilotRequestE2EProvider.cs	Shared .NET test handler that fabricates model endpoint responses for off-network turns.
dotnet/test/E2E/CopilotRequestCancelErrorE2ETests.cs	.NET E2E for handler error mapping and runtime cancellation propagation.
dotnet/src/Types.cs	Adds experimental RequestHandler option to CopilotClientOptions.
dotnet/src/Generated/Rpc.cs	Adds generated types and client-global registration for llmInference.*.
dotnet/src/Client.cs	Registers client-global handlers and calls LlmInference.SetProviderAsync when configured.

Copilot's findings

Files not reviewed (2)

• go/rpc/zrpc.go: Generated file
• nodejs/package-lock.json: Generated file

• Files reviewed: 62/69 changed files
• Comments generated: 1

[github-actions]

Generated by SDK Consistency Review Agent for issue #1689 · sonnet46 3.1M

[github-actions]

Cross-SDK consistency concern: binary WebSocket frame type is lost

The Go adapter decodes binary frames correctly in routeChunk (line ~603: base64-decodes when params.Binary == true) but drops the binary/text flag before pushing into the Body channel. As a result, SendRequestMessage here always forwards as websocket.MessageText, regardless of the original wire frame type.

All other SDKs carry the binary flag through the request message path:

• .NET: CopilotWebSocketMessage.IsBinary
• Java: CopilotWebSocketMessage.binary()
• Python: str | bytes (type encodes the distinction)
• Node.js: string | Uint8Array (same)
• Rust: CopilotWebSocketMessage.binary: bool

In practice, LLM inference likely only uses text frames, so this may not cause immediate issues. But if binary frames ever appear this will corrupt the upstream message silently. Consider threading the binary flag through the Body channel (e.g. as a struct { data []byte; binary bool }) so SendRequestMessage can use the correct message type.

[github-actions]

Minor cross-SDK naming inconsistency: Code vs ErrorCode

All other SDKs name this field ErrorCode/errorCode/error_code:

• .NET: public string? ErrorCode { get; init; }
• Java: public String errorCode()
• Python: error_code: str | None
• Node.js: readonly errorCode?: string

Go uses the shorter Code, which is not wrong, but this creates a small inconsistency for developers navigating across SDKs. Consider renaming to ErrorCode to match the cross-SDK convention (the Err field name is fine — Err vs Error is established Go idiom).

[github-actions]

API surface difference from other SDKs: extra response parameter

Every other SDK's open_websocket / openWebSocket / OpenWebSocketAsync takes only the context:

// Node.js
protected openWebSocket(ctx: CopilotRequestContext): Promise<CopilotWebSocketHandler>
// .NET
protected virtual Task<CopilotWebSocketHandler> OpenWebSocketAsync(CopilotRequestContext ctx)
// Java
protected CopilotWebSocketHandler openWebSocket(CopilotRequestContext ctx)
// Python
async def open_web_socket(self, ctx: CopilotRequestContext) -> CopilotWebSocketHandler
// Go
OpenWebSocket func(ctx *CopilotRequestContext) (CopilotWebSocketHandler, error)

Rust adds response: CopilotWebSocketResponse as a second argument, which consumers must store and use to write upstream→runtime messages.

This is architecturally necessary for Rust traits (no base-class state for a sendResponseMessage helper), so it isn't a bug. But it's worth documenting clearly in the CopilotRequestHandler trait doc comment — specifically that response must be stored in the returned handler struct and used to call response.send_message(...) in place of the sendResponseMessage base-class helper the other SDKs provide.

[github-actions]

Cross-SDK Consistency Review

Overall, this is an impressively consistent implementation across all six SDK languages — class names, option field names, and WebSocket API shapes are well-aligned. One naming inconsistency in the HTTP hook is worth flagging before merge.

⚠️ HTTP override method: sendHttp/send_http vs sendRequest/send_request/SendRequestAsync

Three SDKs use one semantic name, two use another:

SDK	HTTP override method
Node.js	sendRequest(request: Request, ctx)
.NET	SendRequestAsync(HttpRequestMessage, ctx)
Python	send_request(self, request, ctx)
Rust	send_http(request: CopilotHttpRequest, ctx)
Java	sendHttp(HttpRequest request, ctx)
Go	n/a — struct field Transport http.RoundTripper (idiomatic)

Rust and Java use send_http/sendHttp while the other three SDKs spell it sendRequest/SendRequestAsync/send_request. A developer porting a handler from Node.js to Java would look for sendRequest, not find it, and have to hunt for sendHttp. Aligning all five override-based SDKs on sendRequest/SendRequestAsync/send_request (or alternatively all on sendHttp/send_http) would improve discoverability across the SDK family.

i️ PR description has stale Java examples

The Java usage example in the PR description references types and method names that differ from the actual code:

PR description	Actual code
extends LlmRequestHandler	extends CopilotRequestHandler
LlmRequestContext	CopilotRequestContext
new ForwardingWebSocketHandler(ctx.url(), ctx.headers())	new ForwardingCopilotWebSocketHandler(context)
protected byte[] onSendRequestMessage(byte[] data, boolean binary)	public void sendRequestMessage(CopilotWebSocketMessage message)

This doesn't affect the implementation itself but will confuse readers. Worth updating before merge so the description matches the actual API.

✅ Everything else looks consistent

• Class names: CopilotRequestHandler, CopilotRequestContext, CopilotWebSocketHandler, CopilotWebSocketCloseStatus, ForwardingCopilotWebSocketHandler — consistent across all SDKs ✅
• Client option field: requestHandler/RequestHandler/request_handler — consistent across all SDKs ✅
• WebSocket override: openWebSocket/OpenWebSocketAsync/open_web_socket/open_websocket — consistent (language-case only) ✅
• WebSocket message type CopilotWebSocketMessage and its sendRequestMessage/sendResponseMessage hooks — consistent ✅
• E2E test coverage appears symmetric across all six SDKs ✅

Generated by SDK Consistency Review Agent for issue #1689 · sonnet46 2.2M · ◷

[github-actions]

Generated by SDK Consistency Review Agent for issue #1689 · sonnet46 2.2M

[github-actions]

Naming inconsistency with Node.js / .NET / Python: The HTTP interception hook is called sendHttp here (and send_http in Rust), but the equivalent in the other three SDKs is sendRequest / SendRequestAsync / send_request. A consumer who has read the Node.js docs will look for sendRequest and not find it.

Suggest aligning on one name across all five override-based SDKs — either rename this to sendRequest (matching the majority) or update the other three to sendHttp/send_http.

[github-actions]

Naming inconsistency with Node.js / .NET / Python: The HTTP interception hook is called send_http here (and sendHttp in Java), but the equivalent in the other three SDKs is sendRequest / SendRequestAsync / send_request. Aligning on one name across all five override-based SDKs would improve cross-language discoverability.