Long request upload timeout Error

[dhanush-mk]

Hello,

We have an use case to upload / download large files to/from our application behind envoy proxy LB. Lets take i am downloading or uploading a file to the backend which usually takes 10 min normally, we had our previous nginx ingress controller which uses these annotations which allows 10min timeout for our proxy server
proxy-send-timeout = 600s
proxy-read-timeout = 600s

So in gateway API i created a BackendTrafficPolicy and in http timeouts added requestTimeout = 600s and it worked well for downloading a large file which takes around 10 mins. But for uploading a large file i added many configs like maxStreamDuration under backendtrafficpolicy , requestReceivedTimeout in clienttrafficpolicy, under httproute added timeout section with request and backendRequest field as well, but none worked as if i am uploading from GUI i am getting timeout within 300-350 sec.

[MentalOfCrow]

The 300–350 second cutoff strongly suggests Envoy's downstream stream idle timeout, whose default is 5 minutes. The settings you listed control different timers:

• BackendTrafficPolicy.timeout.http.requestTimeout limits the upstream request/response and explains why extending it helped the download;
• maxStreamDuration limits total upstream stream lifetime;
• ClientTrafficPolicy.timeout.http.requestReceivedTimeout limits the total time allowed to receive the request body;
• ClientTrafficPolicy.timeout.http.streamIdleTimeout is the downstream stream's inactivity timer and defaults to 5 minutes.

Set both downstream upload-related timers on a ClientTrafficPolicy attached to the Gateway (not the backend/HTTPRoute), with some margin over the expected 10 minutes:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: large-upload-client-timeouts
  namespace: your-namespace
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: your-gateway
  timeout:
    http:
      requestReceivedTimeout: 900s
      streamIdleTimeout: 900s

Keep the backend/route request timeout at least as large, or use 0s only if you intentionally want no overall limit. Then verify that the policy is actually attached:

kubectl get clienttrafficpolicy large-upload-client-timeouts \
  -n your-namespace -o yaml

Its status should show an accepted condition for the intended Gateway. A policy in another namespace, targeting the wrong Gateway name/section, or rejected because the installed Envoy Gateway CRDs do not expose that field will have no effect even though kubectl apply succeeded.

Before changing more timers, add or inspect Envoy access logs with %RESPONSE_CODE_DETAILS%. The value tells you which timer fired:

• stream_idle_timeout → raise ClientTrafficPolicy ... streamIdleTimeout or determine why the browser/backend stopped producing data events;
• request_overall_timeout → the total request/route timeout is still lower;
• upstream_max_stream_duration_reached → maxStreamDuration is the remaining limit.

If the upload is continuously transmitting bytes, the stream-idle timer should reset on each data event. If it still reports stream_idle_timeout, check for buffering or a stalled hop in front of Envoy (cloud load balancer, WAF, browser/reverse proxy) and test directly with a throttled curl upload to the Gateway address. Also inspect that outer hop's logs/timeouts; Envoy policies cannot override a 300-second timeout enforced before the request reaches Envoy.

If this identifies the 5-minute cutoff, please mark it as the accepted answer so future upload cases can find the downstream-versus-backend timeout distinction quickly.

[dhanush-mk]

Hello,
Thankyou for the explanation. I have configured the timeout in clienttrafficpolicy and tried, but still it is failing not exactly in 5 min, but in 5.25 min, it is a wired time, not sure from where it is coming. I am sharing you the errors found in envoy logs

timeout:
http:
requestReceivedTimeout: 900s
streamIdleTimeout: 900s

Logs: