Bound HTTP transaction lifetime

CHANGELOG.asciidoc

@@ -69,6 +69,8 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima
 ** Reconciled the `validationRequest` default: the builder default is now `g.inject(0)` to match the `Settings` default (it was previously `''`).
 * Removed `Transaction.open()` in favor of `begin()`, which is now the single transaction-start primitive across embedded and remote contexts.
 * Changed `begin()` and `close()` to be idempotent and calling it when a transaction is already in that state no longer throws.
+* Added `maxTransactionLifetime` setting to Gremlin Server, an absolute cap on the total age of an HTTP transaction that interrupts a running operation and rolls the transaction back when it fires (default 600000ms, set to `0` to disable).
+* Changed the Gremlin Server HTTP transaction idle timer to suspend while an operation is running (so a long-running operation is bounded by `evaluationTimeout` rather than the idle timeout) and to honor `0` as "disable idle reclamation"; the `idleTransactionTimeout` default is now 60000ms.
 * Added configurable CORS `allowedOrigins` setting to Gremlin Server; warns when wildcard origin is used alongside authentication.
 * Standardized `gremlin-go` connection options per the TinkerPop 4.x GLV proposal:
 ** Moved `BasicAuth`/`SigV4Auth`/`SigV4AuthWithCredentials` out of package `gremlingo` into a new `auth` sub-package as `auth.Basic`/`auth.SigV4`/`auth.SigV4WithCredentials`. The flat `gremlingo` functions have been removed; use the `auth` sub-package.

docs/src/dev/provider/index.asciidoc

@@ -1430,14 +1430,21 @@ rejects it with HTTP 400. This prevents cross-graph operations within a single t
 
 ==== Transaction Timeout and Idle Reclamation
 
-Servers implement a configurable transaction timeout (`transactionTimeout`, default 600000ms). The timeout represents
-how long a transaction can sit idle with no requests before the server forcibly rolls it back and removes it. The
-timeout resets on each request received for that transaction, so active transactions are not affected. After a timeout
-fires, any subsequent request with that transaction ID receives a 404 response.
+A transaction can be bounded at three independent scopes, each disabled with `0`. The `evaluationTimeout` bounds a
+single operation. The `idleTransactionTimeout` (default 60000ms) bounds the idle gaps between operations: how long a
+transaction may remain idle, with no operation running or queued, before it is rolled back and removed; once it is
+removed, a subsequent request with that transaction ID receives a 404. The `maxTransactionLifetime` (default 600000ms)
+bounds the total age of the transaction regardless of activity, and so may end a transaction while an operation is still
+running; the in-flight request then receives a 504 (`TransactionException`) and subsequent requests receive a 404.
+
+The defaults bound a transaction without any operator configuration; disabling them is a deliberate choice, and a
+per-request `timeoutMs` is honored as sent rather than overridden by the server. How promptly a running operation is
+actually interrupted when a bound fires is a property of the provider's execution and traversal machinery, not
+guaranteed by these settings.
 
 ==== Transaction Capacity Limits
 
-Servers enforce a configurable maximum number of concurrent open transactions (`maxConcurrentTransactions`, default
+Servers may enforce a configurable maximum number of concurrent open transactions (`maxConcurrentTransactions`, default
 1000). When the limit is reached, new begin requests are rejected with HTTP 503. Slots are freed when transactions
 close via commit, rollback, or timeout.
 

docs/src/reference/gremlin-applications.asciidoc

@@ -930,13 +930,15 @@ The following table describes the various YAML configuration options that Gremli
 |gremlinPool |The number of "Gremlin" threads available to execute actual scripts in a `ScriptEngine`. This pool represents the workers available to handle blocking operations in Gremlin Server. When set to `0`, Gremlin Server will use the value provided by `Runtime.availableProcessors()`. |0
 |host |The name of the host to bind the server to. |localhost
 |idleConnectionTimeout |Time in milliseconds that the server will allow a channel to not receive requests from a client before it automatically closes. If enabled, the value provided should typically exceed the amount of time given to `keepAliveInterval`. Note that while this value is to be provided as milliseconds it will resolve to second precision. Set this value to `0` to disable this feature. |0
+|idleTransactionTimeout |Time in milliseconds that a transaction can remain idle (no operation running or queued) before the server forcibly rolls it back and removes it. The idle timer is suspended while an operation is running, so a long-running operation does not trip it (its duration is instead bounded by `evaluationTimeout`); the timer is armed only once the transaction returns to idle. Set to `0` to disable idle reclamation. |60000
 |keepAliveInterval |Time in milliseconds that the server will allow a channel to not send responses to a client before it sends a "ping" to see if it is still present. If it is present, the client should respond with a "pong" which will thus reset the `idleConnectionTimeout` and keep the channel open. If enabled, this number should be smaller than the value provided to the `idleConnectionTimeout`. Note that while this value is to be provided as milliseconds it will resolve to second precision. Set this value to `0` to disable this feature. |0
 |maxAccumulationBufferComponents |Maximum number of request components that can be aggregated for a message. |1024
 |maxChunkSize |The maximum length of the content or each chunk.  If the content length exceeds this value, the transfer encoding of the decoded request will be converted to 'chunked' and the content will be split into multiple `HttpContent` objects.  If the transfer encoding of the HTTP request is 'chunked' already, each chunk will be split into smaller chunks if the length of the chunk exceeds this value. |8192
 |maxConcurrentTransactions |The maximum number of concurrent open transactions allowed. When this limit is reached, new `g.tx().begin()` requests are rejected with HTTP 503. Slots are freed when transactions close via commit, rollback, or timeout. |1000
 |maxHeaderSize |The maximum length of all headers. |8192
 |maxInitialLineLength |The maximum length of the initial line (e.g.  "GET / HTTP/1.0") processed in a request, which essentially controls the maximum length of the submitted URI. |4096
 |maxParameters |The maximum number of parameters that can be passed on a request. Larger numbers may impact performance for scripts. This configuration only applies to the `HttpChannelizer`. |16
+|maxTransactionLifetime |Absolute cap in milliseconds on the total age of a transaction regardless of activity. Unlike `idleTransactionTimeout`, it fires even while an operation is running, interrupting it and rolling the transaction back. Set to `0` to disable the cap. |600000
 |maxRequestContentLength |The maximum length of the aggregated content for a request message.  Works in concert with `maxChunkSize` where chunked requests are accumulated back into a single message.  A request exceeding this size will return a `413 - Request Entity Too Large` status code. |10485760
 |maxWorkQueueSize |The maximum size the general processing queue can grow before the `gremlinPool` starts to reject requests. |8192
 |metrics.consoleReporter.enabled |Turns on console reporting of metrics. |false
@@ -983,7 +985,6 @@ The following table describes the various YAML configuration options that Gremli
 |strictTransactionManagement |Set to `true` to require `aliases` to be submitted on every requests, where the `aliases` become the scope of transaction management. |false
 |threadPoolBoss |The number of threads available to Gremlin Server for accepting connections. Should always be set to `1`. |1
 |threadPoolWorker |The number of threads available to Gremlin Server for processing non-blocking reads and writes. |1
-|transactionTimeout |Time in milliseconds that a transaction can sit idle (no requests) before the server forcibly rolls it back and removes it. The timeout resets on each request received for that transaction. Set to `0` to disable this feature. |600000
 |useEpollEventLoop |Try to use epoll event loops (works only on Linux os) instead of netty NIO. |false
 |writeBufferHighWaterMark | If the number of bytes in the network send buffer exceeds this value then the channel is no longer writeable, accepting no additional writes until buffer is drained and the `writeBufferLowWaterMark` is met. |65536
 |writeBufferLowWaterMark | Once the number of bytes queued in the network send buffer exceeds the `writeBufferHighWaterMark`, the channel will not become writeable again until the buffer is drained and it drops below this value. |32768
@@ -2259,16 +2260,26 @@ IMPORTANT: Not all graph implementations support explicit transactions (for exam
 Use `TinkerTransactionGraph` or another graph implementation that supports explicit transactions. Attempting to begin
 an explicit transaction on a graph that does not support them will result in an error.
 
-Two settings in the Gremlin Server YAML control transaction resource usage:
-
-* `transactionTimeout` (default: 600000ms) -- How long a transaction can sit idle before the server forcibly rolls it
-  back. The timeout resets on each request received for that transaction.
-* `maxConcurrentTransactions` (default: 1000) -- The maximum number of open transactions allowed. When the limit is
-  reached, new begin requests are rejected with HTTP 503.
+Several settings in the Gremlin Server YAML control transaction resource usage. The `idleTransactionTimeout`
+(default 60000ms) governs how long a transaction can remain idle (no operation running or queued) before the
+server forcibly rolls it back. The idle timer is suspended while an operation is running, so a long-running operation
+does not trip it; it is armed only once the transaction returns to idle. Set it to `0` to disable idle reclamation. The
+`maxTransactionLifetime` (default 600000ms) is an absolute cap on the total age of a transaction regardless of
+activity. Unlike `idleTransactionTimeout`, it fires even while an operation is running, interrupting it and rolling the
+transaction back. It bounds transaction lifetime and concurrency-slot occupancy absolutely; like `evaluationTimeout`,
+its ability to free the underlying worker thread depends on the operation reaching an interruptible point. Set it to
+`0` to disable the cap. Finally, `maxConcurrentTransactions` (default 1000) caps the number of open transactions
+allowed; when the limit is reached, new begin requests are rejected with HTTP 503.
+
+These compose with the per-operation `evaluationTimeout` (and its per-request `timeoutMs` override) to bound a
+transaction at three independent scopes: a single operation (`evaluationTimeout`), the gaps between operations
+(`idleTransactionTimeout`), and the transaction as a whole (`maxTransactionLifetime`). The defaults keep a transaction
+bounded out of the box; disabling all of them is a deliberate operator choice, and a per-request `timeoutMs` is always
+honored as sent.
 
 Each open transaction consumes a dedicated thread on the server to maintain thread-local transaction state for the
-underlying graph. Ensure that clients close transactions promptly and that the `transactionTimeout` is set to reclaim
-abandoned ones. The `transactions` gauge metric can be used to monitor usage.
+underlying graph. Ensure that clients close transactions promptly and that the `idleTransactionTimeout` is set to
+reclaim abandoned ones. The `transactions` gauge metric can be used to monitor usage.
 
 In load-balanced deployments, all requests within a transaction must reach the same server instance because
 transaction state is local to the server that created it. The `X-Transaction-Id` header is available for load

gremlin-server/conf/gremlin-server-transaction.yaml

@@ -31,6 +31,8 @@ metrics: {
   jmxReporter: {enabled: true},
   slf4jReporter: {enabled: true, interval: 180000}}
 strictTransactionManagement: false
+idleTransactionTimeout: 60000
+maxTransactionLifetime: 600000
 idleConnectionTimeout: 0
 keepAliveInterval: 0
 maxInitialLineLength: 4096

gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/Context.java

@@ -57,6 +57,10 @@ public class Context {
     private final Object timeoutExecutorLock = new Object();
     private String transactionId; // initially null for non-transactional requests and begin() calls; set after transaction creation.
     private Map<String, Object> parameters = new HashMap<>(); // only available after string parameters are parsed by grammar.
+    // Set by the transaction's lifetime cap (on the scheduler thread) before it interrupts this request's operation, and
+    // read as the interrupt unwinds the operation (on the transaction worker thread) to report an accurate
+    // transaction-timeout error rather than a generic evaluation timeout. volatile for cross-thread visibility.
+    private volatile boolean closedByLifetimeCap = false;
 
     public Context(final RequestMessage requestMessage, final ChannelHandlerContext ctx,
                    final Settings settings, final GraphManager graphManager,
@@ -143,6 +147,22 @@ public void setTransactionId(final String transactionId) {
         this.transactionId = transactionId;
     }
 
+    /**
+     * Marks this request's operation as having been interrupted because its transaction hit its absolute lifetime cap.
+     * Set by the transaction's lifetime cap before it interrupts the operation.
+     */
+    public void setClosedByLifetimeCap(final boolean closedByLifetimeCap) {
+        this.closedByLifetimeCap = closedByLifetimeCap;
+    }
+
+    /**
+     * Returns {@code true} if this request's operation was interrupted by its transaction's absolute lifetime cap, in
+     * which case the resulting interrupt should be reported as a transaction timeout rather than an evaluation timeout.
+     */
+    public boolean isClosedByLifetimeCap() {
+        return closedByLifetimeCap;
+    }
+
     public Map<String, Object> getParameters() {
         return this.parameters;
     }

gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/Settings.java

@@ -185,10 +185,13 @@ public Settings() {
     public boolean strictTransactionManagement = false;
 
     /**
-     * Time in milliseconds that a transaction can remain idle before it is automatically rolled back.
-     * This prevents resource leaks from abandoned transactions. Default is 600000 (10 minutes).
+     * Time in milliseconds that a transaction can remain idle (no operation running or queued) before it is
+     * automatically rolled back. This prevents resource leaks from abandoned transactions. The idle timer is suspended
+     * while an operation is in progress, so a long-running operation does not trip it (its duration is instead bounded
+     * by {@link #evaluationTimeout}). Set to {@code 0} to disable idle reclamation entirely. Default is 60000
+     * (1 minute).
      */
-    public long transactionTimeout = 600000L;
+    public long idleTransactionTimeout = 60000L;
 
     /**
      * Time in milliseconds to wait for a transaction commit or rollback operation to complete.
@@ -202,6 +205,17 @@ public Settings() {
      */
     public int maxConcurrentTransactions = 1000;
 
+    /**
+     * Absolute ceiling, in milliseconds, on the total age of a transaction regardless of activity. Unlike
+     * {@link #idleTransactionTimeout} (which only reclaims idle transactions), this cap fires even while an operation is
+     * running, interrupting it and rolling the transaction back, so it bounds how long a single transaction can hold its
+     * dedicated worker thread and concurrency slot. The bound on transaction lifetime and slot occupancy is absolute;
+     * the bound on thread occupancy is best-effort in the same way {@link #evaluationTimeout} is, since interrupting a
+     * running operation only takes effect when it reaches an interruptible point. Set to {@code 0} to disable the cap.
+     * Default is 600000 (10 minutes).
+     */
+    public long maxTransactionLifetime = 600000L;
+
     /**
      * The full class name of the {@link Channelizer} to use in Gremlin Server.
      */