FELIX-6859 allow a bounded VirtualThreadPool as the virtual threads executor

http/README.md

@@ -479,33 +479,45 @@ properties can be used (some legacy property names still exist but are not docum
 | `org.apache.felix.http.jetty.selectors` | The number of Jetty selector threads for the connector. Selectors notice and schedule established connections that can make I/O progress. Default is `-1`, which lets Jetty choose: for a sized thread pool Jetty uses `max(1, min(cpus / 2, maxThreads / 16))`, and for a thread pool that is not sized, such as the `VirtualThreadPool`, it uses `max(1, cpus / 2)`. Keep the selector count below the number of carrier threads when virtual threads are active. |
 | `org.apache.felix.http.jetty.threadpool.max` | The maximum number of threads in the Jetty thread pool. Default is `-1`, meaning the property is unset and Jetty's own default applies, which is a `QueuedThreadPool` with 200 platform threads. When `org.apache.felix.http.jetty.virtualthreads.enable` is `true`, this property also selects the type of thread pool, see [Thread pool and virtual threads](#thread-pool-and-virtual-threads). |
 | `org.apache.felix.http.jetty.virtualthreads.enable` | Enables virtual threads in Jetty 12 (JDK 21 or later). Default is `false`. The value of `org.apache.felix.http.jetty.threadpool.max` then selects which thread pool is built, see [Thread pool and virtual threads](#thread-pool-and-virtual-threads). |
+| `org.apache.felix.http.jetty.virtualthreads.max` | The maximum number of virtual thread tasks that run at the same time, or `-1` to leave the number unbounded. Note that unlike `org.apache.felix.http.jetty.threadpool.max` this bounds concurrent tasks, not the number of threads. Only relevant when `org.apache.felix.http.jetty.virtualthreads.enable` is `true`. When set to a positive value, Jetty's preferred setup is used: a `QueuedThreadPool`, sized by `org.apache.felix.http.jetty.threadpool.max`, whose virtual threads executor is a bounded `VirtualThreadPool`. Platform threads then still run the acceptors and the selectors. Default is `-1`. Jetty 12 bundle only, available from version 2.0.8. |
 
 ### Thread pool and virtual threads
 
-The combination of `org.apache.felix.http.jetty.threadpool.max` and
-`org.apache.felix.http.jetty.virtualthreads.enable` decides which thread pool the Jetty 12 bundle
-builds. Both properties are unset by default, which gives a `QueuedThreadPool` with 200 platform
-threads, the Jetty default.
-
-| `virtualthreads.enable` | `threadpool.max` | Thread pool |
-|--|--|--|
-| `false` | unset (`-1`) | Jetty's default `QueuedThreadPool` with 200 platform threads. |
-| `false` | set | A `QueuedThreadPool` with `<max>` platform threads. |
-| `true` | unset (`-1`) | A `QueuedThreadPool` whose virtual threads executor is `Executors.newVirtualThreadPerTaskExecutor()`. The number of concurrent virtual thread tasks is **unbounded**. |
-| `true` | set | A standalone `VirtualThreadPool` with `setMaxConcurrentTasks(<max>)`. A semaphore limits the number of tasks that run at the same time. |
-
-Note that enabling virtual threads without setting `org.apache.felix.http.jetty.threadpool.max`
-yields the unbounded variant, which Jetty warns can exhaust memory during a load spike.
+The combination of `org.apache.felix.http.jetty.threadpool.max`,
+`org.apache.felix.http.jetty.virtualthreads.enable` and
+`org.apache.felix.http.jetty.virtualthreads.max` decides which thread pool the Jetty 12 bundle
+builds. All three properties are unset by default, which gives a `QueuedThreadPool` with 200
+platform threads, the Jetty default.
+
+| `virtualthreads.enable` | `threadpool.max` | `virtualthreads.max` | Thread pool |
+|--|--|--|--|
+| `false` | unset (`-1`) | - | Jetty's default `QueuedThreadPool` with 200 platform threads. |
+| `false` | set | - | A `QueuedThreadPool` with `<max>` platform threads. |
+| `true` | unset (`-1`) | unset (`-1`) | A `QueuedThreadPool` whose virtual threads executor is `Executors.newVirtualThreadPerTaskExecutor()`. The number of concurrent virtual thread tasks is **unbounded**. |
+| `true` | set | unset (`-1`) | A standalone `VirtualThreadPool` with `setMaxConcurrentTasks(<max>)`. A semaphore limits the number of tasks that run at the same time. The pool creates only virtual threads. |
+| `true` | unset or set | set | Jetty's preferred setup: a `QueuedThreadPool`, sized by `threadpool.max`, whose virtual threads executor is a `VirtualThreadPool` with `setMaxConcurrentTasks(<value>)`. Platform threads still run the acceptors and the selectors. Available from Jetty 12 bundle version 2.0.8. |
+
+Note that enabling virtual threads without setting either `org.apache.felix.http.jetty.threadpool.max`
+or `org.apache.felix.http.jetty.virtualthreads.max` yields the unbounded variant, which Jetty warns
+can exhaust memory during a load spike.
+
+Note also that `org.apache.felix.http.jetty.threadpool.max` and
+`org.apache.felix.http.jetty.virtualthreads.max` bound different things:
+`threadpool.max` bounds threads, `virtualthreads.max` bounds the number of tasks that run at the
+same time. The fourth row above is the exception, where `threadpool.max` is reused as a bound on
+concurrent tasks. That is kept for backwards compatibility; `virtualthreads.max` is the properly
+named equivalent.
 
 A few more things to be aware of when virtual threads are enabled:
 
 * With the `QueuedThreadPool` variant, platform threads still run the acceptors and the selectors,
   and Jetty sets the reserved thread count to zero, so it always uses the Produce-Execute-Consume
   mode. The standalone `VirtualThreadPool` only creates virtual threads.
 * Keep `org.apache.felix.http.jetty.selectors` below the number of carrier threads.
-* Jetty documents a third combination as the preferred one: a `QueuedThreadPool` whose virtual
-  threads executor is a bounded `VirtualThreadPool`. There is no Felix HTTP configuration for that
-  combination yet, this is tracked in [FELIX-6859](https://issues.apache.org/jira/browse/FELIX-6859).
+* The combination Jetty documents as the preferred one, a `QueuedThreadPool` whose virtual threads
+  executor is a bounded `VirtualThreadPool`, is the last row of the table. It is configured with
+  `org.apache.felix.http.jetty.virtualthreads.max` and is available from Jetty 12 bundle version
+  2.0.8, see [FELIX-6859](https://issues.apache.org/jira/browse/FELIX-6859).
 
 Virtual threads require JDK 21 or later and are only supported by the Jetty 12 bundle. The Jetty 11
 bundle only honours `org.apache.felix.http.jetty.threadpool.max`.

http/jetty12/src/main/java/org/apache/felix/http/jetty/internal/ConfigMetaTypeProvider.java

@@ -168,9 +168,15 @@ public ObjectClassDefinition getObjectClassDefinition( String id, String locale
         adList.add(new AttributeDefinitionImpl(JettyConfig.FELIX_JETTY_USE_VIRTUAL_THREADS,
                 "Use Virtual Threads",
                 "Use virtual threads in Jetty (JDK 21 or higher). Defaults to false.",
-                -1,
+                false,
                 bundle.getBundleContext().getProperty(JettyConfig.FELIX_JETTY_USE_VIRTUAL_THREADS)));
 
+        adList.add(new AttributeDefinitionImpl(JettyConfig.FELIX_JETTY_VIRTUAL_THREADS_MAX,
+                "Virtual Threads Max Concurrent Tasks",
+                "The maximum number of virtual thread tasks that run at the same time, or -1 to leave the number unbounded. Note that unlike 'Threadpool Max' this bounds concurrent tasks, not the number of threads. Only relevant if 'Use Virtual Threads' is enabled. When set to a positive value, Jetty's preferred setup is used: a QueuedThreadPool, sized by 'Threadpool Max', with a bounded VirtualThreadPool as its virtual threads executor. Defaults to -1.",
+                -1,
+                bundle.getBundleContext().getProperty(JettyConfig.FELIX_JETTY_VIRTUAL_THREADS_MAX)));
+
         adList.add(new AttributeDefinitionImpl(JettyConfig.FELIX_JETTY_ACCEPTORS,
                 "Acceptors",
                 "Number of acceptor threads to use, or -1 for a default value. Acceptors accept new TCP/IP connections. If 0, then the selector threads are used to accept connections.",

http/jetty12/src/main/java/org/apache/felix/http/jetty/internal/JettyConfig.java

@@ -90,6 +90,9 @@ public final class JettyConfig
     /** Felix specific property to enable the use of virtual threads in Jetty */
     public static final String FELIX_JETTY_USE_VIRTUAL_THREADS = "org.apache.felix.http.jetty.virtualthreads.enable";
 
+    /** Felix specific property to bound the number of concurrent virtual thread tasks in Jetty */
+    public static final String FELIX_JETTY_VIRTUAL_THREADS_MAX = "org.apache.felix.http.jetty.virtualthreads.max";
+
     /** Felix specific property to control the number of jetty acceptor threads */
     public static final String FELIX_JETTY_ACCEPTORS = "org.apache.felix.http.jetty.acceptors";
 
@@ -498,6 +501,11 @@ public boolean isUseVirtualThreads() {
         return this.getBooleanProperty(FELIX_JETTY_USE_VIRTUAL_THREADS, false);
     }
 
+    public int getVirtualThreadsMax()
+    {
+        return getIntProperty(FELIX_JETTY_VIRTUAL_THREADS_MAX, -1);
+    }
+
     public int getRequestBufferSize()
     {
         return getIntProperty(FELIX_JETTY_REQUEST_BUFFER_SIZE, 8 * 1024);

http/jetty12/src/main/java/org/apache/felix/http/jetty/internal/JettyService.java

@@ -438,33 +438,66 @@ private void initializeJetty() throws Exception
 
     private Server createServer() throws Exception
     {
-        final int threadPoolMax = this.config.getThreadPoolMax();
-        if (!this.config.isUseVirtualThreads() && threadPoolMax >= 0) {
-            return new Server(new QueuedThreadPool(threadPoolMax));
-        } else if (this.config.isUseVirtualThreads()) {
+        final ThreadPool threadPool = createThreadPool(this.config);
+        return threadPool == null ? new Server() : new Server(threadPool);
+    }
+
+    /**
+     * Selects the Jetty thread pool for the configured combination of
+     * {@link JettyConfig#FELIX_JETTY_THREADPOOL_MAX},
+     * {@link JettyConfig#FELIX_JETTY_USE_VIRTUAL_THREADS} and
+     * {@link JettyConfig#FELIX_JETTY_VIRTUAL_THREADS_MAX}.
+     *
+     * Package private so that the selection can be asserted without starting a server.
+     *
+     * @param config The configuration
+     * @return The thread pool, or {@code null} when none is configured, in which case
+     *         Jetty's own default applies.
+     */
+    static ThreadPool createThreadPool(final JettyConfig config) throws Exception
+    {
+        final int threadPoolMax = config.getThreadPoolMax();
+        if (!config.isUseVirtualThreads()) {
+            return threadPoolMax >= 0 ? new QueuedThreadPool(threadPoolMax) : null;
+        } else {
             // See https://jetty.org/docs/jetty/12/programming-guide/arch/threads.html#thread-pool-virtual-threads
             Method newVirtualThreadPerTaskExecutorMethod = null;
             try {
                 newVirtualThreadPerTaskExecutorMethod = Executors.class.getMethod("newVirtualThreadPerTaskExecutor");
             } catch (NoSuchMethodException e) {
                 throw new IllegalArgumentException("Virtual threads are only available in Java 21 or later, or via preview flags in Java 19-20");
             }
-            if (threadPoolMax >= 0) {
+            final int maxConcurrentTasks = config.getVirtualThreadsMax();
+            if (maxConcurrentTasks > 0) {
+                // Jetty's preferred setup: a QueuedThreadPool, which keeps platform threads for
+                // the acceptors and the selectors, with a bounded VirtualThreadPool as its virtual
+                // threads executor. The VirtualThreadPool is added as a bean because
+                // setVirtualThreadsExecutor() only stores the executor, it does not manage its
+                // life cycle, and an unstarted VirtualThreadPool rejects every task.
+                QueuedThreadPool threadPool = threadPoolMax >= 0 ? new QueuedThreadPool(threadPoolMax) : new QueuedThreadPool();
+                VirtualThreadPool virtualThreadPool = new VirtualThreadPool();
+                virtualThreadPool.setMaxConcurrentTasks(maxConcurrentTasks);
+                threadPool.setVirtualThreadsExecutor(virtualThreadPool);
+                threadPool.addBean(virtualThreadPool);
+                return threadPool;
+            } else if (threadPoolMax >= 0) {
                 // Standalone VirtualThreadPool as the server's thread pool: only virtual
-                // threads, with a semaphore limiting concurrent tasks to threadPoolMax.
+                // threads, with a semaphore limiting concurrent tasks to threadPoolMax. Note
+                // that this branch predates virtualthreads.max and reuses threadpool.max as a
+                // bound on concurrent tasks rather than on threads. It is kept as is for
+                // backwards compatibility; virtualthreads.max above is the properly named
+                // equivalent.
                 VirtualThreadPool threadPool = new VirtualThreadPool();
                 threadPool.setMaxConcurrentTasks(threadPoolMax);
-                return new Server(threadPool);
+                return threadPool;
             } else {
                 // QueuedThreadPool with an unbounded virtual-threads executor: platform
                 // threads still run the acceptors and selectors, tasks run on virtual threads.
                 QueuedThreadPool threadPool = new QueuedThreadPool();
                 final Executor virtualExecutor = (Executor) newVirtualThreadPerTaskExecutorMethod.invoke(null);
                 threadPool.setVirtualThreadsExecutor(virtualExecutor);
-                return new Server(threadPool);
+                return threadPool;
             }
-        } else {
-            return new Server();
         }
     }
 

http/jetty12/src/test/java/org/apache/felix/http/jetty/internal/JettyConfigTest.java

@@ -149,6 +149,23 @@ public class JettyConfigTest
         assertEquals("string2", ((List<String>)toCheck.get("list")).get(1));
     }
 
+    @Test public void testGetDefaultVirtualThreadsMax()
+    {
+        assertEquals(-1, this.config.getVirtualThreadsMax());
+    }
+
+    @Test public void testGetVirtualThreadsMax()
+    {
+        Hashtable<String, Object> props = new Hashtable<>();
+        props.put(JettyConfig.FELIX_JETTY_VIRTUAL_THREADS_MAX, 200);
+        this.config.update(props);
+        assertEquals(200, this.config.getVirtualThreadsMax());
+
+        props.put(JettyConfig.FELIX_JETTY_VIRTUAL_THREADS_MAX, "200");
+        this.config.update(props);
+        assertEquals(200, this.config.getVirtualThreadsMax());
+    }
+
     @Before
     public void setUp()
     {