How do you size and configure a thread pool for a backend service?

How ThreadPoolExecutor grows. Per the Javadoc: if fewer than corePoolSize threads are running, a new task gets a new thread. Otherwise the task is queued. Only if the queue is full does the pool add threads up to maximumPoolSize. If both are full, the RejectedExecutionHandler runs. Consequence: with an unbounded queue, the pool never grows past the core size, and work piles up in memory.

Sizing. For CPU-bound work, start near the number of cores. For work that waits on I/O, more threads help, roughly in proportion to how long each task waits versus computes. Treat any formula as a starting point and measure throughput, latency and queue depth under load. Separate pools for unrelated work, such as calls to two different downstreams, stop one slow dependency from starving the other.

Avoid unbounded defaults. Executors.newCachedThreadPool() is documented as an unbounded thread pool, and newFixedThreadPool uses an unbounded queue. Both can turn a traffic spike into thousands of threads or a huge queue.

Rejection policies. AbortPolicy, the default, throws RejectedExecutionException; CallerRunsPolicy runs the task in the submitting thread, which slows producers down; DiscardPolicy and DiscardOldestPolicy drop work and are only acceptable when losing tasks is fine.

import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

public class PoolConfig {
    public static void main(String[] args) throws InterruptedException {
        int cores = Runtime.getRuntime().availableProcessors();
        ThreadPoolExecutor pool = new ThreadPoolExecutor(
            cores, cores * 4,                               // core and max threads (tune from measurements)
            30, TimeUnit.SECONDS,                           // idle threads above core size time out
            new ArrayBlockingQueue<>(100),                  // bounded queue: backpressure instead of unbounded memory
            new ThreadPoolExecutor.CallerRunsPolicy());     // saturated: the caller runs the task and slows down
        AtomicInteger done = new AtomicInteger();
        for (int i = 0; i < 500; i++) pool.execute(() -> {
            try { Thread.sleep(2); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
            done.incrementAndGet();
        });
        pool.shutdown();                                    // stop accepting, finish queued work
        if (!pool.awaitTermination(10, TimeUnit.SECONDS)) pool.shutdownNow(); // interrupt stragglers
        System.out.println("done=" + done.get());          // 500: nothing was dropped
    }
}

Verified with OpenJDK 21.0.13: it prints done=500, because CallerRunsPolicy slowed the producer instead of dropping tasks.

Shutdown. shutdown() stops new submissions and lets queued work finish; awaitTermination waits; shutdownNow() interrupts what is still running. Tasks must check interruption or they never stop.

Virtual threads. For many concurrent blocking calls, Java 21 virtual threads remove much of the sizing problem, but you still need limits on downstream concurrency, for example a semaphore per dependency. See virtual threads.

Alternate interview scenarios

"Runnable or Callable: which do you submit to a pool?" Callable is "similar to Runnable", as its Javadoc puts it, but returns a result and can throw a checked exception; Runnable does neither. submit accepts both and returns a Future, and execute accepts only Runnable. In both cases an exception is captured by the Future when you use submit, so someone must call get or it is lost; see An async task throws, but nothing is logged.