can't start new thread: 186 companies looked dead

On 1 September 2026 the production board showed a fetch problem on most companies. It looked like employer career sites, or our scrapers, had died. They had not. The EC2 fetch worker could not start OS threads, so every six-hour country cycle failed in about a second and stamped a sticky “fetch problem” flag on the catalog.

The catalog went stale from 30 August onward. 186 of 394 companies were flagged. Germany and the Netherlands took most of the hits (85 and 84). Last healthy country runs: 29 August 2026.

What we saw

The worker container was still “Up.” The six-hour loop still ran. Cycles “finished,” so nothing in the process table looked wedged. The board then went red.

111/111 is bookkeeping, not a scrape. The pool dies immediately; finalize still sets progress to total. Same-second timestamps across countries are sequential scheduling: each country dies in under a second, so a 16-country cycle completes in about two seconds.

First error: 28 August 12:50 UTC. Spike: 29 August 07:00 — 173 errors in one hour. Every scheduled country run from 30 August onward failed in about a second.

What it was not

Three wrong explanations were easy to reach for:

• Employer ATS outages. The error string was the same for almost every company. Career sites do not fail in lockstep like that.
• “Python asyncio uses too much RAM.” Coroutines are cheap. OS threads (~8 MiB stacks) and Chromium are not.
• A reason to rewrite the worker in Go. HTTP clients would get cheaper. The browser fallback would cost the same. The bug was the concurrency model, not the language.

We run the worker on a t4g.micro. Low RAM is the constraint, not the bug. A bigger box would have delayed the same failure.

Nested threads, nested event loops

Fetch already runs in a separate container. The code is not a separate system: the worker imports the same fetch, scrape, and catalog packages as the panel and writes the same Postgres.

The model on 1 September, simplified:

scheduler (one country at a time)
  → OS thread for the country
    → asyncio.run  (event loop #1)
      → ThreadPoolExecutor(max_workers=4)
        → asyncio.run per company  (event loop #2)
          → HTTP to the career board
          → sometimes Playwright Chromium on yet another thread

That is thread-per-company with a nested event loop, not a coroutine pool. Production default was FETCH_SCHEDULE_CONCURRENCY=4. Each of those four workers also sized HTTP limits up to 16.

The sequential path (workers <= 1) already did the right thing: await each company on the country loop. Concurrency 4 never used that path. It opened a ThreadPoolExecutor and called asyncio.run() again inside each worker.

# per company, inside the pool — the bug
async def _inner():
    async with make_fetch_client(concurrency=http_concurrency) as client:
        return await asyncio.wait_for(
            fetch_and_persist_company(...),
            timeout=company_timeout_seconds(),
        )

msg, new_count = asyncio.run(_inner())

RuntimeError: can't start new thread means pthread_create failed — typically ENOMEM or a PID/nproc limit. It does not mean “asyncio RAM is uncontrollable.”

Some career pages have no JSON API. Those fall through to a sync Chromium scrape via asyncio.to_thread. One in-flight generic company could be: country thread + pool thread + nested event loop + extra thread + a browser. That is the RAM and PID amplifier. It is also why the process was already unable to start threads after days of cycles. On 1 September the pool died before boards ran, so the exception string is the thread error, not a hung browser.

A July hang had already taught us to put timeouts on every blocking boundary. Those timeouts do not bound how many OS threads and browsers exist. This incident is that next failure mode.

Why the board said the company was broken

Any attempt message matching — Error: … set a sticky catalog flag. There was no distinction between “this career board returned garbage” and “this process cannot create a thread.” Operators then debug Greenhouse or Ashby instead of the worker.

Runs also looked complete. companies_done == companies_total, the scheduler logged a finished cycle, duration was ~0s, new_jobs was 0. The stats table does not show error_message.

Application logs in Docker are not an archive: they vanish on docker rm -f. Metrics (disk, RAM, health) did not store fetch errors. Durable evidence was Postgres: per-company attempt rows and country-run summaries. Grouping error_message was enough.

SELECT LEFT(error_message, 180), COUNT(*)
FROM company_fetch_attempts
WHERE status = 'error'
GROUP BY 1
ORDER BY 2 DESC;

What we shipped

On 2 September, country fetch stopped starting a ThreadPoolExecutor or calling asyncio.run() per company. The outer country thread remains so the scheduler can join it. Companies run on that thread’s one event loop, bounded by asyncio.Semaphore, sharing one HTTP client.

scheduler (one country at a time)
  → OS thread for the country
    → asyncio.run  (one loop)
      → asyncio.Semaphore(workers)   default 2
        → await fetch_and_persist_company(shared client)
          → HTTP; Playwright capped at 1 browser

• Default FETCH_SCHEDULE_CONCURRENCY is 2, not 4. Do not raise it on a t4g.micro without watching RSS.
• Infra errors such as can't start new thread no longer set the sticky company flag.
• A failed run no longer marks progress as fully “done.”

We did not rewrite the worker in Go. Fetch is already a separate container. Splitting languages without a job contract would duplicate the ATS parsers and the scrape suite. A queued job table is still on the table; it was not this outage’s fix.

The rule

Timeouts bound how long a unit of work may run. They do not bound how many threads, event loops, and browsers you create while it runs. On a small box, asyncio.run() inside a thread pool is a way to spend the process before the timeout ever fires.

One event loop per country. A semaphore for in-flight companies. Playwright at one. Treat “cannot start a thread” as infrastructure, not as a broken employer.