I wrote a scraper with requests a while back, looped it over ten thousand URLs, and watched my CPU sit at basically nothing while the thing crawled along at two pages a second. My first fix — and I’m not proud of this — was wrapping it in a ThreadPoolExecutor and firing off 500 worker threads at once. Three seconds later: OSError: [Errno 24] Too many open files. The ones that didn’t crash outright got my IP subnet flagged by the target’s WAF, and I spent the rest of that day staring at 429s.
Both failures trace back to the same misunderstanding. Network requests are bound by latency, not CPU — your processor isn’t doing anything while it waits for a packet to cross the ocean and come back, which is exactly why synchronous code is slow and exactly why brute-forcing it with more OS threads doesn’t fix the actual bottleneck. Threads cost real memory — each one reserves a meaningful chunk of stack space — and operating systems cap how many file descriptors a process can hold open at once. Throw enough threads at a scraping job and you hit that ceiling before you hit any meaningful throughput gain.
| Architecture | Concurrency model | Memory footprint | Scale | Risk |
|---|---|---|---|---|
Synchronous (requests) |
Blocking, sequential | ~10MB | Extremely low | Stalls on every packet |
Multi-threaded (ThreadPoolExecutor) |
OS threads | 1MB+ per thread | Medium | Socket/FD exhaustion |
Async (asyncio + httpx) |
Event loop, cooperative | ~50MB for 10k tasks | Massive | Only controlled by explicit semaphores |
Installing What You Actually Need #
pip install "httpx[http2]" uvloopTwo things worth flagging up front. asyncio isn’t a package — it’s been built into Python since 3.4, and pip installing it does nothing useful at best. And if you want http2=True to actually work later in this code, you need the h2 package, which only comes along if you install httpx with the http2 extra specifically — plain pip install httpx and then setting http2=True throws an ImportError the moment you try to instantiate the client. uvloop is legitimate as written — on Linux or macOS it swaps in a Cython-optimized event loop that’s genuinely 2 to 4x faster than the stdlib default, though it doesn’t run on Windows.
The Three Pillars of a Scraper That Doesn’t Fall Over #
A production-grade async scraper needs three things non-negotiably: one shared client instance instead of spinning up a new connection pool per request, a semaphore capping how many requests run simultaneously, and real exception handling with backoff instead of letting one bad response take down the whole batch.
import asyncio
import logging
import sys
from typing import List, Optional
import httpx
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[logging.StreamHandler(sys.stdout)]
)
logger = logging.getLogger("AsyncScraper")
TARGET_URLS = [
f"https://httpbin.org/delay/1?id={i}" for i in range(1, 101)
]
async def fetch_page(
client: httpx.AsyncClient,
url: str,
semaphore: asyncio.Semaphore,
max_retries: int = 3
) -> Optional[str]:
async with semaphore:
for attempt in range(1, max_retries + 1):
try:
response = await client.get(url, timeout=10.0)
response.raise_for_status()
logger.info(f"SUCCESS: Fetched {url} [Status: {response.status_code}]")
return response.text
except httpx.HTTPStatusError as e:
logger.warning(f"HTTP ERROR {e.response.status_code} on {url} (Attempt {attempt}/{max_retries})")
if e.response.status_code in [429, 500, 502, 503, 504]:
await asyncio.sleep(2 ** attempt)
else:
break
except (httpx.RequestError, httpx.TimeoutException) as e:
logger.warning(f"NETWORK ERROR on {url}: {type(e).__name__} (Attempt {attempt}/{max_retries})")
await asyncio.sleep(2 ** attempt)
logger.error(f"FAILED PERMANENTLY: Exhausted retries for {url}")
return None
async def main():
limits = httpx.Limits(max_keepalive_connections=20, max_connections=50)
timeout = httpx.Timeout(15.0, connect=5.0)
semaphore = asyncio.Semaphore(10)
async with httpx.AsyncClient(limits=limits, timeout=timeout, http2=True) as client:
tasks = [fetch_page(client, url, semaphore) for url in TARGET_URLS]
logger.info(f"Initiating batch scrape of {len(TARGET_URLS)} endpoints...")
results = await asyncio.gather(*tasks)
successful_scrapes = [r for r in results if r is not None]
logger.info(f"Batch complete. Successfully scraped {len(successful_scrapes)}/{len(TARGET_URLS)} pages.")
if __name__ == "__main__":
import uvloop
uvloop.run(main())What’s Actually Happening Under the Hood #
The httpx.Limits config matters more than it looks — max_keepalive_connections=20 and max_connections=50 mean the client pools and reuses underlying TCP connections instead of opening and tearing one down for every single request. Doing that per-request is expensive — real TCP handshake overhead, and enough of it leaves you with a pile of sockets stuck in TIME_WAIT that your OS has to clean up.
The semaphore is where the actual throttling happens. Hand asyncio.gather a list of 10,000 URLs and it instantiates 10,000 coroutines immediately — but each one has to acquire the semaphore before it does anything, so with Semaphore(10) only ten requests are ever actually in flight at once. The other 9,990 just sit there cooperatively waiting their turn, costing almost nothing while they wait.
The backoff is what keeps you from getting blacklisted. If a target starts throwing 429s or 503s, hammering it immediately again just makes things worse — 2 ** attempt gives you 2 seconds, then 4, then 8, giving the server room to recover instead of you getting permanently shadowbanned for being aggressive about it.
When the Target Renders Everything Client-Side #
Sometimes httpx gets back an empty shell because the actual content only exists after React or Vue hydrates in a real browser. At that point you need Playwright, and the common mistake is launching a brand-new Chromium instance inside every single semaphore-guarded call is the exact same anti-pattern as creating a new httpx.AsyncClient per request — expensive, wasteful, and it throws away the whole point of connection/resource reuse. Launch the browser once, outside the loop, and spin up a fresh context per task instead:
import asyncio
from playwright.async_api import async_playwright
async def scrape_spa(browser, url: str, semaphore: asyncio.Semaphore):
async with semaphore:
context = await browser.new_context()
page = await context.new_page()
try:
await page.goto(url, timeout=30000)
await page.wait_for_selector("#app-root", timeout=10000)
content = await page.content()
print(f"Successfully rendered SPA: {url}")
return content
except Exception as e:
print(f"SPA Render Failed for {url}: {e}")
return None
finally:
await context.close()
async def scrape_all_spas(urls: list[str]):
semaphore = asyncio.Semaphore(3)
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
try:
tasks = [scrape_spa(browser, url, semaphore) for url in urls]
return await asyncio.gather(*tasks)
finally:
await browser.close()One browser process, many lightweight contexts — each context is isolated (separate cookies, storage) without paying the cost of a full new Chromium launch every time. And watch the memory: a single headless browser instance can eat 200–500MB of RAM, so keep that semaphore low — 3 to 5 concurrent, not 10 — or you’ll run your host out of memory long before you run out of URLs.