What is Scrapling?
Scrapling is an open-source Python web scraping framework created by Karim Shoair that brings HTTP requests, stealth browsers, parsing and concurrent crawling into one library. It scrapes static pages, renders JavaScript, interacts with websites and extracts data from the API calls behind them. Crawlers can use reusable sessions, proxy rotation, automatic retries, pause and resume, and built-in exports to JSON, JSONL, CSV and XML.
Fetching is split between a Fetcher class for HTTP requests that can impersonate a browser's TLS fingerprint and headers and use HTTP/3, and a StealthyFetcher for full browser automation with Playwright's Chromium or Google Chrome, fingerprint spoofing and Cloudflare Turnstile handling. The parser supports CSS, XPath, text and regex search, can relocate elements after a website changes, and outputs clean Markdown for RAG pipelines. The spider API follows Scrapy conventions, and a decorator lets existing Scrapy callbacks use the Scrapling parser.
Scrapling requires Python 3.10 or higher and installs with pip. The base package contains only the parser, and browser fetchers need the fetchers extra plus the scrapling install command. A Docker image with all extras and browsers is published on Docker Hub and the GitHub registry. Scrapling also runs from the terminal, in an interactive shell, or for AI agents through its MCP server and Agent Skill. The project is licensed under BSD-3-Clause.
Scrapling screenshots
Explore the interface. Open a screenshot to view it at full size.
Source: official Scrapling project page.
Key capabilities
- Fetchers Fetcher for fast HTTP requests with browser TLS fingerprint impersonation and HTTP/3, and StealthyFetcher for browser automation with fingerprint spoofing.
- Sessions and proxies FetcherSession and StealthySession keep cookies and state across requests, and ProxyRotator rotates proxies across all session types.
- Spiders Scrapy-like spiders with async parse callbacks, concurrency limits, per-domain throttling, pause and resume through a crawl directory, and item streaming.
- Spider templates CrawlSpider, SitemapSpider, XML and CSV feed spiders, and a ShopifySpider for public Shopify stores.
- Exports Built-in exporters for JSON, JSONL, CSV and XML, or your own pipeline through hooks.
- CLI and shell Extract commands that save pages as text, Markdown or HTML without code, and an IPython-based scraping shell.
- AI tooling An MCP server with seventeen tools for AI agents, and an Agent Skill that teaches coding agents the library.
Where Scrapling fits
- Adaptive parsing Elements are relocated after website changes using similarity algorithms, alongside CSS, XPath, text and regex selection.
- Blocking and politeness Blocked requests are detected and retried, AutoThrottle tunes the delay per domain, and an optional flag obeys robots.txt.
- Development mode Responses are cached to disk on the first run and replayed later, so parse logic can be tested without hitting the target again.
- Remote and custom browsers Fetchers can connect to a running browser over CDP or use your own Chromium build.
- Background API capture Browser sessions can record requests and search the saved responses afterwards.
- Network controls Optional DNS-over-HTTPS prevents DNS leaks behind proxies, and built-in ad blocking covers about 3,500 ad and tracker domains.
Install
pip install scraplingReviews of Scrapling
Be the first to share your experience with Scrapling.
Write a review
Loading review form…
Alternatives to Scrapling
AutomaAutoma is a Chrome and Firefox extension that automates browser work with a block editor and a recorder. It handles form filling, scraping to tables, scheduled runs and Google Sheets export, with AGPL source on GitHub.CCamoufoxCamoufox is an open source anti-detect browser built for web scraping and AI agents, intended for use with rotating, preferably residential, proxies.
Crawl4AICrawl4AI is an Apache-2.0 Python crawler built on Playwright that outputs Markdown, JSON, screenshots and PDFs for RAG and agents. Run it as a library, a Docker server with REST and MCP, or use the hosted Crawl4AI Cloud API.