Scraperai

ScraperAI's core insight is that selector-writing is the tedious part of scraping, and that an LLM is a reasonable tool for the job if you constrain the problem. Instead of pointing a model at raw HTML, it minifies the markup first, then asks the model to identify the data and the XPATHs that reach it. It also checks page type and pagination, so the generated config knows whether it is looking at a listing page or a single article, and whether there is a next page to follow. The result is a config file you keep. That serialization is the differentiator. A one-off scrape in a chat window dies with the conversation; a ScraperAI config can be re-run next month, committed to a repo, or handed to a teammate. For anyone who scrapes the same handful of sites on a recurring basis, that is the whole value proposition. The architecture is split into four documented parts — LLMs, Crawler, ParserAI and Scraper — which lets you swap pieces. ChatGPT is supported out of the box, and custom LLM endpoints and custom crawlers are supported too, with Selenium support for pages that need a real browser. Because you supply the LLM key, you pay your provider directly for inference. Where it thins out: project maturity. Version 0.0.2, documentation limited to a short docs site with a demo link, and support that runs through GitHub issues. There is no scheduler, no proxy pool, no hosted infrastructure — you supply the machine and the network. It is also a CLI, so the 'no-code' framing only holds once you are comfortable in a terminal. For high-frequency scraping, per-page LLM calls and your provider's rate limits become the binding constraint, and you should think about whether every run needs model inference at all once the config exists.