Scrapy, a fast high-level web crawling & scraping framework for Python.

Elegant Scraper and Crawler Framework for Golang

Auto_Jobs_Applier_AI_Agent by AIHawk is an AI Agent that automates the jobs application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in an automated and personalized way.

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Pythonic HTML Parsing for Humans™

A scalable web crawler framework for Java.

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Tabula is a tool for liberating data tables trapped inside PDF files

List of libraries, tools and APIs for web scraping and data processing.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Declarative web scraping

Distributed crawler powered by Headless Chrome

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Mechanize is a ruby library that makes automated web interaction easy.

Collection of useful data science topics along with articles, videos, and code

Up-to-date simple useragent faker with real world database

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!