Skip to content
#

web-scraping

Star

Here are 5,771 public repositories matching this topic...

Language: All
Filter by language
All 5,771 Python 2,802 Jupyter Notebook 1,242 JavaScript 461 HTML 256 TypeScript 167 R 119 Java 95 Go 88 PHP 59 Ruby 40
Sort: Most stars
Sort options
Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated
changedetection.io

dgtlmoon / changedetection.io

Star 19k

The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification

notifications monitoring self-hosted web-scraping website-monitor url-monitor change-alert change-detection changedetection website-change-monitor website-change-tracker website-monitoring change-monitoring website-watcher website-change-detector restock-monitor website-change-detection website-change-notification back-in-stock website-defacement-monitoring
  • Updated Nov 10, 2024
  • Python
crawlee

apify / crawlee

Star 15.5k

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright
  • Updated Nov 10, 2024
  • TypeScript
Douyin_TikTok_Download_API

Evil0ctal / Douyin_TikTok_Download_API

Sponsor
Star 9.3k

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper
  • Updated Sep 26, 2024
  • Python
SeleniumBase

seleniumbase / SeleniumBase

Star 5.4k

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.

python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin behave bot-detection unittests web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python
  • Updated Nov 9, 2024
  • Python

apify / crawlee-python

Star 4.5k

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
  • Updated Nov 11, 2024
  • Python

Improve this page

Add a description, image, and links to the web-scraping topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the web-scraping topic, visit your repo's landing page and select "manage topics."

Learn more