Scraping web site to save to DB in Docker
This sample uses iHerb as target site.
-
Docker
-
Docker Compose
-
MariaDB
-
Python
-
Django
-
Scrapy
docker-compose up -d --build
./start.sh
Default user is
USERNAME: root
PASSWORD: initpass
Specify Environmental Valiable by docker-compose for Scrapy Configuration following
- SCRAPY_START_INDEX: 22419
- SCRAPY_NUM_ITEMS: 1000
- SCRAPY_CONCURRENT_ITEMS: 10000
- SCRAPY_CONCURRENT_REQUESTS: 2
- SCRAPY_DOWNLOAD_DELAY: 8.0