| Test, Build and Deploy |  |
| Coverage |  |
| Embark on an exploration of Ogre City apartments for sale historical data here | http://propertydata.lv/ |
Purpose: This application will scrape daily ss.lv website from apartments for sale category in specific city of your choice and store scraped data in postgres database and will send daily email with report.
# docker -v
Docker version 20.10.11, build dea9396
# docker-compose -v
Docker Compose version v2.2.1
- Clone repository
- Create database.ini here is example
[postgresql]
host=<your docker db hostname>
database=<your db name>
user=<your db username>
password=<your db password>
- Create .env.prod file for docker compose
# ws_worker container envs
[email protected]
SENDGRID_API_KEY=<Your SENDGRID API Key>
[email protected]
POSTGRES_PASSWORD=<Your DB Password>- Run docker-compose --env-file .env.prod up -d
make
help 💬 This help message
all runs setup, build and up targets
setup gets database.ini and .env.prod and dowloads last DB bacukp file
build builds all containers
up starts all containers
down stops all containers
clean removes setup and DB files and folders
lt Lists tables sizes in postgres docker allows to test if DB dump was restored correctly- Scrape ss.lv website to extract advert data from Ogre city apartments for sale section
- Store scraped data in postgres database container tables listed_ads and removed_ads for tracking longer price trends
- Daily email is sent which includes advert URLs and key data categorized by room count
- Email contains pdf attachment with basic price analytics for categorized by room count
- Fully automated deployment for dev branche with Github Actions CICD to AWS EC2
- Add tests and test coverage step in CICD and in README.md
- Add WEB service functionality for data explore using Pygwalker and Streamlit
- Add Streamlit web service to CICD
- Add doc and doc coverage step in CICD and in README.md