Skip to content

Latest commit

 

History

History

etl

README.md

ETL Service

ETL for data transfer from PostgreSQL to Elasticsearch.

Run in docker

docker compose --profile infra --profile etl up -d

Run locally

  1. Prepare python environment

  2. Prepare environment variables

cp .env.example .env
  1. Run infrastructure in Docker:
docker compose --profile infra up -d
  1. Run ETL service:
make up

Prepare python environment

  1. If you don't have Poetry installed run:
make poetry-download
  1. Initialize poetry and install pre-commit hooks:
make install
make pre-commit-install
  1. Run formatters and linters. Make sure there is no errors.
make format lint

Makefile usage

Makefile contains a lot of functions for faster development.

1. Download and remove Poetry

To download and install Poetry run:

make poetry-download

To uninstall

make poetry-remove

2. Install all dependencies and pre-commit hooks

Install requirements:

make install

Pre-commit hooks could be installed after git init via

make pre-commit-install

3. Codestyle

Automatic formatting uses black and ruff.

make codestyle

# or use synonym
make format

Codestyle checks only, without rewriting files:

make check-codestyle

Update all dev libraries to the latest version using one command

make update-dev-deps

4. Code security

This command identifies security issues with Safety:

make check-safety

To validate pyproject.toml use

make check-poetry

5. Linting and type checks

Run static linting with ruff and mypy:

make static-lint

6. Tests with coverage

Run tests:

make test

7. All linters

Of course there is a command to rule run all linters in one:

make lint

8. Docker 

make docker-build

which is equivalent to:

make docker-build VERSION=latest

Remove docker image with

make docker-remove

More information about docker.

9. Cleanup

Delete pycache files 

make pycache-remove

Remove package build

make build-remove

Delete .DS_STORE files

make dsstore-remove

Remove .mypycache

make mypycache-remove

Or to remove all above run:

make cleanup

Credits

This project was generated with python-package-template