Implementing best practices for PySpark ETL jobs and applications.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Mass processing data with a complete ETL for .net developers

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

Terraform modules for provisioning and managing AWS Glue resources

This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

Introduction to the data pipeline management with Airflow. Airflow schedule and maintain numerous ETL processes running on a large scale Enterprise Data Warehouse.

An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.

Example project and best practices for Python-based Spark ETL jobs and applications.

Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE

A simple in-memory, configuration driven, data processing pipeline for Apache Spark.

Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and transforms the raw data (ETL process) using Apache spark to meet business requirements and also enables Data Analyst create Data Visualization using Superset. Airflow is used to orchestrate the pipeline

A Python PySpark Projet with Poetry

Sentiment Analysis of Tweets Using ETL process and Elastic Search

Comms processing (ETL) with Apache Flink.

This repo will guide you step-by-step method to create star schema dimensional model.

Ferramenta para Cópia de Dados SQL Server, que foi desenvolvida para auxiliar na geração de arquivos e cópia eficiente de dados entre bases de dados SQL Server.

A declarative, SQL-like DSL for data integration tasks.

Optimized CSV Loader, which replaces a traditional ETL process to load huge CSV dataset to traditional databases