Welcome to the Dataflow Solution Guides!
The Dataflow Solution Guides offer full end-to-end deployment for the most common streaming solutions to run on Dataflow.
This repository contains the following assets for each guide:
- Full Terraform code to spawn all the necessary Google Cloud infrastructure
- Pipelines code in Python, Java and Go (coming soon) for a sample pipeline for each use case
This the list of solution guides available at this moment:
| Guide | Description | Development status |
|---|---|---|
| GenAI & Machine Learning Inference | Real-time inference with local GenAI models, using a GPU | Ready ✅ |
| ETL / Integration | Real-time change data capture from a Spanner database to BigQuery | Ready ✅ |
| Log Replication & Analytics | Real-time log replication into Splunk | Beta 🏭 |
| Marketing Intelligence | Real-time marketing intelligence, using an AutoML model deployed in Vertex | Beta 🏭 |
| Clickstream Analytics | Real-time clickstream analytics with Bigtable enrichment / data hydration | Work in progress 🔨 |
| IoT Analytics | Real-time Internet of Things (IoT) analytics with Bigtable enrichment & models deployed in Vertex AI | Work in progress 🔨 |
| Anomaly Detection | Real-time detection of anomalies in a stream of data leveraging GenAI with models deployed in Vertex AI | Beta 🏭 |
| Ads Analytics | Real-time bidding strategies for programmatic ad buying augmented with predictive ML models | Beta 🏭 |
| Customer Data Platform | Real-time customer data platform that unifies a customer view from different sources. | Beta 🏭 |
| Gaming Analytics | Real-time analyis of gaming data to enhance live gameplay & offer targeting | Beta 🏭 |
terraform: This directory contains the Terraform code for deploying the necessary Google Cloud infrastructure for each use case.pipelines: This directory contains the Python, Java, and Go code for the sample pipelines.use_cases: This directory contains the documentation of each use case
- GitHub Issues: Report any issues or ask questions on the GitHub repository.
- Stack Overflow: Search for existing solutions or ask questions on Stack
Overflow using the
google-cloud-dataflowtag:
Your contributions to this repository are welcome.
- Fork and Pull Request: Fork the repository and submit a pull request with your changes.
- Follow the Contribution Guidelines: Please follow the contribution guidelines outlined in the CONTRIBUTING.md file.
This is not an officially supported Google product. The code in this repository is for demonstrative purposes only.