This is a tool to perform a high number of reads to a Bigtable table for demonstration purposes. It is deployed as a Dataflow template, so it can easily be run as a Dataflow job.

1. Set your environment variables

   TEMPLATE_PATH="gs://cloud-bigtable-dataflow-templates/generate-workload.json"
   INSTANCE_ID=YOUR-INSTANCE-ID
   TABLE_ID=YOUR-TABLE-ID
   REGION=us-central1
   WORKLOAD_RATE=100 # Optional, default to 1000
   WORKLOAD_DURATION=15 # Optional, default to 10
   

2. Run this command to start a job from dataflow template:

   JOB_NAME="generate-bigtable-workload"
   gcloud dataflow flex-template run $JOB_NAME \
   --template-file-gcs-location "$TEMPLATE_PATH" \
   --parameters bigtableInstanceId="$INSTANCE_ID" \
   --parameters bigtableTableId="$TABLE_ID" \
   --region "$REGION" \
   --parameters workloadRate=$WORKLOAD_RATE
   --parameters workloadDurationMinutes=$WORKLOAD_DURATION
   

These instructions are for maintenance of the workload generator, but if you would like to modify this example and deploy the template yourself, you can follow them to do so.

1. Build the project

   mvn clean package -DskipTests
   

2. Set the environment variables. To deploy a version on your project, update these with your own resources as described in the Using Flex Templates documentation.

   export TEMPLATE_PATH="gs://cloud-bigtable-dataflow-templates/generate-workload.json"
   export TEMPLATE_IMAGE="gcr.io/cloud-bigtable-ecosystem/dataflow/generate-workload:latest"
   export LOGS_PATH="gs://cloud-bigtable-dataflow-templates-logs/workload-generator"
   

3. Deploy the template

   gcloud dataflow flex-template build $TEMPLATE_PATH \
   --image-gcr-path "$TEMPLATE_IMAGE" \
   --sdk-language "JAVA" \
   --flex-template-base-image JAVA11 \
   --metadata-file "metadata.json" \
   --jar "target/workload-generator-0.1.jar" \
   --env FLEX_TEMPLATE_JAVA_MAIN_CLASS="bigtable.WorkloadGenerator" \
   --gcs-log-dir="$LOGS_PATH"
   

   Note: Make sure your account or service account has cloudbuild and storage permissions.

If you would like to modify this and run it yourself you can use these commands:

1. Create a Bigtable instance and table

2. Set up the environment variables

   GOOGLE_CLOUD_PROJECT=your-project-id
   INSTANCE_ID=your-instance-id
   REGION=us-central1
   TABLE_ID=your-table-id 
   WORKLOAD_RATE=100 # Optional
   WORKLOAD_DURATION=15 # Optional, default to 10
   

3. Run the command

   mvn compile exec:java -Dexec.mainClass=WorkloadGenerator \
   "-Dexec.args=--bigtableInstanceId=$INSTANCE_ID =--bigtableTableId=$TABLE_ID \
   --runner=dataflow --project=$GOOGLE_CLOUD_PROJECT \
   --region=$REGION" \
   --workloadRate=$WORKLOAD_RATE 
   --workloadDurationMinutes=$WORKLOAD_DURATION