Skip to content

omarismail94/RandomPipes

Repository files navigation

Upload Template to Google Storage

mvn compile exec:java
-Dexec.mainClass=org.omar.$CLASS_NAME
-Dexec.cleanupDaemonThreads=false
-Dexec.args="
--project=$PROJECT_ID
--stagingLocation=gs://$BUCKET_NAME/staging
--tempLocation=gs://$BUCKET_NAME/temp
--templateLocation=gs://$BUCKET_NAME/templates/$TEMPLATE_NAME.json
--runner=DataflowRunner"

Execute on Dataflow

If Template Exists on Google Storage

gcloud dataflow jobs run $JOB_NAME
--gcs-location=gs://$BUCKET_NAME/templates/$TEMPLATE_NAME.json
--region=us-central1
--parameters

To Compile and Run on Dataflow

mvn compile exec:java
-Dexec.mainClass=org.omar.$CLASS_NAME
-Dexec.args="
--project=$PROJECT_ID
--stagingLocation=gs://$BUCKET_NAME/staging
--tempLocation=gs://$BUCKET_NAME/temp
--runner=DataflowRunner
-- "

To Run Locally

mvn clean compile exec:java -Dexec.mainClass=org.omar.$CLASS_NAME

About

Java implementations of the transforms listed in https://beam.apache.org/documentation/programming-guide/

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published