cloud-run-jobs

Execute a Cloud Run job using Workflows

This sample demonstrates how to execute a Cloud Run job from a workflow.

Create a Cloud Run job

This sample uses the Cloud Run jobs parallel processing demo.

Clone the Cloud Run jobs demo app repository:

git clone https://github.com/GoogleCloudPlatform/jobs-demos.git

Change to the directory that contains the sample code:

cd jobs-demos/parallel-processing

Create the Cloud Run job by running the deployment script:

./deploy-parallel-job.sh

The job processes data input files in a Cloud Storage bucket gs://input-[PROJECT_ID].

Deploy the workflow

Create a workflow.yaml file with the provided workflow definition.

The sample workflow accepts a Cloud Storage event as a parameter and uses the event to determine whether to execute the Cloud Run job. If the Cloud Storage object specified in the event is the input data file used by the job, the workflow executes the job. Otherwise, the workflow terminates.

Deploy the workflow:

gcloud workflows deploy cloud-run-job-workflow \
    --location=us-central1 \
    --source=workflow.yaml

Create a trigger

Create an Eventarc trigger that executes the workflow whenever a file is uploaded or overwritten in the Cloud Storage bucket containing the input data file:

PROJECT_ID=$(gcloud config get project)
PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
gcloud eventarc triggers create cloud-run-job-workflow-trigger \
    --location=us \
    --destination-workflow=cloud-run-job-workflow  \
    --destination-workflow-location=us-central1 \
    --event-filters="type=google.cloud.storage.object.v1.finalized" \
    --event-filters="bucket=input-$PROJECT_ID" \
    --service-account="$PROJECT_NUMBER[email protected]"

Trigger the workflow

Test the end-to-end system by updating the input data file in Cloud Storage:

base64 /dev/urandom | head -c 100000 >input_file.txt
gsutil cp input_file.txt gs://input-$PROJECT_ID/input_file.txt

Confirm that the Cloud Run job ran as expected by viewing the job executions:

gcloud run jobs executions list --job=parallel-job