Metropolitano GCS Project

This project will move compressed csv files into a gcs bucket and unzipped them using kestra.

Source

The compressed files are in this repo. Data corresponds from January to May 25.

Build an EL pipeline

The process is represented by these tasks:

• Extract the csv.gz files from github repository
• Load into GCS bucket

Upload csv files to GCS Bucket

• Quick GCP setup
  • Create project account
  • Create your service account, give it permissions (storage.objectAdmin, storage.objectViewer) and add a key
  • Place the key in a safety place in your computer and out of the project folder. Terraform will use it to create resources safely, see infrastructure/variables.tf.
• infrastructure has all the code to create gcp resources
• Spin up Kestra

  • docker compose up -d to spin up kestra and postgres db containers. Execute this command in the folder where docker-compose.yml is placed.

  • Go to http://localhost:18080 and give credentials placed in KESTRA_CONFIGURATION inside docker-compose.yml

  • Copy scripts gcp-kv.yaml and gcp_taxi_scheduled.yalm to kestra flows via curl like this

    curl -u admin@kestra.io:Admin1234 -X POST http://localhost:18080/api/v1/flows/import \
    -F "fileUpload=@kestra/gcp_kv.yaml"
    

Note

It is important to know where you are to import the yaml files sucessfullly.

• Once all containers are up, add a key called GOOGLE_CREDENTIALS_ID in KV Store with your service account in kestra App. Copy the hole json key

• Execute .yaml scrips in kestra App

  • Execute gcp_kv.yaml to create kestra internal variables
  • Execute gcp_scheduled.yaml to perform EL

Warning

Be careful with the logs in kestra, at some point it will show your keys. Your have to pay for the security of the credentials.