isolate prefect resources by environment

[solomon-negusse]

This PR aims to isolate environment-specific resources/config by (a) suffixing Prefect webhook/automation identifiers per Terraform workspace, and (b) allowing New Relic agent configuration to vary by environment. It also adds a small Click-based CLI for running pipeline update flows locally.

Changes:

Add new_relic_environment Terraform variable and pass it into the API ECS task env.
Suffix Prefect webhook/event/automation names by workspace (local.name_suffix) to avoid cross-environment collisions.
Introduce a Click CLI wrapper for pipelines/run_updates.py and document local + Prefect Cloud execution.

[Copilot]

Pull request overview

This PR aims to isolate environment-specific resources/config by (a) suffixing Prefect webhook/automation identifiers per Terraform workspace, and (b) allowing New Relic agent configuration to vary by environment. It also adds a small Click-based CLI for running pipeline update flows locally.

Changes:

• Add new_relic_environment Terraform variable and pass it into the API ECS task env.
• Suffix Prefect webhook/event/automation names by workspace (local.name_suffix) to avoid cross-environment collisions.
• Introduce a Click CLI wrapper for pipelines/run_updates.py and document local + Prefect Cloud execution.

Reviewed changes

Copilot reviewed 8 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
terraform/vars/staging.tfvars	Adds staging override for new_relic_environment.
terraform/variables.tf	Defines new new_relic_environment variable (default production).
terraform/main.tf	Injects NEW_RELIC_ENVIRONMENT into the API task definition; adjusts dask worker container name.
terraform/prefect.tf	Suffixes Prefect webhook/event/automation names by workspace; gates automation enablement.
api/newrelic.ini	Enables New Relic monitoring + distributed tracing for the staging environment.
pipelines/run_updates.py	Adds Click CLI entrypoint and options for selecting update flow/version/flags.
pipelines/pyproject.toml / pipelines/uv.lock	Adds click dependency.
pipelines/README.md	Expands documentation for running tests, local runs, and Prefect Cloud architecture/usage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[danscales]

Looks great! Thanks for updating the README.md with all the relevant info and configurations!