Deployment-specific automation with flow run state template?

[TheoMathurin]

When adding an automation with a flow run state template, one has to choose a flow but cannot select a specific deployment. Therefore the automation would apply to flow runs belonging to all deployments of a flow.

In my case I need to run a specific deployment of my flow, based on the deployment to which the trigger flow run belongs.

If I'm not mistaken, this is not possible right now. There may be fundamental reasons why that could not be. But if it's possible in principle, I think it would be a valuable addition to Prefect's automations.

[zzstoatzz]

hi @TheoMathurin

When adding an automation with a flow run state template, one has to choose a flow but cannot select a specific deployment.

You should be able to go into the JSON tab in the UI's Automation builder and use match_related to restrict the trigger to deployments associated with the selected resource (e.g. flow)

programmatically, this should be possible the same way.

is this helpful?

[bdalpe]

Hi @TheoMathurin! Prefect does support this, but you'll need to use a "Custom" trigger template.

Expand the Evaulation Options accordion and select the deployment in the Filter for events related to multi-select form field.

---

Here's a longer, more detailed explanation of @zzstoatzz's answer:

A flow run state event will have "related" resources associated with it. An example of a flow run state change message will look something like this (with a few fields omitted):

{
  ...
  "resource": {
    "prefect.resource.id": "prefect.flow-run.925eacce-7fe5-4753-8f02-77f1511543db",
    "prefect.resource.name": "cute-kittiwake"
  }
  "related": [
    {
      "prefect.resource.id": "prefect.flow.cb6126db-d528-402f-b439-96637187a8ca",
      "prefect.resource.role": "flow",
      "prefect.resource.name": "hello"
    },
    {
      "prefect.resource.id": "prefect.deployment.37ca4a08-e2d9-4628-a310-cc15a323378e",
      "prefect.resource.role": "deployment",
      "prefect.resource.name": "example"
    },
    {
      "prefect.resource.id": "prefect.work-pool.a67da06b-9427-4a0f-a709-c3b37ac29cbf",
      "prefect.resource.role": "work-pool",
      "prefect.resource.name": "k8s-pool",
      "prefect.work-pool.type": "kubernetes"
    },
    {
      "prefect.resource.id": "prefect.tag.examples",
      "prefect.resource.role": "tag"
    }
  ]
  ...
}

Note the second resource in the related array that describes the deployment ID the event's resource is associated with.

So, to build an automation to run another deployment based on an "upstream" deployment's flow status changing to completed, you can use the match_related field to filter for your flow run completed events for a specific deployment.

It would look something like this in your automation JSON configuration:

...
"match": {
  "prefect.resource.id": [
    "prefect.flow-run.*"
  ]
},
"match_related": {
  "prefect.resource.id": "prefect.deployment.<upstream-deployment-guid>"
},
"expect": [
  "prefect.flow-run.Completed"
],
"for_each": [
  "prefect.resource.id"
],
...

Then, select your deployment you want to run as the action.

Here's a full automation YAML example you could use too:

automations:
  - name: Run downstream flow
    description: Runs a downstream flow when the upstream flow completes
    enabled: true
    trigger:
      type: event
      posture: Reactive
      expect:
        - prefect.flow-run.Completed
      match:
        - prefect.resource.id: "prefect.flow-run.*"
      match_related:
        - prefect.resource.id: "prefect.deployment.<upstream-deployment-guid>"
      threshold: 1
      for_each:
        - prefect.resource.id
    actions:
      - type: run-deployment
        deployment_id: "<downstream-deployment-guid>"
        parameters: {}
        job_variables: {}

Deploy this using prefect automation create --from-file automation.yaml

Take a look at these resources for some more information:

• How to chain deployments with events
• Creating automations
• Define event triggers

[TheoMathurin]

Ok thanks! I had not looked at the custom automation builder. I have not yet tried but it should be ok then.

As an aside, I realise that in the dropdown widgets in the custom builder, deployments appear with their short name, and not their full name (<FLOW_NAME>/<DEPLOYMENT_NAME>). Because I have multiple deployments that share the same deployment name (prod/preprod etc) and are only distinguished when considering the flow name, I see several deployments with the same name and cannot know which is which.

Is it worth adding a feature request to change how deployments appear in these dropdowns, or is it such bad practice to name deployments in this manner that it should never happen?

[bdalpe]

As an aside, I realise that in the dropdown widgets in the custom builder, deployments appear with their short name, and not their full name (<FLOW_NAME>/<DEPLOYMENT_NAME>). Because I have multiple deployments that share the same deployment name (prod/preprod etc) and are only distinguished when considering the flow name, I see several deployments with the same name and cannot know which is which.

IMO, it's a fair point worthy of a UI enhancement. Since you can have multiple deployments with the same name associated to the same flow, it's hard to differentiate. Maybe we can put the GUIDs in parenthesis?

Is it worth adding a feature request to change how deployments appear in these dropdowns, or is it such bad practice to name deployments in this manner that it should never happen?

I'd encourage you to use unique names for the previous point and for best practices, but it's not required.

[bdalpe]

Created #18758 for tracking.

[TheoMathurin]

By GUID you mean the deployment ID?

It would certainly cover every situation, even deployments with the same name associated with the same flow. But I wonder why you would want to do this.

In my case deployments do not have the same (full) name because they're associated with different flows, e.g. flow1/prod and flow2/prod. This was actually a suggestion by Marvin when I asked the question about whether the deployment name should repeat the flow name!