Aborted bluegreen rollout preview service behaviour

[nhs-work]

Not sure if this is best suited under discussions or issues but would like to better understand the current behaviour and intended usage of the preview service, in particular for bluegreen rollout aborts.

I have observed that upon making a new deployment, if it fails the AnalysisRun for Pre Promotion Analysis, Argo Rollouts will help to not promote the preview service but it's rollouts-pod-template-hash will remain pointing at the now failed hash value instead of pointing to the last stable replicaset's hash (which is how the active service behaves). This appears to be due to the difference in implementation of reconcilePreviewService, which does not contain any status checks for an aborted status like reconcileActiveService.

Is this behaviour of remaining at the failed hash an intended behaviour and is there some reason for this? My project currently runs pre-promotion integration tests for a few different apps against the preview services and having preview service selectors that do not exist on any pods will result in test failures.

[sglre6355]

I'm facing the exact same issue in a similar scenario. I think it's reasonable to expect the preview service to point back to the stable version when an AnalysisRun fails, especially given the preview usually scales down unless explicitly configured not to. Looking at the commit history and the docs, the current behavior seems to be intentional though (for example, it says "The Rollout always makes sure that the preview service is sending traffic to the newest ReplicaSet" in the BlueGreen docs).

Now changing this behaviour would probably be a breaking change so I'm wondering if it might make sense to add an optional Rollout field that could modify this behavior. If a maintainer here is happy with this I'm happy to make a PR.

[kostis-codefresh]

Could you explain a bit about the use case here and what the business need is behind this?

Currently it works like this (or at least this is what I understand)

1. Stable/preview is 1.4
2. Preview becomes 1.5, stable stays at 1.4
3. Analysis runs against whatever you want (either 1.5 or 1.4 or both)
4. Analysis fails. Rollout is marked as degraded. All traffic goes back to 1.4 again

End of story.

Are you saying that after this, you still want to run integration tests to 1.5? Is this your scenario?

[sglre6355]

In my case I route traffic with a certain header to preview and all others to stable with an Istio VirtualService. This enables us to make preview accessible for specific users (in my case, users from the QA team) before fully promoting.

With your example, when the analysis fails in step 4 stable does point back to 1.4 but preview doesn't since its pod hash is not updated. This makes the service inaccessible for those specific users. I'm not sure of the reasoning behind this implementation, but I think it's somewhat unintuitive and would like to know if you know why.

Either way, I think it doesn't sound like a bad idea to have an optional field to make preview point back to the old version and would love to hear your perspective (or other approaches to achieve what I'm trying to do if you know any).

[kostis-codefresh]

With your example, when the analysis fails in step 4 stable does point back to 1.4 but preview doesn't since its pod hash is not updated

Do you have a github repo with an example. Because normally this doesn't happen. When a rollout is NOT in progress both preview and stable services should work just fine and they should point back to 1.4

So either there is a bug, or you have a configuration that I am not aware of.