[FLINK-37515] Basic support for Blue/Green deployments #969

schongloo · 2025-04-08T16:29:57Z

What is the purpose of the change

This pull request adds basic support for Blue/Green deployments as outlined by FLIP-503 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337677648).

Brief change log

A new FlinkBlueGreenDeploymentController controller, with the capability of managing the lifecycle of these deployments.
A new corresponding CRD is introduced
The new FlinkBlueGreenDeploymentController will manage this CRD and hide from the user the details of the actual Blue/Green (Active/StandBy) jobs.
Delegate the lifecycle of the actual Jobs to the existing FlinkDeployment controller.

Verifying this change

Test to verify a single basic deployment
Test to verify a proper transition Blue -> Green
Test to verify correct behavior when encountering an error before a transition
Test to verify correct behavior when encountering an error during a transition

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changes to the CustomResourceDescriptors: no
Core observer or reconciler logic that is regularly executed: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs / JavaDocs. UML diagrams for all scenarios added under docs/content/docs

...tor-api/src/main/java/org/apache/flink/kubernetes/operator/api/bluegreen/DeploymentType.java

...main/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentState.java

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

davidradl · 2025-04-10T10:59:56Z

I am concerned about the number of TODOs. Things like dealing with errors and timing conditions should be in place in the initial drop of code - unless we can mark this capability as beta or the like. WDYT @gyfora ?

…d-transition.

schongloo · 2025-04-11T17:45:41Z

@davidradl thanks for your review/comments. I've cleaned up all the outdated/invalid TODOs which pointed me 1 missing unit test, added. At this point the few remaining TODOs do not interfere with the main functionality and are not critical/open for discussion.

… BASIC is supported).Tests to confirm the deployments are deleted accurately after the specified deletion delay.

…xceptions.

hansh0801 · 2025-04-23T09:05:36Z

👍

hansh0801 · 2025-04-29T05:44:12Z

when this feature gonna be released?

schongloo · 2025-04-30T13:25:00Z

Hi @hansh0801, trying to release it ASAP but I also want to make sure the logic and contracts are as robust as possible to minimize changes later. Looks pretty stable in my current testing and I've reopened it for review. Thanks!

hansh0801 · 2025-04-30T13:30:33Z

cool ! and I'm also waiting for blue green phase 2 :)
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337677650

schongloo · 2025-04-30T18:20:23Z

@hansh0801 indeed... Phase 2 will take a little bit longer due to its complexity but already working on it.

...tor-api/src/main/java/org/apache/flink/kubernetes/operator/api/bluegreen/DeploymentType.java

...tor-api/src/main/java/org/apache/flink/kubernetes/operator/api/bluegreen/TransitionMode.java

...src/main/java/org/apache/flink/kubernetes/operator/api/spec/FlinkDeploymentTemplateSpec.java

...ain/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentStatus.java

gyfora · 2025-05-07T11:33:09Z

...ain/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentStatus.java

+    private long deploymentReadyTimestamp;
+
+    /** Information about the TaskManagers for the scale subresource. */
+    private TaskManagerInfo taskManager;


Do we want to support this? I feel that the scale logic using TM count is not really used feature for regular FlinkDeployments either

We could remove it for now and only add it if there is a need

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

gyfora · 2025-05-07T12:58:37Z

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

+        if (deploymentStatus == null) {
+            deploymentStatus = new FlinkBlueGreenDeploymentStatus();
+            return patchStatusUpdateControl(
+                            bgDeployment,
+                            deploymentStatus,
+                            FlinkBlueGreenDeploymentState.INITIALIZING_BLUE,
+                            null)
+                    .rescheduleAfter(100);


The BlueGreenDeployment could simply implement initStatus and this can be removed completely.

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

gyfora · 2025-05-07T13:08:16Z

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

+    private boolean hasSpecChanged(
+            FlinkBlueGreenDeploymentSpec newSpec, FlinkBlueGreenDeploymentStatus deploymentStatus) {
+
+        String lastReconciledSpec = deploymentStatus.getLastReconciledSpec();
+        String newSpecSerialized = SpecUtils.serializeObject(newSpec, "spec");
+
+        // TODO: in FLIP-504 check here the TransitionMode has not been changed
+
+        return !lastReconciledSpec.equals(newSpecSerialized);


We may want to use the DiffBuilder/ existing utilities to get the type of the spec change here and make sure we don't trigger deployments on non-upgrade changes.

Things like operator config changes etc should not go through the Blue/Green flow I believe but simply applied on the currently active deployment.

Sure, let me get familiar with the DiffBuilder and see how to incorporate it

gyfora · 2025-05-07T13:09:22Z

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java

+        boolean deleted =
+                deletedStatus.size() == 1
+                        && deletedStatus.get(0).getKind().equals("FlinkDeployment");
+


do we want to wait here until the deletion is actually finished (the object is not there anymore?) that would require some additional logic here

I'd rather not introduce waiting logic. I immediately reschedule a reconciliation afterwards which will re-execute this portion if the deployment is not yet gone.

davidradl · 2025-05-07T14:32:09Z

...main/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentState.java

+    TRANSITIONING_TO_BLUE,
+
+    /** Identifies the system is transitioning from "Blue" to "Green". */
+    TRANSITIONING_TO_GREEN,


what state are we in during shutdown?

...tes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/SpecUtils.java

davidradl · 2025-05-07T14:45:22Z

...tes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/SpecUtils.java

+            return objectMapper.writeValueAsString(wrapper);
+        } catch (JsonProcessingException e) {
+            throw new RuntimeException(
+                    "Could not serialize " + wrapperKey + ", this indicates a bug...", e);


nit: should be include object.toString() in the error also?

davidradl · 2025-05-07T14:51:01Z

helm/flink-kubernetes-operator/templates/rbac.yaml

@@ -101,6 +103,7 @@ rules:
  - apiGroups:
      - flink.apache.org
    resources:
+      - flinkbluegreendeployments/status
      - flinkdeployments/status


this maybe a basic question. Are flinkbluegreendeployments a type of flinkdeployments. Would we expect to see them as flinkdeployments?

That's the original idea but we shifted to a has-is relationship rather than a is-a one.

To clarify this answer @davidradl . For every FlinkBlueGreenDeployment, there will also be 0-2 FlinkDeployment which are the currently running deployments.

…fied

maheepm-lyft · 2025-06-16T20:08:06Z

Hi @schongloo @davidradl @gyfora , I hope you're doing well. I've been following this PR as blue-green deployments would solve a critical use case for our production needs. I noticed there have been no updates to this PR for a few weeks. Was wondering if there is anything blocking the release of this and if there is something myself or the OSS community could help with (testing, fixing any changes, etc). Thanks for all the work so far! 🙏

gyfora · 2025-06-19T07:45:06Z

Hi @schongloo @davidradl @gyfora , I hope you're doing well. I've been following this PR as blue-green deployments would solve a critical use case for our production needs. I noticed there have been no updates to this PR for a few weeks. Was wondering if there is anything blocking the release of this and if there is something myself or the OSS community could help with (testing, fixing any changes, etc). Thanks for all the work so far! 🙏

@maheepm-lyft , @schongloo is currently on holiday but once he is back we can try to merge this as soon as reasonably possible :)

In the meantime you can help us by trying out the current version and by providing a code review for the PR, that can definitely speed things up.

gyfora · 2025-06-23T07:40:59Z

@schongloo can you please re-open this PR against the main branch?

FLIP-503 first batch

ba62f07

schongloo mentioned this pull request Apr 8, 2025

[FLINK-37515] Basic support for Blue/Green deployments #954

Closed

davidradl reviewed Apr 10, 2025

View reviewed changes

...tor-api/src/main/java/org/apache/flink/kubernetes/operator/api/bluegreen/DeploymentType.java Show resolved Hide resolved

davidradl reviewed Apr 10, 2025

View reviewed changes

...main/java/org/apache/flink/kubernetes/operator/api/status/FlinkBlueGreenDeploymentState.java Show resolved Hide resolved

davidradl reviewed Apr 10, 2025

View reviewed changes

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java Outdated Show resolved Hide resolved

davidradl reviewed Apr 10, 2025

View reviewed changes

...java/org/apache/flink/kubernetes/operator/controller/FlinkBlueGreenDeploymentController.java Outdated Show resolved Hide resolved

schongloo added 2 commits April 11, 2025 10:38

Addressing PR comments. Adding unit test to cover for Spec changes mi…

eb5b24f

…d-transition.

Addressing PR comments/cleaning up comments.

75d787e

schongloo requested a review from davidradl April 11, 2025 17:41

schongloo added 2 commits April 14, 2025 00:28

Adding a TransitionMode to the CRD/API for the future (currently only…

8addc70

… BASIC is supported).Tests to confirm the deployments are deleted accurately after the specified deletion delay.

Addressing the test sync issues that caused intermittent NullPointerE…

6ecc28c

…xceptions.

schongloo marked this pull request as draft April 15, 2025 18:06

Simplifying the retry/abort logic.

1df7747

Addressing edge case restarting after the first deployment fails.

3690b0d

schongloo marked this pull request as ready for review April 30, 2025 13:22

schongloo changed the base branch from release-1.11 to main April 30, 2025 21:14

schongloo changed the base branch from main to release-1.11 April 30, 2025 22:33

schongloo added 3 commits May 5, 2025 09:41

Undoing accidental test removal

24f65e8

Defensive code vs. non-BASIC TransitionMode (only for FLIP-503)

18da43c

Updated rbac.yaml

7d5072a

gyfora requested changes May 7, 2025

View reviewed changes

davidradl reviewed May 7, 2025

View reviewed changes

...tes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/SpecUtils.java Outdated Show resolved Hide resolved

davidradl reviewed May 7, 2025

View reviewed changes

schongloo added 10 commits May 13, 2025 11:36

Addressing PR comments

27ae567

Fixing accidental comment

7bcb244

More PR comments addressed

4834517

Clean up

4f4c39f

Updated Blue/Green CRD

f9245aa

Using the initial savepoint path, on first deployments only, if speci…

dcce6c8

…fied

Clean up

d4b31ea

Clean up

f04723f

Clean up

3698bb3

Addressing PR comments

d62260b

[FLINK-37515] Basic support for Blue/Green deployments #969

Are you sure you want to change the base?

[FLINK-37515] Basic support for Blue/Green deployments #969

Uh oh!

Conversation

schongloo commented Apr 8, 2025

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidradl commented Apr 10, 2025

Uh oh!

schongloo commented Apr 11, 2025

Uh oh!

hansh0801 commented Apr 23, 2025

Uh oh!

hansh0801 commented Apr 29, 2025

Uh oh!

schongloo commented Apr 30, 2025

Uh oh!

hansh0801 commented Apr 30, 2025

Uh oh!

schongloo commented Apr 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maheepm-lyft commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gyfora commented Jun 19, 2025

Uh oh!

gyfora commented Jun 23, 2025

Uh oh!

Uh oh!

maheepm-lyft commented Jun 16, 2025 •

edited

Loading