Client Request Auto-Forwarding

[shijiesheng]

Client Request Auto-Forwarding

Value Proposition

In a multi-cluster Cadence server, domains are only active in one cluster given a workflow execution. As a result, test workers need to connect to the “active” cluster to process test workflows. This requires the test infrastructure to be aware of the activeness of each domain.

With this proposal, we aim to auto forward the request from test clients to domain’s active cluster for simpler testing.

How does auto-forwarding work in Cadence?

When the frontend receives a request, it reads the domain cache and checks if the current cluster is the active cluster. Decided by the redirect policy, it can then forward the request to the frontend of the active cluster.

Each frontend of different clusters are aware of each other through ClusterGroups meta data.

Active Active Domain Edge Case

Specially, for Active-Active domains, active cluster is selected by active cluster selection policy for start workflow APIs and workflow execution records for other workflow APIs. Task polling APIs will NOT be forwarded because neither information exists.

By design, active-active domain requires workers processing in every cluster that the domain exists in. Thus, we do not support testing for active-active through auto-forwarding. Users have to start workers in all active clusters.

Proposal

Client will send a new FeatureFlag `AutoForwardingEnabled`. If it’s set to true, client requests will be auto-forwarded.

message FeatureFlags {
	bool workflow_execution_already_completed_error_enabled = 1;
+	bool auto_forwarding_enabled = 2;
}

How does FeatureFlag work?

Client will send a request with a feature flag header `cadence-client-feature-flags`. This header will store a json string serialized by IDL. (currently in thrift but will be migrated to use proto for cross-language consistency)