Placement API to allow multiple ManifestWorks to be assigned to the same cluster

Hi, I'm working on building a multi-cluster autoscaler.
One idea is to create custom resources on top of ManifestWork, like Deployment for Pod, and scale the replicas of it.
I intend to have the custom resource controller generate name-[UUID] ManifestWorks.

The problem with building a multi-cluster autoscaler based on the above idea is that the Placement API doesn't allow multiple ManifestWork assignments to the same cluster.

If there are cluster-1 and cluster-2, and numberOfClusters is 3, the PlacementDecision would be as follows.

status:
  decisions:
  - clusterName: cluster-1
    reason: ""
  - clusterName: cluster-2
    reason: ""

Are there any plans to add numberOfResources param, for example, in the future to allow for it?

In that case, the PlacementDecision would be as follows.

status:
  decisions:
  - clusterName: cluster-1
    resources: 2
    reason: ""
  - clusterName: cluster-2
    resources: 1
    reason: ""

If that is not planned, I would build a custom scheduler.

[mikeshng]

I think I have a similar enhancement request for Placement #134.

In your example, what does the resources: 2 and resources: 1 represent? Could you clarify how the Placement controller can look at the ManagedClusters and populate those fields?

In my example, "resource" means ManifestWork. So numberOfResources means numberOfManifestWorks.
Probably the following example better expresses my request.

If there are cluster-1 and cluster-2, and numberOfManifestWorks is 3:

status:
  decisions:
  - clusterName: cluster-1
    manifestWorks: 2
    reason: ""
  - clusterName: cluster-2
    manifestWorks: 1
    reason: ""

In this example, two ManifestWorks will be placed in cluster-1 and one ManifestWork in cluster-2.

My request aligns with #134 in modifying the PlacementDecision's output, but it differs in proposing a major change to the Placement API.

[mikeshng]

I think Placement API is meant to be a primitive API that focuses on just selecting clusters, so maybe it's not the right place to express ManifestWork.
Maybe you can use ManifestWorkReplicaSet (or propose an enhancement on it) which already contains a list of Placement references:
https://github.com/open-cluster-management-io/api/blob/main/work/v1alpha1/types_manifestworkreplicaset.go#L59-L64
This is a higher level API that ties ManifestWork and Placement together.

Please also checkout the spread enhancement, because I feel like what you are proposing has some similarities as spread: https://github.com/open-cluster-management-io/enhancements/tree/main/enhancements/sig-architecture/70-spread-policy

The initial idea in building a multi-cluster autoscaler was to coordinate the replicas of Deployment in ManifestWork with the spec.numberOfClusters in Placement.
After considering specific scale-out and scale-in cases, I found this to be complex.

The next idea was to treat the ManifestWork with a Deployment with replicas: 1 as the smallest unit.
To build a multi-cluster autoscaler based on this idea, multiple ManifestWorks must be assigned to the same cluster, just as Deployment assigns multiple Pods to the same node in Kubernetes.
ManifestWorkReplicaSet and Spread Policy (thanks so much for those efforts!) are not useful in my case as they are based on assigning one ManifestWork per cluster.

Anyway, it makes sense that Placement is only responsible for cluster selection, and I think we should build our own scheduler.
If no changes to the Placement API or creation of a new API are planned, feel free to close this issue.
Thank you for discussing this issue.

[qiujian16]

The original plan is such replica distribution feature will be added in ManifestWorkReplicaSet where user can set which resource is distributable. For instance, in manifestworkreplicaset, a user can set a deployment with replica of 9. And if placement select 3 clusters, deployment replica in each cluster (applied by manifestwork) will be 3.

And user can select a different distribution policy (replicate/even distribution/choose only one cluster). This is not implemented yet, I am open for an enhancement proposal in ManifestWorkReplicaSet

Thanks, that seems to be what I am looking for.

Indeed, there are traces of the original plan in the KEP of ManifestWorkReplicaSet.
https://github.com/open-cluster-management-io/enhancements/blob/15f73579f718357fb803d28067662e31edc39a0f/enhancements/sig-architecture/35-work-placement/work-placement.md

Also, the discussion here seems to relate to open-cluster-management-io/enhancements#31.

[github-actions]

This issue is stale because it has been open for 120 days with no activity. After 14 days of inactivity, it will be closed. Remove the stable label to prevent this issue from being closed.

I decided to build our own scheduler and will not work on this.