Sharding application-controller by Application or a Project

[msobkowiak-olx]

Hey there !

Wanted to ask on possibilities to distribute load across multiple application-controllers, our setups usually consist of one ArgoCD instance per cluster, thus is not very beneficial for us to use default sharding option described here:
https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/

I would like to ask if there is any plans to actually change this to shard per application name (using static or consistent (with crash support) hashing, just like on memcached clients or in loadbalancers), which should have the following benefits:

• best possible granularity of the load (with monorepos still being sort of an issue however it is still an issue even now)
• best resource ultilisation (achieving best possible "even" distribution, with exception for large monorepos)
• best possible scaling scenarios (supporting ArgoCD with one or multiple clusters)

Similar can be applied to a project. The hashing can simply take project or app name, calculate the hash and send it to proper slot for processing in the deployed statefulset. This should be possible because of the constraints ArgoCD puts on the Applications and Projects (unique naming, guaranteeing that each entity has unique key for hashing function, the distribution happens onto application-controller pods).

Please note I am aware of the queue tuning on operation and status processors, however we would like to avoid creating too large pods not fitting our workload nodes in the clusters hosting ArgoCD.

Let me know if it makes sense.

[awx-fuyuanchu]

Yeah. It sounds like a more general pain point for most users. One cluster could spawn hundreds of Applications and sharding the Applications in one cluster can help a lot for performance, efficiency, and scalability.

[saprette]

Hello, that is something that we would benefit from.

[andrii-korotkov-verkada]

I suggest to open an enhancement proposal for this. I don't know if this is easily solvable, since resources can move between applications and applications can move between projects.

[andrii-korotkov-verkada]

I think the problem is two things can maintain a different view of the same object in Kubernetes, so there might be conflicting changes.

[msobkowiak-olx]

Isn't this an issue even now on single shard ? I mean you can technically deploy two different applications with different name and exactly the same manifests (we used to hit collisions sometimes during migrations from helm).

[andrii-korotkov-verkada]

There's a cluster lock acquired by a number of places.

[msobkowiak-olx]

Thanks, I mean it would be crucial to understand what exactly this lock synchronizes in this case. Implementing a distributed lock would not be that easy (technically the locking for cross-shard implementation could be externalised to redis or so - that is actually not that hard but prone to process crashes without expiration implementation).

[andrii-korotkov-verkada]

I can bring this up during the contributors meeting.

[andrii-korotkov-verkada]

See #6125 for a similar thing.

[crenshaw-dev]

Most of the work that the Application controller does is cluster-scoped. Breaking it into Application- or Project-sized chunks is very high technical cost for potentially very low reward.

In my opinion, the time is better spent improving the general performance of the Application controller and tuning performance configuration to alleviate issues.

[msobkowiak-olx]

If the applications are consistent (with 700 you would for sure hit those issues) I think the purpose of the locking is to simply have the application refreshed by exactly one application controller at the time. If this is the case honestly I think sharding and locks per app should be possible. Perhaps problematic part could be resharding/adding apps (as this is something that changes often) but I would say the ring can be just composed by application controller names (statefulset would really help here to keep the ring stable) and then consistent hashing could be done by application name.

I guess what the issue is here is that each applicationcontroller on startup needs to have a list of apps assigned as there is no "coordinator" concept here (that is usually an issue with operators pattern - they elect a leader for given partition and here its definetely easier to partition by cluster so we dont re-discover every now and then). To solve that the stateless coordinator concept could be introduced to hash/balance the traffic and pass the requests to applicationcontrollers given by hashing/balancing algorithm (just like in regular loadbalancers) but this then needs to externalize the locks (eg with centralized redis so no concurrent changes are made to the single app with multiple applicationcontroller at the same time).

Again - lazy solution could be allowing to add the cluster with multiple aliases - that point to the same host but different name (ARNs or other unique metadata would then be a bit tricky to use tho).

[jessesuen]

The technical challenge is not just about spreading work around. The challenge is that, to properly reconcile an application, the app-controller needs to understand the full state of the cluster, to maintain what we call the "live state cache." The live state cache is an in-memory cache that stores lightweight metadata (name, namespace, gvk, ownerResources) about all resources, which is then used to:

1. detect when a resource (e.g. Pod) is related to an application so that an application can be queued for reconcilliation
2. reconstruct the application tree during reconciliation

Even if we were to support multiple shards for a single cluster, each shard would still have to maintain its own live state cache, and each shard would use the same amount of memory resources, even if they were handling only a subset of the apps for that cluster. Not to mention, since each live state cache would be independent of the others, there could be inconsistencies between them, which then implies this live state cache would need be split out as a separate component for consistency. Accessing this cache then becomes an RPC call rather than in-memory access, slowing reconciliation.

This has been considered before, but thus far, nothing I've heard can easily overcome the challenges I've described above.

[7onn]

@icy how did you manage to create the same cluster twice? I couln't succeed on this as a new secret requires me to input caData and certData; while the default in-cluster seems to manage this by itself.

I fear that creating them with these hardcoded values would make me vulnerable as it would break my implementation as soon as EKS rotates the certificate. Plus, the config doesn't allow me to reference certs from a file so I could use the one ArgoCD's service account mounts in the pods.

Did you toggle the tlsClientConfig.insecure: true?

[icy]

Hi @7onn ,

Sorry I didn't mention that before to avoid some noise. Here are all things that helped in my situation. I hope that helps.

I use kubectl to enter any argocd (application controller) pod, from within the pod, I can try some basic kubectl commands for example kubectl get pods, kubectl get nodes; these commands work in my setup. Then I create a new ~/.kube/config file (as below). You may add as many configurations as you need, just keep in mind that server: values should be unique. The next step is very trivial

$ argocd login
$ argocd cluster add clone1
$ argocd cluster add clone2

argocd will warn if you want to overwrite some existing cluster. Be careful to answer ;) In my case, I just run with --upsert option and overwrite everything.

Authentication

You may notice the configuration relies on the default service token /var/run/secrets/kubernetes.io/serviceaccount/token. This has some expiration, 6 month or one year (depends on your setup). But that doesn't really matter. I just need to re-run the command argocd cluster add clone1 as above every now and then.

My ~/.kube/config

NOTE: this file is created with the argocd application controller pod.

apiVersion: v1
kind: Config
clusters:
- name: clone1
  cluster:
    server: https://kubernetes.default
    certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- name: clone2
  cluster:
    server: https://kubernetes.default.svc.cluster.local
    certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
users:
- name: clone1-sa
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
- name: clone2-sa
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
contexts:
- name: clone1
  context:
    cluster: clone1
    user: clone1-sa
    namespace: default
- name: clone2
  context:
    cluster: clone2
    user: clone2-sa
    namespace: default
current-context: clone1

[icy]

PS: Please also note I removed the additional clusters (clone2, clone3,...) in the sample. It's just a lot of duplication.