Deploy new version of WMAgent without draining the agent

[mapellidario]

Impact of the new feature

WMAgent

Is your feature request related to a problem? Please describe.

Draining the agents before deploying a new version means that our release cycle is entangled with operations. For examples, if even a few workflows are stuck because of a single Tier1 is struggling, we may have a hard time to complete the drain and then to deploy a new version.

Since our medium-term goal is to streamline wmcore operations, this feels like a good candidate.

Describe the solution you'd like

Draining the agent is also used to cleanup WMBS, CouchDB and condor storage. If we want to stop draining the agent for new version deployment, we also need to:

• Partition tables in WMBS, automatically drop old partitions when they are no longer relevant
• Investigate why WMBS is slow on FNAL agents. Should we create any index? Should we separate disk used by condor, CouchDB and mariadb, so that processes do not compete for I/O?
• Cleanup disk space used by condor schedd, otherwise agents will systematically go in drain when the data disk is 85% full.

When the WMBS schema changes with a new wmagent version, we may need to adjust it on the fly with some "alter table" query, or we can use the current drain procedure. In any case, since the schema is not explicit/static but dynamic, we would need to detect WMBS schema changes

• WMAgent: Implement Schema validation functionalities #11925

Describe alternatives you've considered

We can continue to deploy new wmagent versions after a full agent drain

Additional context

(none)