Feature Request: Use MEM_MAX and QEMURAM in the scheduler

[rclobus]

Recently on the Debian openQA instance we have a mixed set of tests that require either 2GB or 4GB RAM. One of the smaller worker machines has only 8GB available and is unable to handle 2 4GB-builds simultaneously, resulting in out-of-memory errors. If we are lucky, one of the 2GB builds is running there and all is fine.

Feature request: under the assumption that the openQA workers are the only users on the machine, extend the functionality of the scheduler to account for the amount of available RAM.

Existing code:
The workers have an attribute 'MEM_MAX', which is the total amount of memory on the worker machine. Each test has 'QEMURAM' set to the amount of memory that is required for QEMU.

The build can be scheduled to a worker when: MEM_MAX - SumOf(QEMURAM of active workers) - some overhead >= QEMURAM of new build
If this condition is not met, try the other workers. If none of the workers (currently) meet this condition, keep on the scheduled list

For now, I'll reduce the memory to about 3.5GB RAM, which will allow the two builds to be run in parallel again.

Discussed today in the Linux QA call.

openQA could support "external" schedulers with an API change.

Currently when a job starts, the web UI chooses a worker and sends a "start job" message. The worker then runs the job and pipes status messages back to the web UI until the job completes.

With external scheduling, the external scheduler would choose a worker and it would send a "start job" message directly to the worker. The worker then needs to send a "job has started" message to the web UI. The web UI then creates a job ID. From there: the worker sends status messages to the web UI and the external scheduler, until the job completes.

This would allow use of custom schedulers, including Gitlab CI, Slurm or anything else.

[okurz]

This would allow use of custom schedulers, including Gitlab CI, Slurm or anything else.

Depending on the specific use-case it might be easier to just use the already existing feature from
http://open.qa/docs/#_integrating_test_results_from_external_systems