Why fcmaes is not scaling with increasing number of workers?

[de-ranit]

I was trying to optimize the following example function.

evaluate_fcmaes.py :

import sys
from fcmaes import cmaescpp
from numpy.random import Generator, PCG64
from scipy.optimize import Bounds
import numpy as np


def cost_nse(obs, sim):

    return 1.0 - np.sum((obs - sim) ** 2.0) / np.sum((obs - np.mean(obs)) ** 2.0)


def costhand(param_scalar):

    sim = []
    for p_val in param_scalar:
        sim.append(p_val * 0.1 + p_val * 0.2 + p_val * 0.3)

    obs = [0.25, 0.11, 0.76, 0.45, 0.98, 1.2]

    nse = cost_nse(np.array(obs), np.array(sim))

    return 1.0 - (1.0 / (2.0 - nse))

def run_cmaes():
    bnds = Bounds([0.0] * 6, [1.0] * 6)

    WORKERS = sys.argv[1]

    ret = cmaescpp.minimize(
        fun=costhand,
        bounds=bnds,
        x0=[0.5] * 6,
        input_sigma=0.1,
        popsize=1000,
        max_evaluations=20000,
        stop_fitness=1e-6,
        rg=Generator(PCG64(43)),
        workers=WORKERS,
    )

if I set the workers to None, and run python -mtimeit -s 'import evaluate_fcmaes' 'evaluate_fcmaes.run_cmaes()' None, I am getting: 2000 loops, best of 5: 106 usec per loop

if I set the workers to 6, and run python -mtimeit -s 'import evaluate_fcmaes' 'evaluate_fcmaes.run_cmaes()' 6, I am getting: 2000 loops, best of 5: 107 usec per loop

Why I am not getting a significant reduction in time when using more workers? am I using cmaescpp correctly if I want to parallel evaluate costhand and optimize its parameters?

[dietmarwo]

There is an issue regarding Python and multithreading. The C++ version of cmaes uses multi-threading which doesn't work if the fitness function uses the Python interpreter (at least for Python version <= 12).
So for Python fitness functions you should use cmaes instead of cmaescpp.
If you instead use the parallel restart / retry mechanism, this is no issue because this uses multi processing - a separate process for each parallel optimization run.

In general, parallelisation of a single optimization run makes only sense for very expensive fitness functions. The overhead managing the different processes is too high.
So even using cmaes instead of cmaescpp you will see no progress.

When you increase the computational effort, for instance

...
 nse = cost_nse(np.array(obs), np.array(sim))
 for _ in range(1000):
        _ = np.sin(2) 
...

you will see that increasing the number of workers improves the time used significantly - but only using cmaes and not cmaescpp.

For easy optimization problems (fast to evaluate fitness, only one restart required), you don't need parallelization.
If you have many such problems to solve, use Python multiprocessing and execute each optimization single threaded.
fcmaes' built-in parallelisation mechanisms will not help in this case.

[dietmarwo]

Checked if removal of the GIL (global interpreter lock) in Python
would help here. You have to compile Python 3.13 using '--disable-gil' to check:

wget https://www.python.org/ftp/python/3.13.3/Python-3.13.3.tgz
tar -xf Python-3.13.3.tgz
cd Python-3.13.3
./configure --prefix=$HOME/python-3.13-nogil --disable-gil --enable-optimizations
make -j$(nproc)
make install

First I slowed down the fitness function using

    nse = cost_nse(np.array(obs), np.array(sim))
    for _ in range(10000):
        _ = np.sin(2) 

Otherwise process / thread related overhead would dominate.

Result: The most visible change is CPU utilization, now 50% using 16 workers
on a 16 core / 32 thread AMD 9950x. But the time used (80.39 sec) is very
disappointing. Down from 108.22 sec. Comparable to the 1 worker time for
Python 3.12 and 3.8. I have no explanation for that. Fitness functions
written in C++ called from cmaescpp scale well even when the GIL is in place.

Conclusion:

• Prefer to execute optimizations single threaded but parallelize restarts
• If your fitness function is slow and written in Python you can parallelize
  using the workers parameter, but use the python version of the algorithm, in this
  case use cmaes instead of cmaescpp.

python3.13t -X gil=0 :

workers = 1, cmaescpp
time =  108.22 sec, cpu 3.14%

workers = 16, cmaescpp
time =  80.39 sec, cpu 50%

workers = 1, cmaes
time =  113.84 sec, cpu 3.14%

workers = 16, cmaes
time =  8.28 sec, cpu 16 x 2.81%


python3.12 :

workers = 1, cmaescpp
time =  78.84 sec, cpu 3.14%

workers = 16, cmaescpp
time =  84.66 sec, cpu 3.14%

workers = 1, cmaes
time =  88.69 sec, cpu 3.14%

workers = 16, cmaes
time =  6.49 sec, cpu 16 x 2.81%


python3.8 : 

workers = 1, cmaescpp
time =  78.84 sec, cpu 3.14%

workers = 16, cmaescpp
time =  81.09 sec, cpu 3.14%

workers = 1, cmaes
time =  78.07 sec, cpu 3.14%

workers = 16, cmaes
time =  6.83 sec, cpu 16 x 2.81%

[de-ranit]

Thanks a lot for these very useful tips.

My actual fitness function is written in Python and is time-consuming to evaluate. Using cmaes instead of cmaescpp scaled it well. It is also not very clear to me what should be the advantage of using parallel restart / retry mechanism. Can you please let me know or can I read it somewhere?

I have a couple more questions.

1. If I am optimizing my fitness function on a 72-core (physical) machine (with 144 logical cores). Should it help cmaes when the number of workers is set to 144 instead of 72? I tried this with my fitness function and found not much difference when no of workers was set to 72 vs 144.
2. I have a limit on the maximum amount of time I can run a script or job. If this is not sufficient to minimize the cost function, then the job gets aborted. Let's say I want to have a population size of 1000, and max iteration set to 80 (so total 80000 function evaluations are done), and the time limit is not enough to finish it. Is there a way to save the current state in a file (let's say after 50000 evaluations) and then restart from there on a new job and finish the rest 30000 evaluations?

[dietmarwo]

See Simulation my newest tutorial about the topic.
You actually have two options: Either use the ask/tell interface or switch to the Python versions of the optimizers.

It is also not very clear to me what should be the advantage of using parallel restart / retry mechanism

Parallel restart is only useful if more than one restart is required for convergence. If your problem converges using a single run, you have to use parallel evaluation instead - as you do already. Using the ask/tell interface this is easy. You could use any parallelization method, not only what is provided by fcmaes (fcmaes.evaluator.parallel). You could even distribute the workload over multiple CPUs

If I am optimizing my fitness function on a 72-core (physical) machine (with 144 logical cores). Should it help cmaes when the number of workers is set to 144 instead of 72? I tried this with my fitness function and found not much difference when no of workers was set to 72 vs 144.

There are two limitations:

• workers < popsize, ideally popsize is N*workers for an integer N.
• increasing popsize can decrease convergence speed.

So if a single run/restart leads to convergence it may be difficult to utilize all 72 cores.
You could try to apply limited parallelization to the objective evaluation itself, for instance

import os
os.environ.setdefault("OMP_NUM_THREADS", "8")
os.environ.setdefault("NUMEXPR_NUM_THREADS", "8")
os.environ.setdefault("MKL_NUM_THREADS", "8")
numba.set_num_threads(8)

dependent on the library the objective function uses.
Using workers = popsize = 16 then utilizes 128 = 16*8 threads.

Only if there are many decision variables to optimize, it could help to increase popsize = workers instead.

Is there a way to save the current state in a file (let's say after 50000 evaluations) and then restart from there on a new job and finish the rest 30000 evaluations?

The ask/tell interface (see Simulation) enables full control over the optimization process. But "restart from there" doesn't require it, it can simply be achieved by starting the second run using as guess x0, the best solution found so far. Reduce input_sigma to focus the search around the guess.