Cythonize scalar function root finders

[mikofski]

@person142 responsed to #8354 that his preference was for a cython optimize api.

This came up before: #7242.

As you can see I was against the idea. Handling all reasonable halting conditions for a scalar solver is already a PITA, and I think the problem gets much worse for a vectorized scalar solver. IMO it is better to provide a Cython API to a scalar solver and let users handle looping over it.

#8357 only vectorized Newton methods. #8431 allows users to use a "tight loop" in C with Brent, Ridder, and other root finders.

We decided it was okay to split up Newton

Is it okay to split up Newton, Halley's, and secant into different calls in cython_optimize?

I am personally pro this (wish it had been done that way in optimize), but maybe others will feel more strongly that the APIs should match.

We discussed which callbacks to support and the fact that cython_optimize should be pure C so it can free the GIL in this comment

This commit has links to annotated html that shows that zeros_struct and zeros_array are pure C, no Python so they can release the GIL to call Cython prange

The Cython optimize API in #8431 was discussed in this SciPy-Dev post and this one too when I specifically asked about callback signatures.

[pvanmulbregt]

If I understand correctly then:
gh-8354 (Issue) ENH: "vectorize" newton and other scalar methods
gh-8357 (PR) ENH: vectorize scalar zero-search-functions
At one stage a reviewer expressed preference for a different approach, using a cython optimized API. The PR gh-8357, using vectorization and not a cython optimized API, was accepted & merged.
gh-8341 (New PR) ENH: Cython optimize zeros api

What is the implication of the new PR? Are you now staying that vectorization is not the right approach and it should be replaced with a cython optimized API?

[mikofski]

What is the implication of the new PR?

PR #8431 is not new, it was opened in February, around the same time as #8357 .

Are you now saying that vectorization is not the right approach and it should be replaced with a cython optimized API?

No, I still think that for newton, using NumPy was appropriate, because since it is in pure Python, it was amenable to vectorization. Also being a Python function, _array_newton is accessible to anyone without using Cython or compiling C source.

However, in pvlib-python for example brentq is also used, and currently this is not vectorized. For the remaining root-finding functions (brentq, brenth, ridder, bisect), I agree with the original reviewer that Cython optimize API is the right approach.

Also, the Cython optimize API serves other purposes in addition to just vectorizing the calculations. For example, using Cython, it is trivial to parallelize the calculation with low overhead using prange if the caller is pure C and no Python, such as the zeros_struct and zeros_array solvers. And using the Cython optimize API would allow using scipy.LowLevelCallable which I had hoped to work on next.

Are you against adding the Cython optimize API? Is anyone in favor of this? I worry that I have wasted my time, especially if there is need or desire for these features, other than my own.

[rgommers]

Are you against adding the Cython optimize API? Is anyone in favor of this? I worry that I have wasted my time, especially if there is need or desire for these features, other than my own.

I'm not against. The existing cython APIs, especially the linalg one, have been very successful. However, we do need a clean API as well as make sure that this is not going to be a maintenance burden or block new features in the future. Currently the API isn't looking right, and even reviewing is going to be a lot of work. Hence the reluctance.

Would be good to get a sense of what others think of cython_optimize in principle (not the current API) - @person142, @pv, other optimize-interested devs?

[andyfaff]

Would be good to get a sense of what others think of cython_optimize *in principle* (not the current API) - @person142 <https://github.com/person142>, @pv <https://github.com/pv>, other optimize-interested devs?

In `scipy.integrate.quad` the `func` object to integrate can either be a Python callable or a `LowLevelCallable` with one of a few signatures. I think that if any of the `optimize` functions (I'm more interested in minimize related) could offer something similar, then that would be really really good. In orther words the `optimize.minimize` call signature remains more or less the same, it's just that `func` could either be a Python callable or a `LowLevelCallable`. Obviously the backend minimizer would have to be able to make use of that `LowLevelCallable`. Presumably for some minimizers this would mean translating core iterative code to cython (this is one of the targets I have in mind for the object-oriented minimizer PR), those whose functionality is already written in using extensions might be able to benefit more easily. I wouldn't be a fan (-10) of having two different functions for the same thing, e.g. integrate.quad/integrate.cquad

[mikofski]

@andyfaff thanks for your feedback. I copied the integrate.quad signatures based on your suggestion to implement LowLevelCallable, specifically

double func(double x, void *user_data) # 1
double func(int n, double *xx) # 2

My goal as I mentioned in this comment was to use cython_optimize.zeros to dispatch the LowLevelCallable based on it's signature to either zeros_struct (1) or zeros_array (2) but also leave those exposed in the Cython API to use directly in a pure C no Python use case if nogil was desired. I was hoping to do this in two separate PR[s], complete Cython API first, then add dispatcher for LowLevelCallable or Python callable.

Also I tried to keep the solvers signatures identical to their Python counterparts, hence why func, x0, fprime, ... for newton, in which the function and it's derivatives are separated. The signatures for brentq, brenth, ridder, and bisect have the same signature as their Python versions. The only signatures that are slightly different are secant and halley which I've separated from newton as discussed in this comment

[ev-br]

Would be good to get a sense of what others think of cython_optimize in principle (not the current API)

IMO it's very valuable.
There's a question of scope, of course: not sure about the whole optimize, but Cython root-finders, emphatically yes.

I would definitely use Cython brentq, the other scalar root-finders might be lower priority.

[pvanmulbregt]

@mikofski Please take my questions as my attempt to understand what the issue is and what you are proposing, I'm not expressing an opinion. Procedurally, any discussion of this Issue and PR should be in the context of today's SciPy, which has two implementations of newton, rather than the SciPy of January which had one. The PR gh-8431 would then add a 3rd implementation.

What I'd like to see are the use cases, how they are handled in today's SciPy, what the issue is with using current implementations, how that might be changed, and then the benefit(s) of such change. That's why I was asking for a new Issue to be filed, to make all that explicit in today's context.

[andyfaff]

On Tue, 4 Sep 2018 at 21:52, Paul van Mulbregt ***@***.***> wrote: @mikofski <https://github.com/mikofski> Please take my questions as my attempt to understand what the issue is and what you are proposing, I'm not expressing an opinion. Procedurally, any discussion of this Issue and PR should be in the context of today's SciPy, which has two implementations of newton, rather than the SciPy of January which had one. The PR gh-8431 <#8431> would then add a 3rd implementation. What I'd like to see are the use cases, how they are handled in today's SciPy, what the issue is with using current implementations, how that might be changed, and then the benefit(s) of such change. That's why I was asking for a new Issue to be filed, to make all that explicit in today's context.

Paul's reply was a lot more considered than my attempt. This is pretty much my view as well. I'm more interested from a machinery point of view, i.e. how can `minimize`/`differential_evolution`/`basinhopping`/etc be performance enhanced by using `LowLevelCallable`.

[mikofski]

PR #8431 does not attempt to modify any of the minimizers, sorry, only the scalar-function root finders. In addition, it does not currently address LowLevelCallable, although I would like to in a follow on PR.

The use cases for Cython optimize API are twofold:

1. enable brentq and other scalar-function root finders to be used to solve the same callback with over 100,000 different sets of arguments by binding them to C using Cython - although newton and it's siblings were vectorized in ENH: vectorize scalar zero-search-functions #8357, the other root finders, such as brentq were not because they are written in C and not as easily vectorized, an alternative is to use Cython
2. enable all of the scalar-function root finders to take advantage of benefits of a C-binding using Cython, such as:
   • ability to use LowLevelCallable which requires a C-binding for the purpose of using non-python callbacks
   • ability to use prange for embarrassingly parallel calculations in a loop with the same callback and over 100,000 different sets of arguments
   • ability to use non-Python memoryviews to access array elements so that the compiler can optimize loops and increase the performance of solving the same callback with over 100,000 different sets of extra arguments

For example, pvlib-python uses brentq to solve for the maximum power point in the single-diode model of a solar cell. In a typical simulation of a PV system, at 15 second intervals, there could be 1e6 daytime hours that would need a unique solution. Other efforts such as caching/memoizing and interpolating are possible, but the overhead may be greater than vectorizing, embarrassingly parallel loops, or compiler optimizations.

There may be other use cases, and a few individuals have already stated that it may be useful. Also based on the success of other Cython API[s] such as linalg and special then a Cython optimize API might also be

cc: @andyfaff and @pvanmulbregt

[mikofski]

today's SciPy, which has two implementations of newton, rather than the SciPy of January which had one. The PR #8431 would then add a 3rd implementation.

@pvanmulbregt are you concerned that there are too many implementations of newton?

[pvanmulbregt]

I'm concerned about the number of sub-case specializations of newton. Each new sub-case introduces additional maintenance and reduced usage (hence reduced reliability) in the existing implementations.

The title of the Issue "Cythonize scalar function root finders" is implementation-centric rather than user-centric. It's also a little ambiguous on one key point --- whether the cythonized root finders will be replacing the existing python/C root-finders. (The PR gh-8341 doesn't replace, it adds new ones handling a subset of functions.) The discussion may go in different directions based on this point,
so I'll ask it now.
Is it your intent that the Cython-ized versions replace the existing versions?

[mikofski]

No, I do not intend or expect the Cython versions of any of the zero-finders to replace the existing Python API[s] because usage of the Cython versions requires a non-trivial effort by the user to write and compile Cython code, and additional software to be installed, such as the Cython package and a C-compiler, which are not required for users who only wish to use the SciPy Python API[s].

The usage of the Cython API is fundamentally different from the Python API:

Python API

>>> from math import exp
>>> from scipy.optimize import brentq

>>> brentq(lambda x, c0, c1: c0 - exp(x) / c1, a=0.5, b=1.0, args=(1.0, 2.0),
...        xtol=0.001, rtol=0.001, maxiter=10, full_output=True)
(0.68314716056,
       converged: True
            flag: 'converged'
  function_calls: 6
      iterations: 5
            root: 0.68314716056)

A similar calculation using the Cython API would probably not be worth the effort if there were not significant performance improvements.

Cython API

import cython
from libc.math cimport exp
from scipy.optimize.cython_optimize cimport zeros_struct

ctypedef struct test_full_output:
    int funcalls
    int iterations
    int error_num
    double root

ctypedef struct test_params:
    double C0
    double C1

@cython.cdivision(True)
cdef double f(double x, void *args):
    cdef test_params *myargs = <test_params *> args
    return myargs.C0 - exp(x) / myargs.C1

cdef test_full_output show_full_output(dict args, double xa, double xb,
                                       double xtol, double rtol, int mitr):
    cdef test_params myargs = args
    cdef test_full_output full_output
    full_output.root = zeros_struct.brentq(f, xa, xb, <test_params *> &myargs,
        xtol, rtol, mitr, <zeros_struct.scipy_zeros_parameters *> &full_output)
    return full_output

def test_cython_brentq(args={'C0'=1.0, 'C1': 0.2}, xa=0.5, xb=1.0,
                       xtol=1e-3, rtol=1e-3, mitr=10):
    """test Cython brentq with full output"""
    return show_full_output(args, xa, xb, xtol, rtol, mitr)

After compiling the Cython code, then in Python ...

>>> test_cython_brentq()
{'funcalls': 6,
 'iterations': 5,
 'error_num': 0,
 'root': 0.68314716056}

I'm also concerned about the duplicate implementations of Newton. One idea that I think could possibly combine the various Newton implementations would be to implement a Python extension module, based on the current Python extension module for brentq, ..., in zeros.c to replace the Python implementation and call the C source code for Newton, secant, and Halley, which IMHO would be a different PR.

[rgommers]

No, I do not intend or expect the Cython versions of any of the zero-finders to replace the existing Python API[s]

@mikofski the question was not about the API; it's clear that any changes there need to be backwards compatible. It's about the implementations. I.e. will there be duplication of implementation between Python - Cython - C.

One idea that I think could possibly combine the various Newton implementations would be to implement a Python extension module, based on the current Python extension module for brentq, ..., in zeros.c to replace the Python implementation and call the C source code for Newton, secant, and Halley, which IMHO would be a different PR.

Yes that sounds better. If you have a Cython API for newton, then you'd want one of:

• 1 implementation in C, and exposing that from Cython and Python
• 1 implementation in Cython, and calling that from Python (preferred if current implementation is Python-only)