FEAT - add fit_intercept support for LBFGS

[floriankozikowski]

Context of the PR

Closes #325

Contributions of the PR

• add fit_intercept support for LBFGS
• add default warm_start=False, so LBFGS works with debug script in ProxNewton and Poisson Datafit Yields Incorrect Predictions #320

Checks before merging PR

• added documentation for any new feature
• added unit tests
• edited the what's new (if applicable)

[mathurinm]

there may be an issue here, because intercept_update_step does not compute the gradient, but a scaled version of it (it's mulliplied by the stepsize).
The safest way to do it would be to call raw_grad().sum(), which is equivalent to np.ones(n_features) @ raw_grad(), e.g. the gradient with respect to a feature full of ones.

Sounds good @Badr-MOUFAD ?

[Badr-MOUFAD]

Good catch @mathurinm,

I have a little concern because it seems that in some part of the code (Logostic datafit) intercept_update_step account for the stepsize, whereas in other parts (Quadratic, Huber, Poisson, ...) intercept_update_step evaluate the gradient

That being said, I agree that the safest option is to sum(raw_grad(y, Xw)).

[mathurinm]

For Quadratic (hand Huber I guess) it's because the stepsize is 1 (lc is $| X_j |^2 / n = 1$

[Badr-MOUFAD]

you are right @mathurinm, thanks

What should we do for Poisson and Gamma datafits ? they implement intercept_update_step with the stepsize convention, but actually since they are non-quadratic it does't not make sense for them

[floriankozikowski]

@Badr-MOUFAD I tried the refactor (also considering your comment @mathurinm )
Let me know what you think. Didn't find an option to make this refactor shorter

[floriankozikowski]

Btw, looking at the initial issue #320 that initiated this PR, the intercept fitting improved the sklearn speed difference, but it is still slower.

--- Fitting Time Comparison ---
skglm (LBFGS): 8.0444 seconds
sklearn (L-BFGS): 2.4578 seconds

sklearn was 3.27x faster.

I guess in this PR, we only focus on the intercept, but maybe we should open a new issue investigating other causes for this.

If you approve the refactor, I can delete the debug script (issue320) and we can merge.

[Badr-MOUFAD]

Thanks @floriankozikowski for the timing comparison.
Weird that sklearn is x3 faster, but I agree to tackle this issue in a separate PR

Can you pls add the intercept to the unittest and update the whats'new page

[floriankozikowski]

@Badr-MOUFAD thanks for the feedback! It should be complete now and I'd say we can merge. I won't have access to my laptop the next two weeks, so if anything comes up, I will look at it mid-August again.