FEAT Add log-sum penalty

[PABannier]

Context of the PR

This PR adds the log-sum penalty. Details about the penalty can be found in [1].
Closes #108.

Contributions of the PR

• Add the penalty LogSumPenalty

Checks before merging PR

• added documentation for any new feature
• added unittests
• edited the what's new

---

[1] Prater-Bennette, A., Shen, L. and Tripp, E.E., 2022. The Proximity Operator of the Log-Sum Penalty. Journal of Scientific Computing, 93(3), p.67.

[mathurinm]

@PABannier can you resurrect this? It would be a nice addition

[PABannier]

@mathurinm @Badr-MOUFAD The following script converges.

import numpy as np

from skglm.datafits import Quadratic
from skglm.penalties import LogSumPenalty
from skglm import GeneralizedLinearEstimator
from skglm.solvers import AndersonCD, FISTA
from skglm.utils.data import make_correlated_data


random_state = 113
n_samples, n_features = 50, 60

rng = np.random.RandomState(random_state)
X, y, _ = make_correlated_data(n_samples, n_features, random_state=rng)
rng.seed(random_state)

alpha = 1e-5
eps = 1e-2
tol = 1e-5

est = GeneralizedLinearEstimator(
        datafit=Quadratic(),
        penalty=LogSumPenalty(alpha=alpha, eps=eps),
        solver=AndersonCD(tol=tol, ws_strategy="fixpoint", verbose=2),
).fit(X, y)

print("Done.")

I did not find the definition of the subdifferential of the log-sum penalty.

[mathurinm]

For the subdifferential, since the penalty log(1 + |x|) is differentiable left and right at 0, I'd say [-1/eps, 1/eps] at 0, and {sign(x)/(eps + |x|)} elsewhere?

[mathurinm]

We talked about testing the prox procedure with @Badr-MOUFAD and came up with the following suggestion:

• define the prox objective function F for a few values of epsilon and x's (in particular above/below the threshold)
• compute its argmin with the formula
• compute F on a grid of 1000 points with reasonable values (value between 0 and 10 * prox?)
• check that F(prox) is lower than all the 1000 values

WDYT?

[Badr-MOUFAD]

Currently, the unit test checking prox computation of the penalty fails, which hints at (perhaps) a wrong prox result.

[Badr-MOUFAD]

LGTM 🚀

Many thanks @PABannier for the hard work 🙏

[PABannier]

@Badr-MOUFAD Thank you for giving it the final touch :) Much appreciated.