Describe the workflow you want to enable

I'd like to have faster convergence of the "newton-cg" solver of LogisticRegression based on scientific publications with empirical studies as done in A Study on Truncated Newton Methods for Linear Classification (2022) (free pdf version).

Describe your proposed solution

It is about the inner stopping criterion in a truncated Newton solver, i.e. when should the inner solver for "hessian @ coefficients = -gradient" stop.

$eta = \eta$ is the forcing sequence.

Current stopping criterion

$residual ratio = \frac{\rVert res\lVert_1}{\rVert grad \lVert_1} \leq \eta$ with $res = residual = grad - hess @ coef$ and $\eta = \min([0.5, \sqrt{\rVert grad \lVert_1]})$ (this eta is called adaptive forcing sequence.

Proposed stopping criterion

As recommended by Chapter VII.

• Replace residual ratio with the quadratic approximation ratio $j\frac{Q_j - Q_{j-1}}{Q_j}$ and $Q_j = grad @ coef_j + \frac{1}{2} coef_j^T @ hessian @ coef_j$ and $j$ is the inner iteration number.
• Optionally replace L1-norm by L2-norm. For the quadratic ratio, this does not matter much.

Describe alternatives you've considered, if relevant

No response

Additional context

No response