Use DifferentiationInterface for autodiff, allow ADTypes

[gdalle]

Warning

This PR bumps the Julia compat to 1.10 instead of 1.5.

This PR introduces the use of ADTypes for autodiff package selection + DifferentiationInterface for autodiff calls (fixes #132).

• bump package version to 7.9.0
• bump Julia compat to 1.10
• add ADTypes and DifferentiationInterface to dependencies
• modify oncedifferentiable.jl, twicedifferentiable.jl and constraints.jl
• add docs for passing autodiff=ADTypes.AutoSomething()
• adjust count of function/gradient calls?

Tests pass locally but I might be misinterpreting the interface somehow so please double-check. I think we can handle performance improvements in a follow-up PR.

[pkofod]

Looks like I never updated CI here for Github Actions :) I like the simplifications so far

[gdalle]

Do we need complex number support? Because it is not available in every AD backend so at the moment DI is only tested on real numbers.

[gdalle]

I ran the Optim.jl test suite with the version of NLSolversBase from this PR and they seem to pass (https://github.com/gdalle/Optim.jl/actions/runs/12166910179/job/33934234795), except on Julia 1.6 because the recent versions of DI no longer support it.

[pkofod]

CI PR is merged

[gdalle]

Should I do another PR which adds code coverage to the test action?

[pkofod]

@devmotion added codecov. Should be set up now

[gdalle]

Alright, closing and reopening to get coverage (hopefully)

[gdalle]

Any idea why we didn't get a Codecov comment? The report is available here but perhaps it doesn't have the right permissions to post on the PR?

[devmotion]

Might be codecov/codecov-action#1662? A bit unclear what the problem actually is, the same config works fine in other repos. I wonder if the codecov app has to be installed for the comments to appear.

[gdalle]

What's also weird is that the Codecov report (on the website) tells me "No file covered by tests were changed", which is obviously wrong

[devmotion]

https://app.codecov.io/github/JuliaNLSolvers/NLSolversBase.jl/pull/153 states that this PR changes coverage

[gdalle]

Fixed the missed line. Should we add some docs? Or keep it on the down low until downstream tests are running smoothly too?

[pkofod]

Fixed the missed line. Should we add some docs? Or keep it on the down low until downstream tests are running smoothly too?

To be honest, I think adding docs now is better. The risk (that often ends up happening) is that we forget later. It will only be on the dev docs anyway so there's no harm.

[gdalle]

Done

[gdalle]

Sure but I'm assuming we want gradients of real-valued functions, for the purposes of optimization? If those functions have complex inputs, it will fail

julia> using ForwardDiff

julia> norm(x) = sum(abs2, x)
norm (generic function with 1 method)

julia> ForwardDiff.gradient(norm, [0.0 + im])
ERROR: ArgumentError: Cannot create a dual over scalar type ComplexF64. If the type behaves as a scalar, define ForwardDiff.can_dual(::Type{ComplexF64}) = true.

[gdalle]

And that's also why I'm reluctant to promise complex support in DI: every backend handles it differently, sometimes with different conventions, sometimes one operator works (ForwardDiff.derivative) and one doesn't (ForwardDiff.gradient), etc. All of this makes it really hard to obtain homogenous behavior

[pkofod]

I think we require user-written input in the complex case.. @antoine-levitt might remember but it's a long time ago :D I can also check later

[gdalle]

If autodiff is only used for real numbers that will make our life easier. Otherwise we can also add select complex tests to DI

[antoine-levitt]

I don't remember, but it's reasonable to treat complex numbers just like any user defined struct. If the AD backend has support for that (ie it can vjp a CustomType->Real function and give a CustomType) then great, otherwise just ask the user to convert into a vector of real -> Real function

[gdalle]

So what do we do about this PR? At the moment it passes all the tests here and all the ones in Optim.jl. Should we wait until complex numbers have more reliable handling in DI (for instance, erroring when a hessian is requested)?

[pkofod]

So what do we do about this PR? At the moment it passes all the tests here and all the ones in Optim.jl. Should we wait until complex numbers have more reliable handling in DI (for instance, erroring when a hessian is requested)?

Let me have a look over the holiday break. I have to go back and look, because I cannot remember what we actually support here, and if we do support something we cannot just remove it in a feature release.

[gdalle]

Hi @pkofod, gentle bump on this one :) Did you have time to figure out what kind of complex number support you need?

[pkofod]

Hi @pkofod, gentle bump on this one :) Did you have time to figure out what kind of complex number support you need?

I actually did not return to this after the holiday break yet. I will have a look again.

[gdalle]

No stress, take as much time as you need! I'm just excited for Optim to finally have reverse mode autodiff but it can definitely wait 😅

[gdalle]

Hey, gentle ping on this one, let me know if you need any help.

[pkofod]

Hey, gentle ping on this one, let me know if you need any help.

Yes, I have to check. We support complex input "natively", but maybe you're right that it has to be user-provided gradients in that case. I never used the functionality myself! :)

[gdalle]

Judging by this paragraph in the docs, users are invited to bring their own complex gradients or use the "built-in finite differences".

Automatic differentiation support for complex inputs may come when Cassete.jl is ready.

https://julianlsolvers.github.io/Optim.jl/dev/algo/complex/#Differentation

But I guess the "built-in finite differences" are still affected by this PR since we're taking over for autodiff=:finite? So we rely on the ability of FiniteDiff to compute complex gradients?

[pkofod]

But I guess the "built-in finite differences" are still affected by this PR since we're taking over for autodiff=:finite? So we rely on the ability of FiniteDiff to compute complex gradients?

Okay, then I think we are fine 👍

[gdalle]

Okay, then I think we are fine 👍

Should I add tests to DI making sure that complex gradients are correctly forwarded to and computed by FiniteDiff? Right now I only have some (limited) tests for holomorphic functions, and holomorphic functions with real output are trivial.

[pkofod]

Should I add tests to DI making sure that complex gradients are correctly forwarded to and computed by FiniteDiff? Right now I only have some tests for holomorphic functions, and holomorphic functions with real output are trivial.

I suppose that would be nice yes. Alternatively, there should be tests in Optim (if there aren't already) testing the finite diff version of using complex inputs in Optim.

[gdalle]

Alternatively, there should be tests in Optim (if there aren't already) testing the finite diff version of using complex inputs in Optim.

I think there are already some?

https://github.com/JuliaNLSolvers/Optim.jl/blob/b041bd63373cf2403180c6993eff235b5c010670/test/multivariate/complex.jl#L1

[gdalle]

I opened an issue in DI to add complex gradient tests: JuliaDiff/DifferentiationInterface.jl#732

[gdalle]

Tests and other fixes incoming in JuliaDiff/DifferentiationInterface.jl#733

[gdalle]

Merged, now complex gradients are tested with FiniteDiff in the DI test suite.
@pkofod the only question left to settle is that of function/gradient call count, you can just tell me what you want here. Note that beyond ForwardDiff & FiniteDiff, counting function calls may not make much sense: for instance, Enzyme does not actually call the function when computing its gradient.

[gdalle]

@pkofod i think this should be good to go for a final review

[pkofod]

@pkofod i think this should be good to go for a final review

I think I should remove appveyor, I think the integration is closed. I was just wondering at first why tests failed :) I'll have a look

[pkofod]

Merged, now complex gradients are tested with FiniteDiff in the DI test suite. @pkofod the only question left to settle is that of function/gradient call count, you can just tell me what you want here. Note that beyond ForwardDiff & FiniteDiff, counting function calls may not make much sense: for instance, Enzyme does not actually call the function when computing its gradient.

It's one of those things where if you include the f_calls for the finite difference calculations people will complain that it should only be counted towards G and if you don't then people complain that not all f calls are counted. Let me have a look and decide and it'll be a matter of documentation

[gdalle]

I think I should remove appveyor, I think the integration is closed. I was just wondering at first why tests failed :) I'll have a look

On the other hand, it might make sense to run some downstream tests for Optim here?

[gdalle]

@pkofod just let me know when you have time whether you need anything more :)