ReverseDiff support in an extension

[mohamed82008]

It would be cool to make ReverseDiff work with this using the ReverseDiff.@grad macro. Examples: https://github.com/JuliaDiff/ReverseDiff.jl/blob/master/test/MacrosTests.jl#L184

[gdalle]

I thought ReverseDiff was already compatible with ChainRules?

[mohamed82008]

You can "import" rules into ReverseDiff using @grad_from_chainrules. But the macro doesn't support callable structs and it needs this PR (JuliaDiff/ReverseDiff.jl#232) to support custom input types. But we will need an extension anyways because it's an opt-in system.

[gdalle]

So how do we define rules then?

[mohamed82008]

We define an internal function which takes the implicit function as its first argument. Then we define the rule on the internal function.

(f::ImplicitFunction)(args...; kwargs...) = call_implicit_function(f, args...; kwargs...)

[gdalle]

Do we have to move all rules to this weird scheme or is it just for ReverseDiff?
Cause redefining (f::ImplicitFunction)(args...; kwargs...) would be bad

[mohamed82008]

preferably we should move all rules to be consistent, defining that function is just an extra line of code

[gdalle]

The ChainRules docs says ReverseDiff doesn't support calling back into AD. That means we cannot use it for both the implicit function and the conditions.
However, once #80 is merged, we can probably use ReverseDiff for the conditions and another backend for the implicit function. Can we do the other way around? Not sure.

[gdalle]

Thanks to DI we can use ReverseDiff for the conditions, but I don't think it will ever work for the outer backend