Choice of second-order AD and warnings

[gdalle]

This issue is about the machinery for choosing backends and throwing warnings:

OptimizationBase.jl/src/adtypes.jl

Lines 222 to 236 in 2ffab7e

	function generate_adtype(adtype)
	if !(adtype isa SciMLBase.NoAD || adtype isa DifferentiationInterface.SecondOrder ||
	adtype isa AutoZygote)
	soadtype = DifferentiationInterface.SecondOrder(adtype, adtype)
	elseif adtype isa AutoZygote
	soadtype = DifferentiationInterface.SecondOrder(AutoForwardDiff(), adtype)
	elseif adtype isa DifferentiationInterface.SecondOrder
	soadtype = adtype
	adtype = adtype.inner
	elseif adtype isa SciMLBase.NoAD
	soadtype = adtype
	adtype = adtype
	end
	return adtype, soadtype
	end

OptimizationBase.jl/src/cache.jl

Lines 45 to 58 in 2ffab7e

	if !(prob.f.adtype isa DifferentiationInterface.SecondOrder ||
	prob.f.adtype isa AutoZygote) &&
	(SciMLBase.requireshessian(opt) || SciMLBase.requiresconshess(opt) ||
	SciMLBase.requireslagh(opt))
	@warn "The selected optimization algorithm requires second order derivatives, but `SecondOrder` ADtype was not provided.
	So a `SecondOrder` with $(prob.f.adtype) for both inner and outer will be created, this can be suboptimal and not work in some cases so
	an explicit `SecondOrder` ADtype is recommended."
	elseif prob.f.adtype isa AutoZygote &&
	(SciMLBase.requiresconshess(opt) || SciMLBase.requireslagh(opt) ||
	SciMLBase.requireshessian(opt))
	@warn "The selected optimization algorithm requires second order derivatives, but `AutoZygote` ADtype was provided.
	So a `SecondOrder` with `AutoZygote` for inner and `AutoForwardDiff` for outer will be created, for choosing another pair
	an explicit `SecondOrder` ADtype is recommended."
	end

I think that this could be both optimized and simplified due to recent changes in DI.

Nowadays, DI.inner and DI.outer can also be called on backends which are not SecondOrder, they just act as the identity. Thus, you don't need to explicitly create a SecondOrder(adtype, adtype). Passing adtype alone will be equivalent in most cases, and faster in some because it can leverage custom Hessian implementations within a single backend (e.g. SecondOrder(AutoForwardDiff(), AutoForwardDiff()) cannot call ForwardDiff.hessian whereas AutoForwardDiff() can).
Furthermore, DI's hvp and hessian for AutoZygote() already use ForwardDiff-over-Zygote.

Here are my suggestions:

• Simplify the generate_adtype logic and its variants to avoid creating SecondOrder objects altogether.
• Throw a warning based on the modes DI.inner and DI.outer, e.g. when the inner backend is not a reverse mode backend. This can be checked with ADTypes.mode(DI.inner(adtype)) isa Union{ADTypes.ReverseMode,ADTypes.ForwardOrReverseMode}. Of course you also want to allow ForwardDiff so feel free to refine.
• Document this behavior so that users are less confused by the warnings (see this Discourse thread).

What do you think @Vaibhavdixit02?

[Vaibhavdixit02]

This sounds very reasonable, sorry for the long delay in response. I will fix this up.