Loss diverges on using ConvolutionDepthwise layer.

[krraush]

Hi,
I am using the imagenet pre-trained model to finetune for classification.
If I use the DepthWiseconvolution layer, the loss first decreases and then starts to increase within 100 iterations.

However, with everything else as in the above fine tuning (i.e Base LR, solver etc), if i replace the ConvolutionDepthwise with Convolution layer the loss converges.
I tried fiddling with learning rates and solvers, but it didn't help.

I want to use the ConvolutionalDepthwise, as it is ~4 times faster than convolution.

[SuSuXia]

Can you share the ConvolutionDepthwise layer? Thanks.