modernBERT doesn't support Denoising Auto Encoder loss

[zionsoumik]

Getting error:

Model name or path "modernBERT" does not support being as a decoder. Please make sure the decoder model has an "XXXLMHead" class.

@tomaarsen

[tomaarsen]

Hello!

Indeed, the ModernBERT architecture implemented in transformers here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/modernbert/modeling_modernbert.py
does not contain a decoder class (sometimes known as a ...LMHead class, which stands for Language Modeling Head). Because of this, we cannot create a decoder with ModernBERT, so it's not possible to use DenoisingAutoEncoderLoss with ModernBERT as the decoder (but you can use it as the encoder, and another model as the decoder).

I think a ModernBERT implementation for decoders is being worked on, but I'm not sure. Otherwise, the only solution is to implement this yourself, then registering it, and then you can use it as a decoder.

• Tom Aarsen