Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add BertForMaskedLM to support SPLADE Models #2550

Merged
merged 6 commits into from
Oct 7, 2024
Merged

Add BertForMaskedLM to support SPLADE Models #2550

merged 6 commits into from
Oct 7, 2024

Conversation

akshayballal95
Copy link
Contributor

SPLADE models are proving to be quite useful for document retrieval. As there was no Masked LM implementation of BERT, I have added it based on the transformers library implementation. These are the main changes:

  1. Add BertForMaskedLM in the Bert Model, which extends the Bert Model for Masked LM tasks like SPLADE.
  2. Added an example to showcase SPLADE. Merging this example is optional as it is very similar to the Bert example but with SPLADE aggregation formula.

@LaurentMazare LaurentMazare merged commit 937e8ed into huggingface:main Oct 7, 2024
9 of 10 checks passed
@LaurentMazare
Copy link
Collaborator

Thanks!

EricLBuehler pushed a commit to EricLBuehler/candle that referenced this pull request Nov 26, 2024
@akshayballal95 @LaurentMazare @EricLBuehler
Add BertForMaskedLM to support SPLADE Models (huggingface#2550)
0ca3b61 
* add bert for masked lm

* working example

* add example readme

* Clippy fix.

* And apply rustfmt.

---------

Co-authored-by: Laurent <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@akshayballal95 @LaurentMazare