How many epochs are recommended for pretraining?

[JinChengneng]

Hi there! Thanks a lot for your contribution to Pluto, which is a very novel idea.

I've tried to train a pre-trained model by executing with default hyperparameters. The command is

python pretrain.py data/pretrain/train_peptide.csv data/pretrain/dev_peptide.csv

However, the performance of the pre-trained model is far below the results shown in your paper. The same does the provided pre-trained model at Pluto/model/pretrain/*.

Could you please tell me how many epochs did you take to train the pre-trained model or provide a pre-trained model which matches the performance shown in your paper?

Thanks in advance for any replies.