Can't run anything

[jubueche]

Hi,

I tried a bunch of times now, but I can't run anything in this repo. Can you give instructions how to download the model and where to save it in a little more detail?
In general, one end-to-end demonstration on one of the datasets in the paper would be nice where the model is loaded (with a path that is passed).

All the best,
Julian

[zzy14]

Hi Julian,

I have updated the T5 example. Using the default args, you can run T5-base on SST-2 dataset.

[jubueche]

Thanks. I generated the 96 experts (k=20) and ran the ground-truth example. However, I only get 93.46 as a score. Also, I don't understand this line.

I am probably using too small experts/too few k.
[EDIT] If I comment out the line where the model is modified, and run the inference, I get 0.93922. So the baseline seems to be the problem.

All the best,
Julian

[zzy14]

Yes, it seems to be correct. The reporter results in Table 2 are based on T5-Large. You can use larger models to get higher scores.

In this line, we compare the probabilities of two label words, representing "positive" and "negative" respectively. You can check the original paper of T5 for more details (Appendix D).

[jubueche]

Ok, that makes sense. I really appreciate it. I have some more questions but will reach out privately. Thanks a lot again.