Infill a sentence as a continuation of a conditioning token?

[jbmaxwell]

I'd like to be able to do a version of sentence infilling that allows for conditioning the generation on a leading token—i.e., semantically prompting the generated infill sentence. In your estimation, would it be a big job to enable this kind of generation? I'm thinking of the way that initial words like "however", "therefore", "further", and so on can have a strong semantic effect on the kind of sentence infill generated.

[chrisdonahue]

Sorry for the delayed response. I don't think this would be a huge job. All you would have to do is create a new mask function which does exactly this masking task of preserving the first word of a sentence but masking out the rest.

For example, for the sentence:

Therefore, the idea was straightforward.

You would want to produce a training example of the form:

Therefore, [blank] [sep] the idea was straightforward. [answer]

Does this help?

[jbmaxwell]

Ah, right—a custom mask function... makes sense. I can't dig into this for another week, or so, but will give it a go when I'm back on this task. Thanks!

This is a very cool tweak on applied Transformers, btw! 👍

[chrisdonahue]

Thank you!

Also, I just thought about this again and realized that you absolutely don't need a custom mask function for this. You just need to train a standard ILM to perform sentence infilling, and then condition the generation on the initial word.

E.g., pass the model [blank_sentence] [sep] Therefore, as input and have it generate until it produces an [answer] token.

[jbmaxwell]

Ah, okay! I'll give this a go first.

I can't remember what I tried, but I think I was just trying things in the Jupyter notebook and wasn't sure how infill_word or infill_sentence would do this. Sounds like it would just be a matter of writing a new "infill" function(?).