Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix batch tokenization for multiple invoke #313

Open
wants to merge 1 commit into
base: 0.4
Choose a base branch
from
Open

Fix batch tokenization for multiple invoke #313

wants to merge 1 commit into from

Conversation

cadentj
Copy link
Member

@cadentj cadentj commented Dec 30, 2024

Tweaked the indexing on _batch method in LanguageModel.

In this line

batched_inputs["attention_mask"][:1, : attention_mask.shape[1]] = attention_mask

The first [:1] batch index of the re-tokenized batch is set to the running batch's attention mask. This breaks for more than two invokes because attention_mask is shape [n - 1, seq] which is longer than the single index replacement.

Fixed to use negative indexing

batched_inputs["attention_mask"][:-1, : attention_mask.shape[1]] = attention_mask

@JadenFiotto-Kaufman
Copy link
Member

@cadentj This doesn't seem to pass the test_invoke test. in test_lm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants