Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] multi-GPU rerank retrievalタスクにおいてOOM防止のための修正 #51

Merged
merged 4 commits into from
Aug 5, 2024

Conversation

akiFQC
Copy link
Collaborator

@akiFQC akiFQC commented Aug 5, 2024

関連する Issue / PR

#50

PR をマージした後の挙動の変化

  • 緊急のdebugとして、TransfomersEmbedderで複数GPUを使った場合のOOMを防ぐ

挙動の変更を達成するために行ったこと

  • (緊急のdebugとして) torchrunのそれぞれのプロセスで、異なるGPUで類似度行列の計算を行う
  • 類似度行列のchunk sizeを1000000→10000に変更

動作確認

  • テストが通ることを確認した
  • マージ先がdevブランチであることを確認した

@akiFQC akiFQC requested a review from lsz05 August 5, 2024 02:45
@lsz05 lsz05 changed the base branch from main to dev August 5, 2024 02:55
@lsz05 lsz05 changed the title Fix/multi gpu rerank retrieval [Fix] multi-GPU rerank retrievalタスクにおいてOOM防止のためのdefault chunk size変更 Aug 5, 2024
@lsz05 lsz05 changed the title [Fix] multi-GPU rerank retrievalタスクにおいてOOM防止のためのdefault chunk size変更 [Fix] multi-GPU rerank retrievalタスクにおいてOOM防止のための修正 Aug 5, 2024
@akiFQC akiFQC added the bug Something isn't working label Aug 5, 2024
@akiFQC akiFQC self-assigned this Aug 5, 2024
Copy link
Collaborator

@lsz05 lsz05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTMです

@lsz05 lsz05 merged commit 0810589 into dev Aug 5, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants