Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support blockwise fp8 matmul kernel #3267

Merged
merged 3 commits into from
Feb 12, 2025
Merged

support blockwise fp8 matmul kernel #3267

merged 3 commits into from
Feb 12, 2025

Conversation

yizhang2077
Copy link
Collaborator

@yizhang2077 yizhang2077 commented Feb 3, 2025

Motivation

support fp8 blockwise kernel (Currently only supports scale_a block shapes of 1x128 and scale_b block shapes of 128x128 for deepseek v3), mainly from vllm
correctness
python3 tests/test_fp8_blockwise_gemm.py
benchmark
python3 benchmark/bench_fp8_blockwise_gemm.py

Modifications

Checklist

@yizhang2077 yizhang2077 force-pushed the support-fp8-blockwise branch from 0cee1ef to 7b9ee19 Compare February 3, 2025 10:40
@yizhang2077 yizhang2077 force-pushed the support-fp8-blockwise branch from 7b9ee19 to 50bc4d1 Compare February 9, 2025 15:09
@yizhang2077 yizhang2077 changed the title support blockwise fp8 matmul kernel [WIP] support blockwise fp8 matmul kernel Feb 9, 2025
@yizhang2077
Copy link
Collaborator Author

keep tuning

@yizhang2077 yizhang2077 force-pushed the support-fp8-blockwise branch from 50bc4d1 to 00e2441 Compare February 9, 2025 15:12
@yizhang2077 yizhang2077 changed the title [WIP] support blockwise fp8 matmul kernel support blockwise fp8 matmul kernel Feb 12, 2025
@yizhang2077 yizhang2077 force-pushed the support-fp8-blockwise branch from 3f87227 to ef513b5 Compare February 12, 2025 14:37
@yizhang2077 yizhang2077 mentioned this pull request Feb 12, 2025
8 tasks
@zhyncs zhyncs merged commit 640363a into main Feb 12, 2025
8 of 10 checks passed
@zhyncs zhyncs deleted the support-fp8-blockwise branch February 12, 2025 17:49
chongli-uw pushed a commit to chongli-uw/sglang that referenced this pull request Feb 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants