Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RoPE bias #7

Open
JaheimLee opened this issue Apr 11, 2023 · 4 comments
Open

RoPE bias #7

JaheimLee opened this issue Apr 11, 2023 · 4 comments

Comments

@JaheimLee
Copy link

JaheimLee commented Apr 11, 2023 •
edited
Loading

llama应该默认没有启用bias项。但按照苏神最新思路，把q,k的bias项加回来可以明显提升长度外推性能，作者考虑预训练测试一下不
https://kexue.fm/archives/9577
@s-JoL
Copy link
Owner

s-JoL commented Apr 11, 2023

赞，再预训练的时候我会加上

@chensongcan
Copy link

请问目前公开的代码预训练的时候加上bias项了吗

@s-JoL
Copy link
Owner

s-JoL commented Apr 24, 2023

目前加上bias项的还在训练中，将在训练好之后再release出相应代码

@chensongcan
Copy link

这个项目是基于llama从0开始训练是吗

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@s-JoL @JaheimLee @chensongcan