Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2D Linear Layer #196

Open
OneAdder opened this issue Jan 31, 2025 · 1 comment
Open

2D Linear Layer #196

OneAdder opened this issue Jan 31, 2025 · 1 comment

Comments

@OneAdder
Copy link

Proposal: introduce a 2D linear layer to be reused as a building block

It is crucial to have a simple 2D matmul layer in order to be able to build more complex algorithms such as transformers without the need to copypaste back propagation for each matrix multiplication.

Suggestion: I port my implementation here.

Back Story
I have recently started developing ML algorithms in Fortran as a hobby project. My goal is transformers. I have already started to code a bit of MultiHead Attention for neural-fortran. But decided to stop and first think about implementing a linear layer separately, not as a part of MultiHead Attention. Should I a create also a separate issue for Attention?

Discussion: Maybe make it batched right away?

@OneAdder
Copy link
Author

OneAdder commented Feb 3, 2025

#197

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant