Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve U-Net and Transformer implementations #8

Open
cabralpinto opened this issue Aug 29, 2023 · 0 comments
Open

Improve U-Net and Transformer implementations #8

cabralpinto opened this issue Aug 29, 2023 · 0 comments
Labels
enhancement New feature or request help wanted Extra attention is needed question Further information is requested

Comments

@cabralpinto
Copy link
Owner

As the documentation states, the current implementations are neither the most effective or efficient. The U-Net implementation was adapted from the The Annotated Diffusion Model and the Transformer implementation was adapted from Peebles & Xie (2022) (adaptive layer norm block). Although these produce good enough results, ideally the library would provide the best implementations out there for general use.

From what I've read, I think a good choice for the U-Net implementation would be the one used in Imagen for the Text-to-Image model, but there may well be other more recent architectures that would be a better fit. For the Transformer I'm really not sure right now. Any input on this would be greatly appreciated.

@cabralpinto cabralpinto added enhancement New feature or request help wanted Extra attention is needed question Further information is requested labels Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant