This repository contains a Rust implementation of a transformer built from scratch. The transformer is configured for sentiment analysis, and a dataset of movie reviews is included in the repository for training and testing the transformer.
This project implements an encoder-only transformer, built without any external machine learning libraries. The transformer uses word embeddings generated from my WordEmbeddings repository to encode words. Sample word embeddings are included in the repository.
To use this transformer implementation, you must have Rust and Cargo installed on your machine. After installing Rust and Cargo, you can clone this repository to your local machine.
To run the transformer, use the following command:
$ cargo run --release
This command will train the transformer on the movie review dataset and then run tests on a test set. The results of the training and testing will be printed to the console.
This specific model was trained with 12 words & 50 dimensions per sample, 2 encoder blocks, 3 attention heads, a hidden layer size of 400 and a learning rate of 0.001.
Original paper: Attention Is All You Need
Original paper for dataset: Learning Word Vectors for Sentiment Analysis
For more information about this project, read my blog post on transformers.
The source code for the transformer is licensed under the GNU Affero General Public License v3.0 - see the LICENSE file for details.