Lingoflop

---

Lingoflop is a machine translation model based on the original Transformer (Encoder-Decoder), implemented with PyTorch.
For more details, see the paper Attention Is All You Need.

Introduction

• A machine translation model based on the original Transformer (Encoder-Decoder), implemented with PyTorch.
• A standard Encoder-Decoder Transformer built from scratch.
• A WordLevel tokenizer trained with the tokenizers library.
• The opus_books dataset loaded through Hugging Face datasets.
• Support for training, validation, model checkpoint saving, and resuming training.
• TensorBoard logging for monitoring the training process.

Quick Start

Python 3.10 or later is recommended.

Set up the environment:

conda create -n lingoflop python=3.10
conda activate lingoflop
pip install -r requirements.txt

Start training from the project directory:

python train.py

The model can be trained on a single NVIDIA RTX 4090 with 24 GB of VRAM. The default training settings are:

• batch_size = 8
• num_epochs = 20
• lr = 1e-4
• seq_len = 350
• d_model = 512
• lang_src = "en"
• lang_tgt = "fr"

To resume training from a checkpoint, update config.py:

• preload = None: train from scratch.
• preload = "xx": load weights/tmodel_xx.pt and continue training.

Training Outputs

Model weights are saved in the weights/ directory. TensorBoard logs are saved in runs/tmodel.

If TensorBoard is not installed, run:

pip install tensorboard

Launch TensorBoard with:

tensorboard --logdir=runs

Then open the following address in your browser:

http://localhost:6006

Notes

The current implementation includes:

• Encoder self-attention
• Decoder masked self-attention
• Cross-attention
• Feed-forward layers with residual connections and layer normalization

The validation stage uses greedy decoding to generate example translations.

The project uses English-to-French translation as the default example. The opus_books dataset supports additional language pairs; to use a different target language, update lang_tgt in config.py.

PS: To be honest, maintaining a codebase written entirely by hand is not a light task in today's agentic era.