Machine Learning @ UVA: Transformer Workshop

Part 1: Word Embeddings Demo

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install numpy matplotlib

Train a tiny embedding model

python part-1/src/embedding_demo.py --mode train --corpus --window 2 --dim 20 --top-n 30

Train on PTB with a subset + progress

python part-1/src/embedding_demo.py --mode train --corpus part-1/data/ptb.train.txt --max-lines 2000 --progress-every 200 --top-n 200

python part-1/src/embedding_demo.py --mode train --corpus part-1/data/ptb.train.txt --max-tokens 50000 --progress-every 5000

Use a pretrained embedding file

python part-1/src/embedding_demo.py --mode pretrained --pretrained part-1/data/pretrained_vectors.txt --top-n 30

Query nearest neighbors

python part-1/src/embedding_demo.py --mode train --query king queen car

Files

• src/embedding_demo.py: Training + visualization script.
• data/toy_corpus.txt: Small corpus for training in a workshop setting.
• data/pretrained_vectors.txt: Tiny pretrained vectors (word2vec text format).