An LLM tokenizer implemented as a streamly application - #78

Open

twitu wants to merge 10 commits into

composewell:masterfrom

Commits on Feb 2, 2025

Add byte indexing logic
twitu
committed
7dfe7b4

Commits on Feb 14, 2025

Byte mapping maps sequences of bytes to index
twitu
committed
4aad2ed
Count and merge frequent pairs
twitu
committed
05006e8

Commits on Feb 15, 2025

Replace most frequent pair with new index in stream
twitu
committed
82ca052
Stream updated mappings
twitu
committed
0efe1c6
Tokenize string with byte mapping
twitu
committed
90fb23f

Commits on Feb 17, 2025

Fix tokenizer logic
twitu
committed
372d676
Cleanup
twitu
committed
9dc288a
Add example executable to cabal and config
twitu
committed
21655cf
Add run instructions
twitu
committed
52f5a3e