Tokenizer and LLM training study
Clustering the token embeddings of GPT-2, Llama, and FastText, then using the clusters to reframe next-token prediction as next-multi-token prediction.
Private project, source not public
- Studied the token embeddings of GPT-2, Llama, and FastText using K-Means to identify semantic clusters
- Reframed standard next-token prediction as a next-multi-token prediction task using the clusters
- Used the clusters to change the traditional cross-entropy loss to binary cross-entropy (BCE)
- Date
- Aug 2025
- Domain
- Agents & LLMs
- Stack
- BPE, K-Means, PCA, cuML