README.md

✂️Cut-Cross-Entropy-Pytorch

It’s Pytorch-version Cut-Cross-Entropy(CCE) implementation:

1. GEMM
2. LSE-style CCE Forward
3. LSE CCE Backward
4. Linear-Cross-Entropy backward

Here is a blog for explaination of CCE: zhihu

Reference

Cut Your Losses in Large-Vocabulary Language Models