Researchers from Microsoft and Cornell University have published a new training technique called Free Pause Tokens that improves large language models’ ability to predict the next token. This method creates a lightweight parallel prediction stream that shares weights with the model’s main stream through shared gated feedforward networks, adding minimal computation at the end of training. Tests on a 1 billion parameter model show a 2 to 3 centinats improvement in next-token prediction, with training time under 1.14 times the standard time. Inference is not slowed down because the prediction stream can be ignored at runtime, and the method is compatible with FlashAttention without requiring changes to existing infrastructure.
Source: Read the original article
