FIM: Filling in the Middle for Language Models
Failed to add items
Add to basket failed.
Add to Wish List failed.
Remove from Wish List failed.
Follow podcast failed
Unfollow podcast failed
FIM: Filling in the Middle for Language Models
-
Narrated by:
-
By:
About this listen
This 2022 academic paper explores Fill-in-the-Middle (FIM) capabilities in causal decoder-based language models, demonstrating that these models can learn to infill text effectively by simply rearranging parts of the training data. The authors propose a method where a middle section of text is moved to the end of a document during training, showing this data augmentation does not negatively impact the model's original left-to-right generative ability. The research highlights the efficiency of FIM training, suggesting it should be a default practice, and offers best practices and hyperparameters for optimal performance, particularly noting the superiority of character-level span selection and context-level FIM implementation. They also introduce new benchmarks to evaluate infilling performance, emphasizing the importance of sampling-based evaluations over traditional perplexity measures for gauging real-world utility.
Source: https://arxiv.org/pdf/2207.14255