pile

Bases: StreamingPretrainingDataset

The Pile (EleutherAI) for general pretraining.

Streams text from EleutherAI/pile, an 825 GiB diverse open-source language modelling dataset. Uses parquet auto-convert to bypass deprecated custom loading scripts.