fineweb_edu
fineweb_edu
¶
FineWebEduDedup()
¶
Bases: DatasetComponent
Deduplicated FineWeb-Edu from the Track-M preprocessing pipeline.
This dataset is consumed as pre-tokenized memmaps placed under
$root/data/track_m_fineweb_edu_dedup (or ..._<suffix> when
data/suffix is set), with a required train.bin and optional
val.bin. Both files must be contiguous np.uint32 token-id streams
(cl100k_base tokenization), matching the PMD loader format.