dataset
dataset
¶
DatasetConfig(suffix: str = field('data/suffix', default=''))
dataclass
¶
Configuration shared by materialized datasets.
Attributes:
| Name | Type | Description |
|---|---|---|
suffix |
str
|
Optional suffix appended to the registered dataset key when locating the materialized directory. |
DatasetComponent
¶
Configuration contract shared by raw dataset families.
config() -> list[type[Any]]
classmethod
¶
Return configuration schemas needed by this dataset.
Returns:
| Type | Description |
|---|---|
list[type[Any]]
|
The common materialization schema followed by the dataset's owned |
list[type[Any]]
|
source schema, when it has one. |
PretrainingDataset
¶
Bases: StringDataset
Pretraining dataset, identical to Dataset[str] but is tokenized differently. Specifically, this dataset is tokenized irrespective of item boundaries.
StreamingPretrainingDataset
¶
Bases: StreamingStringDataset
Pretraining dataset, identical to Dataset[str] but is tokenized differently. Specifically, this dataset is tokenized irrespective of item boundaries.