tokenize
tokenize
¶
Node-backed tokenization jobs for statically declared datasets.
The materialized binary files are the state. Logged nodes only record offsets
that are safe after a flush, so these jobs use LoggingJob rather than the
model-oriented CheckpointedJob machinery.
TokenizeConfig(val_pct: float = field('data/val_pct', default=0.05), seed: int = field('data/seed', default=2357))
dataclass
¶
Configuration shared by every tokenization strategy.
TokenizeDatasetConfig(val_pct: float = field('data/val_pct', default=0.05), seed: int = field('data/seed', default=2357), block_size: int = field('architecture/block_size', default=512), pad_token: int = field('data/pad_token', default=0), system_prompt: str = field('data/system_prompt', default=''), assistant_only: bool = field('data/assistant_only', default=False), replacement: bool = field('data/tokenize/replacement', default=True), allow_truncation: bool = field('data/allow_truncation', default=True))
dataclass
¶
TokenizePretrainingDatasetConfig(val_pct: float = field('data/val_pct', default=0.05), seed: int = field('data/seed', default=2357), max_samples: int = field('data/max_samples', default=(-1)), max_tokens: int = field('data/max_tokens', default=(-1)))
dataclass
¶
TokenizeContrastiveDatasetConfig(val_pct: float = field('data/val_pct', default=0.05), seed: int = field('data/seed', default=2357), block_size: int = field('architecture/block_size', default=512), pad_token: int = field('data/pad_token', default=0), system_prompt: str = field('data/system_prompt', default=''), assistant_only: bool = field('data/assistant_only', default=False), replacement: bool = field('data/tokenize/replacement', default=True), allow_truncation: bool = field('data/allow_truncation', default=True))
dataclass
¶
TokenizeJob(spec: ExecutionSpec, base: Optional[Node] = None)
¶
Bases: LoggingJob[C], Generic[C]
Materialize one statically declared raw dataset.
Progress is durable only after the output files are flushed and the corresponding node metadata is written. Both resume and branch continue from that durable point; their distinction is only the resulting DAG edge. These jobs deliberately do not infer progress from files or legacy JSON.
output_path: Path
property
¶
Return the legacy-compatible materialized dataset path.
Returns:
| Type | Description |
|---|---|
Path
|
|
config() -> list[type[Any]]
classmethod
¶
Return tokenizer, dataset, and strategy configuration schemas.
Returns:
| Type | Description |
|---|---|
list[type[Any]]
|
Schemas needed to construct this concrete tokenization job. |
state_init() -> None
¶
Initialize fresh progress and reject an existing output path.
state_restore(state: ValueRow) -> None
¶
Restore progress from a node and validate its materialization.
restored_complete() -> bool
¶
Advance resumed tokenization and re-publish completion when present.
Returns:
| Type | Description |
|---|---|
bool
|
Whether the restored materialization was already complete. |
publish(progress: Mapping[str, bool | float | int | str]) -> None
¶
Publish a durable progress point and advance the node.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
progress
|
Mapping[str, bool | float | int | str]
|
Counters describing files already flushed to disk. |
required |
TokenizeBlockwiseDatasetJob(spec: ExecutionSpec, base: Optional[Node] = None)
¶
Bases: TokenizeJob[TokenizeDatasetConfig]
Materialize one indexable dataset as padded token and mask arrays.
run() -> None
¶
Tokenize the configured dataset, continuing from restored rows.
TokenizeContrastiveDatasetJob(spec: ExecutionSpec, base: Optional[Node] = None)
¶
Bases: TokenizeJob[TokenizeContrastiveDatasetConfig]
Materialize one paired dataset as padded positive/negative arrays.
run() -> None
¶
Tokenize paired examples, continuing from restored rows.
TokenizeVariableDatasetJob(spec: ExecutionSpec, base: Optional[Node] = None)
¶
Bases: TokenizeJob[TokenizePretrainingDatasetConfig]
Materialize one streaming dataset as contiguous train/validation tokens.
run() -> None
¶
Tokenize a stream, continuing from restored token offsets.