perplexity_evals

`perplexity_evals` ¶

Perplexity evaluations for existing datasets with validation splits.

These complement the existing RolloutEvaluation counterparts by measuring how well the model predicts validation-set tokens (1/perplexity, higher is better), which is especially useful for tracking forgetting in continual learning experiments.

`MNLIPerplexityEval()` ¶

Bases: PerplexityEvaluation

Perplexity on MNLI validation_matched split.

`QQPPerplexityEval()` ¶

Bases: PerplexityEvaluation

Perplexity on QQP validation split.

`SST2PerplexityEval()` ¶

Bases: PerplexityEvaluation

Perplexity on SST-2 validation split.

`SIQAPerplexityEval()` ¶

Bases: PerplexityEvaluation

Perplexity on Social IQa validation split.

`WinograndePerplexityEval()` ¶

Bases: PerplexityEvaluation

Perplexity on Winogrande validation split.

`FineWebPerplexityEval()` ¶

Bases: PerplexityEvaluation

Perplexity on a sample from FineWeb.

`PileDetoxifyPerplexityEval()` ¶

Bases: PerplexityEvaluation

Perplexity on a sample from tomekkorbak/pile-detoxify.

perplexity_evals