Skip to content

theseus

Home
Getting Started
Tutorial
Tutorial
How-to Guides
How-to Guides
- Analysis System
- Evaluation System
- Remote Configuration
  Remote Configuration
- Adding Things
  Adding Things
  - Overview
  - Adding Components
    Adding Components
    
    Overview
    
    Model
    
    Experiment
    
    Dataset
    
    Evaluation
    
    Analysis Job
    
    Optimizer
    
    Learning Rate Schedule
  - Extending Components
    Extending Components
    
    Overview
    
    Trainer
    
    Module
    
    GPT
    
    Transformer Block
    
    Attention
    
    MLP
    
    Mixture of Experts
Design
Design
Reference
Reference
- analysis
  analysis
  - Index
  - attention
  - base
  - plots
  - trace
- base
  base
  - Index
  - axis
  - chip
  - dag
  - hardware
  - job
  - topology
- checkpoint
- cli
  cli
  - Index
  - app
  - cleanup
  - configurator
    configurator
    
    Index
    
    fields
    
    steps
  - configure
  - interface
    interface
    
    Index
    
    actions
    
    age
    
    cache
    
    chart
    
    data
    
    driver
    
    fields
    
    location
    
    log_data
    
    log_list
    
    log_view
    
    navigation
    
    plot
    
    plot_controls
    
    plot_math
    
    plotly_options
    
    run
    
    state
    
    status
    
    timeline
    
    tree
    
    tree_row
    
    view
    
    views
    
    walk
  - jobs
  - run
  - submit
  - ui
- config
- data
  data
  - Index
  - datasets
    datasets
    
    Index
    
    addition
    
    alpaca
    
    bbq
    
    ccaligned
    
    cfq
    
    clutrr
    
    dataset
    
    fever
    
    fineweb
    
    fineweb_edu
    
    flan
    
    harmfulqa
    
    longbench
    
    longhealth
    
    mmlu
    
    mnli
    
    mtob
    
    openr1_math
    
    pes2o
    
    pg19
    
    pile
    
    pile_detoxify
    
    pile_injected
    
    qqp
    
    siqa
    
    squad
    
    sst2
    
    winogrande
  - tokenize
  - tokenizer
- evaluation
  evaluation
  - Index
  - base
  - datasets
    datasets
    
    Index
    
    addition
    
    alpaca
    
    arc_challenge
    
    arithmetic
    
    bbh
    
    bbq bbq
    On this page
    
    bbq
    
    BBQEval
    
    blimp
    
    ccaligned
    
    cfq
    
    clutrr
    
    fever
    
    gsm8k
    
    hellaswag
    
    longbench
    
    longhealth
    
    math
    
    mmlu
    
    mnli
    
    mtob
    
    perplexity_evals
    
    pes2o
    
    pg19
    
    pg19_lengthgen
    
    pile
    
    pile_injected
    
    qqp
    
    siqa
    
    squad
    
    sst2
    
    tinystories
    
    winogrande
- execute
  execute
  - Index
  - bootstrap
  - bundle
  - combobulator
  - config
  - dispatch
  - provider
    provider
    
    Index
    
    base
    
    slurm
    slurm
    
    Index
    
    availability
    
    provider
    
    ssh
    
    utils
    
    volcano
    volcano
    
    Index
    
    config
    
    manifest
    
    transport
  - run
    run
    
    Index
    
    lineage
    
    run
  - solve
- experiments
  experiments
  - Index
  - benchmark
  - data
    data
    
    Index
    
    blockwise
    
    streaming
  - forking
    forking
    
    Index
    
    pretrain
  - models
    models
    
    Index
    
    gpt
    
    gpt_neox
    
    llama
    
    qwen
    
    qwen_3_5
- inference
  inference
  - Index
  - base
- job
- model
  model
  - Index
  - activations
    activations
    
    Index
    
    swiglu
  - attention
    attention
    
    Index
    
    base
    
    forking
    
    gated_delta
    
    grouped
    
    rope
  - axes
  - block
    block
    
    Index
    
    block
    
    forking
    
    gpt_neox
    
    llama
    
    qwen
    
    qwen_3_5
  - debug
  - layers
    layers
    
    Index
    
    layernorm
    
    mlp
    
    mrope
    
    rmsnorm
    
    rope
  - masks
  - models
    models
    
    Index
    
    base
    
    contrib
    contrib
    
    Index
    
    gpt_neox
    
    llama
    
    marin
    
    qwen
    
    qwen_3_5
    
    qwen_3_5_moe
    
    thoughtbubbles
  - module
  - moe
    moe
    
    Index
    
    base
    
    shared
  - utils
- quick
- registry
- store
- training
  training
  - Index
  - backbone
  - base
  - contrastive
  - flywheel
    flywheel
    
    Index
    
    contrastive
    
    dataset
    
    io
    
    padded
    
    pmd
    
    strategy
    
    stream
  - kl_divergence
  - optimizers
    optimizers
    
    Index
    
    adamw
    
    base
    
    muon
  - profiler
  - schedules
    schedules
    
    Index
    
    base
    
    cosine_rewarm
    
    wsd
    
    wsds
  - utils
About

bbq

`bbq` ¶

`BBQEval()` ¶

Bases: RolloutEvaluation

BBQ bias evaluation (all categories, test split).