Skip to content

Index

data

decode_chat_template(tokens: list[int] | str, encoder: Optional[Tokenizer] = None) -> ChatTemplate

Decode tokens back into a ChatTemplate.

Parses chatml format: <|im_start|>role message<|im_end|>

Parameters:

Name Type Description Default
tokens list[int] | str

Token list or string (if encoder is None, treated as string)

 required
encoder Optional[Tokenizer]

Tokenizer (if None, tokens is treated as the raw string)

 None

encode_chat_template(template: ChatTemplate, encoder: Optional[Tokenizer] = None, system_prompt: Optional[str] = None, prompt: bool = False, *, tokenize: bool = True) -> list[int] | str 

encode_chat_template(template: ChatTemplate, encoder: Optional[Tokenizer] = None, system_prompt: Optional[str] = None, prompt: bool = False, *, tokenize: Literal[False]) -> str

encode_chat_template(template: ChatTemplate, encoder: Tokenizer = ..., system_prompt: Optional[str] = None, prompt: bool = False, *, tokenize: Literal[True] = True) -> list[int]

Encode a chat template as tokens or formatted text.

  • For tiktoken, formatting is always ChatML.
  • For HuggingFace tokenizers, formatting uses tokenizer.apply_chat_template.

Parameters:

Name Type Description Default
template ChatTemplate

List of chat turns

 required
encoder Optional[Tokenizer]

Tokenizer to use. Required when tokenize=True.

 None
system_prompt Optional[str]

Optional system prompt to prepend

 None
prompt bool

If True, append a generation prompt for autoregressive generation

 False
tokenize bool

If True, return token ids. If False, return formatted text.

 True