Dataset processors and filters

Callable objects passed to HF dataset map() and filter() functions.

lotd.TextTokenizer

Applies template and tokenizes texts in a dataset.

__call__ 

__call__(texts: List[str]) -> Dict[str, List[List[int]]]

Tokenizes a batch of text samples.

Parameters:
  • texts (List[str]) –

    list of strings.
Returns:
  • Dict[str, List[List[int]]] –

    dict with input_ids and prompt_mask for all text in a batch.

__init__ 

__init__(tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], template: str = '[CLS]{{text}}[SEP]', max_length: Union[int, None] = None) -> None

Initializes the dataset tokenizer.

Parameters:
  • tokenizer (Union[PreTrainedTokenizer, PreTrainedTokenizerFast]) –

    pre-trained transformers tokenizer.

  • template (str, default: '[CLS]{{text}}[SEP]' ) –

    jinja2 template containing {{text}} keyword.

  • max_length (optional, default: None ) –

    maximum sequence length after truncation.

lotd.ChatTokenizer

Applies chat template and tokenizes messages in a dataset.

__call__ 

__call__(prompts: Union[List[str], List[List[str]]], responses: Union[List[str], List[List[str]]] = [], system: Union[List[str], None] = None) -> Dict[str, List[List[int]]]

Tokenizes chats.

Parameters:
  • prompts (Union[List[str], List[List[str]]]) –

    a list where each item is either a single prompt or multiple prompts in a dialog.

  • responses (Union[List[str], List[List[str]]], default: [] ) –

    a list where each item is either a single response or multiple responses in a dialog.
Returns:
  • Dict[str, List[List[int]]] –

    dict with input_ids and prompt_mask for all text in a batch.

__init__ 

__init__(tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], max_length: Union[int, None] = None) -> None

Initializes the dataset chat tokenizer.

Parameters:
  • tokenizer (Union[PreTrainedTokenizer, PreTrainedTokenizerFast]) –

    pre-trained transformers tokenizer.

  • max_length (optional, default: None ) –

    maximum sequence length after truncation.

lotd.LengthFilter

Filters datasets by max and min length of input_ids.

__call__ 

__call__(input_ids: List[List[int]]) -> List[bool]

Processes a batch of input_ids.

__init__ 

__init__(min_length: int = 0, max_length: Union[int, None] = None) -> None

Initializes the collator.

Parameters:
  • min_length (int, default: 0 ) –

    lowest number of tokens.

  • max_length (Union[int, None], default: None ) –

    highest number of tokens.