Datasets

Ready-to-use pipelines for some popular datasets

lotd.datasets.alpaca 

alpaca(tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], cache_path: Union[str, None] = None, max_length: int = 512, batch_size: int = 16, seed: int = 42, pre: Union[Callable, None] = None, post: Union[Callable, None] = None) -> Tuple[DataLoader, DataLoader, DataLoader]

Downloads and Pre-processes Alpaca dataset for instruction fine-tuning.

Parameters:
  • tokenizer (Union[PreTrainedTokenizer, PreTrainedTokenizerFast]) –

    transformers tokenizer with chat_template parameter set.

  • cache_path (Union[str, None], default: None ) –

    path to load/save dataset.

  • max_length (int, default: 512 ) –

    maximum sequence length for filter.

  • batch_size (int, default: 16 ) –

    dataloaders batch size.

  • seed (int, default: 42 ) –

    splitting and shuffling seed.

  • pre (Union[Callable, None], default: None ) –

    see PadCollator.

  • post (Union[Callable, None], default: None ) –

    see PadCollator.
Returns:
  • Tuple[DataLoader, DataLoader, DataLoader] –

    a tuple of train, validation and test dataloaders.

lotd.datasets.tinystories 

tinystories(tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast], template: str = '[CLS]{{tex}}[SEP]', cache_path: Union[str, None] = None, max_length: int = 512, batch_size: int = 16, seed: int = 42, pre: Union[Callable, None] = None, post: Union[Callable, None] = None) -> Tuple[DataLoader, DataLoader, DataLoader]

Downloads and Pre-processes TinyStories dataset for language modeling. train and validation are merged, shuffled and re-split with 80%/10%/10% ratio.

Parameters:
  • tokenizer (Union[PreTrainedTokenizer, PreTrainedTokenizerFast]) –

    transformers tokenizer.

  • template (str, default: '[CLS]{{tex}}[SEP]' ) –

    prompt template with {{text}} placeholder.

  • cache_path (Union[str, None], default: None ) –

    path to load/save dataset.

  • max_length (int, default: 512 ) –

    maximum sequence length for truncation.

  • batch_size (int, default: 16 ) –

    dataloaders batch size.

  • seed (int, default: 42 ) –

    splitting and shuffling seed.

  • pre (Union[Callable, None], default: None ) –

    see PadCollator.

  • post (Union[Callable, None], default: None ) –

    see PadCollator.
Returns:
  • Tuple[DataLoader, DataLoader, DataLoader] –

    a tuple of train, validation and test dataloaders.