o ̳i*@sdddlmZmZmZddlZddlmZddlm Z ddl m Z m Z ddl m Z Gddde ZdS) )DictListOptionalN) functional)Dataset)CROSS_ENTROPY_IGNORE_IDX PACK_TYPE)tqdmc @seZdZdZdddddededed eed ed df d d Zd!ddZ d efddZ de d e fddZ de d dfddZ de d e fddZde ded e fddZd efddZded eeejffdd ZdS)" PackedDataseta Performs greedy sample packing on a provided dataset. This is done as a single preprocessing step before training begins. Shuffling is done outside of this class on packed samples with a ``Sampler`` as part of the dataloader. Currently, this only supports in-memory map-style datasets. The class loads, tokenizes, and packs examples on initialization - no tokenization is done during training. The general flow on initialization is: load tokenized sample -> add to buffer -> when buffer is long enough, add to ``self.packs``. During training, returns self.packs[idx] as input, label, attention mask, and position ids. The attention mask is a lower triangular block mask to prevent samples from cross-attending within a pack. The position ids indicate the position of each token relative to its sample within a pack. These are all padded to max sequence length, so a batch-wise collator is not needed. A packed sample is made up of individual smaller sequence length samples jammed together within ``max_seq_len``. For example, if max_seq_len is 6 and there are varied length samples:: tokens = [ [S1, S1, S1, S2, S2, pad], [S3, S3, S4, S4, pad, pad], ..., ] To prevent cross-contamination, the following mask would be returned for the first pack in the example:: mask = [ [1, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0], [0, 0, 0, 1, 0, 0], [0, 0, 0, 1, 1, 0], [0, 0, 0, 0, 0, 1], ] The position ids would be:: input_pos = [ [0, 1, 2, 0, 1, 2], [0, 1, 0, 1, 2, 3], ..., ] The identity matrix is used in the mask for pad tokens instead of a causal mask. For position ids for pad tokens, we simply continue to increment from the previous sample normally. Args: ds (Dataset): dataset to sample pack. This should return a dictionary with field "tokens" and "labels" containing the tokenized and label samples. max_seq_len (int): Maximum number of tokens to pack padding_idx (int): padding index for the tokenizer. Default is 0. max_packs (Optional[int]): Maximum number of packs. Default is None, which will create as many packs as possible. split_across_pack (bool): if the last sample in a pack does not fit in ``max_seq_len``, split the sample into the next pack, or move it entirely to the beginning of the next pack. For pre-training, typically this is set to True for general text completion. For fine-tuning, typically this is set to False to avoid truncating sentences in instruct tuning. Default is False. rNF) padding_idx max_packssplit_across_packds max_seq_lenr r r returncCs6||_||_||_||_||_g|_d|_|dS)Nr)rrr r r packsprevious_sample_boundary_pack)selfrrr r r rN/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/datasets/_packed.py__init__Ss  zPackedDataset.__init__csggggd}tjrtjrtjnd}|dkr&ttjddd}jD]}|d|d}}t|}|jkrLj sLt d|d jd |d|7<|d|7<|d fd d t |D7<|d|g7<t|djkr s |}t|djkr r|dkr|t|d_ rnq)t|ddkrLjjdustjjkrɈ|dSdSdS)zIterate through the dataset. Use a buffer to hold samples until max_seq_len, then append the buffer to self.packs as a single "packed" sample. Continue until max_packs or end of dataset.tokenslabels input_posseq_lensrzPacking datasetT)totaldesc dynamic_ncolsrrzDataset sample is too long (z > zA). Please set `split_across_pack=True` or increase `max_seq_len`.rcsg|]}|jqSr)r).0xrrr sz'PackedDataset._pack..rN)torch distributed is_availableis_initializedget_rankr lenrrr ValueErrorrange_should_stop_packing_split_and_add_packupdaterr r _add_pack)r current_packrankpbarsamplerrseq_lenrr"rrfsT   " zPackedDataset._packcCs"|jdurt|j|jkrdSdS)zs