o wiB- @sLddlZddlmZmZmZddlZddlmZddlm Z ddgZ deee de d e d e fd d Z d ee de d eee fddZ d ee de d eee fddZd ee de d eee fddZdejde fddZ d dee de ded eeee effddZdeee dee eefde de d eef ddZdS)!N)DictListTuple)tqdm)loggingfirst_fit_decreasingfirst_fit_shufflebinssbin_sizereturncCs.t|D]\}}t|||kr|SqdS)a Finds the first bin in a list of bins that has enough space to fit a sequence of size 's'. Args: bins: A list of lists, where each inner list represents a bin and contains the current elements in that bin. s: The size of the sequence to be placed in a bin. bin_size: The maximum capacity of each bin. Returns: The index of the first bin that can fit the sequence 's', or -1 if no such bin exists. ) enumeratesum)r r r iabinr^/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/utils/sequence_packing_utils.pyfind_first_bin_that_fitss  rseqlens pack_sizecCsBg}|D]}t|||}|dkr||gq|||q|S)a Packs sequences of varying lengths into bins using the First-Fit algorithm. Args: seqlens: A list of integers, representing the lengths of the sequences to be packed. pack_size: The maximum capacity of each bin. Returns: A list of lists, where each inner list represents a bin and contains the indices of the sequences assigned to that bin. r )rappend)rrresr first_binrrr first_fit,s  rcCst|dd}t||S)a Packs sequences of varying lengths into bins using the First-Fit Decreasing algorithm. This is a variation of the First-Fit algorithm where the sequences are sorted by decreasing length before packing. Args: seqlens: A list of integers, representing the lengths of the sequences to be packed. pack_size: The maximum capacity of each bin. Returns: A list of lists, similar to the output of the 'first_fit' function. T)reverse)sortedr)rrsorted_seqlensrrrrBs cCs"|dd}tj|t||S)a Packs sequences of varying lengths into bins using the First-Fit with Shuffling algorithm. This variation shuffles the order of the sequences before applying the First-Fit algorithm. Args: seqlens: A list of integers, representing the lengths of the sequences to be packed. pack_size: The maximum capacity of each bin. Returns: A list of lists, similar to the output of the 'first_fit' function. N)nprandomshuffler)rrshuffled_seqlensrrrrSs  datasettruncate_seq_lencCstdtt}dg|d}|D]}t|dd}|||||d7<qtdt|g}t|dD] }|t||q?||fS)a Creates a histogram of sequence lengths from a tokenized dataset. This function analyzes the tokenized dataset and creates a histogram showing the distribution of sequence lengths. Args: dataset: A NumPy array containing the tokenized sequences. Each element is a dictionary that contains at minimum the key `input_ids`. truncate_seq_len: The maximum sequence length to consider in the histogram. Returns: sequences: A dictionary where keys are sequence lengths and values are lists of corresponding sequences from the dataset. histogram: A list representing the histogram data (number of sequences for each length). z,Creating histogram from tokenized dataset...r input_idszHistogram of sequence lengths) rinfo collections defaultdictlistlenrdebugrange)r"r# sequencescounts item_dictseq_len histogramrrr create_histes    r2r1packing_algorithmcCstd|dg}t|D] \}}||g|qt|}|||}dd|D}t|t|} t|} tdd|D} t|} | | t| dtt |t||dd|| d} t d t |td t |t||dd d td |d| dd|| fS)a Packs sequences into bins using the specified packing algorithm. This function takes the histogram of sequence lengths, desired pack size, and a string representing the packing algorithm to use. It then calls the corresponding function (e.g., 'first_fit_decreasing') and performs the packing process using only sequence lengths as input (without the actual sequences). Args: histogram: A list representing the histogram data (number of sequences for each length). pack_size: The maximum capacity of each bin. packing_algorithm: One of the supported packing algorithms from ['first_fit_decreasing', 'first_fit_shuffle'] Returns: assignments: A list of lists, where each inner list represents a bin and contains the indices of the sequence lengths assigned to that bin. pack_metadata: A dict that records packing metadata, for instance the max number of samples per bin. zPacking sequences to length z...cSg|]}t|qSr)r.0xrrr z+create_packing_strategy..cSr4r)r*)r6brrrr8r9d)dataset_max_seqlenmax_samples_per_binpacking_factorpacking_efficiencyrmin_packed_seqlenzPacked sequence lengths:z Packing is z.2fz % efficientz>>>>> For pack size z., average number of sequences per pack is n = z.3fz <<<<<) rr&rextendglobalsr*maxminroundrr+)r1rr3 all_seq_lensrcount packing_fn assignmentspacked_seq_lensr? max_seqlenr>rApacking_metadatarrrcreate_packing_strategys2    (rNrJr-pad_idcs4t}tt|dD]~}||}t|dkrtjt|}tdd|D|}ztdd|D|} dd| D} Wn>t yztfdd|D|} Wn"t y} zd} | | d |d7} t | t | d } ~ wwYnw|| f||<q iii}} } tt |t|d D]?\} }ggdg}}}|D]}|||d|||d|t|q||| <|| | <|d d | | <qg}tt|D]}||| || |d }||qtdd|DsJdtdd|DsJd|S)a Fills the packing strategy with actual sequence data based on assignments and sequence information. This function takes the assignments generated by the packing algorithm (containing sequence length indices), the original sequences data, and the pack size. It iterates through the assignments, retrieves the corresponding sequences from the sequences dictionary, and constructs the final output data structure with input IDs, loss masks (if available), and starting indices for each sequence in a packed sequence. Args: assignments: A list of lists, where each inner list represents a bin and contains the indices of the sequence lengths assigned to that bin (output of 'create_packing_strategy'). sequences: A dictionary where keys are sequence lengths and values are lists of corresponding sequences from the dataset (output of 'create_hist'). pack_size: The maximum capacity of each bin. pad_id: The tokenizer's padding token. Returns: output_data: A list of dictionaries, where each dictionary represents a packed sequence with its input IDs, loss mask (if available), and starting indices. r$rcSg|]}|dqS)r%rr5rrrr8r9z)fill_packing_strategy..cSrP) loss_maskrr5rrrr8r9cSsg|] }|dddgqS)r$NFrr5rrrr8scs,g|]fddttdDqS)cs,g|]}|ddkod|kqS)answer_start_idxr$r%r)r6idx)rOr7rrr8sz4fill_packing_strategy...r%)r,r*)r6rO)r7rr8s  z?Key errors loss_mask and answer_start_idx missing in example -  N)totalr )r%rQ seq_start_idcs|]}|d VqdS)rNrr6seqrrr z(fill_packing_strategy..z4Error: There are items left over from the assignmentcsrX)r$NrrYrrrr[r\)dictrr,r*rr permutationarraytolistKeyErrorrerror ValueErrorrrBpoprallvalues)rJr-rrO ifile_handlesr0 per_seq_datapermr%rQerrerr_msgrWoindex assignment _input_ids _loss_mask _seq_start_id seq_length output_datarr/rrTrfill_packing_strategysf           rs)r)r'typingrrrnumpyrr nemo.utilsr PACKING_ALGOSintrrrrr_r2strr]rNrsrrrrs@  """"* 4