o }oi]'@sddlZddlmZmZmZmZddlZddlZddl m Z ddl m Z m Z ddlmZddlmZGddde Zd ed ee e fd ed efd dZdS)N)IteratorListOptionalUnion)DistributedSampler)AudioToBPEDatasetAudioToCharDataset)ASRModel)loggingc@seZdZ    ddededeeded ed ed eed ed dfddZd efddZ d ee j fddZ d e eefddZd efddZdS)SemiSortBatchSamplerTFN* global_rank world_size durations batch_size batch_shuffle drop_lastrandomization_factorseedreturnc Cs|dur d}tdn td|d|dkr!td|d||_||_tj|tjd |_||_ ||_ ||_ d |_ ||_ ||_||_td dS) a  Semi Sorted Batching, as proposed in _SSB ("Speed up training with variable length inputs by efficient batching strategies.", Zhenhao Ge et al. (2021).). The Semi Sorted Batch Sampler (SSB) samples the indices by their duration with the addition of pseudo noise that is sampled from the uniform distribution \mathbb{U}\left[ -delta * r, delta * r \right], where delta is defined as the difference between the maximum and minimum duration and r is the randomization factor that controls the strength of the noise (when r = 0, there will be a strong sorting). The heuristic value of the r according to the experiments from paper is 0.2. The torch calls the set_epoch method from the distributed data loader sampler at the end of each epoch to shuffle the samples according to the seed and epoch number. So the SSB is passed to the dataloader as a sampler with the dataloader's batch size options and the batch_sampler option set to None to disable automatical batching. In this case, the sampler has become an iterator that returns a list of batch indices. Args: global_rank: Rank among all GPUs. world_size: The number of GPUs used. durations: Sample durations parsed from `dataset.manifest_processor`. batch_size: Micro batch size or batch size per singe gpu. batch_shuffle: Batch sort before each epoch. drop_last: Drop the last batch if the number of samples is less than batch size. Defaults to False. randomization_factor: The strength of noise that will be added to the sample duration. If no value is passed, the value 0.2 will be used. seed: Seed for batch shuffleling. Defaults to 42. Raises: ValueError: Wrong randomization factor value. RuntimeError: Unexpected behavior. .. SSB_: https://www.isca-archive.org/interspeech_2021/ge21_interspeech.pdf Ng?zHRandomization factor not found in config, default value 0.1 will be set.zA randomization factor z will be used.gz4Randomization factor must be non-negative but found .)dtyperz&Semi Sorted Batch Sampler will be used)r info ValueErrorrank num_replicasnparrayfloat32rshufflemicro_batch_sizerepochrr_calculate_local_num_batcheslocal_num_batches) selfr rrrrrrrr%a/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/asr_batching.py__init__s"1  zSemiSortBatchSampler.__init__cCsZt|j}|jr|||j8}t||j}|j||j|j}||7}||j}|SN)lenrrr mathceilr)r$init_num_samplesglobal_num_batchesnum_batches_padr#r%r%r&r"fs  z1SemiSortBatchSampler._calculate_local_num_batchesc Cst|j}t|j}|||jd}tjj| |t|jd}t|j|}d}|j rRt||j }tjj t||dd}t ||}t dt|dtt||j }|dkrkt d|d gS|j||j|j} ||jkrt d|d |jd | d | dkrtjjdt|| |j d} tj||| fdd }||jd|j} t|j t| |j } tj| | dd } t| |jkrtdt| d|jd| S)N)lowhighsizerF)replacezDrop last is set to True, so z samples will be dropped.zThe number of all batches is zr, than dataloader will be empty. To avoid this try to decrease batch size or world size or set drop_last to False.z', which is less than the world size of z. SSB Sampler will add zA batches. To avoid this try to decrease batch size or world size.)axiszNumber of calculated indices z4 is not equal to calculated number of local batches r)rmaxrminrrandomuniformr)argsortrr choicedeleter warningr*r+rrandint concatenaterrangesplitr# RuntimeError)r$ max_duration min_durationboundnoisesorted_indicestailexcluder-pad_batches_numbatch_indeces_pad local_indices size_mask local_batchesr%r%r& _make_batchesys\       z"SemiSortBatchSampler._make_batchesccsn|}|jr t}||j|jdtj|j|d}nt d|j}t |D] \}}||Vq+dS)N) generatorr) rNrtorch Generator manual_seedrr!randpermr#arange enumerate)r$rMgindices_indexr%r%r&__iter__s zSemiSortBatchSampler.__iter__cCs|jSr()r#)r$r%r%r&__len__szSemiSortBatchSampler.__len__)TFNr )__name__ __module__ __qualname__intrboolrfloatr'r"rrrNrr[r\r%r%r%r&r s8  JB r modeldatasetconfigrc Cst|tst|tstdt|ddd|jjjD}t|j |j ||d| dd| dd | d d | d d d}|S)ag Instantiates a Semi Sorted (Batch) Sampler. Args: model: ASR Model. dataset: Dataset which allow iterate over all object and parse durations. config: Train, Vaidation or Test dataset config. Raises: ValueError: Wrong dataset type. Returns: SemiSortBatchSampler: Semi Sorted Batch Sampler class. z\Only AudioToCharDataset or AudioToBPEDataset supported with semi sorted batching, but found rcSsg|]}|jqSr%)duration).0sampler%r%r& sz1get_semi_sorted_batch_sampler..rrTrFrNsemi_sort_sampler_seedr )r rrrrrrr) isinstancerrrtypemanifest_processor collectiondatar r rget)rcrdrersamplerr%r%r&get_semi_sorted_batch_samplers$     rr)r*typingrrrrnumpyrrQtorch.utils.data.distributedr'nemo.collections.asr.data.audio_to_textrr%nemo.collections.asr.models.asr_modelr nemo.utilsr r dictrrr%r%r%r&s&   2