o }oih4@sxddlmZddlZddlmmZddlmZddlmZm Z ddl m Z m Z m Z mZmZdgZGdddeZdS))ceilN)nn)Loss typecheck)AcousticEncodedRepresentation LengthsTypeLossType NeuralTypeSpectrogramTypeContrastiveLossc+seZdZeddZeddZeddZ          d4dedededededede de de dededed ed!ed"e d#e d$e d%e d&ed'ed(e f*fd)d* Z d+d,Z e d5d.d/Zd0d1Zd2d3ZZS)6r cCs6tdttdttdtttdtdddS)z(Input types definitions for Contrastive.)BDT)r rr r T)optional) spectrograms spec_masksdecoder_outputsdecoder_lengths)r r rtuplerselfrf/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/losses/ssl_losses/contrastive.py input_typess   zContrastiveLoss.input_typescCsdttdiS)z]Output types definitions for Contrastive. loss: NeuralType(None) loss) elements_type)r rrrrr output_types&szContrastiveLoss.output_typescCsdS)NFrrrrr needs_labels.szContrastiveLoss.needs_labelsdF@皙?sumT?;?皙?0@in_dimproj_dimcombine_time_steps num_negativesquantized_targets codebook_sizeprob_ppl_weight logit_tempreducesample_from_same_utterance_onlysample_from_non_maskedsample_from_codebook group_loss num_groupsquantizer_temp_startquantizer_temp_minquantizer_temp_decaymask_threshold store_ids reduce_ids multiplierc st|||f}||_||_||_|jr)d||||||ddd}t||_||_||_| |_ ||_ | |_ | |_ | |_ | |_||_||_||_||_|js[t||||_dSdS)aQ Loss function representing the contrastive task of identifying the true latent speech representation of the masked spectrogram steps from a set of sampled distractors. Args: in_dim: Number of spectrogram channels. proj_dim: Number of channels in the model outputs. combine_time_steps: How many time steps should be combined into a single representation. num_negatives: Number of sampled negatives for each target. quantized_targets: Bool that determines if the targets should be quantized. codebook_size: Number of vectors in the codebook per group. prob_ppl_weight: Float multiplier on the perplexity loss for target quantization. logit_temp: Float temperature for normalizing logits. reduce: String representing the type of reduction used for cross entropy. sample_from_same_utterance_only: Bool that determines if negatives should be sampled only from same utterance. sample_from_non_masked: Bool that determines if negatives should be sampled from non-masked steps of the spectrogram. sample_from_codebook: Bool that determines if negatives should be sampled from entire codebook. group_loss: Bool that determines if loss should be computed separately for each group in the quantizer codebook. num_groups: Number of groups in the quantizer codebook. quantizer_temp_start: Starting temperature in quantizer. quantizer_temp_min: Minimum temperature in quantizer. quantizer_temp_decay: Decay rate of quantizer temperature per global step. mask_threshold: Float threshold for determining if a time step of the spectrogram is masked based on percent of masked channels. store_ids: Bool that determines if the quantizer ids will be stored to be potentially used by other losses. reduce_ids: Bool that determines if we convert any sequence of consecutive equivalent ids to a single occurence of that id. multiplier: Float multipler on final loss zJnemo.collections.asr.parts.submodules.ssl_quantizers.GumbelVectorQuantizerT)_target_dimvq_dimnum_varsgroupstempcombine_groups time_firstN)super__init__r-r,r/r from_config_dict quantizerr0r1r+r2r3r4r5r:r=r;r<rLinear target_proj)rr)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=quantizer_temp quantizer_cfg __class__rrrG2s> 4  zContrastiveLoss.__init__cCsh|jd}ttj||f|jd|j}||d}|||jf|jdd}|dd}||fS)Nrdevicer)shapetorch multinomialonesrQr,view transpose)rynumhighneg_idxsnegsrrrsample_negativess  z ContrastiveLoss.sample_negativesNcCs4|dd}|dd}tt|jd|jd|jd|jd}|dkrs