o %ݫi_/@sdZddlZddlmmZddlmZddlmZddlm Z ddl m Z zddl Z Wne y>dZed7Ze ewe eZGd d d ejZGd d d ejZdS) a.This lobe enables the integration of fairseq pretrained wav2vec models. Reference: https://arxiv.org/abs/2006.11477 Reference: https://arxiv.org/abs/1904.05862 FairSeq >= 1.0.0 needs to be installed: https://fairseq.readthedocs.io/en/latest/ Authors * Titouan Parcollet 2021 * Salima Mdhaffar 2021 N)nn)length_to_mask) download_file) get_loggerz1Please install Fairseq to use pretrained wav2vec zE.G. run: pip install fairseqcs\eZdZdZ       dfdd ZddZdd d Zd d Zd dZdddZ Z S)FairseqWav2Vec2a This lobe enables the integration of fairseq pretrained wav2vec2.0 models. Source paper: https://arxiv.org/abs/2006.11477 FairSeq >= 0.10.0 needs to be installed: https://fairseq.readthedocs.io/en/latest/ The model can be used as a fixed features extractor or can be finetuned. It will download automatically the model if a url is given (e.g FairSeq repository from GitHub). Arguments --------- pretrained_path : str Path of the pretrained wav2vec2 model. It can be a url or a local path. save_path : str Path and filename of the downloaded model. input_norm : bool (default: None) If True, a layer_norm (affine) will be applied to the input waveform. By default, it is extracted from the checkpoint of the downloaded model in order to match the pretraining conditions. However, if this information is not given in the checkpoint, it has to be given manually. output_norm : bool (default: False) If True, a layer_norm (affine) will be applied to the output obtained from the wav2vec model. freeze : bool (default: False) If True, the model is frozen. If False, the model will be trained alongside with the rest of the pipeline. freeze_feature_extractor : bool (default: False) Whether to prevent feature extraction weights from updating. pretrain : bool (default: True) If True, the model is pretrained with the specified source. If False, the randomly-initialized model is instantiated. dropout : float (default: None) If different from None (0.0 to 1.0), it will override the given fairseq dropout rates. This is useful if the wav2vec2 model has been trained without dropout and one wants to reactivate it for downstream task fine-tuning (better performance observed). layer_drop : float (default: None) If different from None (0.0 to 1.0), it will override the given fairseq layer_drop rate. This is useful if the wav2vec2 model has been trained without layer_drop and one wants to reactivate it for downstream task fine-tuning. Example ------- >>> inputs = torch.rand([10, 600]) >>> model_url = "https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt" >>> save_path = "models_checkpoints/wav2vec2.pt" >>> model = FairseqWav2Vec2(model_url, save_path) >>> outputs = model(inputs) >>> outputs.shape torch.Size([10, 100, 768]) NFTc svtt||i} |s6|dur6i| d<|dur,|| dd<|| dd<|| dd<| dur6| | dd<tjj|g| d\} } } |durbt| ddrT| dj|_nt| dr^| j|_nd |_n||_| d } | |_||_ ||_ ||_ |j rt d |j|jD]}d |_qn|j|j rt d |jj|jjD]}d |_q|s||j|dS) Nmodeldropout dropout_inputattention_dropout layer_drop) arg_overridestask normalizeFrzAspeechbrain.lobes.models.fairseq_wav2vec - wav2vec 2.0 is frozen.zSspeechbrain.lobes.models.fairseq_wav2vec - wav2vec 2.0 feature extractor is frozen.)super__init__rfairseqcheckpoint_utilsload_model_ensemble_and_taskhasattrrrfreeze output_normfreeze_feature_extractorloggerwarningeval parameters requires_gradtrainfeature_extractor reset_layerremove_pretraining_modules)selfpretrained_path save_path input_normrrrpretrainrr overridesrcfgr param __class__\/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/fairseq_wav2vec.pyrVsb              zFairseqWav2Vec2.__init__cCsT|j||d}|jr$t|||WdS1swY|||S)ajTakes an input waveform and return its corresponding wav2vec encoding. Arguments --------- wav : torch.Tensor A batch of audio signals to transform to features. wav_lens : torch.Tensor The lengths corresponding to the input wavs. Returns ------- wav2vec encoded features. )wav_lenN) make_masksrtorchno_gradextract_features)r!wavwav_lens padding_maskr+r+r,forwards    zFairseqWav2Vec2.forwardcCsR|jrt||jdd}|jj||ddd}|jr't||jdd}|S) Extracts the wav2vect embeddingsNF)r4maskx)rF layer_normshaperr1r)r!r2r4outr+r+r,r1sz FairseqWav2Vec2.extract_featurescC6t|dr ||D] }||kr||q dSz+Reinitializes the parameters of the networkreset_parametersNrr@childrenrr!r child_layerr+r+r,r   zFairseqWav2Vec2.reset_layercCs$d|j_d|j_d|j_d|j_dS)z?Remove unneeded modules. Inspired by the same fairseq function.N)r quantizer project_q target_glu final_proj)r!r+r+r,r s z*FairseqWav2Vec2.remove_pretraining_modulesrcCs2d}|durt||jd}t|}|S)aThis method generates the padding masks. Arguments --------- src : tensor The sequence to the encoder (required). wav_len : tensor The relative length of the wav given in SpeechBrain format. pad_idx : int The index for token (default=0). Returns ------- src_key_padding_mask : torch.Tensor The mask for removing pad tokens. Nr7)r/roundr<rbool)r!srcr-pad_idxsrc_key_padding_maskabs_lenr+r+r,r.s zFairseqWav2Vec2.make_masks)NFFFTNN)NNr) __name__ __module__ __qualname____doc__rr5r1rr r. __classcell__r+r+r)r,rs:T rcs@eZdZdZ   d fdd ZddZddZd d ZZS) FairseqWav2Vec1aThis lobes enables the integration of fairseq pretrained wav2vec1.0 models. Arguments --------- pretrained_path : str Path of the pretrained wav2vec1 model. It can be a url or a local path. save_path : str Path and filename of the downloaded model. output_norm : bool (default: True) If True, a layer_norm (affine) will be applied to the output obtained from the wav2vec model. freeze : bool (default: True) If True, the model is frozen. If False, the model will be trained alongside with the rest of the pipeline. pretrain : bool (default: True) If True, the model is pretrained with the specified source. If False, the randomly-initialized model is instantiated. Example ------- >>> inputs = torch.rand([10, 600]) >>> model_url = "" >>> save_path = "models_checkpoints/wav2vec.pt" >>> model = FairseqWav2Vec1(model_url, save_path) >>> outputs = model(inputs) >>> outputs.shape torch.Size([10, 100, 512]) Tc snt||_||_t||tj|g\}}}||_|jd|_|jr+|j |s5| |jdSdSrP) rrrrrrrrrrr) r!r"r#rrr%rr'r r)r+r,rs$    zFairseqWav2Vec1.__init__cCsF|jrt||WdS1swY||S)aTakes an input waveform and return its corresponding wav2vec encoding. Arguments --------- wav : torch.Tensor A batch of audio signals to transform to features. Returns ------- wav2vec encoded features N)rr/r0r1detach)r!r2r+r+r,r5:s    zFairseqWav2Vec1.forwardcCsB|j|}|j|d}|dd}|jrt||j}|S)r6rr7) rrfeature_aggregatorsqueeze transposerr:r;r<)r!r2r=r+r+r,r1Ns  z FairseqWav2Vec1.extract_featurescCr>r?rArCr+r+r,r[rEzFairseqWav2Vec1.reset_layer)TTT) rQrRrSrTrr5r1rrUr+r+r)r,rVs!  rV)rTr/torch.nn.functionalr functionalr:speechbrain.dataio.dataiorspeechbrain.utils.data_utilsrspeechbrain.utils.loggerrr ImportErrorMSGrQrModulerrVr+r+r+r,s$       ^