o %ݫiL-@sdZddlZddlZddlmmZddlZddl m Z ddl m Z m Z ddlmZeeZGddde ZGdd d e ZdS) aThis lobe enables the integration of huggingface pretrained wav2vec2 models. Reference: https://arxiv.org/abs/2006.11477 Reference: https://arxiv.org/abs/1904.05862 Reference: https://arxiv.org/abs/2110.13900 Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html Authors * Titouan Parcollet 2021 * Boumadane Abdelmoumene 2021 * Ha Nguyen 2023 N)_compute_mask_indices)HFTransformersInterfacemake_padding_masks) get_loggercsNeZdZdZ     d fdd ZdgfddZdd d Zdd d ZZS)Wav2Vec2aThis lobe enables the integration of HuggingFace and SpeechBrain pretrained wav2vec2.0/Hubert models. Source paper wav2vec2.0: https://arxiv.org/abs/2006.11477 Source paper Hubert: https://arxiv.org/abs/2106.07447 Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html The model can be used as a fixed feature extractor or can be finetuned. It will download automatically the model from HuggingFace or use a local path. Arguments --------- source : str HuggingFace hub name: e.g "facebook/wav2vec2-large-lv60" save_path : str Path (dir) of the downloaded model. output_norm : bool (default: True) If True, a layer_norm (affine) will be applied to the output obtained from the wav2vec model. freeze : bool (default: True) If True, the model is frozen. If False, the model will be trained alongside with the rest of the pipeline. freeze_feature_extractor : bool (default: False) When freeze = False and freeze_feature_extractor True, the feature_extractor module of the model is Frozen. If False all the wav2vec model will be trained including feature_extractor module. apply_spec_augment : bool (default: False) If True, the model will apply spec augment on the output of feature extractor (inside huggingface Wav2VecModel() class). If False, the model will not apply spec augment. We set this to false to prevent from doing it twice. output_all_hiddens : bool (default: False) If True, the forward function outputs the hidden states from all transformer layers. For example wav2vec2-base has 12 transformer layers and the output is of shape (13, B, T, C), where a projection of the CNN output is added to the beginning. If False, the forward function outputs the hidden states only from the last transformer layer. **kwargs Extra keyword arguments passed to the `from_pretrained` function. Example ------- >>> inputs = torch.rand([10, 600]) >>> model_hub = "facebook/wav2vec2-base-960h" >>> save_path = "savedir" >>> model = Wav2Vec2(model_hub, save_path) >>> outputs = model(inputs) Fc  stjd|||d|||jj_|j||d|jj|_||_ |j s>|j r>t d|jj |jjD]} d| _q8||_||_dS)N)source save_pathfreeze) cache_dirzespeechbrain.lobes.models.huggingface_transformers.wav2vec2 - wav2vec 2.0 feature extractor is frozen.F)super__init__modelconfigapply_spec_augmentload_feature_extractorfeature_extractor do_normalize normalize_wavfreeze_feature_extractorr loggerwarningeval parameters requires_grad output_normoutput_all_hiddens) selfrrrr rrrkwargsparam __class__r n/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/huggingface_transformers/wav2vec2.pyr Ns$      zWav2Vec2.__init__wav2vec2c Cs\i}tj|dd}|D]\}}|D]}|d|vr*|d|dd}|||<qq |S)aFA custom loading ensures SpeechBrain compatibility for Pretrain and model de/serialization. Here, the scope is to remove '.wav2vec2' before loading. Arguments --------- path : str Checkpoint path, file name relative to the repo root. replaceables : List[str] State dict sub-keys that if found, shall be dropped (incl. the 'model.' parent key), elevating key structures. Returns ------- modified_state_dict : see torch.load SpeechBrain-valid deserialized pretrained model. cpu) map_location.zmodel.)torchloaditemsreplace) rpath replaceablesmodified_state_dictorig_state_dictkeyparamstagsave_keyr r r"_modify_state_dictoszWav2Vec2._modify_state_dictNcCsF|jrt|||WdS1swY|||S)aTakes an input waveform and return its corresponding wav2vec encoding. Arguments --------- wav : torch.Tensor (signal) A batch of audio signals to transform to features. wav_lens : torch.Tensor The relative length of the wav given in SpeechBrain format. Returns ------- Wav2vec encoded features. N)r r(no_gradextract_features)rwavwav_lensr r r"forwards    zWav2Vec2.forwardcCst||d}|jrt||jdd}|j|||jd}|jr2tjt |j dd}|jdd}n|j }|j}|j rEt||dd}|S)aTakes an input waveform and return its corresponding wav2vec encoding. Arguments --------- wav : torch.Tensor (signal) A batch of audio signals to transform to features. wav_lens : torch.Tensor The relative length of the wav given in SpeechBrain format. Returns ------- out : torch.Tensor Wav2vec encoded features. wav_lenN)attention_maskoutput_hidden_statesr)dim) rrF layer_normshaperrr(stacklist hidden_stateslast_hidden_stater)rr7r8 padding_maskout norm_shaper r r"r6s zWav2Vec2.extract_features)FFFFFN) __name__ __module__ __qualname____doc__r r4r9r6 __classcell__r r r r"rs3! rcs:eZdZdZ   d fdd Zd dd Zd d ZZS)Wav2Vec2PretrainaThis lobe enables the integration of HuggingFace wav2vec2.0 models to be pretrained. Source paper: https://arxiv.org/abs/2006.11477 Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html The return is an HuggingFace format and the mask indices that contains: https://huggingface.co/transformers/model_doc/wav2vec2.html#wav2vec2forpretraining For instance, it returns the loss that can be accessed with .loss Arguments --------- source : str HuggingFace hub name: e.g "facebook/wav2vec2-large-lv60" save_path : str Path (dir) of the downloaded model. mask_prob : float (default: 0.65) Probability of masking a given frame. Default is taken from the paper. mask_length : float (default: 10) Length (i.e. number of consecutive masked frames). Default is taken from the paper. normalize_wav : bool Whether to normalize input before processing. Example ------- >>> inputs = torch.rand([10, 32000]) >>> model_hub = "facebook/wav2vec2-base-960h" >>> save_path = "savedir" >>> model = Wav2Vec2Pretrain(model_hub, save_path) >>> outputs, _ = model(inputs, wav_lens=None) ? Tcs(tj||dd||_||_||_dS)NT)rrfor_pretraining)r r mask_prob mask_lengthr)rrrrUrVrr r r"r s  zWav2Vec2Pretrain.__init__Nc Cs|j\}}|jrt||j}|j|}t||f|j|j d}t j ||j t j d}t||d}t||f} t j tjjjj||f|jj| d|j t j d} |j||| |d|fS)aTakes an input waveform and return its corresponding wav2vec encoding. Arguments --------- wav : torch.Tensor (signal) A batch of audio signals to transform to features. wav_lens : torch.Tensor The relative length of the wav given in SpeechBrain format. Returns ------- Wav2vec encoded outputs. )rUrV)devicedtyper:) num_negativesmask_time_indices)rZsampled_negative_indicesr=)rCrrArBr _get_feat_extract_output_lengthsitemrrUrVr(tensorrWlongrnpones transformersmodelsr#modeling_wav2vec2_sample_negative_indicesrrY) rr7r8 batch_sizeraw_sequence_lengthsequence_lengthrZtorch_mask_time_indicesrHfull_sentence_indicesnegative_sample_indicesr r r"r9sJ    zWav2Vec2Pretrain.forwardcCs d|_|S)zIf the config needs to be overridden, here is the place Arguments --------- config : Wav2Vec2Config The original config needs to be overridden. Returns ------- Overridden config T)r>)rrr r r"override_config?s z Wav2Vec2Pretrain.override_config)rRrSTrK)rLrMrNrOr r9rlrPr r r r"rQs' ?rQ)rOnumpyr`r(torch.nn.functionalnn functionalrArb.transformers.models.wav2vec2.modeling_wav2vec2r=speechbrain.lobes.models.huggingface_transformers.huggingfacerrspeechbrain.utils.loggerrrLrrrQr r r r"s  -