o %ݫi@s@dZddlZddlmZddlmZeeZGdddeZdS)zThis lobe enables the integration of huggingface pretrained mBART models. Reference: https://arxiv.org/abs/2001.08210 Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html Authors * Ha Nguyen 2023 N)HFTransformersInterface) get_loggercsVeZdZdZ    dfdd ZdddZedd d Zd d Z ddZ Z S)mBARTadThis lobe enables the integration of HuggingFace and SpeechBrain pretrained mBART models. Source paper mBART: https://arxiv.org/abs/2001.08210 Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html The model is normally used as a text decoder of seq2seq models. It will download automatically the model from HuggingFace or use a local path. Arguments --------- source : str HuggingFace hub name: e.g "facebook/mbart-large-50-many-to-many-mmt" save_path : str Path (dir) of the downloaded model. freeze : bool (default: True) If True, the model is frozen. If False, the model will be trained alongside with the rest of the pipeline. target_lang: str (default: fra_Latn (a.k.a French) The target language code according to NLLB model. decoder_only : bool (default: True) If True, only take the decoder part (and/or the lm_head) of the model. This is useful in case one wants to couple a pre-trained speech encoder (e.g. wav2vec) with a text-based pre-trained decoder (e.g. mBART, NLLB). share_input_output_embed : bool (default: True) If True, use the embedded layer as the lm_head. Example ------- >>> src = torch.rand([10, 1, 1024]) >>> tgt = torch.LongTensor([[250008, 313, 25, 525, 773, 21525, 4004, 2]]) >>> model_hub = "facebook/mbart-large-50-many-to-many-mmt" >>> save_path = "savedir" >>> model = mBART(model_hub, save_path) # doctest: +SKIP >>> outputs = model(src, tgt) # doctest: +SKIP Tfr_XXc stj|||dd||_||_||_|j|d|d|r3|jjjjj |jj _ d|jj _ d|jjjj_ |r9|jj` |j D]\}}d|vsJd|vrNd|_ q>d|_ q>dS)NT)source save_pathfreeze seq2seqlm)r pad_tokentgt_langF encoder_attn layer_norm)super__init__ target_lang decoder_onlyshare_input_output_embedload_tokenizermodeldecoder embed_tokensweightlm_head requires_gradencodernamed_parameters) selfrrrrrrkp __class__k/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/huggingface_transformers/mbart.pyr<s,   zmBART.__init__rcCs||d|jjjjj}|jrKt0t|jjdr&|jjj |dj }|jjj||dj }|j | }|WdS1sFwYt|jjdr[|jjj |dj }|jjj||dj }|j |}|S)aThis method implements a forward step for mt task using a wav2vec encoder (same than above, but without the encoder stack) Arguments --------- src : tensor output features from the w2v2 encoder (transcription) tgt : tensor The sequence to the decoder (translation) (required). pad_idx : int The index for token (default=0). Returns ------- dec_out : torch.Tensor Decoder output. rr) inputs_embeds) input_idsencoder_hidden_statesN) custom_paddingrrconfig pad_token_idrtorchno_gradhasattrrlast_hidden_statedetachr)rsrctgtpad_idxdec_outr!r!r"forwardds8    z mBART.forwardNcCs\|jtjtjfvr |}tj||jd}|jjj|||dd}|j |j |j dfS)aThis method implements a decoding step for the transformer model. Arguments --------- tgt : torch.Tensor The sequence to the decoder. encoder_out : torch.Tensor Hidden output of the encoder. enc_len : torch.LongTensor The actual length of encoder states. Returns ------- output : torch.Tensor Output of transformer. cross_attention : torch.Tensor Attention value. )deviceT)r$r%attention_maskoutput_attentions) dtyper)longint64onessizer3rrrr,cross_attentions)rr/ encoder_outenc_lentgt_maskoutputr!r!r"decodes z mBART.decodecCs|}||||k<|S)aAThis method customizes the padding. Default pad_idx of SpeechBrain is 0. However, it happens that some text-based models like mBART reserves 0 for something else, and are trained with specific pad_idx. This method change org_pad to custom_pad Arguments --------- x : torch.Tensor Input tensor with original pad_idx org_pad : int Original pad_idx custom_pad : int Custom pad_idx Returns ------- out : torch.Tensor Padded outputs. )clone)rxorg_pad custom_padoutr!r!r"r&s zmBART.custom_paddingcCs d|_|S)zIf the config needs to be overridden, here is the place. Arguments --------- config : MBartConfig The original config needs to be overridden. Returns ------- Overridden config g?)decoder_layerdrop)rr'r!r!r"override_configs zmBART.override_config)TrTT)r)N) __name__ __module__ __qualname____doc__rr2r)r*rAr&rH __classcell__r!r!rr"rs* (, %r) rLr)=speechbrain.lobes.models.huggingface_transformers.huggingfacerspeechbrain.utils.loggerrrIloggerrr!r!r!r"s