o %ݫi&?@sdZddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z mZmZddlmZddlmZddlmZeeZGd d d ejZd d d ZdS)ueThis lobe is the interface for huggingface transformers models It enables loading config and model via AutoConfig & AutoModel. Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html Authors * Titouan Parcollet 2021, 2022, 2023 * Mirco Ravanelli 2021 * Boumadane Abdelmoumene 2021 * Ju-Chieh Chou 2021 * Artem Ploujnikov 2021, 2022 * Abdel Heba 2021 * Aku Rouhe 2022 * Arseniy Gorin 2022 * Ali Safaya 2022 * Benoit Wang 2022 * Adel Moumen 2022, 2023 * Andreas Nautsch 2022, 2023 * Luca Della Libera 2022 * Heitor Guimarães 2022 * Ha Nguyen 2023 N) model_info)nn) AutoConfigAutoFeatureExtractor AutoModelAutoModelForCausalLMAutoModelForPreTrainingAutoModelForSeq2SeqLMAutoModelWithLMHead AutoTokenizer)length_to_mask)fetch) get_loggercseZdZdZ         d"fdd Z d#dd Zd d Zd d ZddZddZ ddZ ddZ ddZ ddZ ddZddZddZd d!ZZS)$HFTransformersInterfaceaThis lobe provides an interface for integrating any HuggingFace transformer model within SpeechBrain. We use AutoClasses for loading any model from the hub and its necessary components. For example, we build Wav2Vec2 class which inherits HFTransformersInterface for working with HuggingFace's wav2vec models. While Wav2Vec2 can enjoy some already built features like modeling loading, pretrained weights loading, all weights freezing, feature_extractor loading, etc. Users are expected to override the essential forward() function to fit their specific needs. Depending on the HuggingFace transformer model in question, one can also modify the state_dict by overwriting the _modify_state_dict() method, or adapting their config by modifying override_config() method, etc. See: https://huggingface.co/docs/transformers/model_doc/auto https://huggingface.co/docs/transformers/autoclass_tutorial Arguments --------- source : str HuggingFace hub name: e.g "facebook/wav2vec2-large-lv60" save_path : str save directory of the downloaded model. for_pretraining: bool (default: False) If True, build the model for pretraining with_lm_head : bool (default: False) If True, build the model with lm_head with_casual_lm : bool (default: False) If True, build casual lm model seq2seqlm : bool (default: False) If True, build a sequence-to-sequence model with lm_head quantization_config : dict (default: None) Quantization config, extremely useful for deadling with LLM freeze : bool (default: True) If True, the model is frozen. If False, the model will be trained alongside with the rest of the pipeline. cache_dir : str or Path (default: None) Location of HuggingFace cache for storing pre-trained models, to which symlinks are created. device : any, optional Device to migrate the model to. **kwargs Extra keyword arguments passed to the `from_pretrained` function. Example ------- >>> model_hub = "facebook/wav2vec2-base-960h" >>> save_path = "tmp" >>> model = HFTransformersInterface(model_hub, save_path=save_path) FNpretrained_modelsc  st| dd} tj||d| d\|_} ||j|_||_||_|jr+t |_ n|r1t |_ n|r7t |_ n |r=t |_ nt|_ |j|f|| | d| ||_|jrhtdt|jjd||jdS|j|jdS)Ntrust_remote_codeFT) cache_dirreturn_unused_kwargsr) save_pathrdevicez@speechbrain.lobes.models.huggingface_transformers.huggingface - z is frozen.)super__init__getrfrom_pretrainedconfigoverride_configquantization_configfor_pretrainingr auto_classr rr r_from_pretrainedfreezeloggerwarningtypemodel__name__ freeze_modelgradient_checkpointing_disabletrain)selfsourcerr with_lm_headwith_casual_lm seq2seqlmrr!rrkwargsr_unused_kwargs __class__q/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/huggingface_transformers/huggingface.pyr`sJ     z HFTransformersInterface.__init__c Ks|||\}}}|s|jr|j|j|_|r*|jt|||d} || n|js>|jj |f|j||j d||_|durJ|j |dSdS)aThis function manages the source checking and loading of the params. # 1. Is the model from HF or a local path # 2. Is the model pretrained with HF or SpeechBrain # 3. Download (if appropriate) and load with respect to 1. and 2. Arguments --------- source : str HuggingFace hub name: e.g "facebook/wav2vec2-large-lv60" save_path : str Path (dir) of the downloaded model. cache_dir : str Path (dir) in which a downloaded pretrained model configuration should be cached. device : any, optional Device to migrate the model to. **kwargs Extra keyword arguments passed to `from_pretrained` function. )filenamer+savedir)rrrN) _check_model_sourcerr from_configrr%r(r _load_sb_pretrained_parametersrrto) r*r+rrrr/is_sb ckpt_fileis_localckpt_full_pathr3r3r4r s0   z(HFTransformersInterface._from_pretrainedc Csld}t|}d}|sd}t|d|ddd}|rE|tt|d}td d tt|DrBd}t|}n|}n|}|rztd d t|Dr\d}|||fSt|D]} | d rxtj || }d}|||fSqan3t |j } | D]} | j d r| j }d}|||fSq| D]} | j d r| j }d}|||fSq|d} t| )aChecks if the pretrained model has been trained with SpeechBrain and is hosted locally or on a HuggingFace hub. Called as static function in HFTransformersInterface._from_pretrained. Arguments --------- path : str Used as "source"; local path or HuggingFace hub name: e.g "facebook/wav2vec2-large-lv60" save_path : str norm_output (dir) of the downloaded model. Returns ------- is_sb : bool Whether/not the model is deserializable w/ SpeechBrain or not (then, model conversion is needed). checkpoint_filename : str as of HuggingFace documentation: file name relative to the repo root (guaranteed to be here). is_local : bool Whether/not the model is hosted locally or on a HuggingFace hub. Raises ------ ValueError If file is not found rTFz /models--/z--z /snapshotsrcs|]}|dVqdS)).bin .safetensors.ckptNendswith.0Filer3r3r4   z>HFTransformersInterface._check_model_source..csr@)rArBNrDrFr3r3r4rIrJrCrKz< does not contain a .bin, .safetensors or .ckpt checkpoint !)pathlibPathexistsreplaceoslistdirstranyrEpathjoinrsiblings rfilenameFileNotFoundError) r*rTrcheckpoint_filenamer+r=sink local_pathr;rHfileserr_msgr3r3r4r7sd        z+HFTransformersInterface._check_model_sourcecKsdS)aA custom loading ensures SpeechBrain compatibility for pretrain and model. For example, wav2vec2 model pretrained with SB (Wav2Vec2Pretrain) has slightly different keys from Wav2Vec2. This method handle the compatibility between the two. Users should modify this function according to their own tasks. Arguments --------- path : str Checkpoint path, file name relative to the repo root. **kwargs : dict Args to forward Nr3)r*rTr/r3r3r4_modify_state_dict0sz*HFTransformersInterface._modify_state_dictcCs||}|durtj|dd}|jj|dd}|jD]}td|jd|dd |q|jD]}td |d d t |jj d q5dS)aLoads the parameter of a HuggingFace model pretrained with SpeechBrain and the HuggingFace Pretrain Object. It is necessary to perform a custom loading because HuggingFace adds a level to the checkpoint when storing the model breaking the compatibility Pretrain and model de/serialization. For example, a typical Wav2Vec2 checkpoint for a given parameter would be: model.conv.weight.data while for Wav2Vec2Pretrain it is: model.wav2vec2.weight.data (wav2vec2 must be removed before loading). Arguments --------- path : pathlib.Path The full path to the checkpoint. Ncpu) map_locationF)strictzDuring parameter transfer to z loading from z*, the transferred parameters did not have zparameters for the key: zThe param with the key: z is discarded as it zis useless for finetuning this z model.) r^torchloadr%load_state_dict missing_keysr"r#unexpected_keysr$r&)r*rTmodified_state_dictincompatible_keys missing_keyunexpected_keyr3r3r4r9As*     z6HFTransformersInterface._load_sb_pretrained_parameterscKtz?Users should modify this function according to their own tasks.NotImplementedErrorr*r/r3r3r4forwarddzHFTransformersInterface.forwardcKrkrlrmror3r3r4forward_encoderhrqz'HFTransformersInterface.forward_encodercKrkrlrmror3r3r4forward_decoderlrqz'HFTransformersInterface.forward_decodercKrk)zMight be useful for models like mbart, which can exploit SB's beamsearch for inference Users should modify this function according to their own tasks.rmror3r3r4decodepzHFTransformersInterface.decodecKrk)zeCustom encoding for inference Users should modify this function according to their own tasks.rmror3r3r4encodeuruzHFTransformersInterface.encodecCs ||D]}d|_qdS)a Freezes parameters of a model. This should be overridden too, depending on users' needs, for example, adapters use. Arguments --------- model : from AutoModel.from_config Valid HuggingFace transformers model object. FN)eval parameters requires_grad)r*r%paramr3r3r4r'zs z$HFTransformersInterface.freeze_modelcCs|S)a$Users should modify this function according to their own tasks. Arguments --------- config : HuggingFace config object The original config. Returns ------- config : HuggingFace config object Overridden config. r3)r*rr3r3r4rs z'HFTransformersInterface.override_configcKstj|fd|i||_dS)aLoad model's feature_extractor from the hub. Arguments --------- source : str HuggingFace hub name: e.g "facebook/wav2vec2-large-lv60" cache_dir : str Path (dir) in which a downloaded pretrained model configuration should be cached. **kwarg Keyword arguments to pass to the AutoFeatureExtractor.from_pretrained() method. rN)rrfeature_extractor)r*r+rkwargr3r3r4load_feature_extractors  z.HFTransformersInterface.load_feature_extractorcKstj|fi||_dS)aLoad model's tokenizer from the hub. Arguments --------- source : str HuggingFace hub name: e.g "facebook/wav2vec2-large-lv60" **kwarg Keyword arguments to pass to the AutoFeatureExtractor.from_pretrained() method. N)r r tokenizer)r*r+r|r3r3r4load_tokenizers z&HFTransformersInterface.load_tokenizer) rFFFFNFrN)N)r& __module__ __qualname____doc__rr r7r^r9rprrrsrtrvr'rr}r __classcell__r3r3r1r4r1s41F 6Y#rcCs0d}|durt||jd}t|}|S)azThis method generates the padding masks. Arguments --------- src : tensor The sequence to the encoder (required). wav_len : tensor The relative length of the wav given in SpeechBrain format. pad_idx : int The index for token (default=0). Returns ------- src_key_padding_mask : tensor The padding mask. N)rbroundshaper bool)srcwav_lenpad_idxsrc_key_padding_maskabs_lenr3r3r4make_padding_maskss  r)Nr)rrPrLrbhuggingface_hubrr transformersrrrrrr r r speechbrain.dataio.dataior speechbrain.utils.fetchingr speechbrain.utils.loggerrr&r"Modulerrr3r3r3r4s   (