o i)@sdZddlZddlZddlZddlmZddlZddlmZm Z ddl m Z ddl m Z ddlmZddlmZdd lmZd d Zd d ZddZddZddZddZdddZddZe fddZdS)zFinetuning methods.N) OrderedDict)get_model_conf torch_load) ASRInterface) MTInterface)custom_torch_load) TTSInterface)dynamic_importcsZ|D]\}tfdd|Drtddd|_qtdd|}||fS)a$Freeze model parameters according to modules list. Args: model (torch.nn.Module): Main model. modules (List): Specified module(s) to freeze. Return: model (torch.nn.Module) : Updated main model. model_params (filter): Filtered model parameters. c3|]}|VqdSN startswith.0mmodW/home/ubuntu/.local/lib/python3.10/site-packages/espnet/asr/pytorch_backend/asr_init.py z!freeze_modules..z Freezing z). It will not be updated during training.FcSs|jSr ) requires_gradxrrr#sz freeze_modules..)named_parametersanyloggingwarningrfilter parameters)modelmodulesparam model_paramsrrrfreeze_moduless r%csg}g}|D]\}tfdd|Dr||jfg7}qt|ddd}|D]\}tfdd|DrC||jfg7}q,t|ddd}||k}|swtdtd t|t|td t|t|td |S) aRVerify tuples (key, shape) for input model modules match specified modules. Args: model_state_dict (Dict) : Main model state dict. partial_state_dict (Dict): Pre-trained model state dict. modules (List): Specified module(s) to transfer. Return: (bool): Whether transfer learning is allowed. c3r r r r)key_mrrr8rz(transfer_verification..cS|d|dfSNrrrrrrr:z'transfer_verification..keyc3r r r r)key_prrr=rcSr'r(rrrrrr?r*zYSome specified modules from the pre-trained model don't match with the new model modules:z Pre-trained: z New model: r))itemsrshapesortedrerrorsetexit)model_state_dictpartial_state_dictr" model_modulespartial_modulesvalue_mvalue_p module_matchr)r&r-rtransfer_verification(s* r;cs:t}|D]\}tfdd|Dr||<q|S)a&Create state dict with specified modules matching input model modules. Args: model_state_dict (Dict): Pre-trained model state dict. modules (Dict): Specified module(s) to transfer. Return: new_state_dict (Dict): State dict with specified modules weights. c3r r r rr+rrr]rz)get_partial_state_dict..)rr.r)r4r"new_state_dictvaluerr+rget_partial_state_dictOs  r>cCsjt}t|D])\}}|dkr||d<q |dr2|d}d|dd|dd}|||<q |S) zCreate compatible ASR decoder state dict from LM state dict. Args: lm_state_dict (Dict): Pre-trained LM state dict. Return: new_state_dict (Dict): State dict with compatible key names. zpredictor.embed.weightzdec.embed.weightzpredictor.rnn..z dec.decoder._l0)rlistr.r split) lm_state_dictr<r,r=_splitnew_keyrrrget_lm_state_dictcs    rHcsg}g}t|}|D]tfdd|Dr|g7}q |g7}q |r?td|dtdt|td|S)zFilter non-matched modules in model state dict. Args: model_state_dict (Dict): Pre-trained model state dict. modules (List): Specified module(s) to transfer. Return: new_mods (List): Filtered module list. c3s|]}|VqdSr r )rr,rrrrrz!filter_modules..z`Specified module(s) don't match or (partially match) available modules in model. You specified: r?z"The existing modules in model are:r))rCkeysrrr1r3)r4r"new_modsincorrect_mods mods_modelrrrfilter_modules{s"      rMc s|ds |ds |St}d|vrdnd}t|D]J\}tfdddDrcd vrF|td d }td d d ||d | dkrc|ddd|f|dd|df}||<q|S)a'Create a compatible transducer model state dict for transfer learning. If RNN encoder modules from a non-Transducer model are found in the pre-trained model state dict, the corresponding modules keys are renamed for compatibility. Args: model_state_dict (Dict): Pre-trained model state dict encoder_type (str): Type of pre-trained encoder. encoder_units (int): Number of encoder units in pre-trained model. Returns: new_state_dict (Dict): Transducer compatible pre-trained model state dict. p)lstmgrubbirnnrnnc3s|]}|vVqdSr r)rkr+rrrsz:create_transducer_compatible_state_dict..)l_lastnbrnnrVz _l([0-9]+)r)rBr@N) endswithrrCr.rresearchgroupsubreplacesize)r4 encoder_type encoder_unitsr< rnn_key_namer= layer_namerr+r'create_transducer_compatible_state_dicts" ( rcTcCst|tjtj|d\}}}td|t|dr"|j}nd}t|dr,d|_ t |}d|vrG|||||d}t |||d||fS||||}t ||||fS) a+Load the trained model for recognition. Args: model_path (str): Path to model.***.best training (bool): Training mode specification for transducer model. Returns: model (torch.nn.Module): Trained model. train_args (Namespace): Trained model arguments. model.jsonReading model parameters from model_module'espnet.nets.pytorch_backend.e2e_asr:E2Ectc_typebuiltin transducer)training) rospathjoindirnamerinfohasattrrfrhr rr) model_pathrkidimodim train_argsrf model_classr!rrrload_trained_models"     rwc Cstd|tjtj|d}d|vrtt|St ||\}}}t |dr/|j }nd}t |}||||}t ||t|tsQt|tsQt|tsQJ|rbd|j vrbt||j|jS|S)aExtract the trained model state dict for pre-initialization. Args: model_path (str): Path to trained model. new_is_transducer (bool): Whether the new model is Transducer-based. Return: (Dict): Trained model state dict. rerdrnnlmrfrgrj)rrprlrmrnrorHtorchloadrrqrfr r isinstancerrrrc state_dictetypeeunits) rrnew_is_transducer conf_pathrsrtargsrfrvr!rrrget_trained_model_state_dicts0    rcCsdd}|j}|j}|j}|j}t|j} | |||} t| |s"J| } t d||f||ffD]@\} } | durst j | rgt | d|jv}t|| } t|| }|rft| || rf||| | | |q3td| tdq3| | | S)amLoad ASR/MT/TTS model with pre-trained weights for specified modules. Args: idim (int): Input dimension. odim (int): Output dimension. args Namespace: Model arguments. interface (ASRInterface|MTInterface|TTSInterface): Model interface. Return: main_model (torch.nn.Module): Model with pre-initialized weights. cSs8td|d||D] }td|qdS)NzLoading z from model: zOverriding module )rrprIr)r|r"rrrTrrrprint_new_keys)s z,load_trained_modules..print_new_keysz&Model(s) found for pre-initialization.NrjzSpecified model was not found: r))enc_initdec_init enc_init_mods dec_init_modsr rfr{r|rrrlrmisfilerrMr>r;updater1r3load_state_dict)rsrtr interfacerenc_model_pathdec_model_path enc_modules dec_modulesrv main_modelmain_state_dictrrr"r4r5rrrload_trained_moduless@           r)T)__doc__rrlrY collectionsrryespnet.asr.asr_utilsrrespnet.nets.asr_interfacerespnet.nets.mt_interfacer,espnet.nets.pytorch_backend.transducer.utilsrespnet.nets.tts_interfacerespnet.utils.dynamic_importr r%r;r>rHrMrcrwrrrrrrs*      '! +'-