o ߥi@sddlZddlZddlZddlmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZddlmZddlmZd d lmZmZd d lmZejeje jd Gd dde ZGddde jZ dS)N)DictOptionalTuple)Models) TorchModel)Tensor)MODELS) update_conf)Tasks) GlobalCMVNload_kaldi_cmvn)FSMN) module_namecspeZdZdZ     ddededed ed ed eef fd d Z ddZ de ee ffddZ ddZZS) FSMNDecoratorz? A decorator of FSMN for integrating into modelscope framework N' F model_dir cmvn_filebackbone input_dim output_dimtrainingc sZtj|g|Ri|d|_d|_|r ||||||_dS|tj|dd|_dS)ainitialize the fsmn model from the `model_dir` path. Args: model_dir (str): the model path. cmvn_file (str): cmvn file backbone (dict): params related to backbone input_dim (int): input dimension of network output_dim (int): output dimension of network training (bool): training or inference mode Nz config.yaml)model_workspace config_path)super__init__model model_cfg init_modelospathjoin) selfrrrrrrargskwargs __class___/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/audio/kws/nearfield/model.pyrs    zFSMNDecorator.__init__cCst|dr |jdSdS)Ntmp_dir)hasattrr*cleanupr#r(r(r)__del__<s zFSMNDecorator.__del__returncCs$|jdur|dur|j|S|jS)zP Args: input (torch.Tensor): Input tensor (B, T, D) N)rforwardr)r#inputr(r(r)r0@s zFSMNDecorator.forwardc Cs|durt|\}}tt|t|}nd}d}d} |d} |d} |d} |d} |d}|d}|d}|d }|d }t|| | | | |||||| }d}d}t||||| |||}|S) Ninput_affine_dim num_layers linear_dimproj_dim left_order right_order left_stride right_strideoutput_affine_dim)r r torch from_numpyfloatrKWSModel)r#rrrrmeanistd global_cmvn hidden_dim preprocessingr3r4r5r6r7r8r9r:r; classifier activation kws_modelr(r(r)rJs8     zFSMNDecorator.init_model)NNrrF)__name__ __module__ __qualname____doc__strdictintrboolrr.rrr0r __classcell__r(r(r&r)rs.! rcseZdZdZdedededeejdeejdejdejd ejffd d Zd d Z ddZ e j ddde j dfde jde jdee je jffddZddZZS)r?aOur model consists of four parts: 1. global_cmvn: Optional, (idim, idim) 2. preprocessing: feature dimension projection, (idim, hdim) 3. backbone: backbone or feature extractor of the whole network, (hdim, hdim) 4. classifier: output layer or classifier of KWS model, (hdim, odim) 5. activation: nn.Sigmoid for wakeup word nn.Identity for speech command dataset idimodimhdimrBrDrrErFc s>t||_||_||_||_||_||_||_||_ dS)ab Args: idim (int): input dimension of network odim (int): output dimension of network hdim (int): hidden dimension of network global_cmvn (nn.Module): cmvn for input feature, (idim, idim) preprocessing (nn.Module): feature dimension projection, (idim, hdim) backbone (nn.Module): backbone or feature extractor of the whole network, (hdim, hdim) classifier (nn.Module): output layer or classifier of KWS model, (hdim, odim) activation (nn.Module): nn.Identity for training, nn.Sigmoid for inference N) rrrQrRrSrBrDrrErF) r#rQrRrSrBrDrrErFr&r(r)rws  zKWSModel.__init__cCs |jSN)r to_kaldi_netr-r(r(r)rUs zKWSModel.to_kaldi_netcCs |j|SrT)rto_pytorch_net)r# kaldi_filer(r(r)rVs zKWSModel.to_pytorch_netr)dtypexin_cacher/cCsh|jdur ||}|jdur||}|||\}}|jdur&||}|jdur0||}||fSrT)rBrDrrErF)r#rYrZ out_cacher(r(r)r0s        zKWSModel.forwardcCs"|jdur |j|jdSrT)rD fuse_modulesrr-r(r(r)r\s  zKWSModel.fuse_modules)rHrIrJrKrNrnnModulerrUrVr<zerosr>rrr0r\rPr(r(r&r)r?ls<    r?)!r systempfiletypingrrrr<torch.nnr]modelscope.metainformodelscope.modelsrmodelscope.models.basermodelscope.models.builderr"modelscope.utils.audio.audio_utilsr modelscope.utils.constantr cmvnr r fsmnrregister_modulekeyword_spotting"speech_kws_fsmn_char_ctc_nearfieldrr^r?r(r(r(r)s(        T