o i@sddlZddlZddlZddlZddlZddlZddlZddlZddl Z Gddde Z zddl m Z Wn ey?dZYn wGddde jZzddl m Z Wn ey\dZYn wGddde jZd7d d Zd7d d Zd dZddZddZddZe jdfddZddZd8ddZd7dd Zd!d"Zd#d$Zd%d&Zd'd(Z d)d*Z!d+d,Z"d-d.Z# /   0 0 0 1 0 0 0 1 2d9d3d4Z$d5d6Z%dS):Nc@s*eZdZdZd ddZddZddZd S) CompareValueTriggeraTrigger invoked when key value getting bigger or lower than before. Args: key (str) : Key of value. compare_fn ((float, float) -> bool) : Function to compare the values. trigger (tuple(int, str)) : Trigger that decide the comparison interval. epochcCs8ddlm}||_d|_|j||_|||_dS)Nrtraining) chainerr_key _best_valueutil get_trigger_interval_trigger _init_summary _compare_fn)selfkey compare_fntriggerrrH/home/ubuntu/.local/lib/python3.10/site-packages/espnet/asr/asr_utils.py__init__s  zCompareValueTrigger.__init__cCs|j}|j}|j}||vr||||i||sdS|}t||}||jdur5||_dS| |j|r>dS||_dS)zsz!CompareValueTrigger._init_summaryN)r)__name__ __module__ __qualname____doc__rr rrrrrrs   r extensionc@s`eZdZdZ      dddZd d Zd d Zd dZddZddZ ddZ dddZ dS)PlotAttentionReportaPlot attention reporter. Args: att_vis_fn (espnet.nets.*_backend.e2e_asr.E2E.calculate_all_attentions): Function of attention visualization. data (list[tuple(str, dict[str, list[Any]])]): List json utt key items. outdir (str): Directory to save figures. converter (espnet.asr.*_backend.asr.CustomConverter): Function to convert data. device (int | torch.device): Device. reverse (bool): If True, input and output length are reversed. ikey (str): Key to access input (for ASR/ST ikey="input", for MT ikey="output".) iaxis (int): Dimension to access input (for ASR/ST iaxis=0, for MT iaxis=1.) okey (str): Key to access output (for ASR/ST okey="input", MT okay="output".) oaxis (int): Dimension to access output (for ASR/ST oaxis=0, for MT oaxis=0.) subsampling_factor (int): subsampling factor in encoder Finputroutputrc C||_t||_ddt|D|_||_||_||_||_||_ ||_ | |_ | |_ | |_ | |_tj|jsAt|jdSdS)NcSi|]\}}||qSrr.0kvrrr sz0PlotAttentionReport.__init__..) att_vis_fncopydeepcopydata data_dictoutdir converter transformdevicereverseikeyiaxisokeyoaxisfactorospathexistsmakedirs) rr4r7r9r:r;r<r=r>r?r@rAsubsampling_factorrrrrb  zPlotAttentionReport.__init__c Cs|\}}t|trt|d}t|D]?}t||D]6\}}d|j|||df}||||}d|j|||df} t | ||| || |qqt||D]2\}}d|j||f}||||}d|j||f} t | |||j || |ddq[d St|D]0\}}d|j||f}||||}d |j||f} t | ||| || |qd S) z*Plot and save image file of att_ws matrix.rz#%s/%s.ep.{.updater.epoch}.att%d.pngz#%s/%s.ep.{.updater.epoch}.att%d.npyz!%s/%s.ep.{.updater.epoch}.han.pngz!%s/%s.ep.{.updater.epoch}.han.npyT)han_mode%s/%s.ep.{.updater.epoch}.png%s/%s.ep.{.updater.epoch}.npyN) get_attention_weights isinstancelistlenrange enumerater9trim_attention_weightnpsaveformat_plot_and_save_attention) rratt_ws uttid_listnum_encsiidxatt_wfilename np_filenamerrrr s^     zPlotAttentionReport.__call__c Cs|\}}t|trht|d}t|D]+}t||D]"\}}||||}||} |d|||df| |qqt||D]\}}||||}| |} |d||| |qGdSt|D]\}}||||}||} |d||| |qldS)z4Add image files of att_ws matrix to the tensorboard.rz%s_att%dz%s_han%sN) rLrMrNrOrPrQrRdraw_attention_plot add_figuregcf draw_han_plot) rloggersteprWrXrYrZr[r\plotrrrlog_attentionss6          z"PlotAttentionReport.log_attentionscCX|j|jdd\}}||g|j}t|tr |j|}||fS|jdi|}||fS)adReturn attention weights. Returns: numpy.ndarray: attention weights. float. Its shape would be differ from backend. * pytorch-> 1) multi-head case => (B, H, Lmax, Tmax), 2) other case => (B, Lmax, Tmax). * chainer-> (B, Lmax, Tmax) T return_uttidNr)r;r7r:r<rMtupler4)r return_batchrXbatchrWrrrrLs   z)PlotAttentionReport.get_attention_weightsc Cs|jr|j|j}}|j|j}}n|j|j}}|j|j}}t|j|||dd}t|j|||dd}|jdkrH||j}t|j dkr^|ddd|d|f}|S|d|d|f}|S)z7Transform attention matrix with regard to self.reverse.shaperrN) r=r@rAr>r?intr8rBrOrn) ruttidr\enc_keyenc_axisdec_keydec_axisdec_lenenc_lenrrrrRs  z)PlotAttentionReport.trim_attention_weightcCsddl}|dddlm}||tj}t|j dkrEt |dD]\}}| dt|||j |dd| d|d q%n|j |dd| d|d ||S) zPlot the att_w matrix. Returns: matplotlib.pyplot: pyplot object with attention matrix image. rNAggrorauto)aspectz Encoder Index Decoder Index) matplotlibusematplotlib.pyplotpyplotclfastyperSfloat32rOrnrQsubplotimshowxlabelylabel tight_layout)rr\r|plthawrrrr`s"       z'PlotAttentionReport.draw_attention_plotcCsxddl}|dddlm}|t|jdkrrt|dD]Q\}}g}|dt||t |jdD]}| |dd|f| d |q5| ddg|d|jdg|d|d |d ||qnDg}t |jdD]}| |dd|f| d |q{| ddg|d|jdg|d|d |d ||||S) zPlot the att_w matrix for hierarchical attention. Returns: matplotlib.pyplot: pyplot object with attention matrix image. rNrxrorzAtt{}?TzAttention Weightr{)r|r}r~rrrOrnrQrrPrfappendrUylimxlimgridrrlegendr)rr\r|rrrlegendsrZrrrrcs<           z!PlotAttentionReport.draw_han_plotcCs0|r||}n||}|||dSN)rcr`savefigclose)rr\r]rIrrrrrV:s     z,PlotAttentionReport._plot_and_save_attentionNFr*rr+rr)F) r#r$r%r&rr rgrLrRr`rcrVrrrrr)Js   2(r)c@sVeZdZdZ      dddZd d Zd d Zd dZddZddZ ddZ dS) PlotCTCReporta Plot CTC reporter. Args: ctc_vis_fn (espnet.nets.*_backend.e2e_asr.E2E.calculate_all_ctc_probs): Function of CTC visualization. data (list[tuple(str, dict[str, list[Any]])]): List json utt key items. outdir (str): Directory to save figures. converter (espnet.asr.*_backend.asr.CustomConverter): Function to convert data. device (int | torch.device): Device. reverse (bool): If True, input and output length are reversed. ikey (str): Key to access input (for ASR/ST ikey="input", for MT ikey="output".) iaxis (int): Dimension to access input (for ASR/ST iaxis=0, for MT iaxis=1.) okey (str): Key to access output (for ASR/ST okey="input", MT okay="output".) oaxis (int): Dimension to access output (for ASR/ST oaxis=0, for MT oaxis=0.) subsampling_factor (int): subsampling factor in encoder Fr*rr+rc Cr,)NcSr-rrr.rrrr2rr3z*PlotCTCReport.__init__..) ctc_vis_fnr5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErF) rrr7r9r:r;r<r=r>r?r@rArGrrrrarHzPlotCTCReport.__init__c Cs|\}}t|trWt|d}t|D]?}t||D]6\}}d|j|||df}||||}d|j|||df} t | ||| || |qqdSt|D]0\}}d|j||f}||||}d|j||f} t | ||| || |q[dS)z%Plot and save image file of ctc prob.rz#%s/%s.ep.{.updater.epoch}.ctc%d.pngz#%s/%s.ep.{.updater.epoch}.ctc%d.npyrJrKN) get_ctc_probsrMrNrOrPrQr9 trim_ctc_probrSrTrU_plot_and_save_ctc) rr ctc_probsrXrYrZr[ctc_probr]r^rrrr sB    zPlotCTCReport.__call__c Cs|\}}t|trCt|d}t|D]+}t||D]"\}}||||}||} |d|||df| |qqdSt|D]\}}||||}||} |d||| |qGdS)z0Add image files of ctc probs to the tensorboard.rz%s_ctc%dr_N) rrMrNrOrPrQr draw_ctc_plotrarb) rrdrerrXrYrZr[rrfrrr log_ctc_probss&       zPlotCTCReport.log_ctc_probscCrh)zReturn CTC probs. Returns: numpy.ndarray: CTC probs. float. Its shape would be differ from backend. (B, Tmax, vocab). TriNr)r;r7r:r<rMrkr)rrlrXrmprobsrrrrs  zPlotCTCReport.get_ctc_probscCsFt|j||j|jdd}|jdkr||j}|d|}|S)z.Trim CTC posteriors accoding to input lengths.rnrrN)rpr8r>r?rB)rrqprobrwrrrrs "   zPlotCTCReport.trim_ctc_probc Csddl}|dddlm}|tj}|tj|dd}|j \}}t |}|j ddt | dD]#}|dkrP|j||dddfd d d d q9|||dd|fq9|jd dd|jddd|ttdt|dd|ttddd||S)zPlot the ctc_prob matrix. Returns: matplotlib.pyplot: pyplot object with CTC prob matrix image. rNrxr)axis))figsize:zgrey)labelcolorz Input [frame] )fontsize Posteriors )r|r}r~rrrSrrargsortrnarangefiguresetreshapetolistrfrrxticksrNrPrpyticksr) rrr|rtopk_idsn_framesvocab times_probsr[rrrrs*      zPlotCTCReport.draw_ctc_plotcCs ||}|||dSr)rrr)rrr]rrrrrs   z PlotCTCReport._plot_and_save_ctcNr) r#r$r%r&rr rrrrrrrrrrIs  " $rcsDddl}ddlm}dur|jj|jddfdd}|S)zQExtension to restore snapshot. Returns: An extension function. rNrrrcstdSr)_restore_snapshotrload_fnmodelsnapshotrrrestore_snapshotsz*restore_snapshot..restore_snapshot)rr serializersload_npzmake_extension)rrrrrrrrrrs  rcCs8|dur ddl}|jj}|||tdt|dS)Nrzrestored from )rrrlogginginfostr)rrrrrrrr s  rc(ddlm}|jddfdd}|S)zExtension to perform adadelta eps decay. Args: eps_decay (float): Decay rate of eps. Returns: An extension function. rrrrct|dSr)_adadelta_eps_decayr eps_decayrradadelta_eps_decay#z.adadelta_eps_decay..adadelta_eps_decayrrr)rrrrrrr rcCz|jd}t|dr"|j}t|d||tdt|jdS|jD]}|d|9<tdt|dq%dS)Nmainepszadadelta eps decayed to ) updater get_optimizerhasattrrsetattrrrr param_groups)rr optimizer current_epsprrrr*   rcr)zExtension to perform adam lr decay. Args: eps_decay (float): Decay rate of lr. Returns: An extension function. rrrrcrr)_adam_lr_decayrrrr adam_lr_decayDrz$adam_lr_decay..adam_lr_decayr)rrrrrrr8rrcCr)Nrlrzadam lr decayed to ) rrrrrrrrr)rrr current_lrrrrrrKrrzsnapshot.ep.{.updater.epoch}c,ddlm}|jdddfdd}|S)ziExtension to take snapshot of the trainer for pytorch. Returns: An extension function. rr'rrprioritycst|||dSr)_torch_snapshot_objectrUrr]savefunrrtorch_snapshotbsz&torch_snapshot..torch_snapshotchainer.trainingr(r)rr]r(rrrrrYs  rc Csddlm}|}||t|jjdr.t|jjjdr&|jjjj}n|jjj}nt|jjdr=|jjj}n|jj}|j||j dd}| |}d|} t j | |j d} tj| |} z||| t| tj|j |Wt| dSt| w) Nr)DictionarySerializerrmoduler)rrrtmp)prefixdir)chainer.serializersrrTrrrr state_dicttargetrrUtempfilemkdtempoutrCrDjoinshutilmovermtree) rrr]rrsmodel_state_dict snapshot_dictfnrtmpdirtmppathrrrris,     rdr皙?c Csb||d}|||}|D]}|jdur.|j}|t||j} |j| 7_qdS)a'Adds noise from a standard normal distribution to the gradients. The standard deviation (`sigma`) is controlled by the three hyper-parameters below. `sigma` goes to zero (no noise) with more iterations. Args: model (torch.nn.model): Model. iteration (int): Number of iterations. duration (int) {100, 1000}: Number of durations to control the interval of the `sigma` change. eta (float) {0.01, 0.3, 1.0}: The magnitude of `sigma`. scale_factor (float) {0.55}: The scale of `sigma`. rN) parametersgradsizetorchrandntor<) r iterationdurationeta scale_factorintervalsigmaparam_shapenoiserrradd_gradient_noises     rcCs|dur tj|d}n|}t|d}td|t|}Wdn1s+wYt|t r?|}t j di|S|\}}}||t j di|fS)aGet model config information by reading a model config file (model.json). Args: model_path (str): Model path. conf_path (str): Optional model config path. Returns: list[int, int, dict[str, Any]]: Config information loaded from json file. Nz /model.jsonrbzreading a config file from r) rCrDdirnameopenrrjsonloadrMdictargparse Namespace) model_path conf_path model_conffconfsargsidimodimrrrget_model_confs     r#cCs@ddl}dtj|vr|jj||dddS|j||dS)zLoad chainer model parameters. Args: path (str): Model path or snapshot file path to be loaded. model (chainer.Chain): Chainer model. rNrzupdater/model:main/)rD)rrCrDbasenamerr)rDrrrrr chainer_loadsr%cCs4t|drt|j|dSt||dS)zSave torch model states. Args: path (str): Model path to be saved. model (torch.nn.Module): Torch model. rN)rrrTrr)rDrrrr torch_saves r&cr)aReturns a trainer extension to take snapshots of a given object. Args: target (model): Object to serialize. filename (str): Name of the file into which the object is serialized.It can be a format string, where the trainer object is passed to the :meth: `str.format` method. For example, ``'snapshot_{.updater.iteration}'`` is converted to ``'snapshot_10000'`` at the 10,000th iteration. Returns: An extension function. rr'rrrcs ttj|j|dSr)r&rCrDrrrUrr]rrrsnapshot_objects z(snapshot_object..snapshot_objectr)rr]r(r(rr'rr(s  r(cCsfdtj|vrtj|dddd}n tj|ddd}t|dr+|j|~dS||~dS) zLoad torch model states. Args: path (str): Model path or snapshot file path to be loaded. model (torch.nn.Module): Torch model. rcS|Srrstoragelocrrrztorch_load.. map_locationrcSr)rrr*rrrr-r.rN)rCrDr$rrrrload_state_dict)rDrrrrr torch_loads   r2cCsddlm}tj|ddd}||d}||t|jjdr@t|jjjdr5|jjjj|dn&|jjj|dnt|jjdrR|jjj|dn |jj|d|j d |d ~d S) zResume from snapshot for pytorch. Args: snapshot_path (str): Snapshot file path. trainer (chainer.training.Trainer): Chainer's trainer instance. r)NpzDeserializercSr)rrr*rrrr-r.ztorch_resume..r/rrrrrN) rr3rrrrrrr1r) snapshot_pathrr3rdrrr torch_resumes   r6cstttt|ddd}fdd|D}t|d}ddd|D}d|}d |d d}||||fS) zParse hypothesis. Args: hyp (list[dict[str, Any]]): Recognition hypothesis. char_list (list[str]): List of characters. Returns: tuple(str, str, str, float) yseqrNcsg|]}|qSrrr/r[ char_listrr Bz$parse_hypothesis..score cSsg|]}t|qSr)rr8rrrr;Fr<z)rNmaprprrreplace)hypr:tokenid_as_list token_as_listr=tokenidtokentextrr9rparse_hypothesis5s    rHc Cst}|d|d<g|d<t|dD]a\}}t||\}}}} t|ddkr2t|dd} nddi} | dd|7<|| d<|| d <|| d <| | d <|d| |dkrsd | vrjtd | d td| dq|S)aTAdd N-best results to json. Args: js (dict[str, Any]): Groundtruth utterance dict. nbest_hyps_sd (list[dict[str, Any]]): List of hypothesis for multi_speakers: nutts x nspkrs. char_list (list[str]): List of characters. Returns: dict[str, Any]: N-best results added utterance dict. utt2spkr+rrnamer?z[%d]rec_text rec_token rec_tokenidr=rGzgroundtruth: %szprediction : %s) rrQrHrOitemsrkeysrr) js nbest_hypsr:new_jsnrBrKrLrMr=out_dicrrradd_results_to_jsonMs(  rUdbTFinfernoc Cs*t|}|dkrdt|t|jj}n |dkr|}nt||dur.|d}d}n|jd}d}|durI|durI|jd ||}d }n|jd }d }d|d|f}|j|ddd | |d | rm| d || rw| d || d ||j||||| | | | d|ddS)aPlot spectrogram using matplotlib. Args: plt (matplotlib.pyplot): pyplot object. spec (numpy.ndarray): Input stft (Freq, Time) mode (str): db or linear. fs (int): Sample frequency. To convert y-axis to kHz unit. frame_shift (int): The frame shift of stft. To convert x-axis to second unit. bottom (bool):Whether to draw the respective ticks. left (bool): right (bool): top (bool): labelbottom (bool):Whether to draw the respective tick labels. labelleft (bool): labelright (bool): labeltop (bool): cmap (str): Colormap defined in matplotlib. rVrlinearNikHzrbinrrframer)cmapextentz time [{}]z freq [{}]z{})bottomleftrighttop labelbottom labelleft labelrightlabeltopry)rSabslog10finfodtyper ValueErrorrnrrrUrcolorbar set_label tick_paramsr)rspecmodefs frame_shiftr^r_r`rarbrcrdrer\xytoprxtoprr]rrrplot_spectrogramsD #    ruc s*dddddddddd d d }|D]tttrOttjkr?td ttd jtd jt<qtsh|t<td|tdtfddt jDfddt jDt<qS)a Format args for multi-encoder setup. It deals with following situations: (when args.num_encs=2): 1. args.elayers = None -> args.elayers = [4, 4]; 2. args.elayers = 4 -> args.elayers = [4, 4]; 3. args.elayers = [4, 4, 4] -> args.elayers = [4, 4]. blstmpi,1gdoti@rr) etypeelayerseunits subsample dropout_rateatypeadimawinaheads aconv_chans aconv_filtsz%Length mismatch {}: Convert {} to {}.Nz*{} is not specified, use default value {}.z#Type mismatch {}: Convert {} to {}.cg|]}tqSrvarsr/_r r0rrr;z&format_mulenc_args..crrrrrrrr;r) rOrMrrNrOrYrwarningrUrP)r  default_dictrrrformat_mulenc_argssF     $$rr)rrr) rVNNTTTFTTTFrW)&rr5rrrCrrnumpyrSrobjectrrr( ImportErrorr) ExtensionrrrrrrrrTrrrr#r%r&r(r2r6rHrUrurrrrrsn2 z  1   $ '5 P