o i@sRdZddlZddlZddlZddlmZddlmZGdddejZ ddZ dS) a This script is used to provide utility functions designed for multi-speaker ASR. Copyright 2017 Johns Hopkins University (Shinji Watanabe) Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) Most functions can be directly used as in asr_utils.py: CompareValueTrigger, restore_snapshot, adadelta_eps_decay, chainer_load, torch_snapshot, torch_save, torch_resume, AttributeDict, get_model_conf. N) extension)parse_hypothesisc@sJeZdZdZdddZddZddZd d Zd d Zd dZ ddZ dS)PlotAttentionReporta6Plot attention reporter. Args: att_vis_fn (espnet.nets.*_backend.e2e_asr.calculate_all_attentions): Function of attention visualization. data (list[tuple(str, dict[str, dict[str, Any]])]): List json utt key items. outdir (str): Directory to save figures. converter (espnet.asr.*_backend.asr.CustomConverter): CustomConverter object. Function to convert data. device (torch.device): The destination device to send tensor. reverse (bool): If True, input and output length are reversed. FcCsL||_t||_||_||_||_||_tj |js$t |jdSdS)zInitialize PlotAttentionReport.N) att_vis_fncopydeepcopydataoutdir converterdevicereverseospathexistsmakedirs)selfrrr r r r rL/home/ubuntu/.local/lib/python3.10/site-packages/espnet/asr/asr_mix_utils.py__init__(s zPlotAttentionReport.__init__cCsn|}t|D],\}}t|D]#\}}d|j|j|d|df}||||}||||qqdS)z&Plot and save imaged matrix of att_ws.z&%s/%s.ep.{.updater.epoch}.output%d.pngrN)get_attention_weights enumerater rget_attention_weight_plot_and_save_attentionformat)rtrainer att_ws_sdnsatt_wsidxatt_wfilenamerrr__call__3s zPlotAttentionReport.__call__c Csp|}t|D]-\}}t|D]$\}}||||}||}|d|j|d|||qqdS)z3Add image files of attention matrix to tensorboard.z%srN)rrrdraw_attention_plot add_figurergcfclf) rloggersteprrrrr plotrrrlog_attentions@s   z"PlotAttentionReport.log_attentionscCs(||j|jg|j}|j|}|S)alReturn attention weights. Returns: arr_ws_sd (numpy.ndarray): attention weights. It's shape would be differ from bachend.dtype=float * pytorch-> 1) multi-head case => (B, H, Lmax, Tmax). 2) other case => (B, Lmax, Tmax). * chainer-> attention weights (B, Lmax, Tmax). )r transformrr r)rbatchrrrrrJs z)PlotAttentionReport.get_attention_weightscCs|jr&t|j|ddddd}t|j|dd|dd}n"t|j|dd|dd}t|j|ddddd}t|jdkr^|ddd|d|f}|S|d|d|f}|S)z5Transform attention weight in regard to self.reverse.rinputrshapeoutputN)r intrlenr.)rrr spkr_idxdec_lenenc_lenrrrrYs"$""z(PlotAttentionReport.get_attention_weightcCsddl}|dddlm}t|jdkr;t|dD]\}}|dt|||j|dd| d| d qn|j|dd| d| d | |S) zVisualize attention weights matrix. Args: att_w(Tensor): Attention weight matrix. Returns: matplotlib.pyplot: pyplot object with attention matrix image. rNAggr0rauto)aspectz Encoder Indexz Decoder Index) matplotlibusematplotlib.pyplotpyplotr2r.rsubplotimshowxlabelylabel tight_layout)rr r9plthawrrrr#gs      z'PlotAttentionReport.draw_attention_plotcCs ||}|||dS)N)r#savefigclose)rr r!rBrrrrs   z,PlotAttentionReport._plot_and_save_attentionN)F) __name__ __module__ __qualname____doc__rr"r*rrr#rrrrrrs     rcCst}|d|d<t|}g|d<t|D]a}g}||}t|dD]L\}} t| |\} } } } t|d|}|dd|7<| |d<| |d<| |d<| |d <|||dkrntd |d td |dq"|d|q|S) aWAdd N-best results to json. Args: js (dict[str, Any]): Groundtruth utterance dict. nbest_hyps_sd (list[dict[str, Any]]): List of hypothesis for multi_speakers (# Utts x # Spkrs). char_list (list[str]): List of characters. Returns: dict[str, Any]: N-best results added utterance dict. utt2spkr/rnamez[%d]rec_text rec_token rec_tokenidscorezgroundtruth: %stextzprediction : %s) dictr2rangerritemsappendlogginginfo)js nbest_hyps_sd char_listnew_js num_spkrsrtmp_js nbest_hypsnhyprMrNrOrPout_dicrrradd_results_to_jsons,   rb) rJrrVr chainer.trainingrespnet.asr.asr_utilsr Extensionrrbrrrrs    p