o wi 4@sddlmZmZmZmZmZddlZddlmZddl m Z ddl m Z m Z ddlmZddlmZddlmZdd lmZdd lmZd gZd Zd dZddZGdd d eZdS))LiteralOptionalSequence TypeAliasUnionN)CutSet)MixedCut)_bleu_score_compute_bleu_score_update)SacreBLEUScore)AbstractCTCDecoding)AbstractMultiTaskDecoding)AbstractRNNTDecoding)loggingBLEUbleu_tokenizercsddfdd|DS)z~ Helper function for multi tokenizer BLEU evaluation. Looks for `bleu_tokenizer` property to pass to BLEU metric. cSs |jtSN)customgetBLEU_TOKENIZER)cr^/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/asr/metrics/bleu.py _get_lang(s z1_get_bleu_tokenizers_from_cuts.._get_langcs(g|]}t|tr|jn|qSr) isinstancerfirst_non_padding_cut).0rrrr -s(z2_get_bleu_tokenizers_from_cuts..r)cutsrrr_get_bleu_tokenizers_from_cuts"sr cCs6tt|j}|j|g|d|||ddS)N)listrangendimpermute)tensor dim_indexall_dimsrrr_move_dimension_to_the_front0s(r)cseZdZUdZdZeed<edZe ed<         d&d e e e e fd ed ededeeededeffdd Z d'dejdejdejdejdeejdeejdeefddZd(fdd Zd d!Zd)d"d#Zd$d%ZZS)*ra This metric computes numerator, denominator, hypotheses lengths, and target lengths for Overall Bilingual Evaluation Understudy (BLEU) between prediction and reference texts. When doing distributed training/evaluation the result of ``res=BLEU.(predictions, predictions_lengths, targets, target_lengths)`` calls will be all-reduced between all workers using SUM operations. If used with PytorchLightning LightningModule, include bleu_num bleur_den, bleu_pred_len, and bleu_target_len values inside validation_step results. Then aggregate (sum) then at the end of validation epoch to correctly compute validation BLEUR. Example: def validation_step(self, batch, batch_idx): ... bleu_values = self.bleu(predictions, predictions_len, transcript, transcript_len) self.val_outputs = {'val_loss': loss_value, **bleu_values} return self.val_outputs def on_validation_epoch_end(self): ... bleu_num = torch.stack([x['val_bleu_num'] for x in self.val_outputs]).sum() bleu_denom = torch.stack([x['val_bleu_denom'] for x in self.val_outputs]).sum() bleu_pred_len = torch.stack([x['val_bleu_pred_len'] for x in self.val_outputs]).sum() bleu_target_len = torch.stack([x['val_bleu_target_len'] for x in self.val_outputs]).sum() val_bleu = {"val_bleu": self.bleu._compute_bleu(bleu_pred_len, bleu_target_len, bleu_num, bleu_denom)} tensorboard_logs.update(val_bleu) self.val_outputs.clear() # free memory return {'val_loss': val_loss_mean, 'log': tensorboard_logs} Args: decoding: Decoder instance (CTCDecoding, RNNTDecoding, or MultiTaskDecoding) for converting model outputs to text. tokenize: Tokenizer name for BLEU evaluation (affects BLEU score based on language/tokenization). n_gram: Maximum n-gram order for BLEU calculation (default: 4). lowercase: If True, lowercases all input texts before evaluation. weights: Optional sequence of float weights for each n-gram order. smooth: If True, applies smoothing to BLEU calculation. check_cuts_for_tokenizers: If True, will inspect cuts for a `BLEU_TOKENIZERS` attribute for 'on the fly' changes to tokenizer (see `cuts` argument in `update`). log_prediction: If True, logs the first reference and prediction in each batch. batch_dim_index: Index of the batch dimension in input tensors. dist_sync_on_step: If True, synchronizes metric state across distributed workers on each step. Returns: Dictionary containing BLEU score and component statistics (numerator, denominator, prediction_lengths, target_lengths). Tfull_state_update) none13azhintlcharzja-mecabzko-mecab flores101 flores200SacreBLEUTokenr,FNrdecodingrn_gram lowercaseweightssmoothcheck_cuts_for_bleu_tokenizersc  sD||_| |_| |_||_|||_tj|||||| | ddS)N)tokenizer5r6r7r8dist_sync_on_stepsync_on_compute)log_predictionfold_consecutivebatch_dim_indexr4 _init_decode check_cutssuper__init__)selfr4rr5r6r7r8r9r=r>r?r;r<kwargs __class__rrrCfs z BLEU.__init__ predictionspredictions_lengthstargetstargets_lengthspredictions_mask input_idsrc Ks~d} |jr"t||jdksJdt|d|jddt|} t|dkr5|||||ng} |jdkrBt ||j}| } | } t | D]Z\} }| | d| }|j|}| rl| | nd}t| | jg|gg|j|j|j|j|j||\|_|_| r|jr| dkrtdtd|td| | jqRWddS1swYdS) aT Updates metric state. Args: predictions: an integer torch.Tensor of shape ``[Batch, Time, {Vocabulary}]`` (if ``batch_dim_index == 0``) or ``[Time, Batch]`` (if ``batch_dim_index == 1``) predictions_lengths: an integer torch.Tensor of shape ``[Batch]`` targets: an integer torch.Tensor of shape ``[Batch, Time]`` (if ``batch_dim_index == 0``) or ``[Time, Batch]`` (if ``batch_dim_index == 1``) target_lengths: an integer torch.Tensor of shape ``[Batch]`` predictions_mask: a bool torch.Tensor of shape ``[Batch, Time]`` (if ``batch_dim_index == 0``) or ``[Time, Batch]`` (if ``batch_dim_index == 1``). Required for MultiTaskDecoding. input_ids: an int torch.Tensor of shape ``[Batch, Time]`` (if ``batch_dim_index == 0``) or ``[Time, Batch]`` (if ``batch_dim_index == 1``). Required for MultiTaskDecoding. cuts: a CutSet of ``length == batch size``. If `self.check_cuts`, inspects each element for SacreBLEU tokenizer type for corresponding element in batch. If a sequence element is ``None``, the initial tokenizer type from ``BLEU.__init__`` is used. If ``cuts == None`` then all elements in batch are tokenized with initial tokenizer type. Nrz?BLEU metrics configured for multiple tokenizers, but got only 'z' samples for 'z' predictions. zBLEU reference:zBLEU predicted:)rAlenshaper torchno_gradnumeldecoder?r)longcpu enumeratetolistr4decode_ids_to_strr text numerator denominator preds_len target_lenr5_get_tokenizerr=rinfo)rDrHrIrJrKrLrMrrE tokenizers hypothesestargets_cpu_tensortgt_lenths_cpu_tensoridxtgt_lentarget referencetokrrrupdatesH          "z BLEU.updatec szt}|r6|d||d|j|d|j|d|j|d|jiS|d|iS)a Returns BLEU values and component metrics. Args: return_all_metrics: bool flag. On True, BLEU and composite metrics returned. If False, returns only BLEU. Default: True. prefix: str to prepend to metric value keys. Returns: Dict: key-value pairs of BLEU metrics and values. Keys are prepended with prefix flag. bleu bleu_pred_lenbleu_target_lenbleu_num bleu_denom)rBcomputer]detachfloatr^r[r\)rDreturn_all_metricsprefixrlrFrrrqs   z BLEU.computecCst|||||j|j|jSr)r r5r7r8)rDrIrKr[r\rrr _compute_bleuszBLEU._compute_bleucs`jr|dur jS|jjvrtd|djjdtjjj|fddS)Nz"Sample passed BLEU tokenizer key 'z ' but BLEU config only support ''csj|jjSr) tokenizer_lowerr6split)linerD tokenizer_fnrrsz%BLEU._get_tokenizer..)rArx _TOKENIZE_FNKeyErrorkeysgetattr)rDr:rr|rr_s zBLEU._get_tokenizercstd_tjtrfdd_dStjtr!fdd_dStjtr0fdd_dStdtj)Ncsjj||dS)N)encoder_outputencoded_lengths)r4rnnt_decoder_predictions_tensorrHrIrLrMrDrrr~sz#BLEU._init_decode..csjj||jdS)N)decoder_outputsdecoder_lengthsr>)r4ctc_decoder_predictions_tensorr>rrrrr~ s csjj|||ddS)NF)encoder_hidden_statesencoder_input_maskdecoder_input_idsreturn_hypotheses)r4decode_predictions_tensor)rHprediction_lengthsrLrMrrrr~s z.BLEU metric does not support decoding of type )rTrr4rr r TypeErrortyperrrrr@s   zBLEU._init_decode) r,r3FNFFFTrFT)NNN)Trkr)__name__ __module__ __qualname____doc__r*bool__annotations__rr2rrr rr intrrrsrCrQTensorrrjrqrvr_r@ __classcell__rrrFrr5sl  +   ( G  )typingrrrrrrQlhotser lhotse.cutr!torchmetrics.functional.text.bleur r torchmetrics.textr 2nemo.collections.asr.parts.submodules.ctc_decodingr 8nemo.collections.asr.parts.submodules.multitask_decodingr 3nemo.collections.asr.parts.submodules.rnnt_decodingr nemo.utilsr__all__rr r)rrrrrs