o wi&A@sddlZddlmZddlmZddlmZmZddlZ ddl Z ddl m Z ddl mZddlmZmZddl mZdd lmZdd lmZdd lmZd gZGd ddZGdd d eZdS)N) defaultdict)partial)ListOptional)CutSet)MixedCut) DictConfig OmegaConf)nn)PromptedAudioToTextMiniBatch)WER) SerializationMultiTaskMetricc@seZdZdZdZdZdefddZeddZ edd Z e d d Z e d d Z e ddZe ddZe ddZe ddZe ddZddZdeefddZdS)ConstraintParserz5Boolean Parser class for constraint passing in configN constraintcCstdd|}|s|jS||t|dkr|dSg}dg|dg}|rb|}dkr[g}|rZ|}dkrS||}sMt d|| |n| ||s7n| ||s+||}spt d||S) Nz([()])z \1 r()z4Malformed subexpression find in constraint parsing: z"Parser cannot resolve constraint: ) resubstripsplit_no_constraint_resolve_primitiveslenpop_resolve_bools SyntaxErrorappend)selfrarraystackcexprefncr&c/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/asr/metrics/multitask.pyparse_constraint)s2      z!ConstraintParser.parse_constraintcCs |jdur tjtjd|_|jS)N)z==z!=) _primitivesoperatoreqnerr&r&r' primitivesEs zConstraintParser.primitivescCs$|jdur|j|j|jd|_|jS)N)andorxor) _booleans _logical_and _logical_or _logical_xorr-r&r&r'booleansNs zConstraintParser.booleanscCs|s td||| S)Nz:Malformed subexpression find in 'not' constraint parsing:  ValueError)r# propertiesr&r&r' _logical_notXs zConstraintParser._logical_notcCs,|r|std|d|||o||S)Nz:Malformed subexpression find in 'and' constraint parsing: z and r7l_exprr_exprr9r&r&r'r3^zConstraintParser._logical_andcCs,|r|std|d|||p||S)Nz9Malformed subexpression find in 'or' constraint parsing: z or r7r;r&r&r'r4dr>zConstraintParser._logical_orcCs,|r|std|d|||||AS)Nz:Malformed subexpression find in 'xor' constraint parsing: z xor r7r;r&r&r'r5jr>zConstraintParser._logical_xorcCsdS)NTr&)r9r&r&r'rpszConstraintParser._no_constraintcCs||||SNget)r%keyvalr9r&r&r'_static_constrainttsz#ConstraintParser._static_constraintcCs.||}duo||}duo|||Sr?r@)r%key1key2r9 prop_val1 prop_val2r&r&r'_compare_constraintxs z$ConstraintParser._compare_constraintc Cst|D]O\}}|jD]E\}}d|dd|d}}t||t||}} | r>| \} } t|j|| | ||<q |rQ|\} } t|j|| | ||<q q qdS)Nz \.(\S+)\s*z\s*(\S+)z \s*\.(\S+)) enumerater.itemsrmatchgroupsrrIrD) rridxr"noentailequal match_entail match_equalrErFrCr&r&r'rs  z$ConstraintParser._resolve_primitivescCsd}g}|t|kr/||}|dkr t|j||d}|d7}|||d7}|t|ks |}|jD]7\}}d}g}|t|krk||}||kr\t||||d}|d7}|||d7}|t|ksD|}q6t|dkrvdS|dS)Nrnotr)rrr:rr6rKr)rrrNr!r"rOrPr&r&r'rs4       zConstraintParser._resolve_bools)__name__ __module__ __qualname____doc__r)r2strr(propertyr.r6 staticmethodr:r3r4r5rrDrIrrrr&r&r&r'r#s2         rcseZdZdZdejdeffdd Z  dded e j d e j d e j d e e d e e f ddZd e j d e j d e j de j de j de j defddZdddZddZddZZS)ra Wrapper class for managing multiple metrics in multitask ASR/NLP models. This class enables conditional metric computation based on sample properties stored in Lhotse cuts. It's primarily designed for `EncDecMultiTaskModel` but can support any model with a prompt schema. Key Features: 1. **Automatic Model Integration**: Instantiated metrics are automatically added as attributes to the parent model, enabling seamless integration with existing logging infrastructure. 2. **Conditional Metric Updates**: Only samples meeting specific constraints are passed to each metric, avoiding inappropriate metric calculations (e.g., WER for translation tasks). 3. **Flexible Constraint System**: Supports complex logical expressions for determining when metrics should be applied to samples. 4. **Configuration Inheritance**: Global configuration parameters are automatically inherited by all metrics unless explicitly overridden. Args: model (nn.Module): Parent model that will receive metric instances as attributes. Must have a `decoding` attribute for metrics that require decoding. cfg (DictConfig): Configuration dictionary containing metric definitions and constraints. Configuration Format: The configuration should follow this structure: ``' # Global parameters (inherited by all metrics unless overridden) log_predictions: true batch_dim_index: 0 # Metric definitions metrics: wer: _target_: nemo.collections.asr.metrics.WER # Metric class to instantiate constraint: ".task == transcribe" # When to apply this metric use_cer: false # Metric-specific parameters bleu: _target_: nemo.collections.asr.metrics.BLEU constraint: ".task == translate" bleu_tokenizer: "13a" n_gram: 4 ``` Constraint Syntax: Constraints are evaluated against the `custom` dictionary of Lhotse cuts: - **Custom attribute Access**: `.task`, `.lang`, `.domain` - **Comparisons**: `==`, `!=` - **Logical Operations**: `and`, `or`, `not`, `xor` - **Property Comparisons**: `.source_lang == .target_lang` Examples: - `".task == transcribe"` - Apply to transcription tasks - `".task == translate and .source_lang != .target_lang"` - Cross-lingual translation - `"not .task == other"` - Apply to all tasks except 'other' - `".domain == medical or .domain == legal"` - Specific domains Usage Example: ```python # In model initialization if hasattr(cfg, 'multitask_metrics'): self.multitask_metrics = MultiTaskMetric(self, cfg.multitask_metrics) # During training/validation if hasattr(self, 'multitask_metrics'): metrics = self.multitask_metrics.eval( batch=batch, predictions=predictions, predictions_lengths=pred_lengths, predictions_mask=pred_mask, prefix="val", return_all_metrics=True ) self.log_dict(metrics) ``` Note: - Each metric receives the model's `decoding` instance for text decoding operations - Metrics are automatically instantiated for the parent model as attributes (e.g., `model.wer`, `model.bleu`) - Global configuration parameters are inherited unless explicitly overridden per metric - Metrics defined without 'constraint' keyword are called on every prediction sample - Empty batches (no samples matching constraints) are handled by child metrics. modelcfgc stii|_|_t|}t}t}|d D]I\}}|dd}| D] \}} ||vr8| ||<q,|j |d<t |} t ||| t| } | |vrUtd|| | |j|<|||j|<qdS)z Initialize MultiTaskMetric with model and configuration. Args: model (nn.Module): Parent model that will contain metric instances cfg (DictConfig): Configuration containing metric definitions metricsrdecodingzMultiTaskMetric currently only supports one instance of each metric class. Please check your configs for duplicates values of `_target_` entry.N)super__init__ _metric_dict _constr_dictr to_containerrsetrrKrarfrom_config_dictsetattrtype TypeErroraddr() rr]r^parser seen_typesname metric_cfgrkvmetric metric_type __class__r&r'rcs2       zMultiTaskMetric.__init__FNbatch predictionspredictions_lengthspredictions_maskreturn_all_metricsprefixc CsXi}|j|||j|j|t|dd|jd||j|r |dnd|d||S)Nprompt)rxrytargetstargets_lengthsrz input_idscuts_r`)r|r{)update transcripttranscript_lensgetattrrcomputereset)rrwrxryrzr{r| metric_dictr&r&r'eval2s$   zMultiTaskMetric.evalr~rrrc Csh||\}} |jD]%\} } || | | } } | j|| || || || || || | dq dS)N)rxryrzr~rrr) _split_cutsrdrKr)rrxryrzr~rrr cuts_split idx_splitrors cuts_subsetindicesr&r&r'rNs zMultiTaskMetric.updater`c Csi}|jD]=\}}t|tur:|\}}}|r/||d||d||d|iq||d|iq||j||dq|S)Nwerwer_num wer_denom)r{r|)rdrKrjr rr) rr{r| output_dictrorsrrrr&r&r'rhs,     zMultiTaskMetric.computecCsdd|jDdS)NcSsh|]\}}|qSr&)r).0rorsr&r&r' sz(MultiTaskMetric.reset..)rdrKr-r&r&r'rszMultiTaskMetric.resetcCsztttt}}t|D]+\}}t|tr|jn|}|jD]\}}||jr7|| ||| |q q ||fS)a Split cuts based on metric constraints and return filtered subsets. This method evaluates each cut against all metric constraints and creates separate lists of cuts and indices for each metric. Args: cuts (CutSet): Input cuts containing sample metadata Returns: tuple: (cuts_splits, idx_splits) where: - cuts_splits (dict): Maps metric names to lists of matching cuts - idx_splits (dict): Maps metric names to lists of matching indices Note: - Handles both regular cuts and MixedCuts (uses first_non_padding_cut) - A single cut may match multiple metrics - Cuts not matching any constraints are ignored ) rlistrJ isinstancerfirst_non_padding_cutrerKcustomr)rr cuts_splits idx_splitsrNr"rsconstrr&r&r'rs zMultiTaskMetric._split_cuts)FN)Fr`)rVrWrXrYr Modulerrcr torchTensorrboolrZrrrrrr __classcell__r&r&rur'rsHW4  !)r* collectionsr functoolsrtypingrrregexrrlhotser lhotse.cutr omegaconfrr r 7nemo.collections.asr.data.audio_to_text_lhotse_promptedr nemo.collections.asr.metrics.werr nemo.core.classesr __all__rrr&r&r&r's"