o }o™izã@sddlZddlZddlmZmZmZmZddlZddlm Z ddl m Z m Z m Z mZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*Gdd„deeƒZ+dS)éN)ÚDictÚListÚOptionalÚUnion)ÚTrainer)Ú DictConfigÚ ListConfigÚ OmegaConfÚ open_dict)Úaudio_to_text_dataset)Ú_AudioTextDataset)ÚAudioToBPEDALIDataset)ÚLhotseSpeechToTextBpeDataset)ÚRNNTLoss)ÚWER)ÚEncDecRNNTModel)Ú ASRBPEMixin)ÚRNNTBPEDecodingÚRNNTBPEDecodingConfig)Úget_semi_sorted_batch_sampler)Ú!get_lhotse_dataloader_from_config)ÚPretrainedModelInfo)ÚloggingÚ model_utilscs eZdZdZedeefdd„ƒZddede f‡fdd „ Z dd e e efd e d e efd d„Zdd edefdd„Zde efdd„Zdeddfdd„Z‡ZS)ÚEncDecRNNTBPEModelzKBase class for encoder decoder RNNT-based models with subword tokenization.ÚreturncCs g}tdddd}| |¡tdddd}| |¡tdd d d}| |¡td d d d}| |¡tdddd}| |¡tdddd}| |¡tdddd}| |¡tdddd}| |¡tdddd}| |¡tdddd}| |¡td d!d"d}| |¡td#d$d%d}| |¡td&d'd(d}| |¡td)d*d+d}| |¡td,d-d.d}| |¡td/d0d1d}| |¡td2d3d4d}| |¡td5d6d7d}| |¡td8d9d:d}| |¡td;dd?d@d}| |¡tdAdBdCd}| |¡tdDdEdFd}| |¡tdGdHdId}| |¡tdJdKdLd}| |¡tdMdNdOd}| |¡tdPdQdRd}| |¡tdSdTdUd}| |¡tdVdWdXd}| |¡tdYdZd[d}| |¡td\d]d^d}| |¡td_d`dad}| |¡tdbdcddd}| |¡|S)ezÁ This method returns a list of pre-trained model which can be instantiated directly from NVIDIA's NGC cloud. Returns: List of available pre-trained models. Ústt_en_contextnet_256zrFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_contextnet_256zvhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_contextnet_256/versions/1.6.0/files/stt_en_contextnet_256.nemo)Úpretrained_model_nameÚ descriptionÚlocationÚstt_en_contextnet_512zrFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_contextnet_512zvhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_contextnet_512/versions/1.6.0/files/stt_en_contextnet_512.nemoÚstt_en_contextnet_1024zsFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_contextnet_1024zxhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_contextnet_1024/versions/1.9.0/files/stt_en_contextnet_1024.nemoÚstt_en_contextnet_256_mlszvFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_contextnet_256_mlsz~https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_contextnet_256_mls/versions/1.0.0/files/stt_en_contextnet_256_mls.nemoÚstt_en_contextnet_512_mlszvFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_contextnet_512_mlsz~https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_contextnet_512_mls/versions/1.0.0/files/stt_en_contextnet_512_mls.nemoÚstt_en_contextnet_1024_mlszwFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_contextnet_1024_mlsz€https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_contextnet_1024_mls/versions/1.0.0/files/stt_en_contextnet_1024_mls.nemoÚ!stt_en_conformer_transducer_smallz~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_transducer_smallzŽhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_conformer_transducer_small/versions/1.6.0/files/stt_en_conformer_transducer_small.nemoÚ"stt_en_conformer_transducer_mediumzFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_transducer_mediumzhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_conformer_transducer_medium/versions/1.6.0/files/stt_en_conformer_transducer_medium.nemoÚ!stt_en_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_transducer_largezhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_conformer_transducer_large/versions/1.10.0/files/stt_en_conformer_transducer_large.nemoÚ$stt_en_conformer_transducer_large_lszFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_transducer_large_lsz”https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_conformer_transducer_large_ls/versions/1.8.0/files/stt_en_conformer_transducer_large_ls.nemoÚ"stt_en_conformer_transducer_xlargezFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_transducer_xlargez‘https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_conformer_transducer_xlarge/versions/1.10.0/files/stt_en_conformer_transducer_xlarge.nemoÚ#stt_en_conformer_transducer_xxlargez€For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_conformer_transducer_xxlargez’https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_conformer_transducer_xxlarge/versions/1.8.0/files/stt_en_conformer_transducer_xxlarge.nemoÚstt_de_contextnet_1024zsFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_contextnet_1024zxhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_de_contextnet_1024/versions/1.4.0/files/stt_de_contextnet_1024.nemoÚstt_fr_contextnet_1024zsFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_contextnet_1024zvhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_fr_contextnet_1024/versions/1.5/files/stt_fr_contextnet_1024.nemoÚstt_es_contextnet_1024zsFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_contextnet_1024zxhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_es_contextnet_1024/versions/1.8.0/files/stt_es_contextnet_1024.nemoÚ!stt_de_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_de_conformer_transducer_largezŽhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_de_conformer_transducer_large/versions/1.5.0/files/stt_de_conformer_transducer_large.nemoÚ!stt_fr_conformer_transducer_largez‰For details about this model, please visit https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_fr_conformer_transducer_largezŒhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_fr_conformer_transducer_large/versions/1.5/files/stt_fr_conformer_transducer_large.nemoÚ!stt_es_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_es_conformer_transducer_largezŽhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_es_conformer_transducer_large/versions/1.8.0/files/stt_es_conformer_transducer_large.nemoÚ#stt_enes_conformer_transducer_largez€For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_largez’https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_enes_conformer_transducer_large/versions/1.0.0/files/stt_enes_conformer_transducer_large.nemoÚstt_enes_contextnet_largezvFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_contextnet_largez~https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_enes_contextnet_large/versions/1.0.0/files/stt_enes_contextnet_large.nemoÚ!stt_ca_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ca_conformer_transducer_largezhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_ca_conformer_transducer_large/versions/1.11.0/files/stt_ca_conformer_transducer_large.nemoÚ!stt_rw_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_rw_conformer_transducer_largezhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_rw_conformer_transducer_large/versions/1.11.0/files/stt_rw_conformer_transducer_large.nemoÚ*stt_enes_conformer_transducer_large_codeswz‡For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_large_codeswz https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_enes_conformer_transducer_large_codesw/versions/1.0.0/files/stt_enes_conformer_transducer_large_codesw.nemoÚ"stt_kab_conformer_transducer_largezFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_kab_conformer_transducer_largez‘https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_kab_conformer_transducer_large/versions/1.12.0/files/stt_kab_conformer_transducer_large.nemoÚ!stt_be_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_be_conformer_transducer_largezhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_be_conformer_transducer_large/versions/1.12.0/files/stt_be_conformer_transducer_large.nemoÚ!stt_hr_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_hr_conformer_transducer_largezhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_hr_conformer_transducer_large/versions/1.11.0/files/stt_hr_conformer_transducer_large.nemoÚ!stt_it_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_it_conformer_transducer_largezhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_it_conformer_transducer_large/versions/1.13.0/files/stt_it_conformer_transducer_large.nemoÚ!stt_ru_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_ru_conformer_transducer_largezhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_ru_conformer_transducer_large/versions/1.13.0/files/stt_ru_conformer_transducer_large.nemoÚ!stt_eo_conformer_transducer_largez~For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_eo_conformer_transducer_largezhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_eo_conformer_transducer_large/versions/1.14.0/files/stt_eo_conformer_transducer_large.nemoÚ%stt_en_fastconformer_transducer_largez‚For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_fastconformer_transducer_largez–https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_fastconformer_transducer_large/versions/1.0.0/files/stt_en_fastconformer_transducer_large.nemoÚ(stt_en_fastconformer_transducer_large_lsz…For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_fastconformer_transducer_large_lszœhttps://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_fastconformer_transducer_large_ls/versions/1.0.0/files/stt_en_fastconformer_transducer_large_ls.nemoÚ&stt_en_fastconformer_transducer_xlargezƒFor details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_fastconformer_transducer_xlargez™https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_fastconformer_transducer_xlarge/versions/1.20.1/files/stt_en_fastconformer_transducer_xlarge.nemoÚ'stt_en_fastconformer_transducer_xxlargez„For details about this model, please visit https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_en_fastconformer_transducer_xxlargez›https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_fastconformer_transducer_xxlarge/versions/1.20.1/files/stt_en_fastconformer_transducer_xxlarge.nemo)rÚappend)ÚclsÚresultsÚmodel©rDú_/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/models/rnnt_bpe_models.pyÚlist_available_models)sý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý ý z(EncDecRNNTBPEModel.list_available_modelsNÚcfgÚtrainercs²t |¡}t |¡}d|vrtdƒ‚t|tƒst |¡}| |j ¡|j j   ¡}t |ƒt t |ƒƒ|_Wdƒn1s>wYt |jƒt|ƒ|j_Wdƒn1sYwYt |jƒ$t|ƒ|j_t t |ƒƒ|j_|jj|jj_|jj|jj_Wdƒn1sŠwYtƒj||d| |jj¡|j_t|jj|j|j|j d|_t |jd|j! "dd¡|j! "dd ¡d d |_#|jj$r×|j %|j&¡|j '|j#¡dSdS) NÚ tokenizerz:`cfg` must have `tokenizer` config to create a tokenizer !)rGrH©Ú decoding_cfgÚdecoderÚjointrIrÚuse_cerFÚlog_predictionT©ÚdecodingÚbatch_dim_indexrNrOÚdist_sync_on_step)(rÚ#convert_model_config_to_dict_configÚmaybe_update_config_versionÚ ValueErrorÚ isinstancerr ÚcreateÚ_setup_tokenizerrIÚ get_vocabr rÚlistÚlabelsrLÚlenÚ vocab_sizerMÚ num_classesÚ vocabularyÚmodel_defaultsÚ enc_hiddenÚjointnetÚencoder_hiddenÚ pred_hiddenÚsuperÚ__init__Ú#set_decoding_type_according_to_lossrGrQrrÚ_cfgÚgetÚwerÚ fuse_loss_werÚset_lossÚlossÚset_wer)ÚselfrGrHr`©Ú __class__rDrErgsN       ÿ ÿ  üü  û þzEncDecRNNTBPEModel.__init__Únew_tokenizer_dirÚnew_tokenizer_typerKc Cs t|tƒr|dkr |}n td|›ƒ‚d}|dur|}ntj |¡s)td|›ƒ‚| ¡dvr3tdƒ‚t  ||dœ¡}|  |¡|j j   ¡}|j  ¡}t |¡}|jdkr\t|ƒ|d<n tt| ¡ƒƒ|d<t|ƒ|d <|` t |¡|_ |j ¡} t | ¡} t|ƒ| _|`t | ¡|_|`t|j jd d |_|durŸ|jj}t t ¡} t  t !| ¡¡} t "| |¡}| #|¡}t$||j|j |j d |_t%|j|j&j'|j&j(|j&j)d d|_&|j j*sä|jj+durò|jj+dkrò|j  ,|j¡|j  -|j&¡t.|jj ƒ ||j_ Wdƒn 1swYt.|jjƒ | |j_Wdƒn 1s#wYt.|jjƒ ||j_Wdƒn 1s>wYt/ 0d|j j1›d¡dS)aš Changes vocabulary used during RNNT decoding process. Use this method when fine-tuning on from pre-trained model. This method changes only decoder and leaves encoder and pre-processing modules unchanged. For example, you would use it if you want to use pretrained encoder when fine-tuning on data in another language, or when you'd need model to learn capitalization, punctuation and/or special characters. Args: new_tokenizer_dir: Directory path to tokenizer or a config for a new tokenizer (if the tokenizer type is `agg`) new_tokenizer_type: Type of tokenizer. Can be either `agg`, `bpe` or `wpe`. decoding_cfg: A config for the decoder, which is optional. If the decoding type needs to be changed (from say Greedy to Beam decoding etc), the config can be passed here. Returns: None ÚaggzxNew tokenizer dir should be a string unless the tokenizer is `agg`, but this tokenizer type is: NzDNew tokenizer dir must be non-empty path to a directory. But I got: )ÚbpeÚwpez0New tokenizer type must be either `bpe` or `wpe`)ÚdirÚtyper`r_é)r_rJTrPrzChanged decoder to output to z vocabulary.)2rWrrVÚosÚpathÚisdirÚNotADirectoryErrorÚlowerr rXrYrIrZrMÚto_config_dictÚcopyÚdeepcopyÚtokenizer_typerr[Úkeysr]rÚfrom_config_dictrLr^rnrÚnum_classes_with_blankrGrQÚ structuredrÚ to_containerÚmergerhrrrkrRrNrOrlÚjoint_fused_batch_sizermror rÚinfor`) rprsrtrKÚnew_tokenizer_cfgÚ tokenizer_cfgr`Ú joint_configÚnew_joint_configÚdecoder_configÚnew_decoder_configÚ decoding_clsrDrDrEÚchange_vocabularyTsˆ ÿÿ ÿ               üû  ÿ ÿ ÿz$EncDecRNNTBPEModel.change_vocabularyTÚverbosecCs*|dur t d¡|jj}t t¡}t t |¡¡}t  ||¡}|  |¡}t ||j |j |jd|_t|j|jj|jj|jjdd|_|j jsR|jjdur`|jjdkr`|j  |j¡|j  |j¡| dd¡|j _t|jjƒ ||j_Wdƒn1s}wY|r“t d t |jj¡›¡dSdS) a[ Changes decoding strategy used during RNNT decoding process. Args: decoding_cfg: A config for the decoder, which is optional. If the decoding type needs to be changed (from say Greedy to Beam decoding etc), the config can be passed here. verbose: A flag to enable/disable logging. NzONo `decoding_cfg` passed when changing decoding strategy, using internal configrJTrPrÚ temperaturegð?zChanged decoding strategy to )rr‹rGrQr r‡rrXrˆr‰rhrrLrMrIrrkrRrNrOrlrŠrmrnrorjr•r Úto_yaml)rprKr”r’rDrDrEÚchange_decoding_strategyÉs>    üû  ÿÿz+EncDecRNNTBPEModel.change_decoding_strategyÚconfigc Csˆ| d¡r0t|| dd¡s|jn| d¡| dd¡s|jn| d¡t|j| dd¡d|jdStj||j|j|j|j|j  dd¡d }|durJdSt |t ƒrQ|S|d }t |t j jjƒr_d}t|d ƒrh|j}nt|jd d ƒrw|jd j}n |jd jd j}d}| d d¡r¦t |tƒs–tdt|ƒ›ƒ‚t|||ƒ}d|d<d|d<d}t j jj||d|d|| dd¡|| dd ¡| dd¡d S)NÚ use_lhotseÚ do_transcribeFÚ global_rankÚ world_size)rIÚ return_cuts)r›rœÚdatasetrIÚ preprocessor)r˜Ú local_rankr›rœrIÚpreprocessor_cfgÚshuffleÚ collate_fnrÚuse_semi_sorted_batchingzmSemi Sorted Batch sampler can be used with AudioToCharDataset or AudioToBPEDataset but found dataset of type Ú batch_sizeÚ drop_lastÚ num_workersÚ pin_memory) ržr¥ÚsamplerÚ batch_samplerr£r¦r¢r§r¨)rjrr›rœrrIr Ú)get_audio_to_text_bpe_dataset_from_configr rGrWr ÚtorchÚutilsÚdataÚIterableDatasetÚhasattrr£Údatasetsr Ú RuntimeErrorryrÚ DataLoader)rpr˜ržr¢r£rªrDrDrEÚ_setup_dataloader_from_configüsl  þõ ú     ÿÿ    ÷z0EncDecRNNTBPEModel._setup_dataloader_from_configztorch.utils.data.DataLoaderc Cs²d|vr |d}|d}ntj |dd¡}t|dt|dƒƒ}||jj|d| dt|t ¡dƒ¡d | d d ¡|j j  d d¡d œ}| d¡rO| d¡|d<|j t |ƒd}|S)aØ Setup function for a temporary data loader which wraps the provided audio file. Args: config: A python dictionary which contains the following keys: paths2audio_files: (a list) of paths to audio files. The files should be relatively short fragments. Recommended length per file is between 5 and 25 seconds. batch_size: (int) batch size to use during inference. Bigger will result in better throughput performance but would use more memory. temp_dir: (str) A temporary directory where the audio manifest is temporarily stored. Returns: A pytorch DataLoader for the given audio file(s). Úmanifest_filepathr¥Útemp_dirz manifest.jsonÚpaths2audio_filesFr§rzTÚchannel_selectorNÚuse_start_end_token)rµÚ sample_rater¥r¢r§r¨r¸r¹Ú augmentor)r˜) r{r|ÚjoinÚminr]rŸÚ _sample_raterjÚ cpu_countrGÚ validation_dsr´r)rpr˜rµr¥Ú dl_configÚtemporary_datalayerrDrDrEÚ_setup_transcribe_dataloaderBs$  ø z/EncDecRNNTBPEModel._setup_transcribe_dataloader)N)T)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú classmethodrrrFrrrgrÚstrrr“Úboolr—rr´rÃÚ __classcell__rDrDrqrEr&s"s<ü þý üu3Fr),rr{Útypingrrrrr¬Úlightning.pytorchrÚ omegaconfrrr r Únemo.collections.asr.datar Ú'nemo.collections.asr.data.audio_to_textr Ú,nemo.collections.asr.data.audio_to_text_dalir Ú.nemo.collections.asr.data.audio_to_text_lhotserÚ nemo.collections.asr.losses.rnntrÚ nemo.collections.asr.metrics.werrÚ'nemo.collections.asr.models.rnnt_modelsrÚ!nemo.collections.asr.parts.mixinsrÚ3nemo.collections.asr.parts.submodules.rnnt_decodingrrÚ-nemo.collections.asr.parts.utils.asr_batchingrÚ#nemo.collections.common.data.lhotserÚnemo.core.classes.commonrÚ nemo.utilsrrrrDrDrDrEÚs(