o wi @sddlZddlZddlZddlmZmZddlmZddlm Z ddl m Z m Z m Z mZmZmZddlZddlZddlmZddlmZmZddlmZdd lmZdd lmZmZdd lm Z dd l!m"Z"dd l#m$Z$ddl%m&Z&m'Z'ee e(e e"ee e(ee e"fZ)ee e e e e ee ee e e e(e e ffZ*eGdddZ+eGdddZ,ddZ-GdddeZ.GdddeZ/Gddde/Z0dS)N)ABCabstractmethod) dataclasspartial)AnyDictListOptionalTupleUnion) DictConfig) DataLoaderDataset)tqdm)process_augmentations) AudioSegmentChannelSelectorType)manifest_utils) Hypothesis)move_data_to_device)logging logging_modec@seZdZUdZeejed<dZeejed<dZe ed<dZ ee ed<dZ e ed<d Zeed <dZeeed <dZeeed <dS) InternalTranscribeConfigNdevicedtypeF training_mode logging_level dither_valuer pad_to_valuetemp_dirmanifest_filepath)__name__ __module__ __qualname__rr torch__annotations__rrboolrrrfloatr intr!strr"r,r,l/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/asr/parts/mixins/transcription.pyr(s    rc@seZdZUdZeed<dZeed<dZe eed<dZ e ed<dZ e e ed<dZe eed <d Zeed <dZe eeed <dZe eed <dS)TranscribeConfig batch_sizeFreturn_hypothesesN num_workerschannel_selector augmentor timestampsTverbosepartial_hypothesis _internal)r#r$r%r0r*r'r1r(r2r r3rr4r r5r6r7r rr8rr,r,r,r-r.9s    r.c Cs6t||r t||Std|d|d|d|S)a Utility function to get a value from the transcription config. If the value is not present in the transcription config, the default value is returned. Args: trcfg: A dataclass that represents the transcription config. key: The name of the arg to retrieve. default: The default value to return if the key is not present in the transcription config. Returns: The value of the key in the transcription config or the default value. zUsing default value of z for zG because it is not present in the transcription config .)hasattrgetattrrdebug)trcfgkeydefaultr,r,r-#get_value_from_transcription_configIs r@csBeZdZdeeefffdd ZddZddZdd Z Z S) TranscriptionTensorDatasetconfigcsjt|d|_|d|_|dd|_|d|_|jdur*t|jddd|_nd|_t |j|_ dS)N audio_tensorsr3r4 sample_rater) global_rank world_size) super__init__rCr3get augmentor_cfgrDrr4lenlengthselfrB __class__r,r-rIas     z#TranscriptionTensorDataset.__init__cCs*||jkrtd|d|j||S)NzIndex z" out of range for dataset of size )rM IndexErrorget_item)rOindexr,r,r- __getitem__os  z&TranscriptionTensorDataset.__getitem__cCs|jS)N)rM)rOr,r,r-__len__usz"TranscriptionTensorDataset.__len__cCs|j|}|jdur8tjdtjd|j}|jdtj d }t ||j |j |j d}|j|}tj|j|d}tj|jdtjd}||ddfS)NzAudio Augmentations are being applied during inference by moving the tensor onto CPU. This is highly inefficient and therefore not recommended.modecpu)rr) target_srr3)rr)rCr4rwarningrONCErtor&float32numpyrrDr3perturbtensorsamplesshapelong)rOrTrboriginal_dtypesegmentseq_lenr,r,r-rSxs    z#TranscriptionTensorDataset.get_item) r#r$r%rr+rrIrUrVrS __classcell__r,r,rPr-rA`s rAc@s`eZdZdZe        d-deeeee j e fde d e d e d eed ed e dee deedefddZdeefddZ defddZdefddZdeee j ejfdedefddZedeedededeeeffdd Zed!ede fd"d#Zed$edefd%d&Zededefd'd(Zdefd)d*Z d!edede fd+d,Z!dS).TranscriptionMixina An abstract class for transcribe-able models. Creates a template function `transcribe()` that provides an interface to perform transcription of audio tensors or filepaths. The following abstract classes must be implemented by the subclass: - `_transcribe_input_manifest_processing()`: Process the provided input arguments (filepaths only) and return a config dict for the dataloader. The data loader is should generally operate on NeMo manifests. - `_setup_transcribe_dataloader()`: Setup the dataloader for transcription. Receives the output from `_transcribe_input_manifest_processing()`. - `_transcribe_forward()`: Implements the model's custom forward pass to return outputs that are processed by `_transcribe_output_processing()`. - `_transcribe_output_processing()`: Implements the post processing of the model's outputs to return the results to the user. The result can be a list of objects, list of list of objects, tuple of objects, tuple of list of objects, or a dict of list of objects. r/FrNTaudior0r1r2r3r4r6r5override_configreturnc  Ks| durtd|||||||d| } nt| dstd| jdur&t| _| } | jdur2t| _n t| jts"sz1TranscriptionMixin.transcribe..rz&The number of elements in the result (z3) does not match the results of the current batch (z).zGiven output result for transcription is not supported. Please return a list of results, list of list of results, a dict of list of results, or a tuple of list of results.r,)r.r: ValueErrorr8r isinstancetranscribe_generatorlistextenddictitemstuple enumeraterL RuntimeErrorappendNotImplementedError StopIteration)rOrjr0r1r2r3r4r6r5rk config_kwargstranscribe_cfgresults generatorprocessed_outputskviprocessed_outputr,r,r- transcribes8          / zTranscriptionMixin.transcribec cs2|durt}t|dstd|jdurt|_n t|jts%td|}zk|||tI}||j_ t|t sC| ||}n|}t|drN|j }nd}t |d| dD]}t||jj}|||}|||} ~~| V~ qXWdn1s~wYW||dSW||dS||w) z? A generator version of `transcribe` function. Nr8rmrnr6T Transcribing)descdisable)r.r:rrr8rrs_transcribe_on_begintempfileTemporaryDirectoryr!r_transcribe_input_processingr6rrr_transcribe_forward_transcribe_output_processing_transcribe_on_end) rOrjrkrtmpdir dataloaderr6 test_batch model_outputsrr,r,r-rt?sH          z'TranscriptionMixin.transcribe_generatorr=cCsZ|duriSt|ttjtjfr|g}t|tr t|dkr iSt| }|j j dur1|j |j _ |j j dur<|j |j _ t |ddd}|durWt |ddd}t|td}t|dr_||_|j|j _t|drt|jd rt|jjd r|jjj|j _d |jj_t|jd rt|jjd r|jjj|j _d|jj_|t|j _ttj dS) + Internal function to setup the model for transcription. Perform all setup and pre-checks here. Args: audio: Of type `GenericTranscriptionType` trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. Nrr2)r?r0r/rE preprocessor featurizerditherrpad_to)!rsr+npndarrayr&TensorrurLnext parametersr8rrr@minos cpu_countr:r2trainingrrrrrrr evalr get_verbosityr set_verbosityWARNING)rOrjr=_paramsr2 _batch_sizer,r,r-rs8           z'TranscriptionMixin._transcribe_on_begincCs"t|ttfrt|dkrtdn|g}t|dtrSt|dkr)|dds0|ddr=|d|j_t |d}t|}|jj }| |||}| |}|St|dtjtjfrt|}tdd|Drrdd|D}|jj }||||}|||}|Std t|dd ) a Internal function to process the input audio data and return a DataLoader. This function is called by `transcribe()` and `transcribe_generator()` to setup the input data for transcription. Args: audio: Of type `GenericTranscriptionType` trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. Returns: A DataLoader object that is used to iterate over the input audio data. rzInput `audio` is emptyrEz.jsonz.jsonlcSsg|]}t|tjqSr,)rsrr)ro_tensorr,r,r-rqszCTranscriptionMixin._transcribe_input_processing..cSs&g|]}t|tjrt|n|qSr,)rsrrr& as_tensor)ro audio_tensorr,r,r-rqsInput `audio` is of type z[. Only `str` (path to audio file), `np.ndarray`, and `torch.Tensor` are supported as input.)rsruryrLrrr+endswithr8r"r read_manifestr!%_transcribe_input_manifest_processing_setup_transcribe_dataloaderrrr&rany#_transcribe_input_tensor_processing#_setup_transcribe_tensor_dataloadertype)rOrjr= audio_filestmp_dir ds_configtemp_dataloaderrCr,r,r-rs6 (   z/TranscriptionMixin._transcribe_input_processingrCr!cCsd}t|drd|jvr|jj}nt|dr|j}|dur!td|t|dd|t|ddt|d d|d }t|d d}|rB||d <|S) aE Internal function to process the input audio tensors and return a config dict for the dataloader. Args: audio_tensors: A list of numpy or torch tensors. The user must ensure that they satisfy the correct sample rate and channel format. temp_dir: A temporary directory to store intermediate files. trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. Returns: A config dict that is used to setup the dataloader for transcription. NcfgrDzProvided `audio` data contains numpy or torch tensors, however the class does not have `sample_rate` attribute. Please set `sample_rate` attribute to the model explicitly.r0r/r2rr3)rCr0r!r2r3rDr4)r:rrDr{r@)rOrCr!r=rDrr4r,r,r-rs(      z6TranscriptionMixin._transcribe_input_tensor_processingrcCdS)a Internal function to process the input audio filepaths and return a config dict for the dataloader. Args: audio_files: A list of string filepaths for audio files, or a single string filepath for a manifest file. temp_dir: A temporary directory to store intermediate files. trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. Returns: A config dict that is used to setup the dataloader for transcription. Nr,)rOrr!r=r,r,r-rz8TranscriptionMixin._transcribe_input_manifest_processingrBcCr)a Internal function to setup the dataloader for transcription. This function is called by `transcribe()` and `transcribe_generator()` to setup the input data for transcription. Args: config: A config dict that is used to setup the dataloader for transcription. It can be generated either by `_transcribe_input_manifest_processing()` or `_transcribe_input_tensor_processing()`. Returns: A DataLoader object that is used to iterate over the input audio data. Nr,rNr,r,r-r0s z/TranscriptionMixin._setup_transcribe_dataloaderbatchcCr)au Internal function to perform the model's custom forward pass to return outputs that are processed by `_transcribe_output_processing()`. This function is called by `transcribe()` and `transcribe_generator()` to perform the model's forward pass. Args: batch: A batch of input data from the data loader that is used to perform the model's forward pass. trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. Returns: The model's outputs that are processed by `_transcribe_output_processing()`. Nr,)rOrr=r,r,r-r?sz&TranscriptionMixin._transcribe_forwardcCr)a Internal function to process the model's outputs to return the results to the user. This function is called by `transcribe()` and `transcribe_generator()` to process the model's outputs. Args: outputs: The model's outputs that are processed by `_transcribe_forward()`. trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. Returns: The output can be a list of objects, list of list of objects, tuple of objects, tuple of list of objects, or a dict of list of objects. Its type is defined in `TranscriptionReturnType`. Nr,)rOoutputsr=r,r,r-rOrz0TranscriptionMixin._transcribe_output_processingcCs|j|jjdt|dr5t|jdr!t|jjdr!|jj|jj_t|jdr5t|jjdr5|jj|jj_ |jj durDt |jj dSdS) Internal function to teardown the model after transcription. Perform all teardown and post-checks here. Args: trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. rWrrrrN) trainr8rr:rrrrr rrrrrOr=r,r,r-r`s  z%TranscriptionMixin._transcribe_on_endc Cst|}ddlm}t|drt|jdr|jj}nt|dr,td|j |j }ntdd}t |d|d |d ddt ||d d S) a Internal function to setup the dataloader for transcription. This function is called by `transcribe()` and `transcribe_generator()` to setup the input data for transcription. Args: config: A config dict that is used to setup the dataloader for transcription. It can be generated either by `_transcribe_input_manifest_processing()` or `_transcribe_input_tensor_processing()`. trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. Returns: A DataLoader object that is used to iterate over the input audio data. r)_speech_collate_fn tokenizerpad_idtranscribe_pad_idz:Pad id is explicitly set to `model.transcribe_pad_id` = {}zPad id is being set to 0 because it could not be resolved from the tokenizer. This can happen for various reasons, especially for character based models. If pad id is incorrect, please provide the pad id explicitly by setting `model.transcribe_pad_id`.Fr0r2)r)datasetshuffler0r2 pin_memory drop_last collate_fn) rA'nemo.collections.asr.data.audio_to_textrr:rrrinfoformatrrr)rOrBr=rrrr,r,r-rts(    z6TranscriptionMixin._setup_transcribe_tensor_dataloader)r/FrNNTNN)"r#r$r%__doc__r&inference_moder r+r rrrr*r(r rr r.GenericTranscriptionTyperrtrrrrrrrrrrrrrr,r,r,r-ris|   A79 +  ric speZdZdZdeedededeeeffddZ deffdd Z deffd d Z e defd d Z ZS)ASRTranscriptionMixina An abstract class for ASR models that can transcribe audio. This class is a subclass of `TranscriptionMixin` that implements the default implementation of common abstract methods among the speech recognition model classes. The following abstract classes must be implemented by the subclass: - _transcribe_forward(): Implements the model's custom forward pass to return outputs that are processed by `_transcribe_output_processing()`. - _transcribe_output_processing(): Implements the post processing of the model's outputs to return the results to the user. The result can be a list of objects, list of list of objects, tuple of objects, tuple of list of rr!r=rlc Csttj|dddd<}|D]1}t|tr'|ddd}|t|dqt|t r7|t|dqt d t |d Wd n1sKwY|t |d d |t |ddt |dd t |ddt |ddd}t |dd }|rz||d<|S)aB Internal function to process the input audio filepaths and return a config dict for the dataloader. Specializes to ASR models which can have Encoder-Decoder-Joint architectures. Args: audio_files: A list of string filepaths for audio files. temp_dir: A temporary directory to store intermediate files. trcfg: The transcription config dataclass. Subclasses can change this to a different dataclass if needed. Returns: A config dict that is used to setup the dataloader for transcription. z manifest.jsonwzutf-8)encodingi)audio_filepathdurationtext rzC. Only `str` (path to audio file) or `dict` are supported as input.Nr0r/r2rr3 text_fieldr lang_fieldlang)paths2audio_filesr0r!r2r3rrr4) openrpathjoinrsr+writejsondumpsrwrrrr@) rOrr!r=fp audio_fileentryrr4r,r,r-rs0         z;ASRTranscriptionMixin._transcribe_input_manifest_processingcsRt||t|dr|jt|dr|jt|dr'|jdSdS)rencoderdecoderjointN)rHrr:rfreezerr)rOrjr=rPr,r-rs     z*ASRTranscriptionMixin._transcribe_on_begincs\t|t|dr|jjddt|dr|jjddt|dr,|jjdddSdS)rrTrrrN)rHrr:runfreezerrrrPr,r-rs    z(ASRTranscriptionMixin._transcribe_on_endcCstS)z Utility method that returns the default config for transcribe() function. Returns: A dataclass )r.)clsr,r,r-get_transcribe_configsz+ASRTranscriptionMixin.get_transcribe_config)r#r$r%rr r+r.rrrrr classmethodrrhr,r,rPr-rs  ,r)1rrrabcrr dataclassesr functoolsrtypingrrr r r r r_rr& omegaconfr torch.utils.datarrr0nemo.collections.asr.parts.preprocessing.perturbr0nemo.collections.asr.parts.preprocessing.segmentrr nemo.collections.asr.parts.utilsr+nemo.collections.asr.parts.utils.rnnt_utilsr"nemo.collections.common.data.utilsr nemo.utilsrrr+TranscriptionReturnTyperrr.r@rArirr,r,r,r-s@         (62