o }oikN@sddlZddlZddlZddlmZmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlZddlmZddlmZddlmZmZddlmZmZdd lmZdd lmZee e e e e e e e e e fZeGd d d Z eGd ddZ!ddZ"GdddeZ#dS)N)ABCabstractmethod) dataclass)AnyDictListOptionalTupleUnion) DataLoader)tqdm)audio_rttm_mapget_uniqname_from_filepath)PostProcessingParamsload_postprocessing_from_yaml)move_data_to_device)loggingc@seZdZUdZdZeejed<dZeejed<dZ e ed<dZ ee ed<dZ eed <d Zeed <dZeeed <dZeeed <dS)InternalDiarizeConfigzHInternal diarization configuration parameters for diarization inference.NdevicedtypeF training_mode logging_level dither_valuer pad_to_valuetemp_dirmanifest_filepath)__name__ __module__ __qualname____doc__rrtorch__annotations__rrboolrrrfloatrintrstrrr'r'a/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/mixins/diarization.pyr"s    rc@szeZdZUdZdZeed<dZeed<dZ eed<dZ e e ed<d Z eed <d Zeed <dZeed <dZe eed<dS) DiarizeConfigz3Configuration parameters for diarization inference.session_len_sec batch_size num_workersNpostprocessing_yamlTverboseFinclude_tensor_outputspostprocessing_params _internal)rrrr r+r$r"r-r%r.r/rr&r0r#r1r2rr3rr'r'r'r(r)5s       r)c Cs6t||r t||Std|d|d|d|S)a Utility function to get a value from the diarization config. If the value is not present in the diarization config, the default value is returned. Args: diarcfg: A dataclass that represents the diarization config. key: The name of the arg to retrieve. default: The default value to return if the key is not present in the diarization config. Returns: The value of the key in the diarization config or the default value. zUsing default value of z for zE because it is not present in the diarization config .)hasattrgetattrrdebug)diarcfgkeydefaultr'r'r(!get_value_from_diarization_configEs r;c@sLeZdZdZddZe      d(deee ee j e fde d ed eed e d ed eedefddZd eefddZde ede eeeeefffddZdeee efdefddZdefddZde edededeeeffddZedede fdd Zed!efd"d#Zededefd$d%Zdefd&d'ZdS))SpkDiarizationMixina An abstract class for diarize-able models. Creates a template function `diarize()` that provides an interface to perform transcription of audio tensors or filepaths. The following abstract classes must be implemented by the subclass: - `_setup_diarize_dataloader()`: Setup the dataloader for diarization. Receives the output from `_diarize_input_manifest_processing()`. - `_diarize_forward()`: Implements the model's custom forward pass to return outputs that are processed by `_diarize_output_processing()`. - `_diarize_output_processing()`: Implements the post processing of the model's outputs to return the results to the user. The result can be a list of objects, list of list of objects, tuple of objects, tuple of list of objects, or a dict of list of objects. cCs i|_dS)N)_diarize_audio_rttm_map)selfr'r'r(__init__ts zSpkDiarizationMixin.__init__r,FNaudior-r1r/r.r0override_configreturnc Ksz|durt|} td |||||| d|} nt|ds td|jdur)t|_|} | jdur5t| _n t| jts?tdd} zq|j|| d} | D]d} t| tr^| durXg} | | qKt| t r| durpt dd| D} t| d trt | D] \}}| | |q{qKt | t | krt d t | d t | d t | D] \}}| ||qqKW| StyY| Sw)zG Takes paths to audio files and returns speaker labels N)r-r.r0r1r/r2r3y`diarize_cfg must have an `_internal` argument, which must be of an object of type InternalDiarizeConfig or its subclass.Z`diarize_cfg._internal` must be of an object of type InternalDiarizeConfig or its subclass)rAcSsg|]}gqSr'r').0_r'r'r( sz/SpkDiarizationMixin.diarize..rz&The number of elements in the result (z3) does not match the results of the current batch (z).r')rr)r5 ValueErrorr3r isinstancediarize_generatorlistextendtuple enumeratelen RuntimeErrorappend StopIteration)r>r@r-r1r/r.r0rA config_kwargsr2 diarize_cfgresults generatorprocessed_outputsiprocessed_outputr'r'r(diarizewsn          zSpkDiarizationMixin.diarizec csj|durt}t|dstd|jdurt|_n t|jts%td|}z|||te}||j_ t|t sC| ||}n|}t|drN|j }nd}t t|d| dD]5\}}t||jj}t|j||j|d |j} ||} || | |} | V~~ ~ tjqZWdn1swYW||dSW||dS||w) z< A generator version of `diarize` function. Nr3rCrDr0T Diarizing)descdisabler,)r)r5rHr3rrI_diarize_on_begintempfileTemporaryDirectoryrr _diarize_input_processingr0rNr rrrKr=keysr-_diarize_forward_diarize_output_processingr!cuda empty_cache_diarize_on_end) r>r@rArTtmpdir dataloaderr0 batch_idx test_batchuniq_ids pred_outputsrWr'r'r(rJsN           ""z%SpkDiarizationMixin.diarize_generator audio_filescCs4i}|D]}t|}||ddddd}|||<q|S)z Generate manifest style dict if `audio` is a list of paths to audio files. Args: audio_files: A list of paths to audio files. Returns: audio_rttm_map_dict A list of manifest style dicts. rN-infer)uniq_idaudio_filepathoffsetdurationtextlabel)r)r>rnaudio_rttm_map_dict audio_filerqentryr'r'r(_input_audio_to_rttm_processings  z3SpkDiarizationMixin._input_audio_to_rttm_processingr8cCs|duriSt|tr|g}t|trt|dkriSt|ddd}|dur6t|ddd}t|td}t|dr>||_ |j |j _ t|drzt|j drat|j jd ra|j jj|j _d |j j_t|j drzt|j jd rz|j jj|j _d|j j_|t|j _ttjdS) a Internal function to setup the model for diarization. Perform all setup and pre-checks here. Args: audio (Union[str, List[str]]): Of type `GenericDiarizationType` diarcfg (DiarizeConfig): An instance of `DiarizeConfig`. Nrr.r,)r:r- preprocessor featurizerditherrpad_to)rIr&rKrOr;minos cpu_countr5r.trainingr3rr{r|r}rr~revalr get_verbosityr set_verbosityWARNING)r>r@r8r. _batch_sizer'r'r(r^*s.       z%SpkDiarizationMixin._diarize_on_beginc Cst|ttfrt|dkrtdn|g}t|dtrnt|dkr)|dds0|ddrQ|d|j_t |d|_ g}|j D] \}}| |dqDn t|}|j |d|_ |jj}||||}||}|Stdt|dd ) a Internal function to process the input audio data and return a DataLoader. This function is called by `diarize()` and `diarize_generator()` to setup the input data for diarization. Args: audio: Of type `GenericDiarizationType` diarcfg: The diarization config dataclass. Subclasses can change this to a different dataclass if needed. Returns: A DataLoader object that is used to iterate over the input audio data. rzInput `audio` is emptyr,z.jsonz.jsonlrr)rnInput `audio` is of type z8. Only `str` (path to audio file) is supported as input.)rIrKrMrOrHr&endswithr3rr r=itemsrQrzr"_diarize_input_manifest_processing_setup_diarize_dataloadertype) r>r@r8rnrq meta_dicttmp_dir ds_configtemp_dataloaderr'r'r(raZs, (  z-SpkDiarizationMixin._diarize_input_processingrcCsttj|dddd<}|D]1}t|tr'|ddd}|t|dqt|t r7|t|dqt d t |d Wd n1sKwY|t |d d |t |d|j t |dd d}|S)a Internal function to process the input audio filepaths and return a config dict for the dataloader. Args: audio_files: A list of string filepaths for audio files. temp_dir: A temporary directory to store intermediate files. diarcfg: The diarization config dataclass. Subclasses can change this to a different dataclass if needed. Returns: A config dict that is used to setup the dataloader for diarization. z manifest.jsonwzutf-8)encodingi)rrrtru rzC. Only `str` (path to audio file) or `dict` are supported as input.Nr-r,r+r.)paths2audio_filesr-rr+r.)openrpathjoinrIr&writejsondumpsdictrHrr;r+)r>rnrr8fprxryrr'r'r(rs&      z6SpkDiarizationMixin._diarize_input_manifest_processingconfigcCdS)a Internal function to setup the dataloader for diarization. This function is called by `diarize()` and `diarize_generator()` to setup the input data for diarization. Args: config: A config dict that is used to setup the dataloader for diarization. It can be generated by `_diarize_input_manifest_processing()`. Returns: A DataLoader object that is used to iterate over the input audio data. Nr')r>rr'r'r(r z-SpkDiarizationMixin._setup_diarize_dataloaderbatchcCr)a Internal function to perform the model's custom forward pass to return outputs that are processed by `_diarize_output_processing()`. This function is called by `diarize()` and `diarize_generator()` to perform the model's forward pass. Args: batch: A batch of input data from the data loader that is used to perform the model's forward pass. Returns: The model's outputs that are processed by `_diarize_output_processing()`. Nr')r>rr'r'r(rcrz$SpkDiarizationMixin._diarize_forwardcCr)a Internal function to process the model's outputs to return the results to the user. This function is called by `diarize()` and `diarize_generator()` to process the model's outputs. Args: outputs: The model's outputs that are processed by `_diarize_forward()`. uniq_ids: List of unique recording identificators in batch diarcfg: The diarization config dataclass. Subclasses can change this to a different dataclass if needed. Returns: The output can be a list of objects, list of list of objects, tuple of objects, tuple of list of objects. Its type is defined in `GenericDiarizationType`. Nr')r>outputsrlr8r'r'r(rdsz.SpkDiarizationMixin._diarize_output_processingcCs|j|jjdt|dr5t|jdr!t|jjdr!|jj|jj_t|jdr5t|jjdr5|jj|jj_ |jj durDt |jj dSdS)z Internal function to teardown the model after transcription. Perform all teardown and post-checks here. Args: diarcfg: The diarization config dataclass. Subclasses can change this to a different dataclass if needed. )moder{r|r}r~N) trainr3rr5r{r|rr}rr~rrr)r>r8r'r'r(rgs  z#SpkDiarizationMixin._diarize_on_end)r,FNr,FN) rrrr r?r!inference_moder r&rnpndarrayr r%r#rr)GenericDiarizationTyperZrJrr$rzr^rarrrrrcrdrgr'r'r'r(r<\s^ X*B0,  %r<)$rrr_abcrr dataclassesrtypingrrrrr r numpyrr!torch.utils.datar r .nemo.collections.asr.parts.utils.speaker_utilsr r*nemo.collections.asr.parts.utils.vad_utilsrr"nemo.collections.common.data.utilsr nemo.utilsrrrr)r;r<r'r'r'r(s*       (