o }oi@@sddlZddlZddlZddlmZmZmZddlZddl Z ddl Z ddl m Z ddlmZddlmZmZmZddlmZddlmZddlmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(dd l)m*Z*m+Z+m,Z,dd l-m.Z.zddl/Z0dd l1m2Z2m3Z3m4Z4d Z5Wn e6ydZ5Ynwzddl7m8Z8m9Z9m:Z:m;Z;d ZGddde>Z?dS)N)DictListTuple) OmegaConf)convolve)cosinehamminghann)tqdm)process_augmentations) DataAnnotator SpeechSamplerbuild_speaker_samples_mapget_background_noiseget_cleaned_base_pathget_random_offset_indexget_speaker_idsget_speaker_samplesget_split_points_in_alignmentsload_speaker_samplenormalize_audioper_speaker_normalize perturb_audioread_audio_from_bufferread_noise_manifest) read_manifest)get_overlap_range is_overlapmerge_float_intervals)logging)CardioidFamilyDirectionVectorDirectivityPatternTF)att2t_SabineEstimatorbeta_SabineEstimation simulateRIRt2nc@s.eZdZdZddZdedededefdd Zd d Zd d Z ddZ dee fddZ dee dede ee effddZddZdedee defddZdWdedefd d!Zd"ede eeffd#d$Zd%ed&ed'ede eeffd(d)ZdXd+efd,d-Z dWd.ed/ed0ed1ed2ede eejff d3d4Zd5ed6eed7eeefd1efd8d9Zd5ededed:ed;edd?Zd@ej dAe defdBdCZ!dDeedEefdFdGZ"ded:ed@ejdHejde ejejeff dIdJZ# KdYdLedMedNed6eed7eeefdOedPej$dQefdRdSZ%dZdefdUdVZ&dTS)[MultiSpeakerSimulatora Multispeaker Audio Session Simulator - Simulates multispeaker audio sessions using single-speaker audio files and corresponding word alignments. Change Log: v1.0: Dec 2022 - First working verison, supports multispeaker simulation with overlaps, silence and RIR v1.0.1: Feb 2023 - Multi-GPU support for speed up - Faster random sampling routine - Fixed sentence duration bug - Silence and overlap length sampling algorithms are updated to guarantee `mean_silence` approximation v1.0.2: March 2023 - Added support for segment-level gain perturbation and session-level white-noise perturbation - Modified speaker sampling mechanism to include as many speakers as possible in each data-generation run - Added chunking mechanism to avoid freezing in multiprocessing processes v1.1.0 March 2023 - Faster audio-file loading with maximum audio duration parameter - Re-organized MultiSpeakerSimulator class and moved util functions to util files. v1.1.1 March 2023 - Changed `silence_mean` to use exactly the same sampling equation as `overlap_mean`. Args: cfg: OmegaConf configuration loaded from yaml file. Parameters: manifest_filepath (str): Manifest file with paths to single speaker audio files sr (int): Sampling rate of the input audio files from the manifest random_seed (int): Seed to random number generator session_config: num_speakers (int): Number of unique speakers per multispeaker audio session num_sessions (int): Number of sessions to simulate session_length (int): Length of each simulated multispeaker audio session (seconds). Short sessions (e.g. ~240 seconds) tend to fall short of the expected overlap-ratio and silence-ratio. session_params: max_audio_read_sec (int): The maximum audio length in second when loading an audio file. The bigger the number, the slower the reading speed. Should be greater than 2.5 second. sentence_length_params (list): k,p values for a negative_binomial distribution which is sampled to get the sentence length (in number of words) dominance_var (float): Variance in speaker dominance (where each speaker's dominance is sampled from a normal distribution centered on 1/`num_speakers`, and then the dominance values are together normalized to 1) min_dominance (float): Minimum percentage of speaking time per speaker (note that this can cause the dominance of the other speakers to be slightly reduced) turn_prob (float): Probability of switching speakers after each utterance mean_silence (float): Mean proportion of silence to speaking time in the audio session. Should be in range [0, 1). mean_silence_var (float): Variance for mean silence in all audio sessions. This value should be 0 <= mean_silence_var < mean_silence * (1 - mean_silence). per_silence_var (float): Variance for each silence in an audio session, set large values (e.g., 20) for de-correlation. per_silence_min (float): Minimum duration for each silence, default to 0. per_silence_max (float): Maximum duration for each silence, default to -1 for no maximum. mean_overlap (float): Mean proportion of overlap in the overall non-silence duration. Should be in range [0, 1) and recommend [0, 0.15] range for accurate results. mean_overlap_var (float): Variance for mean overlap in all audio sessions. This value should be 0 <= mean_overlap_var < mean_overlap * (1 - mean_overlap). per_overlap_var (float): Variance for per overlap in each session, set large values to de-correlate silence lengths with the latest speech segment lengths per_overlap_min (float): Minimum per overlap duration in seconds per_overlap_max (float): Maximum per overlap duration in seconds, set -1 for no maximum start_window (bool): Whether to window the start of sentences to smooth the audio signal (and remove silence at the start of the clip) window_type (str): Type of windowing used when segmenting utterances ("hamming", "hann", "cosine") window_size (float): Length of window at the start or the end of segmented utterance (seconds) start_buffer (float): Buffer of silence before the start of the sentence (to avoid cutting off speech or starting abruptly) split_buffer (float): Split RTTM labels if greater than twice this amount of silence (to avoid long gaps between utterances as being labelled as speech) release_buffer (float): Buffer before window at end of sentence (to avoid cutting off speech or ending abruptly) normalize (bool): Normalize speaker volumes normalization_type (str): Normalizing speakers ("equal" - same volume per speaker, "var" - variable volume per speaker) normalization_var (str): Variance in speaker volume (sample from standard deviation centered at 1) min_volume (float): Minimum speaker volume (only used when variable normalization is used) max_volume (float): Maximum speaker volume (only used when variable normalization is used) end_buffer (float): Buffer at the end of the session to leave blank outputs: output_dir (str): Output directory for audio sessions and corresponding label files output_filename (str): Output filename for the wav and RTTM files overwrite_output (bool): If true, delete the output directory if it exists output_precision (int): Number of decimal places in output files background_noise: add_bg (bool): Add ambient background noise if true background_manifest (str): Path to background noise manifest file snr (int): SNR for background noise (using average speaker power), set `snr_min` and `snr_max` values to enable random SNR snr_min (int): Min random SNR for background noise (using average speaker power), set `null` to use fixed SNR snr_max (int): Max random SNR for background noise (using average speaker power), set `null` to use fixed SNR segment_augmentor: add_seg_aug (bool): Set True to enable augmentation on each speech segment (Default: False) segmentor: gain: prob (float): Probability range (uniform distribution) gain augmentation for individual segment min_gain_dbfs (float): minimum gain in terms of dB max_gain_dbfs (float): maximum gain in terms of dB session_augmentor: add_sess_aug: (bool) set True to enable audio augmentation on the whole session (Default: False) segmentor: white_noise: prob (float): Probability of adding white noise (Default: 1.0) min_level (float): minimum gain in terms of dB max_level (float): maximum gain in terms of dB speaker_enforcement: enforce_num_speakers (bool): Enforce that all requested speakers are present in the output wav file enforce_time (list): Percentage of the way through the audio session that enforcement mode is triggered (sampled between time 1 and 2) segment_manifest: (parameters for regenerating the segment manifest file) window (float): Window length for segmentation shift (float): Shift length for segmentation step_count (int): Number of the unit segments you want to create per utterance deci (int): Rounding decimals for segment manifest file cCs||_t||_t||_t|jjj|_t |j|_ g|_ d|_ d|_ g|_g|_d|_g|_ddt|jjjjD|_d|_d|_d|_|jjjj|_|jjjdd|_d|_d|_t j!"rht #dnt #d |_$i|_%|jjjd d |_&|jjj'd dr|jjj'j(rt)|jjj'j*d |_'nd|_'|jjj+d dr|jjj+j,rt)|jjj+j*d |_+nd|_+|-|j.|jjjj/|jjjj|j 0|jjj1d|_2|jdd|_3|jjdd|_4|5|_6dS)NcSg|]}dqSr.0nr,r,]/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/data/data_simulation.py z2MultiSpeakerSimulator.__init__..r turn_prob_ming?cudacpuadd_missing_overlapF augmentor) augmenter)num_sess num_speakersall_speaker_ids random_seed num_workersmultiprocessing_chunksizei')7_paramsr annotatorr samplerrdata_simulatormanifest_filepath _manifestr_speaker_samples_noise_samples _sentence_text_words _alignments_min_alignment_count_merged_speech_intervalsrangesession_configr:_furthest_sample_missing_overlapbase_manifest_filepathsegment_manifest_filepathsession_paramsmax_audio_read_sec_max_audio_read_secget_turn_prob_min_volume _speaker_idstorchr4 is_availabledevice_device_audio_read_buffer_dictr6segment_augmentor add_seg_augr r7session_augmentor add_sess_aug _check_args_init_speaker_permutations num_sessionskeysr<_permutated_speaker_indsr=r?_init_chunk_count chunk_countselfcfgr,r,r0__init__sb            zMultiSpeakerSimulator.__init__r9r:r;r<c Cstj|tt|}tt|||}||}t|D]@}||kr(|} n|} | dkr9td| d|d|dkrJtj t|d| } nt | tj t|d| f} || 8}qt d|dt d|d |d | ||S) a Initialize the speaker permutations for the number of speakers in the session. When generating the simulated sessions, we want to include as many speakers as possible. This function generates a set of permutations that can be used to sweep all speakers in the source dataset to make sure we maximize the total number of speakers included in the simulated sessions. Args: num_sess (int): Number of sessions to generate num_speakers (int): Number of speakers in each session all_speaker_ids (list): List of all speaker IDs Returns: permuted_inds (np.array): Array of permuted speaker indices to use for each session Dimensions: (num_sess, num_speakers) rz seq_len is z at count z and should be greater than 0NzTotal z speakers in the source dataset.z%Initialized speaker permutations for z sessions with z speakers each.)nprandomseedlenlistintceilrN ValueError permutationhstackrinforeshape) rlr9r:r;r<all_speaker_id_countsperm_set_count target_countcountseq_len permuted_indsr,r,r0res"   "  z0MultiSpeakerSimulator._init_speaker_permutationscCstt|jjjj|jS)z Initialize the chunk count for multi-processing to prevent over-flow of job counts. The multi-processing pipeline can freeze if there are more than approximately 10,000 jobs in the pipeline at the same time. )rtrorur@rCrOrfr?rlr,r,r0ri*sz'MultiSpeakerSimulator._init_chunk_countcCs>|jjjjdkr td|jjjjdks|jjjjdkr td|jjjjdks0|jjjjdkr4td|jjjj|jkrQ|jjjj dkrQt d|j|jjj_|jjjj dkr]td|jjjj ddkrktd d|jjjj dkr~dkstd td |jjjjdks|jjjjdkrtd |jjjjdks|jjjjdkrtd |jjjjdkrtd |jjjjdkr|jjjj|jjjjd|jjjjkrtd|jjjjdkrtd|jjjjdkrtd|jjjjdkr|jjjj|jjjjd|jjjjkrtd|jjjjdkrtd|jjjjdks0|jjjjdkr4td|jjjjddksJ|jjjjddkrNtd|jjjjddksd|jjjjddkrhtd|jjjj|jjjjdkr{td|jjjjdvr|jjjjdurtdt|jdkrtddS)zO Checks YAML arguments to ensure they are within valid ranges. r>zMAt least one speaker is required for making audio sessions (num_speakers < 1)rz$Turn probability is outside of [0,1]TzTurn probability is less than {self._turn_prob_min} while enforce_num_speakers=True, which may result in excessive session lengths. Forcing turn_prob to 0.5.g@z4Max audio read time must be greater than 2.5 secondszfk (number of success until the exp. ends) in Sentence length parameter value must be a positive numberzQp (success probability) value in sentence length parameter must be in range (0,1]z Mean overlap is outside of [0,1]z Mean silence is outside of [0,1]z$Mean silence variance is not below 0zJMean silence variance should be lower than mean_silence * (1-mean_silence)zPer silence variance is below 0z*Mean overlap variance is not larger than 0zJMean overlap variance should be lower than mean_overlap * (1-mean_overlap)z)Per overlap variance is not larger than 0z%Minimum dominance is outside of [0,1]z-Speaker enforcement start is outside of [0,1]z+Speaker enforcement end is outside of [0,1]zManifest file is empty. Check that the source path is correct.)r@rCrOr: ExceptionrT turn_probrXspeaker_enforcementenforce_num_speakersrwarningrUsentence_length_params mean_overlap mean_silencemean_silence_varper_silence_varmean_overlap_varper_overlap_var min_dominance enforce_time window_typerrrErr,r,r0rd2s      z!MultiSpeakerSimulator._check_argscCs&d|_g|_g|_i|_tjdS)za Clear the system memory. Cache data for audio files and alignments are removed. N)rHrJrKr_r[r4 empty_cacherr,r,r0clean_ups zMultiSpeakerSimulator.clean_upreturncCsd|jjjj}tjj||jjjj|jjjjd}tj |dtj d}t |}|dkr@t t |D]}|||jjjj7<q1||d|jjjj|jjjj}t t |D]}|||jjjj7<|dkrv||||d||<qX|S)z Get the dominance value for each speaker, accounting for the dominance variance and the minimum per-speaker dominance. Returns: dominance (list): Per-speaker dominance ?locscalesizera_mina_maxr>)r@rCrOr:rorpnormalrT dominance_varclipinfsumrNrrr)rldominance_mean dominancetotalir,r,r0_get_speaker_dominances0     z,MultiSpeakerSimulator._get_speaker_dominancebase_speaker_dominancefactorcCsg}t|jjjjD]}|j|dkr||q t|dkrmt |}tt|dddD]}||||d||<q.|D] }|||||<q?|t |}tdt|D]}||||d||<qXd}||fS|}d}||fS)a Increase speaker dominance for unrepresented speakers (used only in enforce mode). Increases the dominance for these speakers by the input factor (and then re-normalizes the probabilities to 1). Args: base_speaker_dominance (list): Dominance values for each speaker. factor (int): Factor to increase dominance of unrepresented speakers by. Returns: dominance (list): Per-speaker dominance enforce (bool): Whether to keep enforce mode turned on rr>TF) rNr@rCrOr:rPappendrrrocopyr)rlrrincrease_percentrrenforcer,r,r0_increase_speaker_dominances&   z1MultiSpeakerSimulator._increase_speaker_dominancecCs|jjjjdkrt|jjjj|_dS|jjjjdkrGtj j d|jjjj |jjjjd|_tj t |j|jjjj|jjjjd|_dSdS)zc Set the volume for each speaker (either equal volume or variable speaker volume). equalvariablerrrN)r@rCrTnormalization_typeroonesrOr:rYrprnormalization_varrarray min_volume max_volumetolistrr,r,r0_set_speaker_volumes      z)MultiSpeakerSimulator._set_speaker_volume prev_speakerrcCs|jjjjdkr|durd}|S|}|Stjdd|jjjjkr'|dur'|S|}||krJtjdd}d}|||krF|d7}|||ks<||ks-|S)a6 Get the next speaker (accounting for turn probability and dominance distribution). Args: prev_speaker (int): Previous speaker turn. dominance (list): Dominance values for each speaker. Returns: prev_speaker/speaker_turn (int): Speaker turn r>Nr) r@rCrOr:rorpuniformrTr)rlrr speaker_turnrandr,r,r0_get_next_speakers"   z'MultiSpeakerSimulator._get_next_speakerF window_amountstartcCs|jjjjdkrt|d}n"|jjjjdkrt|d}n|jjjjdkr-t|d}ntdt | |j }|rB|d|S||dS)aw Get window curve to alleviate abrupt change of time-series signal when segmenting audio samples. Args: window_amount (int): Window length (in terms of number of samples). start (bool): If true, return the first half of the window. Returns: window (tensor): Half window (either first half or second half) rr)r rrN) r@rCrTrrr rrr[ from_numpytor^)rlrrwindowr,r,r0 _get_windows   z!MultiSpeakerSimulator._get_windowfirst_alignmentcCst|jjjj|jjj}t|jjjj|jjj}||kr&d}d}||fS|||kr6||}d}||fS|||}||fS)aP Get the start cutoff and window length for smoothing the start of the sentence. Args: first_alignment (int): Start of the first word (in terms of number of samples). Returns: start_cutoff (int): Amount into the audio clip to start window_amount (int): Window length r)rtr@rCrT window_sizesr start_buffer)rlrrr start_cutoffr,r,r0_get_start_buffer_and_window2s   z2MultiSpeakerSimulator._get_start_buffer_and_windowcurrent_sample_cursorremaining_dur_samplesremaining_len_audio_filecCst|jjjj|jjj}t|jjjj|jjj}|||kr'||}d}n||||kr5|||}||krA|}d}||fS|||krK||}||fS)aN Get the end buffer and window length for smoothing the end of the sentence. Args: current_sample_cursor (int): Current location in the target file (in terms of number of samples). remaining_dur_samples (int): Remaining duration in the target file (in terms of number of samples). remaining_len_audio_file (int): Length remaining in audio file (in terms of number of samples). Returns: release_buffer (int): Amount after the end of the last alignment to include window_amount (int): Window length r)rtr@rCrTrrrelease_buffer)rlrrrrrr,r,r0_get_end_buffer_and_windowJs    z0MultiSpeakerSimulator._get_end_buffer_and_windowr num_missingcCsbtt|jD] }|j|dkr|d7}q|dkr/t|jjjj|d|jjjjdSdS)z Check if any speakers were not included in the clip and display a warning. Args: num_missing (int): Number of missing speakers. rr>zFspeakers were included in the clip instead of the requested amount of N) rNrrrPwarningswarnr@rCrOr:)rlrkr,r,r0_check_missing_speakersks z-MultiSpeakerSimulator._check_missing_speakersaudio_manifestsentence_word_countmax_word_count_in_sentencemax_samples_in_sentence random_offsetc Csl|rtjjdt|dd}nd}t|d|d|jjj}||\} } |jjj j s/d} t|j } || } ||} d\}}}d}|}d}|| kr|| kr|t|dkrt|d||jjj| }||| krmnv|d|}|dkr||dkr|ng|j ||j t| d|jjjt| d|jjj|d||dkr|d7}|d7}qG|jdkr|j|7_n |jd |7_|d7}|d7}|}||7}|| kr|| kr|t|dksW|jjj jd ur+| dkr|j| d d }|j |j|_ t|j t|| | | |fd|_ t|j || | | |fd|j|_ nt|j || | |fd|j|_ |t|dkr|jjj jd ur||| t|| |d \}}t|j || || ||fd|j|_ |dkr|j|d d }| ||}| |||}t||||}t|j |fd|j|_ ~||t|j fS)aV Add audio file to current sentence (up to the desired number of words). Uses the alignments to segment the audio file. NOTE: 0 index is always silence in `audio_manifest['words']`, so we choose `offset_idx=1` as the first word Args: audio_manifest (dict): Line from manifest file for current audio file audio_file (tensor): Current loaded audio file sentence_word_count (int): Running count for number of words in sentence max_word_count_in_sentence (int): Maximum count for number of words in sentence max_samples_in_sentence (int): Maximum length for sentence in terms of samples Returns: sentence_word_count+current_word_count (int): Running word count len(self._sentence) (tensor): Current length of the audio file r>words)lowhigh alignmentsr)rrrr(r NT)rF)rorprandintrrrtr@rCrrrT start_windowrHrJrrKfloatrIrrrr^r[catmultiplyr)rlr audio_filerrrr offset_idxrrstart_window_amountsentence_samplesrremaining_durationprev_dur_samples dur_samplescurr_dur_samplescurrent_word_countword_idx silence_countwordrrend_window_amount sig_startsig_endwindowed_audio_filer,r,r0 _add_file|s        $    zMultiSpeakerSimulator._add_filer speaker_idsspeaker_wav_align_mapc Csrtj|jjjjd|jjjjdd}tjdtj |j d|_ d|_ gg|_ |_d\}}||kr||krt||||jd}t||jd|j|jd} t||j| |j |j|jdd \} } }|jjjjrmt| | |j|j d } ||| |||\}}||kr||ks6|jjjjrtt|j dkrt|j |j|jjjj|jjjt|j d } t |j | ||j!|j d |_ d Sd Sd S)a\ Build a new sentence by attaching utterance samples together until the sentence has reached a desired length. While generating the sentence, alignment information is used to segment the audio. Args: speaker_turn (int): Current speaker turn. speaker_ids (list): LibriSpeech speaker IDs for each speaker in the current session. speaker_wav_align_map (dict): Dictionary containing speaker IDs and their corresponding wav filepath and alignments. max_samples_in_sentence (int): Maximum length for sentence in terms of samples rr>)dtyper]r(rr)rrrmin_alignment_count)raudio_read_buffer_dict offset_minrUrT)r buffer_dict offset_indexr]rUr read_subsetr])rr split_bufferrsentence_audio_len)sentence_audiosplitsrvolumer]N)"rorpnegative_binomialr@rCrTrr[zerosfloat64r^rHrIrJrKrrLrr_rVrr`rarr normalizemaxabsrrrrrrrY) rlrrrrslrrrrrrrr,r,r0_build_sentencesp   "#  z%MultiSpeakerSimulator._build_sentencelengthsession_len_samplesprev_len_samplesrcCst||}|jj|} |j|| } ||kr|dur| r|j| } || } | dkr:| d| 8} |jd| 7_d} | |j|kr[| |j|| 8} |j|j|| 7_|j|} ||} |}| |}d}t| |g| |grt| |g| |g}t|d|dd}|| kr|j| |7_|jj |7_ | S|j |}||||kr|st|||} | S||} | S)a% Returns new overlapped (or shifted) start position after inserting overlap or silence. Args: speaker_turn (int): The integer index of the current speaker turn. prev_speaker (int): The integer index of the previous speaker turn. start (int): Current start of the audio file being inserted. length (int): Length of the audio file being inserted. session_len_samples (int): Maximum length of the session in terms of number of samples prev_len_samples (int): Length of previous sentence (in terms of number of samples) enforce (bool): Whether speaker enforcement mode is being used Returns: new_start (int): New starting position in the session accounting for overlap or silence Nrr>) rBrunning_speech_len_samplessilence_vs_overlap_selectorsample_from_overlap_modelrQrPrrrrunning_overlap_len_samplessample_from_silence_model)rlrrrr r rrrunning_len_samplesnon_silence_len_samples add_overlapdesired_overlap_amount new_start prev_startprev_endnew_endoverlap_amount overlap_rangesilence_amountr,r,r0_add_silence_or_overlapWs>     z-MultiSpeakerSimulator._add_silence_or_overlaprsnrcCs6|jd|jjj|jj|jj||jt|j d}|S)z Get meta data for the current session. Args: array (np.ndarray): audio array snr (float): signal-to-noise ratio Returns: dict: meta data r)duration silence_mean overlap_meanbg_snrrspeaker_volumes) shaper@rCrrBsess_silence_meansess_overlap_meanrZrsrY)rlrr  meta_datar,r,r0_get_session_meta_datas z,MultiSpeakerSimulator._get_session_meta_data rttm_listrc Csg}|D]}dd|D}|t|dt|dgqt||_tdd|jD}||jjj|}t ||jjj}t ||jjj} || fS)a- Calculate the total speech and silence duration in the current session using RTTM file. Args: rttm_list (list): List of RTTM timestamps running_len_samples (int): Total number of samples generated so far in the current session Returns: sess_speech_len_rttm (int): The total number of speech samples in the current session sess_silence_len_rttm (int): The total number of silence samples in the current session cSsg|]}|qSr,r,)r.tokenr,r,r0r1r2zHMultiSpeakerSimulator._get_session_silence_from_rttm..rr>cSsg|] }|d|dqS)r>rr,)r.xr,r,r0r1s) splitrrrrMrr@rCrrt) rlr+rall_sample_listx_rawr-total_speech_in_secstotal_silence_in_secssess_speech_lensess_silence_lenr,r,r0_get_session_silence_from_rttms  z4MultiSpeakerSimulator._get_session_silence_from_rttm is_speechcCsx||}|t|kr&tjj|d|t|f}tjj|d|t|f}||||j7<d|||<|||fS)a Add a sentence to the session array containing time-series signal. Args: start (int): Starting position in the session length (int): Length of the sentence array (torch.Tensor): Session array is_speech (torch.Tensor): Session array containing speech/non-speech labels Returns: array (torch.Tensor): Session array in torch.Tensor format is_speech (torch.Tensor): Session array containing speech/non-speech labels in torch.Tensor format rr>)rrr[nn functionalpadrH)rlrr rr6endr,r,r0_add_sentence_to_arrays   z,MultiSpeakerSimulator._add_sentence_to_arrayr)idxbasepathfilename noise_samplesr]enforce_counterc  Cs"|jjj} tj| |||_|} t| } | d\} } d}|j ||_ ddt |jjjjD|_d|_tj|jjjjd|jjjjd}|jjjj}t|jjjj|jjj}t||j}t||j}|j|j| |ks|r~| ||kr|r|| |\} }|r|d7}| || }|| }|rt!d}n||jjj"j#|jjjkrn|$||||t%|j&}|j'||| ||| |d}|j(||||d \}}}|j j)|j*|j+||jjj||jjj||d }|j j,d -||j j.|j/t0j12||d ||jjj||jjj||t0j12||d t0j12||dd}|j j,d3||j j4|j*|j+|||t!||jjjd}|j j,d-|t5| |} |j6|j j,d | d\|j_7|j_8| |j|<|}|} | |ks|s|jjj9j:rt;||jjj|j9|jrt%|j dkrt?||dkd}t@t%|||j |jA|jjj=jB|jjj=jC|jjj=jD| ||jd \}}||7}ntEdd}tF|}tG|r|HI}tJKt0j12||d ||jjj|j jL|||jM||dd~|N||fS)a _generate_session function without RIR simulation. Generate a multispeaker audio session and corresponding label files. Args: idx (int): Index for current session (out of total number of sessions). basepath (str): Path to output directory. filename (str): Filename for output files. speaker_ids (list): List of speaker IDs that will be used in this session. speaker_wav_align_map (dict): Dictionary containing speaker IDs and their corresponding wav filepath and alignments. noise_samples (list): List of randomly sampled noise source files that will be used for generating this session. device (torch.device): Device to use for generating this session. enforce_counter (int): In enforcement mode, dominance is increased by a factor of enforce_counter for unrepresented speakers rNcSr*r+r,r-r,r,r0r1 r2z;MultiSpeakerSimulator._generate_session..rr>rrrrr r rr)rr rr6)rrrr: speaker_idrttm.wav.rttm.ctm)text wav_filenamerr rB rttm_filepath ctm_filepathjson)rr session_namerBrctm)r+rrr) len_array power_arrayr?rsnr_minsnr_maxbackground_noise_snrrqr]z9No background noise samples found in self._noise_samples.N/Arr r=r>r))Or@rCr<rorprqr^rrrrAinit_annotation_listsrGrNrOr:rP_missing_silencerrrrrtsession_lengthrr[rrrBget_session_silence_meanget_session_overlap_meanrrrrT end_bufferr rrrHrr;create_new_rttm_entryrJrK annote_listsextendcreate_new_json_entryrIospathjoinrcreate_new_ctm_entrymaximumr5rrunning_silence_len_samplesrbrcrr]background_noiseadd_bgmeanrr_rQrRr rvr is_tensorr5numpysfwritewrite_annotation_filesr*r)rlr<r=r>rrr?r]r@r<speaker_dominancerrrrrrr rr6rrr rr:new_rttm_entriesnew_json_entrynew_ctm_entriesavg_power_arraybgr r,r,r0_generate_sessions                    X      " z'MultiSpeakerSimulator._generate_sessionNc Cstd|dur|jjj}tj||jjjj }t ||jjjj d}t |jtj|dtjj|jd}g}|jjjj}t|jjjj|jjjjd}g}t|D]F} |jjjjd| } t| |j|jd} t | |jd } |j!j"|d } t#j$%rt#&d | t#j$'}n|j(}|)| || | | | |fqT|jd krd|_*d|_t|j+D]}g}||j,t-|d |j,|}}t||D]/} d dt|jjjj.D|_/i|_0|jd kr|)|j1|j2g|| Rq|)|| q|jd krtj3|}n|}t4|d|d d|j+d|d dd|ddt5|dD]+}|jd kr/|6\}} n|j!j"|d |_7|j2|\}} |j8j9|| d|:q q|;|j8j<|dtd|dS)z Generate several multispeaker audio sessions and corresponding list files. Args: random_seed (int): random seed for reproducibility zGenerating Diarization SessionsN)overwrite_outputz params.yaml) max_workers)rhbackground_manifest_)sess_idxspeaker_samplespermutated_speaker_inds)rr{)noise_manifestzcuda:r>cSr*r+r,r-r,r,r0r1r2z;MultiSpeakerSimulator.generate_sessions..[/z] Waiting jobs from z 2z to jobs)descunitr)r=r>)r=z6Data simulation has been completed, results saved at: )=rryr@rCr<rorprqoutputs output_dirrrvrsaverarbrc concurrentfuturesProcessPoolExecutorr=rOrfrrgrhrxrNoutput_filenamerrFrhrrBsample_noise_manifestr[r4r\r] device_countr^rrErjr?minr:rPr_submitru as_completedr rrresultrGrAadd_to_filename_listsrshutdownwrite_filelist_files)rlr<rr=tprrfsource_noise_manifestqueuerzr>rrr?r] chunk_idxstt_idxend_idx generatorfuturer,r,r0generate_sessionss              (  z'MultiSpeakerSimulator.generate_sessions)Fr+r)N)'__name__ __module__ __qualname____doc__rnrtrrerirdrrrrboolrrrrrrrdictr[Tensorrstrrrsr rrondarrayr*r5r;r]rurr,r,r,r0r'AszC+e " %  !    T G !  r) abs_weights)Tdiffrr) rorr@rCrrrndimrr&rNrprrrgrhvstack noise_src_posrrrrabsorbtion_paramsrT60att_diffatt_maxrr$r#r&r%)rl room_sz_tmprr pos_src_tmprj mic_pos_tmpmic_posrrrrrrrbetarTmaxnb_imgRIRRIR_padr,r,r0_generate_rir_gpuRIRYsZ $ (  (     z-RIRMultiSpeakerSimulator._generate_rir_gpuRIRrc Cs|jjjjj}|jjj}t|jjjjj }|j dkr?t |j d}t |j dD]}tj||df||df||<q*n|}t|jjjjj}|j dkrt |j d|j df}t |j dD]}t |j dD]}tj|||df|||df||<qmqdn|}t||\} } tj||t| | d} |jjjjrt||jjjjjf}|D]} | | q|jjjjj} |jjjjjdkrtj} tdddd }t|| d }t|jjjjj }|j dkrt |j d|j df}t |j dD] }t |j dD]}tj|||df|||df||<qqn|}| j!|j"|d | #d}| j$D]}|D]} | j dd|krG| j dd}q4q0| j$|fS) z Create simulated RIR using the pyroomacoustics library Returns: RIR (tensor): Generated RIR RIR_pad (int): Length of padding added when convolving the RIR with an audio file r)rr>r)fs materials max_orderrZT)azimuth colatitudedegrees) orientation pattern_enum) directivity)%r@rCrrrrrorrrrrr&rNrprrprainverse_sabineShoeBoxMaterialrgrhrr add_sourcerrr"OMNIr!r radd_microphone_arrayT compute_rirrir)rlrt60rrrrrrr e_absorptionrroomposrdir_vecdir_objrrrir_padchannelr,r,r0_generate_rir_pyroomacousticssf  $ (   *  z6RIRMultiSpeakerSimulator._generate_rir_pyroomacousticsrrc Csg}d}t|jjjjjD]F}|jjjjdkr(t||||dt|f }n|jjjjdkrAt||||dt| }t||krKt|}| t |q ||fS)a Augment one sentence (or background noise segment) using a synthetic RIR. Args: input (torch.tensor): Input audio. speaker_turn (int): Current speaker turn. RIR (torch.tensor): Room Impulse Response. Returns: output_sound (list): List of tensors containing augmented audio length (int): Length of output audio channels (or of the longest if they have different lengths) rrNr) rNr@rCrrrrrrrrrr[tensor)rlinputrr output_soundr r out_channelr,r,r0 _convolve_rirs "" z&RIRMultiSpeakerSimulator._convolve_rirr)r<r=r>rrr?r]r@c & Cs |jjj} tj| |||_|} t| } | d\} } d}|j ||_ ddt |jjjjD|_|jjjjdkrH|\}}n|jjjjdkrW|\}}ntdtj|jjjjd|jjjjd }|jjjj}t|jjjj|jjj}t||jjjjj f}t|}| |ks|r| ||kr|r|!| |\} }|r|d 7}|"|| }|| |}|rt#d }n||jjj$j%|jjjkrn|&|||||'|j(||\}}|j)||| ||| |d }||}|t*|krtj+j,-|ddd|t*|f}tj+j,-|d|t*|f}d |||<t |jjjjj D]}t*||}|||||f||7<q&|j .|j/|j0||jjj||jjj||}|j j1d 2||j 3|j4t5j67||d ||jjj||jjj||t5j67||dt5j67||d}|j j1d8||j 9|||||jjj} |j j1d2| t:| |} | |j|<|}|} | |ks|s|jjj;j<rt=||jjj|j;}|jjj>j?rLt*|j dkrt@||d kd}!tAt*||!|j |jB|jjj>jC|jjj>jD|jjj>jE| ||jd \}"}#||"7}|jFd}|G||!\}"}#|'|"d|\}$}%t |jjjjj D]}|dd|f|$|d|7<q5nd}#tH|}tI|r^|JK}tLMt5j67||d ||jjj|j jN|||jO||#dd~|P||fS)aR Generate a multispeaker audio session and corresponding label files. Args: idx (int): Index for current session (out of total number of sessions). basepath (str): Path to output directory. filename (str): Filename for output files. speaker_ids (list): List of speaker IDs that will be used in this session. speaker_wav_align_map (dict): Dictionary containing speaker IDs and their corresponding wav filepath and alignments. noise_samples (list): List of randomly sampled noise source files that will be used for generating this session. device (torch.device): Device to use for generating this session. enforce_counter (int): In enforcement mode, dominance is increased by a factor of enforce_counter for unrepresented speakers rNcSr*r+r,r-r,r,r0r1r2z>RIRMultiSpeakerSimulator._generate_session..rrrrr>rrArCrDrErFrKrMr)rNrrTrUrV)Qr@rCr<rorprqr^rrrrArWrGrNrOr:rPrrrrrrrrrrtrYrr[rrrrrrrTr\r rrHrrrr7r8r9r]rJrKr^r_r`rIrarbrcrrdrerbrcrrgrhrirr_rQrRr r&_get_backgroundrrjr5rkrlrmrnr*r)&rlr<r=r>rrr?r]r@r<rorrrrrrrrr rr6rraugmented_sentencer rr:rlen_chrprqrrrsrtr  augmented_bgryr,r,r0rus            $        Q     (  " z*RIRMultiSpeakerSimulator._generate_sessionr)rrrrrnrrrr[rrtrrsrrrr]ru __classcell__r,r,rr0rs2 39 C! r)@rrartypingrrrrkro soundfilerlr[ omegaconfr scipy.signalrscipy.signal.windowsrrr r 0nemo.collections.asr.parts.preprocessing.perturbr 6nemo.collections.asr.parts.utils.data_simulation_utilsr r rrrrrrrrrrrrr/nemo.collections.asr.parts.utils.manifest_utilsr.nemo.collections.asr.parts.utils.speaker_utilsrrr nemo.utilsrrrpyroomacoustics.directivitiesr r!r"rrrr#r$r%r&robjectr'rr,r,r,r0sP    D    I