o }oi@sddlmZmZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZmZmZmZddlmZgd ZGd d d e ZGd d d e ZGddde ZGddde ZGddde ZdS))DictOptionalN)Trainer) DictConfig)AudioToAudioModel)PretrainedModelInfo typecheck) AudioSignal LengthsTypeLossType NeuralType)logging)EncMaskDecAudioToAudioModel%ScoreBasedGenerativeAudioToAudioModelPredictiveAudioToAudioModel#SchroedingerBridgeAudioToAudioModelFlowMatchingAudioToAudioModelcseZdZdZddedeffdd Zedee e ffdd Z edee e ffd d Z e dd d ZddZddede fddZedeefddZZS)raClass for encoder-mask-decoder audio processing models. The model consists of the following blocks: - encoder: transforms input multi-channel audio signal into an encoded representation (analysis transform) - mask_estimator: estimates a mask used by signal processor - mask_processor: mask-based signal processor, combines the encoded input and the estimated mask - decoder: transforms processor output into the time domain (synthesis transform) Ncfgtrainercsd|_|dur |j|_tj||d|jj|_t|jj|_t|jj|_t|jj |_ t|jj |_ d|jvrKt dt|jj |_ nt dd|_ t|jdrm|jjdurmt dt|jj|_nt dd|_|dS) Nrrmixture_consistencyzUsing mixture consistencyzMixture consistency not usedchannel_augmentzUsing channel augmentationzChannel augmentation not used) world_sizesuper__init___cfg sample_raterfrom_config_dictencodermask_estimatormask_processordecoderr debugrhasattrrrchannel_augmentationsetup_optimization_flagsselfrr __class__]/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/audio/models/enhancement.pyr/s(       z$EncMaskDecAudioToAudioModel.__init__returncC(tdt|jdttdtdddSNBCTfreqr1Toptional input_signal input_lengthr r rtupler r(r+r+r, input_typesQ  z'EncMaskDecAudioToAudioModel.input_typescCr.Nr0r4r1Tr6 output_signal output_lengthr;r=r+r+r, output_typesZr?z(EncMaskDecAudioToAudioModel.output_typesc Cs|d}|j||d\}}|j||d\}}|j|||d\}} |jdur-|j||d}|j|| d\}} |j||d}|| fS)a> Forward pass of the model. Args: input_signal: Tensor that represents a batch of raw audio signals, of shape [B, T] or [B, T, C]. T here represents timesteps, with 1 second of audio represented as `self.sample_rate` number of floating point values. input_signal_length: Vector of length B, that contains the individual lengths of the audio sequences. Returns: Output signal `output` in the time domain and the length of the output signal `output_length`. inputr:)rGr:maskN)mixtureestimaterG batch_length)sizerr r!rr"match_batch_length) r(r9r:rLencodedencoded_lengthrH_ processedprocessed_lengthr+r+r,forwardcs  z#EncMaskDecAudioToAudioModel.forwardc Cst|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}|jr<|jdur<|j|d}|j||d\}}|j|||d}| d || d |j j d d | d t j |jjt jd|S)Nr9r: target_signal B T -> B 1 T)rGr8rJtargetr: train_loss learning_raterlr global_stepdtype) isinstancedictndimeinops rearrangetrainingr%rTlosslog _optimizer param_groupstorchtensorrr]float32) r(batch batch_idxr9r:rUrQprocessed_signalrfr+r+r, training_steps"         z)EncMaskDecAudioToAudioModel.training_steprvaldataloader_idxtagc Ct|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}|j||d\} }|j| ||d} t|dr_||jvr_|j|| D] \} } | j | ||d qR| d t j |jjt jd |d | iS Nr9r:rUrVrWr8rXmetricspredsrYr:r]r^_lossr`rarbrcrdrTrfr$rvitemsupdatergrjrkrr]rl) r(rmrnrrrsr9r:rUrQrorfnamemetricr+r+r,evaluation_steps       z+EncMaskDecAudioToAudioModel.evaluation_stepcCsg}|S)z This method returns a list of pre-trained model which can be instantiated directly from NVIDIA's NGC cloud. Returns: List of available pre-trained models. r+)clsresultsr+r+r,list_available_modelssz1EncMaskDecAudioToAudioModel.list_available_modelsNrrq)__name__ __module__ __qualname____doc__rrrpropertyrstrr r>rDrrTrpintr classmethodrrr __classcell__r+r+r)r,r%s " %""rcseZdZdZddedeffdd Zedee e ffdd Z edee e ffd d Z e dd d ZddZddede fddZZS)rzqThis models aims to directly estimate the coefficients in the encoded domain by applying a neural model. Nrrcstj||d|jj|_||jj|_||jj|_||jj|_|jdd|_ |jdd|_ | t d|jjt d|j t d|j dS) Nrnormalize_inputFeps:0yE>Initialized %s normalize_input: %s eps: %s)rrrrrrr" estimatorgetrrr&r r#r*rr'r)r+r,rs z$PredictiveAudioToAudioModel.__init__r-cCr.r/r;r=r+r+r,r>z'PredictiveAudioToAudioModel.input_typescCr.r@r;r=r+r+r,rDrz(PredictiveAudioToAudioModel.output_typesc Cs|d}|jrtj|ddd}|||j}|j||d\}}|j||d\}}|j||d\} } |jr;| |} |j | |d} | | fS)a Forward pass of the model. Args: input_signal: time-domain signal input_length: valid length of each example in the batch Returns: Output signal `output` in the time domain and the length of the output signal `output_length`. rErETdimkeepdimrFrK) rMrrjamaxabsrrrr"rN) r(r9r:rL norm_scalerOrP estimatedestimated_lengthoutputrCr+r+r,rTs z#PredictiveAudioToAudioModel.forwardc Cst|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}|j||d\}}|j|||d}|d||d |jj d d |d t j |j j t jd |S)Nr9r:rUrVrWr8rXrZr[rr\r]r^)r`rarbrcrdrTrfrgrhrirjrkrr]rl) r(rmrnr9r:rUrQrBrfr+r+r,rp*s        z)PredictiveAudioToAudioModel.training_steprrqrrrsc Crtrurz) r(rmrnrrrsr9r:rUrQrBrfr}r~r+r+r,rGs       z+PredictiveAudioToAudioModel.evaluation_steprr)rrrrrrrrrrr r>rDrrTrprrrr+r+r)r,rs $rcseZdZdZddedeffdd Zedee e ffdd Z edee e ffd d Z e edd d Ze e dee dee ededde deiddddZddZddede fddZZS)raThis models is using a score-based diffusion process to generate an encoded representation of the enhanced signal. The model consists of the following blocks: - encoder: transforms input multi-channel audio signal into an encoded representation (analysis transform) - estimator: neural model, estimates a score for the diffusion process - sde: stochastic differential equation (SDE) defining the forward and reverse diffusion process - sampler: sampler for the reverse diffusion process, estimates coefficients of the target signal - decoder: transforms sampler output into the time domain (synthesis transform) Nrrcstj||d|jj|_||jj|_||jj|_||jj|_||jj|_d|jj vr7t dd|jj vrAt dt j j |jj |j|jd|_ |jdd|_|jd |_|jdurjtd |j|jd d |_|td |jjtd|jtd|jdS)NrsdezDSDE should be defined in the model config, not in the sampler configscore_estimatorzPScore estimator should be defined in the model config, not in the sampler config)rrrFmax_utts_evaluation_metricsJMetrics will be evaluated on first %d examples of the evaluation datasets.rrrrr)rrrrrrr"rrsampler ValueErrorhydrautils instantiaterrrr warningrr&r#r*rr'r)r+r,rus.    z.ScoreBasedGenerativeAudioToAudioModel.__init__r-cCr.r/r;r=r+r+r,r>rz1ScoreBasedGenerativeAudioToAudioModel.input_typescCr.r@r;r=r+r+r,rDrz2ScoreBasedGenerativeAudioToAudioModel.output_typesc C|d}|jrtj|ddd}|||j}|j||d\}}|j|||d\}}|j||d\} } |jr<| |} |j | |d} | | fS)aIForward pass of the model. Forward pass of the model aplies the following steps: - encoder to obtain the encoded representation of the input signal - sampler to generate the estimated coefficients of the target signal - decoder to transform the sampler output into the time domain Args: input_signal: Tensor that represents a batch of time-domain audio signals, of shape [B, C, T]. T here represents timesteps, with 1 second of audio represented as `self.sample_rate` number of floating point values. input_signal_length: Vector of length B, contains the individual lengths of the audio sequences. Returns: Output `output_signal` in the time domain and the length of the output signal `output_length`. rErTrrF) prior_meanscore_condition state_lengthrK rMrrjrrrrrr"rN r(r9r:rLrrOrP generatedgenerated_lengthrrCr+r+r,rTs  z-ScoreBasedGenerativeAudioToAudioModel.forwardr0r1rUr9r:rfr>rDcCs|d}|jr tj|ddd}|||j}|||j}|j||d\}}|j||d\}} |jj||j d} |jj ||| d\} } t |} | | | }tj ||gdd }|j ||| d \}}|| }| }|j|||d }|S) zRandomly generate a time step for each example in the batch, estimate the score and calculate the loss value. Note that this step does not include sampler. rrTrrFrMdevice)statertimerrGr: conditionrX)rMrrjrrrrr generate_timerperturb_kernel_params randn_likecatrrf)r(rUr9r: batch_sizer input_enc input_enc_len target_encrQsde_timepk_meanpk_stdz_norm perturbed_encestimator_input score_est score_len score_refrfr+r+r,_steps"   z+ScoreBasedGenerativeAudioToAudioModel._stepcCst|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}|j|||d}|d||d|jjd d |d t j |j j t j d |S Nr9r:rUrVrWrrZr[rr\r]r^r`rarbrcrdrrgrhrirjrkrr]rlr(rmrnr9r:rUrQrfr+r+r,rps        z3ScoreBasedGenerativeAudioToAudioModel.training_steprrqrrrscCs|t|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}|j|||d} d} |jdurEd} |d } n%tt |j ||} |j ||| j } | |jk} t |j| |d } | r|j |d| d f|d| d \}}t|d r||j vr|j ||D]\}}|j||d| d f|d| d q|dtj|jjtjd|d| iSNr9r:rUrVrWrFTr.r8rvrwr]r^ryr`rarbrcrdrrrMnextiterrv num_examplesminrTr$r{r|rgrjrkrr]rlr(rmrnrrrsr9r:rUrQrfupdate_metricsrfirst_metric_namenum_examples_evaluatedrBr}r~r+r+r,r5s>            z5ScoreBasedGenerativeAudioToAudioModel.evaluation_steprrrrrrrrrrrrr r>rDrrjinference_moderTr r<r r rrprrrr+r+r)r,ris* ,,    4rcsFeZdZdZd&dedeffdd Zedee e ffdd Z edee e ffd d Z e ed&d d Ze e dee ededdde dee ededddded&ddZed'ddZe e dee dee ededde deidd&ddZddZd(d"ed#e fd$d%ZZS))raThis models uses a flow matching process to generate an encoded representation of the enhanced signal. The model consists of the following blocks: - encoder: transforms input multi-channel audio signal into an encoded representation (analysis transform) - estimator: neural model, estimates a score for the diffusion process - flow: ordinary differential equation (ODE) defining a flow and a vector field. - sampler: sampler for the inference process, estimates coefficients of the target signal - decoder: transforms sampler output into the time domain (synthesis transform) - ssl_pretrain_masking: if it is defined, perform the ssl pretrain masking for self reconstruction in the training process Nrrcs`tj||d|jj|_||jj|_||jj|_||jj|_||jj|_t j j |jj |jd|_ |j dd|_|j ddurWtd||jj|_nd|_|j dd|_|j d |_|jdurutd |j|j d d |_|td |jjtd|jdutd|jtd|jtd|jtd|jdS)Nr)rp_cond?ssl_pretrain_maskingz5SSL-pretrain_masking is found and will be initializedrFrrrrzInitialized %sz doing SSL-pretraining: %sz p_cond: %sz normalize_input: %sz loss: %sz eps: %s)rrrrrrr"rflowrrrrrrr r#rrrrrr&r*rrfr'r)r+r,r{s6   z&FlowMatchingAudioToAudioModel.__init__r-cCr.r/r;r=r+r+r,r>rz)FlowMatchingAudioToAudioModel.input_typescCr.r@r;r=r+r+r,rDrz*FlowMatchingAudioToAudioModel.output_typescC|j||ddS)aForward pass of the model to generate samples from the target distribution. This is used for inference mode only, and it explicitly disables SSL masking to the input. Args: input_signal: Tensor that represents a batch of raw audio signals, of shape [B, T] or [B, T, C]. T here represents timesteps, with 1 second of audio represented as `self.sample_rate` number of floating point values. input_signal_length: Vector of length B, that contains the individual lengths of the audio sequences. Returns: Output signal `output` in the time domain and the length of the output signal `output_length`. Fr9r:enable_ssl_maskingforward_internalr(r9r:r+r+r,rTsz%FlowMatchingAudioToAudioModel.forwardr0r1Tr6r8rArcCr)aForward pass of the model to generate samples from the target distribution. This is used for eval mode only, and it enables SSL masking to the input. Args: input_signal: Tensor that represents a batch of raw audio signals, of shape [B, T] or [B, T, C]. T here represents timesteps, with 1 second of audio represented as `self.sample_rate` number of floating point values. input_signal_length: Vector of length B, that contains the individual lengths of the audio sequences. Returns: Output signal `output` in the time domain and the length of the output signal `output_length`. Trrrr+r+r, forward_evalsz*FlowMatchingAudioToAudioModel.forward_evalFc Cs|d}|jrtj|ddd}|||j}|j||d\}}|jdkr-t|}n|r;|j dur;|j ||d}t ||j j }|j |||d \} } |j| | d\} } |jr^| |} |j| |d } | | fS) aInternal forward pass of the model. Args: input_signal: Tensor that represents a batch of raw audio signals, of shape [B, T] or [B, T, C]. T here represents timesteps, with 1 second of audio represented as `self.sample_rate` number of floating point values. input_signal_length: Vector of length B, that contains the individual lengths of the audio sequences. enable_ssl_masking: Whether to enable SSL masking of the input. If using SSL pretraining, masking is applied to the input signal. If not using SSL pretraining, masking is not applied. Returns: Output signal `output` in the time domain and the length of the output signal `output_length`. rErTrrFrN input_speclength)restimator_conditionrrK)rMrrjrrrrr zeros_likerrr sigma_startrr"rN) r(r9r:rrLrrOrP init_staterrrrCr+r+r,rs$    z.FlowMatchingAudioToAudioModel.forward_internalrrfcCs&|d}|jr tj|ddd}|||j}|||j}|j||d\}}|j||d\}} |jdur>|j||d}t t ||j k d} || |j}t|} |jj|dj |jd } |jj| | |d } tj| |gd d }|j||| d \}}|jj| | || d}|j|||dS)NrrTrrFrz B -> B 1 1 1)r)r)rx_startx_endrrr)rrrpointrX)rMrrjrrrrrrcrdrandrfloattorrrrsamplerr vector_fieldrf)r(rUr9r:rrrrrrQkeep_conditionsrrrrrJ estimate_lenconditional_vector_fieldr+r+r,r"s$   z#FlowMatchingAudioToAudioModel._stepcCst|tr|d}|d}|d|}n|\}}}}|jdkr't|d}|jdkr2t|d}|j|||d}|d||d|j j d d |d t j |j jt jd |Sr)r`rarclonerbrcrdrrgrhrirjrkrr]rlrr+r+r,rpVs       z+FlowMatchingAudioToAudioModel.training_steprrqrrrscCst|tr|d}|d}|d|}n|\}}}}|jdkr't|d}|jdkr2t|d}|j|||d} d} |jdurId} | d } n%t t |j ||} |j ||| j } | |jk} t|j| | d } | r|j|d| d f|d| d \}}t|d r||j vr|j ||D]\}}|j||d| d f|d| d q|dtj|jjtjd|d| iSr)r`rarrrbrcrdrrrMrrrvrrrr$r{r|rgrjrkrr]rlrr+r+r,rosF           z-FlowMatchingAudioToAudioModel.evaluation_stepr)NFr)rrrrrrrrrrr r>rDrrjrrTr r<r rrr rrprrrr+r+r)r,rnsB 5    6    *rcseZdZdZddedeffdd Zedee e ffdd Z edee e ffd d Z e edd d Ze e dee dee edede dee dee dedddddZddZddede fddZZS)ruThis models is using a Schrödinger Bridge process to generate an encoded representation of the enhanced signal. The model consists of the following blocks: - encoder: transforms input audio signal into an encoded representation (analysis transform) - estimator: neural model, estimates the coefficients for the SB process - noise_schedule: defines the path between the clean and noisy signals - sampler: sampler for the reverse process, estimates coefficients of the target signal - decoder: transforms sampler output into the time domain (synthesis transform) References: Schrödinger Bridge for Generative Speech Enhancement, https://arxiv.org/abs/2407.16074 Nrrcstj||d|jj|_||jj|_||jj|_||jj|_|jj|_||jj |_ t j j |jj |j |j|jd|_ |jdd|_|jd|_|jdur]td|jd|jvrs||jj|_|jdd |_nd|_d |_d |jvr||jj|_|jd d |_nd|_d |_|jdur|jdus|jdurtd |jdd|_|td|jjtd|jtd|jtd|jtd|jtd|jtd|jtd|jtd|jdS)Nr)noise_schedulerestimator_outputrFrr loss_encodedloss_encoded_weightr loss_timeloss_time_weightzREither ``loss`` or ``loss_encoded`` and ``loss_time`` should be defined, not both.rrrz estimator_output: %sz normalize_input: %sz loss: %sz loss_encoded: %sz loss_encoded_weight: %sz loss_time: %sz loss_time_weight: %sz eps: %s)rrrrrrr"rrrrrrrrrrr rrrrrrfrrr&r#r*rr'r)r+r,rsV     z,SchroedingerBridgeAudioToAudioModel.__init__r-cCr.r/r;r=r+r+r,r>z/SchroedingerBridgeAudioToAudioModel.input_typescCr.r@r;r=r+r+r,rD rz0SchroedingerBridgeAudioToAudioModel.output_typesc Cr)aOForward pass of the model. Forward pass of the model consists of the following steps - encoder to obtain the encoded representation of the input signal - sampler to generate the estimated coefficients of the target signal - decoder to transform the estimated output into the time domain Args: input_signal: Tensor that represents a batch of time-domain audio signals, of shape [B, C, T]. T here represents timesteps, with 1 second of audio represented as `self.sample_rate` number of floating point values. input_signal_length: Vector of length B, contains the individual lengths of the audio sequences. Returns: Output `output_signal` in the time domain and the length of the output signal `output_length`. rErTrrF)rrrrKrrr+r+r,rTs  z+SchroedingerBridgeAudioToAudioModel.forwardr0r1r)rfrrrcCsh|d}|jr tj|ddd}|||j}|||j}|j||d\}}|j||d\}} |jj||j d} |jj | d\} } } |jj | d\}}}| |d|d|j}| |d|d|j}| d d d d }| d d d d }||||}| ||||j}| d d d d }t |}|||}tj||gd d }|j||| d \}}|jdkr&|jdur|j|||d}d}}ned}|jdur|j|||d}||j|7}nd}|jdur#t|j||d\}} Wdn 1swY|d }|j||d}|j|||d}||j|7}n d}n td|jd|||fS)zRandomly generate time step for each example in the batch, run neural estimator to estimate the target and calculate the loss. rrTrrFr)rrVrErrrrdata_predictionNrXrrKz Output type z is not implemented)rMrrjrrrrrrr get_alphas get_sigmasviewrrrrrfrrrrdisable_checksr"rNrNotImplementedError)r(rUr9r:rrrrrrQ process_timealpha_t alpha_bar_t alpha_t_maxsigma_t sigma_bar_t sigma_t_max weight_target weight_inputmean_xstd_xrx_trrJrrfrrestimate_signalrLr+r+r,r@sZ           z)SchroedingerBridgeAudioToAudioModel._stepc Cst|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}|j|||d\}}} |d||d|jjd d |d t j |j j t j d |durc|d || durm|d| |S)Nr9r:rUrVrWrrZr[rr\r]r^train_loss_encodedtrain_loss_timer) r(rmrnr9r:rUrQrfrrr+r+r,rps(          z1SchroedingerBridgeAudioToAudioModel.training_steprrqrrrscCst|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}|j|||d^} }d} |jdurGd} |d } n%tt |j ||} |j ||| j } | |jk} t |j| |d } | r|j |d| d f|d| d \}}t|d r||j vr|j ||D]\}}|j||d| d f|d| d q|dtj|jjtjd|d| iSrrrr+r+r,rs>            z3SchroedingerBridgeAudioToAudioModel.evaluation_steprrrr+r+r)r,rs.F-      c"r)typingrrrcrrjlightning.pytorchr omegaconfr,nemo.collections.audio.models.audio_to_audiornemo.core.classes.commonrrnemo.core.neural_typesr r r r nemo.utilsr __all__rrrrrr+r+r+r,s,     6@