o wiW*@sdZddlmZmZddlZddlmZddlZddl m Z ddl m m Z ddlmZddlmZddlmZddlmZmZddlmZmZdd lmZmZd Zd Zeed ZGd ddejZ GdddeZ!dS)a Implementation of Maxine BNR2 denoising network Maxine Background Noise Removal (BNR) 2.0 is an audio background noise removal model from NVIDIA. This is the second generation of BNR from Maxine Audio Effects SDK. BNR 2.0 removes unwanted noises from audio improving speech intelligibility and also improving the speech recognition accuracy of various ASR systems under noisy environments. BNR 2.0 uses the SEASR architecture described in https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10837982 )DictOptionalN)Trainer) DictConfig)AudioToAudioModel)apply_weight_norm_lstmremove_weight_norm_lstm)PretrainedModelInfo typecheck) AudioSignal NeuralTypei> ics<eZdZdZ dfd d Zd d Zd dZddZZS)_Seasrz*Internal implementation of the model classF@?c s|tkrtdt||_||_|d|_|j|_|r dnd} tj d|j||dd|_ tj |jdd |_ tj |j|jdd d |_tj |j|jd | d |_tj |jdd |_tj |j|jdd d |_tj |j|jd | d |_tj |jdd |_tj |j|jdd d |_tj |j|jd | d |_tj |jdd |_tj d |j|j|jd |d|_tj |j|jdd |d|_tj |j|jdd |d|_tj |j|jd d|_t|j|j|_t|_tj|jd||d|_ t!|_"dS)NzCurrently only 16k is supportedrsameF) kernel_sizestridebiasgMbP?)epsT) num_layers batch_first)rpadding)rdropout)rrr )r)rr)#SUPPORTED_SAMPLE_RATEAssertionErrorsuper__init__f1f2f3 gru_nodesnnConv1dconv1d BatchNorm1dbn0GRU feature_gru0 conv1d_out1bn1 feature_gru1 conv1d_out2bn2 feature_gru2 conv1d_out3bn3 denoise_gru denoise_gru_1 denoise_gru_2 denoise_gru_3Linear denoise_maskSigmoidmask_actConvTranspose1dinv_convTanhinv_conv_activation) self sample_rate hidden_nodes streamingrr%r&rr r __class__e/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/audio/models/maxine/bnr.pyr$2s6   " z_Seasr.__init__cKsn|d}t||}||}t||}||}|| ddd\}}t| |}| |}| | ddd\}}t| |} || } | ddd} | ddd}||\} }t| ||| fd} || \} }| |} || \} }| | } || \} }| | } || \} }||| }| ddd}||} || } || } | S)Nx0rrr)getFrelur+r- leaky_relur0r1r2permuter3r4r5r6r7r/torchcatr8r9r:r;r?r=rArC)rDkwargsrLxc0xc1fg1_xc2fg2xc3fg0xixi1xi2maskrJrJrKforwardZs8       z_Seasr.forwardcCdd}||dS)z2Apply weight normalization module from all layers.cSsNt|tjjtjjfrtjj|dSt|tjjtjjfr%t |dSdSN) isinstancerRr)r*r<utils weight_normLSTMr.rmrJrJrK_apply_weight_norms  z4_Seasr.apply_weight_norm.._apply_weight_normNapply)rDrjrJrJrKapply_weight_normsz_Seasr.apply_weight_normcCrb)z3Remove weight normalization module from all layers.cSsjz*t|tjjtjjfrtjj|WdSt|tjjtjjfr(t |WdSWdSt y4YdSwrc) rdrRr)r*r<reremove_weight_normrgr.r ValueErrorrhrJrJrK_remove_weight_norms z6_Seasr.remove_weight_norm.._remove_weight_normNrk)rDrprJrJrKrns z_Seasr.remove_weight_norm)rFrrrrr) __name__ __module__ __qualname____doc__r$rarmrn __classcell__rJrJrHrKr/s(( rcseZdZdZddedeffdd Zedee e ffdd Z edee e ffd d Z e d d ZddZddede fddZedeefddZZS)BNR2z!Implementation of the BNR 2 modelNcfgtrainercsd|_|dur |j|_tj||d|jj|_|t|j|_t|jdr;t|jj dr=|jj j r?|j dSdSdSdS)Nr)rwrxtrainenable_weight_norm) world_sizer#r$_cfgrEsetup_optimization_flagsrseasrhasattrryrzrm)rDrwrxrHrJrKr$s     z BNR2.__init__returncCdtdt|jdiS)N input_signalBCTfreqr r rErDrJrJrK input_types zBNR2.input_typescCr)N output_signalrrrrrJrJrK output_typesrzBNR2.output_typescCsj|jdkr|jddkrtdn |jdkrtd|j|jdtdkr.tdt|jj|d S) a Forward pass of the model. Args: input_signal: Tensor that represents a batch of raw audio signals, of shape [B, T] or [B, T, C]. T here represents timesteps, with 1 second of audio represented as `self.sample_rate` number of floating point values. Returns: Output signal `output` in the time domain and the length of the output signal `output_length`. rrzBThis network currently only supports single channel audio signals.rzKInvalid shape for input signal (received {}, supported [B, 1, T] or [B, T])rz&Input samples must be a multiple of {})rL)ndimshaperoformatSUPPORTED_INPUT_ALIGN_SAMPLESr~ra)rDrrJrJrKras  z BNR2.forwardc Cst|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}|j|d}|j|||d}|d||d |jj d d |d t j |j j t jd |S)Nr input_length target_signalr B T -> B 1 Trtargetestimater train_loss learning_raterlr global_stepdtype)rddictreinops rearrangeralosslog _optimizer param_groupsrRtensorrxrfloat32) rDbatch batch_idxrrrrXpredicted_audiorrJrJrK training_steps         zBNR2.training_steprvaldataloader_idxtagc Cst|tr|d}|d}|d}n|\}}}}|jdkr#t|d}|jdkr.t|d}||d} |j|| |d} t|dr[||jvr[|j||D] \} } | j | ||d qN| d t j |j jt jd |d | iS) Nrrrrrrrmetrics)predsrrrr_loss)rdrrrrrrritemsupdaterrRrrxrr) rDrrrrrrrrXprocessed_signalrnamemetricrJrJrKevaluation_steps        zBNR2.evaluation_stepcCsdS)z This method returns a list of pre-trained model which can be instantiated directly from NVIDIA's NGC cloud. Returns: List of available pre-trained models. NrJ)clsrJrJrKlist_available_modelss zBNR2.list_available_modelsrc)rr)rqrrrsrtrrr$propertyrstrr rrr rarintr classmethodrr rrurJrJrHrKrvs rv)"rttypingrrrlightning.pytorchpytorchpltrRtorch.nnr)torch.nn.functional functionalrNr omegaconfr,nemo.collections.audio.models.audio_to_audior)nemo.collections.audio.parts.utils.maxinerrnemo.core.classes.commonr r nemo.core.neural_typesr r r!SUPPORTED_INPUT_ALIGN_MSrLightningModulerrvrJrJrJrKs$       m