o ߥi6@s4ddlZddlZddlmZmZddlZddlmZddlmm Z ddl m Z ddl mZmZddlmZmZddlmZmZddlmZddlmZd Zejeje jd Gd d d eZd dZGdddej Z!Gdddej"Z#GdddZ$Gdddej Z%Gdddej Z&Gdddej Z'dS)N)AnyDict)Models)MODELS TorchModel)MossFormerModuleScaledSinuEmbedding)CumulativeLayerNormGlobalLayerNorm)Tensor)Tasks:0yE>) module_namecsReZdZdZdeffdd ZdedeeeffddZ dd d Z d d Z Z S) MossFormerzqLibrary to support MossFormer speech separation. Args: model_dir (str): the model path. model_dirc stj|g|Ri|t|d|dd|_t|dd|d|d|dd|_t|d|dt|d |d |d |d |d |d|d|d|dd|_|d|_ dS)N kernel_size out_channels)rr in_channelsstridebias)rrrrr num_blocksd_model attn_dropout group_size query_key_dimexpansion_factorcausalnormnum_spks)rr) super__init__EncoderencoderDecoderdecoderMossFormerMaskNet MossFormerMmask_netr)selfrargskwargs __class__a/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/audio/separation/mossformer.pyr!!s0   zMossFormer.__init__inputsreturncs|}|}t|gj}||tjfddtjDdd}|d}|d}||krEt |ddd||f}|S|ddd|ddf}|S)Ncs g|] }|dqS))r% unsqueeze).0ir)sep_hr.r/ ?sz&MossFormer.forward..r2dimrr) r#r(torchstackrcatrangesizeFpad)r)r0mix_west_mask est_sourcet_origint_estr.r6r/forward7s"     zMossFormer.forwardNcCs|s|j}|s td}|jjtjtj|d|ddd|j jtjtj|d|ddd|j jtjtj|d|ddddS)Ncpuz encoder.bin) map_locationT)strictz decoder.binz masknet.bin) rr;devicer#load_state_dictloadospathjoinr%r()r) load_pathrKr.r.r/load_check_pointNs,  zMossFormer.load_check_pointcCst|j|j|jdS)N)r#r%masknet)dictr#r%r()r)r.r.r/as_dict`s zMossFormer.as_dict)NN) __name__ __module__ __qualname____doc__strr!r rrrGrRrU __classcell__r.r.r,r/rs  rcCsL|dkr t||ddS|dkrt|ddS|dkr!tjd|ddSt|S) z5Just a wrapper to select the normalization type. glnT)elementwise_affineclnlnrr eps)r r nn GroupNorm BatchNorm1d)rr:shaper.r.r/ select_normes  rfcsFeZdZdZ   d dededeffdd Zd ejfd d ZZ S)r"abConvolutional Encoder Layer. Args: kernel_size: Length of filters. in_channels: Number of input channels. out_channels: Number of output channels. Examples: >>> x = torch.randn(2, 1000) >>> encoder = Encoder(kernel_size=4, out_channels=64) >>> h = encoder(x) >>> h.shape # torch.Size([2, 64, 499]) @rrrrcs4tt|tj||||dddd|_||_dS)NrgrF)rrrrgroupsr)r r"r!rbConv1dconv1dr)r)rrrr,r.r/r!s zEncoder.__init__xcCs0|jdkr tj|dd}||}t|}|S)ayReturn the encoded output. Args: x: Input tensor with dimensionality [B, L]. Returns: Encoded tensor with dimensionality [B, N, T_out]. where B = Batchsize L = Number of timepoints N = Number of filters T_out = Number of timepoints at the output of the encoder rr9)rr;r3rkr@relur)rlr.r.r/rGs   zEncoder.forward)rgrhr) rVrWrXrYintr!r;r rGr[r.r.r,r/r"ssr"cs,eZdZdZfddZfddZZS)r$aA decoder layer that consists of ConvTranspose1d. Args: kernel_size: Length of filters. in_channels: Number of input channels. out_channels: Number of output channels. Example --------- >>> x = torch.randn(2, 100, 1000) >>> decoder = Decoder(kernel_size=4, in_channels=100, out_channels=1) >>> h = decoder(x) >>> h.shape torch.Size([2, 1003]) cstt|j|i|dSN)r r$r!)r)r*r+r,r.r/r!szDecoder.__init__csr|dvrtd|jt|dkr|nt|d}t|dkr2tj|dd}|St|}|S)zReturn the decoded output. Args: x: Input tensor with dimensionality [B, N, L]. where, B = Batchsize, N = number of filters L = time points )rgz{} accept 3/4D tensor as inputrqrr9) r: RuntimeErrorformatrVr rGr;r3squeezernr,r.r/rGs $ zDecoder.forward)rVrWrXrYr!rGr[r.r.r,r/r$s r$c@s eZdZdZddZddZdS) IdentityBlockzThis block is used when we want to have identity transformation within the Dual_path block. Example ------- >>> x = torch.randn(10, 100) >>> IB = IdentityBlock() >>> xhat = IB(x) cKsdSrpr.)r)r+r.r.r/_init__zIdentityBlock._init__cCs|Srpr.rnr.r.r/__call__rwzIdentityBlock.__call__N)rVrWrXrYrvrxr.r.r.r/rus rucs>eZdZdZ      d fdd Zd ejfd d ZZS)r'aThis class implements the transformer encoder. Args: num_blocks : int Number of mossformer blocks to include. d_model : int The dimension of the input embedding. attn_dropout : float Dropout for the self-attention (Optional). group_size: int the chunk size query_key_dim: int the attention vector dimension expansion_factor: int the expansion factor for the linear projection in conv module causal: bool true for causal / false for non causal Example ------- >>> import torch >>> x = torch.rand((8, 60, 512)) #B, S, N >>> net = MossFormerM(num_blocks=8, d_model=512) >>> output, _ = net(x) >>> output.shape torch.Size([8, 60, 512]) N皙?@Fc sBtt|||||||d|_ddl}|jjj|dd|_dS)N)r:depthrrrrrrgư>r`) r r!r mossformerM speechbrainnnet normalization LayerNormr) r)rrrrrrrsbr,r.r/r!s zMossFormerM.__init__srccCs||}||}|S)z Args: src: Tensor shape [B, S, N], where, B = Batchsize, S = time points N = number of filters The sequence to the encoder layer (required). )r~r)r)routputr.r.r/rGs zMossFormerM.forward)Nryrzr{r|F rVrWrXrYr!r;r rGr[r.r.r,r/r'sr'cs6eZdZdZ  d fdd ZdejfddZZS) ComputeAttentionaRComputation block for dual-path processing. Args: att_mdl : torch.nn.module Model to process within the chunks. out_channels : int Dimensionality of attention model. norm : str Normalization type. skip_connection : bool Skip connection around the attention module. Example --------- >>> att_block = MossFormerM(num_blocks=8, d_model=512) >>> comp_att = ComputeAttention(att_block, 512) >>> x = torch.randn(10, 64, 512) >>> x = comp_att(x) >>> x.shape torch.Size([10, 64, 512]) r_Tcs>tt|||_||_||_|durt||d|_dSdS)Nrq)r rr!att_mdlskip_connectionrrfatt_norm)r)rrrrr,r.r/r!<szComputeAttention.__init__rlcCsX|ddd}||}|ddd}|jdur!||}|jr(||}|}|S)aReturns the output tensor. Args: x: Input tensor of dimension [B, S, N]. Returns: out: Output tensor of dimension [B, S, N]. where, B = Batchsize, N = number of filters S = time points rrgrN)permute contiguousrrrr)r)rlatt_outoutr.r.r/rGMs   zComputeAttention.forward)r_Trr.r.r,r/r%s rcs:eZdZdZ    d fdd Zdejfdd ZZS) r&aThe dual path model which is the basis for dualpathrnn, sepformer, dptnet. Args: in_channels : int Number of channels at the output of the encoder. out_channels : int Number of channels that would be inputted to the intra and inter blocks. att_model : torch.nn.module Attention model to process the input sequence. norm : str Normalization type. num_spks : int Number of sources (speakers). skip_connection : bool Skip connection around attention module. use_global_pos_enc : bool Global positional encodings. Example --------- >>> mossformer_block = MossFormerM(num_blocks=8, d_model=512) >>> mossformer_masknet = MossFormerMaskNet(64, 64, att_model, num_spks=2) >>> x = torch.randn(10, 64, 2000) >>> x = mossformer_masknet(x) >>> x.shape torch.Size([2, 10, 64, 2000]) r_rgTcstt|||_t||d|_tj||ddd|_||_ |j r&t ||_ t t||||d|_tj|||dd|_tj||ddd|_t|_t|_tt||dt|_tt||dt|_dS)NrqrF)r)r)r)r r&r!rrfrrbrjconv1d_encoderuse_global_pos_encrpos_enccopydeepcopyrmdl conv1d_out conv1_decoderPReLUpreluReLU activation SequentialTanhrSigmoid output_gate)r)rr att_modelrrrrr,r.r/r!s>      zMossFormerMaskNet.__init__rlc Cs||}||}|jr$|}|dd}||}|dd}||}||}||}||}|j\}}}| ||j d|}| || |}| |}|j\}}}| ||j ||}||}|dd}|S)aQReturns the output tensor. Args: x: Input tensor of dimension [B, N, S]. Returns: out: Output tensor of dimension [spks, B, N, S] where, spks = Number of speakers B = Batchsize, N = number of filters S = the number of time frames rr2r)rrr transposerrrrreviewrrrrr) r)rlbaseembb_snLr.r.r/rGs(             zMossFormerMaskNet.forward)r_rgTTrr.r.r,r/r&ks!(r&)(rrNtypingrrr;torch.nnrbtorch.nn.functional functionalr@modelscope.metainformodelscope.modelsrr3modelscope.models.audio.separation.mossformer_blockrr9modelscope.models.audio.separation.mossformer_conv_moduler r modelscope.models.baser modelscope.utils.constantr EPSregister_modulespeech_separation(speech_mossformer_separation_temporal_8krrfModuler"ConvTranspose1dr$rur'rr&r.r.r.r/s2    K6*AF