o wiR@sddlZddlmZmZmZddlZddlZddlZddlm m Z ddl m Z mZddlmZmZddlmZmZmZmZmZddlmZGdddeZGd d d eZGd d d eZGd ddej jZdS)N)DictOptionalSequence)activation_registrymask_sequence_tensor) NeuralModule typecheck) FloatType LengthsType NeuralTypeSpectrogramTypeVoidType)loggingcsveZdZdZddddedeffddZedeee ffd d Z edeee ffd d Z e dddZ ZS)/SpectrogramNoiseConditionalScoreNetworkPlusPlusanThis model handles complex-valued inputs by stacking real and imaginary components. Stacked tensor is processed using NCSN++ and the output is projected to generate real and imaginary components of the output channels. Args: in_channels: number of input complex-valued channels out_channels: number of output complex-valued channels ) in_channels out_channelsrrc st|dkrtd|||_|dkrtd|||_|}d|j|d<|d<td i||_tj j d|jd|jdd|_ t d|jjt d |jt d |jdS) NrzKNumber of input channels needs to be larger or equal to one, current value zLNumber of output channels needs to be larger or equal to one, current value rrrr kernel_sizeInitialized %s withz in_channels: %sz out_channels: %s)super__init__ ValueErrorrrcopy$NoiseConditionalScoreNetworkPlusPlusncsnpptorchnnConv2doutput_projectionrdebug __class____name__)selfrrkwargs ncsnpp_paramsr#rk/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/audio/parts/submodules/ncsnpp.pyr's( z8SpectrogramNoiseConditionalScoreNetworkPlusPlus.__init__returncC,tdttdtddtdtdddS+Returns definitions of module output ports.BCDTr/Toptionalinput input_length condition)r r r r r%rrr) input_typesH z;SpectrogramNoiseConditionalScoreNetworkPlusPlus.input_typescCtdttdtdddSr-r.r3Tr4)output output_length)r r r r:rrr) output_typesQ z B (C RI) F Tr6rr)shaper RuntimeErrorrstackrealimageinops rearrangerr!reshaperpermuteview_as_complex contiguous) r%r7r8r9r/C_inr1r2input_real_imagr?r@rrr)forwardYs   z7SpectrogramNoiseConditionalScoreNetworkPlusPlus.forward)NN)r$ __module__ __qualname____doc__intrpropertyrstrr r;rArrT __classcell__rrr(r)rs !rcseZdZdZ          d+d ed ed edeededededededede ede effdd Z ddZ e de dide diddejdejfd d!Zedeee ffd"d#Zedeee ffd$d%Ze d d&dejd'e ejd(e ejfd)d*ZZS),ra3Implementation of Noise Conditional Score Network (NCSN++) architecture. References: - Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, NeurIPS 2021 - Brock et al., Large scale GAN training for high fidelity natural image synthesis, ICLR 2018 swishrr^r_r_rFh㈵>F0@N nonlinearityrrchannelsnum_res_blocksnum_resolutions init_scaleconditioned_on_timefourier_embedding_scale dropout_rate pad_time_topad_dimension_toc  s`tt||_||_tjjddd|_tjjddd|_ ||_ ||_ ||_ ||_ ||_||_| p8d|j|_| p@d|j|_|jrutjt|j d| dtj|j dd|j dd|jtj|j dd|j dd|_tj|_|j ddD]}|jtjj|j |d d q|j| |j|jr|ddndd }tj|_t|j dd|j d dD]!\}}t|D]}td|dkr|n||d |}|j|qqtj|_tt|j d dt|j ddD]$\}}tt|D]}td||dkr |n|d |}|j|qqtj|_ |j ddD]}|j tjj||d d q't!|j|jksCJt!|j|j|j ksQJt!|j|j|j ks_Jt!|j |jksjJ|"t#$d|j%j&t#$d|j t#$d|j t#$d|j t#$d|j t#$d|jt#$d|jt#$d|jt#$d|jdS)Ng?bilinear) scale_factormoderr)embedding_sizescalerFrr) activationrjrgdiffusion_step_embedding_dim)in_chout_ch)rrz in_channels: %sz out_channels: %sz channels: %sz num_res_blocks: %sz num_resolutions: %sz conditioned_on_time: %sz pad_time_to: %sz pad_dimension_to: %sr)'rrrrsrgrrUpsample downsampleupsamplerrrdrerfrhrkrl SequentialGaussianFourierProjectionLineartime_embedding ModuleList input_pyramidappendr input_blocksziprangeResnetBlockBigGANPlusPlus output_blocksreversedprojection_blockslen init_weights_rr"r#r$)r%rcrrrdrerfrgrhrirjrkrl_ch block_paramsrurvnblockr(rr)r|sr $     &   ." z-NoiseConditionalScoreNetworkPlusPlus.__init__cCs|D]"}t|tjjtjjfr&tjj|j|j dur&tjj |j q|j D] }tjjj|j|j dq*|D]}||urCq)r r r r:rrr)rArBz1NoiseConditionalScoreNetworkPlusPlus.output_types)r9r8r9cCs|jd|jks J|j^}}}|j|d}|dur-t|jdg|jd|j}|}|durmt|jdkrJtdt|jdt |j|jd|jdkretdt |jd t |jd | t |}|g}t |j dD] } |||dqwd d t||jD}g} t|d} t|j} t|D]A\} } | | td } t| |} t |jD]}t| | |} t| |} | | q| |j dk}|s|| } |d}qg}tt| |jD].\}}| j|jkr|| || } |d}| |td } || |} t| |} q|| g}t|t|j D]\}}||} |t!j"| |jdddq+t#|}|jdd|jddksYJ|ddddd|d|f}||fS)a4Forward pass of the model. Args: input: input tensor, shjae (B, C, D, T) input_length: length of the valid time steps for each example in the batch, shape (B,) condition: scalar condition (time) for the model, will be embedded using `self.time_embedding` r)r7Nrrrz.Expected conditon to be a 1-dim tensor, got a z-dim tensor of shape z Condition z and input z' should match along the batch dimensioncSsg|]\}}||qSrr).0imagerrrr) Nsz@NoiseConditionalScoreNetworkPlusPlus.forward..@r)size)$rGrrr LongTensortodevicerrtupler}logrrfrrxrr zeros_likeiterr enumeratemathsqrtrrenextceillongrrryrr interpolatesum)r%r7r8r9rr1r2lengthspyramidresolution_numhistoryhiddenrrfinal_resolution to_projectresidualrimagestensor projectionresultrrr)rT'sj  $           "" z,NoiseConditionalScoreNetworkPlusPlus.forward) r\rrr]rrFr`FrarbNN)r$rUrVrWrZrXrfloatboolrrrrr rTensorrrYrr;rArTr[rrr(r)rtsx      prcsheZdZdZddedeffdd Zedee e ffd d Z edee e ffd d Z d dZ ZS)r{znGaussian Fourier embeddings for input scalars. The input scalars are typically time or noise levels. r_?rprqcs*ttjjt||dd|_dS)NF) requires_grad)rrrr ParameterrandnW)r%rprqr(rr)rs  z"GaussianFourierProjection.__init__r*cCdtdtiS)r-r7r3)r r r:rrr)r; z%GaussianFourierProjection.input_typescCr)r-r?)r/r1)r r r:rrr)rArz&GaussianFourierProjection.output_typescCsJ|dddf|jdddfdtj}tjt|t|gddS)NrrrrC)rrpircatsincos)r%r7x_projrrr)rTs,z!GaussianFourierProjection.forward)r_r)r$rUrVrWrXrrrYrrZr r;rArTr[rrr(r)r{sr{cseZdZdZ      ddejjdeded eed e d e d eed eede ffdd Z ddZ ddej deej fddZ ZS)ra!Implementation of a ResNet block for the BigGAN model. References: - Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, NeurIPS 2021 - Brock et al., Large scale GAN training for high fidelity natural image synthesis, ICLR 2018 Nr`皙?ư>rsrurvrtrgrj in_num_groupsout_num_groupsepsc s t|p t|dd}|pt|dd}||_tjtjj||| d||_tjj ||ddd|_ |durKtj|tj ||t j jd|_tjtjj||| d|tj|tjj ||ddd|_||krvtjj ||dd |_||_||_||_|dS) aN Args: activation (torch.nn.Module): activation layer (ReLU, SiLU, etc) in_ch (int): number of channels in the input image out_ch (int, optional): number of channels in the output image diffusion_step_embedding_dim (int, optional): dimension of diffusion timestep embedding. Defaults to None (no embedding). dropout_rate (float, optional): dropout rate. Defaults to 0.1. init_scale (float, optional): scaling for weight initialization. Defaults to 0.0. in_num_groups (int, optional): num_groups in the first GroupNorm. Defaults to min(in_ch // 4, 32) out_num_groups (int, optional): num_groups in the second GroupNorm. Defaults to min(out_ch // 4, 32) eps (float, optional): eps parameter of GroupNorms. Defaults to 1e-6. rF ) num_groups num_channelsrrEr)rrrpaddingNzbatch dim -> batch dim 1 1r)rrminrgrrrz GroupNorm input_blockr middle_convr|rLlayers Rearrangediffusion_step_projectionDropout output_blockresidual_projectionactrurvr) r%rsrurvrtrgrjrrrr(rr)rs6     z"ResnetBlockBigGANPlusPlus.__init__cCsn|D]"}t|tjjtjjfr&tjj|j|j dur&tjj |j qtjjj|j dj|j ddS)zWeight initializationNrrr) rrrrr r|rrrrrrrgrrrr)rs   z'ResnetBlockBigGANPlusPlus.init_weights_xdiffusion_time_embeddingcCs\||}||}|dur|||}||}|j|jkr%||}||tdS)zForward pass of the model. Args: x: input tensor diffusion_time_embedding: embedding of the diffusion time step Returns: Output tensor Nr)rrrrrGrrr)r%rrhrrr)rTs    z!ResnetBlockBigGANPlusPlus.forward)Nr`rNNr)N)r$rUrVrWrrModulerXrrrrrrTr[rrr(r)rs:   ;$ r)rtypingrrrrLeinops.layers.torchrtorch.nn.functionalr functionalr#nemo.collections.common.parts.utilsrrnemo.core.classesrrnemo.core.neural_typesr r r r r nemo.utilsrrrr{rrrrrr)s  W