o ߥi"@sddlZddlmZddlZddlZddlmZddl mm Z ddl m Z mZGdddejZd/ddZd d Zd d Zd dZddZGdddejZGdddejZGdddejeZGdddejZGdddejZGdddeZGdddejZdd ZGd!d"d"ejZGd#d$d$ejZ Gd%d&d&ejZ!Gd'd(d(e!Z"Gd)d*d*ejZ#Gd+d,d,e Z$Gd-d.d.eZ%dS)0N)abstractmethod)PretrainedConfigPreTrainedModelcseZdZfddZZS) GroupNormcstt|||jSN)superrforwardfloattypedtypeselfx __class__g/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/multi_modal/guided_diffusion/unet.pyrszGroupNorm.forward)__name__ __module__ __qualname__r __classcell__rrrrrsr'cCs|d}tt| tjd|tjd|j|jd}|dddf|d}tj t |t |gdd}|drRtj |t |ddddfgdd}|S) aY Create sinusoidal timestep embeddings. :param timesteps: a 1-D Tensor of N indices, one per batch element. These may be fractional. :param dim: the dimension of the output. :param max_period: controls the minimum frequency of the embeddings. :return: an [N x dim] Tensor of positional embeddings. r)startendr )deviceNdim) thexpmathlogarangefloat32torr catcossin zeros_like) timestepsr max_periodhalffreqsargs embeddingrrrtimestep_embeddings r1cCLt|tjtjtjfr"|jj|j_|jdur$|jj|j_dSdSdS)z/ Convert primitive modules to float16. N) isinstancennConv1dConv2dConv3dweightdatar-biasllrrrconvert_module_to_f16+  r=cCr2)zP Convert primitive modules to float32, undoing convert_module_to_f16(). N) r3r4r5r6r7r8r9r r:r;rrrconvert_module_to_f325r>r?cOsV|dkr tj|i|S|dkrtj|i|S|dkr$tj|i|Std|)z4 Create a 1D, 2D, or 3D convolution module. rrzunsupported dimensions: )r4r5r6r7 ValueError)dimsr/kwargsrrrconv_nd?srDcCs4|rt|t|}tj|t|g|RS||S)a Evaluate a function without caching intermediate activations, allowing for reduced memory at the expense of extra compute in the backward pass. :param func: the function to evaluate. :param inputs: the argument sequence to pass to `func`. :param params: a sequence of parameters `func` depends on but does not explicitly take as arguments. :param flag: if False, disable gradient checkpointing. )tupleCheckpointFunctionapplylen)funcinputsparamsflagr/rrr checkpointLs rMc s>eZdZdZ d dedededeffdd Zd d ZZS) AttentionPool2dzS Adapted from CLIP: https://github.com/openai/CLIP/blob/main/clip/model.py N spacial_dim embed_dimnum_heads_channels output_dimcspttt||dd|d|_td|d|d|_td||p'|d|_ |||_ t |j |_ dS)Nrr?r@) r__init__r4 Parameterr randnpositional_embeddingrDqkv_projc_proj num_heads QKVAttention attention)r rOrPrQrRrrrrTbs  zAttentionPool2d.__init__cCs|j^}}}|||d}tj|jddd|gdd}||jdddddf|j}||}| |}| |}|dddddfS)NrT)rkeepdimrr) shapereshaper r'meanrWr&r rXr\rY)r rbc_spatialrrrrqs $   zAttentionPool2d.forwardr)rrr__doc__intrTrrrrrrrN]s rNc@seZdZdZeddZdS) TimestepBlockzT Any module where forward() takes timestep embeddings as a second argument. cCsdS)zJ Apply the module to `x` given `emb` timestep embeddings. Nrr rembrrrrszTimestepBlock.forwardN)rrrrdrrrrrrrf|srfc@seZdZdZddZdS)TimestepEmbedSequentialzt A sequential module that passes timestep embeddings to the children that support it as an extra input. cCs,|D]}t|tr|||}q||}q|Sr)r3rf)r rrhlayerrrrrs    zTimestepEmbedSequential.forwardN)rrrrdrrrrrris ric*eZdZdZdfdd ZddZZS) UpsampleaB An upsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determining if a convolution is applied. :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then upsampling occurs in the inner-two dimensions. rNcsJt||_|p ||_||_||_|r#t||j|jddd|_dSdS)Nr@rpadding)rrTchannels out_channelsuse_convrBrDconv)r rorqrBrprrrrTs   zUpsample.__init__cCst|jd|jks J|jdkr(tj||jd|jdd|jddfdd}ntj|ddd}|jr8||}|S)Nrr@rnearestmode) scale_factorrv)r^rorBF interpolaterqrrr rrrrs $ zUpsample.forwardrNrrrrdrTrrrrrrrls  rlcrk) DownsampleaE A downsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determining if a convolution is applied. :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then downsampling occurs in the inner-two dimensions. rNcs|t||_|p ||_||_||_|dkrdnd}|r,t||j|jd|dd|_dS|j|jks4Jtj ||d|_dS)Nr@r)rrrr)stridern) kernel_sizer}) rrTrorprqrBrDopr4 AvgPool2d)r rorqrBrpr}rrrrTs"   zDownsample.__init__cCs|jd|jks J||S)Nr)r^rorr rrrrs zDownsample.forwardrzr{rrrrr|s r|cs@eZdZdZ       d fdd ZddZd d ZZS) ResBlocka A residual block that can optionally change the number of channels. :param channels: the number of input channels. :param emb_channels: the number of timestep embedding channels. :param dropout: the rate of dropout. :param out_channels: if specified, the number of out channels. :param use_conv: if True and out_channels is specified, use a spatial convolution instead of a smaller 1x1 convolution to change the channels in the skip connection. :param dims: determines if the signal is 1D, 2D, or 3D. :param use_checkpoint: if True, use gradient checkpointing on this module. :param up: if True, use this block for upsampling. :param down: if True, use this block for downsampling. NFrc st||_||_||_|p||_||_||_||_t t d|t t |||jddd|_| p4| |_| rGt|d||_t|d||_n| rXt|d||_t|d||_nt |_|_t t t ||rod|jn|j|_t t d|jt t j|dt ||j|jddd|_t j|jdj|j|krt |_dS|rt |||jddd|_dSt |||jd|_dS) N r@rrmFr)pr)rrTro emb_channelsdropoutrprquse_checkpointuse_scale_shift_normr4 SequentialrSiLUrD in_layersupdownrlh_updx_updr|IdentityLinear emb_layersDropout out_layersinitzeros_r8skip_connection) r rorrrprqrrBrupdownrrrrTs\          zResBlock.__init__cCst|j||f||jS)a  Apply the block to a Tensor, conditioned on a timestep embedding. :param x: an [N x C x ...] Tensor of features. :param emb: an [N x emb_channels] Tensor of timestep embeddings. :return: an [N x C x ...] Tensor of outputs. rM_forward parametersrrgrrrr,szResBlock.forwardc Cs|jr#|jdd|jd}}||}||}||}||}n||}|||j}t|jt|jkrI|d}t|jt|jks;|j rr|j d|j dd}}t j |ddd\} } ||d| | }||}n ||}| |}| ||S)Nr).Nrrrr)rrrrrr r rHr^rrr chunkr) r rrhin_restin_convhemb_outout_normout_restscaleshiftrrrr7s&      zResBlock._forward)NFFrFFFrrrrdrTrrrrrrrrsA rcs:eZdZdZ    d fdd ZddZd d ZZS) AttentionBlocka An attention block that allows spatial positions to attend to each other. Originally ported from here, but adapted to the N-d case. https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/models/unet.py#L66. rrFcst||_|dkr||_n||dks Jd|d||||_||_td||_td||dd|_|rAt |j|_ nt |j|_ td||d|_ t j|j jdS)Nrrzq,k,v channels z' is not divisible by num_head_channels rrr@)rrTrorZrrnormrDqkvr[r\QKVAttentionLegacyproj_outr4rrr8)r rorZnum_head_channelsruse_new_attention_orderrrrrTVs    zAttentionBlock.__init__cCst|j|f||jSrrr rrrruszAttentionBlock.forwardcCsV|j^}}}|||d}|||}||}||}||j||g|RS)Nr)r^r_rrr\r)r rrarbspatialrrrrrrys   zAttentionBlock._forward)rrFFrrrrrrNs rcCsL|dj^}}}tt|}d||d|}|jt|g7_dS)a( A counter for the `thop` package to count the operations in an attention operation. Meant to be used like: macs, params = thop.profile( model, inputs=(inputs, timestamps), custom_ops={QKVAttention: QKVAttention.count_flops}, ) rrN)r^renpprod total_opsr DoubleTensor)model_xyrarbr num_spatial matmul_opsrrrcount_flops_attns rc4eZdZdZfddZddZeddZZS)rzh A module which performs QKV attention. Matches legacy QKVAttention + input/ouput heads shaping ct||_dSrrrTn_headsr rrrrrT  zQKVAttentionLegacy.__init__c Cs|j\}}}|d|jdksJ|d|j}|||j|d|j|dd\}}}dtt|} td|| || } tj| dd | j } td| |} | |d|S)z Apply QKV attention. :param qkv: an [N x (H * 3 * C) x T] tensor of Qs, Ks, and Vs. :return: an [N x (H * C) x T] tensor after attention. r@rrr bct,bcs->btsr bts,bcs->bct) r^rr_splitr"sqrtr einsumsoftmaxr r r r rbswidthlengthchqkvrr8arrrrs  zQKVAttentionLegacy.forwardcC t|||Srrrrrrrr count_flops zQKVAttentionLegacy.count_flops rrrrdrTr staticmethodrrrrrrrs  rcr)r[zP A module which performs QKV attention and splits in a different order. crrrrrrrrTrzQKVAttention.__init__c Cs|j\}}}|d|jdksJ|d|j}|jddd\}}}dtt|} td|| ||j|||| ||j||} tj| dd | j } td| | ||j||} | |d|S)z Apply QKV attention. :param qkv: an [N x (3 * H * C) x T] tensor of Qs, Ks, and Vs. :return: an [N x (H * C) x T] tensor after attention. r@rrrrrr) r^rrr"rr rviewrr r r r_rrrrrs zQKVAttention.forwardcCrrrrrrrrrzQKVAttention.count_flopsrrrrrr[s  r[csVeZdZdZ           dfd d Zd d ZddZdddZZS) UNetModela The full UNet model with attention and timestep embedding. :param in_channels: channels in the input Tensor. :param model_channels: base channel count for the model. :param out_channels: channels in the output Tensor. :param num_res_blocks: number of residual blocks per downsample. :param attention_resolutions: a collection of downsample rates at which attention will take place. May be a set, list, or tuple. For example, if this contains 4, then at 4x downsampling, attention will be used. :param dropout: the dropout probability. :param channel_mult: channel multiplier for each level of the UNet. :param conv_resample: if True, use learned convolutions for upsampling and downsampling. :param dims: determines if the signal is 1D, 2D, or 3D. :param num_classes: if specified (as an int), then this model will be class-conditional with `num_classes` classes. :param use_checkpoint: use gradient checkpointing to reduce memory usage. :param num_heads: the number of attention heads in each attention layer. :param num_head_channels: if specified, ignore num_heads and instead use a fixed channel width per attention head. :param num_heads_upsample: works with num_heads to set a different number of heads for upsampling. Deprecated. :param use_scale_shift_norm: use a FiLM-like conditioning mechanism. :param resblock_updown: use residual blocks for up/downsampling. :param use_new_attention_order: use a different attention pattern for potentially increased efficiency. rrrrsTrNFrrc st|dkr |}||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | r1t jnt j|_||_||_||_|d}tt||tt|||_|j dur`t| ||_t|d|}}ttt| ||dddg|_||_ |g}d}t!|D]~\}}t"|D]<}t#|||t||| | |dg}t||}||vr|$t%|| |||d|j$t||j |7_ |$|q|t&|dkr|}|j$t|rt#||||| | |d d nt'|| | |d |}|$||d 9}|j |7_ qtt#|||| | |d t%|| |||dt#|||| | |d |_(|j |7_ tg|_)t*t!|dddD]s\}}t"|dD]g}|+}t#||||t||| | |dg}t||}||vrt|$t%|| |||d|r||kr|}|$|rt#||||| | |d dnt,|| | |d |d }|j)$t||j |7_ qEq;tt-d|tt| ||ddd|_.tj/0|j.dj1dS)Nrrsrr@rrmrprBrrrrZrrTrprBrrrrBrprrBrr)rprBrrrr)2rrT image_size in_channelsmodel_channelsrpnum_res_blocksattention_resolutionsr channel_mult conv_resample num_classesrr float16r%r rZrnum_heads_upsampler4rrr time_embed Embedding label_embre ModuleListrirD input_blocks _feature_size enumeraterangerappendrrHr| middle_block output_blockslistpoprlroutrrr8) r rrrrprrrrrrBrruse_fp16rZrrrresblock_updownrtime_embed_dimrinput_chinput_block_chansdslevelmult_layersout_chiichrrrrTsJ                 )zUNetModel.__init__cC(|jt|jt|jtdSz< Convert the torso of the model to float16. N)rrGr=rrr rrrconvert_to_fp16  zUNetModel.convert_to_fp16cCrz< Convert the torso of the model to float32. N)rrGr?rrrrrrconvert_to_fp32rzUNetModel.convert_to_fp32cCs|du|jduks Jdg}|t||j}|jdur/|j|jdfks(J|||}||j}|jD] }|||}| |q8| ||}|j D]}t j ||gdd}|||}qN||j}||S)a Apply the model to an input batch. :param x: an [N x C x ...] Tensor of inputs. :param timesteps: a 1-D batch of timesteps. :param y: an [N] Tensor of labels, if class-conditional. :return: an [N x C x ...] Tensor of outputs. Nzs>       !"t4"$ T)