o ̳i&@sFddlmZmZddlZddlmmZddlmZmZddl m Z Gdddej Z Gdddej Z Gd d d ej Zd ed ej fd dZdeded ej fddZGdddej ZGdddej Zdededed ej fddZGdddej ZGdddej ZGdddej ZGd d!d!ej Zd"ed efd#d$ZdS)%)ListTupleN)nnTensor)MultiHeadAttentioncsreZdZdZdeeeefdejdejffdd Zde de fd d Z de de fd d Z d e de fddZ Z S)FluxAutoencodera  The image autoencoder for Flux diffusion models. Args: img_shape (Tuple[int, int, int]): The shape of the input image (without the batch dimension). encoder (nn.Module): The encoder module. decoder (nn.Module): The decoder module. img_shapeencoderdecodercs t||_||_||_dSN)super__init__ _img_shaper r )selfrr r  __class__V/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/flux/_autoencoder.pyr s  zFluxAutoencoder.__init__xreturncC|||S)z Args: x (Tensor): input image of shape [bsz, ch_in, img resolution, img resolution] Returns: Tensor: output image of the same shape )decodeencoderrrrrforward&szFluxAutoencoder.forwardcCs"|jdd|jks J||S)a  Encode images into their latent representations. Args: x (Tensor): input images (shape = [bsz, ch_in, img resolution, img resolution]) Returns: Tensor: latent encodings (shape = [bsz, ch_z, latent resolution, latent resolution]) N)shaperr rrrrr0s zFluxAutoencoder.encodezcCs ||S)a Decode latent representations into images. Args: z (Tensor): latent encodings (shape = [bsz, ch_z, latent resolution, latent resolution]) Returns: Tensor: output images (shape = [bsz, ch_in, img resolution, img resolution]) )r )rrrrrr=s zFluxAutoencoder.decode)__name__ __module__ __qualname____doc__rintrModuler rrrr __classcell__rrrrrs    rc PeZdZdZdededeedededef fdd Zd ed efd d Z Z S) FluxEncodera  The encoder half of the Flux diffusion model's image autoencoder. Args: ch_in (int): The number of channels of the input image. ch_z (int): The number of latent channels (dimension of the latent vector `z`). channels (List[int]): The number of output channels for each downsample block. n_layers_per_down_block (int): Number of resnet layers per upsample block. scale_factor (float): Constant for scaling `z`. shift_factor (float): Constant for shifting `z`. ch_inch_zchannelsn_layers_per_down_block scale_factor shift_factorcszt||_||_tj|ddddd|_tfddtt D|_ t d|_ t dd||_dS) Nrr kernel_sizestridepaddingc Dg|]}t|dkr|dnd||tdkdqS)rr)n_layersr'ch_out downsample) DownBlocklen.0ir)r*rr gz(FluxEncoder.__init__..)r r r+r,rConv2dconv_in ModuleListranger7down mid_blockmid end_blockend)rr'r(r)r*r+r,rr;rr Ws    zFluxEncoder.__init__rrcCsJ||}|jD]}||}q||}||}t|}|j||jS)z Args: x (Tensor): input images (shape = [bsz, ch_in, img resolution, img resolution]) Returns: Tensor: latent encodings (shape = [bsz, ch_z, latent resolution, latent resolution]) )rArDrFrHdiagonal_gaussianr+r,)rrhblockrrrrrvs     zFluxEncoder.forward rrr r!r"rfloatr rrr$rrrrr&J  r&c r%) FluxDecodera The encoder half of the Flux diffusion model's image autoencoder. Args: ch_out (int): The number of channels of the output image. ch_z (int): The number of latent channels (dimension of the latent vector `z`). channels (List[int]): The number of output channels for each upsample block. n_layers_per_up_block (int): Number of resnet layers per upsample block. scale_factor (float): Constant for scaling `z`. shift_factor (float): Constant for shifting `z`. r4r(r)n_layers_per_up_blockr+r,csvt||_||_tj|ddddd|_td|_t fddt t D|_ t d||_dS)Nrr-rr.c r2)rr)r3r'r4upsample)UpBlockr7r8r)rPrrr<r=z(FluxDecoder.__init__..r>)r r r+r,rr@rArErFrBrCr7uprGrH)rr4r(r)rPr+r,rrSrr s    zFluxDecoder.__init__rrcCsF||j|j}||}||}|jD]}||}q||}|S)z Args: z (Tensor): latent encodings (shape = [bsz, ch_z, latent resolution, latent resolution]) Returns: Tensor: output images (shape = [bsz, ch_in, img resolution, img resolution]) )r+r,rArFrTrH)rrrJrKrrrrrs     zFluxDecoder.forwardrLrrrrrOrNrOchrcCs"tt||dt|t||dS)Nr'r4)r Sequential ResnetLayer AttnLayer)rUrrrrEs   rEr'r4c Cs0ttjd|dddttj||ddddS)N ư>T num_groups num_channelsepsaffiner-rr.)rrW GroupNormSiLUr@rVrrrrGs rGc@eZdZdedededeffdd Zdedefd d ZZS) r6r3r'r4r5c8tt||||_|rt||_dSt|_dSr )r r resnet_layerslayers DownsamplerIdentityr5)rr3r'r4r5rrrr   zDownBlock.__init__rrcCrr )r5rfrrrrrzDownBlock.forward rrr r"boolr rrr$rrrrr6r6crc) rRr3r'r4rQcrdr )r r rerfUpsamplerrhrQ)rr3r'r4rQrrrr rizUpBlock.__init__rrcCrr )rQrfrrrrrrjzUpBlock.forwardrkrrrrrRrmrRncstjfddt|DS)Ncs$g|]}t|dkr ndqS)rrV)rXr8rVrrr<sz!resnet_layers..)rrWrC)ror'r4rrVrres  rec4eZdZdeffdd ZdedefddZZS)rYdimc sdt||_tjd|ddd|_t|dd|t||t||t||t||dd |_dS)NrZr[Tr\rF) embed_dim num_heads num_kv_headshead_dimq_projk_projv_proj output_proj is_causal) r r rqrranormrLinearattn)rrqrrrr s      zAttnLayer.__init__rrcCsj|j\}}}}|}||}td|}|||||}|||}|||||}td|}||S)Nz bchw -> bhwcz bhwc -> bchw)rr{torcheinsumreshaper})rrbcrJwresidualrrrrs    zAttnLayer.forwardrrr r"r rrr$rrrrrYsrYcs8eZdZdedeffdd ZdedefddZZS) rXr'r4c sttjtjd|dddttj||ddddtjd|dddttj||ddddg|_||kr>t|_ dStj||dddd|_ dS) NrZr[Tr\r-rr.r) r r rrWrarbr@mainrhshortcut)rr'r4rrrr s  zResnetLayer.__init__rrcCs||||Sr )rrrrrrr(szResnetLayer.forwardrrrrrrXsrXcrp)rgrUcs$ttj||dddd|_dS)Nr-r?rr.r r rr@convrrUrrrr - zDownsample.__init__rrcCs|tj|ddddS)N)rrrrconstantr)modevalue)rFpadrrrrr1szDownsample.forwardrrrrrrg,rgcrp)rnrUcs$ttj||dddd|_dS)Nr-rr.rrrrrr 6rzUpsample.__init__rrcCs|tj|dddS)Ng@nearest)r+r)rr interpolaterrrrr:szUpsample.forwardrrrrrrn5rrnrcCs4tj|ddd\}}td|}||t|S)Nr?r)rqg?)r~chunkexp randn_like)rmeanlogvarstdrrrrI>srI)typingrrr~torch.nn.functionalr functionalrrtorchtune.modules.attentionrr#rr&rOr"rErGr6rRrerYrXrgrnrIrrrrs$ 9==   $