o Gi_@sddlZddlmZddlZddlmZddlmZmZddl m Z ddl m Z e eZGdd d e ZGd d d e eZGd d d ejZGdddejZGdddejZGdddejZGdddejZGdddejZGdddejZdS)N)Callable) ConfigMixinregister_to_config)logging) ModelMixinc seZdZdZdedffdd ZddejdeedBd eejfd d Z  dd e e j Bde dede de dBf ddZede e j BdBfddZZS) MultiAdaptera MultiAdapter is a wrapper model that contains multiple adapter models and merges their outputs according to user-assigned weighting. This model inherits from [`ModelMixin`]. Check the superclass documentation for common methods such as downloading or saving. Args: adapters (`list[T2IAdapter]`, *optional*, defaults to None): A list of `T2IAdapter` model instances. adapters T2IAdaptercstt|t||_t||_t|dkrtdt|dkr&td|dj }|dj }t dt|D],}||j |ksG||j |krctd|d|d|d||j d|d ||j q7||_ ||_ dS) NrzExpecting at least one adapterrzQFor a single adapter, please use the `T2IAdapter` class instead of `MultiAdapter`zjExpecting all adapters to have the same downscaling behavior, but got: adapters[0].total_downscale_factor=z adapters[0].downscale_factor=z adapter[`z`].total_downscale_factor=z`].downscale_factor=) superr __init__len num_adapternn ModuleListr ValueErrortotal_downscale_factordownscale_factorrange)selfr $first_adapter_total_downscale_factorfirst_adapter_downscale_factoridx __class__L/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/adapter.pyr )s<       zMultiAdapter.__init__Nxsadapter_weightsreturnc Cs|durtd|jg|j}nt|}d}t|||jD]6\}}}||}|dur@|}tt|D] }|||||<q4qtt|D]}|||||7<qFq|S)az Args: xs (`torch.Tensor`): A tensor of shape (batch, channel, height, width) representing input images for multiple adapter models, concatenated along dimension 1(channel dimension). The `channel` dimension should be equal to `num_adapter` * number of channel per image. adapter_weights (`list[float]`, *optional*, defaults to None): A list of floats representing the weights which will be multiplied by each adapter's output before summing them together. If `None`, equal weights will be used for all adapters. Nr)torchtensorrzipr rr) rrr accume_statexwadapterfeaturesirrrforwardLs  zMultiAdapter.forwardTsave_directoryis_main_process save_functionsafe_serializationvariantc CsBd}|}|jD]}|j|||||d|d7}|d|}qdS)a6 Save a model and its configuration file to a specified directory, allowing it to be re-loaded with the `[`~models.adapter.MultiAdapter.from_pretrained`]` class method. Args: save_directory (`str` or `os.PathLike`): The directory where the model will be saved. If the directory does not exist, it will be created. is_main_process (`bool`, optional, defaults=True): Indicates whether current process is the main process or not. Useful for distributed training (e.g., TPUs) and need to call this function on all processes. In this case, set `is_main_process=True` only for the main process to avoid race conditions. save_function (`Callable`): Function used to save the state dictionary. Useful for distributed training (e.g., TPUs) to replace `torch.save` with another method. Can also be configured using`DIFFUSERS_SAVE_MODE` environment variable. safe_serialization (`bool`, optional, defaults=True): If `True`, save the model using `safetensors`. If `False`, save the model with `pickle`. variant (`str`, *optional*): If specified, weights are saved in the format `pytorch_model..bin`. r)r,r-r.r/r_N)r save_pretrained) rr+r,r-r.r/rmodel_path_to_saver'rrrr1is zMultiAdapter.save_pretrainedpretrained_model_pathcKsd}g}|}tj|r+tj|fi|}|||d7}|d|}tj|s tt|d|dt|dkrOt dtj |d|dd||S) a Instantiate a pretrained `MultiAdapter` model from multiple pre-trained adapter models. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, set it back to training mode using `model.train()`. Warnings: *Weights from XXX not initialized from pretrained model* means that the weights of XXX are not pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning. *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, so those weights are discarded. Args: pretrained_model_path (`os.PathLike`): A path to a *directory* containing model weights saved using [`~diffusers.models.adapter.MultiAdapter.save_pretrained`], e.g., `./my_model_directory/adapter`. torch_dtype (`torch.dtype`, *optional*): Override the default `torch.dtype` and load the model under this dtype. output_loading_info(`bool`, *optional*, defaults to `False`): Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages. device_map (`str` or `dict[str, int | str | torch.device]`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For more information about each option see [designing a device map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map). max_memory (`Dict`, *optional*): A dictionary mapping device identifiers to their maximum memory. Default to the maximum memory available for each GPU and the available CPU RAM if unset. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading by not initializing the weights and only loading the pre-trained weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch, setting this argument to `True` will raise an error. variant (`str`, *optional*): If specified, load weights from a `variant` file (*e.g.* pytorch_model..bin). `variant` will be ignored when using `from_flax`. use_safetensors (`bool`, *optional*, defaults to `None`): If `None`, the `safetensors` weights will be downloaded if available **and** if`safetensors` library is installed. If `True`, the model will be forcibly loaded from`safetensors` weights. If `False`, `safetensors` is not used. rrr0z adapters loaded from .zNo T2IAdapters found under z. Expected at least _0) ospathisdirr from_pretrainedappendloggerinforrdirname)clsr3kwargsrr model_path_to_loadr'rrrr9s-    zMultiAdapter.from_pretrainedN)TNTN)__name__ __module__ __qualname____doc__listr r!Tensorfloatr*strr6PathLikeboolrr1 classmethodr9 __classcell__rrrrr s* (#  *"r c seZdZdZedgddddfdedeed ed ed ef fd d Zde j dee j fddZ e ddZ e ddZZS)r a\ A simple ResNet-like model that accepts images containing control signals such as keyposes and depth. The model generates multiple feature maps that are used as additional conditioning in [`UNet2DConditionModel`]. The model's architecture follows the original implementation of [Adapter](https://github.com/TencentARC/T2I-Adapter/blob/686de4681515662c0ac2ffa07bf5dda83af1038a/ldm/modules/encoders/adapter.py#L97) and [AdapterLight](https://github.com/TencentARC/T2I-Adapter/blob/686de4681515662c0ac2ffa07bf5dda83af1038a/ldm/modules/encoders/adapter.py#L235). This model inherits from [`ModelMixin`]. Check the superclass documentation for the common methods, such as downloading or saving. Args: in_channels (`int`, *optional*, defaults to `3`): The number of channels in the adapter's input (*control image*). Set it to 1 if you're using a gray scale image. channels (`list[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The number of channels in each downsample block's output hidden state. The `len(block_out_channels)` determines the number of downsample blocks in the adapter. num_res_blocks (`int`, *optional*, defaults to `2`): Number of ResNet blocks in each downsample block. downscale_factor (`int`, *optional*, defaults to `8`): A factor that determines the total downscale factor of the Adapter. adapter_type (`str`, *optional*, defaults to `full_adapter`): Adapter type (`full_adapter` or `full_adapter_xl` or `light_adapter`) to use. @rRr full_adapter in_channelschannelsnum_res_blocksr adapter_typecsnt|dkrt|||||_dS|dkr!t|||||_dS|dkr/t|||||_dStd|d)NrTfull_adapter_xl light_adapterzUnsupported adapter_type: 'zH'. Choose either 'full_adapter' or 'full_adapter_xl' or 'light_adapter'.)r r FullAdapterr' FullAdapterXL LightAdapterr)rrUrVrWrrXrrrr s  zT2IAdapter.__init__r%r cCs ||S)a This function processes the input tensor `x` through the adapter model and returns a list of feature tensors, each representing information extracted at a different scale from the input. The length of the list is determined by the number of downsample blocks in the Adapter, as specified by the `channels` and `num_res_blocks` parameters during initialization. )r'rr%rrrr* s zT2IAdapter.forwardcCs|jjSrA)r'rrrrrrsz!T2IAdapter.total_downscale_factorcCs |jjjS)zThe downscale factor applied in the T2I-Adapter's initial pixel unshuffle operation. If an input image's dimensions are not evenly divisible by the downscale_factor then an exception will be raised. )r' unshufflerr_rrrrs zT2IAdapter.downscale_factor)rBrCrDrErintrFrIr r!rGr*propertyrrrMrrrrr s0 r c ^eZdZdZdgdddfdedeeded effd d Zd ejd eejfddZ Z S)r[2 See [`T2IAdapter`] for more information. rNrOrrSrUrVrWrcst||d}t||_tj|dddd|_ttddgfddt dt D|_ |dt d|_ dS)NrrrNr kernel_sizepaddingcs(g|]}t|d|ddqSrTdown) AdapterBlock.0r)rVrWrr 8z(FullAdapter.__init__..) r r rPixelUnshuffler`Conv2dconv_inrrkrrbodyrrrUrVrWrrrnrr 's      zFullAdapter.__init__r%r cC:||}||}g}|jD] }||}||q|S)a This method processes the input tensor `x` through the FullAdapter model and performs operations including pixel unshuffling, convolution, and a stack of AdapterBlocks. It returns a list of feature tensors, each capturing information at a different stage of processing within the FullAdapter model. The number of feature tensors in the list is determined by the number of downsample blocks specified during initialization. r`rsrtr:rr%r(blockrrrr*As    zFullAdapter.forward rBrCrDrErarFr r!rGr*rMrrrrr[" "r[c rc)r\rdrNrOrrUrVrWrc st||d}t||_tj||dddd|_g|_tt |D]<}|dkr=|j t ||d|||q&|dkrT|j t ||d|||ddq&|j t |||||q&t |j|_|d|_ dS)NrrrNrreTri)r r rrqr`rrrsrtrrr:rkrr)rrUrVrWrr)rrrr Ys   "&zFullAdapterXL.__init__r%r cCrv)z This method takes the tensor x as input and processes it through FullAdapterXL model. It consists of operations including unshuffling pixels, applying convolution layer and appending each block into list of feature tensors. rwrxrrrr*us    zFullAdapterXL.forwardrzrrrrr\Ts "r\c JeZdZdZd dedededeffdd Zd ejd ejfd d Z Z S)rka# An AdapterBlock is a helper model that contains multiple ResNet-like blocks. It is used in the `FullAdapter` and `FullAdapterXL` models. Args: in_channels (`int`): Number of channels of AdapterBlock's input. out_channels (`int`): Number of channels of AdapterBlock's output. num_res_blocks (`int`): Number of ResNet blocks in the AdapterBlock. down (`bool`, *optional*, defaults to `False`): If `True`, perform downsampling on AdapterBlock's input. FrU out_channelsrWrjcshtd|_|rtjdddd|_d|_|kr#tj|dd|_tjfddt|D|_ dS)NrTrfstride ceil_moderrfcg|]}tqSr)AdapterResnetBlockrmr0r~rrroz)AdapterBlock.__init__..) r r downsampler AvgPool2din_convrr SequentialrresnetsrrUr~rWrjrrrr s  zAdapterBlock.__init__r%r cCs6|jdur ||}|jdur||}||}|S)a This method takes tensor x as input and performs operations downsampling and convolutional layers if the self.downsample and self.in_conv properties of AdapterBlock model are specified. Then it applies a series of residual blocks to the input tensor. N)rrrr^rrrr*s     zAdapterBlock.forwardF rBrCrDrErarKr r!rGr*rMrrrrrks rkc<eZdZdZdeffdd ZdejdejfddZZ S) rz An `AdapterResnetBlock` is a helper model that implements a ResNet-like block. Args: channels (`int`): Number of channels of AdapterResnetBlock's input and output. rVcs>ttj||ddd|_t|_tj||dd|_dS)NrNrrerr r rrrblock1ReLUactblock2rrVrrrr s  zAdapterResnetBlock.__init__r%r cC"|||}||}||S)z This method takes input tensor x and applies a convolutional layer, ReLU activation, and another convolutional layer on the input tensor. It returns addition with the input tensor. rrrrr%hrrrr* zAdapterResnetBlock.forward rBrCrDrErar r!rGr*rMrrrrrsrc rc)r]rdrN)rPrQrRrSrUrVrWrc st||d}t||_tt|dgfddttdDtdddd|_ |dt|_ dS) Nrrcs(g|]}t||dddqSrh)LightAdapterBlockrlrnrrrorpz)LightAdapter.__init__..rTri) r r rrqr`rrrrrtrrurrnrr s     zLightAdapter.__init__r%r cCs0||}g}|jD] }||}||q |S)z This method takes the input tensor x and performs downscaling and appends it in list of feature tensors. Each feature tensor corresponds to a different level of processing within the LightAdapter. )r`rtr:rxrrrr*s   zLightAdapter.forwardrzrrrrr]r{r]c r})ra< A `LightAdapterBlock` is a helper model that contains multiple `LightAdapterResnetBlocks`. It is used in the `LightAdapter` model. Args: in_channels (`int`): Number of channels of LightAdapterBlock's input. out_channels (`int`): Number of channels of LightAdapterBlock's output. num_res_blocks (`int`): Number of LightAdapterResnetBlocks in the LightAdapterBlock. down (`bool`, *optional*, defaults to `False`): If `True`, perform downsampling on LightAdapterBlock's input. FrUr~rWrjcstt|dd|_|rtjdddd|_tj|dd|_tjfddt|D|_ tj|dd|_ dS) NrrTrrrcrr)LightAdapterResnetBlockr mid_channelsrrrorz.LightAdapterBlock.__init__..) r r rrrrrrrrrout_convrrrrr s zLightAdapterBlock.__init__r%r cCs6|jdur ||}||}||}||}|S)z This method takes tensor x as input and performs downsampling if required. Then it applies in convolution layer, a sequence of residual blocks, and out convolutional layer. N)rrrrr^rrrr*s     zLightAdapterBlock.forwardrrrrrrrs  rcr) ra A `LightAdapterResnetBlock` is a helper model that implements a ResNet-like block with a slightly different architecture than `AdapterResnetBlock`. Args: channels (`int`): Number of channels of LightAdapterResnetBlock's input and output. rVcs@ttj||ddd|_t|_tj||ddd|_dS)NrNrrerrrrrr 8s  z LightAdapterResnetBlock.__init__r%r cCr)z This function takes input tensor x and processes it through one convolutional layer, ReLU activation, and another convolutional layer and adds it to input tensor. rrrrrr*>rzLightAdapterResnetBlock.forwardrrrrrr.s r)r6typingrr!torch.nnrconfiguration_utilsrrutilsrmodeling_utilsr get_loggerrBr;r r Moduler[r\rkrr]rrrrrrs$     =J220/+