o Gi%@sRddlZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZd d lmZmZmZmZd d lmZmZmZd d lmZeeZGdddeZGdddeZGdddeZGdddeZGdddeZ GdddeZ!GdddeZ"Gddde"Z#Gddde"Z$Gd d!d!e"Z%Gd"d#d#e"Z&dS)$N)Any) FrozenDict)ClassifierFreeGuidance)ControlNetModelUNet2DConditionModel)EulerDiscreteScheduler)logging) BlockStateLoopSequentialPipelineBlocksModularPipelineBlocks PipelineState) ComponentSpec InputParam OutputParam) StableDiffusionXLModularPipelinec @speZdZdZedeefddZedefddZ edeefddZ e d e d ed ed efd dZdS)#StableDiffusionXLLoopBeforeDenoiserstable-diffusion-xlreturncC tdtgSN schedulerrrselfrk/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/modular_pipelines/stable_diffusion_xl/denoise.pyexpected_components+z7StableDiffusionXLLoopBeforeDenoiser.expected_componentscC dS)Nzstep within the denoising loop that prepare the latent input for the denoiser. This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` object (e.g. `StableDiffusionXLDenoiseLoopWrapper`)rrrrr description1z/StableDiffusionXLLoopBeforeDenoiser.descriptioncCstddtjddgS)NlatentsT^The initial latents to use for the denoising process. Can be generated in prepare_latent step.required type_hintr"rtorchTensorrrrrinputs9sz*StableDiffusionXLLoopBeforeDenoiser.inputs components block_stateitcCs|j|j||_||fSN)rscale_model_inputr$scaled_latentsrr-r.r/r0rrr__call__Dsz,StableDiffusionXLLoopBeforeDenoiser.__call__N)__name__ __module__ __qualname__ model_namepropertylistrrstrr"r,r*no_gradrr intr5rrrrr(s rc @s|eZdZdZedeefddZedefddZ edeefddZ e d d Z e d ed ed edefddZdS)*StableDiffusionXLInpaintLoopBeforeDenoiserrrcCtdttdtgSNrunetrrrrrrrrOz>StableDiffusionXLInpaintLoopBeforeDenoiser.expected_componentscCr!)Nzstep within the denoising loop that prepare the latent input for the denoiser (for inpainting workflow only). This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` objectrrrrrr"Vr#z6StableDiffusionXLInpaintLoopBeforeDenoiser.descriptioncCs8tddtjddtdtjdBddtdtjdBd dgS) Nr$Tr%r&mask{The mask to use for the denoising process, for inpainting task only. Can be generated in vae_encode or prepare_latent step.r(r"masked_image_latentszThe masked image latents to use for the denoising process, for inpainting task only. Can be generated in vae_encode or prepare_latent step.r)rrrrr,]s"z1StableDiffusionXLInpaintLoopBeforeDenoiser.inputscCs|j}|dkrO|jdus|jdurtd|jjd}|jjd}|jjd}||||krQtd|jjd|jjjd|d|d|d |||d dSdS) N zKmask and masked_image_latents must be provided for inpainting-specific UnetrzCIncorrect configuration settings! The config of `components.unet`: z expects z& but received `num_channels_latents`: z + `num_channels_mask`: z + `num_channels_masked_image`: z = zV. Please verify the config of `components.unet` or your `mask_image` or `image` input.) num_channels_unetrErH ValueErrorr$shaperBconfig in_channels)r-r.rJnum_channels_latentsnum_channels_masknum_channels_masked_imagerrr check_inputsrs.     z7StableDiffusionXLInpaintLoopBeforeDenoiser.check_inputsr-r.r/r0cCsL||||j|j||_|jdkr"tj|j|j|j gdd|_||fS)NrIr)dim) rRrr2r$r3rJr*catrErHr4rrrr5s  z3StableDiffusionXLInpaintLoopBeforeDenoiser.__call__N)r6r7r8r9r:r;rrr<r"r, staticmethodrRr*r=rr r>r5rrrrr?Ls  r?c @s|eZdZdZedeefddZedefddZ edee ee ffddZ e d ed ed ed edef d dZdS)StableDiffusionXLLoopDenoiserrrcCs"tdttddiddtdtgS)Nguiderguidance_scale@ from_configrMdefault_creation_methodrB)rrrrrrrrrs z1StableDiffusionXLLoopDenoiser.expected_componentscCr!)NzStep within the denoising loop that denoise the latents with guidance. This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` object (e.g. `StableDiffusionXLDenoiseLoopWrapper`)rrrrrr"r#z)StableDiffusionXLLoopDenoiser.descriptioncCs4tdtddtddtdtjdBddtd d d gS) Ncross_attention_kwargsnum_inference_stepsTgThe number of inference steps to use for the denoising process. Can be generated in set_timesteps step.r& timestep_condzThe guidance scale embedding to use for Latent Consistency Models(LCMs). Can be generated in prepare_additional_conditioning step.rGdenoiser_input_fieldsAll conditional model inputs that need to be prepared with guider. It should contain prompt_embeds/negative_prompt_embeds, add_time_ids/negative_add_time_ids, pooled_prompt_embeds/negative_pooled_prompt_embeds, and ip_adapter_embeds/negative_ip_adapter_embeds (optional).please add `kwargs_type=denoiser_input_fields` to their parameter spec (`OutputParam`) when they are created and added to the pipeline state kwargs_typer")rr>r*r+rrrrr,s"z$StableDiffusionXLLoopDenoiser.inputsr-r.r/r0c st|ddt|ddft|ddt|ddft|ddt|ddft|ddt|ddfd }|jj||j|d |j|}|D]2|j|jfd d |D}|d}|j|j |||j |j |d dd_ |j |jqA||d|_ ||fS)N prompt_embedsnegative_prompt_embeds add_time_idsnegative_add_time_idspooled_prompt_embedsnegative_pooled_prompt_embedsip_adapter_embedsnegative_ip_adapter_embedsretime_ids text_embeds image_embedsstepr^timestepcsi|]}|t|qSr)getattr).0 input_nameguider_state_batchrr sz:StableDiffusionXLLoopDenoiser.__call__..F)encoder_hidden_statesr`r]added_cond_kwargs return_dictr)rtrW set_stater^prepare_inputsprepare_modelsrBkeyspopr3r`r] noise_predcleanup_models) rr-r.r/r0 guider_inputs guider_state cond_kwargsrerrwrr5sB           z&StableDiffusionXLLoopDenoiser.__call__N)r6r7r8r9r:r;rrr<r"tuplerr,r*r=rr r>rr5rrrrrVs( rVc @seZdZdZedeefddZedefddZ edee ee ffddZ e gfd d Zed ed ed edefddZdS)'StableDiffusionXLControlNetLoopDenoiserrrcC*tdttddiddtdttdtgS)NrWrXrYrZr[rB controlnet)rrrrrrrrrr z;StableDiffusionXLControlNetLoopDenoiser.expected_componentscCr!)Nzstep within the denoising loop that denoise the latents with guidance (with controlnet). This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` object (e.g. `StableDiffusionXLDenoiseLoopWrapper`)rrrrrr"r#z3StableDiffusionXLControlNetLoopDenoiser.descriptionc Csztdtddtjddtdtddtd dtd dtd dttd dtd tjdBddtddtddtdddtdddg S)Nr]controlnet_condTzgThe control image to use for the denoising process. Can be generated in prepare_controlnet_inputs step.r&conditioning_scalez}The controlnet conditioning scale value to use for the denoising process. Can be generated in prepare_controlnet_inputs step.rG guess_modezjThe guess mode value to use for the denoising process. Can be generated in prepare_controlnet_inputs step.controlnet_keepzpThe controlnet keep values to use for the denoising process. Can be generated in prepare_controlnet_inputs step.r`zThe guidance scale embedding to use for Latent Consistency Models(LCMs), can be generated by prepare_additional_conditioning stepr^r_rarbrccontrolnet_kwargszadditional kwargs for controlnet (e.g. control_type_idx and control_type from the controlnet union input step )please add `kwargs_type=controlnet_kwargs` to their parameter spec (`OutputParam`) when they are created and added to the pipeline state)rr*r+floatboolr;r>rrrrr,sX z.StableDiffusionXLControlNetLoopDenoiser.inputscKFtt|j}i}|D]\}}||vr ||vr |||<q|Sr1setinspect signature parametersritemsfuncexclude_kwargskwargsaccepted_kwargs extra_kwargskeyvaluerrrprepare_extra_kwargsWz~s zDStableDiffusionXLControlNetLoopDenoiser.__call__..rrq)rornrpF)rzrrrr{r|cSsg|]}t|qSr)r* zeros_like)rudrrrrs)rzr`r]r{down_block_additional_residualsmid_block_additional_residualr|)"rrforwardrrt isinstancerr;zipr cond_scaledown_block_res_samples_zerosmid_block_res_sample_zerosrWr}r^r~rrBrornhasattrrpris_conditionalr3rerr*rr`r]rr)rr-r.r/r0extra_controlnet_kwargsrcontrolnet_cond_scalerrxr{controlnet_added_cond_kwargsdown_block_res_samplesmid_block_res_samplerrrr5as                 z0StableDiffusionXLControlNetLoopDenoiser.__call__N)r6r7r8r9r:r;rrr<r"rrr,rUrr*r=rr r>r5rrrrrs 9 rc @seZdZdZedeefddZedefddZ edee ee ffddZ edee fd d Zegfd d Zed edededefddZdS)"StableDiffusionXLLoopAfterDenoiserrrcCrrrrrrrrr z6StableDiffusionXLLoopAfterDenoiser.expected_componentscCr!)Nzstep within the denoising loop that update the latents. This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` object (e.g. `StableDiffusionXLDenoiseLoopWrapper`)rrrrrr"r#z.StableDiffusionXLLoopAfterDenoiser.descriptioncCstdddtdgS)Netadefault generator)rrrrrr,s z)StableDiffusionXLLoopAfterDenoiser.inputscCtdtjddgSNr$zThe denoised latentsrGrr*r+rrrrintermediate_outputsz7StableDiffusionXLLoopAfterDenoiser.intermediate_outputscKrr1rrrrrrrz7StableDiffusionXLLoopAfterDenoiser.prepare_extra_kwargsr-r.r/r0cCs|j|jj|j|jd|_|jj|_|jj|j ||jfi|jddid|_|jj|jkr=t j j r=|j|j|_||fS)Nrrr|Fr)rrrrrrextra_step_kwargsr$dtype latents_dtyperr*backendsmps is_availabletor4rrrr5s&  z+StableDiffusionXLLoopAfterDenoiser.__call__N)r6r7r8r9r:r;rrr<r"rrr,rrrUrr*r=rr r>r5rrrrrs rc @seZdZdZedeefddZedefddZ edee ee ffddZ edee fd d Zegfd d Zd dZededededefddZdS))StableDiffusionXLInpaintLoopAfterDenoiserrrcCr@rArCrrrrrrDz=StableDiffusionXLInpaintLoopAfterDenoiser.expected_componentscCr!)Nzstep within the denoising loop that update the latents (for inpainting workflow only). This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` object (e.g. `StableDiffusionXLDenoiseLoopWrapper`)rrrrrr""r#z5StableDiffusionXLInpaintLoopAfterDenoiser.descriptionc CsZtdddtdtddtjddtd tjdBd d td tjdBd d tdtjdBdd gS)Nrrrr timestepsTWThe timesteps to use for the denoising process. Can be generated in set_timesteps step.r&rErFrGnoisezhThe noise added to the image latents, for inpainting task only. Can be generated in prepare_latent step. image_latentszThe image latents to use for the denoising process, for inpainting/image-to-image task only. Can be generated in vae_encode or prepare_latent step.r)rrrrr,*s0 z0StableDiffusionXLInpaintLoopAfterDenoiser.inputscCrrrrrrrrFrz>StableDiffusionXLInpaintLoopAfterDenoiser.intermediate_outputscKrr1rrrrrrJrz>StableDiffusionXLInpaintLoopAfterDenoiser.prepare_extra_kwargscCsf|jdkr/|jdurtd|jj|jdur!td|jj|jdur1td|jjdSdS)Nz(image_latents is required for this step zmask is required for this step z noise is required for this step )rJrrK __class__r6rEr)rr-r.rrrrRTs    z6StableDiffusionXLInpaintLoopAfterDenoiser.check_inputsr-r.r/r0cCs||||j|jj|j|jd|_|jj|_ |jj|j ||jfi|jddid|_|jj|j krCt j j rC|j|j |_|jdkr||j|_|t|jdkrm|j|d|_|j|j|jt |jg|_d|j|j|j|j|_||fS)Nrr|Frrr)rRrrrrrrrr$rrrr*rrrrrJrinit_latents_properlenrnoise_timestep add_noisertensorrEr4rrrr5]s@     z2StableDiffusionXLInpaintLoopAfterDenoiser.__call__N)r6r7r8r9r:r;rrr<r"rrr,rrrUrrRr*r=rr r>r5rrrrrs  rc@sleZdZdZedefddZedeefddZ edee fddZ e d ed edefd d Zd S)#StableDiffusionXLDenoiseLoopWrapperrrcCr!)NzPipeline block that iteratively denoise the latents over `timesteps`. The specific steps with each iteration can be customized with `sub_blocks` attributesrrrrrr"r#z/StableDiffusionXLDenoiseLoopWrapper.descriptioncCr)NrWrXrYrZr[rrB)rrrrrrrrrloop_expected_componentsrzrrrr loop_inputssz/StableDiffusionXLDenoiseLoopWrapper.loop_inputsr-statecCs||}|jjjdurdnd|_|jr|jn|jtt |j |j |j j d|_|j|j d;}t|j D]-\}}|j||||d\}}|t |j dkse|d|jkri|d|j j dkri|qs0       $GrSAnD