o Gi_@sddlmZddlmZddlZddlZddlmZddl m Z ddl m Z m Z ddlmZdd lmZmZdd lmZmZdd lmZmZmZmZdd lmZd dlmZddlm Z erkddl!m"m#Z$dZ%ndZ%e&e'Z(dZ)eGdddeZ*GdddeZ+dS)) dataclass)AnyN)Image)tqdm) CLIPTextModel CLIPTokenizer)PipelineImageInput) AutoencoderKLUNet2DConditionModel) DDIMScheduler LCMScheduler) BaseOutputis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipeline)MarigoldImageProcessorTFa Examples: ```py >>> import diffusers >>> import torch >>> pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained( ... "prs-eth/marigold-iid-appearance-v1-1", variant="fp16", torch_dtype=torch.float16 ... ).to("cuda") >>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg") >>> intrinsics = pipe(image) >>> vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties) >>> vis[0]["albedo"].save("einstein_albedo.png") >>> vis[0]["roughness"].save("einstein_roughness.png") >>> vis[0]["metallicity"].save("einstein_metallicity.png") ``` ```py >>> import diffusers >>> import torch >>> pipe = diffusers.MarigoldIntrinsicsPipeline.from_pretrained( ... "prs-eth/marigold-iid-lighting-v1-1", variant="fp16", torch_dtype=torch.float16 ... ).to("cuda") >>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg") >>> intrinsics = pipe(image) >>> vis = pipe.image_processor.visualize_intrinsics(intrinsics.prediction, pipe.target_properties) >>> vis[0]["albedo"].save("einstein_albedo.png") >>> vis[0]["shading"].save("einstein_shading.png") >>> vis[0]["residual"].save("einstein_residual.png") ``` c@sDeZdZUdZejejBed<dejBejBed<dejBed<dS)MarigoldIntrinsicsOutputuS Output class for Marigold Intrinsic Image Decomposition pipeline. Args: prediction (`np.ndarray`, `torch.Tensor`): Predicted image intrinsics with values in the range [0, 1]. The shape is `(numimages * numtargets) × 3 × height × width` for `torch.Tensor` or `(numimages * numtargets) × height × width × 3` for `np.ndarray`, where `numtargets` corresponds to the number of predicted target modalities of the intrinsic image decomposition. uncertainty (`None`, `np.ndarray`, `torch.Tensor`): Uncertainty maps computed from the ensemble, with values in the range [0, 1]. The shape is `(numimages * numtargets) × 3 × height × width` for `torch.Tensor` or `(numimages * numtargets) × height × width × 3` for `np.ndarray`. latent (`None`, `torch.Tensor`): Latent features corresponding to the predictions, compatible with the `latents` argument of the pipeline. The shape is `(numimages * numensemble) × (numtargets * 4) × latentheight × latentwidth`. predictionN uncertaintylatent) __name__ __module__ __qualname____doc__npndarraytorchTensor__annotations__r$r$m/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/marigold/pipeline_marigold_intrinsics.pyr_s rc!s"eZdZdZdZdZ    d7dededee Bde d e d e dBd e e efdBd edBd edBffdd ZeddZdedededede de dede e efdBdejdBdejeejBdBde dedefdd Zejjd8d"d#Zeee  $  ! % % $    & ' ' !d9dededBdededBd(ede de dede e efdBdejeejBdBdejeejBdBde ded)ed*efd+d,Z dejdejdBdejdBdedede!ejejff d-d.Z"d/ejdejfd0d1Z#e$ ' 2d:d3ejded4e de!ejejdBffd5d6Z%Z&S);MarigoldIntrinsicsPipelinea Pipeline for Intrinsic Image Decomposition (IID) using the Marigold method: https://marigoldcomputervision.github.io. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: unet (`UNet2DConditionModel`): Conditional U-Net to denoise the targets latent, conditioned on image latent. vae (`AutoencoderKL`): Variational Auto-Encoder (VAE) Model to encode and decode images and predictions to and from latent representations. scheduler (`DDIMScheduler` or `LCMScheduler`): A scheduler to be used in combination with `unet` to denoise the encoded image latents. text_encoder (`CLIPTextModel`): Text-encoder, for empty text embedding. tokenizer (`CLIPTokenizer`): CLIP tokenizer. prediction_type (`str`, *optional*): Type of predictions made by the model. target_properties (`dict[str, Any]`, *optional*): Properties of the predicted modalities, such as `target_names`, a `list[str]` used to define the number, order and names of the predicted modalities, and any other metadata that may be required to interpret the predictions. default_denoising_steps (`int`, *optional*): The minimum number of denoising diffusion steps that are required to produce a prediction of reasonable quality with the given model. This value must be set in the model config. When the pipeline is called without explicitly setting `num_inference_steps`, the default value is used. This is required to ensure reasonable results with various model flavors compatible with the pipeline, such as those relying on very short denoising schedules (`LCMScheduler`) and those with full diffusion schedules (`DDIMScheduler`). default_processing_resolution (`int`, *optional*): The recommended value of the `processing_resolution` parameter of the pipeline. This value must be set in the model config. When the pipeline is called without explicitly setting `processing_resolution`, the default value is used. This is required to ensure reasonable results with various model flavors trained with varying optimal processing resolution values. ztext_encoder->unet->vae) intrinsicsNunetvae scheduler text_encoder tokenizerprediction_typetarget_propertiesdefault_denoising_stepsdefault_processing_resolutionc st||jvrtd|d|jd|j|||||d|j|||| dt|ddr;dt|j j j dnd |_ ||_ ||_| |_d|_t|j d |_dS) Nz*Potentially unsupported `prediction_type='z&'`; values supported by the pipeline: .)r(r)r*r+r,)r-r.r/r0r)rr)vae_scale_factor)super__init__supported_prediction_typesloggerwarningregister_modulesregister_to_configgetattrlenr)configblock_out_channelsr3r.r/r0empty_text_embeddingrimage_processor) selfr(r)r*r+r,r-r.r/r0 __class__r$r%r5s4 (z#MarigoldIntrinsicsPipeline.__init__cCs|jjj|jjjS)N)r(r= out_channelsr)latent_channels)rAr$r$r% n_targetssz$MarigoldIntrinsicsPipeline.n_targetsimagenum_inference_steps ensemble_sizeprocessing_resolutionresample_method_inputresample_method_output batch_sizeensembling_kwargslatents generator output_typeoutput_uncertaintyreturnc  sdt|jjjd} | |jkrtd|jd| d|dur$td|dkr,td|dkr4td|dkr=td |dkrG| rGtd |durOtd |d krWtd ||jd krgtd|jd|dvrotd|dvrwtd|dkrtd| dvrtd| durdurtd|durt|t stdd|vr|ddvrtdd }d\}}t|t s|g}t |D]{\}}t|t j st|r|jdvrtd|d|jd|jdd\}}d}|jd kr|jd }nt|tjr|j\}}d}n td!|d"t|d|dur||}}n||f||fkr7td#|d$||fd%||f||7}q| durt| sKtd&| d kr[td'| jd|d krt||}|||}|||}|d ks{|d krtd(|d)|d*||}}||jd|j}||jd|j}|||jjj||f}| j|krtd+| jd,|ddurtt rt||krtd-tfd.d/Dstd0|Sttjstd1td|S)2Nrrz/`vae_scale_factor` computed at initialization (z) differs from the actual one (z).zW`num_inference_steps` is not specified and could not be resolved from the model config.z'`num_inference_steps` must be positive.z!`ensemble_size` must be positive.zk`ensemble_size` == 2 results are similar to no ensembling (1); consider increasing the value to at least 3.zpComputing uncertainty by setting `output_uncertainty=True` also requires setting `ensemble_size` greater than 1.zY`processing_resolution` is not specified and could not be resolved from the model config.rzx`processing_resolution` must be non-negative: 0 for native resolution, or any positive value for downsampled processing.z.`processing_resolution` must be a multiple of r1)nearestz nearest-exactbilinearbicubicareazy`resample_method_input` takes string values compatible with PIL library: nearest, nearest-exact, bilinear, bicubic, area.zz`resample_method_output` takes string values compatible with PIL library: nearest, nearest-exact, bilinear, bicubic, area.z`batch_size` must be positive.)ptrz*`output_type` must be one of `pt` or `np`.z2`latents` and `generator` cannot be used together.z)`ensembling_kwargs` must be a dictionary. reductionmedianmeanzF`ensembling_kwargs['reduction']` can be either `'median'` or `'mean'`.)NN)rrz`image[z(]` has unsupported dimensions or shape: r]zUnsupported `image[z ]` type: z Input `image[z]` has incompatible dimensions z with the previous images z!`latents` must be a torch.Tensor.z/`latents` has unsupported dimensions or shape: z*Extreme aspect ratio of the input image: [z x ]z`latents` has unexpected shape=z expected=z^The number of generators must match the total number of ensemble members for all input images.c3s$|] }|jjdjjkVqdS)rN)devicetype).0grPr$r% Ms"z:MarigoldIntrinsicsPipeline.check_inputs..z;`generator` device placement is not consistent in the list.zUnsupported generator type: )r<r)r=r>r3 ValueErrorr7r8 isinstancedictlist enumeraterr r! is_tensorndimshapersizeradimmaxr(rDall Generator)rArGrHrIrJrKrLrMrNrOrPrQrRactual_vae_scale_factor num_imagesWHiimgH_iW_iN_imax_orignew_Hnew_Wwhshape_expectedr$rdr% check_inputss                     z'MarigoldIntrinsicsPipeline.check_inputsTcCst|ds i|_nt|jtstdt|jdtdi|j}|d||d<|d||d<|dur>t|fi|S|durKtdd|i|Std) N_progress_bar_configz=`self._progress_bar_config` should be of type `dict`, but is r1descleavetotalz/Either `total` or `iterable` has to be defined.r$)hasattrrrgrhrfragetr)rAiterablerrrprogress_bar_configr$r$r% progress_barTs  z'MarigoldIntrinsicsPipeline.progress_barrrUrFmatch_input_resolution output_latent return_dictc%sZj}j}|dur j}|durj}||||||| | |  }jdurEd}j|djjddd}|j |} |d_j |||||\}}} || | |\}}~jj ||dd d }g}jtd||dd d D]\}|||}|||}|jd}|d|} jj||d jjjd dd D]'}!tj||gd d}"j|"|!| d dd}#jj|#|!|| dj}trtq||q{tj|dd}~~~~~~ ~"~#|j||jj j!j"g|jddRtjfddtdjdDdd|sd}j #|d}$|d krbj||jgjd dRfddt|Dt$\}$tjddr`tj|$dd}$nd}$|rj j%||d d|$durrj j%|$||d d}$| dkrj &|$durrj &|$}$'|s|$|fSt(|$|dS)aj Function invoked when calling the pipeline. Args: image (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`), `list[torch.Tensor]`: An input image or images used as an input for the intrinsic decomposition task. For arrays and tensors, the expected value range is between `[0, 1]`. Passing a batch of images is possible by providing a four-dimensional array or a tensor. Additionally, a list of images of two- or three-dimensional arrays or tensors can be passed. In the latter case, all list elements must have the same width and height. num_inference_steps (`int`, *optional*, defaults to `None`): Number of denoising diffusion steps during inference. The default value `None` results in automatic selection. ensemble_size (`int`, defaults to `1`): Number of ensemble predictions. Higher values result in measurable improvements and visual degradation. processing_resolution (`int`, *optional*, defaults to `None`): Effective processing resolution. When set to `0`, matches the larger input image dimension. This produces crisper predictions, but may also lead to the overall loss of global context. The default value `None` resolves to the optimal value from the model config. match_input_resolution (`bool`, *optional*, defaults to `True`): When enabled, the output prediction is resized to match the input dimensions. When disabled, the longer side of the output will equal to `processing_resolution`. resample_method_input (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize input images to `processing_resolution`. The accepted values are: `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. resample_method_output (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize output predictions to match the input resolution. The accepted values are `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. batch_size (`int`, *optional*, defaults to `1`): Batch size; only matters when setting `ensemble_size` or passing a tensor of images. ensembling_kwargs (`dict`, *optional*, defaults to `None`) Extra dictionary with arguments for precise ensembling control. The following options are available: - reduction (`str`, *optional*, defaults to `"median"`): Defines the ensembling function applied in every pixel location, can be either `"median"` or `"mean"`. latents (`torch.Tensor`, *optional*, defaults to `None`): Latent noise tensors to replace the random initialization. These can be taken from the previous function call's output. generator (`torch.Generator`, or `list[torch.Generator]`, *optional*, defaults to `None`): Random number generator object to ensure reproducibility. output_type (`str`, *optional*, defaults to `"np"`): Preferred format of the output's `prediction` and the optional `uncertainty` fields. The accepted values are: `"np"` (numpy array) or `"pt"` (torch tensor). output_uncertainty (`bool`, *optional*, defaults to `False`): When enabled, the output's `uncertainty` field contains the predictive uncertainty map, provided that the `ensemble_size` argument is set to a value above 2. output_latent (`bool`, *optional*, defaults to `False`): When enabled, the output's `latent` field contains the latent codes corresponding to the predictions within the ensemble. These codes can be saved, modified, and used for subsequent calls with the `latents` argument. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.marigold.MarigoldIntrinsicsOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.marigold.MarigoldIntrinsicsOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.marigold.MarigoldIntrinsicsOutput`] is returned, otherwise a `tuple` is returned where the first element is the prediction, the second element is the uncertainty (or `None`), and the third is the latent (or `None`). N do_not_padTrX)padding max_length truncationreturn_tensorsr)r`dtyperzMarigold predictions...)rr)r`FzDiffusion steps...ro)encoder_hidden_statesrrdrcs"g|] }||qSr$)decode_predictionrbrw)rMpred_latent_for_decodingrAr$r% 1sz7MarigoldIntrinsicsPipeline.__call__..cs(g|]}j|fipiqSr$)ensemble_intrinsicsr)rNrRrrAr$r%rI)is_aar)rrr))_execution_devicerr/r0rr?r,model_max_length input_idstor+r@ preprocessprepare_latentsrepeatrrangermr* set_timesteps timestepsr!catr(step prev_sample XLA_AVAILABLExm mark_stepappendreshaperFr)r=rE unpad_imagezipresize_antialias pt_to_numpymaybe_free_model_hooksr)%rArGrHrIrJrrKrLrMrNrOrPrQrRrrr`rrtprompt text_inputstext_input_idsroriginal_resolution image_latent pred_latentbatch_empty_text_embedding pred_latentsrwbatch_image_latentbatch_pred_latenteffective_batch_sizetextt batch_latentnoiserr$)rMrNrRrrrAr%__call__gsR                  z#MarigoldIntrinsicsPipeline.__call__c sddtjfddtdjdDdd}|jjj}|j|dd}|j\}}} } |} | durHt|j || | f||j |j d} || fS)NcSs,t|dr |jSt|dr|jStd)N latent_distrOz3Could not access latents of provided encoder_output)rrmoderOAttributeError)encoder_outputr$r$r%retrieve_latents{s   zDMarigoldIntrinsicsPipeline.prepare_latents..retrieve_latentsc s(g|]}j||qSr$)r)encoderrMrGrrAr$r%rrz>MarigoldIntrinsicsPipeline.prepare_latents..rr)rPr`r) r!rrrmr)r=scaling_factorrepeat_interleaverrFr`r) rArGrOrPrIrMrN_ECrvrurr$rr%rss&z*MarigoldIntrinsicsPipeline.prepare_latentsrcCsz|dks|jd|jjjkrtd|jjjd|jd|jj||jjjddd}t |d d }|d d }|S) Nr]rz Expecting 4D tensor of shape [B,z ,H,W]; got r1F)rrgg?g@) rormr)r=rErfdecoderr!clip)rArrr$r$r%rs  z,MarigoldIntrinsicsPipeline.decode_predictionr[targetsrYc Cs|dks |jddkrtd|jd|dvr"td|d|j\}}}}}d}|d krDtj|d d } |r@tj|d d }| |fS|d krktj|d d dj} |rbt|| }tj|d d j}| d } | |fStd|d)a Ensembles the intrinsic decomposition represented by the `targets` tensor with expected shape `(B, T, 3, H, W)`, where B is the number of ensemble members for a given prediction of size `(H x W)`, and T is the number of predicted targets. Args: targets (`torch.Tensor`): Input ensemble of intrinsic image decomposition maps. output_uncertainty (`bool`, *optional*, defaults to `False`): Whether to output uncertainty map. reduction (`str`, *optional*, defaults to `"mean"`): Reduction method used to ensemble aligned predictions. The accepted values are: `"median"` and `"mean"`. Returns: A tensor of aligned and ensembled intrinsic decomposition maps with shape `(T, 3, H, W)` and optionally a tensor of uncertainties of shape `(T, 3, H, W)`. rrz.Expecting 4D tensor of shape [B,T,3,H,W]; got r1rZzUnrecognized reduction method: Nr\rrr[T)rokeepdim) rormrfr!r\stdr[valuesabssqueeze) rrRrYBT_rvrurrr$r$r%rs&  z.MarigoldIntrinsicsPipeline.ensemble_intrinsics)NNNN)NNNT)NrNTrUrUrNNNrFFT)Fr[)'rrrrmodel_cpu_offload_seqr6r r r r rrstrrhrintr5propertyrFr r!r"rrriboolrcompilerdisablerno_gradrEXAMPLE_DOC_STRINGrtuplerr staticmethodr __classcell__r$r$rBr%r&xs &   ,               & r&), dataclassesrtypingrnumpyrr!PILr tqdm.autor transformersrrr@r modelsr r schedulersr r utilsrrrrutils.torch_utilsrpipeline_utilsrmarigold_image_processingrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrr7rrr&r$r$r$r%s.         %