o piq@sddlmZddlmZddlmZmZmZmZm Z m Z ddl Z ddl Z ddlmZddlmZddlmZmZdd lmZdd lmZmZdd lmZmZdd lmZmZm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(e)e*Z+dZ,eGdddeZ-Gddde&Z.dS)) dataclass)partial)AnyDictListOptionalTupleUnionN)Image)tqdm) CLIPTextModel CLIPTokenizer)PipelineImageInput) AutoencoderKLUNet2DConditionModel) DDIMScheduler LCMScheduler) BaseOutputloggingreplace_example_docstring)is_scipy_available) randn_tensor)DiffusionPipeline)MarigoldImageProcessoraI Examples: ```py >>> import diffusers >>> import torch >>> pipe = diffusers.MarigoldDepthPipeline.from_pretrained( ... "prs-eth/marigold-depth-lcm-v1-0", variant="fp16", torch_dtype=torch.float16 ... ).to("cuda") >>> image = diffusers.utils.load_image("https://marigoldmonodepth.github.io/images/einstein.jpg") >>> depth = pipe(image) >>> vis = pipe.image_processor.visualize_depth(depth.prediction) >>> vis[0].save("einstein_depth.png") >>> depth_16bit = pipe.image_processor.export_depth_to_16bit_png(depth.prediction) >>> depth_16bit[0].save("einstein_depth_16bit.png") ``` c@sNeZdZUdZeejejfe d<edejejfe d<edejfe d<dS)MarigoldDepthOutputa9 Output class for Marigold monocular depth prediction pipeline. Args: prediction (`np.ndarray`, `torch.Tensor`): Predicted depth maps with values in the range [0, 1]. The shape is always $numimages imes 1 imes height imes width$, regardless of whether the images were passed as a 4D array or a list. uncertainty (`None`, `np.ndarray`, `torch.Tensor`): Uncertainty maps computed from the ensemble, with values in the range [0, 1]. The shape is $numimages imes 1 imes height imes width$. latent (`None`, `torch.Tensor`): Latent features corresponding to the predictions, compatible with the `latents` argument of the pipeline. The shape is $numimages * numensemble imes 4 imes latentheight imes latentwidth$. predictionN uncertaintylatent) __name__ __module__ __qualname____doc__r npndarraytorchTensor__annotations__r*r*r/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/marigold/pipeline_marigold_depth.pyrJs rc!sDeZdZdZdZdZ     d>dededee e fd e d e d e ed e ed e ede ede effdd Zdededededededede eeefde ejde eejeejfdededefdd Zd?d!d"Zeee  #   $ $ #    % & & d@dede edede ed'edededede eeefde eejeejfde eejeejfdeded(ed)efd*d+Zdejde ejde ejdededeejejff d,d-Z d.ejdejfd/d0Z!e"   & 1 2 3 4 5dAd6ejd ed eded7ed8e#d9ed:e#d;edeeje ejffdunet->vae)depth disparityNTunetvae scheduler text_encoder tokenizerprediction_typescale_invariantshift_invariantdefault_denoising_stepsdefault_processing_resolutionc st||jvrtd|d|jd|j|||||d|j|||| | ddt|jj j d|_ ||_ ||_ | |_| |_d|_t|j d|_dS) Nz*Potentially unsupported `prediction_type='z&'`; values supported by the pipeline: .)r/r0r1r2r3)r4r5r6r7r8rr)vae_scale_factor)super__init__supported_prediction_typesloggerwarningregister_modulesregister_to_configlenr0configblock_out_channelsr:r5r6r7r8empty_text_embeddingrimage_processor) selfr/r0r1r2r3r4r5r6r7r8 __class__r*r+r<s8 zMarigoldDepthPipeline.__init__imagenum_inference_steps ensemble_sizeprocessing_resolutionresample_method_inputresample_method_output batch_sizeensembling_kwargslatents generator output_typeoutput_uncertaintyreturnc  s|durtd|dkrtd|dkrtd|dkr!td|dkr2|js+|jr2ts2td|dkr<| r4s"z5MarigoldDepthPipeline.check_inputs..z;`generator` device placement is not consistent in the list.zUnsupported generator type: ) ValueErrorr>r?r5r6r ImportErrorr: isinstancedictlist enumerater%r&r' is_tensorndimshaper sizeredimmaxr0rClatent_channelsrBall Generator)rGrJrKrLrMrNrOrPrQrRrSrTrU num_imagesWHiimgH_iW_iN_imax_orignew_Hnew_Wwhshape_expectedr*rhr+ check_inputss                     z"MarigoldDepthPipeline.check_inputscCst|ds i|_nt|jtstdt|jdtdi|j}|d||d<|d||d<|dur>t|fi|S|durKtdd|i|Std) N_progress_bar_configz=`self._progress_bar_config` should be of type `dict`, but is r9descleavetotalz/Either `total` or `iterable` has to be defined.r*)hasattrrrlrmrjregetr )rGiterablerrrprogress_bar_configr*r*r+ progress_bar;s  z"MarigoldDepthPipeline.progress_barrrYr%Fmatch_input_resolution output_latent return_dictc$sj}j}|dur j}|durj}||||||| | |  }jdurEd}j|djjddd}|j |} |d_j |||||\}}} || | |\}~jj ||dd d }g}jtd||dd d D]V}|||}||}|jd}|d|}jj||d jjjd dd D]!} tj||gd d}!j|!| |d dd}"jj|"| || dj}q||q{tj|dd~~~~~~~!~"tjfddtdjdDdd|sdj |d}#|d krAj||gjd dRfddt|Dt\}#tjddr?tj|#dd}#nd}#|r`j j||d d|#dur`r`j j|#||d d}#| dkryj |#duryryj |#}# |s|#fSt!|#dS)a Function invoked when calling the pipeline. Args: image (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`), `List[torch.Tensor]`: An input image or images used as an input for the depth estimation task. For arrays and tensors, the expected value range is between `[0, 1]`. Passing a batch of images is possible by providing a four-dimensional array or a tensor. Additionally, a list of images of two- or three-dimensional arrays or tensors can be passed. In the latter case, all list elements must have the same width and height. num_inference_steps (`int`, *optional*, defaults to `None`): Number of denoising diffusion steps during inference. The default value `None` results in automatic selection. The number of steps should be at least 10 with the full Marigold models, and between 1 and 4 for Marigold-LCM models. ensemble_size (`int`, defaults to `1`): Number of ensemble predictions. Recommended values are 5 and higher for better precision, or 1 for faster inference. processing_resolution (`int`, *optional*, defaults to `None`): Effective processing resolution. When set to `0`, matches the larger input image dimension. This produces crisper predictions, but may also lead to the overall loss of global context. The default value `None` resolves to the optimal value from the model config. match_input_resolution (`bool`, *optional*, defaults to `True`): When enabled, the output prediction is resized to match the input dimensions. When disabled, the longer side of the output will equal to `processing_resolution`. resample_method_input (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize input images to `processing_resolution`. The accepted values are: `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. resample_method_output (`str`, *optional*, defaults to `"bilinear"`): Resampling method used to resize output predictions to match the input resolution. The accepted values are `"nearest"`, `"nearest-exact"`, `"bilinear"`, `"bicubic"`, or `"area"`. batch_size (`int`, *optional*, defaults to `1`): Batch size; only matters when setting `ensemble_size` or passing a tensor of images. ensembling_kwargs (`dict`, *optional*, defaults to `None`) Extra dictionary with arguments for precise ensembling control. The following options are available: - reduction (`str`, *optional*, defaults to `"median"`): Defines the ensembling function applied in every pixel location, can be either `"median"` or `"mean"`. - regularizer_strength (`float`, *optional*, defaults to `0.02`): Strength of the regularizer that pulls the aligned predictions to the unit range from 0 to 1. - max_iter (`int`, *optional*, defaults to `2`): Maximum number of the alignment solver steps. Refer to `scipy.optimize.minimize` function, `options` argument. - tol (`float`, *optional*, defaults to `1e-3`): Alignment solver tolerance. The solver stops when the tolerance is reached. - max_res (`int`, *optional*, defaults to `None`): Resolution at which the alignment is performed; `None` matches the `processing_resolution`. latents (`torch.Tensor`, or `List[torch.Tensor]`, *optional*, defaults to `None`): Latent noise tensors to replace the random initialization. These can be taken from the previous function call's output. generator (`torch.Generator`, or `List[torch.Generator]`, *optional*, defaults to `None`): Random number generator object to ensure reproducibility. output_type (`str`, *optional*, defaults to `"np"`): Preferred format of the output's `prediction` and the optional `uncertainty` fields. The accepted values are: `"np"` (numpy array) or `"pt"` (torch tensor). output_uncertainty (`bool`, *optional*, defaults to `False`): When enabled, the output's `uncertainty` field contains the predictive uncertainty map, provided that the `ensemble_size` argument is set to a value above 2. output_latent (`bool`, *optional*, defaults to `False`): When enabled, the output's `latent` field contains the latent codes corresponding to the predictions within the ensemble. These codes can be saved, modified, and used for subsequent calls with the `latents` argument. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.marigold.MarigoldDepthOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.marigold.MarigoldDepthOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.marigold.MarigoldDepthOutput`] is returned, otherwise a `tuple` is returned where the first element is the prediction, the second element is the uncertainty (or `None`), and the third is the latent (or `None`). N do_not_padTr\)padding max_length truncationreturn_tensorsr)rddtyperzMarigold predictions...)rr)rdFzDiffusion steps...rt)encoder_hidden_statesrrhcs"g|] }||qSr*)decode_predictionrfr|)rP pred_latentrGr*r+ sz2MarigoldDepthPipeline.__call__..cs0g|]}j|jjfipiqSr*)ensemble_depthr5r6r)rQrUrrGr*r+r0s)is_aar%)rrr )"_execution_devicerr7r8rrEr3model_max_length input_idstor2rF preprocessprepare_latentsrepeatrrangerrr1 set_timesteps timestepsr'catr/step prev_sampleappend unpad_imagereshapezipresize_antialias pt_to_numpymaybe_free_model_hooksr)$rGrJrKrLrMrrNrOrPrQrRrSrTrUrrrdrryprompt text_inputstext_input_idsroriginal_resolution image_latentbatch_empty_text_embedding pred_latentsr|batch_image_latentbatch_pred_latenteffective_batch_sizetextt batch_latentnoiserr*)rPrQrUrrrGr+__call__Ms\                zMarigoldDepthPipeline.__call__cs~ddtjfddtdjdDdd}|jjj}|j|dd}|}|dur;t|j||j |j d}||fS)NcSs,t|dr |jSt|dr|jStd)N latent_distrRz3Could not access latents of provided encoder_output)rrmoderRAttributeError)encoder_outputr*r*r+retrieve_latentshs   z?MarigoldDepthPipeline.prepare_latents..retrieve_latentsc s(g|]}j||qSr*)r0encoderrPrJrrGr*r+rqsz9MarigoldDepthPipeline.prepare_latents..rr)rSrdr) r'rrrrr0rCscaling_factorrepeat_interleaverrdr)rGrJrRrSrLrPrrr*rr+r`s$z%MarigoldDepthPipeline.prepare_latentsrcCs|dks|jd|jjjkrtd|jjjd|jd|jj||jjjddd}|jdd d }t |d d }|d d }|S)Nrarz Expecting 4D tensor of shape [B,z ,H,W]; got r9F)rrTrtkeepdimg?g@) rtrrr0rCrvrjdecoderr_r'clip)rGrrr*r*r+rs  z'MarigoldDepthPipeline.decode_predictionr`{Gz?rMbP?r-r]regularizer_strengthmax_itertolmax_resc  s|dks |jddkrtd|jddvr"tdd s* r*tddtjf fd d dtjd tjd tjf fd d ddtjdtd ttjt tjfffdd d tjdtjd t ffdd dtjf fdd } p } |jd| r| |} || }||d\}} | } r r| }n rd}ntd| |j dd}|||}|r| |} || fS)a Ensembles the depth maps represented by the `depth` tensor with expected shape `(B, 1, H, W)`, where B is the number of ensemble members for a given prediction of size `(H x W)`. Even though the function is designed for depth maps, it can also be used with disparity maps as long as the input tensor values are non-negative. The alignment happens when the predictions have one or more degrees of freedom, that is when they are either affine-invariant (`scale_invariant=True` and `shift_invariant=True`), or just scale-invariant (only `scale_invariant=True`). For absolute predictions (`scale_invariant=False` and `shift_invariant=False`) alignment is skipped and only ensembling is performed. Args: depth (`torch.Tensor`): Input ensemble depth maps. scale_invariant (`bool`, *optional*, defaults to `True`): Whether to treat predictions as scale-invariant. shift_invariant (`bool`, *optional*, defaults to `True`): Whether to treat predictions as shift-invariant. output_uncertainty (`bool`, *optional*, defaults to `False`): Whether to output uncertainty map. reduction (`str`, *optional*, defaults to `"median"`): Reduction method used to ensemble aligned predictions. The accepted values are: `"mean"` and `"median"`. regularizer_strength (`float`, *optional*, defaults to `0.02`): Strength of the regularizer that pulls the aligned predictions to the unit range from 0 to 1. max_iter (`int`, *optional*, defaults to `2`): Maximum number of the alignment solver steps. Refer to `scipy.optimize.minimize` function, `options` argument. tol (`float`, *optional*, defaults to `1e-3`): Alignment solver tolerance. The solver stops when the tolerance is reached. max_res (`int`, *optional*, defaults to `1024`): Resolution at which the alignment is performed; `None` matches the `processing_resolution`. Returns: A tensor of aligned and ensembled depth maps and optionally a tensor of uncertainties of the same shape: `(1, 1, H, W)`. rarz,Expecting 4D tensor of shape [B,1,H,W]; got r9r^Unrecognized reduction method: z1Pure shift-invariant ensembling is not supported.r-cs|djddj}|djddj}r6r6d||jdd}| |}t||f}|SrHd|jdd}|}|St d)Nrrrư>minUnrecognized alignment.) rrvaluesruclampr'rcpunumpyrj)r-init_mininit_maxinit_sinit_tparamrLr5r6r*r+ init_params  z8MarigoldDepthPipeline.ensemble_depth..init_paramrrVcsr0r0t|d\}}t||ddd}t||ddd}|||}|SrFt||ddd}||}|Std)Nrrr)r%splitr' from_numpyrviewrj)r-rsroutrr*r+aligns z3MarigoldDepthPipeline.ensemble_depth..alignF depth_alignedreturn_uncertaintycsd}dkrtj|ddd}|rtj|ddd}||fSdkr=tj|dddj}|r9tjt||dddj}||fStdd)Nr_rTrr`rr9)r'r_stdr`rabsrj)rrrr)r]r*r+ensemblesz6MarigoldDepthPipeline.ensemble_depth..ensemblec sd}||}ttD]\}}||||}||d7}qdkrP|dd\}}d|} d|} || | 7}|S)NgrrFrr) r' combinationsaranger_sqrtitemrrru) rr-costrr|jdiffr_err_nearerr_far)rrrLrr*r+cost_fns z5MarigoldDepthPipeline.ensemble_depth..cost_fncsrddl}|tj}dur t|jddkr t|d}|}|jj t |d|dddd}|j S) NrrrX)r-BFGSF)maxiterdisp)methodroptions) scipyrr'float32rurrrresize_to_max_edgeoptimizeminimizerx)r-rdepth_to_alignrres)rrrrrr*r+ compute_params  z;MarigoldDepthPipeline.ensemble_depth..compute_paramrrrrrN)F)rtrrrjr'r(r%r&boolrrfloatrurr)r-r5r6rUr]rrrrrrequires_aligningrr depth_max depth_min depth_ranger*) rrrrLrrrr]rr5r6rr+rsF.$$    z$MarigoldDepthPipeline.ensemble_depth)NTTNN)NNNT)NrNTrYrYrNNNr%FFT)TTFr`rrrr)&r!r"r#r$model_cpu_offload_seqr=rrr rrr r rstrrintr<rrrr'r(rxrrrno_gradrEXAMPLE_DOC_STRINGrrrr staticmethodrr __classcell__r*r*rHr+r,`s.)     /            %  r,)/ dataclassesr functoolsrtypingrrrrrr rr%r'PILr tqdm.autor transformersr r rFrmodelsrr schedulersrrutilsrrrutils.import_utilsrutils.torch_utilsrpipeline_utilsrmarigold_image_processingr get_loggerr!r>rrr,r*r*r*r+s*