o Gi4@sddlZddlmZddlZddlZddlZddlm Z m Z ddl m Z ddl mZddlmZmZmZmZddlmZd d lmZd d lmZerWddlmmZd ZndZee Z!dZ"eGdddeZ#GdddeZ$dS)N) dataclass)CLIPTextModelWithProjection CLIPTokenizer)PriorTransformer)HeunDiscreteScheduler) BaseOutputis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DiffusionPipeline) ShapERendererTFa Examples: ```py >>> import torch >>> from diffusers import DiffusionPipeline >>> from diffusers.utils import export_to_gif >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> repo = "openai/shap-e" >>> pipe = DiffusionPipeline.from_pretrained(repo, torch_dtype=torch.float16) >>> pipe = pipe.to(device) >>> guidance_scale = 15.0 >>> prompt = "a shark" >>> images = pipe( ... prompt, ... guidance_scale=guidance_scale, ... num_inference_steps=64, ... frame_size=256, ... ).images >>> gif_path = export_to_gif(images[0], "shark_3d.gif") ``` c@s4eZdZUdZeeejjeeejBe d<dS)ShapEPipelineOutputz Output class for [`ShapEPipeline`] and [`ShapEImg2ImgPipeline`]. Args: images (`torch.Tensor`) A list of images for 3D rendering. imagesN) __name__ __module__ __qualname____doc__listPILImagenpndarray__annotations__rr^/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/shap_e/pipeline_shap_e.pyrJs &rcseZdZdZdZdgZdededede de f fdd Z d d Z d d Z eee        d dedededejeejBdBdejdBdedededBdefddZZS)! ShapEPipelinea Pipeline for generating latent representation of a 3D asset and rendering with the NeRF method. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). Args: prior ([`PriorTransformer`]): The canonical unCLIP prior to approximate the image embedding from the text embedding. text_encoder ([`~transformers.CLIPTextModelWithProjection`]): Frozen text-encoder. tokenizer ([`~transformers.CLIPTokenizer`]): A `CLIPTokenizer` to tokenize text. scheduler ([`HeunDiscreteScheduler`]): A scheduler to be used in combination with the `prior` model to generate image embedding. shap_e_renderer ([`ShapERenderer`]): Shap-E renderer projects the generated latents into parameters of a MLP to create 3D objects with the NeRF rendering method. ztext_encoder->priorshap_e_rendererprior text_encoder tokenizer schedulercs"t|j|||||ddS)N)r!r"r#r$r )super__init__register_modules)selfr!r"r#r$r  __class__rrr&os  zShapEPipeline.__init__cCsR|dur t||||d}n|j|krtd|jd|||}||j}|S)N) generatordevicedtypezUnexpected latents shape, got z , expected )r shape ValueErrortoinit_noise_sigma)r(r.r-r,r+latentsr$rrrprepare_latentss   zShapEPipeline.prepare_latentsc Cst|tr t|ndd|j_|j|d|jjddd}|j}|j|dddj}|jd |jd krVt ||sV|j |dd|jjdd f}t d |jjd || ||} | j} | j|dd } | tjj| d dd } |rt| } t| | g} t| jd| } | S)Nrr max_lengthTpt)paddingr4 truncationreturn_tensorslongest)r6r8z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: dim)r<keepdim) isinstancerlenr# pad_token_idmodel_max_length input_idsr.torchequal batch_decodeloggerwarningr"r0 text_embedsrepeat_interleavelinalgnorm zeros_likecatmathsqrt) r(promptr,num_images_per_promptdo_classifier_free_guidance text_inputstext_input_idsuntruncated_ids removed_texttext_encoder_output prompt_embedsnegative_prompt_embedsrrr_encode_prompts< $ zShapEPipeline._encode_promptrN@@pilTrPrQnum_inference_stepsr+r2guidance_scale frame_size output_type return_dictc s\t|trd} nt|trt|} n tdt|j} | |} |dk} || || } jj || djj }j j j }j j j}| ||f| j| ||j}||jd||}t|D]M\}}| rst|gdn|}j||}j ||| dj}|j|jddd\}}| r|d\}}||||}jj|||d j}trtqe |d vrtd ||d krt!|d Sg}|dkrt|D]\}}j"#|dddf| }|$|qn6t|D]\}}j"j%|dddf| |d}|$|qt&|}|'(}|dkr#fdd|D}| s)|fSt!|d S)aq The call function to the pipeline for generation. Args: prompt (`str` or `list[str]`): The prompt or prompts to guide the image generation. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. num_inference_steps (`int`, *optional*, defaults to 25): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. guidance_scale (`float`, *optional*, defaults to 4.0): A higher guidance scale value encourages the model to generate images closely linked to the text `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`. frame_size (`int`, *optional*, default to 64): The width and height of each image frame of the generated 3D output. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated image. Choose between `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`), `"latent"` (`torch.Tensor`), or mesh ([`MeshDecoderOutput`]). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.shap_e.pipeline_shap_e.ShapEPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated images. rz2`prompt` has to be of type `str` or `list` but is g?)r,rr )timestepproj_embeddingr;)rdsample)rr^latentmeshzUOnly the output types `pil`, `np`, `latent` and `mesh` are supported not output_type=rg)rrhN)sizer^csg|]}|qSr) numpy_to_pil).0imager(rr Tsz*ShapEPipeline.__call__..))r>strrr?r/type_execution_devicerZr$ set_timesteps timestepsr!confignum_embeddings embedding_dimr3r-reshaper. enumerate progress_barrCrMscale_model_inputpredicted_image_embeddingsplitchunkstep prev_sample XLA_AVAILABLExm mark_stepmaybe_free_model_hooksrr decode_to_meshappenddecode_to_imagestackcpunumpy)r(rPrQr_r+r2r`rarbrc batch_sizer,rRrXrsrurvitlatent_model_inputscaled_model_input noise_pred_noise_pred_uncondrrgrhrlrrmr__call__s 4                zShapEPipeline.__call__)rr[NNr\r]r^T)rrrrmodel_cpu_offload_seq_exclude_from_cpu_offloadrrrrrr&r3rZrCno_gradr EXAMPLE_DOC_STRINGroint GeneratorrTensorfloatboolr __classcell__rrr)rrWs\ 1  r)%rN dataclassesrrr PIL.ImagerrC transformersrrmodelsr schedulersrutilsrr r r utils.torch_utilsr pipeline_utilsrrendererrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrrFrrrrrrrs*