o ۷id@s2ddlZddlZddlZddlZddlmZmZddlmZddl m Z m Z m Z ddl mZmZmZmZddlmZdd lmZmZeeZd ed efd d Zidddddgddddddgddddddgddddddgddddddgddddddgdddddd d!ddddd"d#gd$gd%dd$gd&d'd(d'dd d)d*d)dd)gdd+d,d-d.d/d+gd0d1d2d-d.d/d1gd0d3d4d5d6d7d8d9d:d;d$dd8d?d$gd@d"dAdBdCdDdEd"gdFd#dGdBdCdDdEd#gdFdHdIdBdddJdKdLgdMidNdOdBdPddQd6dNgdMdRdSdBdTddUd6dRgdMdVdWdBdXddYd6dVgdMdZd[d\d]gd^d_d`dad5d=d>d8d?d`gd@dbdcd5d>d>d8d?dbgd@ddded5d8d>d8d?ddgd@dfdgdhddfgddidjdkddigddldmdkddlgddndodkdd dpdqdkdd drdsdkddrgddtdudvddwgddxdydzdd{dd|d}d~dd ddddd3ggddddd-dd iZGdddeZeddGdddedZddd efddZ ddd efddZ!dej"dde#e$e%e&e'e(e)e*e+e,ej-ddg e.e/e0gej1dde2e3gd'gdndpgdde4ge5gej+dde3gdgdigdde6e7ge8gej"dde3gd+gdtgdwdej1ddge5ge9e:e3gdgdigdddZ;ddeded efddZGdddZ?dS)N)asdict dataclass)Any) create_repohf_hub_download upload_file)EntryNotFoundErrorHfHubHTTPErrorRepositoryNotFoundErrorRevisionNotFoundError)HUGGINGFACE_CO_RESOLVE_ENDPOINT) InputParam OutputParamnamereturncCs|ddS)z,Convert snake_case name to Title Case label._ )replacetitlerrc/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/modular_pipelines/mellon_node_utils.py_name_to_labelsrimageImageinput)labeltypedisplayrequired_block_paramsimagesImagesoutput control_imagez Control ImagelatentsLatents image_latentsz Image Latentsfirst_frame_latentszFirst Frame Latentslatents_previewzLatents Previewlatent)rrr image_latents_with_strengthheightwidthstrength)falsetrue)rrrr onChanger! embeddingszText Embeddings image_embedszImage EmbeddingspromptPromptstringtextarea)rrr defaultr!negative_promptzNegative Promptguidance_scalezGuidance Scalefloatsliderg@g?g>@g?)rrr r:minmaxstepStrengthg?g{Gz?)rrr:r?r@rAr!Heightinti@)rrr:r?rAr!WidthseedSeedrandom generator)rrr:r?r@r r!num_inference_stepsStepsd num_framesFramesQilayersLayers output_typez Output Typedropdownnp)r[pilpt)rrr:optionscontrolnet_conditioning_scalezControlnet Conditioning Scalecontrol_guidance_startzControl Guidance Startcontrol_guidance_endzControl Guidance EndvideosVideosvideovaeVAEdiffusers_auto_model image_encoderz Image Encoderunetz Denoise Model scheduler Scheduler controlnetzControlNet Model text_encodersz Text Encodersdiffusers_auto_models text_encodercontrolnet_bundle ControlNetcustom_controlnetcontrolnet_image ip_adapterz IP Adaptercustom_ip_adapterguiderGuider custom_guider)FT)rrr r2docDocc@seZdZdZdefddZdS)MellonParamMetazEMetaclass that enables MellonParam.template_name(**overrides) syntax.rcs,tvrdfdd }|Stdd)Ncs>t}d|di||}|dur||d<di|S)Nrr:r)MELLON_PARAM_TEMPLATESget)r: overridestemplateparamsclsrrrfactorys z,MellonParamMeta.__getattr__..factoryz,type object 'MellonParam' has no attribute ''N)r|AttributeError)rrrrrr __getattr__szMellonParamMeta.__getattr__N)__name__ __module__ __qualname____doc__strrrrrrr{sr{T)frozenc@seZdZUdZeed<eed<eed<dZedBed<dZeed<dZ e dBed<dZ e dBed <dZ e dBed <dZ eed <dZeed <dZeeefdBed <dZeed<dZeed<dZeeeBdBed<deeeffddZGdddZGdddZdS) MellonParamat Parameter definition for Mellon nodes. Usage: ```python # From template (standard diffuser params) MellonParam.seed() MellonParam.prompt(default="a cat") MellonParam.latents(display="output") # Generic inputs (for custom blocks) MellonParam.Input.slider("my_scale", default=1.0, min=0.0, max=2.0) MellonParam.Input.dropdown("mode", options=["fast", "slow"]) # Generic outputs MellonParam.Output.image("result_images") # Fully custom MellonParam(name="custom", label="Custom", type="float", default=0.5) ``` rrrNr r:r?r@rAr^value fieldOptionsr2onSignalr!rcCst|}dd|DS)zMConvert to dict for Mellon schema, excluding None values and internal fields.cSs&i|]\}}|dur|dvr||qS)N)rr!r).0kvrrr (s&z'MellonParam.to_dict..)ritems)selfdatarrrto_dict%szMellonParam.to_dictc@s"eZdZdZededdfddZed"dededdfd d Zed#ded eededdfd dZ e d$dede de de de ddf ddZ e d$dede de de de ddf ddZ ed%dede ddfddZed&dededdfddZedededdfddZededdfd d!Zd S)'zMellonParam.Inputz$input UI elements for custom blocks.rrrcCt|t|dddS)z image input.rrrrrr rrrrrrr0zMellonParam.Input.imager9r:cCst|t|dd|dS)ztext input as textarea.r7r8)rrrr r:rrrr:rrrtextbox5szMellonParam.Input.textboxNr^cCs8|r|s|d}|s d}|s|g}t|t|d||dS)zdropdown selection.rr9r7)rrrr^rr)rrr^r:rrrrZ<szMellonParam.Input.dropdownrr?r@rAc Cspt|tp |duo t|t}|rdnd}|dur|}|dur |}|dur*|r(dnd}t|t||d||||dS)z slider input.Nr=rErCrr>)rrrr r:r?r@rA isinstancer=rrrrr:r?r@rAis_float param_typerrrr>Gs$  zMellonParam.Input.sliderc CsBt|tp |duo t|t}|rdnd}t|t||||||dS)znumber input (no slider).Nr=rE)rrrr:r?r@rArrrrrnumber_s  zMellonParam.Input.numberrIc Cst|t|dd|dddS)z!seed input with randomize button.rErLrrK)rrrr r:r?r@rrrrrrIjszMellonParam.Input.seedFcCst|t|d|dS)zboolean checkbox.boolean)rrrrrrrrrcheckboxwrzMellonParam.Input.checkboxrcCt|t||ddS)z'custom type input for node connections.rrrrrrrrr custom_type|rzMellonParam.Input.custom_typecCr)z%model input for diffusers components.rgrrrrrrrmodelrzMellonParam.Input.model)r9)NN)rNNN)rIrF)rrrr classmethodrrrlistrZr=r>rrErIboolrrrrrrrInput-s\"     rc@seZdZdZededdfddZededdfddZededdfd d Zeded eddfd d Z ededdfddZ dS)zMellonParam.Outputz%output UI elements for custom blocks.rrrcCr)z image output.rr$rrrrrrrrzMellonParam.Output.imagecCr)z video output.rdr$rrrrrrrdrzMellonParam.Output.videocCr)z text output.r7r$rrrrrrtextrzMellonParam.Output.textrcCr)z(custom type output for node connections.r$rrrrrrrrzMellonParam.Output.custom_typecCr)z&model output for diffusers components.rgr$rrrrrrrrzMellonParam.Output.modelN) rrrrrrrrdrrrrrrrOutputsr)rrrrr__annotations__r r:rr?r=r@rAr^rrdictr2rr!rrrrrrrrrs&      \r) metaclass input_paramrcCs|j}|j}|r |dnd}|j}t|tr|S|}|dkr%tj|S|dkr3tjj||p0ddS|dkrAtjj ||p>ddS|dkrOtjj ||pLd dS|d kr]tjj ||pZd dS|d krktjj ||phd dS|d krytjj ||pvd dS|dkrtj|Stjj|ddS)ah Convert an InputParam to a MellonParam using metadata. Args: input_param: An InputParam with optional metadata containing either: - {"mellon": ""} for simple types (image, textbox, slider, etc.) - {"mellon": MellonParam(...)} for full control over UI configuration Returns: MellonParam instance mellonNrrr9)r:rZr>rrrIrFrcustomr)rmetadatar}r:rrrrrrZr>rrIrrr)rrr mellon_valuer: mellon_typerrrinput_param_to_mellon_params0    r output_paramrcCs~|j}|j}|r |dnd}|dkrtj|S|dkr#tj|S|dkr-tj|S|dkr7tj|Stjj |ddS) a7 Convert an OutputParam to a MellonParam using metadata. Args: output_param: An OutputParam with optional metadata={"mellon": ""} where type is one of: image, video, text, model. If metadata is None or unknown, maps to "custom". Returns: MellonParam instance rNrrdrrrr) rrr}rrrrdrrr)rrrrrrroutput_param_to_mellon_params     r)r denoiseinputs model_inputsoutputsrequired_inputsrequired_model_inputs block_name vae_encoderdecode)rlrrrodecoder *rmarkercCs||r|S||S)z4Add required marker to label if not already present.)endswith)rrrrr mark_required?s  r node_spec node_typec Csi}g}g}g}|dg}|dg}|dgD]}|} |j|vr-t| d| d<| ||j<||jq|dgD]}|} |j|vrRt| d| d<| ||j<||jq?|dgD]}|j|vrrd|j} n|j} ||| <|| qd|||||d|d S) a Convert a node spec dict into Mellon format. A node spec is how we define a Mellon diffusers node in code. This function converts it into the `params` map format that Mellon UI expects. The `params` map is a dict where keys are parameter names and values are UI configuration: ```python {"seed": {"label": "Seed", "type": "int", "default": 0}} ``` For Modular Mellon nodes, we need to distinguish: - `inputs`: Pipeline inputs (e.g., seed, prompt, image) - `model_inputs`: Model components (e.g., unet, vae, scheduler) - `outputs`: Node outputs (e.g., latents, images) The node spec also includes: - `required_inputs` / `required_model_inputs`: Which params are required (marked with *) - `block_name`: The modular pipeline block this node corresponds to on backend We provide factory methods for common parameters (e.g., `MellonParam.seed()`, `MellonParam.unet()`) so you don't have to manually specify all the UI configuration. Args: node_spec: Dict with `inputs`, `model_inputs`, `outputs` (lists of MellonParam), plus `required_inputs`, `required_model_inputs`, `block_name`. node_type: The node type string (e.g., "denoise", "controlnet") Returns: Dict with: - `params`: Flat dict of all params in Mellon UI format - `input_names`: List of input parameter names - `model_input_names`: List of model input parameter names - `output_names`: List of output parameter names - `block_name`: The backend block name - `node_type`: The node type Example: ```python node_spec = { "inputs": [MellonParam.seed(), MellonParam.prompt()], "model_inputs": [MellonParam.unet()], "outputs": [MellonParam.latents(display="output")], "required_inputs": ["prompt"], "required_model_inputs": ["unet"], "block_name": "denoise", } result = node_spec_to_mellon_dict(node_spec, "denoise") # Returns: # { # "params": { # "seed": {"label": "Seed", "type": "int", "default": 0}, # "prompt": {"label": "Prompt *", "type": "string", "default": ""}, # * marks required # "unet": {"label": "Denoise Model *", "type": "diffusers_auto_model", "display": "input"}, # "latents": {"label": "Latents", "type": "latents", "display": "output"}, # }, # "input_names": ["seed", "prompt"], # "model_input_names": ["unet"], # "output_names": ["latents"], # "block_name": "denoise", # "node_type": "denoise", # } ``` rrrrrrout_r)r input_namesmodel_input_names output_namesrr)r}rrrappend) rrrrrrrrp param_dict output_namerrrnode_spec_to_mellon_dictFs>B         rc @seZdZdZdZ   d-deeeeefdBfdededefd d Ze d eeeffd d Z d efddZ d eeeffddZ e deeefd dfddZd efddZdeejBfddZe deejBd dfddZd.deejBdefdd Ze d!eejBd dfd"d#Ze    $d/d%eeeeeffdBdededed df d&d'Ze   d0d(ed)eeefdBd*eeefdBd dfd+d,ZdS)1MellonPipelineConfiga Configuration for an entire Mellon pipeline containing multiple nodes. Accepts node specs as dicts with inputs/model_inputs/outputs lists of MellonParam, converts them to Mellon-ready format, and handles save/load to Hub. Example: ```python config = MellonPipelineConfig( node_specs={ "denoise": { "inputs": [MellonParam.seed(), MellonParam.prompt()], "model_inputs": [MellonParam.unet()], "outputs": [MellonParam.latents(display="output")], "required_inputs": ["prompt"], "required_model_inputs": ["unet"], "block_name": "denoise", }, "decoder": { "inputs": [MellonParam.latents(display="input")], "outputs": [MellonParam.images()], "block_name": "decoder", }, }, label="My Pipeline", default_repo="user/my-pipeline", default_dtype="float16", ) # Access Mellon format dict denoise = config.node_params["denoise"] input_names = denoise["input_names"] params = denoise["params"] # Save to Hub config.save("./my_config", push_to_hub=True, repo_id="user/my-pipeline") # Load from Hub loaded = MellonPipelineConfig.load("user/my-pipeline") ``` zmellon_pipeline_config.jsonr9 node_specsNr default_repo default_dtypecCs||_||_||_||_dS)a Args: node_specs: Dict mapping node_type to node spec or None. Node spec has: inputs, model_inputs, outputs, required_inputs, required_model_inputs, block_name (all optional) label: Human-readable label for the pipeline default_repo: Default HuggingFace repo for this pipeline default_dtype: Default dtype (e.g., "float16", "bfloat16") Nrrrr)rrrrrrrr__init__s zMellonPipelineConfig.__init__rcCsL|jdur|jSi}|jD]\}}|durd||<qt||||<q|S)z+Lazily compute node_params from node_specs.N)r _node_paramsrr)rrrspecrrr node_paramss  z MellonPipelineConfig.node_paramscCsd|jd|jd|jdg}|jD]T\}}|dur'|d|dqdd|d gD}d d|d gD}d d|d gD}|d|d|d||d||d|qd|S)NzMellonPipelineConfig(label=z, default_repo=z, default_dtype=)z z: NonecSg|]}|jqSrrrrrrr z1MellonPipelineConfig.__repr__..rcSrrrrrrrrrrcSrrrrrrrrrr:z inputs: z model_inputs: z outputs:  )rrrrrrr}join)rlinesrrrrrrrr__repr__s zMellonPipelineConfig.__repr__cCs|j|j|j|jdS)z*Convert to a JSON-serializable dictionary.rrrrrrrrrrs zMellonPipelineConfig.to_dictrcCsL||}d|_|di|_|dd|_|dd|_|dd|_|S)z Create from a dictionary (loaded from JSON). Note: The mellon_params are already in Mellon format when loading from JSON. Nrrr9rr)__new__rr}rrrr)rrinstancerrr from_dict s zMellonPipelineConfig.from_dictcCstj|ddddS)zSerialize to JSON string.r F)indent sort_keysr)jsondumpsrrrrrto_json_string/sz#MellonPipelineConfig.to_json_stringjson_file_pathcCsBt|ddd}||WddS1swYdS)zSave to a JSON file.wutf-8encodingN)openwriter)rrwriterrrr to_json_file3s"z!MellonPipelineConfig.to_json_filecCsBt|ddd }t|}Wdn1swY||S)zLoad from a JSON file.rrrN)rrloadr)rrreaderrrrrfrom_json_file8s  z#MellonPipelineConfig.from_json_fileFsave_directory push_to_hubc Kstj|rtd|dtj|ddtj||j}||t d||rq| dd}| dd}| d d }| d d}| d | tjj d } t | d||dj} t||j| ||pcd|dt d| dSdS)z/Save the mellon pipeline config to a directory.zProvided path (z#) should be a directory, not a fileT)exist_okzPipeline config saved to commit_messageNprivate create_prFtokenrepo_id)rrr zUpload MellonPipelineConfig)path_or_fileobj path_in_repor r rr zPipeline config pushed to hub: )ospathisfileAssertionErrormakedirsr config_namerloggerinfopopsplitseprr r) rrrkwargs output_pathrrr r r rrrsave?s.      zMellonPipelineConfig.savepretrained_model_name_or_pathc Ks|dd}|dd}|dd}|dd}|dd}|d d}|d d} |d d} |d d} t|}tj|rC|} ntj|rctj||j} tj| sbtd |jd|nzt ||j|||| || | ||d } Wnnt yt|dt yt| d|dt yt|d|jdt y} z td|d| d} ~ wtytdtd|d|jdtytd|d|d|jdwz|| WStjtfytd| d w)!z=Load a pipeline config from a local path or Hugging Face Hub. cache_dirN local_dirlocal_dir_use_symlinksautoforce_downloadFproxiesr local_files_onlyrevision subfolderzNo file named z found in ) filenamerr"r#r$r r%r&rr z is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo with `token` or log in with `hf auth login`.z is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/z' for available revisions.z& does not appear to have a file named .z:There was a specific connection error when trying to load z: zWe couldn't connect to 'zM' to load this model, couldn't find it in the cached files and it looks like z- is not the path to a directory containing a z file. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/diffusers/installation#offline-mode'.zCan't load config for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filezThe config file at 'z' is not a valid JSON file.)rrrrrisdirrrEnvironmentErrorrr r rr ValueErrorr rrJSONDecodeErrorUnicodeDecodeError)rrrrrr r"r#r r$r%r& config_fileerrrrrr[s                  zMellonPipelineConfig.loadbfloat16rc Cs|durt}t|j}dtttfdtttfdBfdd}i}|D](\} } | dur1d|| <q$| d} | dus>| |vrCd|| <q$|| || || <q$|||pVt|dd||d S) zb Create MellonPipelineConfig by matching template against actual pipeline blocks. N template_specrcst|jt|jt|jfdd|dgD}fdd|dgD}fdd|dgD}dd |Dd d |Dfd d|d gD}fd d|dgD}||||||ddS)zFFilter template spec params based on what the block actually supports.c2g|]}|jdustfdd|jDr|qS)Nc3|]}|vVqdSrrrrblock_input_namesrr ]MellonPipelineConfig.from_blocks..filter_spec_for_block...r!allrr5rrr zSMellonPipelineConfig.from_blocks..filter_spec_for_block..rcr2)Nc3r3rrr4block_component_namesrrr7r8r9r:rr=rrrr<rcr2)Nc3r3rrr4block_output_namesrrr7r8r9r:rr?rrrr<rcSh|]}|jqSrrrrrr rzRMellonPipelineConfig.from_blocks..filter_spec_for_block..cSrArrrrrrrBrcg|]}|vr|qSrrrr)filtered_input_namesrrrrcrCrrrD)filtered_model_input_namesrrrrFrrr)setrintermediate_output_namescomponent_namesr})r1blockfiltered_inputsfiltered_model_inputsfiltered_outputsfiltered_required_inputsfiltered_required_model_inputsr)r>r6r@rErGrfilter_spec_for_blocks6             z?MellonPipelineConfig.from_blocks..filter_spec_for_blockr model_namer9r)DEFAULT_NODE_SPECSr sub_blocksrrrr}getattr) rblocksrrrr sub_block_maprQrrr1rrrr from_blockss(  &- z MellonPipelineConfig.from_blocks node_label input_types output_typescCsJ|dur|jj}ddd|D}|duri}|dur i}g}g}g}|jD]-} | jdur1q)| j|vrCt| } d|| ji| _td| jd| j| t | q)|j D]!} | jdurbqZ| j|vrtt| } d|| ji| _| t | qZ|j } | D] } | tj| q| t|||ggdd } |d| i|d S) a Create a MellonPipelineConfig from a custom block. Args: block: A block instance with `inputs`, `outputs`, and `expected_components`/`component_names` properties. Each InputParam/OutputParam should have metadata={"mellon": ""} where type is one of: image, video, text, checkbox, number, slider, dropdown, model. If metadata is None, maps to "custom". node_label: The display label for the node. Defaults to block class name with spaces. input_types: Optional dict mapping input param names to mellon types. Overrides the block's metadata if provided. Example: {"prompt": "textbox", "image": "image"} output_types: Optional dict mapping output param names to mellon types. Overrides the block's metadata if provided. Example: {"prompt": "text", "images": "image"} Returns: MellonPipelineConfig instance Nr9cSs g|] }|r d|n|qS)r)isupper)rcrrrrs z:MellonPipelineConfig.from_custom_block..rz processing input: z , metadata: rr)rr) __class__rrstriprrcopyrprintrrrrrJrrrry)rrKrYrZr[ class_namerrrrrrJcomponent_namerrrrfrom_custom_blocksP         z&MellonPipelineConfig.from_custom_block)r9r9r9r)Nr9r9r0)NNN)rrrrrrrrrpropertyrrrrrrrPathLikerrrrrrXrdrrrrrsz*    Q Qr)r)@r`rloggingr dataclassesrrtypingrhuggingface_hubrrrhuggingface_hub.utilsrr r r utilsr modular_pipeline_utilsrr getLoggerrrrrr|rr{rrrr3r.r-rIrNrRr<r/r,r(r)rprirvrjr&r*ryrrer5r;rmr"rbrSrrrrrrrrs       #$+2:CLT\enw$&' - ./5< BC  IM(,      Q&n