o ۷i`@s ddlZddlZddlZddlmZddlmZmZddlm Z ddl m Z m Z m Z mZmZmZddlZddlZddlmZmZddlmZmZdd lmZdd lmZmZmZdd l m!Z!erb e"e#Z$d Z%Gd ddeZ&eGdddZ'eGdddZ(ide)dddde)ddde*dddde*ddd e*d!dd"e*d#d$dd%e*d&d'dd(ej+d)dd*e,e-d+dd,e-d-d.dd/ej.j.e,ej.j.Bdd0dd1ej/d2dd3ej/d4dd5e)d6d7dd8e0e)e fd9dd:dd:d;d<d=ej.j.dd>de*d?dej.j.dd@de-dAdBde-dCdDde-dCdEde*dFdGdej/ddHdej/ddIdej/dJdej/dKdej/ddLde*d&dMdej1ej2dNddO Z3e,ej.j.dPde,ej.j.dQdej/dRdej/d:dSdTej/d:dUdTej/d:dVdTej/d:dWdTej/dXddYZ4eGdZd[d[Z5eGd\d]d]Z6d^d_Z7d`daZ8ddddeZ9ddfdgZ:ddhdiZ;ddkdlZdqdrZ? s   ddtduZ@dvdwZAdxdyZBdze,eCe)e,e5fd{e,e5fd|d}ZDd~e,eCe)e,e6fd{e,e6fddZEd{e0e)e ffddZFdS)N) OrderedDict) dataclassfield) UnionType)AnyLiteralTypeUnionget_args get_origin)InvalidSpecifier SpecifierSet) ConfigMixin FrozenDict)_is_single_file_path_or_url)DIFFUSERS_LOAD_ID_FIELDSis_torch_availablelogging)_is_package_availablea{model_description} ## Example Usage [TODO] ## Pipeline Architecture This modular pipeline is composed of the following blocks: {blocks_description} {trigger_inputs_section} ## Model Components {components_description} {configs_section} {io_specification_section} c@seZdZddZddZdS)InsertableDictcsDt|}fdd|D}|||f||||S)Ncs g|] \}}|kr||fqSr.0kvkeyrh/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/modular_pipelines/modular_pipeline_utils.py @ z)InsertableDict.insert..)listitemsinsertclearupdate)selfrvalueindexr"rrrr#<s  zInsertableDict.insertc Cs|sdSg}t|D]4\}\}}t|tr#d|jd|jd}n d|jjd|jjd}||dt|d|dq d d |d S) NzInsertableDict()zz.cSsg|] }|dur dn|qS)NrWrrprrrr|)rHrer4)r&partsrrPrrOs  zComponentSpec.load_idrOcCsb|}t|}|dkr|S|d}t|D]\}}|t|kr.|dkr(dn||||<q|S)a Decode a load_id string back into a dictionary of loading fields and values. Args: load_id: The load_id string to decode, format: "pretrained_model_name_or_path|subfolder|variant|revision" where None values are represented as "null" Returns: Dict mapping loading field names to their values. e.g. { "pretrained_model_name_or_path": "path/to/repo", "subfolder": "subfolder", "variant": "variant", "revision": "revision" } If a segment value is "null", it's replaced with None. Returns None if load_id is "null" (indicating component not created with `load` method). rWrkN)redictfromkeyssplitr,len)rarOreresultrlr5partrrrr`s   zComponentSpec.decode_load_idcKs|jdus t|jtstd|p|jpi}t|jtr'|jj|fi|}n4t |jj j }i}| D] \}}||vrA|||<q5| D] \}}||vrR|||<qF|jdi|}d|_ t|drg|j|_|S)z/Create component using from_config with config.Nz?`type_hint` is required when using from_config creation method.rWr=r)r;r-r.r]r= issubclassrrGinspect signature__init__ parametersr"rVrY)r&r=kwargsrTsignature_params init_kwargsrrrrrcreate s(  zComponentSpec.createc sfddDfddD}|dd}|dur%tdt|}|r8jdur8tdjjdurKtjtjj sKddjdurzd d l m }|j |fi|}Wnt yz}z td jd |d}~ww|j_n@|rtjd ntjd}tjtjj sddz ||fi|}Wnt y}z td jd|d}~ww|_|D] \} } t| | qLjj|_|S)z%Load component using from_pretrained.cs i|] }|vr||qSr)poprr)rxrr )r z&ComponentSpec.load..c s i|] }||t|qSr)getrfr})passed_loading_kwargsr&rrr~+r rANz`pretrained_model_name_or_path` info is required when using `load` method (you can directly set it in `pretrained_model_name_or_path` field of the ComponentSpec or pass it as an argument)zW`type_hint` is required when loading a single file model but is missing for component: torch_dtyper) AutoModelzUnable to load z without `type_hint`: from_single_filerFz using load method: )rer|r]rr;r:rsrZr[r\ diffusersrrF Exceptionr1rfrAr"setattrrOrV) r&rx load_kwargsrAis_single_filerrTe load_methodrrr)rxrr&rload&sP        zComponentSpec.loadN)"r0r/r8__doc__r:str__annotations__r;rr<r=rrrAr!rCrDrErHrrIrMrQrS classmethodrrcrepropertyrOrmr`r{rrrrrr9`s2 $$ ; $&r9c@s2eZdZUdZeed<eed<dZedBed<dS) ConfigSpecz5Specification for a pipeline configuration parameter.r:r?Nr<)r0r/r8rrrrr<rrrrrbs rpromptTz0The prompt or prompts to guide image generation.)r;requiredr<negative_promptz8The prompt or prompts not to guide the image generation.)r;r<max_sequence_lengthiz,Maximum sequence length for prompt encoding.)r;r?r<heightz,The height in pixels of the generated image.widthz+The width in pixels of the generated image.num_inference_steps2zThe number of denoising steps.num_images_per_promptz,The number of images to generate per prompt. generatorz-Torch generator for deterministic generation.sigmasz(Custom sigmas for the denoising process.strengthg?z Strength for img2img/inpainting.imagezJReference image(s) for denoising. Can be a single image or list of images.latentsz1Pre-generated noisy latents for image generation. timestepsz$Timesteps for the denoising process. output_typepilz!Output format: 'pil', 'np', 'pt'.attention_kwargsz+Additional kwargs for attention processors.denoiser_input_fieldsz[conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.)r: kwargs_typer< mask_imagezMask image for inpainting.z(Padding for mask cropping in inpainting.z*Control image for ControlNet conditioning.gz"When to start applying ControlNet.g?z!When to stop applying ControlNet.z"Scale for ControlNet conditioning.z*Number of layers to extract from the imagez\text embeddings used to guide the image generation. Can be generated from text_encoder step.zFmask for the text embeddings. Can be generated from text_encoder step.zenegative text embeddings used to guide the image generation. Can be generated from text_encoder step.zOmask for the negative text embeddings. Can be generated from text_encoder step.zYimage latents used to guide the image generation. Can be generated from vae_encoder step.zNumber of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt. Can be generated in input step.z>The dtype of the model inputs, can be generated in input step.) padding_mask_crop control_imagecontrol_guidance_startcontrol_guidance_endcontrolnet_conditioning_scalelayers prompt_embedsprompt_embeds_masknegative_prompt_embedsnegative_prompt_embeds_mask image_latents batch_sizedtypezGenerated images.zThe generated videos.zDenoised latents.zThe prompt embeddings.)r;rr<zThe encoder attention mask.zThe negative prompt embeddings.z$The negative prompt embeddings mask.z-The latent representation of the input image.)imagesvideosrrrrrrc@seZdZUdZdZeed<dZeed<dZ eed<dZ e ed<dZ eed <dZ eed <dZeeefed <d d ZeddededdfddZdS) InputParamz%Specification for an input parameter.Nr:r;r?FrrBr<rr@cCs&d|jd|jr dndd|jdS)N<: roptionalz , default=>)r:rr?rPrrrr75s&zInputParam.__repr__ template_namenoterUcKx|tvr td|dt|}|d|d|}|r.d|vr.|dd|d|d<|||dd|i|S) >> inputs = [ ... InputParam(name="prompt", required=True), ... InputParam(name="image", required=True), ... InputParam(name="guidance_scale", required=False, default=7.5), ... InputParam(name="num_inference_steps", required=False, default=50) ... ] >>> format_inputs_short(inputs) 'prompt, image, guidance_scale=7.5, num_inference_steps=50' cSsg|]}|jr|qSrrrparamrrrrrgz'format_inputs_short..cSsg|]}|js|qSrrrrrrrrgr*css|]}|jVqdSrr:rrrr sz&format_inputs_short..css"|] }|jd|jVqdS)=N)r:r?rrrrrs )r4)inputsrequired_inputsoptional_inputs required_str optional_str inputs_strrrrformat_inputs_shortqsrc Csg}|D]*}|j|vr|d|jdq|jdur&|jdur&d|j}n|j}||qdd|D}g}g}|D]} | j|vrJ|| jq<|| jqsz-format_intermediates_short..z - inputs: r*z - modified: z - outputs:  z (none))r:r2rr4) intermediate_inputsrequired_intermediate_inputsintermediate_outputs input_partsrinp_name inputs_setmodified_partsnew_output_partsoutrqrrrformat_intermediates_shorts.    rArgsscCs@|sdSd|}d|d}d|d}g}dd}dd} |||d |D]p} | jtkr6|| jnd} | jd urH| jd urHd | jn| j} || d | d } t| drn| jsn| d7} | jd urn| d| j7} | d7} | jrt dd| j}| |||}| d||7} n| d|d7} || q*d |S)aFormat a list of InputParam or OutputParam objects into a readable string representation. Args: params: list of InputParam or OutputParam objects to format header: Header text to use (e.g. "Args" or "Returns") indent_level: Number of spaces to indent each parameter line (default: 4) max_line_length: Maximum length for each line before wrapping (default: 115) Returns: A formatted string representing all parameters rB rcSJt|ts t|turddt|D}d|St|dr!|jSt|S)NcS$g|]}t|dr |jnt|qSr0rYr0rrtrrrr$z7format_params..get_type_str.. | r0 r-rr r r r4rYr0rr; type_strsrrr get_type_str z#format_params..get_type_strc Ss|}g}g}d}|D]-}t||rdnd}|r0|||kr0|d||g}t|}q ||||7}q |rD|d|d||S)zFWrap text while preserving markdown links and maintaining indentation.rrrr)rorpr2r4) textindent max_lengthwordslines current_linecurrent_lengthword word_lengthrrr wrap_texts   z format_params..wrap_text:N** (``r , *optional*z, defaults to z):z \[(.*?)\]\((https?://[^\s\)]+)\)z[\1](\2)rzTODO: Add description.) r2r;rr:rrYrr?r<resubr4)paramsheader indent_levelmax_line_length base_indent param_indent desc_indentformatted_paramsrrrtype_strr: param_strdesc wrapped_descrrr format_paramss4   &     rcCt|d||S)aFormat a list of InputParam objects into a readable string representation. Args: input_params: list of InputParam objects to format indent_level: Number of spaces to indent each parameter line (default: 4) max_line_length: Maximum length for each line before wrapping (default: 115) Returns: A formatted string representing all input parameters Inputsr) input_paramsr r rrrformat_input_params rcCr)aFormat a list of OutputParam objects into a readable string representation. Args: output_params: list of OutputParam objects to format indent_level: Number of spaces to indent each parameter line (default: 4) max_line_length: Maximum length for each line before wrapping (default: 115) Returns: A formatted string representing all output parameters Outputsr) output_paramsr r rrrformat_output_paramsrrrc Cs|sdSdd}|rd|dgng}|D]\}|jtkr!||jnd}|jdur3|jdur3d|jn|j}d|d|d }t|d rY|jsY|d 7}|jdurY|d |jd 7}|d 7}|jrc|jnd}|d|7}||qd |S)asFormat a list of InputParam or OutputParam objects as a markdown bullet-point list. Suitable for model cards rendered on Hugging Face Hub. Args: params: list of InputParam or OutputParam objects to format header: Header text (e.g. "Inputs" or "Outputs") Returns: A formatted markdown string, or empty string if params is empty. rBcSr)NcSrrrrrrrr=rz@format_params_markdown..get_type_str..rr0rrrrrr;rz,format_params_markdown..get_type_strrz:** Nz- `z` (`rrrz, defaults to `r+zNo description providedrr) r;rr:rrYrr?r<r2r4) r r rrrrr:rrrrrformat_params_markdown,s" &   r cCs|sdSd|}d|d}g}||d|r|dt|D]d\}}t|jdr1|jjnt|j} ||jd| d} |jrL| d|j7} g} |D]} t || } | re| | d | qR| rs| d d | d 7} || |r|t |d kr|dq#d |S)aFormat a list of ComponentSpec objects into a readable string representation. Args: components: list of ComponentSpec objects to format indent_level: Number of spaces to indent each component line (default: 4) max_line_length: Maximum length for each line before wrapping (default: 115) add_empty_lines: Whether to add empty lines between components (default: True) Returns: A formatted string representing all components rBrrz Components:r0r`)rrz [r*]rr) r2r,rYr;r0rr:r<rerfr4rp) componentsr r add_empty_linesrcomponent_indentformatted_componentsr5rT type_namecomponent_descloading_field_values field_name field_valuerrrformat_componentsTs6        r,c Cs|sdSd|}d|d}g}||d|r|dt|D]/\}}||jd|jd} |jr>| d|j7} || |rR|t|dkrR|dq#d |S) aFormat a list of ConfigSpec objects into a readable string representation. Args: configs: list of ConfigSpec objects to format indent_level: Number of spaces to indent each config line (default: 4) max_line_length: Maximum length for each line before wrapping (default: 115) add_empty_lines: Whether to add empty lines between configs (default: True) Returns: A formatted string representing all configs rBrrzConfigs:z (default: r+rrr)r2r,r:r?r<rpr4) configsr r r$r config_indentformatted_configsr5r= config_descrrrformat_configss"      r1cCs|durdSdg}|D].\}}dd|D}|r2ddd|D}|d |d |q |d |d q d |S) zFormat a workflow map into a readable string representation. Args: workflow_map: Dictionary mapping workflow names to trigger inputs Returns: A formatted string representing all workflows NrBzSupported workflows:cSsg|]\}}|r|qSrrrrrrrz#format_workflow..r*cs|] }d|dVqdSrNrrrrrrz"format_workflow..z - `z `: requires z*`: default (no additional inputs required)r)r"r4r2) workflow_mapr workflow_nametrigger_inputsrrrrrformat_workflows  r9rBc Csd}|r |d|d7}|r%|d}ddd|D}||d7}|r:t|dkr:t|dd d } || d7}|rOt|dkrOt|dd d } || d7}|t|dd 7}|d7}|t|dd 7}|S) a  Generates a formatted documentation string describing the pipeline block's parameters and structure. Args: inputs: list of input parameters intermediate_inputs: list of intermediate input parameters outputs: list of output parameters description (str, *optional*): Description of the block class_name (str, *optional*): Name of the class to include in the documentation expected_components (list[ComponentSpec], *optional*): list of expected components expected_configs (list[ConfigSpec], *optional*): list of expected configurations Returns: str: A formatted string containing information about components, configs, call parameters, intermediate inputs/outputs, and final outputs. rBzclass  rcss|] }d|VqdS)z N)rstriprlinerrrrr5z"make_doc_string..rrFr r$)r )stripror4rpr,r1rr) routputsr< class_nameexpected_componentsexpected_configsoutput desc_lines aligned_desccomponents_str configs_strrrrmake_doc_strings"   rIc Cs|duri}n t|tstdt|}|siSi}|D]_\}}t|\}}|s2t|d|ryzt|}Wnt yR}z td|d|d|d}~ww|dkrdtd|d|d n|j |d d syt|d |d |d|||<q|S)NzZRequirements must be provided as a dictionary mapping package names to version specifiers.zO was specified in the requirements but wasn't found in the current environment.zRequirement specifier 'z' for z is invalid.zN/Az Version of z2 could not be determined to validate requirement 'z '. Things might work unexpected.T) prereleasesz requirement 'z,' is not satisfied by the installed version z. Things might work unexpected.) r-rmr]_normalize_requirementsr"rr^r_r r contains) reqsnormalized_reqsfinalreq specified_ver req_availablereq_actual_ver specifiererrrrr_validate_requirementss>    rVcs6|siStdtttfffdd |S)Nmappingcs|D]\}}t|tr|qt|}|std|dur$dnt|}|r6|ds6d|}|}|dur|sH|rH||<n8|r|r||krztd t d||g}Wnt yyt d|d|d|d |d Ynwt||<q||<qdS) Nz)Requirement package name cannot be empty.rB)rrr!~z==,zConflicting requirements for 'z ' detected: 'z' vs 'z '. Keeping 'z'.)r"r-rmrr?r] startswithrr r4filterr r^r_)rWpkgspecpkg_namespec_str existing_spec combined_spec _accumulate normalizedrrrd.s4        z,_normalize_requirements.._accumulate)rrmrr)rMrrcrrK(s !rKnamed_input_listsrUcGsi}i}|D]l\}}|D]e}|jdur|jdurd|j}n|j}||vri||}|jdurV|jdurV|j|jkrVtd|d|jd||d|jd|d|jd |jdurh|jdurh|||<|||<q |||<|||<q qt|S) a Combines multiple lists of InputParam objects from different blocks. For duplicate inputs, updates only if current default value is None and new default value is not None. Warns if multiple non-None default values exist for the same input. Args: named_input_lists: List of tuples containing (block_name, input_param_list) pairs Returns: List[InputParam]: Combined list of unique InputParam objects Nrz3Multiple different default values found for input 'z': z (from block 'z') and z '). Using r))r:rr?warningswarnr!values)rf combined_dict value_sources block_namer input_param input_name current_paramrrrcombine_inputsTsB        rpnamed_output_listscGsVi}|D] \}}|D]}|j|vs||jjdur#|jdur#|||j<q qt|S)aV Combines multiple lists of OutputParam objects from different blocks. For duplicate outputs, keeps the first occurrence of each output name. Args: named_output_lists: List of tuples containing (block_name, output_param_list) pairs Returns: List[OutputParam]: Combined list of unique OutputParam objects N)r:rr!ri)rqrjrlr@ output_paramrrrcombine_outputss    rsc0 s2|jj}|dd}t|dd}g}t|ddpi}|rWt|D]4\}\}}|jj} t|ddr:|jdd nd} ||d d |d | d | rV|d| q"|r^d |nd} t|dg} | rt | d dd} | dd }|rdd|dD}ddt|D}d |}nd}nd}t|dg}d}|rt |d dd}|dd }|rd|}t|dddu}|r||j }g}|j}t|dd}|dur||kr|nd}|d|D]^\}}t|z||}Wnty|d|d |d!|d"Yqw|j}fd#d|D}|d|d t|dd$} || r6| nd%|d|d"q|j}!|durN|n|j}"t|!d&} t|"d'}#| r`| nd(}$|#rg|#nd)}%|d*|$d+|%d |}&d}'nO|j}(|j})t|(d&} t|)d'}#| r| nd(}$|#r|#nd)}%d,|$d+|%}&d}'t|d-r|jrtd.d|jD}*|*rd/ d0d1|*D}+d2|+d}'d3d4g},t|d5r|jr|,|j|r&t|j }-td6d1|-Dr|,d7td8d1|-Dr|,d9td:d1|-Dr|,d;tdd1d?DrD|,d7tfd@d1dADrU|,d9tfdBd1dCDrf|,d;tfdDd1dEDsw|,d=n|,d=t|j}.dF|dG|dH|.dI}/||/| |||&|'|,dJS)Ka Generate model card content for a modular pipeline. This function creates a comprehensive model card with descriptions of the pipeline's architecture, components, configurations, inputs, and outputs. Args: blocks: The pipeline's blocks object containing all pipeline specifications Returns: Dict[str, Any]: A dictionary containing formatted content sections: - pipeline_name: Name of the pipeline - model_description: Overall description with pipeline type - blocks_description: Detailed architecture of blocks - components_description: List of required components - configs_section: Configuration parameters section - io_specification_section: Input/Output specification (per-workflow or unified) - trigger_inputs_section: Conditional execution information - tags: List of relevant tags for the model card Blocksz Pipeliner<zA modular diffusion pipeline. sub_blocksNrBrrrz. **z** (`r!z - zNo blocks defined.rBFr>z Components: cSsg|] }|r|qSr)r?r<rrrrrjz7generate_modular_model_card_content..cSs"g|] \}}|dd|qS)rz. r)rr5r=rrrrs"z No specific components required.zFNo specific components required. Components can be loaded dynamically.rCz Configs: z ## Configuration Parameters _workflow_maprz ## Workflow Input Specification z
z z%*Could not resolve workflow blocks.* z
csg|] }|jvr|qSrrrh)trigger_input_namesrrrs)r zNo additional inputs required.rrzNo specific inputs defined.zStandard pipeline outputs.z ## Input/Output Specification r:z## Input/Output Specification r8cSsg|]}|dur|qSrrrrrrrr2r*csr3r4rrrrrrr5z6generate_modular_model_card_content..z~ ### Conditional Execution This pipeline contains blocks that are selected at runtime based on inputs: - **Trigger Inputs**: zmodular-diffusersr model_namecs|]}d|vVqdS) inpaintingNrrwfrrrr(rzcsry) image2imageNrr{rrrr*r}zimage-to-imagecsry) controlnetNrr{rrrr,r}rcsry) text2imageNrr{rrrr.r}z text-to-imagec3|]}|vVqdSrrrtriggersrrr2r})maskrc3rrrrrrrr4r})rrc3rrrrrrrr6r})rcontrolnet_condc3rrrrrrrr8r})rrrrupThis is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework. **Pipeline Type**: z **Description**: z This pipeline uses a z8-block architecture that can be customized and extended.) pipeline_namemodel_descriptionblocks_descriptioncomponents_descriptionconfigs_sectionio_specification_sectiontrigger_inputs_sectiontags)r1r0replacerfr,r"r<ror2r4r,r?r1rvr@setkeys get_workflowrrr rYr8sortedrxanyrpru)0blocksblocks_class_namerr<blocks_desc_partsrur5r:block block_class block_descrr#rGrrenumerated_linesr-rrHconfigs_description has_workflowsr6rlblocks_outputsblocks_intermediateshared_outputswf_namer8workflow_blocks wf_inputstrigger_paramsr all_inputs all_outputs outputs_strinputs_descriptionoutputs_descriptionrrrr@trigger_inputs_listtrigger_inputs_strrworkflow_names block_countrr)rwrr#generate_modular_model_card_contents                                 r)rrr)rr)r)rrT)rBNNN)Grtrrg collectionsr dataclassesrrtypesrtypingrrrr r r PIL.ImagePILrZpackaging.specifiersr r configuration_utilsrrloaders.single_file_utilsrutilsrrrutils.import_utilsr get_loggerr0r^MODULAR_MODEL_CARD_TEMPLATErr9rrint Generatorr!floatImageTensorrmrfloat32rrrrrrrrrr r,r1r9rIrVrKtuplerprsrrrrrs      %  !%).37;@ DJP  )## 2 P   ( 7& 8'&,&+