o ۷iFm@sddlZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZddlmZddlmZdd lmZmZmZdd lmZmZdd lmZdd lmZmZm Z dd l!m"Z"ee#Z$GdddZ%dS)N) OrderedDict) init_logger)BaseLayerWithLoRA) LoRAModel)LoRALayerWeightsPackedLoRALayerWeights) PEFTHelper) LoRARequest)get_adapter_absolute_pathget_supported_lora_modulesreplace_submodule)MergedColumnParallelLinearQKVParallelLinear) LoRAConfig)*_expand_expected_modules_for_packed_layers_match_target_modulesfrom_layer_diffusion)stable_lora_int_idc @seZdZdZ   d@dejdejdejde d e dBd e f d d Z d e e fddZd ee ee ffddZde de d ee dBfddZdAdedBd e d dfddZddZde d e fddZed e fddZded eeeffd d!Zd"ejd ee fd#d$Zd%ed dfd&d'Zd(e d dfd)d*Zd+ed,e d e e!BdBfd-d.Z"de d/e d e#fd0d1Z$de d/e d dfd2d3Z%dBd4d5Z&dBd6d7Z'ded e#fd8d9Z(de d e#fd:d;Z)d ee fdd?Z+dS)CDiffusionLoRAManagerzManager for LoRA adapters in diffusion models. Reuses vLLM's LoRA infrastructure, adapted for diffusion pipelines. Uses LRU cache management similar to LRUCacheLoRAModelManager. N?pipelinedevicedtypemax_cached_adapters lora_path lora_scalecCs||_||_||_||_||_t|j|j|_||_ i|_ d|_ i|_ t |_t|_i|_d|_td|||||durZtd||tdt||d}|||dSdS)a0 Initialize the DiffusionLoRAManager. Args: max_cached_adapters: Maximum number of LoRA adapters to keep in the CPU-side cache (LRU). This mirrors vLLM's `max_cpu_loras` and is exposed to users via `OmniDiffusionConfig.max_cpu_loras`. NrzcInitializing DiffusionLoRAManager: device=%s, dtype=%s, max_cached_adapters=%d, static_lora_path=%sz:Loading LoRA during initialization from %s with scale %.2fstatic) lora_name lora_int_idr)rrr_compute_supported_lora_modules_supported_lora_modules_compute_packed_modules_mapping_packed_modules_mappingr_expected_lora_modulesr_registered_adapters_active_adapter_id_adapter_scalesr_adapter_access_orderset_pinned_adapters _lora_modules_max_lora_rankloggerinfor rset_active_adapter)selfrrrrrr init_requestr2V/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm_omni/diffusion/lora/manager.py__init__)sB  zDiffusionLoRAManager.__init__returncCsZtt|j}d}|jD]\}}t|tr#d}||ddq|r+|d|S)a1Compute supported LoRA module suffixes for this pipeline. vLLM's get_supported_lora_modules() returns suffixes for LinearBase modules. After this manager replaces layers with BaseLayerWithLoRA wrappers, those LinearBase modules become nested under ".base_layer", which would cause get_supported_lora_modules() to return "base_layer". To make adapter loading stable across multiple adapters, we also accept suffixes from existing BaseLayerWithLoRA wrappers and drop "base_layer" when appropriate. FT. base_layer) r)r r named_modules isinstanceraddsplitdiscard)r0 supportedhas_lora_wrappersnamemoduler2r2r3r ls   z4DiffusionLoRAManager._compute_supported_lora_modulesc Csi}|jD]R}t|dd}t|tsq|D]?\}}t|tr$|s%qt|ttfr5t dd|Ds6qt|}|s=q| |}|durK|||<q||krXt d||||qq|S)avCollect packed->sublayer mappings from the diffusion model. vLLM models declare `packed_modules_mapping` on the model class. For diffusion pipelines, we attach the same mapping on the transformer module(s) that implement packed (fused) projections, so LoRA loading can accept checkpoints trained against the logical sub-projections. packed_modules_mappingNcss|]}t|tVqdSN)r:str).0sr2r2r3 szGDiffusionLoRAManager._compute_packed_modules_mapping..z=Conflicting packed_modules_mapping for %s: %s vs %s; using %s) rmodulesgetattrr:dictitemsrDlisttupleallgetr-warning)r0mappingrApacked packed_name sub_namessub_names_listexistingr2r2r3r"s6     z4DiffusionLoRAManager._compute_packed_modules_mappingpacked_module_suffixn_slicescCs<|j|}|s dSt||krtd|t||dS|S)NzWpacked_modules_mapping[%s] has %d slices but layer expects %d; skipping sublayer lookup)r#rOlenr-rP)r0rWrX sub_suffixesr2r2r3_get_packed_sublayer_suffixess  z2DiffusionLoRAManager._get_packed_sublayer_suffixes lora_requestc Cs|durtd|dStd|r td|dS|j}td||j|j|t |j |j ||j vrGt d||j| |n|||||dS)zSet the active LoRA adapter for the pipeline. Args: lora_request: The LoRA request, or None to deactivate all adapters. lora_scale: The external scale for the LoRA adapter. Nz8No lora_request provided, deactivating all LoRA adaptersgzDReceived a request with LoRA scale 0; deactivating all LoRA adapterszMSetting active adapter: id=%d, name=%s, path=%s, scale=%.2f, cache_size=%d/%dz#Loading new adapter: id=%d, name=%s)r-debug_deactivate_all_adaptersmathiscloserPrrrrYr%rr. add_adapter_touch_adapter_info_activate_adapter)r0r\r adapter_idr2r2r3r/s.      z'DiffusionLoRAManager.set_active_adaptercCst|j|<|j|dS)z)Update the current caching ordering info.N)timer( move_to_endr0rdr2r2r3rbsz(DiffusionLoRAManager._touch_adapter_infordcCst|}||j|<dS)zUpdate the adapter scale for a given adapter ID. To avoid potential issues with using Floats as keys, for now, we round float values to 3 decimal points. N)r_get_rounded_scaler')r0rdrscaler2r2r3_update_adapter_scales z*DiffusionLoRAManager._update_adapter_scalecCs t|dS)zNormalizes a lora scale for use as a key in the _adapter_scales dict; for now we just round scales to 3 decimal places. )round)rr2r2r3rhs z'DiffusionLoRAManager._get_rounded_scalec Cs|jstdtd|jt|j}td|tj|d|jd}t d|j |j |j t j||j||jd|jd|jdd }t d|jt|jt|j|jD]}|qU||fS) Nz:No supported LoRA modules found in the diffusion pipeline.zSupported LoRA modules: %szResolved LoRA path: %s)max_position_embeddingstensorizer_config_dictz:Loaded PEFT config: r=%d, lora_alpha=%d, target_modules=%scpu)expected_lora_modules peft_helper lora_model_idrrmodel_vocab_sizernweights_mapperz4Loaded LoRA model: id=%d, num_modules=%d, modules=%s)r$ ValueErrorr-r]r rrfrom_local_dirrnr.r lora_alphatarget_modulesrfrom_local_checkpointrridrYlorasrLkeysvaluesoptimize)r0r\rrq lora_modellorar2r2r3 _load_adaptersH     z"DiffusionLoRAManager._load_adapterrAcCs(t|tr gdSt|trddgSgS)a_Return a packed_modules_list suitable for vLLM LoRA can_replace_layer(). Diffusion transformers frequently use packed projection layers like QKVParallelLinear (fused QKV). vLLM's LoRA replacement logic relies on `packed_modules_list` length to decide between single-slice vs packed LoRA layer implementations. )qkv01)r:rr )r0rAr2r2r3_get_packed_modules_lists  z-DiffusionLoRAManager._get_packed_modules_listrqc s||jt|dd}ddt|tr|r|n t|tr#|r#|dtdtffdd }t|jd|j |j dd}d D]}t |j |sFq=t|j |}t|t jsSq=|jdd D]\}}t|tsid |d vrjqY|d |} | |jvr}td | qY||} dusdur|| } | st| dkr| d \} } }||t| }|dur|D]}| r| d |n|}||rd} nq| sqYt|d|| dd}||urt|trt|||||j| <td| t|jqYq=dS)Nry module_namer5cs6durddl}||duSdurdSt|S)NrT)regexsearchr)rretarget_modules_listtarget_modules_patternr2r3_matches_target:s  zGDiffusionLoRAManager._replace_layers_with_lora.._matches_targetrF max_lora_rank max_loras max_cpu_loras lora_dtypefully_sharded_loras) transformer transformer_2dit)remove_duplicater8r6z#Layer %s already replaced, skippingT)layerr lora_configpacked_modules_list model_configzReplaced layer: %s -> %s)_ensure_max_lora_rankrwrIr:rDrLboolrr,rrhasattrrnnModuler9rr<r+r-r]rrY rpartitionr[rr type__name__)r0rqryrrcomponent_name componentrrAfull_module_namershould_replaceprefix_ packed_suffixrZ sub_suffix sub_full_name lora_layerr2rr3_replace_layers_with_lora/st           z.DiffusionLoRAManager._replace_layers_with_loramin_rankcCs||jkrdS|dkrtd|td|j|||_|js"dSt|jd|j|jdd}|jD] }|j d|ddq3|j durV|j }|j |}d|_ | ||dSdS) aEnsure LoRA buffers can accommodate adapters up to `min_rank`. We allocate per-layer LoRA buffers once when we first replace layers. If a later adapter has a larger rank, we need to reinitialize those buffers and re-apply the currently active adapter. NrzInvalid LoRA rank: z"Increasing max LoRA rank: %d -> %drFr)rrr) r,rur-r.r+rrrr~create_lora_weightsr&r'rc)r0rrr active_id active_scaler2r2r3r|s.    z*DiffusionLoRAManager._ensure_max_lora_rankrrcCs`||}|dur |Sd|vr|dddn|}||}|dur$|S|dd}||S)aBest-effort lookup for LoRA weights by name. Tries: - Full module name (e.g. transformer.blocks.0.attn.to_qkv) - Relative name without the top-level component (e.g. blocks.0.attn.to_qkv) - Suffix-only name (e.g. to_qkv) Nr6rr7)get_lorar<)r0rr lora_weightscomponent_relative_name module_suffixr2r2r3_get_lora_weightss   z&DiffusionLoRAManager._get_lora_weightsricCs,t|}|j|k}|j||k}|o|S)z?True if the adapter_id is active and the current scale matches.)rrhr&r'rO)r0rdri rounded_scale is_active matches_scaler2r2r3_is_active_at_scales  z(DiffusionLoRAManager._is_active_at_scalecs||rtd|dStd||j|}|jD]L\}}|||}|durt|dd}|dkr| d\}} } | | |} | durP| dqg} d} | D],}|ra|d|n|}|||}|durvd} t |t rvd}| t |tr|ndqV| s| dqg}g}| D]}|dur|d|dq||j||jq|jd||d td || n| dqt |t r|j}fd d |jD}|jd||d td |qt|dd}|dkrQt|dd}|dur| dqt|}|jjd|kr%td||jjd|| dqttj|jt|dd}|jg|}fdd |D}|jd||d td|q|j}|jd|j|d td||jj|jjq||_||dS)Nz0Adapter %d already active at scale %.3f skippingzActivating adapter: id=%drXrr6rFT)indexlora_alora_bz;Activated packed LoRA for %s via submodules=%s (scale=%.2f)cs g|] }|dur dn|qSrCr2rEbrir2r3 sz:DiffusionLoRAManager._activate_adapter..z)Activated packed LoRA for %s (scale=%.2f) output_sliceszQSkipping LoRA for %s due to shape mismatch: lora_b[0]=%d != sum(output_slices)=%d)dimcsg|]}|qSr2r2rrr2r3rsz5Activated fused LoRA for packed layer %s (scale=%.2f)zCActivated LoRA for %s: lora_a shape=%s, lora_b shape=%s, scale=%.2f)rr-r]r.r%r+rKrrIrr[ reset_lorar:rappendrrrset_lorasumshaperPrLtorchr<r&rj)r0rdrirrrrrXrrrrZ sub_loras any_foundrrsub_lora lora_a_list lora_b_listrtotalb_splits scaled_lora_br2rr3rcs                         z&DiffusionLoRAManager._activate_adaptercCs@tdt|j|jD]}|dqd|_tddS)Nz$Deactivating all adapters: %d layersrzAll adapters deactivated)r-r.rYr+r~rr&r])r0rr2r2r3r^5s  z-DiffusionLoRAManager._deactivate_all_adapterscstjjdkrBfddjD}|s!tdjdS|d}td|tjj|tjjdks dSdS)z]Evict unpinned registered adapters until we have room for a new adapter to be loaded.rcsg|] }|jvr|qSr2)r*)rEaidr0r2r3rAsz?DiffusionLoRAManager._evict_for_new_adapter..zjCache full (%d) but all adapters are pinned; cannot evict. Increase max_cached_adapters or unpin adapters.rz*Evicting LRU adapter: id=%d (cache: %d/%d)N) rYr%rr(r}r-rPr.remove_adapter)r0evict_candidateslru_adapter_idr2rr3_evict_for_new_adapter<s" z+DiffusionLoRAManager._evict_for_new_adaptercCs|j}||jvrtd|dStd||j|||\}}||||j|<| |td|t |j|j dS)zG Add a new adapter to the cache without activating it. z'Adapter %d already registered, skippingFz"Adding new adapter: id=%d, name=%sz#Adapter %d added, cache size: %d/%dT) rr%r-r]r.rrrrbrrYr)r0r\rdrrqr2r2r3raSs     z DiffusionLoRAManager.add_adaptercCs||jvr td|dStd||j|kr||j|=|j|d|j|d|j |td|t |j|j dS)z3 Remove an adapter from the cache. z#Adapter %d not found, cannot removeFzRemoving adapter: id=%dNz%Adapter %d removed, cache size: %d/%dT) r%r-r]r.r&r^r'popr(r*r=rYrrgr2r2r3ros"     z#DiffusionLoRAManager.remove_adaptercCst|jS)z&Return list of registered adapter ids.)rLr%r}rr2r2r3 list_adapterssz"DiffusionLoRAManager.list_adapterscCsP||jvr td|dS|j|t|j|<|j|td|dS)z4Mark an adapter as pinned so it will not be evicted.z Adapter %d not found, cannot pinFz'Pinned adapter id=%d (won't be evicted)T) r%r-r]r*r;rer(rfr.rgr2r2r3 pin_adapters     z DiffusionLoRAManager.pin_adapter)rNr)r)r5N),r __module__ __qualname____doc__rrrrrintrDfloatr4r)r rJrLr"r[r r/rbrj staticmethodrhrMrrrrrrrrrrrrcr^rrarrrr2r2r2r3r"sb  C$$  1M&   t r)&r_re collectionsrrtorch.nnr vllm.loggerrvllm.lora.layersrvllm.lora.lora_modelrvllm.lora.lora_weightsrrvllm.lora.peft_helperrvllm.lora.requestr vllm.lora.utilsr r r !vllm.model_executor.layers.linearr rvllm_omni.config.lorarvllm_omni.diffusion.lora.utilsrrrvllm_omni.lora.utilsrrr-rr2r2r2r3s$