o پi><@sddlmZmZddlmZddlmZddlZddlmZddl m Z m Z ddl m Z ddlmZmZmZmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZddlm Z ee!Z"dejj#dej$dejj#fddZ%     d1de&ej'de(e)efde*e)dej+de,de,dej-dej-de.de.d ej-dBd!e.d"e.dejj'fd#d$Z/dedgdd%de.d&e.d'edBd(e dBd)e*ee)ej'ge.fd!e.ddfd*d+Z0   d2d,eejj'Bd-ee1e)ej$fddfdej+dej-dBd"e.de.d.ee)ge1e)eeffdBdefd/d0Z2dS)3)Callable Generator)chain)AnyN)nn) DeviceMeshinit_device_mesh)distribute_tensor)CPUOffloadPolicy FSDPModuleMixedPrecisionPolicy fully_shard)_IncompatibleKeys)get_param_names_mappinghf_to_custom_state_dictset_default_torch_dtype)safetensors_weights_iterator)current_platform) init_logger)set_mixed_precision_policy actual_paramtensorreturncCsR|j}z |j||dd}Wnty|||}Ynw|j|jd|_|S)NF requires_grad) __class____new__ TypeError__dict__updater)rrcls new_paramr"b/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/runtime/loader/fsdp_load.py_make_param_like(s r$FT model_cls init_paramsweight_dir_listdevicehsdp_replicate_dimhsdp_shard_dim param_dtype reduce_dtype cpu_offloadfsdp_inference output_dtypepin_cpu_memorystrictc  Cs^|r|ntj} t| || dd}t| || |dt| $td|di|}Wdn1s3wYWdn1sBwY| }trTd}t d|rv||}| s`|}d}t tj ||fdd }t ||d |||j | d t|}t|j}t||||| ||d t||D]\}}|jrtd |dt|tjjrd|_q|S)aLoad a model with optional FSDP (Fully Sharded Data Parallel) support. Args: param_dtype: Data type for model parameters, also used for: - Model initialization context (set_default_torch_dtype) - FSDP mixed precision policy - Weight loading and casting reduce_dtype: Data type for gradient reduction in FSDP mixed precision. strict: If True, enforce strict state dict loading (all keys must match). F)cast_forward_inputs)r+r,r/ mp_policymetaNz6Disabling FSDP for MPS platform as it's not compatible) replicateshard) mesh_shapemesh_dim_namesT)r-reshard_after_forwardr3meshfsdp_shard_conditionsr0)r1r-param_names_mappingzUnexpected param or buffer z on meta device.r")torchbfloat16r rrr(ris_mpsloggerinfor device_type shard_model_fsdp_shard_conditionsrrr=%load_model_from_full_model_state_dictrnamed_parameters named_buffersis_meta RuntimeError isinstancer Parameterr)r%r&r'r(r)r*r+r,r-r.r/r0r1default_torch_dtyper3modeluse_fsdp world_size device_meshweight_iteratorparam_names_mapping_fnnpr"r"r#maybe_load_fsdp_model8sp   rV)r:r3r;r<r0r:r3r;r<c s|dus t|dkrtdt|jdS|||d}|r$t|d|d<d}tt|D]\t fdd|DrJt fi||d 7}q.|dkrSt d t |fi|dS) ao Utility to shard a model with FSDP using the PyTorch Distributed fully_shard API. This method will over the model's named modules from the bottom-up and apply shard modules based on whether they meet any of the criteria from shard_conditions. Args: model (TransformerDecoder): Model to shard with FSDP. cpu_offload (bool): If set to True, FSDP will offload parameters, gradients, and optimizer states to CPU. reshard_after_forward (bool): Whether to reshard parameters and buffers after the forward pass. Setting this to True corresponds to the FULL_SHARD sharding strategy from FSDP1, while setting it to False corresponds to the SHARD_GRAD_OP sharding strategy. mesh (Optional[DeviceMesh]): Device mesh to use for FSDP sharding under multiple parallelism. Default to None. fsdp_shard_conditions (List[Callable[[str, nn.Module], bool]]): A list of functions to determine which modules to shard with FSDP. pin_cpu_memory (bool): If set to True, FSDP will pin the CPU memory of the offloaded parameters. NrzPThe FSDP shard condition list is empty or None. No modules will be sharded in %s)r:r;r3) pin_memoryoffload_policycsg|]}|qSr"r").0shard_conditionmrTr"r# szshard_model..r5zXNo layer modules were sharded. Please check if shard conditions are working as expected.) lenrAwarningtype__name__r reversedlist named_modulesanyr ValueError) rNr-r:r3r;r<r0 fsdp_kwargsnum_layers_shardedr"r[r#rDs.rDrNfull_sd_iteratorr=cs|}t|}t||\} } t|tp tdd|D} t| } i} | D]}| |}| |}|durP|s>| rFt d|dt d|dq+|rT|n|j}t|ds|j||d }| |}|durrt|d dnd}|dur|dus~Jtj|||d }t|d d }t||}|s|sd }||_||||j}n|}|r| s|}n|j||d }t||j|j}|r|d }d }tj||d| |<q+| |_t | t | }|rt d|gd}|D]ktfdd|Dst !d|t dd|d| }dvs dvr$tj"}ntj#}t|ds?||||d }|r>| s>|}n||||d }t||j|j}|rU|}t|| <q|j$| |ddS)a$ Converting full state dict into a sharded state dict and loading it into FSDP model (if training) or normal huggingface model Args: model (Union[FSDPModule, torch.nn.Module]): Model to generate fully qualified names for cpu_state_dict full_sd_iterator (Generator): an iterator yielding (param_name, tensor) pairs device (torch.device): device used to move full state dict tensors param_dtype (torch.dtype): dtype used to move full state dict tensors. If none, respect original dtype from checkpoint strict (bool): flag to check if to load the model in strict mode cpu_offload (bool): flag to check if FSDP offload is enabled param_names_mapping (Optional[Callable[[str], str]]): a function that maps full param name to sharded param name Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys css|]}t|dVqdS)rQN)hasattr)rYrUr"r"r# s  z8load_model_from_full_model_state_dict..Nz Parameter zQ not found in custom model state dict. The hf to custom mapping may be incorrect.z Parameter 'zY' from checkpoint not found in model; skipping. This is expected for optional parameters.rQ)r(dtype weight_loaderrFcpurz0Found unloaded parameters in meta state dict: %s) gate_compresswcscaleswtscalebiasc3s|]}|vVqdS)Nr")rYpatternnew_param_namer"r#rkUsz3Unsupported new parameter: %s. Allowed patterns: %szNew parameter 'z9' is not supported. Currently only parameters containing z are allowed.rprqT)r1assign)% state_dictdictrGrrKr revaluessortedkeysgetrfrAr_rlrjtogetattrr> empty_liker$is_floating_point is_complexrdatarnr rQ placementsrrLreverse_param_names_mappingseterror ones_like zeros_likeload_state_dict)rNrir(r+r1r-r=meta_sd param_dictcustom_param_sdr is_fsdp_modelsorted_param_names sharded_sdtarget_param_name full_tensormeta_sharded_param target_dtyperrmsharded_tensorr temp_param unused_keysALLOWED_NEW_PARAM_PATTERNS init_liker"rtr#rFs                   rF)FFNTT)FFN)3collections.abcrr itertoolsrtypingrr>rtorch.distributedrrtorch.distributed._tensorr torch.distributed.fsdpr r r r torch.nn.modules.moduler*sglang.multimodal_gen.runtime.loader.utilsrrr1sglang.multimodal_gen.runtime.loader.weight_utilsr'sglang.multimodal_gen.runtime.platformsr1sglang.multimodal_gen.runtime.utils.logging_utilsrsglang.multimodal_gen.utilsrrarArLTensorr$r`Modulerxstrrcr(intrlboolrVrDtuplerFr"r"r"r#s                 ` F