o پij5 @s6dZddlZddlZddlZddlZddlmZmZddlm Z ddl Z ddl Z ddl Z ddlmZddlmZddlmZz ddlmZd ZWn eySd ZYnwdd lmZdd lmZeeZeZd6ddZ e GdddeZ!d7de"e Bde"dBfddZ#de$e"de"de"d e$e"fddZ%de$e"d e$e"fddZ&dZ'de"d e(fdd Z)d efde$e"d!e(d"e(d ee*e"e j+fddffd#d$Z,d%e"d&e"d e-fd'd(Z. d8de$e"d!e(d ee*e"e j+fddffd)d*Z/d+e j+d,e j+d dfd-d.Z0d/e"d0e-d e"dBfd1d2Z1d3ee*e"e j+fd e"fd4d5Z2dS)9zMUtilities for downloading, loading, initializing and verifying model weights.N) GeneratorIterable)Path) safe_open)DTensor)tqdm)SafetensorsStreamerTF)get_local_torch_device) init_loggerreturncCs:dtjvrz ddl}dtj_WdStyYdSwdS)z#automatically activates hf_transferHF_HUB_ENABLE_HF_TRANSFERrNT)osenviron hf_transferhuggingface_hub constantsr ImportError)rre/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/runtime/loader/weight_utils.pyenable_hf_transfer(s  rcseZdZfddZZS) DisabledTqdmcsd|d<tj|i|dS)NTdisable)super__init__)selfargskwargs __class__rrr9szDisabledTqdm.__init__)__name__ __module__ __qualname__r __classcell__rrrrr7srmodel_name_or_path cache_dircCsl|pt}t|}tjtj|dd|dd}t|  }||d}t j tj ||dd}|S)NT)exist_ok/-z.locki)mode)temp_dirstrr makedirspathdirnamereplacehashlibsha256encode hexdigestfilelockFileLockjoin)r#r$lock_dir model_name hash_namelock_file_namelockrrrget_lock>s  r;hf_weights_files hf_folder index_filecstj||}tj|s|St|}t|d}Wdn1s%wYt|D]}tj|||q/fdd|D}|S)N weight_mapcsg|]}|vr|qSrr.0fweight_files_in_indexrr az6filter_duplicate_safetensors_files..) r r,r5isfileopenjsonloadsetadd)r<r=r>index_file_namerBr? weight_namerrCr"filter_duplicate_safetensors_filesPs  rOcsgdfdd|D}|S)z Exclude files that are not needed for inference. See https://github.com/huggingface/transformers/blob/v4.34.0/src/transformers/trainer.py#L227-L233 )ztraining_args.binz optimizer.binz optimizer.ptz scheduler.ptz scaler.ptcs&g|]tfddDsqS)c3s|]}|VqdSN)endswith)rAxrBrr sszCfilter_files_not_needed_for_inference...)any)rA blacklistrSrrErs z9filter_files_not_needed_for_inference..r)r<rrVr%filter_files_not_needed_for_inferencees  rXz_{desc}: {percentage:3.0f}% Completed | {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}] file_pathc Csz"t|ddd}t|}WdWdS1swYWdStyB}ztd|t|jt|WYd}~dSd}~ww)z Validate that a safetensors file is readable and not corrupted. Args: file_path: Path to the safetensors file Returns: True if file is valid, False if corrupted ptcpu frameworkdeviceNTz0Corrupted safetensors file detected: %s - %s: %sF) rlistkeys Exceptionloggererrortyperr*)rYrB_errr_validate_safetensors_files"  rgto_cpuuse_runai_model_streamerc cstj p tjdk}|rdntt}dd|D}|r|D]c}zHtj|rVtj |}t |t dtj |tj|rUt |t dtj |ntj|rkt |t dtj |Wq"ty}z t d||WYd }~q"d }~wwtd t|d d d|Dd |rt+} | || D]\} } |r| | fVq| | |fVqWd d S1swYd St|d| tdD]+} t| d|d} | D] } | | }| |fVqWd n 1swYqd S)z7Iterate over the weights in the model safetensor files.rr[cSsg|]}t|s|qSr)rg)rAst_filerrrrEs z0safetensors_weights_iterator..zRemoved corrupted symlink: %szRemoved corrupted blob: %szRemoved corrupted file: %sz&Failed to remove corrupted file %s: %sNFound z9 corrupted safetensors file(s). Files have been removed: cSsg|]}tj|qSr)r r,basenamer@rrrrErFz?. Please retry - the files will be re-downloaded automatically.z%Loading safetensors checkpoint shardsdescr bar_formatrZr\)torch distributedis_initializedget_rankr*r r r,islinkrealpathremoverbinforlexistsrGrawarning RuntimeErrorlenr stream_files get_tensorsclonedetachtor _BAR_FORMATrr` get_tensor)r<rhri enable_tqdmr^corrupted_filesrY blob_pathrfstreamernametensorrjrBparamrrrsafetensors_weights_iteratorsx             "    rbin_filer^c Csjz tj||ddWSty4}zdt|vr/tdtj|tj||ddWYd}~Sd}~ww)a`Load a PyTorch checkpoint file, handling legacy tar format. PyTorch 2.6 changed the default of weights_only from False to True. Legacy tar format files cannot be loaded with weights_only=True. This function tries weights_only=True first, then falls back to False for legacy tar format files from trusted sources (HuggingFace Hub). T) map_location weights_onlyzlegacy .tar formatz6Loading %s with weights_only=False (legacy tar format)FN) rprJrzr*rbryr r,rl)rr^rfrrr _load_pt_files  rccsd|rdntt}tj ptjdk}t|d| tdD]}t||}| EdH~q dS)z3Iterate over the weights in the model bin/pt files.r[rzLoading pt checkpoint shardsrmN) r*r rprqrrrsrrritems)r<rhr^rrstaterrrpt_weights_iterators rr loaded_weightcCsz7|dkr|dkr|j|WdS||ks/Jd|d|d|j|WdSty?w)zDefault weight loader.zAttempted to load weight (z) into parameter ()N)numeldatafill_itemsizecopy_ra)rrrrrdefault_weight_loaders  rr params_dictc sdr$tddd}||vr"tdd|ddS|Sdd g}d d g}|D]J}|rxtfd d |DrRd|dd|d|}n |d|}||vrttd|dd|d|d dS|Sq.S)aRemap the name of FP8 k/v_scale parameters. This function handles the remapping of FP8 k/v_scale parameter names. It detects if the given name ends with a suffix and attempts to remap it to the expected name format in the model. If the remapped name is not found in the params_dict, a warning is printed and None is returned. Args: name (str): The original loaded checkpoint parameter name. params_dict (dict): Dictionary containing the model's named parameters. Returns: str: The remapped parameter name if successful, or the original name if no remapping is needed. None: If the remapped name is not found in params_dict. z .kv_scalezDEPRECATED. Found kv_scale in the checkpoint. This format is deprecated in favor of separate k_scale and v_scale tensors and will be removed in a future release. Functionally, we will remap kv_scale to k_scale and duplicate k_scale to v_scalez .attn.k_scalez'Found kv_scale in the checkpoint (e.g. z6), but not found the expected name in the model (e.g. z). kv_scale is not loaded.Nz.k_scalez.v_scalez.self_attn.k_proj.k_scalez.self_attn.v_proj.v_scalec3s|]}|vVqdSrPr)rA mo_scale_namerrrrTAsz,maybe_remap_kv_scale_name..z .self_attn.r_projz.self_attn.attnz.attnrkz in the checkpoint (e.g. z). z is not loaded.)rQrb warning_oncer.rU)rr remapped_namepossible_scale_namesmodelopt_scale_names scale_namerrrmaybe_remap_kv_scale_namesF   r named_paramscCsvt}t|dddD]*\}}|||}t|tr#|j}|| d t jjq |S)aCompute a SHA-256 checksum for a set of (name, tensor) pairs. Used to verify the correctness of weight refitting. After a refit, compare the checksum of the in-GPU model weights against the checksum of the on-disk tensors or the tensors in the training engine. cSs|dS)Nrr)rRrrr`sz*compute_weights_checksum..)key)r/r0sortedupdater1r isinstancer _local_tensorr[ contiguousreshapeviewrpuint8numpyrr2)rhasherrrtrrrcompute_weights_checksumVs  (r)r NrP)T)3__doc__r/rIr tempfilecollections.abcrrpathlibrr3huggingface_hub.constantsrrpsafetensors.torchrtorch.distributed.tensorr tqdm.autorrunai_model_streamerrHAS_RUNAI_MODEL_STREAMERr)sglang.multimodal_gen.runtime.distributedr 1sglang.multimodal_gen.runtime.utils.logging_utilsr rrb gettempdirr)rrr*r;r_rOrXrboolrgtupleTensorrdictrrrrrrrrrs            D >