o پi^1@sdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z mZddlZddlmZeeZdZdedede ed ed e ef d d Zd ed e efddZd efddZd ed e efddZd ed e efddZ ddd edede ed dfddZd ed dfddZ d ed e efddZ!d eded dfdd Z"d!ed"ed efd#d$Z#   %ded ede ede ed ed eee eff d&d'Z$d ed(e ed)ed efd*d+Z%d,ed-ed efd.d/Z&d0ed1ed eee effd2d3Z'd0ed efd4d5Z(dfd7e ed8e)d efd9d:Z*d ed eee effd;d<Z+d ed(e ed)ed eee effd=d>Z, %dgd ed?ed efd@dAZ-d!ed efdBdCZ.d!ed efdDdEZ/d ed eee effdFdGZ0d ed(e ed eee ee effdHdIZ1d)edJe ed e)fdKdLZ2d)ed edMed dfdNdOZ3d)edPedQe ed efdRdSZ4dTedUe ed)ed efdVdWZ5 ddd)edXe ed efdYdZZ6d)edXe ed e)fd[d\Z7 ]dhd)edUe edXe ede ed^e)d ef d_d`Z8daed dfdbdcZ9dS)ia CI-specific weight validation and cache cleanup utilities. This module contains validation and cleanup logic that is ONLY used in CI environments. These functions handle: - Validating safetensors files for corruption - Checking for missing shards in sharded models - Cleaning up corrupted files (selective or full cache deletion) - Automatic retry logic for corrupted downloads - Validating config/tokenizer files completeness to enable offline mode For regular users, weight_utils.py provides simple download functionality without the overhead of validation and automatic cleanup. The CI-specific behavior is gated by is_in_ci() checks in weight_utils.py. N)ListOptionalTuple)log_info_on_rank05repo_idfilenamerevisionallow_remote_checkreturnc Cs|s td||dSzddlm}|}|j|||d}td|||p%d||WStyH}ztd|||p:d|WYd}~dSd}~ww) aV Check if a file exists on Hugging Face Hub for a specific revision. Args: repo_id: Repository ID (e.g., "meta-llama/Llama-2-7b-hf") filename: File name to check (e.g., "hf_quant_config.json") revision: Git revision (commit hash, branch, or tag). None means default branch. allow_remote_check: Whether remote checks are allowed (e.g., CI validation phase) Returns: True if file exists on hub, False if it doesn't exist, None if we cannot determine (network error or remote check not allowed - be conservative and assume incomplete) z9Remote check disabled for %s/%s, returning None (unknown)Nr)HfApi)rrr z0Remote file check: %s/%s (revision=%s) exists=%sdefaultzZFailed to check remote file existence for %s/%s (revision=%s): %s. Will treat as optional.)loggerdebughuggingface_hubr file_exists Exception)rrr r r apiexistser`/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/model_loader/ci_weight_validation.py_remote_file_exists)s< r snapshot_dircCsN|rtj|s dStj|d}t|ddd}d|dS)a Get the path to validation marker file for a snapshot. Marker is stored in /tmp to avoid permission issues with HF cache directory. Marker key is sha256(snapshot_dir) to avoid any collisions regardless of model_name_or_path format. Args: snapshot_dir: Path to snapshot directory Returns: Path to marker file or None if snapshot_dir is invalid N/utf-8 z/tmp/sglang_hf_validation_.json) ospathisdirrealpathrstriphashlibsha256encode hexdigest)rnormalized_dirdir_hashrrr_get_validation_marker_path\s  r)cCs8tjdtjdd}tj|d}tj|dd|S)a( Get the directory for per-run validation markers. These markers are specific to the current CI run and are not shared across runners. They are stored in a temporary directory that is cleaned up after the run completes. Returns: Path to per-run marker directory RUNNER_TEMPTMPDIRz/tmpsglang_ci_offline_markersTexist_ok)renvirongetrjoinmakedirs)base_dir marker_dirrrr_get_per_run_marker_dirys r5cCs\|rtj|s dStj|d}t|ddd}t }tj ||dS)aj Get the path to per-run validation marker file for a snapshot. Per-run markers are specific to the current CI run and are not shared across runners. This prevents cross-runner cache state pollution. Args: snapshot_dir: Path to snapshot directory Returns: Path to per-run marker file or None if snapshot_dir is invalid Nrrrr) rrr r!r"r#r$r%r&r5r1)rr'r(r4rrr_get_per_run_marker_paths  r6c st|}|r tj|sdSzCt|ddd }t|Wdn1s&wYtts3WdSgd}t fdd|DsEWdS dd urOWdSWSt yk}zt d ||WYd}~dSd}~ww) z Read per-run validation marker for a snapshot. Args: snapshot_dir: Path to snapshot directory Returns: Marker dict if exists and valid, None otherwise Nrrencoding) timestampmodel_id snapshot_hashvalidation_passedc3|]}|vVqdSNr).0kmarkerrr z'_read_per_run_marker..r=Tz)Failed to read per-run marker from %s: %s)r6rrropenjsonload isinstancedictallr0rrrr marker_pathf required_keysrrrBr_read_per_run_markers(   rPr;required_filesc CsPt|}|s tddSddlm}tj|}|d||d|p&gd}zAtj |}tj |ddt j d d |d d d }|j } tj||ddWdn1sWwYt| |td|WdSty} z1td|| zdtvrtj| rt| Wn tyYnwWYd} ~ dSWYd} ~ dSd} ~ ww)z Write per-run validation marker for a snapshot. Args: snapshot_dir: Path to snapshot directory model_id: Model identifier required_files: List of required files that were validated z1Cannot write per-run marker: invalid snapshot_dirNrdatetimeZT)r:r;r<r=rQr-wrF.tmpmoder9dirdeletesuffixindentzWrote per-run marker to %sz(Failed to write per-run marker to %s: %s temp_path)r6rrrSrrbasenameutcnow isoformatdirnamer2tempfileNamedTemporaryFilenamerGdumpreplacerwarninglocalsrremove) rr;rQrMrSr<rCr4rNr_rrrr_write_per_run_markersP        rlc Csrt|}|r5tj|r7zt|td|WdSty4}ztd||WYd}~dSd}~wwdSdS)zv Remove per-run validation marker for a snapshot. Args: snapshot_dir: Path to snapshot directory zRemoved per-run marker: %sz&Failed to remove per-run marker %s: %sN) r6rrrrkrrrri)rrMrrrr_remove_per_run_markers rmc st|}|sdStj|sdSz^t|ddd }t|Wdn1s(wYtts5WdSgd}t fdd|DsGWdSdt krYt d dt WdS d d urlt d d WdSWStjtfy}zt d ||WYd}~dSd}~ww)a Read validation marker for a snapshot. Args: snapshot_dir: Path to snapshot directory Returns: Marker dict with keys: version, validated_at, validation_passed None if marker doesn't exist or is invalid or validation_passed is not True Nr7rr8version validated_atr=c3r>r?r)r@keyrBrrrD&rEz*_read_validation_marker..roz>Validation marker version mismatch: %s != %s, will re-validater=Tz?Validation marker has validation_passed=%s, treating as invalidz*Failed to read validation marker at %s: %s)r)rrrrFrGrHrIrJrKVALIDATION_MARKER_VERSIONrrr0JSONDecodeErrorOSErrorrLrrBr_read_validation_marker sB     rupassedc CsF|sdSt|}|stddSddlm}t|d|d}zBtj |}tj |ddt j d d |d d d }|j }tj||ddWdn1sQwYt||td||WdSty}z1td||zdtvrtj|rt|Wn tyYnwWYd}~dSWYd}~dSd}~ww)a- Write validation marker for a snapshot (atomic write). IMPORTANT: We only cache successful validations. Failed validations are NOT cached to allow retry after files are downloaded. Args: snapshot_dir: Path to snapshot directory passed: Whether validation passed Nz)Cannot write marker: invalid snapshot_dirrrRrTrnTr-rUrFrVrWr\r]z)Wrote validation marker to %s (passed=%s)z+Failed to write validation marker to %s: %sr_)r)rrrSrrrarbrrrcr2rdrerfrGrgrhrrirjrrk) rrvrMrSrCr4rNr_rrrr_write_validation_markerBsN       rw file_path file_namec CsNtj|std||dStj|std||dSztj|}|dkr3td||WdSWntyN}ztd||WYd}~dSd}~wwz!t |dd d }t |WdWd S1siwYWd St j y}ztd |||WYd}~dSd}~wt y}ztd |||WYd}~dSd}~ww)z Validate that a JSON file exists, is non-empty, and can be parsed. Args: file_path: Path to the JSON file file_name: Name of the file (for logging) Returns: True if the file is valid, False otherwise z'CI cache validation: %s not found at %sFz)CI cache validation: %s is not a file: %srz$CI cache validation: %s is empty: %sz.CI cache validation: Cannot get size of %s: %sNr7rr8Tz2CI cache validation: %s is not valid JSON: %s - %sz/CI cache validation: Failed to read %s: %s - %s)rrrrrisfilerigetsizertrFrGrHrsr)rxry file_sizerrNrrr_validate_json_filezsZ     r}Fc! CsBg}ddg}|D]}tj||}t||s||qtj|d}tj|r3t|ds3|dtj|d} tj| } d} |duoVd|voVtj| oV|d } | rc|rct|d||d} | d ur| s|d |pod d t t d |d|p}d dn3t| ds|dn(| dur| rt| ds|dn| rt| ds|dt d|pd|pd dD]} tj|| }tj|rt|| s|| dnqtj|d}tj|rt|ds|dtj|d}tj|rt|ds|dtj|d}tj|rzt |ddd }t |}Wdn 1s.wY|di}|rt|trt}|D]\}}t|trgd|vrg|dd}||d qI|D]5}tj||}tj|s||d!t d"|d#|qktj|s||d$qkWnt jttfy}z t d%|WYd}~nd}~wwgd&}d}|D]F}tj||}tj|r tj|r |d'krt||rd }n!qztj|dkrd }WnWqty Yqwq|s|d(t|dk} | |fS))a Validate that critical config and tokenizer files exist and are valid. This checks for: - config.json (required) - tokenizer_config.json (required) - generation_config.json (optional but validated if present) - hf_quant_config.json (conditionally required based on Hub) - for FP4/FP8/ModelOpt - quantize_config.json / quant_config.json (optional but validated if present) - for AWQ/GPTQ - params.json (optional but validated if present) - for Mistral native format - preprocessor_config.json (optional but validated if present) - for vision models - trust_remote_code dynamic modules (required if auto_map present in config.json) - At least one tokenizer file: tokenizer.json, tokenizer.model, or tiktoken.model Args: snapshot_dir: Path to the model snapshot directory model_id: Model repository ID (e.g., "meta-llama/Llama-2-7b-hf"), used for remote checks revision: Git revision (commit hash), used for remote checks allow_remote_check: Whether to check Hub for file existence to determine requirements Returns: Tuple of (is_valid, missing_files) - is_valid: True if all required files are present and valid - missing_files: List of missing or invalid file names config.jsontokenizer_config.jsonzgeneration_config.jsonz+generation_config.json (exists but invalid)hf_quant_config.jsonNr)rrr r Tz;hf_quant_config.json (required: exists on Hub for revision r z but missing locally)z!Hub has hf_quant_config.json for z revision zH but local snapshot missing it. Cache incomplete, will not write marker.z)hf_quant_config.json (exists but invalid)FzpCannot verify hf_quant_config.json on Hub for %s (revision=%s), treating as optional since remote status unknownunknown)zquantize_config.jsonzquant_config.jsonz (exists but invalid)z params.jsonz params.json (exists but invalid)preprocessor_config.jsonz-preprocessor_config.json (exists but invalid)r7rr8auto_map.r.pyz! (required for trust_remote_code)z$Custom module file not in snapshot:  for z (exists but not a file)+Failed to check auto_map in config.json: %stokenizer.jsontokenizer.modelztiktoken.modelrztokenizer file)rrr1r}appendrisabs startswithrrrrrFrGrHr0rIrJsetitemsstrsplitaddrzrsrtKeyErrorr{len)!rr;r r missing_filesrQryrxgeneration_config_pathhf_quant_config_pathlocal_hf_quant_existsremote_hf_quant_exists is_hf_repoquant_config_namequant_config_pathparams_json_pathpreprocessor_config_path config_pathrNconfigr custom_filesrqvalue module_name custom_filecustom_file_pathrtokenizer_filestokenizer_foundtokenizer_filetokenizer_pathis_validrrr$_validate_config_and_tokenizer_filess                           r weight_filesmodel_name_or_pathcCs|s ttd|dSt|}|dur/t|}|r tj|nd}ttd|d|dStj|}ddl}|jj }ttd |d |d |t ||||d \} } | settd | d|dSt ||\} } } | s|ttd| d|dSttd|t |dddS)aq Validate local cache completeness (config/tokenizer/weights) and determine if offline mode can be safely enabled. This function uses a snapshot-level marker to cache validation results, so the heavy validation is done at most once per snapshot per runner. This function checks: 1. Validation marker (if exists and version matches, skip re-validation) 2. Config and tokenizer files (config.json, tokenizer_config.json, etc.) 3. Weight files (safetensors shards, index files, corruption check) If all are present and valid, it returns True to signal that offline mode can be safely enabled. IMPORTANT: This should be called BEFORE any HF operations, and if it returns True, the caller should set HF_HUB_OFFLINE=1 for the server subprocess env ONLY (not global environment). Args: snapshot_dir: Path to the model snapshot directory weight_files: List of weight file paths to validate (must be non-empty) model_name_or_path: Model identifier for logging Returns: True if cache is complete and offline mode can be enabled, False otherwise zJCI_OFFLINE: No weight files provided, skip offline, keep online allowed - FNrzCI_OFFLINE: Marker hit (marker=z6), skip re-validation, offline mode will be enabled - TrzBCI_OFFLINE: No marker found, performing full validation (snapshot=z, allow_remote_check=z) - rr;r r z+CI_OFFLINE: Missing config/tokenizer files z&, skip offline, keep online allowed - z&CI_OFFLINE: Weight validation failed (z'), skip offline, keep online allowed - zDCI_OFFLINE: Cache validation PASSED, offline mode will be enabled - )rv) rrrur)rrr`huggingface_hub.constants constantsHF_HUB_OFFLINEr_validate_sharded_modelrw)rrrrCrM marker_namer rr config_validmissing_config_files weights_valid error_msg_rrr0ci_validate_cache_and_enable_offline_if_completesj!    rcomponent_namecomponent_infocCsH|}d|vr dSd|vrdSd|vrdSd|vrdSd|vr"dSdS)a< Infer component type from component name and info. Args: component_name: Name of the component (e.g., "scheduler", "tokenizer") component_info: Component info from model_index.json (e.g., ["diffusers", "SchedulerClass"]) Returns: Component type string for validation rules scheduler tokenizerimage_processorfeature_extractor processormodel)lower)rr name_lowerrrr_infer_component_types r component_dircomponent_typec Cs|dkr#ddg}|D]}tj||}t||rd|fSq d|fS|dkrgd}tj|d}t|dr;d|fStj|d }tj|rgtj|rgztj|d kr\d|fWSWn tyfYnwtj|d }tj|d }t|d rtj|rd|fSd|fS|d vrddg}|D]}tj||}t||rd|fSqd|fSdg}tj|d} t| drd|fSd|fS)a( Check if component has required config files based on type. Args: component_dir: Path to component directory component_type: Type of component (scheduler, tokenizer, processor, model, etc.) Returns: Tuple of (has_valid_config, list_of_candidates_tried) rzscheduler_config.jsonr~TFr)rrzvocab.json+merges.txtrrrz vocab.jsonz merges.txt)rrrr)rrr1r}rrzr{rt) rr candidates candidatecandidate_pathtokenizer_json_pathtokenizer_model_path vocab_path merges_pathrrrr_check_component_config sX         rcCs4gd}|D]}ttj||}|rdSqdS)z Check if component directory has weight files. Args: component_dir: Path to component directory Returns: True if weight files found, False otherwise ) *.safetensors*.binz*.ptz*.pthTF) glob_moduleglobrrr1)rweight_patternspatternrrrr_check_component_weights[s r componentsmax_showcCsDt||kr d|S|d|}t||}d|d|dS)z Format component list with truncation. Args: components: List of component names max_show: Maximum number to show before truncating Returns: Formatted string like "comp1, comp2, comp3" or "comp1, comp2, +3 more" , Nz, +z more)rr1)rrshown remainingrrr_format_component_listos   rc Cs8tjddk}tj|d}tj|sdSzt|ddd }t|}Wdn1s/wYWn%tj t fyZ}z|rOd d |fWYd}~SWYd}~d Sd}~wwd d | D}|shdSg}g}g} g} | D]N\} } tj|| } tj | s| | qtt| | }t| |\}}|s| | |rd|}| | d|dqt|dv}|rt| }|s| | qt|s|s| rg}|rt|}|r| d|n| d||r|rd| }| d|n t|}| d|| rt| }| d|d d|fSdS)a Validate diffusion model (diffusers pipeline) cache completeness. This validation is based on model_index.json as the single source of truth. Error reporting uses coarse-grained error codes unless verbose mode is enabled. Error codes: - DIFFUSERS_INVALID_INDEX: model_index.json missing or corrupted - DIFFUSERS_INVALID_COMPONENTS: model_index.json has no valid components - DIFFUSERS_MISSING_COMPONENT: component directory or config missing - DIFFUSERS_MISSING_WEIGHTS: component weights missing Args: snapshot_dir: Path to the model snapshot directory Returns: Tuple of (is_valid, error_message) - (True, None) if validation passed - (False, error_code_with_components) if validation failed SGLANG_CI_VALIDATE_VERBOSE1zmodel_index.json)Fz3DIFFUSERS_INVALID_INDEX: model_index.json not foundr7rr8NFz8DIFFUSERS_INVALID_INDEX: model_index.json parse error - )Fz3DIFFUSERS_INVALID_INDEX: model_index.json corruptedcSs*i|]\}}|dst|tr||qS)r)rrIlist)r@rAvrrr sz-_validate_diffusion_model..)Fz9DIFFUSERS_INVALID_COMPONENTS: no valid components definedrz (tried: ))rrrrrz$DIFFUSERS_MISSING_COMPONENT (dirs): z"DIFFUSERS_MISSING_COMPONENT(dir): z; z'DIFFUSERS_MISSING_COMPONENT (configs): z"DIFFUSERS_MISSING_COMPONENT(cfg): zDIFFUSERS_MISSING_WEIGHTS: z | TN)rr/r0rr1rrFrGrHrsrtrr rrrrr)rverbosemodel_index_pathrN model_indexrr missing_dirsmissing_configsmissing_configs_verbosemissing_weightsrrrrhas_valid_configconfig_candidatescandidates_str needs_weights has_weightserrorsdir_str config_str weight_strrrr_validate_diffusion_models         rc Csz|sdStj|}tjddk}t||||d\}}|s*d|}dd|fSt||\}} } |s;dd| fSd S) a= Validate cache and return detailed reason for failure. This function performs validation without relying on shared validation markers. Used by prevalidate_cached_models.py to provide detailed feedback. Args: snapshot_dir: Path to the model snapshot directory weight_files: List of weight file paths to validate model_name_or_path: Model identifier for logging Returns: Tuple of (success, reason): - (True, None) if validation passed - (False, reason_str) if validation failed with specific reason )FzNo weight files providedrrrrFz Missing config/tokenizer files: zWeight validation failed: r)rrr`r/r0rr1r) rrrr r rrmissing_files_strrrrrrr#validate_cache_with_detailed_reasons"   rrequires_hf_quant_configc& sddg}|D]}tjtj|sdSqgd}tfdd|D}|s*dStjd}tj|rz|t|ddd  }t|}Wd n1sOwY|d i} | rt | t rt } | D]\} } t | t rd | vr| d d } | | dqh| D]*}tj|}tj|std|WdStj|std|WdSqWntjttfy}z td|WYd }~nd }~wwtjd}tj|}|rLzLt|ddd  }t|}Wd n1swY|di}|r+t |}|D]}tj|}tj|s)td|WdSqWntjttfyK}ztd||WYd }~dSd }~wwttjd}|s[dStd}i}|D]A}tj|}||}|r|d}t|d}t|d}|d|} | |vr|t d|| <|| d|qd| D]*\} }!|!d}|!d}"t td|d}#|#|"}$|$rtdt |$| dSq|rtjd }%tj|%sdSd!S)"a& Lightweight runtime validation for cache completeness. This is used during test runs to ensure the current runner's cache is complete before enabling offline mode. Much faster than full validation as it only checks file existence, not corruption. Args: snapshot_dir: Path to the model snapshot directory requires_hf_quant_config: If True, hf_quant_config.json must exist (required for modelopt quantization) Returns: True if cache is complete, False otherwise r~rFrc3s&|]}tjtj|VqdSr?)rrrr1)r@fnamerrrrDTs z-validate_cache_lightweight..r7rr8Nrrrrz-Custom module file not in snapshot: %s for %sz,Custom module path exists but not a file: %srzmodel.safetensors.index.json weight_mapz/Index validation failed: missing shard %s in %sz$Failed to validate index file %s: %srz"(.*?)-(\d+)-of-(\d+)\.safetensors$r\-of-)total found_shardsrrz7Shard validation failed: missing shards %s in %s for %srT)!rrrr1anyrFrGrHr0rIrJrrrrrrrrzrsrtrvaluesrrrecompiler`matchgroupintrangesorted)&rrrQrr has_tokenizerrrNrrrrqrrrrr index_path has_index index_datarrequired_shards shard_name shard_pathsafetensors_files shard_pattern shard_groups base_namerprefixshard_id total_shards group_key group_inforexpected_shardsmissing_shards hf_quant_pathrrrvalidate_cache_lightweight1s                  rc Csz#tj|ddd}t|}WdWdS1swYWdStyC}ztd|t|jt |WYd}~dSd}~ww)z Validate that a safetensors file is readable and not corrupted. Args: file_path: Path to the safetensors file Returns: True if the file is valid, False if corrupted ptcpu) frameworkdeviceNTz0Corrupted safetensors file detected: %s - %s: %sF) safetensors safe_openrkeysrrritype__name__r)rxrNrrrrr_validate_safetensors_files"  r&c Csbzddl}|j|ddddWdSty0}ztd|t|jt|WYd}~dSd}~ww)a Validate that a PyTorch .bin file is readable and not corrupted. This catches corruption issues like truncated downloads or invalid archives that would cause errors like: "RuntimeError: PytorchStreamReader failed reading file data/X: invalid header or archive is corrupted" Args: file_path: Path to the .bin file Returns: True if the file is valid, False if corrupted rNrTF) map_location weights_onlymmapz0Corrupted PyTorch bin file detected: %s - %s: %s)torchrHrrrir$r%r)rxr*rrrr_validate_pytorch_bin_filesr+c Csddt|D}|sdS|D]}tj||}tj|rgtj|sgztj|}t|t d|tj|rBt|Wnt y\}z t d||WYd}~nd}~wwdd|d fSzft | }t |}Wdn1s|wY|d i}|sWqt|} g} | D]} tj|| } tj| s| | q| rdd t| d |d | ddt| dkrdndfWSWqty}zt d||dd|dfWYd}~Sd}~wt y }z t d||WYd}~qd}~wwdS)ad Check if all files listed in safetensors index files actually exist on disk. This catches cases where the snapshot directory exists but files are missing (e.g., due to incomplete downloads or corrupted cache). Args: snapshot_dir: Path to the model snapshot directory Returns: Tuple of (all_exist, error_message) cSsg|] }|dr|qS).safetensors.index.json)endswithr@rNrrr s  z,_check_index_files_exist..rz/Removed broken index symlink: %s (blob missing)z&Failed to remove broken symlink %s: %sNFzBroken index file symlink: z (cleaned up, will re-download)rzMissing z file(s) from index : rz...z Failed to read index file %s: %sz Index file z unreadable (will re-download))rlistdirrr1islinkrr!rkrrirerrorrFrGrHr0rrrrFileNotFoundError) r index_files index_filer  blob_pathrrNr rrQrryrxrrr_check_index_files_exist sn           4   r9cCst|\}}|s d|gfStd}i}|D]Q}tj|}||}|rg|d} |d} |d} | d| d| } | |vrN| t| | ggd|| <t|d } || d  | || d  |qg}| D]\} }|d }t |d }t t d|d}||}|rdd | dt |gfS|ddkr|d D] }t|s| |qn|ddkr|d D] }t|s| |q|ddkrtj||dd}tj|sddtj|gfSqn|rdddd|D|fSddgfS)a` Validate that all model shards are present and not corrupted. Args: snapshot_dir: Path to the model snapshot directory weight_files: List of weight file paths Returns: Tuple of (is_valid, error_message, corrupted_files) - corrupted_files: List of file paths that are corrupted (for selective cleanup) Fz'(.*?)-(\d+)-of-(\d+)\.(safetensors|bin)rrrr)rrr[rfilesr\rr;rzMissing shards in r0r[r!binrr,zMissing index file: zCorrupted shard files: cSsg|]}tj|qSr)rrr`r.rrrr/sz+_validate_sharded_model..TN)r9rrrrr`rrrrrrrrr&r+r1r)rrindex_check_valid index_errorrrrNrrrtotal_shards_strr[rrcorrupted_filesrrrrrr7rrrr_s|                   rr@c Csd}|D]o}zPtj|r.z)Downloaded model files are corrupted for r0z=. The corrupted files have been removed. Will retry download.F)extendrrrrr1r-rr&rr`r+rCrr)rRrSrrrr@rNrrTr _validate_weights_after_downloads8rV cache_dircCst|dd}z,ddl}|p|jj}tj |r8tj |d}tj |ddtj |d|dWSWn t yBYnwtj d rNd |Sd |S) aM Generate a unique lock file path for download coordination. In CI environments where multiple containers share an NFS-mounted HF cache, the lock file is placed on the shared cache directory so ALL containers coordinate on the same lock. This prevents cross-container .incomplete file race conditions. Falls back to /dev/shm (container-local) for non-CI or when the cache dir is not accessible. Args: model_name_or_path: Model identifier cache_dir: HF cache directory (None to use default) Returns: Path to the lock file Nrz .sglang_locksTr- download_z.lockz/dev/shmz/dev/shm/sglang_download_lock_z/tmp/sglang_download_lock_) r#r$r%r&rr HF_HUB_CACHErrr r1r2r)rrWkey_hashreffective_cache_dirlock_dirrrr_get_lock_file_paths      r^c Cs(zzddl}|p |jj}|jjdg|d}tj||d}tj|s)WdSt tj|d}d}|D]4}zt ||d7}t dtj|Wq7tyk} zt d tj|| WYd} ~ q7d} ~ ww|dkrxt d ||||WSty} z t d | WYd} ~ dSd} ~ ww) a Remove stale .incomplete files from the model's blobs directory. This is lighter than _cleanup_corrupted_model_cache (which deletes the entire cache). We only remove .incomplete files so snapshot_download starts fresh on retry, preserving any successfully downloaded blobs. Args: model_name_or_path: Model identifier (e.g., "meta-llama/Llama-2-7b-hf") cache_dir: HF cache directory (None to use default) Returns: Number of .incomplete files removed rNmodelsrrMrNrzRemoved incomplete blob: %sz'Failed to remove incomplete blob %s: %sz.Cleaned up %d .incomplete blob(s) for %s in %sz'Failed to clean up incomplete blobs: %s)rrrZREPO_ID_SEPARATORr1rrrr rrrkrrr`rtrir) rrWrr\repo_folder_namerOrPremovedrNrrrr_cleanup_incomplete_blobssF    rcr max_retriescCsddl}ddl}ddlm}ddlm} Gddd| } t||} tdt | |j | dd d } td t ||  td t |z&dd l m } | ||||}|durmtdt ||WWdSWnty}z td|WYd}~nd}~wwt||}|dkrtd||d}t|D]}z|||||| ||jjdd}WnWttfy}zHtdt |d||t|j|||dkrdd|}td|t||t|WYd}~qtd|d|dt|jd||d}~wwt|||}|r|WdS||dkr4ttd|d|dd|dqtd|d|d |WdS1sLwYdS)!a CI-specific download with validation and automatic retry on corruption. This function handles the download of model weights in CI environments, with automatic validation and retry logic for handling corrupted downloads. Uses filelock.FileLock on the shared HF cache directory to coordinate downloads across all processes AND all containers sharing the same NFS-mounted cache. Only one process downloads at a time; others wait for the lock then use the cached result. Args: model_name_or_path: The model name or path allow_patterns: The allowed patterns for weight files ignore_patterns: The patterns to filter out weight files cache_dir: The cache directory to store model weights revision: The revision of the model max_retries: Maximum number of download retries if corruption is detected Returns: str: The path to the downloaded model weights Raises: RuntimeError: If download fails after max_retries attempts rN)snapshot_download)tqdmcseZdZfddZZS)z;ci_download_with_validation_and_retry..DisabledTqdmcsd|d<tj|i|dS)NTdisable)super__init__)selfargskwargs __class__rrriLszDci_download_with_validation_and_retry..DisabledTqdm.__init__)r% __module__ __qualname__ri __classcell__rrrmr DisabledTqdmKsrrz,[CI Download] Process %d using lock file: %si)timeoutrXz7[CI Download] Process %d waiting to acquire lock for %sz-[CI Download] Process %d ACQUIRED lock for %s)$_find_local_hf_snapshot_dir_unlockedzf[CI Download] Process %d found cached model after acquiring lock (downloaded by another container): %sz>[CI Download] Re-check for cached model failed (non-fatal): %szO[CI Download] Pre-download cleanup: removed %d stale .incomplete file(s) for %sr)rSignore_patternsrW tqdm_classr local_files_only max_workerszJ[CI Download] Process %d hit download error (attempt %d/%d) for %s: %s: %s r\zB[CI Download] Cleaning up .incomplete files and retrying in %ds...zDownload failed for z after z. attempts due to download errors. Last error: r0zRetrying download for z (attempt rz)...z/Downloaded model files are still corrupted for zm attempts. This may indicate a persistent issue with the model files on Hugging Face Hub or network problems.)filelockrrre tqdm.autorfr^rrArgetpidFileLock$sglang.srt.model_loader.weight_utilsrurrrcrrrr5rtrir$r%timesleep RuntimeErrorrVr)rrSrvrWr rdr{rrerfrrlock_file_pathlockru cached_pathrcleanedrRattemptbackoffrrrr%ci_download_with_validation_and_retry%s!            !c&r model_pathc Csddlm}|s dStj|rdSzddl}|jj}tj||jj dg| d}tj|s7WdStj|d}tj|sGWdSg}t |D]N}tj||}tj|s^qNt tj|d} | D]} tj| ssqjt| s||| qjt tj|d} | D]} tj| sqt| s|| qqN|rtd t||t||WdSWdSty} z td | WYd} ~ dSd} ~ ww) a Validate and clean corrupted safetensors files in HF cache before loading. This function is needed because HFRunner (used in tests) calls transformers' from_pretrained() directly, which bypasses SGLang's weight validation. Corrupted cached files can cause cryptic errors like "EOF while parsing" from safetensors. Only runs in CI to avoid overhead for regular users. Args: model_path: Model identifier (e.g., "meta-llama/Llama-2-7b") r)is_in_ciNr_r snapshotsrrzRHFRunner: Found %d corrupted weight file(s) for %s. Removing to force re-download.z*HF cache validation failed (non-fatal): %s) sglang.utilsrrrr rrrZr1r`rr2rrrr&rr+rrirrCrr)rrrrWrI snapshots_dirr@r<rrsf_file bin_filesbin_filerrrrci_validate_and_clean_hf_cachesj           rr?)NNF)r)F)r):__doc__rrr#rGloggingrrrGrdrtypingrrrr!sglang.srt.utilsr getLoggerr%rrrrboolrr)r5r6rJrPrrlrmrurwr}rrrrrrrrrrr&r+r9rrCrJrQrVr^rcrrrrrrsT   3% 5788 S m N } 3 " R a > # l 9 +; 7