o پi@@sddlZddlZddlmZmZddlmZmZmZm Z ddl Z ddl m Z ddl mZeeZGdddeejZeGdd d ZdS) N) dataclassfield)AnyListOptionalUnion)ModelOptConfig)is_hipc@sTeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdS) LoadFormatautopt safetensorsnpcachedummy sharded_stategguf bitsandbytesmistrallayeredflash_rljaxremoteremote_instancerdma local_cachedfastsafetensorsprivateN)__name__ __module__ __qualname__AUTOPT SAFETENSORSNPCACHEDUMMY SHARDED_STATEGGUF BITSANDBYTESMISTRALLAYEREDFLASH_RLJAXREMOTEREMOTE_INSTANCERDMA LOCAL_CACHEDFASTSAFETENSORSPRIVATEr2r2R/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/configs/load_config.pyr s&r c@sdeZdZUdZejZeeefe d<dZ e ee d<e e dZe eee fe d<dZe eeeefe d<dZe ee d<d Zee d <dZe ee d <dZe ee d <dZe ee d <dZe eee d<dZe ee d<dZe ee d<dZe ee d<dZe ee d<dZe ee d<dZe ee d<dZ e ee d<dZ!e ee d<ddZ"dddZ#dS) LoadConfiga download_dir: Directory to download and load the weights, default to the default cache directory of huggingface. load_format: The format of the model weights to load: "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. "pt" will load the weights in the pytorch bin format. "safetensors" will load the weights in the safetensors format. "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. "dummy" will initialize the weights with random values, which is mainly for profiling. "bitsandbytes" will load nf4 type weights. "flash_rl" will load weights with support for RL training with quantized models, enabling efficient weight reloading. ignore_patterns: The list of patterns to ignore when loading the model. Default to "original/**/*" to avoid repeated loading of llama's checkpoints. decryption_key_file: If set, decrypts the output files with a password read from this file (after PBKDF2). decrypt_max_concurrency: The maximum number of concurrent processes to decrypt the safetensor files. -1 means no limit. # ModelOpt-specific loading options modelopt_checkpoint_restore_path: Optional[str] = None modelopt_checkpoint_save_path: Optional[str] = None modelopt_export_path: Optional[str] = None load_formatN download_dir)default_factorymodel_loader_extra_configignore_patternsdecryption_key_filedecrypt_max_concurrencytp_rank.remote_instance_weight_loader_seed_instance_ip8remote_instance_weight_loader_seed_instance_service_port6remote_instance_weight_loader_send_weights_group_ports%remote_instance_weight_loader_backend-remote_instance_weight_loader_transfer_engine modelopt_checkpoint_restore_pathmodelopt_checkpoint_save_pathmodelopt_export_pathmodelopt_configrl_quant_profiledraft_model_idxcCs|jpi}t|trt||_||jdur(t|jdkr(t d|jndg|_|j dur>t |j |j |jd|_ dSdS)Nrzs z2LoadConfig._verify_load_format..z load format 'z7' is not supported in ROCm. Supported load formats are )rLr5rMlowerr r __members__ ValueError)rTr5rocm_supported_load_formatr2rYr3rPvs    zLoadConfig._verify_load_format)rVN)$rrr__doc__r r r5rrM__annotations__r6rrdictr8r9rr:r<intr=r>r?r@rArBrrCrDrErFrrGrHrUrPr2r2r2r3r4$s.  r4)enumlogging dataclassesrrtypingrrrrrN"sglang.srt.configs.modelopt_configrsglang.srt.utilsr getLoggerrrRrMEnumr r4r2r2r2r3s