o پi@svdZddlZddlZddlZddlmZmZmZm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZddlmZmZddlmZeeZejdejfd d Zd ed eefd dZ d ede e ej!effddZ"d edefddZ#dej!d efddZ$ddZ%dej&de'fddZ(ddddej)j*de ej)j+d e'd!ed"efd#e ed$e eeefddfd%d&Z,dS)'z+Utilities for selecting and loading models.N)AnyCallableDictIterableListOptionalTupleType)nn)get_class_from_dynamic_module) ModelConfig ModelImpl)deep_gemm_wrapperdtypeccs(t}t|dVt|dS)z0Sets the default torch dtype to the given dtype.N)torchget_default_dtypeset_default_dtype)r old_dtyperQ/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/model_loader/utils.pyset_default_torch_dtypes  r model_config architecturescst|D]y\}}|dkr qtjddpt}fddt|dddD}tt|d}|durBd|vr>td |d |d}jt j kr]t |d rY| sYtd |d d||<jt j kr}t |d rs| sst|dtd|d||<q|S)NTransformersForCausalLMauto_mapcs$i|]\}}|t|jjdqS))revision)r model_pathr).0namemodulerrr -s  z-resolve_transformers_arch..cSs|dS)Nrr)xrrr1sz+resolve_transformers_arch..)key AutoModelzCannot find model module. 'z' is not a registered model in the Transformers library (only relevant if the model is meant to be in Transformers) and 'AutoModel' is not present in the model config's 'auto_map' (relevant if the model is custom).is_backend_compatiblez#The Transformers implementation of z is not compatible with SGLang.z` has no SGlang implementation and the Transformers implementation is not compatible with SGLang.z%s has no SGLang implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.) enumerategetattr hf_configdictsorteditems transformers ValueError model_implr TRANSFORMERShasattrr&AUTOloggerwarning)rriarchr auto_modules model_modulerr rresolve_transformers_archsJ        r9returncsddlm}t|jdg}gd}|jdur"|j|vr"d|vr"dg}|tfdd|D}|jtj kr;d g}n |rC|jtj krHt ||}| |S) Nr) ModelRegistryr)fp8zcompressed-tensors gptq_marlin awq_marlinquark_int4fp8_moeMixtralForCausalLMQuantMixtralForCausalLMc3s|]}|vVqdS)Nr)rr6supported_archsrr osz)get_model_architecture..MindSporeForCausalLM) sglang.srt.models.registryr;r(r) quantizationget_supported_archsanyr/r MINDSPOREr0r9resolve_model_cls)rr;rmixtral_supportedis_native_supportedrrBrget_model_architectureYs     rNcCs t|dS)N)rNr rrrget_architecture_class_namexs rPmodelcCs:t|dr|jjddkr|jdddS|dSdS)Npost_load_weightsrDeepseekV3ForCausalLMNextNT)is_nextn)r1r)rrR)rQrrrrrR|s  rRcCstjo tjo |duS)zFShould we requant fp8 weights into UE8M0 format when loading the modelN)rENABLE_JIT_DEEPGEMMDEEPGEMM_SCALE_UE8M0)weight_block_sizerrr$should_deepgemm_weight_requant_ue8m0s rXweightcCs"t|dd}|dur dS|jdkS)aReturn True if we should load the given weight asynchronously. For host (CPU) tensors, using a threadpool can overlap H2D copies and improve throughput. For device tensors, threading often adds overhead (e.g., GIL contention) without benefit, so we do it synchronously. deviceNFcpu)r(type)rYrZrrrshould_async_loads  r]r) func_args func_kwargsexecutorfutures use_asyncfunc.r^r_cCsD|duri}|r||j|g|Ri|dS||i|dS)aSubmit a task to the executor if async loading is enabled. Parameters (keyword-only): - executor: ThreadPoolExecutor used to submit background tasks - futures: a list collecting the submitted Future objects - use_async: whether to submit to executor or run inline - func: the callable to run - func_args: positional args for the callable (defaults to empty tuple) - func_kwargs: keyword args for the callable (defaults to empty dict) N)appendsubmit)r`rarbrcr^r_rrrmaybe_executor_submits "rf)-__doc__concurrent.futures concurrent contextlibloggingtypingrrrrrrrr rr-r !transformers.dynamic_module_utilsr sglang.srt.configs.model_configr r sglang.srt.layersr getLogger__name__r3contextmanagerrrliststrr9ModulerNrPrRrXTensorboolr]raThreadPoolExecutorFuturerfrrrrsJ(     ;