o پi!@sddlmZddlZddlmZmZddlmZmZm Z m Z m Z m Z ddl Z ddl mZerBddlmZddlmZmZddlmZGd d d eZGd d d eZGd ddeZGdddeZdddZdS)) annotationsN)ABCabstractmethod) TYPE_CHECKINGAnyDictListOptionalType)nn)MoeRunnerConfig) CombineInputDispatchOutput) WeightsMapperc@s2eZdZdZdddZeddd Zdd d ZdS)QuantizeMethodBasez+Base class for different quantized methods.layertorch.nn.ModulecOt)zXCreate weights for a layer. The weights will be set as attributes of the layer.NotImplementedError)selfr weight_argsextra_weight_attrsr^/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/quantization/base_config.pycreate_weightsz!QuantizeMethodBase.create_weightsreturn torch.TensorcOr)zxApply the weights in layer to the input tensor. Expects create_weights to have been called before on the layer.r)rrargskwargsrrrapplyszQuantizeMethodBase.apply nn.ModuleNonecCdS)zwProcess the weight after loading. This can be used for example, to transpose weights for computation. Nr)rrrrrprocess_weights_after_loading#sz0QuantizeMethodBase.process_weights_after_loadingN)rr)rrrr)rr"rr#)__name__ __module__ __qualname____doc__rrr!r%rrrrrs   rc@s,eZdZdZdd d Ze ddddZdS)LinearMethodBasez:Base class for different (maybe quantized) linear methods.rrinput_size_per_partitionintoutput_partition_sizes List[int] input_size output_size params_dtype torch.dtypecKr)aCreate weights for a linear layer. The weights will be set as attributes of the layer. Args: layer: The layer that is using the LinearMethodBase factory. input_size_per_partition: Size of the weight input dim on rank X. output_partition_sizes: Sizes of the output dim of each logical weight on rank X. E.g., output_partition_sizes for QKVLinear is a list contains the width of Wq, Wk, Wv on rank X. input_size: Size of the input dim of the weight across all ranks. output_size: Size of the output dim of the weight across all ranks. params_dtype: Datatype of the parameters. r)rrr+r-r/r0r1rrrrr.szLinearMethodBase.create_weightsNxrbiasOptional[torch.Tensor]rcCr)zwApply the weights in layer to the input tensor. Expects create_weights to have been called before on the layer.r)rrr3r4rrrr!G zLinearMethodBase.apply) rrr+r,r-r.r/r,r0r,r1r2N)rrr3rr4r5rr)r&r'r(r)rrr!rrrrr*+s  r*c@s.eZdZdd d Zdd dZedddZdS)FusedMoEMethodBaserr num_expertsr, hidden_sizeintermediate_size_per_partitionr1r2cKtr7r)rrr9r:r;r1rrrrrU z!FusedMoEMethodBase.create_weightsmoe_runner_configr cCr<r7r)rrr>rrrcreate_moe_runner`sz$FusedMoEMethodBase.create_moe_runnerdispatch_outputrrr cCr<r7r)rrr@rrrr!eszFusedMoEMethodBase.applyN) rrr9r,r:r,r;r,r1r2)rrr>r )rrr@rrr )r&r'r(rr?rr!rrrrr8Ss   r8cseZdZdZfddZed.ddZed/d d Zeed0d d Z e ed1ddZ eed2ddZ ed3ddZ ed3ddZe d4ddZe d5d d!Zed6d&d'Zed1d(d)Zd7d,d-ZZS)8QuantizationConfigz$Base class for quantization configs.cstt|_dSr7)super__init__dictpacked_modules_mappingr __class__rrrCqs  zQuantizationConfig.__init__rstrcCr)z Name of the quantization method.rrFrrrget_namevzQuantizationConfig.get_nameList[torch.dtype]cCr)z$List of supported activation dtypes.rrFrrrget_supported_act_dtypes{rKz+QuantizationConfig.get_supported_act_dtypesr,cCr)zMinimum GPU capability to support the quantization method. E.g., 70 for Volta, 75 for Turing, 80 for Ampere. This requirement is due to the custom CUDA kernels used by the quantization method. r)clsrrrget_min_capabilityr6z%QuantizationConfig.get_min_capability List[str]cCr)z7List of filenames to search for in the model directory.rrrrrget_config_filenamesz'QuantizationConfig.get_config_filenamesconfigDict[str, Any]'QuantizationConfig'cCr)z;Create a config class from the model's quantization config.r)rNrSrrr from_configrRzQuantizationConfig.from_config Optional[str]cCr$)z Detects if this quantization method can support a given checkpoint format by overriding the user specified quantization method -- this method should only be overwritten by subclasses in exceptional circumstances Nr)rN hf_quant_cfg user_quantrrroverride_quantization_methodsz/QuantizationConfig.override_quantization_methodcCsp|durdS|dd}|dkr"d|vrdSd|vs d|vr"d S|d ddkr,dS|d dd kr6d SdS) z3Shared ModelOpt quantization method override logic.N quant_algomodeloptFP8 modelopt_fp8NVFP4FP4 modelopt_fp4 quant_method)getupper)rNhf_quant_configrYr[rrr&_modelopt_override_quantization_methodsz9QuantizationConfig._modelopt_override_quantization_methodkeysrcCs.|D] }||vr||Sqtd|d)z1Get a value from the model's quantization config.zCannot find any of z$ in the model's quantization config.) ValueError)rSrhkeyrrr get_from_keyss  z QuantizationConfig.get_from_keysdefaultcCs&zt||WSty|YSw)z:Get a optional value from the model's quantization config.)rArkri)rSrhrlrrrget_from_keys_ors  z#QuantizationConfig.get_from_keys_orrrprefixOptional[QuantizeMethodBase]cCr)a7Get the quantize method to use for the quantized layer. Args: layer: The layer for the quant method. prefix: The full name of the layer in the state dict Returns: The quantize method. None if the given layer doesn't support quant method. r)rrrnrrrget_quant_methods z#QuantizationConfig.get_quant_methodcCr)zvReturns the activation function names that should be post-scaled. For now, this is only used by AWQ. rrFrrrget_scaled_act_namesrz'QuantizationConfig.get_scaled_act_nameshf_to_sglang_mapper'WeightsMapper'cCr$)a% Interface for models to update module names referenced in quantization configs in order to reflect the sglang model structure :param hf_to_sglang_mapper: maps from hf model structure (the assumed structure of the qconfig) to sglang model structure Nr)rrrrrrapply_weight_name_mapperr=z+QuantizationConfig.apply_weight_name_mapper)rrI)rrL)rr,)rrP)rSrTrrU)rrW)rSrTrhrPrr)rSrTrhrPrlrrr)rrrnrIrro)rrrs)r&r'r(r)rCrrJrM classmethodrO staticmethodrQrVrZrgrkrmrprqrt __classcell__rrrGrrAns:          rA method_classType[QuantizeMethodBase]rboolcCs,ttdd}t|dd}|duo||uS)z Not all quant methods have embedding implemented, so we need to check that it exists for our given method. We check this by making sure the function has been changed from the base implementation. embeddingN)inspectgetattr_staticr)rxbase_embeddingclass_embeddingrrr method_has_implemented_embeddingsr)rxryrrz) __future__rr|abcrrtypingrrrrr r torchr sglang.srt.layers.moe.moe_runnerr &sglang.srt.layers.moe.token_dispatcherr rsglang.srt.models.utilsrrr*r8rArrrrrs     (