o پi@sddlmZddlZddlZddlmZddlmZddlZddl m Z ddl m Z ddl mZddlmZeeZeGd d d ZdS) ) annotationsN) dataclass)Any)is_nunchaku_available)current_platform) init_logger) StoreBooleanc@s~eZdZUdZdZded<dZded<dZded<dZd ed <dZ ded <dddZ dddZ e dddZ edddZdS)NunchakuSVDQuantArgszCLI-facing configuration for Nunchaku (SVDQuant) inference. This is intentionally lightweight and only contains arguments needed to construct `runtime.layers.quantization.nunchaku_config.NunchakuConfig`. Fboolenable_svdquantNz str | Nonequantized_model_pathquantization_precisionz int | Nonequantization_rankquantization_act_unsignedreturnNonecCs|jr |js d|_|jr|jsdSd}d}tj|j}td|}|r6|\}}|dkr0dnd}t|}|j durO|p>d|_ |rOt d|j d|j|j durj|pWd |_ |rlt d |j d|jdSdSdS) z6infer precision and rank from filename if not providedTNzsvdq-(int4|fp4)_r(\d+)fp4nvfp4int4z#inferred --quantization-precision: z from --quantized-model-path: zinferred --quantization-rank: ) r r ospathbasenameresearchgroupsintr loggerinfor)selfinferred_precision inferred_rankfilenamematchp_strr_strr&^/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/configs/quantization.py_adjust_config$s<          z#NunchakuSVDQuantArgs._adjust_configc Cs ||js dStstdtj}g}t|D]/}tj |\}}|dkr9| d|d||dq|dvrK| d|d||dq|rYtdd |d |j s`td t sgtd |jd vrutd|jd|jdkrtd|jddS)NzONunchaku SVDQuant is only supported on NVIDIA CUDA GPUs (Ampere SM8x or SM12x). zcuda:z (SMz , Hopper)) )zNunchaku SVDQuant is currently only supported on Ampere (SM8x) or SM12x GPUs; Hopper (SM90) is not supported. Unsupported devices: z, z*. Disable it with --enable-svdquant false.z;--enable-svdquant requires --quantized-model-path to be setzNunchaku is enabled, but not installed. Please refer to https://nunchaku.tech/docs/nunchaku/installation/installation.html for detailed installation methods.)rrz"Invalid --quantization-precision: z. Must be one of: int4, nvfp4rzInvalid --quantization-rank: z . Must be > 0)r(r ris_cuda ValueErrortorchcuda device_countrangeget_device_capabilityappendjoinr rr r)rr1 unsupportedimajorminorr&r&r'validateIsN      zNunchakuSVDQuantArgs.validatecCsd|jdttjdd|jdttjdd|jdtddd|jd tdd d|jd ttjd ddS) Nz--enable-svdquantz0Enable Nunchaku SVDQuant (W4A4-style) inference.)actiondefaulthelpz--quantized-model-pathzqPath to pre-quantized Nunchaku weights. Can be a single .safetensors file or a directory containing .safetensors.)typer<r=z--quantization-precisionzfQuantization precision: int4 or nvfp4. If not specified, inferred from model path or defaults to int4.z--quantization-rankz`SVD low-rank dimension (e.g., 32). If not specified, inferred from model path or defaults to 32.z--quantization-act-unsignedz4Use unsigned activation quantization (if supported).) add_argumentrr r strr rr)parserr&r&r' add_cli_args}s<  z!NunchakuSVDQuantArgs.add_cli_argskwargsdict[str, Any]'NunchakuSVDQuantArgs'c CsD|t|d|j|d|j|d|dt|d|jdS)Nr r r rr)r r r rr)r getr r r)clsrCr&r&r' from_dicts zNunchakuSVDQuantArgs.from_dict)rr)rCrDrrE)__name__ __module__ __qualname____doc__r __annotations__r r rrr(r: staticmethodrB classmethodrHr&r&r&r'r s        %4 #r ) __future__rrr dataclassesrtypingrr/Isglang.multimodal_gen.runtime.layers.quantization.configs.nunchaku_configr'sglang.multimodal_gen.runtime.platformsr1sglang.multimodal_gen.runtime.utils.logging_utilsrsglang.multimodal_gen.utilsrrIrr r&r&r&r's