o پi4 @s dZddlZddlZddlmZddlZddlmZddlmm Z ddl m Z ddl mZmZmZddlmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZm Z dd l!m"Z"eZ#eZ$eZ%eZ&eZ'eZ(e#sye(rdd l)m*Z*m+Z+m,Z,ne'rdd l)m*Z*m-Z-m+Z+m,Z,erddl.Z.e/e0Z1GdddeZ2GdddeZ3GdddeZ4Gdddej5Z6GdddeZ7GdddeZ8Gdddej5Z9e:ej:dde4e6e8dZ;    d+d ed$eej?d%ej5f d&d'Z@d(e fd)d*ZAdS),z&Fused operators for activation layers.N)Optional)PretrainedConfig)divideget_tensor_model_parallel_rank$get_tensor_model_parallel_world_size)envs)QuantizationConfig)MultiPlatformOp)get_global_server_args)cpu_has_amx_supportis_cpuis_cudais_hipis_npuis_xpuset_weight_attrs)resolve_obj_by_qualname) gelu_and_mulgelu_tanh_and_mul silu_and_mul)r gelu_quickrrcseZdZfddZdejdejfddZdejdejfddZdejdejfd d Zdejdejfd d Z dejdejfd dZ Z S) SiluAndMulcs.tj|i|tjdur|j|_dSdSN)super__init__r rl_on_policy_targetforward_native_forward_method)selfargskwargs __class__P/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/activation.pyr@s  zSiluAndMul.__init__xreturncCs4|jdd}t|dd|f|d|dfS)N.)shapeFsilurr%dr#r#r$rEs&zSiluAndMul.forward_nativecCD|jdd}|jdd|f}tj||j|jd}t|||SNr'r(dtypedevicer)torchemptyr1r2rrr%r- output_shapeoutr#r#r$ forward_cudaI  zSiluAndMul.forward_cudacCs tr tjj|}|S||Sr)_is_cpu_amx_availabler4ops sgl_kernelsilu_and_mul_cpurrr%r8r#r#r$ forward_cpuPs zSiluAndMul.forward_cpucCst|}|Sr) torch_npu npu_swiglur?r#r#r$ forward_npuWs zSiluAndMul.forward_npucCr.r/r3r6r#r#r$ forward_xpu[r:zSiluAndMul.forward_xpu) __name__ __module__ __qualname__rr4Tensorrr9r@rCrD __classcell__r#r#r!r$r?s rcseZdZdfdd ZdejdejfddZdejdejfdd Zdejdejfd d Zdejdejfd d Z dejdejfddZ dejdejfddZ Z S) GeluAndMultanhcst||_dSr)rr approximate)rrLr!r#r$rds  zGeluAndMul.__init__r%r&cCsn|jdd}|jdd|f}tj||j|jd}|jdkr't|||S|jdkr3t|||Std)Nr'r(r0rKnonez$GeluAndMul only support tanh or none) r)r4r5r1r2rLrr RuntimeErrorr6r#r#r$ _forward_implhs   zGeluAndMul._forward_implcCs:|jdd}tj|dd|f|jd|d|dfS)Nr'r(.rL)r)r*gelurLr,r#r#r$rts,zGeluAndMul.forward_nativecCsBtr|jdkrtjj|Str|jdkrtjj|S||S)NrKrM)r;rLr4r<r=gelu_tanh_and_mul_cpugelu_and_mul_cpurrr%r#r#r$r@xs  zGeluAndMul.forward_cpucC ||SrrOrTr#r#r$r9 zGeluAndMul.forward_cudacCrUrrVrTr#r#r$rDrWzGeluAndMul.forward_xpucCs<tjr ||Stj|d|jdkrdnddd\}}|S)Nr'rKrT)dimrL activate_left)r"SGLANG_NPU_FORWARD_NATIVE_GELUTANHgetrrA npu_geglurL)rr%y_npugelu_npur#r#r$rCs   zGeluAndMul.forward_npu)rK) rErFrGrr4rHrOrr@r9rDrCrIr#r#r!r$rJcs rJc@s8eZdZdejdejfddZdejdejfddZdS)NewGELUr%r&c Cs:tdtj}d|dt||dt|dS)Ng@?g?gHm?g@)mathsqrtpir4rKpow)rr%cr#r#r$rs*zNewGELU.forward_nativecCrUrrrTr#r#r$r9s zNewGELU.forward_cudaN)rErFrGr4rHrr9r#r#r#r$r`sr`c@s&eZdZdZdejdejfddZdS)ReLU2zQ Applies the squared Rectified Linear Unit function. y = max(0, x)^2 r%r&cCst|}||Sr)r*relurTr#r#r$forwards z ReLU2.forwardN)rErFrG__doc__r4rHrjr#r#r#r$rhsrhc@sdeZdZdejdejfddZdejdejfddZdejdejfddZdejdejfd d Zd S) QuickGELUr%r&cCs|td|S)NgZd;?)r4sigmoidrTr#r#r$rszQuickGELU.forward_nativecCrUrrgrTr#r#r$r9rWzQuickGELU.forward_cudacCs$tj|j|j|jd}t|||S)Nr0)r4r5r)r1r2rr?r#r#r$ forward_hips zQuickGELU.forward_hipcCs t|Sr)rA npu_fast_gelurTr#r#r$rCrWzQuickGELU.forward_npuN) rErFrGr4rHrr9rnrCr#r#r#r$rls rlc seZdZdZddddejdfdededed ed ejd ef fd d Z dej dej fddZ dej dej fddZ dej dej fddZ ZS)XIELUz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?ragưF alpha_p_init alpha_n_initbetaepsr1with_vector_loadsc stttttj||ddd|_ ttttj|||ddd|_ | dtj||d| dtj||d||_ t |j |_t |j |_d|_zFddl}tjj|_d}zddlm} | |j|_|d7}Wnty} z|d | d 7}|j|_WYd} ~ nd} ~ wwt |WdSty} zWYd} ~ dSd} ~ ww) Nr1rXrrsrtzUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.)!rrnn Parameterr4logexptensor unsqueezealpha_palpha_nregister_bufferrufloatrsdetachcpuitem _beta_scalarrt _eps_scalar_xielu_cuda_obj xielu.opsclassesxielurp torch._dynamorw _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_once) rrqrrrsrtr1rurmsgrwerrr!r#r$rsL     zXIELU.__init__r%r&c Cshtj|j}|jtj|j}t|dk||||j|tt ||j |||j|SNr) rx functionalsoftplusr~rsrr4whereexpm1minrt)rr%r~rr#r#r$ _xielu_pythons$zXIELU._xielu_pythoncCs|jdus Jd|j}|dkr|d}|dks|dkr-|dd|d}||jkr:td||j|j||j |j |j |j |j }||S)z>Firewall function to prevent torch.compile from seeing .item()Nz"XIELU CUDA object must not be Nonerr'rXzWarning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s). Note: For SGLang this may be expected if sending[B*S,D] instead of [B,S,D].)rr)rYr}viewsizerrrjr~rrrru)rr%original_shaperesultr#r#r$rs,     zXIELU._xielu_cudainputcCs8|jdur|jrtjs||Std||S)Nz:torch._dynamo is compiling, using Python version of xIELU.) rr r4_dynamo is_compilingrrrr)rrr#r#r$rjs   z XIELU.forward)rErFrGrkr4bfloat16rr1boolrrHrrrjrIr#r#r!r$rps06 rpc sleZdZdZ  ddejdededee j ffdd Z d e j d e j fd d Z dejde j fddZZS)ScaledActivationzqAn activation function with post-scale parameters. This is used for some quantization methods like AWQ. TN act_moduleintermediate_sizeinput_is_parallel params_dtypecslt||_||_|rt}t||}n|}|dur t}t tj ||d|_ t |j d|j idS)Nrv weight_loader)rractrrrr4get_default_dtyperxryr5scalesrr)rrrrrtp_sizeintermediate_size_per_partitionr!r#r$r+s   zScaledActivation.__init__r%r&cCs|||jSr)rrrTr#r#r$rjAszScaledActivation.forwardparam loaded_weightcCsP|j}|jrt}|jd}||}|d||}|j|jks!J||dSr)datarrr)narrowcopy_)rrr param_datatp_rank shard_size start_idxr#r#r$rDs zScaledActivation.weight_loader)TN)rErFrGrkrxModuleintrrr4r1rrHrjryrrIr#r#r!r$r%s rrKrP)rQgelu_pytorch_tanhgelu_newrelu2rT act_fn_name quant_configrrrr&cCs^|}|tvrtd|dt|}|dur-||vr-|dur&tdt||||S|S)z#Get an activation function by name.zActivation function z is not supported.NzDintermediate_size must be specified for scaled activation functions.)lower_ACTIVATION_REGISTRY ValueErrorget_scaled_act_namesr)rrrrract_fnr#r#r$ get_act_fnXsrconfigcCs>t|dr|jdur|j}|dsJdt|StS)N$sbert_ce_default_activation_functionztorch.nn.modules.zVLoading of activation functions is restricted to torch.nn.modules for security reasons)hasattrr startswithrrxIdentity)r function_namer#r#r$%get_cross_encoder_activation_functionqs   r)NNTN)Brkloggingrbtypingrr4torch.nnrxtorch.nn.functionalrr* transformersrsglang.srt.distributedrrrsglang.srt.environr*sglang.srt.layers.quantization.base_configrsglang.srt.layers.utilsr sglang.srt.server_argsr sglang.srt.utilsr r r rrrr sglang.utilsr_is_cuda_is_npur;_is_cpu_is_hip_is_xpur=rrrrrA getLoggerrErrrJr`rrhrlrprGELUrstrrrr1rrr#r#r#r$sv        $  $/  n+