o ۾iY8 @sTdZddlZddlmZddlmmZddlmZddl m Z ddl m Z m Z ddlmZdejdejd ed ejfd d Zdejd ejdejd ed eejejff ddZdejdejdejd ed ejf ddZ d#dedejdefddZe d Gddde Ze dGddde Ze dGdd d e ZGd!d"d"ejZdS)$zCustom normalization layers.N)rocm_aiter_ops)CustomOp)rms_norm_batch_invariantvllm_is_batch_invariant)current_platformxweightvariance_epsilonreturncCs<ddlm}trt|||St|}||||||SNr) _custom_ops)vllmr rrtorch empty_likerms_norm)rrr opsoutrX/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/layers/layernorm.pyrs   rresidualcCsBddlm}trt||||||fS|||||||fSr )r r rrfused_add_rms_norm)rrrr rrrrr#s  rbiascCs,ddlm}t|}|||||||Sr )r r rr poly_norm)rrrr rrrrrr8s  rFwith_fused_adddtype use_aitercCs8|o |tjtjfv}|r|rtjS|rtjS|rtStSN)rfloat16bfloat16rrms_norm2d_with_addrrrrrrrrdispatch_rocm_rmsnorm_funcHsr!cseZdZdZ    ddedededBded ejdBd df fd d Z e   d d ej dededejdej dBdej dBdedBd ej e ej ej fBfddZ d!d ej dej dBd ej e ej ej fBfddZ d!d ej dej dBd ej e ej ej fBfddZ d!d ej dej dBd ej e ej ej fBfddZ d!d ej dej dBd ej e ej ej fBfddZd efddZZS)"RMSNormzRoot mean square normalization. Computes x -> w * x / sqrt(E[x^2] + eps) where w is the learned weight. Refer to https://arxiv.org/abs/1910.07467 ư>NT hidden_sizeepsvar_hidden_size has_weightrr cst||_||_||krdn||_|pt}||_tj||d|_ |jr/t |j |_ t rIt}td||d|_td||d|_dSdS)NrFr T)super__init__r$r variance_size_overriderget_default_dtyper'onesrnn Parameterris_rocmris_rmsnorm_enabledr!rocm_norm_funcrocm_norm_func_with_add)selfr$r%r&r'r weight_dtypeaiter_rmsnorm_enabled __class__rrr*fs*   zRMSNorm.__init__rr orig_dtyperrr+c Cs|tj}|dur||}||}|jd|kr'td|d|jd|dur.|}n||krs    zGemmaRMSNorm.forward_cudar#r)rVrWrXrYrZr[r*r]rr^rfr_rgrMrPrarrr7rrcs`     rcrms_norm_gatedcseZdZdZ     ddedededBded ejdBd ej dBf fd d Z d dZ ddej dej dBdej fddZ ddej dej dBdej fddZZS) RMSNormGatedzRMS Normalization with optional gating. This is a native PyTorch implementation that supports: - Standard RMS normalization - Group RMS normalization - Optional gating with SiLU activation h㈵>NFr$r% group_sizenorm_before_gatedevicercsX||d}t||_ttj|fi||_|dd||_ ||_ | dS)aInitialize RMSNormGated. Args: hidden_size: Size of the hidden dimension eps: Epsilon for numerical stability group_size: If not None, do GroupNorm with each group having group_size elements. group_size=None is equivalent to group_size=hidden_size (i.e. there's only 1 group). norm_before_gate: If True and z is provided: out = norm(x) * silu(z) If False and z is provided: out = norm(x * silu(z)) device: Device to create parameters on dtype: Data type for parameters )rsrrN) r)r*r%r.r/remptyrregister_parameterrqrrreset_parameters)r4r$r%rqrrrsrfactory_kwargsr7rrr*^s    zRMSNormGated.__init__cCstjj|jdSr)rr.initones_r)r4rrrrv~szRMSNormGated.reset_parametersrzr cCs|dur|js|t|}|jdur-|djddd}|t||j}||j }n*ddl m }||d|jd }|djddd}|t||j}||d |j }|dure|jre|t|}|S) a Native PyTorch implementation of RMS normalization with gating. Args: x: Input tensor z: Optional gating tensor Returns: Normalized (and optionally gated) tensor If z is not None: - norm_before_gate=True: out = norm(x) * silu(z) - norm_before_gate=False: out = norm(x * silu(z)) Nr<r;Tr=r) rearrangez... (g d) -> ... g d)dz... g d -> ... (g d)) rrFsilurqrDrErrFr%reinopsr{)r4rrzrHx_normedrr{x_grouprrrrMs   zRMSNormGated.forward_nativec Cs,ddlm}|||j|j||j|j|jdS)Nr) rmsnorm_fn)rzr%rqrr)2vllm.model_executor.layers.fla.ops.layernorm_guardrrrr%rqrr)r4rrzrrrrrPs zRMSNormGated.forward_cuda)rpNFNNr)rVrWrXrYrZr[r\rrsrr*rvr^rMrPrarrr7rroRsJ   +rocs<eZdZdZd dedeffdd Zdejfdd Z Z S) LayerNormz Layer Normalization. r#r>r%csJt||_||_ttj|tjd|_ ttj |tjd|_ dS)Nr() r)r*r>r%r.r/rr-rArrdr)r4r>r%r7rrr*s zLayerNorm.__init__rcCs&t||jf|j|j|j|Sr)r} layer_normr[r>rrr%type_as)r4rrrrforwards zLayerNorm.forwardrm) rVrWrXrYrZr[r*rr^rrarrr7rrsr)F)rYrtorch.nnr.torch.nn.functional functionalr}vllm._aiter_opsrvllm.model_executor.custom_opr*vllm.model_executor.layers.batch_invariantrrvllm.platformsrr^r[rr_rrr\rr!registerr"rcroModulerrrrrsn        ^h