o ̳i@s<ddlZddlmmZddlmZGdddejZdS)N)nncsFeZdZdZd dededdffdd Zd ejdejfd d Z Z S) RMSNormz Root Mean Square Normalization in fp32. See: https://pytorch.org/docs/stable/generated/torch.nn.RMSNorm.html Args: dim (int): embedding size eps (float): small value to avoid division by zero. Default: 1e-6 ư>dimepsreturnNcs.t|f|_||_tt||_dS)N) super__init__normalized_shaperr Parametertorchonesscale)selfrr __class__N/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/modules/rms_norm.pyr s zRMSNorm.__init__xcCs$tj||j|j|jd|jS)z Args: x (torch.Tensor): input tensor to normalize Returns: torch.Tensor: The normalized and scaled tensor having the same shape as ``x``. )r weightr)Frms_normfloatr rrtodtype)rrrrrforwards zRMSNorm.forward)r) __name__ __module__ __qualname____doc__intrr r Tensorr __classcell__rrrrr s r)r torch.nn.functionalr functionalrModulerrrrrs