o i @sddlmZddlZddlmZddlmZddeejdefd d Z dd ejd ejdeejdefd dZ Gdddejj Z dS))OptionalN)nn)_is_triton_availableư>weightepscCsFtsJddlm}tr|js|dur|jrtd||||S)a) RMS Normalization along the last dimension. This is similar to torch.nn.functional.normalize but with eps being added instead of max. Expects x contiguous of shape (..., dim), and returns normalized data of the same shape. For each dim-length vector x, the result has x / sqrt( x*x.sum() + eps) If weights are included, they are a contiguous parameter of length dim which multiplies the result. This functionality is experimental. Its API might be changed without warnings. Use it at your own risk. )_rms_norm_forwardNGradients not supported.)r_triton.rmsnorm_kernelsr torchis_grad_enabled requires_grad ValueError)xrrr rH/home/ubuntu/.local/lib/python3.10/site-packages/xformers/ops/rmsnorm.pyrms_norm s   rrycCsNtr|js|js|dur|jrtdtsJddlm}|||||S)a= An addition fused with rms_norm. z = rms_norm_add(x, y, weight, eps) is equivalent to x += y z = rms_norm(x, weight, eps) where x, y and z are all contiguous. This functionality is experimental. Its API might be changed without warnings. Use it at your own risk. Nr r )_rms_norm_add_forward)r rrrrr r)rrrrrrrr rms_norm_add*s  rcsVeZdZdZddededeffdd Zd ej fd d Z d ej d ej fd dZ Z S)RMSNorma( RMS Normalization layer along the last dimension. This is similar to torch.nn.functional.normalize but with eps being added instead of max. Expects contiguous input of shape (..., dim), and returns normalized data of the same shape. For each dim-length vector x, the result has x / sqrt( x*x.sum() + eps) If weights are included, they are a parameter of length dim which multiplies the result. This functionality is experimental. Its API might be changed without warnings. Use it at your own risk. Trdiminclude_weightrcs4t||_|rtt||_dSd|_dSN)super__init__rr Parameterr onesr)selfrrr __class__rrr[s  zRMSNorm.__init__rcCst||j|jSr)rrr)r rrrrforwardcszRMSNorm.forwardrcCst|||j|jS)z An addition fused with forward. z = layer.increment_and_forward_(x, y) is equivalent to x += y z = layer(x) )rrr)r rrrrrincrement_and_forward_fs zRMSNorm.increment_and_forward_)Tr) __name__ __module__ __qualname____doc__intboolfloatrr Tensorr#r$ __classcell__rrr!rrHs r)r) typingrr rrr,r+rrModulerrrrrs