o if @sdZddlmZmZmZddlZ  ddedeedeedeejj effdd Z Gd d d ejj Z Gd d d ejj Z Gdddejj Z dS)z*Normalization modules for X-former blocks.)DictOptionalTupleNnormalization_typeepspartialreturncCsxtd|dur|ndiftjjd|dur|ndift|dur|nd|dur&|nddftd|dur2|ndifd}||S) a\Get normalization module and arguments given parameters. Args: normalization_type: Normalization module type. eps: Value added to the denominator. partial: Value defining the part of the input used for RMS stats (RMSNorm). Return: : Normalization module class : Normalization module arguments rN?g-q=h㈵>g)rr) basic_norm layer_normrms_norm scale_norm) BasicNormtorchnn LayerNormRMSNorm ScaleNorm)rrrnormrh/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/asr_transducer/encoder/modules/normalization.pyget_normalizationsrcsHeZdZdZ d dededdffdd Zd ejdejfd d Z Z S) rzBasicNorm module definition. Reference: https://github.com/k2-fsa/icefall/pull/288 Args: normalized_shape: Expected size. eps: Value added to the denominator for numerical stability. r normalized_shaperrNcs*ttjt||_dS)zConstruct a BasicNorm object.N) super__init__rr Parametertensorlogdetachrselfrr __class__rrr:s  zBasicNorm.__init__xcCs,tj|dddd|jd}||S)zCompute basic normalization. Args: x: Input sequences. (B, T, D_hidden) Returns: : Output sequences. (B, T, D_hidden) Tdimkeepdim)rmeanpowrexp)r!r$scalesrrrforwardDs$ zBasicNorm.forward)r __name__ __module__ __qualname____doc__intfloatrrTensorr/ __classcell__rrr"rr/s  rc sNeZdZdZ  ddedededdffd d Zd ejdejfd d Z Z S)raRMSNorm module definition. Reference: https://arxiv.org/pdf/1910.07467.pdf Args: normalized_shape: Expected size. eps: Value added to the denominator for numerical stability. partial: Value defining the part of the input used for RMS stats. r rrrrNcsVt||_d|krdkrnndnd|_||_||_tjt ||_ dS)zConstruct a RMSNorm object.rTFN) rrrrprrrronesscale)r!rrrr"rrr_s "zRMSNorm.__init__r$cCs|jr%t|j|j}tj|||j|gdd\}}|jdddd}|}n |jdddd}|j}||d}|j|||j}|S)zCompute RMS normalization. Args: x: Input sequences. (B, T, D_hidden) Returns: x: Output sequences. (B, T, D_hidden) r&)r(r%Tr'r*) rr5rr;rsplitrr=r)r!r$ partial_size partial_x_norm_xd_xrms_xrrrr/ps   zRMSNorm.forward)r r9r0rrr"rrSsrcsFeZdZdZd dededdffdd Zd ejdejfd d Z Z S) rzScaleNorm module definition. Reference: https://arxiv.org/pdf/1910.05895.pdf Args: normalized_shape: Expected size. eps: Value added to the denominator for numerical stability. r rrrNcs,t||_tjt|d|_dS)zConstruct a ScaleNorm object.g?N)rrrrrrrr=r r"rrrs zScaleNorm.__init__r$cCs(|jtj|dddj|jd}||S)zCompute scale normalization. Args: x: Input sequences. (B, T, D_hidden) Returns: : Output sequences. (B, T, D_hidden) r&Tr')min)r=rrclampr)r!r$rrrrr/s zScaleNorm.forward)r r0rrr"rrs r)NN)r4typingrrrrstrr6rModulerrrrrrrrs" '$9