o iJ@sdZddlZddlmZ         d d ed ed ed ededededededededejj fddZ Gdddejj Z Gdddejj Z Gdddejj Z Gdddejj ZdS)!z$Activation functions for Transducer.N)parse皙ɿ?{Gz?activation_typeftswish_thresholdftswish_mean_shifthardtanh_min_valhardtanh_max_valleakyrelu_neg_slope smish_alpha smish_beta softplus_betasoftplus_threshold swish_betareturnc  Csttj} t||dftjj||dftjjd|ift|| | tdkdftjjiftjj ift ||dft | | tdkdftjj iftjj ifd } | |\} }| d i|S) aReturn activation function. Args: activation_type: Activation function type. ftswish_threshold: Threshold value for FTSwish activation formulation. ftswish_mean_shift: Mean shifting value for FTSwish activation formulation. hardtanh_min_val: Minimum value of the linear region range for HardTanh. hardtanh_max_val: Maximum value of the linear region range for HardTanh. leakyrelu_neg_slope: Negative slope value for LeakyReLU activation formulation. smish_alpha: Alpha value for Smish activation fomulation. smish_beta: Beta value for Smish activation formulation. softplus_beta: Beta value for softplus activation formulation in Mish. softplus_threshold: Values above this revert to a linear function in Mish. swish_beta: Beta value for Swish variant formulation. Returns: : Activation function. ) threshold mean_shift)min_valmax_valnegative_slopez1.9)rr use_builtin)alphabetaz1.8)rr) ftswishhardtanh leaky_relumishreluselusmishswishtanhidentityN)Vtorch __version__FTSwishnnHardtanh LeakyReLUMishReLUSELUSmishSwishTanhIdentity)r r r r r rrrrrr torch_version activationsact_funcact_argsr'r'U/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/asr_transducer/activation.pyget_activations2        r;csFeZdZdZd dededdffdd Zd ejdejfd d ZZ S)r+aFlatten-T Swish activation definition. FTSwish(x) = x * sigmoid(x) + threshold where FTSwish(x) < 0 = threshold Reference: https://arxiv.org/abs/1812.06247 Args: threshold: Threshold value for FTSwish activation formulation. (threshold < 0) mean_shift: Mean shifting value for FTSwish activation formulation. (applied only if != 0, disabled by default) rrrrrNcs*t|dks Jd||_||_dS)Nrz*FTSwish threshold parameter should be < 0.)super__init__rr)selfrr __class__r'r:r=Zs  zFTSwish.__init__xcCsP|t||j}t|dk|tj|jg|jd}|jdkr&||j|S)Forward computation.r)device)r)sigmoidrwheretensorrCrsub_r>rAr'r'r:forwardbs "  zFTSwish.forward)rr __name__ __module__ __qualname____doc__floatr=r)TensorrI __classcell__r'r'r?r:r+Ksr+c sPeZdZdZ   ddedededd ffd d Zd ej dej fd dZ Z S)r/aMish activation definition. Mish(x) = x * tanh(softplus(x)) Reference: https://arxiv.org/abs/1908.08681. Args: softplus_beta: Beta value for softplus activation formulation. (Usually 0 > softplus_beta >= 2) softplus_threshold: Values above this revert to a linear function. (Usually 10 > softplus_threshold >= 20) use_builtin: Whether to use PyTorch activation function if available. rrFrrrrNcsNt|rtj_dStj_tjj||d_ fdd_dS)N)rrcs||SN)r%softplusrAr>r'r:zMish.__init__..) r<r=r)r,r/r r4r%SoftplusrS)r>rrrr?rUr:r=}s  z Mish.__init__rAcC ||SrB)r rHr'r'r:rI z Mish.forward)rrF) rKrLrMrNrOintboolr=r)rPrIrQr'r'r?r:r/msr/csFeZdZdZd dededdffdd Zd ejdejfd d ZZ S) r2aSmish activation definition. Smish(x) = (alpha * x) * tanh(log(1 + sigmoid(beta * x))) where alpha > 0 and beta > 0 Reference: https://www.mdpi.com/2079-9292/11/4/540/htm. Args: alpha: Alpha value for Smish activation fomulation. (Usually, alpha = 1. If alpha <= 0, set value to 1). beta: Beta value for Smish activation formulation. (Usually, beta = 1. If beta <= 0, set value to 1). rrrrNcsLttj_|dkr|nd_|dkr|nd_fdd_dS)Nrc s*j|tdtj|S)Nr^)rr%r)logrDrrTrUr'r:rVs z Smish.__init__..) r<r=r)r,r4r%rrr#)r>rrr?rUr:r=s  zSmish.__init__rAcCrYrZ)r#rHr'r'r:rIr[z Smish.forward)rrrJr'r'r?r:r2s r2csFeZdZdZd dededdffdd Zd ejdejfd d Z Z S)r3aSwish activation definition. Swish(x) = (beta * x) * sigmoid(x) where beta = 1 defines standard Swish activation. References: https://arxiv.org/abs/2108.12943 / https://arxiv.org/abs/1710.05941v1. E-swish variant: https://arxiv.org/abs/1801.07145. Args: beta: Beta parameter for E-Swish. (beta >= 1. If beta < 1, use standard Swish). use_builtin: Whether to use PyTorch function if available. rFrrrNcsLt|_|dkrfdd_dS|rtj_dSdd_dS)Nr^csj|t|SrR)rr)rDrTrUr'r:rVrWz Swish.__init__..cSs|t|SrR)r)rDrTr'r'r:rVs)r<r=rr$r)r,SiLU)r>rrr?rUr:r=s zSwish.__init__rAcCrYrZ)r$rHr'r'r:rIr[z Swish.forward)rF) rKrLrMrNrOr]r=r)rPrIrQr'r'r?r:r3s r3) rrrrrrrrrr)rNr)packaging.versionrr(strrOr\r,Moduler;r+r/r2r3r'r'r'r:sT     D"'!