o wie@sddlZddlmZddlZddlmZmZddlmZee Z Gdddej Z Gdd d ej Z Gd d d ej ZGd d d ej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddeZideded d!d"fd#ed$e d%ed&d'ifd(e d)ed*ed+ejd,ed-ed.ed/ejd0ed1ejd2ejd3ejejejejd4ZeeZ d5d6Z!e!d%Z"e!d$Z#e!dZ$e!d#Z%e!d.Z&e!d3Z'e!d-Z(e!d,Z)dS)7N) OrderedDict)Tensornn)loggingc@"eZdZdZdedefddZdS)PytorchGELUTanha& A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. inputreturncCstjj|ddS)Ntanh) approximate)r functionalgeluselfr rU/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/activations.pyforward$szPytorchGELUTanh.forwardN__name__ __module__ __qualname____doc__rrrrrrrsrc@r)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 r r c Cs6d|dttdtj|dt|dS)N??@Hm?g@)torchr mathsqrtpipowrrrrr.s6zNewGELUActivation.forwardNrrrrrr(srcsLeZdZdZd deffdd Zdedefdd Zdedefd d ZZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 Fuse_gelu_pythoncs(t|r |j|_dStjj|_dSN)super__init__ _gelu_pythonactrr r)rr$ __class__rrr':s  zGELUActivation.__init__r r cCs |ddt|tdS)Nrrr)rerfrr rrrrr(As zGELUActivation._gelu_pythoncC ||Sr%r)rrrrrD zGELUActivation.forward)F) rrrrboolr'rr(r __classcell__rrr*rr#2s r#c@r)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs r r cCs*d|dt|ddd||S)Nrrg3E?r)rr rrrrrMs*zFastGELUActivation.forwardNrrrrrr2Hr2c@r)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs r r cCs|td|S)NgZd;?)rsigmoidrrrrrVszQuickGELUActivation.forwardNrrrrrr4Qr3r4cs<eZdZdZdedeffdd Zdedefdd ZZS) ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxcs8||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorr&r'r7r8)rr7r8r*rrr'gs   zClippedGELUActivation.__init__xr cCstt||j|jSr%)rcliprr7r8)rr;rrrrozClippedGELUActivation.forward) rrrrfloatr'rrr1rrr*rr6Zs r6cs2eZdZdZfddZdedefddZZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cs ttdtj|_dS)N)r&r'rr r!precomputed_constantrr*rrr'{s zAccurateGELUActivation.__init__r r c Cs,d|dt|j|dt|dS)Nrrr)rr rAr"rrrrrs,zAccurateGELUActivation.forward)rrrrr'rrr1rrr*rr?ss r?csDeZdZdZfddZdedefddZdedefdd ZZS) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish csttjj|_dSr%)r&r'rr mishr)rBr*rrr's zMishActivation.__init__r r cCs|ttj|Sr%)rr rr softplusrrrr _mish_pythonr=zMishActivation._mish_pythoncCr-r%r.rrrrrr/zMishActivation.forward) rrrrr'rrGrr1rrr*rrDs  rDc@r)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. r r cCs|Sr%rrrrrrszLinearActivation.forwardNrrrrrrHr3rHc@seZdZdZdddZdS)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹۞? ^/ ?cCs*|||td}ddt|S)Nrrr)divrr rr,)rr musigmarrrrszLaplaceActivation.forwardN)rJrKrrrrrrrrrrIsrIc@seZdZdZddZdS)ReLUSquaredActivationz` Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668v2 cCstj|}t|}|Sr%)rr relursquare)rr relu_appliedsquaredrrrrs  zReLUSquaredActivation.forwardNrOrrrrrPs rPcseZdZfddZZS)ClassInstantiercs4t|}t|tr |n|if\}}|di|S)Nr)r& __getitem__ isinstancetuple)rkeycontentclskwargsr*rrrVs zClassInstantier.__getitem__)rrrrVr1rrr*rrUsrUrgelu_10i )r7r8 gelu_fastgelu_new gelu_pythonr$Tgelu_pytorch_tanh gelu_accuratelaplace leaky_relulinearrE quick_gelurQrelu2relu6r5silu)swishr prelucCs,|tvrt|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_stringrrrget_activationsrr)*r collectionsrrrrutilsr get_loggerrloggerModulerrr#r2r4r6r?rDrHrIrPrU LeakyReLUReLUReLU6SigmoidSiLUTanhPReLUACT2CLSrmrrrar`rr_rgrjrE linear_actrrrrs