o  ¢iGã@s<ddlZddlZddlmZddlmZGdd„deƒZdS)éN)ÚTensor)ÚModulec@s"eZdZdZdedefdd„ZdS)ÚGeluNewzAGELU approximation, called `gelu_new` in many transformer models.ÚinputÚreturnc Cs6d|dt t dtj¡|dt |d¡¡S)zE Shapes: input - (batch, seq_len, width) gà?gð?g@g÷Hmâä¦?g@)ÚtorchÚtanhÚmathÚsqrtÚpiÚpow)Úselfr©rú[/home/ubuntu/.local/lib/python3.10/site-packages/curated_transformers/models/activations.pyÚforward sÿÿÿÿýÿzGeluNew.forwardN)Ú__name__Ú __module__Ú __qualname__Ú__doc__rrrrrrrsr)r rrÚtorch.nnrrrrrrÚs