o Gio@sddlZddlmmZddlmZddlmZddlmZm Z er(ddl Z ej ej ej ej ejdZdedejfd d ZGd d d ejZGd ddejZ GdddejZGdddejZGdddejZGdddejZdS)N)nn) deprecate)is_torch_npu_availableis_torch_version)swishsilumishgelureluact_fnreturncCs6|}|tvr t|Std|dtt)zHelper function to get activation function from string. Args: act_fn (str): Name of activation function. Returns: nn.Module: Activation function. zactivation function z not found in ACT2FN mapping )lowerACT2CLS ValueErrorlistkeys)r rP/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/activations.pyget_activation$s  rcs6eZdZdZfddZdejdejfddZZS)FP32SiLUzH SiLU activation function with input upcasted to torch.float32. cstdSN)super__init__)self __class__rrr:szFP32SiLU.__init__inputsr cCstj|dd|jS)NF)inplace)Frfloattodtype)rrrrrforward=szFP32SiLU.forward) __name__ __module__ __qualname____doc__rtorchTensorr# __classcell__rrrrr5s rc sReZdZdZddedededeffdd Zd ej d ej fd d Z ddZ Z S)GELUa GELU activation function with tanh approximation support with `approximate="tanh"`. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. approximate (`str`, *optional*, defaults to `"none"`): If `"tanh"`, use tanh approximation. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. noneTdim_indim_out approximatebiascs&ttj|||d|_||_dSNr0)rrrLinearprojr/)rr-r.r/r0rrrrLs  z GELU.__init__gater cCsJ|jjdkrtddrtj|jtjd|jdj|j dStj||jdS)Nmps<2.0.0r")r/) devicetyperrr r!r(float32r/r"rr5rrrr Qs$z GELU.gelucCs||}||}|Sr)r4r r hidden_statesrrrr#Ws  z GELU.forward)r,T) r$r%r&r'intstrboolrr(r)r r#r*rrrrr+As  r+csNeZdZdZddededeffdd Zdejd ejfd d Z d d Z Z S)GEGLUaN A [variant](https://huggingface.co/papers/2002.05202) of the gated linear unit activation function. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. Tr-r.r0cs$ttj||d|d|_dSNrr2rrrr3r4rr-r.r0rrrrgs zGEGLU.__init__r5r cCs>|jjdkrtddrt|jtjdj|jdSt|S)Nr6r7r8r9) r:r;rrr r!r(r<r"r=rrrr ks z GEGLU.gelucOspt|dks|dddurd}tdd|||}tr(tj|ddddS|jddd \}}|||S) NrscalezThe `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.z1.0.0)dimr/rrJ) lengetrr4r torch_npu npu_gegluchunkr )rr?argskwargsdeprecation_messager5rrrr#qs  z GEGLU.forwardT) r$r%r&r'r@rBrr(r)r r#r*rrrrrC]s  rCcs8eZdZdZd dededeffdd Zdd ZZS) SwiGLUa A [variant](https://huggingface.co/papers/2002.05202) of the gated linear unit activation function. It's similar to `GEGLU` but uses SiLU / Swish instead of GeLU. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. Tr-r.r0cs.ttj||d|d|_t|_dSrD)rrrr3r4SiLU activationrFrrrrs zSwiGLU.__init__cCs*||}|jddd\}}|||S)NrrHrK)r4rPrW)rr?r5rrrr#s zSwiGLU.forwardrT) r$r%r&r'r@rBrr#r*rrrrrU~s rUcsFeZdZdZd dededeffdd Zdejd ejfd d Z Z S) ApproximateGELUa The approximate form of the Gaussian Error Linear Unit (GELU). For more details, see section 2 of this [paper](https://huggingface.co/papers/1606.08415). Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. Tr-r.r0cs ttj|||d|_dSr1rErFrrrrs zApproximateGELU.__init__xr cCs||}|td|S)NgZd;?)r4r(sigmoid)rrYrrrr#s zApproximateGELU.forwardrT) r$r%r&r'r@rBrr(r)r#r*rrrrrXs rXc s8eZdZd dedededeffdd Zd d ZZS) LinearActivationTrr-r.r0rWcs*ttj|||d|_t||_dSr1)rrrr3r4rrW)rr-r.r0rWrrrrs zLinearActivation.__init__cCs||}||Sr)r4rWr>rrrr#s  zLinearActivation.forward)Tr) r$r%r&r@rBrArr#r*rrrrr[s r[)r(torch.nn.functionalr functionalrutilsrutils.import_utilsrrrNrVMishr+ReLUrrAModulerrrCrUrXr[rrrrs(    !