o pi[@sddlZddlmmZddlmZddlmZddlmZer&ddl Z e e e e e dZdedejfd d ZGd d d ejZGd ddejZ GdddejZGdddejZGdddejZdS)N)nn) deprecate)is_torch_npu_available)swishsilumishgelureluact_fnreturncCs&|}|tvr t|Std|)zHelper function to get activation function from string. Args: act_fn (str): Name of activation function. Returns: nn.Module: Activation function. z!Unsupported activation function: )lowerACTIVATION_FUNCTIONS ValueError)r rZ/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/activations.pyget_activation$s rcs6eZdZdZfddZdejdejfddZZS)FP32SiLUzH SiLU activation function with input upcasted to torch.float32. cstdSN)super__init__)self __class__rrr:szFP32SiLU.__init__inputsr cCstj|dd|jS)NF)inplace)Frfloattodtype)rrrrrforward=szFP32SiLU.forward) __name__ __module__ __qualname____doc__rtorchTensorr __classcell__rrrrr5s rc sReZdZdZddedededeffdd Zd ej d ej fd d Z ddZ Z S)GELUa GELU activation function with tanh approximation support with `approximate="tanh"`. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. approximate (`str`, *optional*, defaults to `"none"`): If `"tanh"`, use tanh approximation. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. noneTdim_indim_out approximatebiascs&ttj|||d|_||_dSNr-)rrrLinearprojr,)rr*r+r,r-rrrrLs  z GELU.__init__gater cCs@|jjdkrtj||jdStj|jtjd|jdj|jdS)Nmps)r,r) devicetyperr r,rr%float32rrr2rrrr Qs $z GELU.gelucCs||}||}|Sr)r1r )r hidden_statesrrrr Ws  z GELU.forward)r)T) r!r"r#r$intstrboolrr%r&r r r'rrrrr(As  r(csNeZdZdZddededeffdd Zdejd ejfd d Z d d Z Z S)GEGLUaF A [variant](https://arxiv.org/abs/2002.05202) of the gated linear unit activation function. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. Tr*r+r-cs$ttj||d|d|_dSNrr/rrrr0r1rr*r+r-rrrrgs zGEGLU.__init__r2r cCs4|jjdkr t|St|jtjdj|jdS)Nr3r4)r5r6rr rr%r7rr8rrrr ks  z GEGLU.gelucOspt|dks|dddurd}tdd|||}tr(tj|ddddS|jddd \}}|||S) NrscalezThe `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.z1.0.0)dimr,rrD) lengetrr1r torch_npu npu_gegluchunkr )rr9argskwargsdeprecation_messager2rrrr qs  z GEGLU.forwardT) r!r"r#r$r:r<rr%r&r r r'rrrrr=]s  r=cs8eZdZdZd dededeffdd Zdd ZZS) SwiGLUa A [variant](https://arxiv.org/abs/2002.05202) of the gated linear unit activation function. It's similar to `GEGLU` but uses SiLU / Swish instead of GeLU. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. Tr*r+r-cs.ttj||d|d|_t|_dSr>)rrrr0r1SiLU activationr@rrrrs zSwiGLU.__init__cCs*||}|jddd\}}|||S)NrrBrE)r1rJrQ)rr9r2rrrr s zSwiGLU.forwardrN) r!r"r#r$r:r<rr r'rrrrrO~s rOcsFeZdZdZd dededeffdd Zdejd ejfd d Z Z S) ApproximateGELUa The approximate form of the Gaussian Error Linear Unit (GELU). For more details, see section 2 of this [paper](https://arxiv.org/abs/1606.08415). Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. Tr*r+r-cs ttj|||d|_dSr.r?r@rrrrs zApproximateGELU.__init__xr cCs||}|td|S)NgZd;?)r1r%sigmoid)rrSrrrr s zApproximateGELU.forwardrN) r!r"r#r$r:r<rr%r&r r'rrrrrRs rR)r%torch.nn.functionalr functionalrutilsrutils.import_utilsrrHrPMishr(ReLUrr;Modulerrr=rOrRrrrrs&     !