o iC@sddlmZddlmZmZmZmZmZddlZddl m m Z ddlm Z ddl mZmZddlmZmZmZddlmZmZejjrnejd d d ejd ejd eejdejdeejdeejejejff ddZeGdddeZeGdddeZGdddejjZ GdddejjZ!GdddZ"Gddde"Z#Gdd d e"Z$d ejd ejd eejdejdeejd!ejd"eejdejfd#d$Z%eGd%d&d&Z&d'e&de'fd(d)Z(d'e&de'fd*d+Z)d'e&de'fd,d-Z*e#e d.d/e*gd0Z+e#e!d.d1e(e)gd0Z,e$ed2d3d4e(e)gd0Z-e$e%d.d5gd0Z.dee/e/ffd6d7Z0dd8d ejd ejd eejdejdeejd!ejd"eejd'ee"dejfd9d:Z1d ejd;ejdZ2Gd?d@d@e j3Z4dS)A) dataclass)DictOptionalSequenceTupleUnionN)nn) custom_bwd custom_fwd) BaseOperatorget_xformers_operatorregister_operator) stack_or_noneunbindz%xformers::dual_gemm_silu_identity_mulcudaxw1b1w2b2returncCsL||j}|dur ||7}||j}|dur||7}t||}|||fSN)TFsilu)rrrrrx1x2x4rJ/home/ubuntu/.local/lib/python3.10/site-packages/xformers/ops/swiglu_op.py dual_gemm_silu_identity_mul_cudas   r!c@seZdZedZdZdZdS)DualGemmSiluOpdual_gemm_silu_identity_mulswigludual_gemm_siluN__name__ __module__ __qualname__r OPERATOROPERATOR_CATEGORYNAMErrrr r"'r"c@seZdZedZdZdZdS)GemmFusedSumOpgemm_fused_operand_sumr$Nr&rrrr r..r-r.c@s8eZdZdZdZdZddZeddZedd Z d S) _SwiGLUDecomposedFuncap This is just an example implementation with all operations explicited. This implementation is worse than pytorch, because pytorch is able to fuse some operations (eg the linear forward ...) that are decomposed here. The time measurements were made on the ViT-Giant setting: - A100/f16 - input: [4440, 1536] - hidden: [4440, 4096] decomposedFcCs@ddt| }||d|d||jS)Nr )torchexpfloattodtype)dyrsigmrrr _silu_backwardEs(z$_SwiGLUDecomposedFunc._silu_backwardc Csr||dd|} ||dd|} t| } | | } | |dd|} ||||||||| | | | | | SN) transposerrsave_for_backward)clsctxrrrrrw3b3rrx3rx5rrr forwardKs  z_SwiGLUDecomposedFunc.forwardc Cs|j}|jr|}dd|jD}|\ }}}}}} } } } } }}|| }|dd|}|d}|| }|| }||| }||}|dd|}|d}|||7}|dd|}|d}|||||||fS)NcSsg|]}|qSr)r4).0trrr \sz2_SwiGLUDecomposedFunc.backward..r;r<r) saved_tensors FORCE_BW_F32r4r=sumr9)r?r@dx5rIrrrrrrArBrrrCrrDdx4dw3db3dx3dx2dx1dxdw2db2dw1db1rrr backwardWs$     z_SwiGLUDecomposedFunc.backwardN) r'r(r)__doc__r,rJr9 classmethodrErXrrrr r05s   r0c @sneZdZdZeeddddZedej dej de d e ej e ej ffd d Z eeddd d ZdS)_SwiGLUFusedFunczfused.pyr) device_typec CsVt|||||\} } } t| ||} |||||| | |du|du|dug|_| Sr)r"r*rlinearr>bias) r?r@rrrrrrArBrrrrDrrr rEps z_SwiGLUFusedFunc.forwardr7rr^rcCs8|s |dd|dfSt|dd|\}}||fSr:)r=r.r*)r7rr^dwdbrrr _linear_bwzsz_SwiGLUFusedFunc._linear_bwcCs|j\}}}}}}t||gdd} ||} tjj||| \} } | d\} }~~~ |j|| |jdd\}}~ ~| dur| sCJ| sIJ| |j dd|j dg} | | j dd| j dg| }| | j dd| j dg dd|}|j| | j dd| j dg||jdd\}}| dg|j d\}}|jdr| d| j dg}tj|dd\}}n2d}}n-||}tj || || jdd|d|j|||jdd\}}|j| ||jdd\}}|||||||fS) Nrdimr r^r;r<)betaalphaout)rIrr2opsxformers silu_bw_fusedrrar^ is_contiguousviewshaper=addmmr5r6)r?r@rLrrrrArrw1w2rMdx1dx2rrRrQrNrOrSdw1dw2db1db2rVrTrWrUrrr rXs:  "*&   z_SwiGLUFusedFunc.backwardN)r'r(r)r,rZr rE staticmethodr2Tensorboolrrrar rXrrrr r[ms$  r[c@s\eZdZdZdedefddZdddefd d Zd ee j de j fd d Z defddZ dS)SwiGLUOpzABase class for any swiglu operator in :attr:`xformers.ops.swiglu`packed_weightsnamecCs||_||_||_||_dSr)r,PACKED_WEIGHTSop constraints)selfr{rxryr|rrr __init__s zSwiGLUOp.__init__r{SwiGLUOpDispatchrcs(|jrjsdStfdd|jDS)NFc3s|]}|VqdSrr)rFcr{rr sz$SwiGLUOp.supports..)rzrxallr|r}r{rrr supportss zSwiGLUOp.supportsargscGstr)NotImplementedError)r}rrrr __call__zSwiGLUOp.__call__cCs d|jS)Nz SwiGLUOp:)r,r}rrr __str__ zSwiGLUOp.__str__N) r'r(r)rYrvstrr~rrr2rurrrrrr rws rwcs.eZdZdddeffdd ZddZZS)_ForwardToPythonAutogradFuncr{rrcs t|Sr)superrr __class__rr rrz%_ForwardToPythonAutogradFunc.supportscOs|jj|i|Sr)r{applyr}rkwargsrrr rsz%_ForwardToPythonAutogradFunc.__call__)r'r(r)rvrr __classcell__rrrr rsrc@seZdZddZddZdS)_ForwardToFunccOs|j|i|Srrrrrr rz_ForwardToFunc.__call__cCs|jjdkrdSdS)Nno_such_operatorz not built available)r{r'rrrr infos z_ForwardToFunc.infoN)r'r(r)rrrrrr rs rrArBc Cs8t|||}t|||}t||} t| ||Sr)rr]r) rrrrrrArBrrhiddenrrr _eager_functional_swiglus rc@seZdZUdZeejefed<ej ed<e ej ed<e ed<e ed<e de fdd Zed ejd ejd e ejd ejde ejdejde ejddfddZdS)rz\Dispatcher to automatically select the best operator in :attr:`xformers.ops.swiglu` devicer6dtype_autocast_gpurx bias_enabledrcCs(ttg}|D] }||r|SqtS)zsComputes the best operator Returns: SwiGLUOp: The best operator for the configuration )SwiGLUPackedFusedOp SwiGLUFusedOpr SwiGLUEagerOp)r} prioritiesr{rrr r{s zSwiGLUOpDispatch.oprrrrrrArBcCsLt|j|jt||fdddutrtn|j|duo#|duo#|dudS)Nrrb)rr6rxrr)rrr6rr2is_autocast_enabledget_autocast_gpu_dtype)rrrrrrArBrrr from_argumentss  zSwiGLUOpDispatch.from_argumentsN)r'r(r)rYrr2rr__annotations__r6rrvpropertyrwr{rtrurrrrr rs6  rr{cCs8t|jtr |jn|jj}|dkotj|jddkS)Nrr) isinstancerrtyper2rget_device_capability)r{r\rrr _only_sm80srcCs*tjtjg}|j|vp|jduo|j|vSr)r2halfbfloat16r6r)r{ HALF_DTYPESrrr _only_half_or_autocasts  rcCs|jSr)rrrrr _bias_enabledrrFr1)r|fusedswiglu_packedwTz fused.p.cppeagercCsddtfDS)NcSsi|]}|j|qSr)r,r)rFr{rrr 7sz_info..)rrrrr _info6rrrc Cs|jdd}|d|jdg}|jdks|j|jkr(td|jd|j|durC|jdks;|jd|jdkrCtd|j|dur^|jdksV|jd|jdkr^td |j|jdksm|jd|jdkrutd |j|dur|jdks|jd|jdkrtd |jd |j|durt|||||||j}|js||||||||g|dSt||fdd } |dur|durt||fdd } | durt dn d} |dur|dusJ| durt d||| | ||g|dS)a< Computes a SwiGLU block given the weights/bias of the 3 linear layers. - It is recommended to keep ``op=None`` so the best implementation available for the inputs will be used. :Equivalent pytorch code: .. code-block:: python x1 = F.linear(x, w1, b1) x2 = F.linear(x, w2, b2) hidden = F.silu(x1) * x2 return F.linear(hidden, w3, b3) :Packing weights: To allow faster implementations, it's recommended to have w1/w2 come from the same storage, as in: .. code-block:: python w1, w2 = xformers.ops.unbind(w12, 0) :Supported hardware: This operator is only optimized on A100+ on ``torch.half`` or ``torch.bfloat16`` (autocast is supported), and will fallback to a functional pytorch implementation otherwise. Nr<rdzInvalid shapes for w1: z / w2: r rzInvalid shapes for b1: zInvalid shapes for b2: zInvalid shape for w3: Invalid shapes for w3:  / b3: rbz!b1/b2 needs to be properly packedz!w1/w2 needs to be properly packed) rnreshapendim ValueErrorrrr{rzrr) rrrrrrArBr{ batch_shaperpb1b2rrr r$:s<*"r$rprcCs|jdd}|d|jdg}|dur0|jdks$|jd|jdkr0td|jd|j|js7Jd||||||g|dS)a Computes a SwiGLU block given the weights/bias of the 3 linear layers. :Equivalent pytorch code: .. code-block:: python x1 = F.linear(x, w1, b1) x2 = F.linear(x, w2, b2) hidden = F.silu(x1) * x2 return F.linear(hidden, w3, b3) :Supported hardware: This operator is only optimized on A100+ on ``torch.half`` or ``torch.bfloat16`` (autocast is supported), and will fallback to a functional pytorch implementation otherwise. Nr<r rrrzNot implemented PACKED_WEIGHTS)rnrrrrz)rrprrArBr{rrrr swiglu_packedsrcseZdZdZ  ddddededeeded ed df fd d Zd ej d ej fddZ d e ej eej ej eej ej eej ffddZ d e ej eej ej eej ffddZ ZS)SwiGLUz A Module that encapsulates the call to :attr:`xformers.ops.swiglu`, and holds the weights for the 3 linear layers NT) _pack_weights in_featureshidden_features out_featuresr^rrcst|p|}|p |}||rtj|d||d|_nd|_tj|||d|_tj|||d|_tj|||d|_||_||_ ||_ d|_ dS)aqCreate a SwiGLU module Args: in_features (int): Number of features of the input hidden_features (int): Number of hidden features out_features (Optional[int], optional): Number of features of the input. Defaults to None. bias (bool, optional): Whether linear layers also include a bias. Defaults to True. rdreN) rr~rLinearw12rrrArrrr{)r}rrrr^rrrr r~s  zSwiGLU.__init__rcCs\|jdur |jdur |jjsJdt|g|Rd|jiSt|g|Rd|jiS)zComputes :attr:`swiglu` with the module's weights Args: x (torch.Tensor): A Tensor of shape ``[..., in_features]`` Returns: torch.Tensor: A Tensor of shape ``[..., out_features]`` Nz5_pack_weights and self.op.PACKED_WEIGHTS should matchr{)rr{rzr_packed_ordered_paramsr$_ordered_params)r}rrrr rEs zSwiGLU.forwardcCs|jdur?|jj}|jj}t|d|jdd|jdgdd\}}|dur:t|d|jddgdd\}}nd\}}n|jj|jj}}|jj|jj}}|||||jj|jjfS)z:Used for testing - returns ordered arguments for operatorsNrdrr rb)NN) rweightr^rrmrnrrrA)r}rprrrrrrrr rs&  & zSwiGLU._ordered_paramscCs||jdus Jd |jj}|jj}|d|jdd|jdg}d}|dur4|d|jddg}|||jj|jjfS)Nz0Packed weights are only available when using w12rdrr )rrr^rmrnrA)r}rp b1b2_paramrrrr rs zSwiGLU._packed_ordered_params)NT)r'r(r)rYintrrvr~r2rurErrrrrrrr rsP # %r)5 dataclassesrtypingrrrrrr2torch.nn.functionalr functionalr torch.ampr r commonr r rrrversionhiplibraryregister_kernelrur!r"r.autogradFunctionr0r[rwrrrrrvrrr_SwiGLUDecomposedOprrrrrr$rModulerrrrr s    8>  0     M (