o ib @s,UddlZddlmZddlmZmZddlmZddlm Z zddl m Z m Z m Z mZmZmZmZd Zd e d d d id e ddd ie ddd e je ddd idd e ddd ie je ddd e je ddd ie je ddd idd e je jBe ddddiid e je jBe ddddiid e je jBe ddddiid e je jBe ddddiid e je jBe dd ddiid e je jBe dd!ddiid" Zeeeee efe ffed#<eeWneyd$Zd%d&ZGd'd(d(Z d)d*Zd+d,ZYnwd-eed.efd/d0Zd-ed.dfd1d2Zgd3ZdS)4N)partial)OptionalUnion)lazy_import_flash_attention)flash_attention_forward)DeviceLayerRepositoryMode get_kernelregister_kernel_mappingreplace_kernel_forward_from_hubuse_kernel_forward_from_hubTcudaz!kernels-community/deformable-detrMultiScaleDeformableAttention)repo_id layer_namezkernels-community/moe Llama4TextMoezkernels-community/liger_kernels LigerRMSNorm)rrocmzmedmekk/triton-llama-mlpTritonLlamaMLPzkernels-community/megablocksMegaBlocksMoeMLPzahadnagy/megablockszkernels-community/activationFastGELUz>=0.0.4,<0.1.0)rrversion QuickGELUNewGELUSiluz>=0.1.0GeluGeluTanh) rrRMSNormMLPrrrrSiLUGeLUr_KERNEL_MAPPINGFcOs dd}|S)NcSs|S)N)clsr%r%Y/home/ubuntu/.local/lib/python3.10/site-packages/transformers/integrations/hub_kernels.py decoratorsz.use_kernel_forward_from_hub..decoratorr%)argskwargsr(r%r%r'rsrc@seZdZddZdS)r cOtd)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeError)selfr)r*r%r%r'__init__zLayerRepository.__init__N)__name__ __module__ __qualname__r/r%r%r%r'r s r cOr+)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.r,r)r*r%r%r'rsrcOr+)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.r,r4r%r%r'r r0r attn_implementationreturncCs|duo td|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)r5r%r%r' is_kernelsr9c CsRt|sdSts tdddlm}ddlm}d}|}d|vr,|d\}}||}d|vr<|d\}}| }n|}d}| }| d\}}}| }|rV| nd}zt ||d } Wnt yw} z t d |d | d d} ~ wwt| d r|durt}t|| d} t| ddn |durt| |} ||| |||ddS)zALoad and register the kernel associated to `attn_implementation`.Nz`kernels` is either not installed or uses an incompatible version. Please install the latest version with `pip install -U kernels`.r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|:@)revisionz-An error occurred while trying to load from 'z': .flash_attn_varlen_func)implementationT) force_importflash_attention_2)r9_kernels_available ImportError masking_utilsr:modeling_utilsr;splitgetstrip partitionr Exception ValueErrorhasattrrrrgetattrregister) r5r:r;attention_wrapperactual_attn_namer kernel_name_revkernelekernel_functionr%r%r'load_and_register_kernelsJ        rZ)r rr r)r7 functoolsrtypingrrmodeling_flash_attention_utilsrflash_attentionrkernelsr r r r r rrrE INFERENCETRAINING TORCH_COMPILEr$dictstr__annotations__rFboolr9rZ__all__r%r%r%r's   $        & f   4