o پi-@sddlmZddlZddlZddlmZddlmZmZm Z ddl Z ddl Z ddl mZddlmZer:ddlmZeeZGdd d ejZGd d d ejZGd d d ejZGddde ZGdddZGdddeZdS)) annotationsN) lru_cache) TYPE_CHECKINGAny NamedTuple) init_logger)resolve_obj_by_qualname) AttentionImplc@seZdZeZeZeZeZeZ eZ eZ eZ eZ eZeZeZeZddZedddZdS) AttentionBackendEnumcCs |jSN)namelowerselfre/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/runtime/platforms/interface.py__str__)s zAttentionBackendEnum.__str__returnboolcCs |tjtjtjtjtjtjhvSr )r SLIDING_TILE_ATTNVIDEO_SPARSE_ATTNSPARSE_VIDEO_GEN_2_ATTN VMOBA_ATTNSLA_ATTN SAGE_SLA_ATTNrrrr is_sparse,szAttentionBackendEnum.is_sparseNrr)__name__ __module__ __qualname__enumautoFA2FAr TORCH_SDPA SAGE_ATTN SAGE_ATTN_3rrrAITERrr NO_ATTENTIONrpropertyrrrrrr s"r c@sTeZdZeZeZeZeZeZ eZ eZ eZ eZ dS) PlatformEnumN)rrrr r!CUDAROCMTPUCPUMPSNPUMUSAOOT UNSPECIFIEDrrrrr*8s r*c@s$eZdZeZeZeZdS) CpuArchEnumN)rrrr r!X86ARMr3rrrrr4Ds r4c@s2eZdZUded<ded<d ddZd dd Zd S) DeviceCapabilityintmajorminorrstrcCs|jd|jS)N.)r9r:rrrras_version_strNszDeviceCapability.as_version_strcCs.d|jkr dksJJ|jd|jS)z Express device capability as an integer ````. It is assumed that the minor version is always a single digit. r )r:r9rrrrto_intQszDeviceCapability.to_intNrr;)rr8)rrr__annotations__r=r?rrrrr7Js  r7c@s eZdZUded<ded<ded<dZded<d Zded <d Zded <gZd ed<eddd~ddZ eddd~ddZ eddd~ddZ eddd~ddZ eddd~ddZ eeddddZeedddd Zeeddd!d"Zed~d#d$Zed~d%d&Zeddd~d'd(Zeddd~d)d*Zeddd~d+dZ d~d,d-Zeddd~d.d/Zeddd~d0d1Zeddd2d3Zeddd~d4d5Zeeddd~d6d7Zedd9d:ZeddAdBZe CdddFdGZe CdddJdKZedddLdMZ edddNdOZ!eedddddPdQZ"eddddSdTZ#eddddUdVZ$eddYdZZ%ed[d\Z&eddd`daZ'eddcddZ(eddfdgZ)e dddjdkZ*e C l m dddrdsZ+eddtduZ,eddwdxZ-ed~dydzZ.dd|d}Z/dS)Platformr*_enumr; device_name device_typeNztorch.device | Nonedevicer. dispatch_keyinductorsimple_compile_backendz list[str]supported_quantization)maxsizerrcC|Sr )is_cuda_staticrrrris_cudaozPlatform.is_cudacC |jtjkSr )rCr*r0rrrris_npus zPlatform.is_npucCrMr )is_rocm_staticrrrris_rocmwrPzPlatform.is_rocmcCrQr )rCr*r-rrrris_tpu{rSzPlatform.is_tpucCrQr )rCr*r.rrrris_cpurSzPlatform.is_cpucC|sdStjddkS)NFrr>rNtorchcudaget_device_capabilityclsrrr is_blackwellzPlatform.is_blackwellcCs|sdStjdkS)NF) rrYr]rrr is_hopperszPlatform.is_hoppercCrX)NFr rYr]rrris_sm120r`zPlatform.is_sm120cCt|ddtjkSNrC)getattrr*r+r]rrrrNzPlatform.is_cuda_staticcCrerf)rgr*r,r]rrrrTrhzPlatform.is_rocm_staticcCttdo tjS)Nhpu)hasattrrZrj is_availablerrrris_hpuzPlatform.is_hpucCri)Nxpu)rkrZrorlrrrris_xpurnzPlatform.is_xpucCri)Nnpu)rkrZrqrlrrrrrRrncCrQr )rCr*r2rrrris_out_of_trees zPlatform.is_out_of_treecCs|jtjtjtjfvS)z5Stateless version of :func:`torch.cuda.is_available`.)rCr*r+r,r1rrrr is_cuda_alikeszPlatform.is_cuda_alikecCrQr )rCr*r/rrrris_mpsrSzPlatform.is_mpscCs,z ttdo tjWStyYdSw)NmusaF)rkrZrurlModuleNotFoundErrorrrrris_musas  zPlatform.is_musacCrMr )rUrrrris_hiprPzPlatform.is_hipcCdS)NTrr]rrris_amp_supportedzPlatform.is_amp_supported torch.devicecCtr NotImplementedErrorr]rrrget_local_torch_deviceszPlatform.get_local_torch_deviceselected_backendAttentionBackendEnum | None head_sizer8dtype torch.dtypecCry)z,Get the attention backend class of a device.r)r^rrrrrrget_attn_backend_cls_strsz!Platform.get_attn_backend_cls_strr device_idDeviceCapability | NonecCry)z>Stateless version of :func:`torch.cuda.get_device_capability`.Nrr^rrrrr\szPlatform.get_device_capability capabilitytuple[int, int] | intcCs6|j|d}|dur dSt|tr||kS||kS)z Test whether this platform is compatible with a device capability. The ``capability`` argument can either be: - A tuple ``(major, minor)``. - An integer ````. (See :meth:`DeviceCapability.to_int`) )rNF)r\ isinstancetupler?)r^rrcurrent_capabilityrrrhas_device_capabilitys   zPlatform.has_device_capabilitycCr})zGet the name of a device.r~rrrrget_device_namer{zPlatform.get_device_namecCr})z.Get the uuid of a device, e.g. the PCI bus ID.r~rrrrget_device_uuidr{zPlatform.get_device_uuidcCr})z*Get the total memory of a device in bytes.r~rrrrget_device_total_memorysz Platform.get_device_total_memory local_rankcCs`|s|rtd|S|rtd|S|r"td|S|r+tdStdS)Nr[rqrumpscpu)rOrUrZrFrRrwrt)rrrrr get_devices     zPlatform.get_devicecCs8|rdS|r dS|rdS|rdStd)NncclhcclmcclgloozGNo Accelerators(AMD/NV/MTT GPU, AMD MI instinct accelerators) available)rsrRrwrtrrrrr!get_torch_distributed_backend_strsz*Platform.get_torch_distributed_backend_str enforce_eager bool | NonecCr})zF Check if the current platform supports async output. r~)r^rrrris_async_output_supported!z"Platform.is_async_output_supportedcCs tjddS)aA device-specific wrapper of `torch.inference_mode`. This wrapper is recommended because some hardware backends such as TPU do not support `torch.inference_mode`. In such a case, they will fall back to `torch.no_grad` by overriding this method. T)mode)rZinference_moder]rrrr(s zPlatform.inference_modeseed int | NoneNonecCs<|durt|tj|t|tj|dSdS)z Set the seed of each random module. `torch.manual_seed` will set seed on all devices. Loosely based on: https://github.com/Lightning-AI/pytorch-lightning/blob/2.4.0/src/lightning/fabric/utilities/seed.py#L20 N)randomrnprZ manual_seedr[manual_seed_all)r^rrrrseed_everything2s    zPlatform.seed_everything model_archcCry)a  Verify whether the current platform supports the specified model architecture. - This will raise an Error or Warning based on the model support on the current platform. - By default all models are considered supported. Nr)r^rrrrverify_model_arch@s zPlatform.verify_model_archquantcCs.|jr||jvrt|d|jddSdS)zW Verify whether the quantization is supported by the current platform. z, quantization is currently not supported in r<N)rJ ValueErrorrD)r^rrrrverify_quantizationLszPlatform.verify_quantizationtorch.types.Device | NonefloatcCr})z3 Return the memory usage in bytes. r~)r^rFrrrget_current_memory_usageWsz!Platform.get_current_memory_usageFT distributed empty_cache cpu_grouprcCr})z5 Return the available memory in GiB. r~)r^rrrrrrrget_available_gpu_memory`s z!Platform.get_available_gpu_memorycCry)zW Get device specific communicator class for distributed communication. znsglang.multimodal_gen.runtime.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBaserr]rrrget_device_communicator_clsmrz$Platform.get_device_communicator_clsr4cCstjS)z1Get the CPU architecture of the current platform.)r4r3r]rrrget_cpu_architecturetszPlatform.get_cpu_architecturecCry)zKWhether to enable DIT layerwise offload by default on the current platform.Trr]rrr/enable_dit_layerwise_offload_for_wan_by_defaultyr{z8Platform.enable_dit_layerwise_offload_for_wan_by_defaultr cOs|j|i|}t|Sr )rr)rargskwargsattention_cls_strrrrget_attn_backend~szPlatform.get_attn_backendr)rr|)rrrr8rrrr;)r)rr8rr)rrrr8rr)rr8rr;)rr8rr8)rr8rr|r@)rrrrr )rrrr)rr;rr)rr;rr)rFrrr)rFTN) rr8rrrrrrrr)rr4)rr )0rrrrArFrGrIrJrrOrRrUrVrW classmethodr_rbrdrNrTrmrprrrsrtrwrxrzrrr\rrrrrrrrrrrrrrrrrrrrrrB[s                                     rBc@seZdZejZdZdS)UnspecifiedPlatformrN)rrrr*r3rCrErrrrrsr) __future__rr r functoolsrtypingrrrnumpyrrZ1sglang.multimodal_gen.runtime.utils.logging_utilsrsglang.multimodal_gen.utilsrIsglang.multimodal_gen.runtime.layers.attention.backends.attention_backendr rloggerEnumr r*r4r7rBrrrrrs(      *