o Ti4 @s$ddlmZmZGdddeZdS)) CUDAOpBuilderinstalled_cuda_versioncs\eZdZdZdZdfdd ZddZdfd d Zd d Zd dZ ddZ ddZ Z S)InferenceBuilderDS_BUILD_TRANSFORMER_INFERENCEtransformer_inferenceNcs$|dur|jn|}tj|ddS)N)name)NAMEsuper__init__)selfr __class__b/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/ops/op_builder/transformer_inference.pyr szInferenceBuilder.__init__cCsd|jdS)Nz$deepspeed.ops.transformer.inference._op)rr rrr absolute_nameszInferenceBuilder.absolute_nameFcszddl}Wnty|r|dYdSwd}|s\|jr\t\}}t|jj dd}|j dj }|dkrG|rE|dd}|dkr\|d ksS|d kr\|rZ|d d}t |oc|S) Nz?Please install torch if trying to pre-compile inference kernelsFT.zDNVIDIA Inference is only supported on Pascal and newer architectures z6On Ampere and higher architectures please use CUDA 11+)torch ImportErrorwarningis_rocm_pytorchcuda is_availablerintversionsplitget_device_propertiesmajorr is_compatible)r verboser cuda_okaysys_cuda_major_torch_cuda_majorcuda_capabilityr rrr#s,      zInferenceBuilder.is_compatiblecCsbg}g}dd|DD]}t|ddkr||q ||q t|dkr/|d||S)NcSsg|]}|dqS)r)r ).0ccrrr /sz/InferenceBuilder.filter_ccs..rrzFiltered compute capabilities )rappendlenr)r ccs ccs_retained ccs_prunedr+rrr filter_ccs,s   zInferenceBuilder.filter_ccscCsgdS)N) z.csrc/transformer/inference/csrc/pt_binding.cppz'csrc/transformer/inference/csrc/gelu.cuz'csrc/transformer/inference/csrc/relu.cuz-csrc/transformer/inference/csrc/layer_norm.cuz+csrc/transformer/inference/csrc/rms_norm.cuz*csrc/transformer/inference/csrc/softmax.cuz-csrc/transformer/inference/csrc/dequantize.cuz7csrc/transformer/inference/csrc/apply_rotary_pos_emb.cuz,csrc/transformer/inference/csrc/transform.cuz0csrc/transformer/inference/csrc/pointwise_ops.currrrrsources8zInferenceBuilder.sourcescCs|sdgSgS)Nz-lcurand)rrrrr extra_ldflagsFszInferenceBuilder.extra_ldflagscCsddgS)Nz#csrc/transformer/inference/includesz csrc/includesrrrrr include_pathsLr4zInferenceBuilder.include_paths)N)F) __name__ __module__ __qualname__ BUILD_VARrr rr#r2r3r5r6 __classcell__rrr rr s rN)builderrrrrrrrs