o پi@srddlmZddlZddlZddlmZmZmZddlm Z m Z m Z ddd Z dd d Z      dd dZdS))OptionalN) gemm_kernel!gemm_kernel_descriptor_persistentgemm_kernel_persistent) check_device check_dim check_input?c st|t||gtd|td||dur&t|t|gtd||jd|jdks4Jd|j|jks>Jd|dur^|jd|jdksPJd|jd|jdks^Jd|j\}|j\}|j}|ro|n |tjkrv|ntj}|dus|j|ksJd |durtjf|j |d n|}tj d j } dur| nt | fd d } t| |||||d|d|d|d|d|d||d|S)a GEMM operation with SM constraint by Triton. C = alpha * (a @ b.T) + beta * C Args: a: The first input matrix. Shape: (M, K) b: The second input matrix. Shape: (K, N) c: The output matrix. Shape: (M, N). In-place epilogue is supported. Expected to be out_dtype (if not specified, same as a.dtype, but fp8 --> bf16). alpha: The scaling factor for the product of a and b. beta: The scaling factor for the output matrix c. out_dtype: The dtype of the output matrix. Default: fp8 --> bf16. Otherwise, same as a.dtype. num_sms: The number of SMs to use for the computation. Nrr'Incompatible dimensions between a and b#Incompatible dtypes between a and b'Incompatible dimensions between a and c'Incompatible dimensions between b and c+Incompatible dtypes between c and out_dtypedevicedtypecudac(tt|dt|dfSN BLOCK_SIZE_M BLOCK_SIZE_NmintritoncdivMETAMNnum_smsX/home/ubuntu/.local/lib/python3.10/site-packages/flashinfer/triton/sm_constraint_gemm.pyG z!gemm_persistent..)alphabetaNUM_SMS)r rrshapertorch float8_e4m3fnbfloat16emptyrrget_device_propertiesmulti_processor_countrrstride) abcr(r) out_dtyper#Krr*gridr$r r%gemm_persistents^        "r9c st|t||gtd|td||dur&t|t|gtd||jd|jdks4Jd|j|jks>Jd|dur^|jd|jdksPJd|jd|jdks^Jd|j\}|j\}|j}|ro|n |tjkrv|ntj}|dus|j|ksJd |durtjf|j |d n|}fd d }t |||||| d| d| d| d| d| d||d |S)a GEMM operation without SM constraint by Triton. C = alpha * (a @ b.T) + beta * C Args: a: The first input matrix. Shape: (M, K) b: The second input matrix. Shape: (K, N) c: The output matrix. Shape: (M, N). In-place epilogue is supported. Expected to be out_dtype (if not specified, same as a.dtype, but fp8 --> bf16). alpha: The scaling factor for the product of a and b. beta: The scaling factor for the output matrix c. out_dtype: The dtype of the output matrix. Default: fp8 --> bf16. Otherwise, same as a.dtype. num_sms: The number of SMs to use for the computation. r Nrrr rrrrrcs"t|dt|dfSr)rrrr!r"r$r%r&szgemm..)r(r)) r rrr+rr,r-r.r/rrr2) r3r4r5r(r)r6r7rr8r$r:r%gemmbsX        "r;Fc st|t|t||gtd|td||dur*t|t|gtd||jd|jdks8Jd|j|jksBJd|durb|jd|jdksTJd|jd|jdksbJd|j\}|j\}|j} |rs|n | tjkrz| ntj}| tjkr|d ksJd d ksJd n|d ksJd d ksJd |durtjf|j |d n|}tj d j } dur| nt | dtdtdttfdd} t| fdd} t| ||||||d| tjkrdnddd dd |d|S)a GEMM operation with SM constraint by Triton. Requires TMA support and descriptor creation. C = alpha * (a @ b.T) + beta * C Note: - K and N must be greater than 16B. - Support float16, float8_e4m3fn, bfloat16. - float32 is not supported due to performance issues. Args: a: The first input matrix. Shape: (M, K) b: The second input matrix. Shape: (N, K) c: The output matrix. Shape: (M, N). In-place epilogue is supported. Expected to be out_dtype (if not specified, same as a.dtype, but fp8 --> bf16). alpha: The scaling factor for the product of a and b. beta: The scaling factor for the output matrix c. out_dtype: The dtype of the output matrix. Default: fp8 --> bf16. Otherwise, same as a.dtype. num_sms: The number of SMs to use for the computation. EPILOGUE_SUBTILE: Whether to use the epilogue subtile optimization. r Nrr rrrrzLeast chunk size must be 16Brrsize alignmentstreamcSstj|dtjdS)Nrr)r,r/int8)r>r?r@r$r$r%alloc_fnsz,gemm_descriptor_persistent..alloc_fncrrrrr r$r%r&r'z,gemm_descriptor_persistent..@)r*rr BLOCK_SIZE_K GROUP_SIZE_M num_stages num_warpsEPILOGUE_SUBTILE)r rrr+rr,r-r.r/rrr0r1rintrr set_allocatorrfloat32) r3r4r5r(r)r6r#rJr7rr*rBr8r$r r%gemm_descriptor_persistentsj         " rN)Nr r NN)Nr r N)Nr r NNF)typingrr,rkernels.sm_constraint_gemmrrrutilsrrr r9r;rNr$r$r$r%s   TN