o پi@sddlZddlmZddlmZmZddlZddlmZddl m Z ddl m Z m Z mZdd d Zejd d Ze  ddejdededeedeejejff ddZe ddejdejdedejfddZdS)N)SimpleNamespace)OptionalTuple)flashinfer_api)#gen_mxfp8_quantization_sm100_module)device_support_pdlregister_custom_opregister_fake_opcCs,||d||}|ddd}||S)Nr) total_row total_columnrow_size padded_row padded_columnrrO/home/ubuntu/.local/lib/python3.10/site-packages/flashinfer/fp8_quantization.py _compute_swizzled_layout_sf_sizesrcsttddd   ddtjdtdtd ttd ttjtjff fd d }t d  ddtjdtdtd ttjtjffd d}tddd ddtjdtjdtd tjffdd }t d ddtjdtjdtd tjfdd}t ||dS)Nz flashinfer::mxfp8_quantize_sm100) mutates_argsT inputis_sf_swizzled_layout alignment enable_pdlreturnc s<|jjdkr?tj|jtj|jd}|r"t|jd|jddd}n|d}tj|ftj|jd}||||||fS|durHt |j}||jd}|jd}||d||} tjg|jdd| Rtj |jd}|rt|| dd}n|| d}tj|ftj|jd} ||||||||fS) aKQuantize input tensor to MxFP8 format. Args: input (torch.Tensor): Input tensor of shape [M, K] with dtype fp16/bf16/fp8_quantized. is_sf_swizzled_layout (bool, optional): Whether to use swizzled layout for scale factors. Defaults to True. alignment (int, optional): sfVecSize. Defaults to 32. Note that alignment is not used in the host kernel. enable_pdl (Optional[bool], optional): Whether to enable PDL (Programmatic Dependent Launch). If None, automatically detects based on device capability. Defaults to None. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - Quantized tensor of shape [M, K] with dtype FLOAT8_E4M3 - Scale factors tensor with shape determined by layout and sf_vec_size cpudtypedevicerrrr N) r!typetorchemptyshapeuint8rnumelmxfp8_quantize_hostr float8_e4m3fnmxfp8_quantize) rrrrout_val out_sf_sizeout_sfmkpadded_kmodulerrmxfp8_quantize_sm100sL     zAget_mxfp8_quantization_sm100_module..mxfp8_quantize_sm100cSs8|j\}}|j||gtjd|j||dgtjdfS)Nr r)r& new_emptyr$int64int32)rrrr/r0rrr_fake_mxfp8_quantize_sm100[s zGget_mxfp8_quantization_sm100_module.._fake_mxfp8_quantize_sm100z'flashinfer::mxfp8_dequantize_host_sm100)r scale_tensorcs*tj|jtj|jd}|||||S)aDequantize input tensor from MxFP8 format. Args: input (torch.Tensor): Input tensor of shape [M, K] with dtype FLOAT8_E4M3. scale_tensor (torch.Tensor): Scale factors tensor with shape determined by layout and sf_vec_size. is_sf_swizzled_layout (bool, optional): Whether to use swizzled layout for scale factors. Defaults to True. Returns: torch.Tensor: Dequantized float tensor of shape [M, K] with dtype float32. r)r$r%r&float32r!mxfp8_dequantize_host)rr:routr2rrmxfp8_dequantize_host_sm100gszHget_mxfp8_quantization_sm100_module..mxfp8_dequantize_host_sm100cSs |j|jd|jdgtjdS)Nrrr5)r6r&r$r;rr:rrrr!_fake_mxfp8_dequantize_host_sm100s zNget_mxfp8_quantization_sm100_module.._fake_mxfp8_dequantize_host_sm100)r4r>TrN)TrT) rbuild_and_loadr r$Tensorboolintrrr r)r4r9r>r@rr2r#get_mxfp8_quantization_sm100_modulesx =   rGTrrrrrrcCsJd}|jd|dks J|durt|j}t||||\}}||fS)aQuantize input tensor to MxFP8 format. This function implements MxFP8 quantization that converts input tensors to a compressed MxFP8 format with associated scale factors. It supports various input data types and scale factor layouts. Args: input (torch.Tensor): Input tensor of shape [M, K] with dtype fp16/bf16/fp8_quantized. is_sf_swizzled_layout (bool, optional): Whether to use swizzled layout for scale factors. Defaults to True. alignment (int, optional): sfVecSize. Defaults to 32. enable_pdl (Optional[bool], optional): Whether to enable PDL (Programmatic Dependent Launch). If None, automatically detects based on device capability. Defaults to None. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - Quantized tensor of shape [M, K] with dtype FLOAT8_E4M3 - Scale factors tensor with shape determined by layout and sf_vec_size rr"rN)r&rr!rGr4)rrrr sf_vec_sizex_qsfrrrr+s r+r:cCst|||S)avDequantize input tensor from MxFP8 format. This function performs dequantization by converting a packed FP8 tensor in MxFP8 format back to float values using the associated scale factors. Args: input (torch.Tensor): Packed FP8 tensor in MxFP8 format of shape [M, K] with dtype FLOAT8_E4M3. scale_tensor (torch.Tensor): Scale factors tensor with shape determined by layout and sf_vec_size. is_sf_swizzled_layout (bool, optional): Whether scale factors use swizzled layout. Defaults to True. Returns: torch.Tensor: Dequantized float tensor of shape [M, K] with dtype float32. )rGr>r?rrrr<s r<)r rArB) functoolstypesrtypingrrr$ api_loggingrjit.fp8_quantizationrutilsrr r rcacherGrDrErFr+r<rrrrsH     { $