o i8&@sddlZddlmZddlZddlmZejddZedededed ejd ej j j gd ed ed ededededddZ ddZ ejddZ          dzdedededeeded ed!eed"ed#ed$ed%ed&ed'ed(ed)ed*ed+ed,ef$d-d.Ze d/          dzdedededeeded ed!eed"ed#ed$ed%ed&ed'ed(ed)ed*ed+ed,ef$d0d1Z  d{d2ed3ed4ed5ed6ed7eed8eejd,efd9d:Ze d;  d{d2ed3ed4ed5ed6ed7eed8eejd,efdZe d?d4ed,eeffd@d1ZdAedBedCedDed,ef dEdFZe dGdAedBedCedDed,ef dHd1ZdAedBedCedDedIedJed7eedKeed8eejd,efdLdMZe dNdAedBedCedDedIedJed7eedKeed8eejd,efdOd1ZedPdQZdRdSZe dTdUedVedWedXefdYdZZd4ed[ed\ed,efd]d^Ze d_d4ed[ed\ed,efd`d1Zd2edaedbed4edceddedeed7eed8ejfdfdgZe dhd2edaedbed4edceddedeed7eed8ejd,efdid1Ze djdkedledmedned*edoedped8ejd,efdqd1Zd4ed[ed,efdrdsZe dtd4ed[ed,efdud1Zd2edaed4edced7eed8ejf dvdwZ e dxd2edaed4edced7eed8ejd,efdyd1ZdS)|N)Optional)TensortorchaoFRAGMENTzrowwise_scaled_linear_sparse_cutlass_f8f8(Tensor input, Tensor input_scale, Tensor weight, Tensor weight_meta, Tensor weight_scale, Tensor? bias=None, ScalarType? out_dtype=None) -> TensorzLto_sparse_semi_structured_cutlass_sm9x_f8(Tensor weight) -> (Tensor, Tensor)z\swizzle_mm(Tensor mat1, Tensor mat2, bool mat1_is_swizzled, bool mat2_is_swizzled) -> Tensorzswizzle_scaled_mm(Tensor mat1, Tensor mat2, bool mat1_is_swizzled, bool mat2_is_swizzled, Tensor scale_a, Tensor scale_b, Tensor? bias=None, Tensor? scale_result=None, ScalarType? out_dtype=None) -> TensorzImx_fp8_bf16(Tensor a, Tensor b, Tensor a_scale, Tensor b_scale) -> Tensor)tagsa6qscaled_dot_product(Tensor query, Tensor key, Tensor value, Tensor? attn_mask=None, float dropout_p=0.0, bool is_causal=False, float? scale=None, float q_scale=1.0, int q_zp=0, float k_scale=1.0, int k_zp=0, float v_scale=1.0, int v_zp=0, float a_scale=1.0, int a_zp=0, float o_scale=1.0, int o_zp=0) -> Tensorzida8w4_linear_prepack_cpu(Tensor weight, Tensor scales, Tensor qzeros) -> (Tensor, Tensor, Tensor, Tensor)zda8w4_linear_cpu(Tensor input, Tensor input_scales, Tensor input_qzeros, Tensor weight, Tensor weight_scales, Tensor weight_qzeros, Tensor compensation, Tensor? bias, ScalarType output_dtype) -> Tensorz_scaled_embedding_bag(Tensor qweight, Tensor indices, Tensor offsets, Tensor weight_scale, float o_scale, int mode, bool include_last_offset, ScalarType output_dtype) -> TensorzKfloat8_linear_prepack_cpu(Tensor weight, Tensor scales) -> (Tensor, Tensor)zfloat8_linear_cpu(Tensor input, Tensor input_scales, Tensor weight, Tensor weight_scales, Tensor? bias, ScalarType output_dtype) -> Tensorcfdd}|S)Ncstj|SN)torchlibrary register_fakefuncname?/home/ubuntu/.local/lib/python3.10/site-packages/torchao/ops.py decorator4sz%register_custom_op..decoratorrrrrrrregister_custom_op3 rcr)Ncstjjdd|S)Nr) mutates_args)r r custom_opr rrrr;sz*register_custom_op_impl..decoratorrrrrrregister_custom_op_impl:rrcCs&tjtj}|jd|j}|S)N )r cudaget_device_propertiescurrent_devicemajorminor) device_propscompute_capabilityrrrcached_compute_capabilityAsr!F?querykeyvalue attn_mask dropout_p is_causalscaleq_scaleq_zpk_scalek_zpv_scalev_zpa_scalea_zpo_scaleo_zpreturncCs0tjjj|||||||||| | | | | |||S)a Quantized SDPA with quantized inputs and outputs. Arguments query: input query tensor, key: input key tensor, value: input value tensor, attn_mask: attention mask tensor, dropout_p: dropout probability, is_causal: causal flag, scale: scaling factor applied prior to softmax, q_scale: scale for query from linear quantization, q_zp: zero point for query from linear quantization, k_scale: scale for key from linear quantization, k_zp: zero point of key from linear quantization, v_scale: zero point for value from linear quantization, v_zp: zero point of value from linear quantization, a_scale: scale for attention from softmax quantization, a_zp: zero point for attention from softmax quantization, o_scale: scale for output from linear quantization, o_zp: zero point for output from linear quantization, Returns output of quantized SDPA )r opsrqscaled_dot_productdefaultr$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4rrrr7Hs& *r7ztorchao::qscaled_dot_productcCs|Srrr9rrr_sr:input input_scaleweight weight_meta weight_scalebias out_dtypec Cstjjj|||||||S)a. CUTLASS-based row-wise scaled F8F8 linear operator, for sparsified weight case. Args: input: quantized input tensor, in row-major layout. input_scale: scale factors for input tensor, has to be tensor of the same shape as the input tensor, minus the last dimension. weight: sparsified quantized weight matrix, in row-major layout. weight_meta: sparsify metadata for weight tensor. weight_scale: scale factors for weight tensor, one value per row of weight matrix (thus also tensor of the same shape as the weight tensor, minus the last dimension). bias: an optional vector of size equal to number of rows of weight tensor, or None. out_dtype: optional data type for output tensor. Returns: output: result tensor, in row-major layout. )r r6r)rowwise_scaled_linear_sparse_cutlass_f8f8r8)r;r<r=r>r?r@rArrrrBs rBz2torchao::rowwise_scaled_linear_sparse_cutlass_f8f8c CsB|dur|n|j}|j}tjg|jdd|jdR||dS)Nrdtypedevice)rErFr emptyshape) r;r<r=r>r?r@rArErFrrrr:s *cCstjjj|S)a CUTLASS-based conversion from sparsified input tensor to corresponding compressed tensor, along with corresponding metadata tensor. Args: weight: input tensor, in row-major layout. Returns: weight_compressed: compressed weight tensor, with sparsity eliminated, in row-major layout. weight_meta: metadata tensor, describing the sparsity structure of the input tensor, also in row-major layout. )r r6r)to_sparse_semi_structured_cutlass_sm9x_f8r8r=rrrrIs rIz2torchao::to_sparse_semi_structured_cutlass_sm9x_f8cCs<||d|dd|j|dt|dddtjdfS)NrrE) new_emptymaxr charrJrrrr:s"mat1mat2mat1_is_swizzledmat2_is_swizzledcCstjjj||||S)zO Similar to torch.mm but Tensor inputs can be SwizzleTensor instances. )r r6r swizzle_mmr8rSrTrUrVrrrrWs rWztorchao::swizzle_mmcC||jd|jdSNrrKrPrHrXrrrr:sscale_ascale_b scale_resultc C tjjj||||||||| S)zP Similar to torch.mm but Tensor inputs can be SwizzleTensor instances. )r r6rswizzle_scaled_mmr8 rSrTrUrVr\r]r@r^rArrrr`s r`ztorchao::swizzle_scaled_mmc CrYrZr[rarrrr:s cCsttdr tjtjfStjfS)zGTODO: when e8m0 is hardened and major release lets remove uint8 supportfloat8_e8m0fnu)hasattrr uint8rbrrrr _get_dtypes%s  recs>t}tj|vfddtj|vfdddS)Nc djS)Nz4A_scale tensor must be uint8 or float8_e8m0fnu, got rOr)A_scalerr2 z%_check_scale_dtypes..crf)Nz4B_scale tensor must be uint8 or float8_e8m0fnu, got rOr)B_scalerrrh6ri)rer _checkrE)rgrjallowed_dtypesr)rgrjr_check_scale_dtypes-s  rmztorchao::mx_fp8_bf16ABrgrjcCs$tj|d|dftj|jdS)zMeta impl for mx_fp8_bf16rrKrD)r rGsizebfloat16rF)rnrorgrjrrrmeta_mx_fp8_bf16:s$rrscalesqzeroscCstjjj|||S)z Prepack weights for DA8W4 linear operator on CPU. Args: weight: weight tensor. scales: scales for weight tensor. qzeros: zero points for weight tensor. Returns: packed weight, scales, and zero points. )r r6rda8w4_linear_prepack_cpur8r=rsrtrrrru@sruz!torchao::da8w4_linear_prepack_cpucCs|||tfSr)r rrvrrrr:Qs input_scales input_qzeros weight_scales weight_qzeros compensationc Cr_)a DA8W4 linear operator on CPU. Args: input: input tensor. input_scales: scales for input tensor. input_qzeros: zero points for input tensor. weight: weight tensor. weight_scales: scales for weight tensor. weight_qzeros: zero points for weight tensor. compensation: compensation tensor for weight. bias: optional bias tensor. out_dtype: output data type. Returns: output tensor in out_dtype. )r r6rda8w4_linear_cpur8) r;rwrxr=ryrzr{r@rArrrr|Vs r|ztorchao::da8w4_linear_cpuc CsL|dksJ|d|dd} |jg|jdd| Rd|iS)NrrLrCrEdimrprPrH) r;rwrxr=ryrzr{r@rANrrrr:}s $ztorchao::_scaled_embedding_bagqweightindicesoffsetsw_scalesmodeinclude_last_offsetc Cs0|dksJ|jdd}|j||jd|dS)NTrrKrO)rHrP) rrrrr3rrrA batch_sizerrrr:s cCstjjj||S)z Prepack weights for float8 linear operator on CPU. Args: weight: weight tensor. scales: scales for weight tensor. Returns: packed weight, packed scales )r r6rfloat8_linear_prepack_cpur8r=rsrrrrs rz"torchao::float8_linear_prepack_cpucCs||fSrrrrrrr:scCstjjj||||||S)aH float8 linear operator on CPU. Args: input: input tensor. input_scales: scales for input tensor. weight: weight tensor. weight_scales: scales for weight tensor. bias: optional bias tensor. out_dtype: output data type. Returns: output tensor in out_dtype. )r r6rfloat8_linear_cpur8)r;rwr=ryr@rArrrrs rztorchao::float8_linear_cpucCs^|dvsJ|dkr|d|dn|d}|jg|jdd|Rd|iS)N)rLr}r}rr~rCrEr)r;rwr=ryr@rArrrrr:s *$)Nr"FNr#rr#rr#rr#rr#r)NN)! functoolstypingrr rr Librarylibdefine_CTagneeds_fixed_stride_orderrr lru_cacher!floatboolintr7r:rErBrIrWr`rermrrrur|rrrrrrs           ?                      '