o im@sddlmZmZmZmZddlZddlmZddl m Z m Z ddl m Z dZejo0ejjduZejejejejhZe d+dejdejd efd d Zedde jdfd ejd ede deedejf ddZ edde jddfdejdejd ede deed edejfddZ!d ejdejfddZ"d ejdejdejfddZ#dejdejdeedffddZ$dd Z%d!ed"edefd#d$Z&dejd%eeeefdejfd&d'Z'd(ejfd)d*Z(dS),)IterableOptionalTupleUnionN)AsyncCollectiveTensor all_reduce)ScalingGranularityg-q=Famax float8_dtyperound_scales_to_power_of_2cCsZ|tj}|tvrt|jtj|td}|tj}nt d||r+t |}|S)zConverts the amax value of a tensor to the fp8 scale. Args: amax: The amax value of the tensor. float8_dtype: The float8 dtype. round_scales_to_power_of_2: if true, round scaling factor down to the nearest power of 2. )minUnsupported float8_dtype: ) totorchfloat64 FP8_TYPESfinfomaxclampEPSfloat32 ValueError_round_scale_down_to_power_of_2)r r r resrO/home/ubuntu/.local/lib/python3.10/site-packages/torchao/float8/float8_utils.py amax_to_scales rx reduce_amaxscaling_granularity axiswise_dimreturncCs|tjurtt|}n|tjusJd|dusJdtjt||dd}|rWtrW|dur8| nd}|durFt t t n|}t |d|}t|trW|}|S)N unsupportedT)dimkeepdimMAX)r TENSORWISErrabsAXISWISEr distis_initialized get_grouplistrangeget_world_sizer isinstancerwait)rr device_meshrr r pggrouprrrtensor_to_amax8s    r4 hp_tensorcCst|||||}t|||dS)a Compute scaling factor for the given high precision tensor. Args: hp_tensor: high precision tensor float8_dtype: the float8 dtype to use reduce_amax: whether to reduce the max(abs(hp_tensor)) value across distributed ranks scaling_granularity: Defines the scaling granularity axiswise_dim: if axiswise granularity is used, defines the dim to scale across round_scales_to_power_of_2: if true, round scaling factor down to the nearest power of 2. )r )r4r)r5r rr1rr r r rrrtensor_to_scaleUsr6cCs<|tvrt|j}|j| |d}||Std|)aConverts a tensor to a saturated fp8 tensor. Note: The default behavior in PyTorch for casting to `float8_e4m3fn` and `e5m2` is to not saturate. In this context, we should saturate. A common case where we want to saturate is when the history of a tensor has a maximum value of `amax1`, and the current amax value is `amax2`, where `amax1 < amax2`. This is common when using delayed scaling. )r rr )rrrrrrr)rr max_valuerrrto_fp8_saturatedvs   r8ycCs.tj|}tj||}dt||S)zComputes the error between two tensors in dB. For more details see: https://en.wikipedia.org/wiki/Signal-to-noise_ratio Args: x: The original tensor. y: The tensor to compare to the original tensor. )rlinalg vector_normlog10)rr9PsPnrrr compute_errors r@tensor.cCsb|tvr t|j}ntd||jj|jd}t||k }|dk }||fS)zCalculate FP8 tensor stats Args: tensor: The tensor to calculate stats for. float8_dtype: The float8 dtype. Returns: A tuple containing the number of zeros and the number of max values. r )dtyper) rrrrr_datar _orig_dtyper'sumitem)rAr FP8_MAXtensor_orig_typenum_maxnum_zerorrrfp8_tensor_statisticss rKcCs0t|dks Jd|d|dko|ddkS)Nz%is_row_major only supports 2D tensorsr)len)striderrr is_row_majorsrPsizealignment_valuecCsd|d||S)a Returns the minimum alignment value that is greater than or equal to the given size. Args: size: The size of the data to be aligned. alignment_value: The alignment value to be used. Returns: int: The minimum alignment value that is greater than or equal to the given size. Usage: ``` >>> _get_min_alignment(10, 8) 16 ``` rMr)rQrRrrr_get_min_alignmentsrSdimscCs~|dksJ|j\}}t|tr|f}d|vrt|dn|}d|vr)t|dn|}||}||}tjj|d|d|fS)aP Pads a 2D tensor with zeros to ensure that its dimensions are multiples of 16, which is required `torch._scaled_mm` Args: tensor: The tensor to pad. dims: Dimensions to pad. Returns: torch.Tensor: The padded tensor. Usage: ``` >>> pad_tensor_for_matmul(torch.randn((10, 10)), dims=0).shape torch.Size([16, 10]) >>> pad_tensor_for_matmul(torch.randn((10, 10)), dims=1).shape torch.Size([10, 16]) >>> pad_tensor_for_matmul(torch.randn((10, 10)), dims=(0, 1)).shape torch.Size([16, 16]) ``` rLrrM) r#shaper/intrSrnn functionalpad)rArTdim1dim2 dim1_aligned dim2_alignedpad_dim1pad_dim2rrrpad_tensor_for_matmuls  rascalecCs*|jtjks Jdttt|S)Nzscale must be float32 tensor)rBrrexp2floorlog2)rbrrrrsr)F))typingrrrrrtorch.distributed distributedr))torch.distributed._functional_collectivesrrtorchao.float8.configrrcuda is_availableversionhipIS_ROCM float8_e4m3fn float8_e5m2float8_e4m3fnuzfloat8_e5m2fnuzrno_gradTensorrBboolrr&rWr4r6r8r@rKrPrSrarrrrrs         (