o 2wi#I,@s0 ddlmZddlZddlZddlmZddlmZm Z m Z ddl Z ddl Z ddl mZddlmZddlmZmZdd lmZiZ ejejejfejejfejejfejejej fej!ej"fejejejfej#ej$ej%fd Z&ej'ej(fej)ej*fej+ej,fej-ej.fej'ej(fej)ej*fd Z/ej0ej1ej2fej3ej4ej5fej6ej7ej8fej9ej:ej;fejfej?ej@ejAfd ZBGd ddZCGdddZDGdddZEe jFdddZGe jHIdkrde jfddZJn ddlKZKde jfddZJe jLeGdddZMdde jfddZNdd!d"ZOdd#d$ZPdd%d&ZQdd(d)ZRdd+d,ZSdd/d0ZTdd2d3ZUed4eVd5dd6d7ZWd8ee e jfd9d:ZXd;edZZde edr get_cusparser7rrrrr"szCusparse_Context.initializecCr$r r%r(rrrr*r+zCusparse_Context.get_instance)r2r3r4r&rr"r5r*rrrrrBs rBr:)r8acCs tj|Sr )r9r: device_ofrDrrr_cuda_device_ofs rGcCstSr ) contextlib nullcontextrFrrrrGrdtyper@cGsn|jt|}tt|}t|ttj}t j j ||d}t j ||t|d|}d|_|j|_|S)N)shape)rKcountT)itemsizerr cget_managed_ptrr=c_size_tcastPOINTERc_intnp ctypeslibas_arrayr9 frombufferviewis_pagedr8 page_deviceid)rKr@rL num_bytescuda_ptrc_ptr new_arrayoutrrr get_pagedsr`FAcCsB|jsJd|r d}n|j}tt|t|jt|dS)Nz%Only paged tensors can be prefetched!r,) rYrZr cprefetchget_ptrr=rPnbytesc_int32)rar.deviceidrrrr-s $r-TcCsd}|jtjkrttd|dd}t|}n|jtjkr-ttd|dd}t|}|dur8t d|t|dd}|rN|rNt ||durNt ||t |t ||t | |jsd|jrktjdSdS)Nc_fp32_uint8zFunction not implemented: is_managedF)rKr9float32getattrr r=c_floatuint8c_uint8NotImplementedErrorr-rcc_int64numelrYr: synchronize) func_nameraBvalueprefetchfunccvaluerjrrrelementwise_funcs$       rzcCstd|d|dS)Nfillrz)rarvr@rwrrrr{r{cCstd||ddS)N_mulrr|)rarur@rrrr~r}r~cCs|rdnd}d|}|s|dkr|sd|nd|d}t|d|}d|}|dkr/|S|d}t|d|dg|||dS) Nrr ?r)r9linspacerrrtolist)signed total_bitsadd_zerosign total_valuesvaluesgaplrrrcreate_linear_maps    0r+ew?c Cszddlm}Wnty}ztd|d}~ww|rC|t|dddd}dgd}|t|dddd }n&|t|dddd}dgd }|t|dddd }|||}t|}|j }|| }| d ksJ|S) Nr)normzZScipy is required for `create_normal_map`. Install `bitsandbytes` with the `[test]` extra.g? r,rr) scipy.statsr ImportErrorppfr9rrrsortrmaxrr) offsetuse_extra_valueriev1v2v3vrrrrcreate_normal_maps.  $  "    rrcCs|}|}|rdnd}||||ksJg}g}ttd|| d||dD] \} } |d| q)g} ttjddg|d} d|d} td|D]I}| D]D}|dkrZdnd}tt|D]\} }||d| d 7}qb|dkr~|d| }n |d|| d }| ||r| | qRqNt| d|ksJ| |dkrdt| }t|D]} | dq| t | }|| }|S)Nr rr)repeatrr) enumeraterangeappendlist itertoolsproductlenrr9tensorr)r exponent_bitsprecision_bitsrephas_signevaluespvaluesivalrlstbiasevalue bit_patternrvpvalrcoderrrcreate_fp8_map2sD *        rc Csg}|d}d||d}t|D]U}t|r!d|||dn d|||dd}tjdd|tjd}|dd|ddd} |d|d || 7}|rg|d|d | | 7}q|d krtjdd|dtjd}|dd|ddd} |d|d || 7}|r|d|d | | 7}|d |d t|d|ksJd t|} t| D]}|d q|tj |tjdS) a+ Creates the dynamic quantiztion map. The dynamic data type is made up of a dynamic exponent and fraction. As the exponent increase from 0 to -7 the number of bits available for the fraction shrinks. This is a generalization of the dynamic type where a certain number of the bits and be reserved for the linear quantization region (the fraction). n determines the maximum number of exponent bits. For more details see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561] r rg?rKNr,@ rrr) rintr9rrkrrrrr) rmax_exponent_bitsrdata non_sign_bitsadditional_itemsrfraction_items boundariesmeansrrrrcreate_dynamic_map]s:        rzDThis function is deprecated and will be removed in a future release.)categorycCsnt|d|dd}|}|ddt|}t|D]}|dq|t|}||}|S)Nrr ) num_quantilesrr) estimate_quantilesrrrrrrabsr)rarqrrrrrcreate_quantile_maps    rtensorscCsd}t}|D]}|durt|dds||jM}||jjq|s.tddd|Dt|dkr@td d d|D|S) apVerifies that the input tensors are all on the same device. An input tensor may also be marked as `paged`, in which case the device placement is ignored. Args: tensors (`Iterable[Optional[torch.Tensor]]`): A list of tensors to verify. Raises: `RuntimeError`: Raised when the verification fails. Returns: `Literal[True]` TNrYFzZAll input tensors need to be on the same GPU, but found some tensors to not be on a GPU: cSg|]}|j|jfqSrrLr@.0r/rrr zis_on_gpu..r zcInput tensors need to be on the same GPU, but found the following tensor and device combinations: cSrrrrrrrrr)setrlis_cudaaddr@r8rr)ron_gpugpu_idsr/rrr is_on_gpus   rrreturncCsttj|jjSr )r=r>r9_C_cuda_getCurrentRawStreamr@r8)rrrr_get_tensor_streamsrcCs|durdSt|S)zGets the memory address of the first element of a tenso Args: A (`Optional[Tensor]`): A PyTorch tensor. Returns: `Optional[ct.c_void_p]`: A pointer to the underlying tensor data. N)r=r>data_ptr)rarrrrcs rc`?rr_rc CsX|dkrtd|d|dkrtd||dkr)|dkr)dd|}|dur7tjd tj|jd }t|Lt||g|jtjkr]t t |t |t |t |n#|jtjkrxt t |t |t |t |ntd |jWdn1swY|dkrtd|}td d ||j}||}|S)a Estimates 256 equidistant quantiles on the input tensor eCDF. Uses SRAM-Quantiles algorithm to quickly estimate 256 equidistant quantiles via the eCDF of the input tensor `A`. This is a fast but approximate algorithm and the extreme quantiles close to 0 and 1 have high variance / large estimation errors. These large errors can be avoided by using the offset variable which trims the distribution. The default offset value of 1/512 ensures minimum entropy encoding -- it trims 1/512 = 0.2% from each side of the distrivution. An offset value of 0.01 to 0.02 usually has a much lower error but is not a minimum entropy encoding. Given an offset of 0.02 equidistance points in the range [0.02, 0.98] are used for the quantiles. Parameters ---------- A : torch.Tensor The input tensor. Any shape. out : torch.Tensor Tensor with the 256 estimated quantiles. offset : float The offset for the first and last quantile from 0 and 1. Default: 1/(2*num_quantiles) num_quantiles : int The number of equally spaced quantiles. Returns ------- torch.Tensor: The 256 quantiles in float32 datatype. rzQQuantile estimation needs at least 256 values in the Tensor, but Tensor had only z values.zgCurrently only a maximum of 256 equally spaced quantiles are supported, but the argument num_quantiles=rr rN)rrJzNot supported data type r)rrrpr9zerosrkr@rGrrKr cestimate_quantiles_fp32rcr=rmrSfloat16cestimate_quantiles_fp16roundrlongto)rar_rrstepidxrrrrs4 #    * * rc@seZdZdZdZddeDZgdZ       dddZd d Ze d e e e fd e jd dfddZdddZddZddZdS) QuantStatezWcontainer for quantization state components to work with Params4bit and similar classes)fp4nf4cCsg|]}d|qS)bitsandbytes__r)rxrrrr%szQuantState.) absmax quant_map nested_absmaxnested_quant_map quant_state quant_type blocksizerKrLnested_blocksize nested_dtype nested_offsetNc Cs>||_||_||_||_||_||_||_||_|du|_dSr ) rrLrrKrrrstate2nested) rrrLrrrrKrrrrrr5s zQuantState.__init__cCsR|jr|j|j|j|j|j|jg|jg}||S|j|j|j|jd|jg}||S)a$ ensures compatibility with older quant state scheme with nested lists. assumes the following layout: state = [qabsmax, input_shape, A.dtype, blocksize, [offset, state2], quant_type] state2 = [absmax, input_shape, A.dtype, blocksize, None, quant_type] N)rrrLrKrrrr)rr list_reprrrr __getitem__Js  zQuantState.__getitem__qs_dictr@rc Cspdd|D}t|sd|vrtdt|dks'|ddd|jvr3td |jd |dt|dkrG|d}|t||d d |D}t|  |j s\Jd |vrt t|d|}||d ||d|d|tt |dd}nd\}}||d|d||d|d|tt |d|ddurt |dnd||d}|S)aO unpacks components of state_dict into QuantState where necessary, convert into strings, torch.dtype, ints, etc. qs_dict: based on state_dict, with only relevant keys, striped of prefixes. item with key `quant_state.bitsandbytes__[nf4/fp4]` may contain minor and non-tensor quant state items. cSs(g|]\}}d|vrt|tjr|qS)r isinstancer9rrkrrrrrjs(z(QuantState.from_dict..rzws z(QuantState.from_dict..rrrrr)rrrrKNNrrrrKrLN)rrrrrKrLrr)itemsr ValueErrorrvalid_qs_type_keysupdater poprkeysissubset valid_qs_keysr9rfloatrrlSize)r)rr@qs_key first_qs_keyrrrrrr from_dict^s@ $        zQuantState.from_dictFcCs|j|j|j|jt|jdt|jd}|j r6| |j j|j j|j j t|j jd|j d|s:|Sdd|D}dd|D}t||d|j<|S)z returns dict of tensors and strings to use in serialization via _save_to_state_dict() param: packed -- returns dict[str, torch.Tensor] for state_dict fit for safetensors saving ztorch.)rrrrrKrL)rrrrrcSs"i|] \}}t|tjr||qSrrrrrrr"z&QuantState.as_dict..cSs"i|] \}}t|tjs||qSrrrrrrrrzquant_state.bitsandbytes__)rrrrstrrKstriptuplerLrr rcloneritemrr )rpackedrqs_packed_dictnon_tensor_dictrrras_dicts,  zQuantState.as_dictcCs\|j||_|j||_|jr,|j||_|jj||j_|jj||j_dSdSr )rrrrrr)rr@rrrrsz QuantState.tocCst|tsdStj|j|jddo^|j|jko^tj|j|jddo^|j|jko^|j|jko^|j |j ko^|j durC|j durC|j |j kn|j |j uo^|j durY|j durY|j |j kS|j |j uS)NFgư>)atol) rrr9allcloserrLrrKrrrr)rotherrrr__eq__s,        zQuantState.__eq__)NNNNNNNr1)r2r3r4__doc__valid_quant_typesr rrrr5dictrrr9r@rrrr"rrrrr!s& " 2 rrrc Cs|durdtvrt|jtd<td}tjjj|||j|\}}|rJ| }||8}t||dd\} } t | |j|jdd||j || d} nt ||j|jdd||j d} |durb| |n|}|duro| | j | _ || fS) aWQuantize a tensor in blocks of values. The input tensor is quantized by dividing it into blocks of `blocksize` values. The the absolute maximum value within these blocks is calculated for scaling the non-linear quantization. Args: A (`torch.Tensor`): The input tensor. Supports `float16`, `bfloat16`, or `float32` datatypes. code (`torch.Tensor`, *optional*): A mapping describing the low-bit data type. Defaults to a signed 8-bit dynamic type. For more details, see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561]. absmax (`torch.Tensor`, *optional*): A tensor to use to store the absmax values. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 4096. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. nested (`bool`, *optional*): Whether to additionally quantize the absmax values. Defaults to False. Raises: ValueError: Raised when the input data type is not supported. Returns: `Tuple[torch.Tensor, QuantState]`: A tuple containing the quantization results. - `torch.Tensor`: The quantized tensor. - [`QuantState`]: The state object used to undo the quantization. NdynamicF)rrT)copy)rrrrKrrrrrrK) name2qmaprrr@r9ops bitsandbytesquantize_blockwisedefaultmeanrrKcopy_r) rarrr_rr_out_absmaxrqabsmaxrrrrrr-s4#   r-rrcCs|dus |dus J|dur#|dur#dtvrt|jtd<td}|dur0t|||tjd}|j}|jrLt |j|j }||j 7}|j tjkrL| }|durgtjjj j|||j|j|j|j |d|Stjjj |||j|j|j|j S)aDequantize a tensor in blocks of values. The input tensor is dequantized by dividing it into blocks of `blocksize` values. The the absolute maximum value within these blocks is used for scaling the non-linear dequantization. Args: A (`torch.Tensor`): The quantized input tensor. quant_state ([`QuantState`], *optional*): The quantization state as returned by [`quantize_blockwise`]. Required if `absmax` is not provided. absmax (`torch.Tensor`, *optional*): A tensor containing the scaling values. Required if `quant_state` is not provided and ignored otherwise. code (`torch.Tensor`, *optional*): A mapping describing the low-bit data type. Defaults to a signed 8-bit dynamic type. For more details, see (8-Bit Approximations for Parallelism in Deep Learning)[https://arxiv.org/abs/1511.04561]. Ignored when `quant_state` is provided. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 4096. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. Ignored when `quant_state` is provided. Raises: ValueError: Raised when the input data type is not supported. Returns: `torch.Tensor`: The dequantized tensor. The datatype is indicated by `quant_state.dtype` and defaults to `torch.float32`. Nr'r)r_)r*rrr@rr9rkrrdequantize_blockwiserrrKrr+r,r_rrr.)rarrrr_rrrrrr5s<)      r5@cCs|durd}d}|dkr gd}n(|dkrgd}n|dkr$gd}n|dkr:|d kr6gd ddd }ntd |durFtd |dtj||d}|||dks^J|S)Nr:r)rg 6GgfgTFٿgI4ҿg০ǿg Org__?g`\?g?g@g?g4?g`?g`v"?rr)rg?g @g(@g@g@rg@rgg g(ggggint4)rrrr rrr,iiaf4r6)rg|8geg:Kڞ׿gH2퓊cпg}Yu-ÿgQ #(DrgF?g`_?g 0E?gL_߹E?gƶ=?ga@?gкv -?rr,z94-bit AbnormalFloats currently only support blocksize 64.z Typename z not supportedr@)rpr9rdiv_rrrr)typenamer@rrrrr get_4bit_typefs,   rDcCt|||||d|SNr quantize_4bitrarr_rcompress_statistics quant_storagerrr quantize_fp4rLcCrENrrGrIrrr quantize_nf4rMrOrc Cs|j}tjjj||||\}} t||jd} |r6| } t | | dd\} } ~ t | ||j || || | d}n t | ||j || |d}|durJ| |n|}|durW| |j |_ ||fS)aQuantize tensor A in blocks of 4-bit values. Quantizes tensor A by dividing it into blocks which are independently quantized. Args: A (`torch.Tensor`): The input tensor. Supports `float16`, `bfloat16`, or `float32` datatypes. absmax (`torch.Tensor`, *optional*): A tensor to use to store the absmax values. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 64. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. compress_statistics (`bool`, *optional*): Whether to additionally quantize the absmax values. Defaults to False. quant_type (`str`, *optional*): The data type to use: `nf4` or `fp4`. Defaults to `fp4`. quant_storage (`torch.dtype`, *optional*): The dtype of the tensor used to store the result. Defaults to `torch.uint8`. Raises: ValueError: Raised when the input data type is not supported. Returns: Tuple[`torch.Tensor`, `QuantState`]: A tuple containing the quantization results. - `torch.Tensor`: The quantized tensor with packed 4-bit values. - [`QuantState`]: The state object used to undo the quantization. r@r)r)rrLrKrrrrr)rrLrKrrrN)rLr9r+r,rHr.rDr@r/r-rrKr0r)rarr_rrJrrK input_shaper1r2rrr3rstaterrrrHsD   rHcCt|||||dSrFdequantize_4bitrarrr_rrrrdequantize_fp4rVcCrRrNrSrUrrrdequantize_nf4%rWrXc Cs|dur|dur |dusJt||j|j||d}n|j}|jr6t|j|j}||j7}|jtj kr6| }|durNtj j j j|||j|j|j|j|dntj j j |||j|j|j|j}|jddkrj|S|S)aDequantizes a packed 4-bit quantized tensor. The input tensor is dequantized by dividing it into blocks of `blocksize` values. The the absolute maximum value within these blocks is used for scaling the non-linear dequantization. Args: A (`torch.Tensor`): The quantized input tensor. quant_state ([`QuantState`], *optional*): The quantization state as returned by [`quantize_4bit`]. Required if `absmax` is not provided. absmax (`torch.Tensor`, *optional*): A tensor containing the scaling values. Required if `quant_state` is not provided and ignored otherwise. out (`torch.Tensor`, *optional*): A tensor to use to store the result. blocksize (`int`, *optional*): The size of the blocks. Defaults to 64. Valid values are 64, 128, 256, 512, 1024, 2048, and 4096. quant_type (`str`, *optional*): The data type to use: `nf4` or `fp4`. Defaults to `fp4`. Raises: ValueError: Raised when the input data type or blocksize is not supported. Returns: `torch.Tensor`: The dequantized tensor. N)rrLrKrrr4rr )rrLrKrrr5rrr9rkrr+r,rTr_rrr.r/)rarrr_rrrrrrT/s<"      rTcCsx|durdtvrt|jtd<td}||j}t|}|jtjkr,| }||}t |||}|||ffS)Nr') r*rrr@r9rrrKrkrquantize_no_absmax)rarr_rinprrrquantizexs    r[rQcCs~|dus |dus J|dur)|dur)dtvrt|jtd<td}||j}|dur1||f}t||d|}||dS)Nr'r r)r*rrr@dequantize_no_absmax)rarQrrr_rrr dequantizes  r]c Csxt|.|durtj|tjd}t||gtt|t|t|t | Wd|S1s5wY|S)a Quantizes input tensor to 8-bit. Quantizes the 32-bit input tensor `A` to the 8-bit output tensor `out` using the quantization map `code`. Parameters ---------- A : torch.Tensor The input tensor. code : torch.Tensor The quantization map. out : torch.Tensor, optional The output tensor. Needs to be of type byte. Returns ------- torch.Tensor: Quantized 8-bit tensor. Nr) rGr9 zeros_likernrr cquantizercr=rSrr)rarr_rrrrYs  ( rYc Cst|4|durtj|tjd}t|||gt|}tt|t|t|t | |Wd|S1s;wY|S)a Dequantizes the 8-bit tensor to 32-bit. Dequantizes the 8-bit tensor `A` to the 32-bit tensor `out` via the quantization map `code`. Parameters ---------- A : torch.Tensor The 8-bit input tensor. code : torch.Tensor The quantization map. out : torch.Tensor The 32-bit output tensor. Returns ------- torch.Tensor: 32-bit output tensor. Nr) rGr9r^rkrrr cdequantizercr=rSrr)rarr_streamrrrr\s * r\rroptimizer_namegrstate1beta1epsrlrrbeta2beta3alpha weight_decay gnorm_scale unorm_vec max_unormcCsfd}|dkrt|j}d}|jtjkrt|d}n.|jtjkr*t|d}n!|jtjkr?t t|dkr?t|d}n t d|jd|jt |||||gt |Q|t |t |t |t |t |t|t|t|t| t| t| t|t| t|t|t| t|t|WddS1swYdS) az Performs an inplace optimizer update with one or two optimizer states. Universal optimizer update for 32-bit state and 32/16-bit gradients/weights. Parameters ---------- optimizer_name : str The name of the optimizer: {adam}. g : torch.Tensor Gradient tensor. p : torch.Tensor Parameter tensor. state1 : torch.Tensor Optimizer state 1. beta1 : float Optimizer beta1. eps : float Optimizer epsilon. weight_decay : float Weight decay. step : int Current optimizer step. lr : float The learning rate. state2 : torch.Tensor Optimizer state 2. beta2 : float Optimizer beta2. beta3 : float Optimizer beta3. alpha : float Optimizer alpha. gnorm_scale : float The factor to rescale the gradient to the max clip value. unorm_vec : torch.Tensor The tensor for the update norm. max_unorm : float The maximum update norm relative to the weight norm. skip_zeros : bool Whether to skip zero-valued gradients or not (default: False). rNrr r:rAGradient+optimizer bit data type combination not supported: grad , optimizer )r9rrrrKrkstr2optimizer32bitrbfloat16rr rrGrcr=rmrec_boolrr)rbrcrrdrerfrrgrrhrirjrkrlrmrn skip_zeros param_norm optim_funcrrroptimizer_update_32bitsH>    "rwzyThis function is deprecated and will be removed in a future release. Please use optimizer_update_8bit_blockwise instead. qmap1qmap2max1max2new_max1new_max2cCs d}|dkrt|j}t|t|||||| | | | ||g |jtjkr|jtjkrt |dt |t |t |t |t |t |t |t |t |t |t |t | t | t | t | t | t |t |t |t |t |nk|jtjkr|jtjkrt |dt |t |t |t |t |t |t |t |t |t |t |t | t | t | t | t | t |t |t |t |t |ntd|jd|jWddSWddS1swYdS)a Performs an inplace Adam update. Universal Adam update for 32/8-bit state and 32/16-bit gradients/weights. Uses AdamW formulation if weight decay > 0.0. Parameters ---------- optimizer_name : str The name of the optimizer. Choices {adam, momentum} g : torch.Tensor Gradient tensor. p : torch.Tensor Parameter tensor. state1 : torch.Tensor Adam state 1. state2 : torch.Tensor Adam state 2. beta1 : float Adam beta1. beta2 : float Adam beta2. eps : float Adam epsilon. weight_decay : float Weight decay. step : int Current optimizer step. lr : float The learning rate. qmap1 : torch.Tensor Quantization map for first Adam state. qmap2 : torch.Tensor Quantization map for second Adam state. max1 : torch.Tensor Max value for first Adam state update. max2 : torch.Tensor Max value for second Adam state update. new_max1 : torch.Tensor Max value for the next Adam update of the first state. new_max2 : torch.Tensor Max value for the next Adam update of the second state. gnorm_scale : float The factor to rescale the gradient to the max clip value. unorm_vec : torch.Tensor The tensor for the update norm. max_unorm : float The maximum update norm relative to the weight norm. rrr rorpN)r9rrrrGrrKrkrnstr2optimizer8bitrcr=rmrerrrr )rbrcrrdrrerhrfrrgrxryrzr{r|r}rkrlrmrnrurrroptimizer_update_8bitGsxM     "rabsmax1absmax2cCsvd}|jtjkr|jtjkrt|d}n:|jtjkr(|jtjkr(t|d}n'|jtjkrC|jtjkrCtt|dkrCt|d}n td|jd|jt ||||| | ||gt |R|t |t |t |t |t |t |t |t |t | t | t | t | t | t |t |t |t |t |t |WddS1swYdS)Nrr r:rrorp)rKr9rkrnstr2optimizer8bit_blockwiserrrrr rrGrcr=rmrersrr)rbrcrrdrrerhrirjrfrrgrxryrrrkrlrtrvrrroptimizer_update_8bit_blockwisesH    "rgrad gnorm_vec percentilec Cst|Mt||g|jtjkr&tt|t|t |t | n$|jtj krAt t|t|t |t | n t d|jdWdn1sTwYt||d}t|\}}t||}d}||krz||}|||fS)aApplies percentile clipping grad: torch.Tensor The gradient tensor. gnorm_vec: torch.Tensor Vector of gradient norms. 100 elements expected. step: int The current optimization steps (number of past gradient norms). zGradient type z not supported!Ndr)rGrrKr9rkr cpercentile_clipping_g32rcr=rerrrcpercentile_clipping_g16r sqrtr) rrrr current_gnormvalsr clip_valuerlrrrpercentile_clippings4      r histogramindex1index2sourcecCst|jdks J|jtjksJ|jtjksJ|jtjks!J|jtjks)J|jjdks1J|jjdks9J|jjdksAJ|jjdksIJt |jd}t | }t ||||gt t|t|t|t|||dS)Nrr:r)rrLrKr9rkint32r@typer=rerrrr chistogram_scatter_add_2drc)rrrrmaxdim1nrrrhistogram_scatter_add_2d:s(rc Cstjs tj|j|ks|j|kr td|jd|j|j}|j}|}|} d} t|dkr|t|dkr||sI| sI|jd|jdkrId} n|rZ| sZ|jd|jdkrZd} n|rk| rk|jd|jdkrkd} n|s{| r{|jd|jdkr{d} nt|dkrt|dkr|s| s|jd|jdkrd} n|r| s|jd|jdkrd} ny|r| r|jd|jdkrd} nh|s| r|jd|jdkrd} nWt|dkr#t|dkr#|s| s|jd|jdkrd} n8|r| s|jd|jdkrd} n'|r| r|jd|jdkrd} n|s#| r#|jd|jdkr#d} |durc|j} | sbt|dkrbt|dkrb| d|dkrb| d|dkrb|d|dkrb|d|dkrbd} nt|dkrt|dkr|s| s|d|df} n|r| r|d|df} n|r| s|d|df} n|s| r|d|df} nt|dkrt|dkr|s| s|d|d|df} n|r| r|d|d|df} ny|r| s|d|d|df} ng|s| r|d|d|df} nUt|dkrXt|dkrX|s#| s#|d|d|df} n5|r5| r5|d|d|df} n#|rG| sG|d|d|df} n|sX| rX|d|d|df} | sltd |d |d |d | d | S) Nz3Expected torch.int8 input tensors A and B, but got  and Trr rFr:z?Tensor dimensions incorrect for matrix mulitiplication: A x B:  x z with transpose for A x B: r) r9r:is_initializedinitrK TypeErrorrLrr ) rarur_ transposed_A transposed_B expected_typesAsBtAtBcorrectsoutrrr check_matmulMs  "" "H            rruc Cs~|durtd|j}|jrt||j|j}|dur/tjjj j |||j ||j |j |d|Stjjj |||j ||j |j S)NzIstate cannot be None. gemv_4bit() requires the state from quantize_4bit()r4)r rrr5rrr9r+r, gemv_4bitr_rLrrr.)rarur_rrrQrrrrrs0  rcCsvt|||||}|durtj|tj|jd}t|jdkr>t|jdkr>|jd|jdkr>|jd|jdkr>t|||S|j}|j}|rUt|dkrU|d|df}n|rht|dkrh|d|d|df}|ryt|dkry|d|df}n|rt|dkr|d|d|df}t|dkr4|d|jdkrd}n |d|jdkrd}t|jdkr|d|jdkrd}n)|d|jdkrd}n|d|jdkrd}n |d|jdkrd}t|dkr|d}||rdnd} nt|dkrt|dkr|d|d}|d} |d} |d} ||r,dnd} |d} nHt|dkr|t|dksDJ|d|dkrV|d|dks`t d|d |d}d}|d} |d}|d|d} | } |d} | } t |j}t |||gt|t|t|t| t|t| t|t|t|t| t| t|  |S) NsizerKr@r:rrr FTzMOnly bsi,bso->io supported for tensor contractions, but dims for A x B were: r)rr9rrr@rrL batched_igemmstrider r6r*r?rr cigemmr=rsrerc)rarur_rrrrrrldbmrldaldcptrrrrigemms(   $rcCst|jdkrt|jdkstd|jd|jt|||||}|dur0tj|tj|jd}|r=| d}d}nV| }|d|jdkrU| }| d}n>|d|jdkrgd }| d}n,|ddkrx| }| d}n|ddkr| }| d}n | }| d}|r| d}d}n8| }|d|jdkr| }| d}d}n|d|jdkr| d}d }n | }| d}d}|jd} |jd} |jd} |jd} | } |jd|jd}|jd|jd}|jd|jd}t |j}t|||gt|t|t|t| t| t| t|t|t|t|t|t| t|t|t|t| |S) Nr:z@Expected 3-dimensional tensors for bmm, but got shapes A and B: rrr FrrT)rrLr rr9rrr@ is_contiguousr contiguousr6r*r?rr cbatched_igemmr=rsrercc_longc_uint32)rarur_rrrrsr num_batchrrrrstrideAstrideBstrideCrrrrr+s           rcCs2|durtjjj||||Stjjj||S)aLPerforms an 8-bit integer matrix multiplication. A linear transformation is applied such that `out = A @ B.T`. When possible, integer tensor core hardware is utilized to accelerate the operation. Args: A (`torch.Tensor`): The first matrix operand with the data type `torch.int8`. B (`torch.Tensor`): The second matrix operand with the data type `torch.int8`. out (`torch.Tensor`, *optional*): A pre-allocated tensor used to store the result. dtype (`torch.dtype`, *optional*): The expected data type of the output. Defaults to `torch.int32`. Raises: `NotImplementedError`: The operation is not supported in the current environment. `RuntimeError`: Raised when the cannot be completed for any other reason. Returns: `torch.Tensor`: The result of the operation. N)r9r+r,int8_linear_matmulr_r.)rarur_rKrrrrsr row_stats col_statsrcCs2tjjjj|||tj|d}|dur||S|S)aPerforms dequantization on the result of a quantized int8 matrix multiplication. Args: A (`torch.Tensor` with dtype `torch.int32`): The result of a quantized int8 matrix multiplication. row_stats (`torch.Tensor`): The row-wise quantization statistics for the lhs operand of the matrix multiplication. col_stats (`torch.Tensor`): The column-wise quantization statistics for the rhs operand of the matrix multiplication. out (`torch.Tensor`, *optional*): A pre-allocated tensor to store the output of the operation. bias (`torch.Tensor`, *optional*): An optional bias vector to add to the result. Returns: `torch.Tensor`: The dequantized result with an optional bias, with dtype `torch.float16`. )rKrN)r9r+r,int8_mm_dequantr.rr0)rarrr_rresultrrrrs r nnz_block_ptrcCs|sJd}|dus|dur?|d|jd}|dkr)||k}||d|dur2t||}|dur?|jddd}|||fS)a "Determine the quantization statistics for input matrix `A` in accordance to the `LLM.int8()` algorithm. The row-wise and column-wise absmax values are determined. For more information, see the [LLM.int8() paper](https://arxiv.org/abs/2208.07339). This function is useful for training, but for inference it is advised to use [`get_row_absmax`] instead. The column-wise quantization scales are not typically needed in inference scenarios. Args: A (`torch.Tensor` with dtype `torch.float16`): Input tensor. row_stats (`torch.Tensor`, *optional*): If provided, calculation of row statistics is skipped. col_stats (`torch.Tensor`, *optional*): If provided, calculation of column statistics is skipped. nnz_block_ptr (`torch.Tensor`, *optional*): Not used. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `Tuple[torch.Tensor, torch.Tensor, Optional[torch.Tensor]]`: A tuple containing quantization statistics. - `torch.Tensor` with dtype `torch.float32`: The row-wise quantization statistics. - `torch.Tensor` with dtype `torch.float32`: The column-wise quantization statistics. - `torch.Tensor` with dtype `torch.bool`, *optional*: A mask indicating the locations of outliers in the input tensor. Nr,rrFdimkeepdim)is_floating_pointrrXrL masked_fill_get_row_absmaxamaxr)rarrr threshold outlier_maskabsArrrget_colrow_absmaxs "   rc Cs|jtjksJt|jdd}|jd}tj|ftj|jd}t|gt |"t t |t |t |t |t |t|Wd|S1sOwY|S)aTDetermine the quantization statistics for input matrix `A` in accordance to the `LLM.int8()` algorithm. For more information, see the [LLM.int8() paper](https://arxiv.org/abs/2208.07339). Args: A (`torch.Tensor` with dtype `torch.float16`): The input matrix. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `torch.Tensor` with dtype `torch.float32`: The absolute maximum value for each row, with outliers ignored. Nr,rJ)rKr9rrrLemptyrkr@rrGr cget_row_statsrcr=rmrer)rarrowscolsrrrrrs$      rc @s4eZdZdedededejdejdejf ddZd S) COOSparseTensorrrnnzrowidxcolidxrcCs|jtjksJ|jtjksJ|jtjksJ||ks J||ks(J||ks0J||_||_||_||_||_ ||_ dSr ) rKr9rrrrrrrrrr)rrrrrrrrrrr"s zCOOSparseTensor.__init__N)r2r3r4rr9rrrrrrr!src@eZdZddZdS)CSRSparseTensorcCs|jtjksJ|jtjksJ|jtjksJ||ks J||ks(J||dks2J||_||_||_||_||_ ||_ dSNr ) rKr9rrrrrrrrowptrrr)rrrrrrrrrrr5 zCSRSparseTensor.__init__Nr2r3r4rrrrrr4 rc@r)CSCSparseTensorcCs|jtjksJ|jtjksJ|jtjksJ||ks J||ks(J||dks2J||_||_||_||_||_ ||_ dSr) rKr9rrrrrrrcolptrrr)rrrrrrrrrrrFrzCSCSparseTensor.__init__NrrrrrrErrcCsztj|jdd\}}|dtj|jdftj|jjd}|j| | dd| dt |j|j |j||j|jSNT return_countsr rJr)r8srcr)r9uniqueradd_rrrr@scatter_rrcumsum_rrrrr)cooArcountsrrrrcoo2csrVs   rcCst|j\}}|j|}|j|}tj|dd\}}|dtj|jdftj |jj d}|j | | dd|dt|j|j|j|||Sr)r9rrrrrrrrrr@rrrrrrr)rr col2rowidxrr colvaluesrrrrrcoo2csc_s    rcCsLtj|ftj|d}tj|ftj|d}tj|f||d}t||||||S)NrJ)r9rrr)rrrr@rKrrrrrr coo_zerosksrout_colout_rowcCsT|durtd|durtd|durtd|dur tdtjjjj||dS)aLDetermine the quantization statistics for input matrix `A` in accordance to the `LLM.int8()` algorithm. The statistics are determined both row-wise and column-wise (transposed). For more information, see the [LLM.int8() paper](https://arxiv.org/abs/2208.07339). This function is useful for training, but for inference it is advised to use [`int8_vectorwise_quant`] instead. This implementation performs additional column-wise transposed calculations which are not optimized. Args: A (`torch.Tensor` with dtype `torch.float16`): The input matrix. col_stats (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the column-wise quantization scales. row_stats (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the row-wise quantization scales. out_col (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the column-wise quantized data. out_row (`torch.Tensor`, *optional*): A pre-allocated tensor to hold the row-wise quantized data. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, Optional[torch.Tensor]]`: A tuple containing the quantized tensor and relevant statistics. - `torch.Tensor` with dtype `torch.int8`: The row-wise quantized data. - `torch.Tensor` with dtype `torch.int8`: The column-wise quantized data. - `torch.Tensor` with dtype `torch.float32`: The row-wise quantization scales. - `torch.Tensor` with dtype `torch.float32`: The column-wise quantization scales. - `torch.Tensor` with dtype `torch.int32`, *optional*: A list of column indices which contain outlier features. NzUrow_stats must be None. int8_double_quant() does not support pre-allocated row_stats.zUcol_stats must be None. int8_double_quant() does not support pre-allocated col_stats.zQout_col must be None. int8_double_quant() does not support pre-allocated out_col.zQout_row must be None. int8_double_quant() does not support pre-allocated out_row.)r)r r9r+r,int8_double_quantr.)rarrrrrrrrrrs'rstatscCtjjj||S)aYDequantizes a tensor with dtype `torch.int8` to `torch.float32`. Args: A (`torch.Tensor` with dtype `torch.int8`): The quantized int8 tensor. stats (`torch.Tensor` with dtype `torch.float32`): The row-wise quantization statistics. Returns: `torch.Tensor` with dtype `torch.float32`: The dequantized tensor. )r9r+r,int8_vectorwise_dequantr.)rarrrrrs rcCr)awQuantizes a tensor with dtype `torch.float16` to `torch.int8` in accordance to the `LLM.int8()` algorithm. For more information, see the [LLM.int8() paper](https://arxiv.org/abs/2208.07339). Args: A (`torch.Tensor` with dtype `torch.float16`): The input tensor. threshold (`float`, *optional*): An optional threshold for sparse decomposition of outlier features. No outliers are held back when 0.0. Defaults to 0.0. Returns: `Tuple[torch.Tensor, torch.Tensor, Optional[torch.Tensor]]`: A tuple containing the quantized tensor and relevant statistics. - `torch.Tensor` with dtype `torch.int8`: The quantized data. - `torch.Tensor` with dtype `torch.float32`: The quantization scales. - `torch.Tensor` with dtype `torch.int32`, *optional*: A list of column indices which contain outlier features. )r9r+r,int8_vectorwise_quantr.)rarrrrrsrrcCst|ts2|jr|jtjksJdt|jd|jd||d |d | d}|durFtj |j |jdf|j |jd}|j}|j|ksRJ|j|ks[J|j |ksdJ|j|jdksnJ|rtdnd}||r}dnd}|jd}tj}t|j}t|j} t|j } t|} t|} t|j} t|j }t|j}t|jd}t|}t|}t|j|j|j ||gt||| | | ||||| || t| |S)Nz8Tensor must be `COOSparseTensor or a PyTorch COO tensor.rr )rrrrrrr@rKFT) rr is_sparselayoutr9 sparse_coorL_nnzindicesrrrrr@rKrrrrrrrrrBr*r7rcr=rerr cspmm_coors)rrur_rrrrr ptrRowidx ptrColidx ptrValuesptrBptrCcnnzcrowsAccolsAccolsBcldbcldcrrrspmm_coosf            rcCs|durtj|j|jdf|j|jjd}|j}|j |ks!J|j |ks*J|j |ks3J|j |jdksFJ|j d|j| rLdnd}| |rUdnd}|jd}tj|jdd\}} | d} tj| dd\} } | } | } | dd ksJd | dd |jtjtjfvsJt| } t| }t| }t|j}t|j }t|j}t|}t|}t|}t| }t|j}t|j}t|j }t|jd}t|jd}t|}t|}t|Yt|j|j |j|||g|jtjkr t||| |||||||||||n)|jtjkrAt||| |||||||||||Wd|SWd|SWd|S1sUwY|S) Nr rrz vs FTr) descending z)Current max count per row is 8 but found r)r9rrrLr@rrKrrrrrrrrrcumsumrrrint8rcr=rerGrr cspmm_coo_very_sparse_naive_fp16 cspmm_coo_very_sparse_naive_int8)rru dequant_statsr_rrrrrrr max_countmax_idx ptrOffset ptrMaxCount ptrMaxIdxrrrrrptrDequantStats cnnz_rowsrrrcrowsBrrrrrrspmm_coo_very_sparse s"&            & & &&rg_@ztThis function is deprecated and will be removed in a future release. Consider using `int8_vectorwise_quant` instead.vectorc Cs$|dkrt|}t||dtj}||fS|dvr>tjt||dd}t|t|tj}||fS|dkru|j }|}|| }|dkrWd}d |}| }t||} t||| | }||fS|d vr|j }|}tj||ddtj ||dd}d||dk<d |}tj ||dd}t||} t||| | }||fS|d krt Bt|} tj| |dd}|d }| | | k} t|| } | | | | || <t||ttj}Wd||fS1swY||fSdS) Nlinear)rrowTr zeropointrr go@)vector-zeropoint row-zeropointtruncated-vectorgffffff?)r9rrrrrr rCrKminaminno_grad expand_asr) rrrrzxqrKdynaqxminxzpxabsxrrrrrvectorwise_quant` sV      r,cCs|dkr||tt}|||S|dkr(d||}|||S|dkrqd||}|}t|jdkrIt|jdkrI|d}t|jdkr\t|jdkr\|d}t|jdkrh||9}n||9}||S|dkr|}t|jdkrt|jdkr|d}t|jdkrt|jdkr|d}t|jdkr|d|9}n|d|9}|d|9}||S|d kr|}t|jdkrt|jdkr|d}t|jdkrt|jdkr|d}t|jdkr|||tt9}n |||tt9}||S|d vrd|}t|jdkr/t|jdkr/|d}t|jdkrDt|jdkrD|d}t|jdkrS||t9}n||t9}||t9}||SdS) Nrrrrr:rrrr)r r)r!rrrrLsqueezer/)r&S1S2rKrrrrrrvectorwise_mm_dequant sd                      r0r1)T)NTr )TrT)rT)Trrr)Trr)r)Nrr)NNNr&F)NNNNr&F)Nr6)NNNr6)NNNr6rr)NNNN) NrrrrrNrF)rrNr)rrF)r)NFFN)NFF)NNNr)r)NNNNr)r r)collections.abcrctypesr=rmathrtypingrrrnumpyrTr9rtyping_extensionsrbitsandbytes.utilsr r cextensionr r*cadam32bit_grad_fp32cadam32bit_grad_fp16cadam32bit_grad_bf16cmomentum32bit_grad_32cmomentum32bit_grad_16crmsprop32bit_grad_32crmsprop32bit_grad_16clion32bit_grad_fp32clion32bit_grad_fp16clion32bit_grad_bf16cadagrad32bit_grad_32cadagrad32bit_grad_16cademamix32bit_grad_fp32cademamix32bit_grad_fp16cademamix32bit_grad_bf16rqcadam_static_8bit_grad_32cadam_static_8bit_grad_16cmomentum_static_8bit_grad_32cmomentum_static_8bit_grad_16crmsprop_static_8bit_grad_32crmsprop_static_8bit_grad_16clion_static_8bit_grad_32clion_static_8bit_grad_16r~cadam_8bit_blockwise_grad_fp32cadam_8bit_blockwise_grad_fp16cadam_8bit_blockwise_grad_bf16"cmomentum_8bit_blockwise_grad_fp32"cmomentum_8bit_blockwise_grad_fp16"cmomentum_8bit_blockwise_grad_bf16!crmsprop_8bit_blockwise_grad_fp32!crmsprop_8bit_blockwise_grad_fp16!crmsprop_8bit_blockwise_grad_bf16clion_8bit_blockwise_grad_fp32clion_8bit_blockwise_grad_fp16clion_8bit_blockwise_grad_bf16!cadagrad_8bit_blockwise_grad_fp32!cadagrad_8bit_blockwise_grad_fp16!cadagrad_8bit_blockwise_grad_bf16"cademamix_8bit_blockwise_grad_fp32"cademamix_8bit_blockwise_grad_fp16"cademamix_8bit_blockwise_grad_bf16rrr6rBr@FIRST_CUDA_DEVICEr: device_countrGrHrkr`r-rzr{r~rrrr FutureWarningrrr>rrcrrrrr-rr5rDrnrLrOrHrVrXrTr[r]rYr\rrwrrrrr rrrrrrrrrrrrrrhalfrrrrrrr!r,r0rrrrs*     %"      + 7 $  C4  I MT      O     I  $ $(      g            @ )W ( h (`  9%    3  CR ,