o ٷiǍ @s dZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl Z ddl mZddlmZedZ d*d d Zd+ddZd,ddZd,ddZd+ddZddZid d d iddgdfddZddZd d!Z " # $ $ %d-d&d'Zid d d d"d#d"d$d$d%ddgf d(d)ZdS).zWeightOnly for onnxrt adaptor.N) numpy_helper)np_dtype_to_tensor_dtype) ONNXModel)simple_progress_barneural_compressorc Cs.||d} tj|jd| fdd} |jdd|d|} |jd| g} g} i}d}|d kr[|d d d d d f|d d dd d fd >B}|d d d | f| d d d d f<n|dkrb|} n td |d t| d|| f} t|d|f}|jtjks|jtj ksJt j j |jddt |j|j|dd}| |j| ||d urB|dkr|d}n`|d krtj|jddd ddd}t|jd||d}|d d d }|dd d }||d d@||B||d <||d d@||d >B||d <ntd |d t||ddf}t j j |jddd |j|dd}| |j| ||d|d<|d|d<||d<||d<|dkr_||d<t j j | d | j| dd}| |t j j|f| |j|jr|jdt|ndt|dd|}|| fS)aBBuild MatMulNBits node. Args: node: original matmul node weight_shape: original weight shape num_bits (int): num_bits group_size (int): how many elements share one scale/zp k_blocks (int): block number q_weight (array): quantized weight scale (array): scale zero_point (array): zero point accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). Returns: matmul_weight_only_node: MatMulNBits node new_inits: initializers of the new node ruint8dtyper_QG MatMulNBitsNz8MatMulNBits does not have kernel support for num_bits = ._scaleTname data_typedimsvalsraw_zpKNbits block_sizeaccuracy_levelz com.microsoft)inputsoutputsrdomain)npzerosshapeinputloggererrorreshaper float32float16onnxhelper make_tensorrtobytesappendrastypefullarangeravel ValueError make_nodeoutputstr)node weight_shapenum_bits group_sizek_blocksq_weightscale zero_pointr" blob_sizepacked q_weight_name input_names new_initskwargsop_typeq_weight_pairs scale_tensor packed_zpidxeven_idxodd_idx zp_tensorq_weight_tensormatmul_weight_only_noderTj/home/ubuntu/.local/lib/python3.10/site-packages/onnxruntime/quantization/neural_compressor/weight_only.pymake_matmul_weight_only_node,s 0&      $*      $rVr asymint?c st|d|f}|dks|dkrd|ddn|dkr8|dkr)d|ddnd|dkr6d|d ndtj|ddd |}tj|ddd |}|dkrtt|t|}t|j} |dk} || d tj | | <|d krt | jn tj|jd d d|d>} nHt|j} t fdd||||k D| ||k<|d krt | j|| ntdtt | j|| d } tj|| jd } tj|| | dtj| | | dtj| | dtj| | d| | | fS)a Quantize tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quantized weight scale: scale zero_point: zero point rrXuintrrrsymTaxiskeepdimsg@rYr r csg|] }t|qSrT)float.0imaxqminqrTrU z quant_tensor..out)r&r,minmaxmaximumabsonesr(r4float64r'arrayflattentolistroundminimum empty_liker divideaddclip) datar>r?schemer ratiorminrmax max_rangerBmaskrCrArTrdrU quant_tensors<   . & , rc&Cs@t|d|ftj}d|d}d}tj|dddd}t||}t|t|}tj|ddd}tj |ddd} tj|ddd} tj||ddd} tj | j |j d} || k} ||| | || | | <d| }t t| ||||}||||}tj||dddd}d}d }d}t|D]}tj | j |j d}t|||||g|j d}|| k} || | || || <t t|||||}||}tj|ddd}tj||ddd}tj||ddd}t| ||d}| || ||}|| |||}||||}tj||dddd} t| }!t|}"t|!|"kd}#||#d d f||#d d f<| |#||#<||#||#<||#||#<qt | |d|d }$|tj}tj||j d}%tj|||%d tj|%|$|%d tj|%|%d tj |%|||%d |%||$fS) aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 32. Returns: output: quantized weight scale: scale zero_point: zero point rrrrTr]r 皙?Nr ri)r&r,r4r-sumsqrtrxrnrkrlror(r ryrtrangerqsubtractwhererprvrw)&rzr>r?rerfsum_x2av_xweightsr}r~sum_wsum_xiscalerrB quant_datadiffbest_madnsteprdeltarrminis_ iscale_newfactorquant_data_newmul_weights_quant_data_newsum_lsum_l2sum_xlD this_scalethis_minmadmad_1 best_mad_1idx_to_replacerCrArTrTrUquant_tensor_k_quant_cpusd  (      rc(Cszddl}ddl}|jr||}|d|f|j}d|d}d}|j|dddd}| ||}| || |} |j |ddd} |j |ddd} |j| ddd} |j| |ddd} |j| j|jd}| | k}||| || |||<d|}||||| ||}||| |}|j| |dddd}d }d }d}t|D]}|j| j|jd}||||||g|jd}| | k}|| || |||<||||| ||}| |}|j|ddd}|j||ddd}|j||ddd}|| ||d}| || ||} || |||}!| ||!|}|j| |dddd}"||"}#||}$||#|$kd}%||%ddf||%ddf<|"|%||%<| |%||%<|!|%| |%<q|| |d|d }&||j}|j||jd}'|j|||'d |j |'|&|'d |j|'|'d |j|'|||'d |'||&fWStd t|||WStytdt|||YSw)aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. Returns: output: quantized weight scale: scale zero_point: zero point rNrrrTr]r rrr rizqTry to use k-quant quantization on CUDA. However, CUDA is not available.Fall back to k-quant quantization on CPU.zNow we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.) cupytorchcuda is_availableasarrayr,r4r-rrrxrnrkrlror(r ryrtrrqrrrprvrwgetr*warningr ImportErrorinfo)(rzr>r?cprrerfrrrr}r~rrrrrBrrrrrrrrrrrrrrrrrrrrrrCrArTrTrUquant_tensor_k_quant_cudas    (     rc Cs2|j}t||||||\}}} t||| |S)aQuant dequant tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quant-dequant weight )r(rr&r,) rzr>r?r{r r| org_shapeweightrBzprTrTrU qdq_tensorysrcCsH|dkr|S|j}||}||d}|dkr"t|d|fdfd}|S)aPad tensor rowi so that it can be is divisible by group_size. Args: weight (array): weight group_size (int): how many elements share one scale/zp k_blocks (int): the number of block Returns: weight: paded weight rr)rrconstant)r(r&pad)rr?r@ org_w_shape padded_rowspad_lenrTrTrU pad_tensors  rCPUExecutionProviderk_quantc  Cst|}|jdurtj|jnd} g} g} tdd|D} d} |D]B}|jdvr8| d7} t| | |jdvrj| |j ddurj| |j idkrj| |j d}t j|| d }t|jd krlq'|j}|j |vr||j d }||j d }||j d }|j}|dkr|n|d}|dd|d}||j d}t|||}|dkp|dk}|r |dkrt|j||\}}}nt|j|||d| |j dd\}}}t||||||d|||dks|dkr|nd|d \}}||| || |nUt|j|||d| |j dd}t||ddf}t|}|d|dddf|}tj j!|j dd|d|t"||j|#dd}|$||j |j d<|dkrj|%|q'|&| |'| |(|S)aQuant the model with round to nearst method. Args: model (ModelProto or ONNXModel): onnx model weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'RTN' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". ratios (dict, optional): percentile of clip. Defaults to {}. accuracy_level (int): accuracy level. Support 0 (unset),1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel NcSsg|] }|jdvr|qS)MatMul)rJrarTrTrUrgsz rtn_quantize..rrrfp32)base_dirrr r?r{rrrrr[r rX r<r=r>r?r@rArBrCr"rYr r Tr))r model_pathospathdirnamelennodesrJrget_initializerr)rrrto_arraycopyr(r get_initializer_share_numrrTrrVr4add_initializersr3rr&r, transposer/r0r1rr2add_initializerremove_initializer add_nodes remove_nodestopological_sort)model weight_configr>r?r{ratiosr" providers algorithmr new_nodesr total_numcurr_idr< weight_tensorrr rr@init_share_numsatisfy_MatMulNBits_conditionrArBr q_matmul_noderHrRrTrTrU rtn_quantizes$        "       rc CsX|j}|dkrt|d|fn|}tjtt|tjt|ddd|dd}|S)zGet the scale of weight.rrTr]rr^)r(r&r,meanrnrl)rr?rrBrTrTrUget_weight_scales2rc sddlm}ddlmt}tjdkr%|dr%ddlm }| ||j r6t j |j|jdd d d d |j sDtj|j||d n tj|jd||d }d d|D}~g} t|D]\} } |dkru| d|j|kru| |fSt|dkst| dtrt| dt|ksJdt|dt| dt| dtr| tfdd| dDq_t| dtjr| tddt|| dgd dDq_| tfddt|| dd dDq_| |fS)asPrepare inputs for weight only quantization. Args: model (ModelProto or ONNXModel): onnx model n_samples (int, optional): calibration sample number. -1 means all samples. dataloader (object): dataloader for calibration. providers (list): providers to use Returns: inputs: prepared inputs. so: session options r) find_specrto_numpy) onnxruntime_extensions)get_library_path _augment.onnxTFsave_as_external_dataall_tensors_to_one_fileconvert_attributercSg|]}|jqSrTrrarTrTrUrgGz"prepare_inputs..rzInput number mismatch, require z but get cg|] \}}||fqSrTrT)rbrinp_datarrTrUrgTrhcSsg|]\}}||fqSrTrTrbrinprTrTrUrgVsstrictcrrTrTrrrTrUrgXrh)importlib.utilrutilrortSessionOptionssys version_inforrregister_custom_ops_libraryis_large_modelr/ save_modelrrInferenceSessionSerializeToString get_inputs enumerate batch_sizer isinstancedictr3itemsr&ndarrayzip) r n_samples dataloaderrrsorsession inputs_namesr#rcrzrTrrUprepare_inputs#sF     &*,r{Gz?FTc # sZd|ddddfdd} |j} | |\} } t|dk}d|||f<d||d d f<|r[tt|d d d }||d d f}||d d fd d |f}t|}t|}|tt|}t| d}|||f|7<tjtj |j }|}t d| d|D]}t ||| d}||}t |||d d f}t|}t|}t|}|||||f}t |D]}||d d f}|||f}|d kr|||dkr| ||||||d d f\} } | tt|d d tjf| | d| } | ||d d f<|| d|d||d d f<|| |}!||d d d fttj||d |fdd tj|!dd 8<|!||d d f<q||||d d f<|d|||d d f<||d d d ft||d ||f|8<q|rt|}"||"d d f}t||j}~|S) aQuant the weight with GPTQ method. Args: W (array): weight. H (array): Hessian matrix. num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". blocksize (int, optional): blocksize to quantize weight. percdamp (float, optional): percent of the average Hessian diagonal to use for dampening. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. Returns: Q: fake quantized weight rrdg?g333333@cs>|j}stj|dd}t|jd}ttj|dd|}ttj|dd|}dkrItt ||}|dk}t |rI|| ||<|dk|dk@}d||<d||<||}dkrpt |jdd}nt | |}rt |jdgt d}ttD]`}d|} | |} | |} | | } dkrt | | n|} tt || | d}||8}tt |}t|d}||k}t |r||||<| |||<| |||<qs|d}t||}t||}dgdgt|d}t||}t||}||fS)Nrrrr\rrinf)r(r& expand_dimsrrr'rurkrmrlrnanyrortr`rrYrypowerrrepeatrr,)rrtmpxminxmaxrBzerobestrcpxmin1xmax1scale1zero1qerrr(gridre maxshrinkmsenorm perchannelr{rTrU find_params~sX             zgptq..find_paramsrNrr)r(r&diagargsort zeros_likerr6linalgcholeskyinvrrrkrdeepcopyryrtnewaxisrrmatmulrr,)#WHr>r?r{ blocksizepercdampactorderr*r,r-r(rBrdeadpermLossesQdampr.Hinvi1i2countW1Q1Err1Losses1Hinv1rcwdr%err1invpermrTr'rUgptq\sd 0         (6  D6 rNc*st|}|jdurtj|jnd}t|||| \}}~t|jj j }| dd|Dg}| D]%}|j dvrY||jidkrY||jidddkrY||jd q4tt|}|||jrvtj|j|jd d d d d |jstj|j|| dn tj|jd || d}t|D]\}}tt||dg}g}|j|D]O}|j dvr||jidkr||jidddkr||jddurt !||"|jjd|}t|j#dkrq||||"|jqt|d krqdd|D}d |D]<}|$|g|d j#d t%&dj#dffdd|D}7t%'dfdd|D}q t(|||d dD]\}}}|j|vrr||jd}||jd}||jd}|dkry|n|j#d }|j)}t*|||||||| | | d }||jd}|+|jd} |dk}!|!r|j#}"|"d |d|}#t,|||#}t-|j.|||d\}}$}%t/||"|||#|0d|$0||dkr|%nd| d \}&}'|1|'|2||3|&n)tj4j5|jdd|d |t6||j#|0|7d d!}(|8|(|(j|jd<| dkr$|9|qRq| ||jj j :||;|jrOd d"l|jd |S)#aQuant the model with GPTQ method. Args: model (ModelProto or ONNXModel): onnx model dataloader (object): dataloader for calibration. weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'GPTQ' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". n_samples (int, optional): calibration sample number. percdamp (float, optional): percent of the average Hessian diagonal to use for dampening. blocksize (int, optional): blocksize to quantize weight. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel NrcSrrTrrarTrTrUrgrz!gptq_quantize..rrrGPTQrrTFrrrrcSs&g|]}t|jd|jdfqSr)r&r'r(rarTrTrUrgOs&rcsg|] }|qSrTrTra)nsamplesrrTrUrgUrhcsg|] }|tjqSrT)r&r6rra)rrTrUrgXsrr r?r{)r>r?r{r9r:r;r*r,rr[r rXrr r r)load_external_data_for_model)?rrrrrrrr4rgraphr:remove_tensors_from_outputsrrJrrr3r)listsetadd_tensors_to_outputsrr/rrrrrrrinput_name_to_nodesrrrget_noder(runr&r,rr r rNrrrrrVr4r remove_nodeadd_noder0r1rr2rr MergeFromronnx.external_data_helperrRsplit)*rrrr>r?r{r r:r9r;r*r,r"rrr#r org_output output_namesr<rrN input_name node_listrrHsrzr8r rArrrrr@rBrrrHrRrRrT)rrQrrU gptq_quantizes/                   ? rerP)rrWrXrYrZ)rrW)rrWrXrrFFT)__doc__rloggingrrnumpyr&r/r onnx.helperr onnxruntimer onnx_modelrrr getLoggerr*rVrrrrrrrrrNrerTrTrTrUsl      s 6 I [ w<