o 3wi@sddlZddlZddlZddlZddZGdddZddd Zd ed eeeffd d Z   dddZ ddZ ddZ dddddZ dde DZdS)Nc CsFt|tjjs Jt}||j}||jvrt |j}|j ||j |t |j dkrR|j ddkrH|j d|jjdksHJ|j d|j|<dS|dd|djd}t |ddd}t|ddkjttt |djdd}t|dkd}t||g}||j|<dS|jD]}|qdS)Nr) reduction_dimzscore)dim) isinstancetorchnnLinear OutlierTracer get_instance get_hvalueweighthvalue2outlier_idxfind_outlier_dimsoutliersappendhvalueslennumelmaxshapeviewabssumlistrangewherecatuniquehooksremove) moduleinputtracerhvalue outlier_idxmergeddims outlier_idx2hookr-O/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/bitsandbytes/utils.py outlier_hooks(     0  r/c@sDeZdZdZddZddZddZdd Zd d Ze d d Z dS)r NcCstd)NzCall get_instance() instead) RuntimeErrorselfr-r-r.__init__,szOutlierTracer.__init__cCs`d|_d|_g|_g|_i|_d|_g|_|D]\}}t|t j j r-|j | tqdS)NT)last_wcurrent_outlier_dimsrrr initializedr" named_modulesr r r r rregister_forward_pre_hookr/)r2modelnmr-r-r. initialize/szOutlierTracer.initializecCs t|ddS)Nr6F)getattrr1r-r-r.is_initialized<s zOutlierTracer.is_initializedcCs|jSN)datastoragedata_ptr)r2rr-r-r.r?szOutlierTracer.get_hvaluecCs6|s tddS||}||jvr|j|SdS)Nz$Outlier tracer is not initialized...)r>printrr)r2rr'r-r-r. get_outliersBs   zOutlierTracer.get_outlierscCs|jdur |||_|jSr?) _instance__new__)clsr-r-r.rLs  zOutlierTracer.get_instance) __name__ __module__ __qualname__rEr3r<r>rrD classmethodrr-r-r-r.r )s  r @FcCs|rtjd|jd|f|jdS||}|}|}|||}||} | } | } | | | } |durJtj| |dd\} }|St | |kd}|S)Nrr)sizedevice)kr) r randintrrNlongmeanstdtopkrr)rrrrTrdmr;mmmstdzmrSstdmstdstdzstdvalidxr-r-r.rSs     rcommand_stringreturncs(ddfdd}||\}}||fS)NcSstdd|DS)Ncss|] }|dVqdS)zUTF-8N)decodestrip).0 to_decoder-r-r. lsz6execute_and_return.._decode..)tuple)subprocess_err_out_tupler-r-r._decodeksz#execute_and_return.._decodecs"tjt|tjtjdS)N)stdoutstderr) subprocessPopenshlexsplitPIPE communicate)r^rgr-r.&execute_and_return_decoded_std_streamsnszBexecute_and_return..execute_and_return_decoded_std_streamsr-)r^rqstd_outstd_errr-rpr.execute_and_returnjs  rtlm_headc Cs|D]V\}}tt|dkrt|||||t|tjjrZ||vrZ|j |}||j |j |j du|j |<|rH|j |j |_ |j |j |_ |durZt||d}|durZ||q|S)a Replace linear modules with a new Linear module. Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. linear_replacement (`torch.nn.Module`): The linear module that replaces the old one. Only expects standard arguments. If other arguments need to be passed, use a lambda. skip_modules (`List[str]`, *optional*, defaults to `lm_head`): List of modules names not to convert. Defaults to `lm_head`. copy_weights (`bool`): Copy the weights from the old linear module to the new one post_processing_function (`str`): A function name of the replacement linear class that is called after processing. rN)named_childrenrrchildrenreplace_linearr r r r _modules in_features out_featuresbiasrr=) r9linear_replacement skip_modules copy_weightspost_processing_functionnamer$ old_modulefuncr-r-r.ry{s&   rycCs,t|}|d}tjt|tjd}|S)z Pack a dictionary into a torch tensor for storing quant_state items in state_dict. Parameters: - source_dict: The dictionary to be packed. Returns: A torch tensor containing the packed data. utf-8)dtype)jsondumpsencoder tensorruint8) source_dictjson_str json_bytes tensor_datar-r-r.pack_dict_to_tensors rcCs(t|}|d}t|}|S)z Unpack a torch tensor into a Python dictionary. Parameters: - tensor_data: The torch tensor containing the packed data. Returns: A Python dictionary containing the unpacked data. r)bytescpunumpyr`rloads)rrr unpacked_dictr-r-r.unpack_tensor_to_dicts  rrr)rowcol32 col_turing col_amperecCsi|]\}}||qSr-r-)rbrr\r-r-r. sr)rrLNF)ruFN)rrlrjr r/r rstrrertryrr"LINEAR_8BIT_WEIGHTS_FORMAT_MAPPINGitems*INVERSE_LINEAR_8BIT_WEIGHTS_FORMAT_MAPPINGr-r-r-r.s ! * -