o Ti@sddlZddlZddlmZddlmmZddlmZddl m Z ddl m Z m Z ddlmZGdddejZGd d d ejZdS) N)Optional)get_accelerator) Quantizer FP_Quantize)QuantizationConfigc seZdZdZ    ddeejdedede fdd Z d ejfd d Z d ejfddZ dddZ ddZddZddZddZd ddZfddZZS)!QuantizedParametera Quantized parameter class that implements weight quantization. Weights are stored in quantized form on GPUs, and can be dequantized on-the-fly when needed by the model. The weights are actually quantized during any `.to(device)`. Arguments: data (Tensor): parameter tensor. requires_grad (bool, optional): if the parameter requires gradient. Defaults to False and is not supported to be True. Argument provided only for interface compatibility with torch.nn.Parameter. quantization_config (QuantizationConfig, optional): quantizer (Quantizer, optional): Defaults to FP_Quantize but can be any quantizer that implements deepspeed.ops.fp_quantizer.Quantizer. This argument is also required since the quantizer is stashed in the Parameter itself, some models may clone the Parameter by passing an attribute __dict__. For an example, see tests/unit/linear/test_quant_param.py::TestQuantParam::test_hf_clone NFdata requires_gradquantization_config quantizercCsn|rtd|durtd}tj|||}|durtn||_|dur)||_nt|jd|_| ||S)Nz;requires_grad=True is not supported with QuantizedParameterrr ) ValueErrortorchemptyTensor_make_subclassrr r r_ensure_quantized)clsr r r r selfrQ/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/linear/quantization.py__new__%s  zQuantizedParameter.__new__tensorcCst|r@|j|jjkrBtt|j|jj |j |jj |jj d|_ Wdn1s2wY|j|jjksDJdSdSdS)Nq_bitsq_mantisa_bits) ron_acceleratordtyper q_dtypestreamcurrent_streamdevicer quantizer r mantissa_bits)rrrrrr:s  z$QuantizedParameter._ensure_quantizedreturncCsxt|jr9|jj|jjkr9tt|jj|j j |j|jj |jj dWdS1s4wY|jS)zW Return a tensor containing the dequantized weights of this parameter. rN) rrr rr rr r!r"r dequantizerr$)rrrr dequantizedCs  zQuantizedParameter.dequantizedcCs:t|ddr|r|t|_dS|d|_dSdS)N ds_offloadFcpu)getattrtorcurrent_device_namer )rrevertrrroffloadNs zQuantizedParameter.offloadcCs(|j}|j|d<|j|d<|j|d<|S)Nr r r )__dict__r r r rstaterrr __getstate__Us    zQuantizedParameter.__getstate__cCs,|d|_|d|_|d|_|d|_dS)Nr r r r )r r r r r0rrr __setstate__\s   zQuantizedParameter.__setstate__cCsXt|t|}|}||t|d|_t|d|_t|d|_|S)Nr r r ) typerr2r3copydeepcopyr r r )rmemo new_instancer1rrr __deepcopy__bs zQuantizedParameter.__deepcopy__cCs(t|t|}|}|||SN)r4rr2r3)rr8r1rrr__copy__ks zQuantizedParameter.__copy__cCs.|durdn|}|jj||d|j||dS)Ncuda) non_blocking)r r+)rr"r=rrrr<qszQuantizedParameter.cudacs2tj|i|}|jj|i||||S)zt Move the parameter to the given device. Then, if the device is a cuda device, quantize it. )superr+r r)rargskwargsr __class__rrr+vs zQuantizedParameter.to)NFNN)F)NF)__name__ __module__ __qualname____doc__rrrboolrrrrr'r.r2r3r9r;r<r+ __classcell__rrrArrs0   rc sReZdZdZddejfdedededeffdd Z d ej d ej fd d Z Z S)QuantizedLinearz Linear layer that implements weight quantization. Parameters are stored via `QuantizedParameter` and are dequantized on-the-fly during any forward pass. FN input_dim output_dimbiasr cs<tj||||d|tjksJdt|jj|d|_dS)N)rLrz¤tly only supports bfloat16 dtyper )r>__init__rbfloat16rweightr )rrJrKrLr rrArrrMszQuantizedLinear.__init__inputr%cCst||j|jSr:)FlinearrOr'rL)rrPrrrforwardszQuantizedLinear.forward) rCrDrErFrrNintrGrrMrrSrHrrrArrIs  rI)r5rtorch.nnnntorch.nn.functional functionalrQtypingrdeepspeed.acceleratorrdeepspeed.ops.fp_quantizerrrconfigr ParameterrLinearrIrrrrs    o