o Ti@sLddlmZmZddlmZddlZeGdddZeGdddZdS)) dataclassfield)ListNc@steZdZUdZdZeed<dZeed<dZ eed<dZ e ed <d Z eed <dZ e ed <ed ddZeeed<dS) LoRAConfigaN Configuration settings for LoRAOptimizedLinear. Attributes: lora_r (int): LoRA attention dimension, also known as the rank. Defaults is 64. lora_alpha (float): LoRA scaling factor, default is 16. base_weight_sharding (int): The degree to which the base weights are sharded, should typically be set to the data-parallel world size to maximize the memory reduction benefits. Defaults to 1, which means this feature is disabled. offload (bool): offload frozen parameters to cpu when not in use offload_ratio (float): ratio of parameters to offload to cpu when not in use delay_lora_init (bool): initialize lora parameters at time of model init or allow manual init later target_mods (str): target module names to apply LoRA to, defaults to llama-3.1 arch @lora_rg0@ lora_alphabase_weight_shardingFoffloadg offload_ratiodelay_lora_initcCsgdS)N)q_projk_projv_projo_proj gate_projup_proj down_projrrrK/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/linear/config.py#szLoRAConfig.)default_factory target_modsN)__name__ __module__ __qualname____doc__rint__annotations__rfloatr r boolr r rrrstrrrrrr s       rc@sFeZdZUdZdZeed<dZeed<dZeed<e j Z e j ed<d S) QuantizationConfigax Configuration settings for quantization for LoRAOptimizedLinear, QuantizedLinear, and QuantizedParameter Attributes: q_bits (int): The number of bits used for quantization. Default is 8. mantissa_bits (int): The number of bits reserved for the mantissa in fixed-point quantization. Default is 3. group_size (int): The number of elements used for quantization. Default is 512. q_dtype (torch.dtype): The data type to quantize to. Default is uint8. (in CUDA, buffers are allocated as uint8, but inside the kernels the quantization is done to fp8) q_bits mantissa_bitsi group_sizeq_dtypeN) rrrrr%rrr'r(torchuint8r)dtyperrrrr#&s   r#) dataclassesrrtypingrr*rr#rrrrs