o d۷i< @sJddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZddlZddlZddlmZmZedddZerHdd lmZgd Zd Zd ed edfddZded efddZdIded efddZdejj dedejj fddZ!ded efddZ"ded ed efd!d"Z# dJd#eed$eed e egeffd%d&Z$Gd'd(d(Z% dKd)ej&d*e e'd+e'd e ej&fd,d-Z(dKd.d/Z)d0ej&d ej&fd1d2Z*d3ej&d4e e'd ej&fd5d6Z+ej,d7e-d8ed9efd:d;Z.ej,dd?Z/Gd@dAdAe0Z1dBed e'fdCdDZ2dBed e'fdEdFZ3dBed e'fdGdHZ4dS)LN)wraps)MappingProxyType) TYPE_CHECKINGAnyCallableDictIterableListMappingOptionalTypeVar) AutoConfigPretrainedConfigTr)boundModelCompressor)"infer_compressor_from_model_configfix_fsdp_module_nametensor_follows_mask_structurereplace_moduleis_compressed_tensors_config getattr_chain deprecated Aliasablecombine_shards shard_tensor pack_bitmasksunpack_bitmasks patch_attr patch_attrsParameterizedDefaultDictget_num_attn_headsget_num_kv_heads get_head_dim_fsdp_wrapped_modulepretrained_model_name_or_pathreturnrcCsfddlm}ddlm}t|}||}|durdS|d}|j|fi|}|j||d}|S)a Given a path to a model config, extract a sparsity config if it exists and return the associated ModelCompressor :param pretrained_model_name_or_path: path to model config on disk or HF hub :return: matching compressor if config contains a sparsity config rr)CompressionConfigNformatconfig) compressed_tensors.compressorsrcompressed_tensors.configr(r from_pretrainedparse_sparsity_configgetload_from_registry)r&rr(r+sparsity_configr) compressorr4V/home/ubuntu/vllm_env/lib/python3.10/site-packages/compressed_tensors/utils/helpers.pyrCs    rnamecCs|tdddtdS)z Remove FSDP wrapper prefixes from a module name Accounts for scenario where FSDP_WRAPPER_NAME is at the end of the name, as well as in the middle. :param name: name to strip :return: stripped name .)replaceFSDP_WRAPPER_NAME)r6r4r4r5r[sr2:4maskcCsPttt|d\}}|d|}|dkjdd}t||ks&t dS)a, :param tensor: tensor to check :param mask: mask structure to check for, in the format "n:m" :return: True if the tensor follows the mask structure, False otherwise. Note, some weights can incidentally be zero, so we check for atleast n zeros in each chunk of size m :rdimT) tuplemapintsplitviewsumtorchallitem ValueError)tensorr<nm zero_countsr4r4r5rhs  rmodel new_modulecCsTd|vr|ddd}|t|dd}||}nd}|}|}t|||dS)Nr7r?rr8)rsplitlen get_submodulesetattr)rPr6rQ parent_name child_nameparentr4r4r5rs rcompression_configcCs.z ddlm}t||WStyYdSw)z Returns True if CompressedTensorsConfig is available from transformers and compression_config is an instance of CompressedTensorsConfig See: https://github.com/huggingface/transformers/pull/31704 r)CompressedTensorsConfigF)&transformers.utils.quantization_configrZ isinstance ImportError)rYrZr4r4r5rs    robj chain_strc Ost|dkr d}|d}n d|vrd}|d}nd}|d}|}|D]}t||s9|r0|St|d|t||}q#|S)z Chain multiple getattr calls, separated by `.` :param obj: base object whose attributes are being retrieved :param chain_str: attribute names separated by `.` :param default: default value, throw error otherwise r?TrdefaultFr7z object has no attribute )rSrEhasattrAttributeErrorgetattr) r^r_argskwargs has_defaultr` attr_namesres attr_namer4r4r5rs      r future_namemessagecsdtdtffdd }|S)z Decorator to mark functions as deprecated :param new_function: Function called in place of deprecated function :param message: Deprecation message, replaces default deprecation message funcr'csFdurjddurdd7tfdd}|S)Nz6 is deprecated and will be removed in a future releasez . Please use z instead.cstjtdd|i|S)N) stacklevel)warningswarnDeprecationWarning)rdre)rlrkr4r5wrappedsz.deprecated..decorator..wrapped)__name__r)rlrrrjrk)rlr5 decorators zdeprecated..decorator)r)rjrkrur4rtr5rs rc@s:eZdZdZedeeeffddZddZddZ d S) rz A mixin for enums to allow aliasing of enum members Example: >>> class MyClass(Aliasable, int, Enum): >>> ... r'cCstN)NotImplementedErrorr4r4r4r5 get_aliasesszAliasable.get_aliasescCslt||jr |}|j|jkp||j|j||j|jkS|}||j|j}|||}||kSrv)r\ __class__rxvaluer0)selfotheraliases self_value other_valuer4r4r5__eq__s   zAliasable.__eq__cCs|j|j|j}t|Srv)r}r0rzhash)r{canonical_valuer4r4r5__hash__szAliasable.__hash__N) rs __module__ __qualname____doc__ staticmethodrstrrxrrr4r4r4r5rs  rrL shard_sizesrAcCsTt|||kr tdg}d}|D]}||}||||}|||}q|S)a Shards a tensor into a list of tensors along a given dimension. raises: ValueError: If the sum of shard_sizes does not match the size of the tensor along the given dimension. :param tensor: The input tensor to shard. :param shard_sizes : List of sizes for each shard along the specified dimension. :param dim : The dimension along which to shard the tensor. :returns: A list of tensors sharded along the specified dimension. zSSum of shard_sizes must equal the size of the tensor along the specified dimension.r)rGsizerKnarrowappend)rLrrAshards start_idxrend_idxshardr4r4r5rs rcs|stddd|D}t|dkrtdt|dj}tfdd|D|<tj||dj|djd }d}|D]}|j}| || |||7}q=|S) z Combine decompressed shards along a given dimension using `narrow`. :param shards: List of decompressed shard tensors. :param dim: Dimension to combine along (default: 0). :return: Combined decompressed tensor. zThe list of shards is empty.cSsh|]}|jqSr4)dtype.0rr4r4r5 $sz!combine_shards..r?z$All shards must have the same dtype.rc3s|]}|jVqdSrv)shaperr@r4r5 *sz!combine_shards..)rdevice) rKrSlistrrGrHzerosrrrcopy_)rrA shard_dtypes total_shapecombined shard_offsetr shard_sizer4r@r5rs   r bytemaskscCs"tj|ddd}t|}|S)a Converts a bytemask tensor to a bitmask tensor to reduce memory. Shape RxC will be compressed to R x ceil(C/8) :param bytemasks: mask tensor where each byte corresponds to a weight :return: mask tensor where each bit corresounds to a weight r>little)axisbitorder)numpypackbitsrH from_numpy)rpacked_bits_numpypacked_bits_torchr4r4r5r9s rpacked_bitmasksoriginal_shapecCs8tj|d|ddd}t||t}|S)a# Converts a bitmask tensor back to a bytemask tensor for use during decompression :param packed_bitmasks: mask tensor where each bit corresponds to a weight :param original_shape: dense shape to decompress to :return: boolean mask of weights in the original dense shape r>r)rcountr)r unpackbitscpurHrreshapeastypebool)rr unpacked_bitsunpacked_bitmasks_torchr4r4r5rGs rbaseattrrzc csrt}t|||}t|||zdVW||ur!t|||dSt||dS||ur3t|||wt||w)a Patch the value of an object attribute. Original value is restored upon exit :param base: object which has the attribute to patch :param attr: name of the the attribute to patch :param value: used to replace original value Usage: >>> from types import SimpleNamespace >>> obj = SimpleNamespace() >>> with patch_attr(obj, "attribute", "value"): ... assert obj.attribute == "value" >>> assert not hasattr(obj, "attribute") N)objectrcrUdelattr)rrrz _sentineloriginal_valuer4r4r5ras   rbasesvaluesccs\t}t||D] \}}|t|||q dVWddS1s'wYdS)a Same as `patch_attr` but for a list of objects to patch Patch attribute for a list of objects with list of values. Original values are restored upon exit :param bases: objects which has the attribute to patch :param attr: name of the the attribute to patch :param values: used to replace original values. Must be same length as bases Usage: >>> from types import SimpleNamespace >>> obj1 = SimpleNamespace() >>> obj2 = SimpleNamespace() >>> with patch_attr([obj1, obj2], "attribute", ["value1", "value2"]): ... assert obj1.attribute == "value1" ... assert obj2.attribute == "value2" >>> assert not hasattr(obj1, "attribute") >>> assert not hasattr(obj2, "attribute") N) contextlib ExitStackzip enter_contextr)rrrstackrrzr4r4r5r ~s  "r c@sVeZdZdZdeegeffddZdedefddZeid d e defd d Z d S)r!a  Similar to `collections.DefaultDict`, but upon fetching a key which is missing, the key is passed as arguments to the `default_factory` :param default_factory: function which takes a key as input and returns the corresponding default value default_factorycCs||_ti|_dSrv)rr_factory_kwargs)r{rr4r4r5__init__sz!ParameterizedDefaultDict.__init__keyr'cCs>t|tr|j|i|j}n |j|fi|j}|||<|Srv)r\rBrr)r{rrzr4r4r5 __missing__s z$ParameterizedDefaultDict.__missing__)factory_kwargsrcGs8t|d| ||WdS1swYdS)a" Similar to `__getitem__`, but allows passing kwargs to factory function :param \*args: args whose tuple will value will be treated as key :param factory_kwargs: keyword arguments to pass to `default_factory` :return: dictionary entry for given key rN)r)r{rrdr4r4r5r0s$zParameterizedDefaultDict.getN) rsrrrrrrrrr r0r4r4r4r5r!s  r!r+cC>t|dr|jSt|drt|dr|j|jStd|)z Get the number of attention heads used by a model :param config: model config :return: num_attention_heads of model num_attention_heads hidden_sizehead_dimzCannot determine num_attention_heads from config. Config must define either `num_attention_heads` or both `hidden_size` and `head_dim`. )rarrrrKr*r4r4r5r"  r"cCst|dr|jStd|)z Get the number of key-value attention heads used by a model :param config: model config :return: num_key_value_heads of model num_key_value_headsz\Cannot determine num_key_value_heads from config. Config must define `num_key_value_heads`. )rarrKr*r4r4r5r#s r#cCr)z Get the number of dimensions used by the attention heads of a model :param config: model config :return: head_dim of model rrrz}Cannot determine head_dim from config. Config must define either `head_dim` or both `hidden_size` and `num_attention_heads`. )rarrrrKr*r4r4r5r$rr$)r;)NN)r)5rro functoolsrtypesrtypingrrrrrr r r r rrH transformersr rrr,r__all__r:rrrrrnnModulerrrrrTensorrDrrrrcontextmanagerrrr dictr!r"r#r$r4r4r4r5sv  ,           !  !