o *iB@sUddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZeeZejed<gdZe eeefZ   d'd ejjd e eed e eed e ed edee eejjff ddZ   d'd ejjd e eed e eed e ed edee eejjejjff ddZdedejjd e eede efddZde edBdefddZ!  d(d ejjd e eed e eededee e ejjf ddZ"e#dfdedejjd e eeefd e eeefd e edef ddZ$ d)d ejjd e eeefdede ejjdef d d!Z%d)ded"ed e edefd#d$Z&dejjd"edefd%d&Z'dS)*N) defaultdict) Generator)IterableListMappingOptionalTupleUnion)InternalModule_LOGGER)match_named_modulesmatch_named_parameters match_targetsmatch_modules_setget_lowest_common_ancestor_nameis_matchis_narrow_matchFmodeltargetsignorefused warn_on_failreturnc cs|pg}|pg}t|}|D]%\}}|D]}t||||dr5||h8}t||||ds3||fVnqq|rK|D]}td|d|jjq;dSdS)a Yields names and modules which match `targets` but do not match `ignore`. Values are returned in order of `model.named_modules()` :param model: model containing submodules to match against :param targets: target strings, potentially containing "re:" prefixes :param ignore: targets to ignore, potentially containing "re:" prefixes :fused: optional mapping from suffixes of fused modules to the suffixes of their corresponding shards. See `compressed_tensors.utils.match.is_match` :param warn_on_fail: if True, warns if any targets do not match any modules in model :return: generator of module names and modules )rCould not match `` in instance of N)set named_modulesrr warning __class____name__) rrrrrunmatched_targetsnamemoduletargetr$[/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/compressed_tensors/utils/match.pyr +s(  r c #s|pg}|pg}t|}|D]>\}}t|trq|jddD]-\}} |d||D]} t| rM|| h8}tfdd|DsM|| fVq.q!q|rd|D]} td| d|j j qTdSdS) a Yields parameters which match `targets` but do not match `ignore`. Values are returned in order of `model.named_modules()` :param model: model containing params to match against :param targets: target strings, potentially containing "re:" prefixes :param ignore: targets to ignore, potentially containing "re:" prefixes :fused: optional mapping from suffixes of fused modules to the suffixes of their corresponding shards. See `compressed_tensors.utils.match.is_match` :param warn_on_fail: if True, warns if any targets do not match any params in model :return: generator of fully-qualified param names, parent modules, and params F)recurse.c3s|] }t|VqdSN _match_name.0ignr param_fqnr$r% ssz)match_named_parameters..rrN) rr isinstancer named_parametersr*anyr rrr) rrrrrr module_namer" param_nameparamr#r$r.r%r Rs0     r r!r"cCsr|pg}t|tr gSt|ddd}g}|D] }t||r#||q|D]}t||r6||vr6||q&|S)av Returns the targets that match the given name and module. :param name: the name of the module :param module: the module to match :param targets: the target strings, potentially containing "re:" prefixes :return: the targets that match the given name and module Outputs are ordered by type: exact name match, regex name match, class name match cSs d|v|fS)Nre:r$)xr$r$r%s zmatch_targets..)key)r1r sortedr*append _match_class)r!r"rmatched_targetsr#r$r$r%r}s     rnamescCs`dd|D}t|dkrdSdt|d}dt|d}tj||g}|d|dS)a Given a list of names, returns the lowest-scope common name ignoring Nones. Implementation is a small alteration of os.path.commonprefix https://docs.python.org/3/library/os.path.html#os.path.commonprefix ([s1, s2]->prefix->result) # case 0: multiple modules: [abc.a., abc.b.] -> .abc. -> abc # case 1: single module: [abc.] -> .abc. -> abc # case 2: substring modules: [abc., ab.] -> .ab -> "" # case 3: parent & child: [ab., ab.a.] -> .ab. -> ab cSsg|]}|dur|qSr(r$)r,r!r$r$r% sz3get_lowest_common_ancestor_name..rr')lenminmaxospath commonprefixrfind)r?s1s2 common_prefixr$r$r%rs rTerror_on_module_rematchc #s*|pg}|pg}ttd}t|}|D]Y\}}t}|D]<} t||| |r\t||g} |sI| |krIfdd|DVtt|} t|}| || }|| h8}|| hO}q t|dkrp|rptd|d|dq|t|krydS|sfdd|DVdStd t||d |) aR Yields modules grouped by parent context. We group by parent context so that we can return ALL matches of a specific target that can be paired with another target. This is most relevant in the case of MoE modules with multiple modules for each expert i.e. post_attention_layernorm <-> mlp.expert.N.gate_proj, mlp.expert.N.up_proj for all N. The parent context will differ from one layer to another while being the same for one expert to another. Each returned group is a list (of lists) with the same size and order as `targets` while all matches for each target and the overall order of the groups are ordered in the same way as `model.named_modules` E.g. the following targets would yield modules belonging to the following layers: ```python3 match_modules_set(model, ["q_proj", "k_proj", "v_proj"]) == ( [ [`layers.0.self_attn.q_proj`], [`layers.0.self_attn.k_proj`], [`layers.0.self_attn.v_proj`], ], [ [`layers.1.self_attn.q_proj`], [`layers.1.self_attn.k_proj`], [`layers.1.self_attn.v_proj`], ], ... ) ``` This can be used to match layers to their corresponding downstream counterparts. For example, matching layer norms to their subsequent linear layers ```python3 for norm, q, k, v in match_modules_set(model, (norm_tgt, q_tgt, k_tgt, v_tgt)): fuse_norm_linears(*norm, [*q, *k, *v]) ``` Alternatively for MoE you would get multiple matches per target per group, E.g. ```python3 targets = [ "post_attention_layernorm", "up_proj", "down_proj" ] match_modules_set(model, targets) == ( [ [layers.0.post_attention_layernorm], [ `layers.0.mlp.experts.0.up_proj`, `layers.0.mlp.experts.1.up_proj`, ... ], [ `layers.0.mlp.experts.0.down_proj`, `layers.0.mlp.experts.1.down_proj`, ... ] ], # <- first yield [ [layers.1.post_attention_layernorm], [ `layers.1.mlp.experts.0.up_proj`, `layers.1.mlp.experts.1.up_proj`, ... ], [ `layers.1.mlp.experts.0.down_proj`, `layers.1.mlp.experts.1.down_proj`, ... ] ], ... ) ``` :param model: model containing modules to match against :param targets: target strings, potentially containing "re:" prefixes :param ignore: targets to ignore, potentially containing "re:" prefixes :param error_on_module_rematch: if True, errors when a module gets matched to multiple targets, if False, no error. (Defaults to True) Ncg|]}|qSr$r$r,r#matchesr$r%r@@z%match_modules_set..rBzmodule: z$ was matched with multiple targets: zT which is unexpected disable this check by setting `error_on_module_rematch = False`crNr$r$rOrPr$r%r@WrRz:Found a final incomplete set with matches found for keys: z but no matches found for keys: ) rlistrrrrr<rC ValueError) rrrrMparent_contextr r!r"matched_targets_for_cur_moduler#new_parent_contextr$rPr%rsT^     rcsjt|tr|gn|}t|tr|gn|}tt o4tfdd|Do4tfdd|D S)a Returns true if either module name or module parent classes match against target and the module is not an internal module. The name and module may refer to a fused module defined by vLLM. In these cases, a `fused` mapping must be provided. For example, in `vllm/model_executor/models/llama.py`: ```python packed_modules_mapping = { "qkv_proj": ["q_proj", "k_proj", "v_proj"], "gate_up_proj": ["gate_proj", "up_proj"] } ``` :param name: name of module :param module: module to match :param target: target which matches name or module, potentially contains regex :fused: optional mapping from suffixes of fused modules to the suffixes of their corresponding shards c3&|]}t|pt|VqdSr(r*r=rOrr"r!r$r%r0s  zis_match..c3rXr(rYr+rZr$r%r0s )r1strr r3)r!r"rrrr$rZr%ras rcs`t|tr|gn|}durn|ddd|tfdd|DS)a Checks if any of the targets narrowly match the module. A target narrowly matches a module if the target matches the module, but does not match the module's parent :param model: model containing both module and its parent :param targets: target strings, potentially containing "re:" prefixes :param name: name of module to match :param module: module to match. If none is provided, then get module from model :return: True if any of the targets narrow match the module Nr'rBrc3s*|]}t|ot| VqdSr()rrOr"r!parent parent_namer$r%r0s  z"is_narrow_match..)r1r[ get_submodulersplitr3)rrr!r"r$r\r%rs rr#csn|dur#|D]}||r"||tfdd||DSqdr3td|duS|kS)a Returns true if target string begins with "re:" and regex matches or if target string exactly matches name. If the name refers to a fused module defined by vLLM, a `fused` mapping must be provided. :param name: name of module :param target: target name, potentially contains regex :fused: optional mapping from suffixes of fused modules to the suffixes of their corresponding shards Nc3s|] }t|VqdSr(r))r, shard_suffix name_strippedr#r$r%r0s  z_match_name..r7)endswith removesuffixr3 startswithrematch removeprefix)r!r#r fused_suffixr$rbr%r*s     r*cstfdd|jjDS)a Returns true if any torch parent class names match the target string exactly. A special exception is made for vllm's `LinearBase` class which matches `Linear` :param module: module to match :param target: target which matches name or module c3s:|]}t|tjjo|jkp|jdkodkVqdS) LinearBaseLinearN) issubclasstorchnnModuler)r,clsr#r$r%r0s  z_match_class..)r3r__mro__)r"r#r$rrr%r=s r=)NNF)NTr()(loggingrFrg collectionsrcollections.abcrtypingrrrrrr rn!compressed_tensors.utils.internalr getLoggerrr Logger__annotations____all__r[ FusedMapppingrorpboolr Parameterr rrSrrtuplerrr*r=r$r$r$r%s         *   +  &   ) ,