o Ti!#@sddlZddlmZddlmZddlZedZedZdZddZ d d Z d d Z Gd dde Z Gddde dZGdddZdefddZdS)N)abstractmethod)TypeInferenceModelLayerContainer PARAM_MAPPINGcfdd}|S)z6 Normal getter implementation for a property. cst|ddSN__)getattrselfclsnameparamo/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/parameter_base.py param_gettersz'make_param_getter..param_getterr)rrrrr rmake_param_gettersrcr)zl Setter implementation that will call complete component to potentially finalize the parameter. cs$t|dd||dSr)setattrcomplete_componentr valuer rr param_setter$s z'make_param_setter..param_setterr)rrrrr rmake_param_settersrcCs dd}|S)zC Setter implementation that will raise an error if called. cSstd)Nz'Cannot set a ParametrizedList directly.) ValueErrorrrrrparamlist_setter0sz.make_readonly_setter..paramlist_setterr)rrrrmake_readonly_setter+srcs,eZdZdZfddZfddZZS)ParameterMetaclassa$ MetaClass for the ParameterBase base class. This class will parse the `src_params` attribute and create properties for each of the dependencies. A dependency can either be represented as a string, which is interpreted as a named Tensor, or a `ParametrizedList` subclass. c s|di}dd|D}t|}|D]+\}}t|tr5t|ds)Jdtt||t||<qtt||t ||||<qt ||||} || _ | S)N__annotations__cSs&i|]\}}t|tjtfr||qSr) issubclasstorchTensorParametrizedList).0name annotationrrr As z.ParameterMetaclass.__new__.. count_attrzTParametrizedList must have a count_attr attribute to access on the inference module.) getitemslenrr"hasattrpropertyrrrsuper__new__n_dependencies) clsrbasesattrs annotations dependenciesr/d_named_typenew_cls __class__rrr.>s"  zParameterMetaclass.__new__cstj|i|}|j|i|t|dd|jD]G\}}t|trUt|j |j s __module__ __qualname____doc__r.r; __classcell__rrr8rr6s rc@sHeZdZdZ dededdfddZedej fdd Z d d d Z dS) ParameterBasea% A ParameterBase allows us to consolidate tracking the dependencies of loading a parameter from a checkpoint into a single object. This class should not be used directly, but rather subclassed and the `src_params` attribute set to a list of strings and/or `ParametrizedList`s. modelparent_containerreturnNcCs||_d|_t||_dS)a Direct constructor. This should not be called from client code. Args: model (InferenceModel): Inference model that will be used to shard and transform the parameter in `finalize`. parent_container (LayerContainer): The parent container that this parameter is a member of. We will build a weakref to this container to call the finalization callback. rN)r=completed_componentsweakrefrefrI)r rHrIrrrr<~s zParameterBase.__init__cCsdS)z Finalize the parameter after all of its source parameters have been set. This method will be automatically called when all inputs have been set. It should return the Tensor with all transformations performed on it. Nrr rrrfinalizeszParameterBase.finalizecCs:|jd7_|j|jkrdS|}|||dS)a Mark a component as completed. This should be called by the relevant setter of a direct property or a ParametrizedList. This method will automatically call `finalize` when all dependencies have been met and then call the finalization callback on the parent container. Once the finalization callback has been called, the parameter will be replaced with the `dst_param` attribute on the parent container, and this instance will be destroyed. N)rKr/rNrIfinalization_callback)r finalized_paramrrrrs  z ParameterBase.complete_component)rJN) r>rCrDrErrr<rr r!rNrrrrrrGksrG) metaclassc@sTeZdZUdZeed< eed< dededdfddZdd Zd d Z d d Z dS)r"a A ParametrizedList is a list of parameters that are dependencies of a `ParameterBase` but may vary in length depending on the model configuration (rather than architecture). For example, a MoE layer may have different number of experts depending on the size of the model. This class is used to manage these lists and provide integer indexing of a single component rather than accessing names directly. For example, it tends to be more natural to access the 8th expert with `experts[8]` rather than a name like `expert_8`, especially as an attribute. To inherit from this class, set static variables `name` and `count_attr`. ```python class MyParametrizedList(ParametrizedList): count_attr: str = "my_list_count" ``` In the above example, `my_list_count` should be an accessible attribute of the inference model (i.e. via `self.inference_model.my_list_count`). NOTE: There are some APIs in which this type cannot be used as if it is just a list of Tensors. For example, `torch.cat(param_list)` will not work. However, you can make it compatible with a tuple wrapper: `torch.cat(tuple(param_list))` n_paramsrrJNcCs(||_d|_t||_dg||_dS)z Constructor. Should not be called from client code. Args: param (ParameterBase): The owning parameter. n_params (int): The number of parameters this list contains. This should be rN)rS set_paramsrLrMr_params)r rrSrrrr<s zParametrizedList.__init__cCs |j|SN)rU)r indexrrr __getitem__ zParametrizedList.__getitem__cCsN|j|dur td||j|<|jd7_|j|jkrdS|dS)NzCannot set a parameter twice.rO)rUrrTrSrr)r rWrrrr __setitem__s  zParametrizedList.__setitem__cCs t|jSrV)iterrUr rrr__iter__rYzParametrizedList.__iter__) r>rCrDrEintrrGr<rXrZr\rrrrr"s   r"attrcsGfdddt}|S)a Helper to create a subclass of ParametrizedList with the desired `count_attr`. In this manner, we can annotate the type of a Parameter dependency with the following: ```python class CustomParameter(ParameterBase): dependency_list: ParamList("dependencies_count_name") ``` where "dependencies_count_name" is the name of the attribute on the inference model. cseZdZUZeed<dS)z+ParamList..ParametrizedListInstancer'N)r>rCrDr'strrrr^rrParametrizedListInstances ra)r")r^rarr`r ParamListsrb)rLabcrtypingrr rr MAPPING_KEYrrrtyperrGr"r_rbrrrrs     5<F