o Ti @s\ddlmZddlZdZejejejejejej ej ej ej ej d ZGdddejZdS))DictN_ds_core_param_key) z torch.float32z torch.float64z torch.float16ztorch.bfloat16z torch.int64z torch.int32z torch.int16z torch.int8z torch.uint8z torch.boolcsreZdZdZefddZfddZedej ddfdd Z edd d Z e de eej ffd d ZZS)InferenceParameterz An extension of the torch.Tensor class to support our inference focused features. One important thing to note here is that an InferenceParam can be used a torch.Tensor, but outputs of torch.Tensor operations will not be InferenceParams. cs8tj||g|Ri|}t|drt|d|j|S)N _aux_attrs)super__new__hasattrsetattr aux_attrs)clstensorargskwargs new_tensor __class__^/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/inference_parameter.pyr!s zInferenceParameter.__new__cstj|i|}t|drt|d|jz't|d}|jD]\}}|j|i|}t|||||j|<q"W|SY|S)Nrr)rtorr r torchdeviceitems)selfr rr_nameattrnew_attrrrrr(s   zInferenceParameter.to core_paramreturncKslt|}t|d||D]%\}}t||rtd|dt|tjs-td|dt|||q|S)z1 Create the inference parameter. rz Attribute z already exists on param.z must be a tensor.)rr rr ValueError isinstancerTensor)r rrparam attr_namerrrr initialize7s   zInferenceParameter.initializecKs.t|vr tdtdtj|tfi|S)zW All kwargs must be torch.Tensors and must include the core parameter. z&Must provide core parameter, with key .) CORE_PARAMrrr$)rrrrrinitialize_rawJsz!InferenceParameter.initialize_rawcCs|jS)z5 Dictionary of auxiliary attributes. )r)rrrrr TszInferenceParameter.aux_attrs)rr)__name__ __module__ __qualname____doc__ staticmethodrr classmethodrr!r$r'propertyrstrr __classcell__rrrrrs   "r)typingrrr&float32float64float16bfloat16int64int32int16int8uint8bool STR_TO_DTYPEr!rrrrrs