o Ti@sddlmZddlZddlmZddlmZmZ  ddejde de d e d ee d ee d eejfd dZ  dde de de d e d ee d ee d e fddZ dS))OptionalN) ShardingType) shard_paramget_shard_endpointsparam shard_rank num_shards head_size n_heads_q n_heads_kvreturncCs |dur |dur td|dur+|jdd|dkrtd|jd|d}d}n||k}||kr7td|rDt|tj||d|dS||dkrNtd |jd||d |kr_td ||krm||dkrmtd ||kr{||dkr{td ||k}|durdS|d||}|||d} |rt|tj|||d}t| tj||d |d} tj|| gddS||} || } | | || d|} | || ||| d|} || }||}||| ||| d|}t|tj|| |d}tj|| | gddS)a  Utility method for sharding a QKV parameter. Both biases and weights are supported. It is assumed that the layout of the parameter is such that all Q heads, all K heads, and all V heads are contiguous with respect to each other. Args: param (torch.Tensor): The parameter to shard. shard_rank (int): Which shard of the partitioned tensor to return. num_shards (int): The total number of shards the parameter is distributed across. head_size (int): The size of each head. n_heads_q (int): The number of query heads. This only needs to be passed if the number of query and key/value heads are different. If passed without n_heads_kv, default MHA partitioning will be used. n_heads_kv (int): The number of key/value heads. This only needs to be passed if the number of query and key/value heads are different. This argument should not be passed without n_heads_q (we want to explicitly opt into GQA sharding). N1n_heads_kv should not be passed without n_heads_qrzMHA param shape is not correctTz?There must be at least as many query heads as there are shards.)num_concatenated_matrices granularityz8Must be an even ratio between query and key/value heads.zGQA param shape is not correctzACurrently do not support uneven partitioning of KV heads for GQA.zRCurrently do not support distributing KV heads across different numbers of shards.r)dimr) ValueErrorshaperrOUTER_DIMENSIONtorchcat)rrr r r r mha_shardingeven_kv_shardingq_paramkv_paramq_sharding_degreekv_headk_paramv_paramq_sharding_rankq_factorq_chunkr%m/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/sharding/qkv.pyshard_qkv_paramsn   r' in_featurescCs|dur |dur td|dup||k}|dur"|||kr"td|r6t||||d}|d|ddS||krT||dkrDtd||}||} ||d | S||dkr^td ||} || } |||} t| | | |d} | d| dd |S) aq Helper to calculate the expected output projection dimension of a QKV projection matrix. Args: in_features (int): The model dimension. shard_rank (int): Which rank to return the corresponding size for. num_shards (int): The total number of shards the parameter is distributed across. head_size (int): The size of each head. n_heads_q (int): The number of query heads. This only needs to be passed if the number of query and key/value heads are different. If passed without n_heads_kv, default MHA partitioning will be used. n_heads_kv (int): The number of key/value heads. This only needs to be passed if the number of query and key/value heads are different. This argument cannot be passed without also passing n_heads_q (we want to explicitly opt into GQA sharding). Nrz:in_features is not consistent with n_heads_q and head_sizerrrrz:The KV heads must be evenly distributed across the shards.rzHA shared KV head must always partition across the same number of shards.)rr)r(rr r r r r endpointsn_local_groups group_sizeq_split_degree q_split_ranksplit_granularity q_endpointsr%r%r&qkv_out_featuresss*   r0)NN) typingrrtypesrutilsrrTensorintr'r0r%r%r%r&sD    i