o TÃiH ã @sBddlmZmZ d dedededeedeeeff dd „ZdS) é)ÚOptionalÚTupleNÚ shard_rankÚ num_shardsÚ n_heads_qÚ n_heads_kvÚreturnc Cs||krtdƒ‚|dus||kr(||}||}||kr$|d|dfS||fS||dkr2tdƒ‚||kr@||dkr@tdƒ‚||krN||dkrNtdƒ‚||}||krb||}||}||fS||} || } || }|| }| |kr||ddfS|dfS)a¯ Helper to determine the number of local heads of a given shard. Args: shard_rank (int): The rank of the shard. num_shards (int): The total number of shards that attention is distributed over. n_heads_q (int): The number of query heads. n_heads_kv (int): The number of key/value heads. If not passed, it is assumed that the number of query and key/value heads are the same. zCThere must be at least as many attention heads as there are shards.Nérz8Must be an even ratio between query and key/value heads.z]If splitting a group across multiple shards, we must be able to distribute the groups evenly.z@If parallelizing groups, must be able to evenly distribute them.)Ú ValueError) rrrrÚ base_headsÚ extra_headsÚq_ratioÚlocal_kv_headsÚ local_q_headsÚgroup_sharding_sizeÚgroup_rank_idx©rún/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/sharding/attn.pyÚget_local_heads s8 ÿ r)N)ÚtypingrrÚintrrrrrÚsýÿþý ý