o TiY@s`ddlZddlmZddlmZmZdejdededejfd d Zd edededefd d Z dS)N) ShardingType) shard_paramget_shard_endpointsparam shard_rank num_shardsreturncCst|tj||ddS)a Utility method for sharding an unembed parameter. We shard unembeddings on the vocab dimension with the expectation of an all-gather to produce the full results. TODO(cmikeh2): Really ideal would be if MII could have access to the comm and we would do an A2A and sharded sampling. Args: param (torch.Tensor): The parameter to shard. Should be of shape [vocab_size, model_dim] shard_rank (int): Which shard of the partitioned tensor to return. num_shards (int): The total number of shards the parameter is distributed across. Returns: torch.Tensor: The sharded parameter of shape [sharded_vocab_size, model_dim] r granularity)rrOUTER_DIMENSION)rrrr q/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/sharding/unembed.pyshard_unembed_param sr vocab_sizecCst|||dd\}}||S)aE Utility method for determining the sharded vocab size of a sharded unembed parameter. Args: vocab_size (int): The size of the vocabulary. shard_rank (int): Which shard of the partitioned tensor to return. num_shards (int): The total number of shards the parameter is distributed across. rr )r)rrr start_idxend_idxr r rsharded_unembed_dims r) torchtypesrutilsrrTensorintrrr r r rs