o TiO @sddlmZddlZddlmZmZddlmZmZ  ddej de d e d e d e d ej f d dZ ddej de d e d e d eej f ddZ de d e de d e fddZdS))OptionalN) ShardingTypeDEFAULT_SHARD_GRANULARITY) shard_paramget_shard_endpointsFparam shard_rank num_shardsgatedis_moereturncCs>|rdnd}|rt|tj||td|dSt|tj|||dS)a Utility method for sharding an MLP 1 parameter. Both biases and weights are supported, as well as for fused weights for MoE. Args: param (torch.Tensor): The parameter to shard. shard_rank (int): Which shard of the partitioned tensor to return. num_shards (int): The total number of shards the parameter is distributed across. gated (bool): Whether or not the parameter is from a gated MLP. r) granularity bias_dims)r)rrOUTER_DIMENSIONr)rr r r r rrm/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/model_implementations/sharding/mlp.pyshard_mlp_1_params rcCs:|rdnd}t|j|kr|dkr|SdSt|tj||S)ah Utility method for sharding an MLP 2 parameter. Args: param (torch.Tensor): The parameter to shard. shard_rank (int): Which shard of the partitioned tensor to return. num_shards (int): The total number of shards the parameter is distributed across. is_moe (bool): Whether or not the parameter is from a MoE model. rrrN)lenshaperrINNER_DIMENSION)rr r r bias_dim_sizerrrshard_mlp_2_param*s rintermediate_sizecCst|||}|d|dS)aV Utility method for getting the size of the intermediate dimension of a sharded MLP. Args: intermediate_size (int): The size of the intermediate dimension. num_shards (int): The total number of shards the parameter is distributed across. shard_rank (int): Which shard of the partitioned tensor to return. rr)r)rr r endpointsrrrsharded_intermediate_dimAs r)FF)F)typingrtorchtypesrrutilsrrTensorintboolrrrrrrrs8