o Ti-@sNddlZddlZddlmZd ddZGdddejjZGdd d eZ dS) N)partition_uniformFcCs6|d}tj|||d}|rtdd|DS|S)a+Split a tensor along its last dimension. Adapted from Megatron-LM. Arguments: tensor: input tensor. partitions: list of partition sizes to supply to torch.split contiguous_split_chunks: If True, make each chunk contiguous in memory. dimcss|]}|VqdS)N) contiguous).0chunkr Q/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/runtime/zero/tiling.py sz.split_tensor_along_last_dim..)rtorchsplittuple)tensor partitionscontiguous_split_chunkslast_dim tensor_listr r r split_tensor_along_last_dim s rcs`eZdZdddddejjdffdd ZddZd d Zd d Z d dZ e ddZ Z S) TiledLinearTrFNc  st|dks ||krtd|dks||krtd||_||_||_||_||_||_||_ t ||d|_ t ||d|_ t |j |dksKJt |j |dksVJ|j ddks_J|j ||kshJ|j ||ksqJtj|_t|D]G} |jtj|j | d|j | } t|D]+} | |dkr|nd}|j | d|j | }||| fd|i| }|j| |qq{| dur|| dSdS) a@A replacement for ``torch.nn.Linear`` that works with ZeRO-3 to reduce memory requirements via tiling. TiledLinear breaks the input and output dimensions of a linear layer into tiles that are processed in sequence. This class enables huge linear layers when combined with ZeRO-3 because inactive tiles can be partitioned and offloaded. .. note:: We recommend using as few tiles as necessary. Tiling significantly reduces memory usage, but can reduce throughput for inexpensive layers. This due to the smaller kernels having less parallelism and lower arithmetic intensity, while introducing more frequent synchronization and communication. Args: in_features (int): See ``torch.nn.Linear`` out_features (int): See ``torch.nn.Linear`` bias (bool, optional): See ``torch.nn.Linear`` in_splits (int, optional): The number of tiles along the input dimension. Defaults to 1. out_splits (int, optional): The number of tiles along the output dimension. Defaults to 1. input_is_already_split (bool, optional): If set to ``True``, assume that the ``input_`` in to ``forward()`` is already split into ``in_splits`` chunks. Defaults to ``False``. combine_out_splits (bool, optional): If set to ``False``, do not combine the ``out_splits`` outputs into a single tensor. Defaults to ``True``. linear_cls (class, optional): The underlying class to build individual tiles. Defaults to ``torch.nn.Linear``. init_linear (``torch.nn.Linear``, optional): If set, copy the parameters of ``init_linear``. Useful for debugging. Defaults to ``None``. kwargs (dict, optional): additional keyword arguments to provide to ``linear_cls()``. Raises: RuntimeError: ``in_splits`` must be within the range [1, in_features). RuntimeError: ``out_splits`` must be within the range of [1, out_features). rz,in splits must be in range [1, in_features].z.out splits must be in range [1, out_features].) num_items num_partsrFbiasN)super__init__ RuntimeError in_features out_featuresuse_bias out_splits in_splitsinput_is_already_splitcombine_out_splits partitionin_parts out_partslenr nn ModuleListlinearsrangeappendcopy_params_from)selfrrrr rr!r" linear_cls init_linearkwargsout_id local_out_dimin_id local_bias local_in_dimlocal __class__r r r"s@ /    zTiledLinear.__init__cs|jdkr$|js$t|jd|jfddt|jD}|||}n|jdkr@|}t||jks?Jd|jdt|n|g}dg|j}t|jD]"}t|jD]}|j||||}|j |||||d||<qUqN|j ry| |S|S)Nrcs g|] }|d|qS)rr )rp input_partsr r  z'TiledLinear.forward..z Col splits z does not match input splits r3r1 current_outnew_out) r r!r#shaper*_split_global_inputr&rr)_reduce_local_outputr"_combine_output_splits)r-input_ split_sizesinputsoutputsr1r3 local_outputr r;r forwards6    zTiledLinear.forwardcCs t||S)aPartition an input tensor along the last dimension, aligned with given splits. Subclasses should override this method to account for new input types. Args: input (List[Tensor]): The tensor to partition along the last dimension. split_sizes (List[int]): The size of each partition. Returns: List[Any]: A list of the chunks of ``input``. )r)r-inputrGr r r rCs zTiledLinear._split_global_inputcCs|dur|S||S)aReduce (sum) a new local result into the existing local results. Subclasses should override this method. For a given ``out_id``, this method is called ``in_id-1`` times. The first input split is a simple assignment. Args: in_id (int): The input split that produced ``new_out``. out_id (int): The output split that produced ``new_out``. current_out (Any): The reduced form of all previous ``out_id`` results. new_out (Any): The local result from forward (``in_id``, ``out_id``)e Returns: Any: The combined result of ``current_out`` and ``new_out``. N)clone)r-r3r1r@rAr r r rDsz TiledLinear._reduce_local_outputcCs t||jks Jtj|ddS)zJoin the splits of the output into a single result. Args: outputs (List[Any]): The reduced outputs for each output split. Returns: Any: The combined outputs. r9r)r&rr cat)r-rIr r r rEs z"TiledLinear._combine_output_splitsc Cs|t|dsJ|j|j|jfksJ|jr1t|dsJ|jdus%J|j|jfks0Jn|jdus8Jt|jD]~}|j |}|j |d}t|j D]?}|j |}|j |d}|j ||}|j||||f} t jj|jdd|j| Wdn1swYqP|jdurt jj|jdd|jj|j||jWdn1swYq=dS)aCopy the weight and bias data from ``other``. This is especially useful for reproducible initialization and testing. Equivalent to: .. code-block:: python with torch.no_grad(): self.weight.copy_(other.weight) if self.bias is not None: self.bias.copy_(other.bias) .. note:: If ZeRO-3 is enabled, this is a collective operation and the updated parameters of data-parallel rank 0 will be visible on all ranks. See :class:`deepspeed.zero.GatheredParameters` for more information. Args: other (``torch.nn.Linear``): the linear layer to copy from. weightrNrr) modifier_rank)hasattrrOsizerrrrr*rr%r r$r) deepspeedzeroGatheredParameterscopy_data) r-otherrowrstartrstopcolcstartcstopr6 global_weightr r r r,s2   zTiledLinear.copy_params_from)__name__ __module__ __qualname__r r'LinearrrKrCrDrEno_gradr, __classcell__r r r7r r s^ rcs,eZdZdZfddZfddZZS)TiledLinearReturnBiaszeWrapper for a Linear class that returns its own bias parameter, such as used by Megatron-LM. c st|dur |\}}nd\}}t|tsJt|dksJ|\}}|dus&Jtj||||d}|dur6|}||fS)z3Reduces output tensors, but not the returned bias. N)NNr?) isinstancerr&rrD) r-r3r1r@rA old_tensorold_biasrrr7r r rDs  z*TiledLinearReturnBias._reduce_local_outputcsTdd|D}t|}dd|D}t|dkr$t|}||fSd}||fS)NcSsg|]}|dqS)rr ror r r r=sz@TiledLinearReturnBias._combine_output_splits..cSs g|] }|ddur|dqS)rNr rkr r r r="r>r)rrEr&)r-rItensorsrbiasesrr7r r rEs   z,TiledLinearReturnBias._combine_output_splits)r`rarb__doc__rDrErer r r7r rfs rf)F) r rSdeepspeed.runtime.utilsrr#rr'Modulerrfr r r r s  d