o }oi@sFddlZddlmZddlmZmZmZdedededefdd ZdS) N)Tensor) ProcessGroup all_gatherget_world_sizexseq_dimcp_groupreturnc sft|}fddt|D}z t||dWnty+}ztd|d}~wwtj||dS)a Concatenates tensors from multiple processes along a specified dimension. This function gathers tensors from all processes in the given process group and concatenates them along the specified dimension. Args: x (Tensor): The input tensor to be gathered and concatenated. seq_dim (int): The dimension along which to concatenate the gathered tensors. cp_group (ProcessGroup): The process group containing all the processes involved in the gathering. Returns: Tensor: A tensor resulting from the concatenation of tensors from all processes. Raises: RuntimeError: If the gathering of tensors fails. csg|]}tqS)torch zeros_like).0_rr g/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/sampler/context_parallel.py *sz"cat_outputs_cp..)groupzGathering failed: N)dim)rranger RuntimeErrorr cat)rrr world_sizegathered_tensorser rrcat_outputs_cpsr)r rtorch.distributedrrrintrr r r rs