o }oi @sddlZddlZddlmZddlmZddlmZddlm Z z ddl m Z dZ Wn e efy7dZ Ynw  d dd ZGd d d ejZdS) N) distributed) functional))average_losses_across_data_parallel_group)parallel_stateTFc st}t}t}|r*tjtjjdd}tjtjjdd}||fSfddt |D} fddt |D} t j| |dt j| |d|sZ| |<| |<tj| dd}tj| dd}||fS)a Gathers image and text features across multiple data parallel processes. This function is designed to work in a distributed environment where multiple processes are handling different portions of data. It gathers the image and text features from all processes to form a complete set of features across the entire dataset. This is crucial for calculating loss in models like CLIP, especially when the model is trained in a data parallel fashion. Parameters: image_features (Tensor): A tensor containing the image features. text_features (Tensor): A tensor containing the text features. local_loss (bool, optional): A flag to determine whether to use local loss calculation. Defaults to False. gather_with_grad (bool, optional): A flag to enable gathering with gradient computation. This is not currently working in the latest PyTorch version. Defaults to False. Returns: Tuple[Tensor, Tensor]: A tuple containing the gathered image features and text features across all processes. r)dimcg|]}tqStorch zeros_like.0_)image_featuresr`/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/multimodal/losses/clip_loss.py Gz#gather_features..crrr r ) text_featuresrrrHr)group) rget_data_parallel_world_sizeget_data_parallel_rankget_data_parallel_groupr catrnn all_gatherrangedist) rr local_lossgather_with_graddata_parallel_world_sizedata_parallel_rankdata_parallel_groupall_image_featuresall_text_featuresgathered_image_featuresgathered_text_featuresr)rrrgather_features s"r&cs0eZdZdZ   dfdd ZddZZS)ClipLossuE A custom loss module for CLIP (Contrastive Language–Image Pretraining) training. This module is specifically designed for calculating the loss in CLIP model training, supporting features like local loss calculation, gradient gathering, and label caching for efficiency in a distributed training setup. Parameters: local_loss (bool, optional): If True, calculates loss locally on each data parallel process. Defaults to False. gather_with_grad (bool, optional): If True, gathers gradients during loss calculation. Currently not functional in the latest PyTorch version. Defaults to False. cache_labels (bool, optional): If True, caches labels for reuse in subsequent iterations, improving performance. Defaults to False. Attributes: world_size (int): The size of the data parallel group (number of processes). rank (int): The rank of the current process within the data parallel group. Methods: forward(output_tensor): Computes the loss given the model's output tensor. This involves gathering features across processes, computing logits, and calculating the final cross-entropy loss. Fcs@t||_||_||_d|_i|_t|_ t |_ dS)InitrN) super__init__rr cache_labelsprev_num_logitslabelsrr world_sizerrank)selfrrr+ __class__rrr*rs  zClipLoss.__init__cCs(|\}}}|j}|jdkr5t|||j|j\}}|jr*|||j}|||j} n|||j}|j} n|||j}|||j} |jd} |j| ksR||jvrvt j | |t j d} |jdkrj|jrj| | |j } |j ru| |j|<| |_n|j|} t|| t| | d} t| g} | d| ifS)zForward for lossr)devicedtypeloss)r4r.r&rrTshaper,r-r arangelongr/r+F cross_entropyr)r0 output_tensorrr logit_scaler4r"r#logits_per_imagelogits_per_text num_logitsr- total_loss reduced_lossrrrforwards4        zClipLoss.forward)FFF)__name__ __module__ __qualname____doc__r*rE __classcell__rrr1rr'Wsr')FF)r torch.distributed.nntorch.nnrrrrr<2nemo.collections.nlp.modules.common.megatron.utilsr megatron.corerHAVE_MEGATRON_CORE ImportErrorModuleNotFoundErrorr&Moduler'rrrrs        7