o }oi<@sddlmZmZmZddlZddlmmZddlm Z mZddl m Z ddl m ZddlmZmZGdddeZGdd d eZGd d d eZd e d e fddZd e de fddZddZdS))DictLiteralTupleN)Tensornn) all_gather)MaskedTokenLossReductionMegatronLossReductionc szeZdZdZddedededdffd d Zd eeej fd ej de ej eeej fffd dZ dej fddZ Z S)BERTLossReductionzYBert Loss Function. when add_sop_loss = False, only calculate Masked token loss. FTvalidation_step val_drop_last add_sop_lossreturnNcs4t||_||_||_|st|||_dSdS)N)super__init__r r r rmlm)selfr r r  __class__R/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/bert/loss.pyrs zBERTLossReduction.__init__batch forward_outc Cs|d|d<|js|j||dSddlm}|d|d}}|dus)Jd|}|dkr@t||d }t||d}ntd ||} t | g} | d | ifS) zqPerform Loss calculation on batch. Currently, Context parallelism is not supported for SOP loss. loss_masklm_lossrparallel_state binary_logitsNzAttempting to calculate Sentence Order Prediction Loss but SOP logits are not provideds, Please Make sure you have added binary head. is_randomz$CP is not supported for SOP loss yetavg) r rforward megatron.corerget_context_parallel_world_sizesentence_order_prediction_lossmasked_token_with_zeroNotImplementedError)average_losses_across_data_parallel_group) rrrrlm_loss_ sop_logitscp_sizesop_loss_for_ublm_loss_for_ub loss_for_ub reduced_lossrrrr!(s     zBERTLossReduction.forwardcC|rVd|dvrdd|D}t|}|Sddlm}dd|D}t|dkr4t|jddn tjddgtj d }tj j ||j d d d |d|d }|Stjdtj d S)Taken from: https://github.com/NVIDIA/NeMo/blob/main /nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py#L535-L552 .r rcSg|]}|dqSr r.0xrrr Oz,BERTLossReduction.reduce..rcS$g|]}|dddkr|dqSloss_sum_and_ub_sizerrrr3rrrr6UdimdeviceTwith_context_parallelgrouprtorchcatmeanr"rlenvstacksumtensorcudacurrent_device distributed all_reduceget_data_parallel_grouprlosses_reduced_per_micro_batchr lossrr:rrrreduceI(    zBERTLossReduction.reduce)FTT)__name__ __module__ __qualname____doc__boolrrstrrFrrr!rU __classcell__rrrrr s   !r c seZdZdZ     ddeded ed ed ed d f fdd Zdee e j fde j d e e j ee e j fffddZ d e j fddZZS)HardNegativeRankingLossaE This loss uses hard-negative samples. The difference of this loss to the default MultipleNegativesRankingLoss from Sentence Transformers is that the latter shares the hard negatives as negatives for all examples, whereas this loss uses hard negatives exclusively for the example they are associated. FTr2r>r r num_hard_negativesscalelabel_smoothingrNcs4t||_||_||_||_tj|d|_dSN)rb) rrr r r`rarCrossEntropyLosscross_entropy_loss)rr r r`rarbrrrrqs z HardNegativeRankingLoss.__init__rrcCsvddlm}|}|dkrtd|jdd|j}d|j}|jd|}||}t dd|D} t d d|D} | jd| jddks[Jd | jd| jd| jd| jd|ksvJd | jd| jd|| j} | dd| | d|| d} tj| | d d  | d|} tj| dtj| jd}| |j9} || |}t|g}|d|ifS)NrrrCP is not supported for  yet.cSr1rrr4itemrrrr6r7z3HardNegativeRankingLoss.forward..cSsg|]}|ddqS)rNrrjrrrr6z {} % {} > 0z {} / {} != {}r<dtyper@r )r"rr#r&rr`shapechunkrFstackrGformatrepeatreshaperKzeroslongr@rarer')rrrrr*num_tensors_per_examplecurrent_train_n_passages batch_sizechunksquerykey query_shaperepeated_queryscoreslabelsce_lossr.rrrr!s0    4    zHardNegativeRankingLoss.forwardcCr/)r0r rcSr1r2rr3rrrr6r7z2HardNegativeRankingLoss.reduce..rcSr8r9rr3rrrr6r;r<r>r?TrArCrrErRrrrrUrVzHardNegativeRankingLoss.reduce)FTrr_r>)rWrXrYrZr[intfloatrrr\rFrrr!rUr]rrrrr^hs8   !r^cseZdZdZ       dded ed ed ed ed ededddffdd ZddZ de e e j fde j dee j e e e j fffddZde j fddZZS),BERTInBatchExclusiveHardNegativesRankingLossa This loss uses in-batch negative samples + hard-negative samples. The difference of this loss to the default MultipleNegativesRankingLoss from Sentence Transformers is that the latter shares the hard negatives as negatives for all examples, whereas this loss uses hard negatives exclusively for the example they are associated. This loss is also capable of using in-batch negatives from all ranks during training. FTrr>localr r r`rarbglobal_in_batch_negatives backprop_type)rglobalrNcs@t||_||_||_||_tj|d|_||_ ||_ dSrc) rrr r r`rarrdrerr)rr r r`rarbrrrrrrs  z5BERTInBatchExclusiveHardNegativesRankingLoss.__init__csddlm}|jdkr4fddt|D}t||d||<t j |dd}|St }t j |dd}|S)Nrrrcsg|]}tqSr)rF zeros_like)r4_ local_tensorrrr6s zhBERTInBatchExclusiveHardNegativesRankingLoss._gather_global_in_batch_representations..rCr<) r"r contiguousrrangeget_data_parallel_world_sizeall_gather_no_backproprQget_data_parallel_rankrFrGall_gather_with_backprop)rrrglobal_tensorsrrr'_gather_global_in_batch_representationss     zTBERTInBatchExclusiveHardNegativesRankingLoss._gather_global_in_batch_representationsrrcsPddlm}|}|dkrtd|jd|jr"|js"||}d|j}|j d|}| |t ddD}t d dD}fd dt |jD} t ||dd} t |dt| ddt | jd d j} t j| | gdd } | d d} | |j9} t jt t| t j| jd} || | }t|g}|d|ifS)NrrrrfrgrhcSr1rirrjrrrr6r7zHBERTInBatchExclusiveHardNegativesRankingLoss.forward..cSr1rrrjrrrr6r7cs$g|]tfddDqS)csg|]}|dqS)rhrrjirrr6rlzSBERTInBatchExclusiveHardNegativesRankingLoss.forward...)rFrr)r4r{rrr6srm)axisgg?rnr )r"rr#r&rrr rr`rprqrFrrrmm transposemultiply unsqueezertrIrKTrGclamprarLrwr@rer')rrrrr*rxrzqueries positives hard_negspos_in_batch_negs_scoreshard_negs_scoresrrrr.rrrr!sD            z4BERTInBatchExclusiveHardNegativesRankingLoss.forwardcCr/)r0r rcSr1r2rr3rrrr6#r7zGBERTInBatchExclusiveHardNegativesRankingLoss.reduce..rcSr8r9rr3rrrr6)r;r<r>r?TrArCrrErRrrrrUrVz3BERTInBatchExclusiveHardNegativesRankingLoss.reduce)FTrrr>Fr)rWrXrYrZr[rrrrrrr\rFrrr!rUr]rrrrrsF    -rrLmaskcCsZ|}|}|dkrt|dd}|St|d|d|}|S)aSCalculate masked token loss with consideration of possible NaN. Sometimes when the number of tokens is very small, none of the tokens get masked for prediction. In that case loss mask is all zeros i.e Happens when the entire batch is masked out (Practically when MBS=1 or 2, and the number of tokens in each batch is < 7 ) rrmr>)rrKrFviewru)rLrlossesrrTrrrr%<s "r%sentence_ordercCs.|dd}|d}tj||dd}|S)z)Calculate sentence order prediction loss.rmrh) ignore_index)rrF cross_entropy)rLrrrTrrrr$Ls r$cCsNddlm}tdd|D}tjj||d|tjj|d}|S)z*Reduce a tensor of losses across all GPUs.rrcSsg|] }|dqSr)clonedetachr)r4rTrrrr6Ysz=average_losses_across_data_parallel_group..rC)r"rrFrGrOrPrQget_world_size)rraveraged_lossesrrrr'Us r')typingrrrrFtorch.nn.functionalr functionalrrtorch.distributedrrtorch.distributed.nn.functionalr nemo.lightning.megatron_parallelrr r r^rr%r$r'rrrrs  NX|