o }oi( @sddlZddlmZddlZddlmmZddlm Z dee de de fddZ d ee d ed e defd d Z dej de de de fddZddej de de fddZ ddeej deej de de de de f ddZdS)N)List)PackedSeqParamsnumberscapacityreturncCs t||}|dkr dS|dS)zVFinds the index of largest number that fits into the knapsack with the given capacity.r)bisect)rrindexr c/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/neva/data/sequence_packing.pysearch_for_fits r item_sizessamples max_capacityc Cst|t|ks Jdg}t|dkr|Sttt||ddd\}}t|}t|}|d|kr@td|dd|d |rng}|} t||}|dkrQn|||8}||||} || qG|||sB|S) zGreedy algorithm with binary search for the knapsack problem. Pack as many samples as possible given a maximum capacity and capacities of individual samples. Used if sequence packing is enabled. z5sample lengths and samples must have the same length.rcSs|dS)Nrr )xr r r 1sz!greedy_knapsack..)keyrzknapsack: A sample is larger z than the max_sequence_length .)lenzipsortedlist ValueErrorr popappend) rrr knapsackssorted_item_sizessorted_samplescurrent_knapsackremaining_capacityidxsampler r r greedy_knapsack#s4        r#instance_tokensnum_image_embeddings_per_tilemedia_token_indexcCs*t||k}t||d|}|S)an Predict the effective sequence length, accounting for media embeddings. Args: instance_tokens (torch.Tensor): Token tensor for a single instance. num_image_embeddings_per_tile (int): Number of image embeddings per tile. media_token_index (int): Token ID representing media. Returns: int: Effective sequence length. r)torchsumitemr)r$r%r& num_imagesseqlenr r r predict_seq_lenOs r,@pad_to_multiple_ofcCs t|}||d||}|S)zGet seqlen with paddingr)r)r$r.r+ seqlen_paddedr r r predict_seq_len_with_padding`sr0tokenslabels ignore_indexc Csg}g}g}g} dg} dg} t||D]`\} } t| ||}||d||}||}|dkrCt| d|fdd} t| d|fd|} || || |tjt| tj| j d| || | d|| | d|qtj |dd d}tj |dd d}tj |dd d}tj |tj |j d}d||dk<t| } t| } t| | | | tt| tt| dd }|||||fS) aN Convert tokens, labels, and associated inputs into a packed version with padded sequence parameters. Args: tokens (list[torch.Tensor]): List of token tensors for each instance. labels (list[torch.Tensor]): List of label tensors for each instance. num_image_embeddings_per_tile (int): Number of image embeddings per tile. media_token_index (int): Token ID representing media. ignore_index (int): Value to use for padding labels. pad_to_multiple_of (int): Sequence length will be padded to a multiple of this value. Default is 8. rrconstant)dtypedevicer)dimgthd) cu_seqlens_q cu_seqlens_kvcu_seqlens_q_paddedcu_seqlens_kv_padded max_seqlen_q max_seqlen_kv qkv_format)rr,Fpadrr'arangerintr6cat unsqueeze ones_likefloat IntTensorrmax)r1r2r%r&r3r. packed_tokens packed_labelspacked_position_idsseqlens_padded cu_seqlenscu_seqlens_paddedr$instance_labelsr+r/pad_lenpacked_loss_maskpacked_seq_paramsr r r convert_to_packedgsH          rT)r-)r typingrr'torch.nn.functionalnn functionalr@megatron.core.packed_seq_paramsrrCr r#Tensorr,r0rTr r r r s.   ,