o wiF@sddlmZddlZejfdejdejdededejdejf d d Zd ejd efd dZ dedejdej dejdejdeejejff ddZ ddejdedejfddZ dS))TupleNmasks num_chunks total_length max_chunksdevicereturncCsd}t|}tdd|D}tj||||f|||d} tt||D]@\} \} } tt| | D]2\} \}}t|dkrbt|d||d<|ddkrO||d<| | |d|d| d |fd q0q#| S) a Pads the provided masks to a uniform shape for batching. Args: masks (torch.Tensor): List of tensors containing attention masks for each batch. num_chunks (torch.Tensor): Tensor containing the number of chunks for each mask. total_length (int): Total sequence length for padding. max_chunks (int): Maximum number of chunks to pad each mask to. device (torch.device): Device to place the output tensor on. dtype (torch.dtype): Data type for the output tensor. Default is `torch.bfloat16`. Returns: torch.Tensor: A padded tensor of shape [B, total_length, max_num_media, max_chunks] where `B` is the batch size. ?cSsg|]}t|qS)len).0mr r d/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/vlm/mllama/model/utils.py .sz(_pad_attention_masks..)dtyperrNg)r maxtorchfull enumeratezipminfill_)rrrrrr mask_value batch_size max_num_media padded_masksidx mask_groupchunks media_idxmask chunk_countr r r_pad_attention_maskss&   &r%attention_biasrcCs||kjdd|dS)a Determines whether each row in the attention bias tensor contains masked values. Args: attention_bias (torch.Tensor): A 4D tensor of shape [B, H, S1, S2], where: - B: Batch size. - H: Number of attention heads. - S1: Length of the first sequence. - S2: Length of the second sequence. mask_value (float): The value used to represent masked positions in `attention_bias`. Returns: torch.Tensor: A 4D tensor of shape [B, H, S1, 1], containing boolean values (as a tensor) indicating if each row in the last dimension is fully masked (0 if fully masked, 1 otherwise). rdim).N)anytype_as)r&rr r r_get_full_row_masked_out_maskBsr+text_token_count text_device text_dtype vision_tokenscross_attention_masksc Cs|dusJd|jd}|jd|jdks#Jd|jd|j|jd|jdks9Jd|jd|j||jdksMJd |d |jd|j\}}}} |||d d}t|d d } |j|dd}|| 9}|j||d| j||dfS)a Generates a cross-attention mask for aligning text and vision tokens. Args: text_token_count (int): Number of tokens in the text sequence. text_device (torch.device): Device to place the output tensor on. text_dtype (torch.dtype): Data type for the output tensor. vision_tokens (torch.Tensor): Vision tokens tensor of shape [B, I, T, D] where: - B: Batch size. - I: Number of images. - T: Number of image tokens per image. - D: Dimension of each image token. cross_attention_masks (torch.Tensor): Cross attention masks of shape [B, N, I, C], where: - B: Batch size. - N: Number of text tokens. - I: Number of images. - C: Number of chunks. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - The adjusted cross-attention masks of shape [B, 1, N, I * T]. - The full row mask status tensor of shape [B, 1, N, 1]. NzVision tokens must be providedrrzAMismatch in number of images given and number of masks provided: z vs z:Mismatch between vision tokens and cross-attention masks: zText sequence length z, does not match cross-attention mask length rr )rr')rr)shapeview unsqueezer+repeat_interleaveto) r,r-r.r/r0vision_token_lengthr_ num_imagesrfull_row_mask_statusr r r_generate_cross_attention_maskXs&    r;tokensvision_token_idcCs||kjdd}|dkrtjddtjdSg}|dkr-||t|gn)t t|dD]}|||||dgq5||dt|g|dd}t |D]}|d|ddkrp||d<|d}q`tj |tjdS)a- Create a vision mask from a tensor of tokens and a vision token ID. Args: tokens (torch.Tensor): A 1D tensor of token IDs. vision_token_id (int): The ID of the vision token. Returns: torch.Tensor: A tensor containing vision masks in the format [start, end]. F)as_tuplerrr)rr) nonzerosqueezenumelremptylongappenditemr rangereversedtensor)r=r>vision_token_locations vision_masksi last_mask_end vision_maskr r rcreate_vision_mask_tensors  $   rO)r<) typingrrbfloat16Tensorintrr%floatr+rr;rOr r r rsF   -  7