o }oiM@sddlmZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZddlmZz ddlmZdZ Wn e!yodZdZ YnwddZ"ddZ#ddee$defddZ%GdddeZ&dS))deepcopy)OptionalTupleUnionN) rearrange)parallel_state)BaseInferenceContext)apply_rotary_pos_emb)get_gpt_decoder_block_spec)PackedSeqParams) SelfAttention) ModuleSpec)L2Norm)deprecate_inference_paramsis_fa_min_version)Tensor)_flash_attn_forwardTFcCs|dg}|dg}tdt|D]N}||d}||}||d}||} ||} | |} td| dD]} || |} || || |}||q6|d|kr_|||| q||fS)a Splits cumulative sequence lengths into chunks based on attention_chunk_size. Args: cu_seqlens (list[int]): List of cumulative sequence lengths. cu_seqlens_padded (list[int]): List of padded cumulative sequence lengths. attention_chunk_size (int): The maximum size of each chunk. Returns: Tuple[list[int], list[int]]: A tuple containing the new chunked cumulative sequence lengths and the new chunked padded cumulative sequence lengths. r)rangelenappend) cu_seqlenscu_seqlens_paddedattention_chunk_sizenew_cu_seqlensnew_cu_seqlens_paddedistartend start_padded end_paddedsegment_lengthnum_full_chunksj new_indexnew_index_paddedr'_/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/gpt/model/llama4_utils.pychunkify_cu_seqlens(s&          r)cCs|jd}||d||}||kr4||}tj|g|jddR|j|jd}tj||gdd}||}|j||g|jddR}|dd}|j|jddg|jddR}|S)a Pads and reshapes a tensor for chunked processing. This function takes an input tensor `x` (typically representing query, key, or value in attention mechanisms) and pads its sequence dimension (dim 0) to be a multiple of `attention_chunk_size`. It then reshapes the tensor so that the sequence dimension is split into chunks, and the chunk dimension is combined with the batch dimension. Args: x (torch.Tensor): Input tensor, expected shape [seq_length, batch_size, ...]. attention_chunk_size (int): The desired size of chunks along the sequence dimension. Returns: torch.Tensor: The reshaped tensor with shape [attention_chunk_size, num_chunks * batch_size, ...]. rrN)devicedtype)dimr) shapetorchzerosr*r+catreshape transpose contiguous)xr seq_length pad_seq_lenpad_size pad_tensor num_chunksr'r'r(chunkifyMs ( (r;vp_stagereturnc Csddlm}ddlm}t|d|d}g}|||d}t|jD]M\}}||} t|} |jduo9| d|jdk} t | j j _ | |j |jd | j j _|jr\| s\t| j j j _t| j j j _n d| j j j _d| j j j _|| q ||_|S) zGet llama4 layer specr) AttnMaskType)get_transformer_layer_offsetT)use_transformer_enginer<)r<Nr) is_nope_layerrattn_mask_type)megatron.core.transformer.enumsr>+megatron.core.transformer.transformer_layerr?r enumerate layer_specsrnope_layer_intervalLlama4SelfAttention submodulesself_attentionmodulercausalparams qk_l2_normr q_layernorm k_layernormr) configr<r>r?llama4_layer_specupdated_layer_specsoffsetidx layer_speclayer_noupdated_layer_specrAr'r'r(get_llama4_layer_spec|s,         rYcseZdZdZdfdd Z        dddded ed eed eed eeee eeffd eedeedeedee dee deede eeffddZ Z S)rHz