o Iiy @s<ddlZddlmmZddlmZdddZddZdS)N) rearrangec Cs|dur||n|}|jdtjd}|jdtjd}tj|dd}|}ttj |dtjdd}t |d|||||fS) a Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. unused_mask: (batch, seqlen), bool / int, 1 means the element is allocated but unused. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask + unused_mask. indices: (total_nnz), the indices of masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int seqused: (batch), returns the number of tokens selected in attention_mask + unused_mask. N)dimdtypeF)as_tupler)rzb s ... -> (b s) ...) sumtorchint32nonzeroflattenmaxitemFpadcumsumr) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchindicesmax_seqlen_in_batch cu_seqlensrB/home/ubuntu/.local/lib/python3.10/site-packages/hopper/padding.py unpad_inputs   rcCsF|jdd}tj||g|R|j|jd}|||<t|d|dS)a Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices that represent the non-masked tokens of the original padded input sequence. batch: int, batch size for the padded sequence. seqlen: int, maximum sequence length for the padded sequence. Return: hidden_states: (batch, seqlen, ...) rN)devicerz(b s) ... -> b s ...)b)shaper zerosrrr)rrbatchseqlenroutputrrr pad_input(s "r%)N) r torch.nn.functionalnn functionalreinopsrrr%rrrrs