o iI@spddlmZmZmZmZmZddlZddlmm Z ddlmZddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*ddl+m,Z,e*rddl-m.Z.ddl/m0Z0m1Z1ndZ.e)rddl2m3Z3m4Z4nd\Z4Z3e$rddl5m6Z6ddl7m8Z8e%9e:Z;ddZde?dej>fd d!Z@ "d]d#ejAd$ej>d%ej>d&ej>d'eej>d(eBd)eBfd*d+ZCGd,d-d-ejAZDGd.d/d/ZEd0ej>d1e?fd2d3ZFd4d5ZGd6d7ZHeIe.e3e4fZJd8d9ZKGd:d;d;ejAZLGdd?d?ejAZNGd@dAdAedBdCZOGdDdEdEejAZPGdFdGdGejAZQGdHdIdIejAZRGdJdKdKejAZSGdLdMdMeZTe"GdNdOdOeZUGdPdQdQejAZVe"GdRdSdSeUZW  T d^dUeej>eXej>dfdVee?d'eej>deej>e?ffdWdXZYGdYdZdZeUeZZgd[Z[dS)_)AnyCallableOptional TypedDictUnionN)nn)ACT2FN)Cache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)deprecate_kwarg)is_causal_conv1d_availableis_mamba_2_ssm_available)GraniteMoeHybridConfig)selective_state_update)mamba_chunk_scan_combined mamba_split_conv1d_scan_combined)causal_conv1d_fncausal_conv1d_updateNN) BlockMask)make_flex_block_causal_maskcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2r1{/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py rotate_halfAsr3cCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer3)qkcossin position_ids unsqueeze_dimq_embedk_embedr1r1r2apply_rotary_pos_embHs  r= hidden_statesn_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r+expandreshape)r>r?batchnum_key_value_headsslenhead_dimr1r1r2 repeat_kvcs 0rGmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr(r r')r*dtype)ptrainingr)rGnum_key_value_groupsr,matmul transposer+r functionalsoftmaxfloat32torQrOrS contiguous) rIrJrKrLrMrNrOkwargs key_states value_states attn_weights causal_mask attn_outputr1r1r2eager_attention_forwardos & rbcseZdZdZdedeffdd Zedddd  dd ej d e ej de ej de e de de ej de eej ej fdeej e ej e eej ffddZZS)GraniteMoeHybridAttentionz=Multi-headed attention from 'Attention Is All You Need' paperconfig layer_idxcs t||_||_|durtd|jjd|j|_|j |_ |j |_ |j |j |_ |j |_ |j |j |_d|_|j|_|j |j |j krUtd|j d|j dtj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |jd|_dS)NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).bias)super__init__rdrelogger warning_once __class____name__attention_dropout hidden_sizenum_attention_heads num_headsrFrDrT is_causalattention_multiplierrN ValueErrorrLinearattention_biasq_projk_projv_projo_projselfrdrerlr1r2ris2  z"GraniteMoeHybridAttention.__init__past_key_valuepast_key_values4.58new_nameversionNFr>rMr9 use_cachecache_positionposition_embeddingsr@cKsD|\} } } ||} ||} ||}| | | |j|jdd} | | | |j|jdd} || | |j|jdd}|durF|nd\}}|durWt | | ||\} } |durl|||d}| | ||j |\} }t }|j jdkrzt|j j}||| | ||f|jsdn|j|jd|\}}|| | d}||}||fS) Nrr(r$)r8r7reagerrH)rOrNr')sizerwrxryviewrqrFrVrDr=updatererbrd_attn_implementationrrSrnrNrz)r|r>rMr9rrrrr\bszq_len_ query_statesr]r^r7r8 cache_kwargsattention_interfacerar_r1r1r2forwards>        z!GraniteMoeHybridAttention.forward)NNNFNN)rm __module__ __qualname____doc__rintrirr,Tensorr LongTensorr booltupler __classcell__r1r1r}r2rcs6  rcc @seZdZdZdZejdfdefddZ ddej dej d e d e e e efd eej ej ff d d ZdejfddZdd e e d e fddZdS) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNrdc s0|j|_d|_|j}|j}g|_g|_g|_t|jD]^}|j|dkrS|jt j |j |j d|j |||dg7_|jt j |j|j||dg7_q|jt jggdg7_|jt jggdg7_|j|qfddt|jD|_fddt|jD|_dS) NFmambar(devicerQrc g|] }tjggdqSrr,tensor.0r batch_sizerr1r2  z=HybridMambaAttentionDynamicCache.__init__..crrrrrr1r2rr)layers_block_typehas_previous_state mamba_d_conv mamba_d_state conv_states ssm_statestransformer_layersrangenum_hidden_layersr,zeros mamba_expandromamba_n_groups mamba_n_heads mamba_d_headrappend key_cache value_cache)r|rdrrQrconv_kernel_sizessm_state_sizeir1rr2risB   z)HybridMambaAttentionDynamicCache.__init__r]r^rerr@cCsz|j|jddkr||j|<||j|<ntj|j||gdd|j|<tj|j||gdd|j|<|j||j|fS)Nr'rr(r))rr+rr,r-)r|r]r^rerr1r1r2rs   z'HybridMambaAttentionDynamicCache.updatebeam_idxcCstt|jD]V}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<qdS)zDReorders the cache for beam search, given the selected beam indices.rN) rlenrr index_selectrZrrr)r|rrerr1r1r2 reorder_cache*s     z.HybridMambaAttentionDynamicCache.reorder_cachercCs:||jvr |jdn|}t|j|krdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rrP)rrrr+)r|rer1r1r2get_seq_length7sz/HybridMambaAttentionDynamicCache.get_seq_lengthN)r)rmrrris_compileabler,float16rrirrrdictstrrrrrrrr1r1r1r2rs$ +  r input_tensorpad_sizecCsHt|jdkrddddd|ddfnddd|ddf}tjjj||dddS)z Padding x tensor with `pad_size` on the seq_len dim (dim=1) Assumes that we only have tensors of either size 4 or 3 rconstant)moderL)rr+r,rrWpad)rr pad_shaper1r1r2pad_tensor_by_sizeCs2rcCsXt||}t|jdkr||jdd||jdS||jdd||jd|jdS)z Padding input_tensor with `pad_size` on the seq_len dim (dim=1) and simultaneously splitting it into chunk sequences. Assumes that we only have tensors of either size 4 or 3 r rr'r()rrr+rB)rr chunk_sizer1r1r2reshape_into_chunksNs rcCs|d}|djg||R}tjtj|||jtjddd}||d}tj|dd}tjtj|||jtjddd}||tj }|S)zo More stable segment sum calculation. Uses cumulative sums and masking instead of direct subtractions. r'.NrdiagonalrrPr)) rrAr,trilonesrr masked_fillcumsuminf)rrmask tensor_segsumr1r1r2 segment_sumbs   rcCsN|dur%|jddkr%|jddkr%|j}||dddddf|}|S)zm Tunes out the hidden states for padding tokens, see https://github.com/state-spaces/mamba/issues/66 Nrr)r+rQrZ)r>rMrQr1r1r2apply_mask_to_padding_statesys$ rc seZdZdZdedeffdd Z    ddejde e d e ej d e ejd e ej f d d Z   dde e d e ej d e ejfddZ    dde e d e ej d e ejd e ej fddZZS)GraniteMoeHybridMambaLayeruP Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) The are a few differences between this and Mamba2Mixer: - The variable use_precomputed_states is slightly different due to the hybrid cache structure - There's a few non-obvious bugs fixed with batching in the slow path that exist in main - Some extra variables that our layer doesn't need have been removed - We ported most of the refactors in https://github.com/huggingface/transformers/pull/35154, which is (as of Dec 18, 2024) unmerged rdrecst|j|_|j|_|j|_|j|_t |j |j|_ ||_ |j |_|j|_t|j|_|j|_|j|_|j|_|j|_|j|_dtdf|_d|_d|_ |j d|j|j|_!t"j#|j!|j!|j |j|j!|jdd|_$|j |j!|j}t"j%|j||jd|_&t"'t()|j|_*t(+d|jd}t"'t(,||_-t.|j |jd |_/t"'t()|j|_0t"j%|j |j|jd|_1t2st34d dSt34d dS) NrHrgMbP?g?r(r) in_channels out_channelsrg kernel_sizegroupspaddingrfepsaThe fast path is not available because one of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1dzOThe fast path for GraniteMoeHybrid will be used when running the model on a GPU)5rhrirrqrorrrrrrintermediate_sizeremamba_conv_bias use_conv_bias hidden_act activationractmamba_proj_biasuse_bias rms_norm_epslayer_norm_epsilonrn_groupsrrFmamba_chunk_sizerfloattime_step_limit time_step_min time_step_maxconv_dimrConv1dconv1druin_proj Parameterr,rdt_biasarangelogA_logGraniteMoeHybridRMSNormGatednormDout_projis_fast_path_availablerjrk)r|rdreprojection_sizeAr}r1r2risX    z#GraniteMoeHybridMambaLayer.__init__Nr> cache_paramsrrMseq_idxcCst||}||}|j\}}} |j|j} |duoD|joD|dkoD|j|jjd|j|jjdko8|knoD|duoD|ddk} | r)| dj |j |j |j gdd\} } }t| |j|j|jj d|jj|j} tj | |j | | gdd\}}}t|j }|ddddfdddddfd|j|jjtjd}|dddddfdd|j}|jddddfd|j}|jddddfd|j}|||j|jd|j}|||j|jd|j}|||j |j}t|j|j||||||d|dd }|||j |j}||| }| |ddddf}|St|j }|j!d td fkr>ind |j!i}|j"r||dur|t#||jj d|jj|j|f|j|j$||j|jj|jj%|j j|j j|j|jd d d |}|S|j |j |j |j gdd\} } }|dur| &dd}t'j()||j*|jddf}|j|j+||jdvr|,|| &dddd|f&dd} nt-| &dd|jj d|jj|j|d&dd} t| |} tj | |j | | gdd\}}}t.|||d|j||||||jd||||jdf|j$|jd|d|jdd|\}}|dur:|dur:|j|j+||||d}||| }| |}|S)Nrrr'r).rQT)zr dt_softplusrHrdt_limitF) rrrrrmsnorm_weight rmsnorm_epsoutproj_weight outproj_biasheaddimngroupsnorm_before_gatereturn_final_statesr()siluswish)r.weightrgrr)rrr rrrr )/rrr+rrrrrersqueezesplitrrrqr#rrrgrr,exprrrArFrZrYrrrrrrrrSr!rvariance_epsilonrVrrWrrcopy_rr"r )r|r>rrrMrprojected_statesrseq_lenrgroups_time_state_sizeuse_precomputed_statesgatehidden_states_B_CdtBCrrrhidden_states_reshapedoutdt_limit_kwargshidden_states_B_C_transposedr scan_output ssm_stater1r1r2cuda_kernels_forwards          <"  ^" V   $        z/GraniteMoeHybridMambaLayer.cuda_kernels_forwardc3 s|j\}}}|j}t||}|} | jjjjgdd\} } } |duoQ|joQ|dkoQ|j j jd|j j jdkoE|knoQ|duoQ|ddk} | r|j j j ddd|j j <| dddddf |j j j|j j dddddf<|j j j jjjd}tj|jjddd} jr| jj} | } n8|dur| dd}tj|j|jddf}|j j || dddd|fdd} t| |} tj| jjjjjgdd\}}}tj ! }| r[|j j j}| dddddfddddf} | dd"|| jdj#} j$d "j$jdj#}tjj%| | | j} t&| j'dj'd} |d "jj#jj tj(d }t| d |j |d}|)|jdddddf}|"|jjj|jd*}|)|d|jd}| d |ddddf}|)|dj#}||d j |d}|j j |j j |||)|jdddddf}|"|jjj|jd*}|)|d|jd}|j j j |j|jd }|+|jj#j}|+|jjd}t,||}|+|jj#}j-d "j-jdj#}||| |j}|)|dddddf}ntj%| j$} t&| j'dj'd} |)||dj#!}|)||dj!}|)||dj!}|j.jjdjd }|j.jjdjd }j/|j/j/j-d t0|}|| d }| |j| }fdd||||fD\}}}}|1dddd}tj2|dd}tt3|} |dddddddddddf|dddddddddddf}!|!jdd}"|"d | 1dddddd }#|#jdd}$|$d |dddddfjdd}%t|ddddddddf|}&||&1ddddd }'|'ddddf|d jdd}(| r|j j ddddfj |(jd})n t4|(ddddf})tj5|)|(gdd}(tt3tj|dddddddfd}*|*dd}*|*d |(ddddddfjdd}+|+ddddf|+dddf}(},t|}-|ddddf|(ddddddf}.|-1dddd}/|.d|/d }0|%|0}|)|djj#}||}dkrB|ddd|ddddf}|)||d}|,dur\|dur\|j j |,6|| }17|1 |}2|2S)Nr'r)rr)shiftsdimsrr(.r).NNr r)r* output_sizecsg|] }t|jqSr1)rr)rtrr|r1r2rsz.r rrP)rr)8r+rQrrrrrrqrrrerrollrZrrrr,sumrrrgrrVrrWrrrrrrrrrArFrsoftplusclamprrYrBr[rbmmrrepeat_interleaverrpermuterr zeros_liker-rr)3r| input_statesrrrMrrrrQrr!r"r#r rr)r>r$r%r cache_devicerdAdBdBxrssm_states_reshaped C_reshapedyr D_residualA_cumsumLG_intermediateGM_intermediateMY_diag decay_statesB_decaystatesprevious_states decay_chunk new_statesr+state_decay_outC_times_statesstate_decay_out_permutedY_offr*contextualized_statesr1r1r2 torch_forward~s      @   ,  $"$ $  $P  &*"&0 (& *   z(GraniteMoeHybridMambaLayer.torch_forwardcKstrd|jjjjvr||||||S|durtd|j}|dur@|jddkr@|jddkr@||dddddf |}| ||||S)Ncudaz\`seq_idx` support requires fast path support. Please install `mamba_ssm` and `causal_conv1d`rr) rrrrtyper,NotImplementedErrorrQr+rZrU)r|r>rrrMrr\rQr1r1r2rMs $ z"GraniteMoeHybridMambaLayer.forward)NNNN)NNN)rmrrrrrrir,rrrr IntTensorr,rUrrr1r1r}r2rsV D . Srcs(eZdZdfdd ZdddZZS) rư>cs&ttt||_||_dSrrhrirrr,rrrr|rorr}r1r2ries  z%GraniteMoeHybridRMSNormGated.__init__NcCsj|j}|tj}|dur|tj|tj}|djddd}|t ||j }|j ||SNr(r'T)keepdim) rQrZr,rYrrWrpowmeanrsqrtrr)r|r>r! input_dtypevariancer1r1r2rjs z$GraniteMoeHybridRMSNormGated.forwardrZr)rmrrrirrr1r1r}r2rdsrcs<eZdZdZdeffdd ZdejdejfddZZ S) GraniteMoeHybridMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. rdcsZt|j|_|j|_t|j|_tj |j|jddd|_ tj |j|jdd|_ dS)Nr(Frf) rhriro input_sizeshared_intermediate_sizerrrrru input_linear output_linearr|rdr}r1r2ris  zGraniteMoeHybridMLP.__init__r>r@cCs<||}|jddd}||d|d}||}|S)Nr(r'r)rr)rhchunkrri)r|r>chunked_hidden_statesr1r1r2rs  zGraniteMoeHybridMLP.forward) rmrrrrrir,rrrr1r1r}r2revs rec@s@eZdZUdZejed<ejed<eed<eed<ejed<dS)GraniteFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_krN) rmrrrr,r__annotations__rrYr1r1r1r2rms   rmF)totalcs.eZdZdfdd ZddZddZZS) GraniteMoeHybridRMSNormrZcs&ttt||_||_dS)zF GraniteMoeHybridRMSNorm is equivalent to T5LayerNorm Nr[r\r}r1r2ris  z GraniteMoeHybridRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||Sr]) rQrZr,rYr_r`rarr)r|r>rbrcr1r1r2rs  zGraniteMoeHybridRMSNorm.forwardcCst|jjd|jS)Nz, eps=)rrr+r)r|r1r1r2 extra_reprsz"GraniteMoeHybridRMSNorm.extra_reprrd)rmrrrirrurr1r1r}r2rtsrtcs6eZdZdedededdffdd Zdd ZZS) GraniteMoeHybridParallelExperts num_expertsrfr/r@Ncs6ttt||||_||_||_||_ dS)a Initialize the GraniteMoeHybridParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) rhrirrr,emptyrrwrfr/)r|rwrfr/r}r1r2ris  z(GraniteMoeHybridParallelExperts.__init__cCsP|j|dd}g}t|jD]}|t|||j|qtj|dd}|S)a Forward pass of the GraniteMoeHybridParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rr)) rrrwrFlinearrr,r-)r|inputs expert_size input_list output_listrresultsr1r1r2rs  z'GraniteMoeHybridParallelExperts.forwardrmrrrrirrr1r1r}r2rvsrvcs2eZdZdededeffdd ZddZZS)GraniteMoeHybridTopKGatingrfrwtop_kcs2t||_||_||_tj||dd|_dS)a Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. FrfN)rhrirwrfrrrulayer)r|rfrwrr}r1r2ris z#GraniteMoeHybridTopKGating.__init__cCs||}|j|jdd\}}tj|dd|}tj|d|j g|j |j d}| d|d}| d}|}|} | d\} } | j|jdd} |}|| } | | | ||fS)Nrr)rrQrtrunc) rounding_mode)rrtopkrr,rXtype_asrrrwrQrscatterlongr3tolistflattensortdiv)r|r>logits top_k_logits top_k_indices top_k_gatesrgatesr| top_k_expertsrindex_sorted_experts batch_index batch_gatesr1r1r2rsz"GraniteMoeHybridTopKGating.forwardrr1r1r}r2rsrcs.eZdZdZdeffdd ZddZZS)GraniteMoeHybridMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. rdcslt|j|_|j|_t|j|_t|j |j|jd|_ t|j |j|j|_ t |j|j |j d|_dS)Nr()rfrwr)rhrirorfrrrrrvnum_local_expertsrhrirnum_experts_per_tokrouterrjr}r1r2ri#s    zGraniteMoeHybridMoE.__init__cCs|\}}}|d|}||\}}}}} ||} || |} | jddd} || d| d} || |} | |dddf} tj|||j f| j | j d}| d|| }| |||j }|| fS)a Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. r'r(r)rrNr)rrBrrhrkrrir,rrfrQr index_addr)r| layer_inputrlengthemb_sizerrrr| router_logits expert_inputsr>rlexpert_outputsr layer_outputr1r1r2r6s   zGraniteMoeHybridMoE.forward)rmrrrrrirrr1r1r}r2rsrcseZdZdedeffdd Zedddd dd ejd e ejde e d e e de e de ej de e de e ejejfdeede eje e ejejfffddZZS)GraniteMoeHybridDecoderLayerrdrecst|j|_d|_|jdkrt||_t|j|jd|_ t|j|jd|_ |j |_ t ||_ d|_|j|dkrBt|||_nt|||_|j||_t|dddk|_dS)Nrrrr)rhriro self_attnrrblock_sparse_moertrinput_layernormpost_attention_layernormresidual_multiplierre shared_mlprrrrc layer_typegetattr has_expertsr{r}r1r2riWs      z%GraniteMoeHybridDecoderLayer.__init__r~rrrNFr>rMoutput_attentionsrroutput_router_logitsrr\r@c  Ks|} ||}|jdur|jd||||d| }d} n|jd|||||||d| \}} | ||j}|} ||}|jrN||\} } | ||}n||}d} | ||j}|f}|rf|| f7}|rm|| f7}|S)a0 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs.Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. N)r>rrrM)r>rMrrrrrr1)rrrrrrrr)r|r>rMrrrrrrr\residualself_attn_weightsmoe_hidden_statesroutputsr1r1r2rnsL&        z$GraniteMoeHybridDecoderLayer.forward)NNFFNFN)rmrrrrrirr,rrr rrrrrm FloatTensorrrr1r1r}r2rVs>    rcsJeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ fddZ ZS) GraniteMoeHybridPreTrainedModelrdmodelTrrFcst|t|tr|jjjd|jjdt|t r9|j j dt t d|jd|j_|jj ddSt|trG|jj ddSdS)NrH)r`stdg?r)rh _init_weights isinstancervrdatanormal_rdinitializer_rangerrfill_r,rrrqrrr)r|rIr}r1r2rs    z-GraniteMoeHybridPreTrainedModel._init_weights)rmrrrrrbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph _is_statefulrrr1r1r}r2rs rcsDeZdZUejed<ddeffdd Zee ddZ Z S) GraniteMoeHybridRotaryEmbeddinginv_freqNrdcstt|drt|jtr|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typerWdefaultrF) persistent)rhrihasattrrrrgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrdr rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r|rdrrr}r1r2ris   z(GraniteMoeHybridRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) Nrr'rmpscpuF) device_typeenabledr(r)r )rrrAr+rZrrrWrr,autocastrVr-r7rr8rQ) r|r.r9inv_freq_expandedposition_ids_expandedrfreqsembr7r8r1r1r2rs0&z'GraniteMoeHybridRotaryEmbedding.forwardr) rmrrr,rrrrrino_gradrrrr1r1r}r2rs rcs4eZdZdeffdd Zee           d!deej deej deej dee e e ejfd eejd eed eed eed eedeedeej deede eeffddZ d"de ej dfdej dej de d ef ddZedej dededejdej def ddZdd ZZS)#GraniteMoeHybridModelrdcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_d|_j|_j|_j|_|j|j|_j|_j|_j|_|jdkr]tnd|_|dS)Ncsg|]}t|qSr1)r)rrerdr1r2r z2GraniteMoeHybridModel.__init__..rFrope)rhri pad_token_id padding_idx vocab_sizer Embeddingro embed_tokens ModuleListrrlayersrtrrgradient_checkpointingembedding_multiplierrprqrFr rope_thetaposition_embedding_typer rotary_emb post_initrjr}rr2ris$  zGraniteMoeHybridModel.__init__N input_idsrMr9r inputs_embedsrroutput_hidden_statesr return_dictrr\r@c  Ks2|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |du|duAr4td|jrC|jrC|rCt dd}|durL| |}||j }|r\|dur\t d| durx|durh| nd} t j| | |jd|jd} |dur| d}|||| ||}||| }|}d}|jdur|||}|rdnd}|rdnd}| rdnd}|jD]D}|jd kr|n|}|r||f7}||f||||| | |d | }|d}|r|ddur||df7}| r|d dur||d f7}q||}|r||f7}|r|jsd |_t|||||d S)Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzGraniteMoeHybrid requires an initialized `HybridMambaAttentionDynamicCache` to return a cache. Because one was not provided, no cache will be returned.rrrr1r)rMrrrrrrr'T)last_hidden_staterr> attentionsr)rdrrruse_return_dictrtrrSrjrkrrrr,rr+rr4_update_causal_mask_update_mamba_maskrrrrrr)r|rrMr9rrrrrrrrr\past_seen_tokensr` mamba_maskr>rall_hidden_statesall_self_attnsall_router_logits decoder_layer layer_mask layer_outputsr1r1r2rs                   zGraniteMoeHybridModel.forwardFr%rc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) Nflash_attention_2rHflex_attentionrFsdpa)rpast_key_values_length is_trainingrr')sequence_length target_lengthrQrr)rVxpunpu)rdranyrr,rr&rrr _ignore_causal_mask_sdparSrQr+get_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrrWfinfomin_unmask_unattended) r|rMrrrrrusing_compilable_cacherQrrr` min_dtyper1r1r2rsT           z)GraniteMoeHybridModel._update_causal_maskrrrQrc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr) fill_valuerQrrrrr'r)r*r,rrfullrtriurrBrAcloner+rZr) rMrrrQrrr\r`r mask_length padding_maskr1r1r2r s,  $ 6  zKGraniteMoeHybridModel._prepare_4d_causal_attention_mask_with_cache_positioncCs.|}|ddks|durt|dkrd}|S)zv No need for zeroing states when 1. Cached forward 2. Attending to all inputs rNr)r,all)r|rMrrr1r1r2rs"z(GraniteMoeHybridModel._update_mamba_mask) NNNNNNNNNNN)F)rmrrrrirrrr,rrrr listrrrrmrrrr staticmethodrrQr rrr1r1r}r2rs      {  D 6rr( gate_logitsrwcs|dus t|ts dSt|tr#|djtjfdd|Ddd}tjjj|dd}tj||dd\}}tjj ||}|durStj | dd} tj |dd} nm|j \} } |j d| | } |dddddddf | | | ||fd||}tj| |ddtj|dd} |ddddddf | | | |j dfd|j d}tj||ddtj|dd} |jjdur|jjnd}|j dt|}t| dd|||j df| d}||S)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrcsg|]}|qSr1)rZ)r layer_gatecompute_devicer1r2r?rz,load_balancing_loss_func..r)r'r)rrrr,r-rrWrXrone_hotr`rr+rArBrZr3indexrr4)rrwrrMconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expertrrrexpert_attention_mask router_per_expert_attention_mask device_indexrank overall_lossr1rr2load_balancing_loss_funcsF    &r-c seZdZdgZdeffdd Ze             ddeej deej d eej d ee e e ejfd eejd eej d eedeedeedeedeedeej de eej fde eeffddZ      dddZZS)GraniteMoeHybridForCausalLMzlm_head.weightrdcsXt|t||_|j|_tj|j|jdd|_|j |_ |j |_ |j |_ | dS)NFrf)rhrirrrrrurolm_headrouter_aux_loss_coefrrwrrrjr}r1r2rivs   z$GraniteMoeHybridForCausalLM.__init__NrrrMr9rrlabelsrrrrrrlogits_to_keepr@cKs|dur|n|jj}| dur| n|jj} | dur| n|jj} | dur$| n|jj} |jd|||||||| | | | d |}|d}t| trKt| dn| }| |dd|ddf}||jj }d}|dury| }|j ||fd|jj i|}d}| rt| r|jn|d|j|j|}|dur||j||j7}| s|f|dd}| r|f|}|dur|f|S|St||||j|j|j|jdS) ax labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, GraniteMoeHybridForCausalLM >>> model = GraniteMoeHybridForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrMr9rrrrrrrrrrr'r)lossaux_lossrrr>rrr1)rdrrrrrrrslicer/logits_scalingr loss_functionrr-rrwrr0rZrrrr>r)r|rrMr9rrr1rrrrrrr2r\rr> slice_indicesrr3r4outputr1r1r2rsx(   z#GraniteMoeHybridForCausalLM.forwardTc Ks:|du} | s5|dus|d|jdkr"|dd|jd df}n#|jd|jdkr4|dd|f}n|rEt|j|jd|j|jd}|durn|durn|dd}||dkd| sn|dd|jd df}|dury| ryd|i} nd|i} | |||||d| D] \} } | | vr| | | <q| S)Nr'rrrrr)r9rrrMr) r+rrdrQrrr masked_fill_r[ritems) r|rrrMrrr9rr\ empty_past_kv model_inputsrKrLr1r1r2prepare_inputs_for_generationsB     z9GraniteMoeHybridForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNNNr)NNNNNT)rmrr_tied_weights_keysrrirrr,rrrr rrrrrrrr>rr1r1r}r2r.ssj         pr.)r.rr)Nr)rH)Nr(N)\typingrrrrrr,torch.nn.functionalrrWrytransformers.activationsr cache_utilsr generationr modeling_attn_mask_utilsr modeling_layersr modeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrutils.deprecationrutils.import_utilsrrconfiguration_granitemoehybridr+mamba_ssm.ops.triton.selective_state_updater!mamba_ssm.ops.triton.ssd_combinedr r! causal_conv1dr"r#!torch.nn.attention.flex_attentionr%integrations.flex_attentionr& get_loggerrmrjr3r=rrrGModulerrbrcrrrrrrrrrrermrtrvrrrrrrrr-r.__all__r1r1r1r2s               W`  b-0<q$  V ?