o i@sddlmZmZmZmZddlZddlmmZ ddlmZddl m Z ddl m Z mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(e#rddl)m*Z*ddl+m,Z,e$-e.Z/GdddeddZ0Gdddej1Z2Gdddej1Z3Gdddej1Z4Gdddej1Z5Gd d!d!ej1Z6d"d#Z7dGd$d%Z8d&ej9d'e:d(ej9fd)d*Z; +dHd,ej1d-ej9d.ej9d/ej9d0eej9d1eGd7d8d8eZ?e"Gd9d:d:eZ@Gd;d<dd>e@ZB  ? dId@eej9eCej9dfdAee:d0eej9d(eej9e:ffdBdCZDGdDdEdEe@eZEgdFZFdS)J)CallableOptional TypedDictUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringis_torch_flex_attn_availablelogging)deprecate_kwarg)GraniteMoeSharedConfig) BlockMask)make_flex_block_causal_maskc@s@eZdZUdZejed<ejed<eed<eed<ejed<dS)GraniteFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__int IntTensorr-r-r/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/granitemoeshared/modeling_granitemoeshared.pyr3s   rF)totalcs<eZdZdZdeffdd ZdejdejfddZZ S) GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configcsZt|j|_|j|_t|j|_tj |j|jddd|_ tj |j|jdd|_ dS)NFbias) super__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr1 __class__r-r.r6Us  zGraniteMoeSharedMLP.__init__ hidden_statesreturncCs<||}|jddd}||d|d}||}|S)Nr2dimrr)r=chunkr;r>)r@rCchunked_hidden_statesr-r-r.forward^s  zGraniteMoeSharedMLP.forward) r$r%r&r'rr6r(TensorrJ __classcell__r-r-rAr.r0Ls r0cs.eZdZdfdd ZddZddZZS) GraniteMoeSharedRMSNormư>cs&ttt||_||_dS)zF GraniteMoeSharedRMSNorm is equivalent to T5LayerNorm N)r5r6r Parameterr(onesweightvariance_epsilon)r@r7epsrAr-r.r6gs  z GraniteMoeSharedRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)Nr2rET)keepdim) dtypetor(float32powmeanrsqrtrRrQ)r@rC input_dtypevariancer-r-r.rJos  zGraniteMoeSharedRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tuplerQshaperR)r@r-r-r. extra_reprvsz"GraniteMoeSharedRMSNorm.extra_repr)rN)r$r%r&r6rJr_rLr-r-rAr.rMfsrMcs6eZdZdedededdffdd Zdd ZZS) GraniteMoeSharedParallelExperts num_expertsr8 output_sizerDNcs6ttt||||_||_||_||_ dS)a Initialize the GraniteMoeSharedParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) r5r6rrOr(emptyrQrar8rb)r@rar8rbrAr-r.r6{s  z(GraniteMoeSharedParallelExperts.__init__cCsP|j|dd}g}t|jD]}|t|||j|qtj|dd}|S)a Forward pass of the GraniteMoeSharedParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rrF) splitrangeraappendFlinearrQr(cat)r@inputs expert_size input_list output_listiresultsr-r-r.rJs  z'GraniteMoeSharedParallelExperts.forwardr$r%r&r+r6rJrLr-r-rAr.r`zsr`cs2eZdZdededeffdd ZddZZS)GraniteMoeSharedTopKGatingr8ratop_kcs2t||_||_||_tj||dd|_dS)a Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. Fr3N)r5r6rar8rrrr<layer)r@r8rarrrAr-r.r6s z#GraniteMoeSharedTopKGating.__init__cCs||}|j|jdd\}}tj|dd|}tj|d|j g|j |j d}| d|d}| d}|}|} | d\} } | j|jdd} |}|| } | | | ||fS)NrrFrrUdevicetrunc) rounding_mode)rsfloattopkrrr(softmaxtype_aszerossizerarUruscatterlongsumtolistflattensortdiv)r@rClogits top_k_logits top_k_indices top_k_gatesr|gatesrk top_k_experts_index_sorted_experts batch_index batch_gatesr-r-r.rJsz"GraniteMoeSharedTopKGating.forwardrpr-r-rAr.rqsrqcs.eZdZdZdeffdd ZddZZS)GraniteMoeSharedMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. r1cslt|j|_|j|_t|j|_t|j |j|jd|_ t|j |j|j|_ t |j|j |j d|_dS)Nr2)r8rarr)r5r6r7r8intermediate_sizerr:r;r`num_local_expertsr=r>rqnum_experts_per_tokrouterr?rAr-r.r6s    zGraniteMoeSharedMoE.__init__cCs|\}}}|d|}||\}}}}} ||} || |} | jddd} || d| d} || |} | |dddf} tj|||j f| j | j d}| d|| }| |||j }|| fS)a Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. rEr2rFrrNrt)r}reshaperr=rHr;r>r(r|r8rUru index_addview)r@ layer_inputbszlengthemb_sizerrrrk router_logits expert_inputsrCrIexpert_outputsr| layer_outputr-r-r.rJs   zGraniteMoeSharedMoE.forward)r$r%r&r'rr6rJrLr-r-rAr.rsrcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..NrEr2rF)r^r(ri)xx1x2r-r-r. rotate_halfsrcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkcossin position_ids unsqueeze_dimq_embedk_embedr-r-r.apply_rotary_pos_embs  rrCn_reprDcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r^expandr)rCrbatchnum_key_value_headsslenhead_dimr-r-r. repeat_kv5s 0rmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr2rrE)rGrU)ptrainingr)rnum_key_value_groupsr(matmul transposer^r functionalrzrWrVrUrr contiguous) rrrrrrrkwargs key_states value_states attn_weights causal_mask attn_outputr-r-r.eager_attention_forwardAs & rcseZdZdZddedeeffdd Zeddd d      dd e j d ee j dee j dee de dee j deee j e j fdee j ee j eee j ffddZZS)GraniteMoeSharedAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr1 layer_idxcs t||_||_|durtd|jjd|j|_|j |_ |j |_ |j |j |_ |j |_ |j |j |_d|_|j|_|j |j |j krUtd|j d|j dtj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |j |jd|_tj|j |j |jd|_dS)NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).r3)r5r6r1rlogger warning_oncerBr$attention_dropoutr7num_attention_heads num_headsrrr is_causalattention_multiplierr ValueErrorrr<attention_biasq_projk_projv_projo_projr@r1rrAr-r.r6as2  z"GraniteMoeSharedAttention.__init__past_key_valuepast_key_values4.58new_nameversionFrCrr use_cachecache_positionposition_embeddingsrDcKsD|\} } } ||} ||} ||}| | | |j|jdd} | | | |j|jdd} || | |j|jdd}|durF|nd\}}|durWt | | ||\} } |durl|||d}| | ||j |\} }t }|j jdkrzt|j j}||| | ||f|jsdn|j|jd|\}}|| | d}||}||fS) Nrr2)NN)rrreagerr)rrrE)r}rrrrrrrrrupdaterrr1_attn_implementationrrrrr)r@rCrrrrrrrrq_lenr query_statesrrrr cache_kwargsattention_interfacerrr-r-r.rJs>        z!GraniteMoeSharedAttention.forwardN)NNNFNN)r$r%r&r'rrr+r6rr(rKr)r boolr]rJrLr-r-rAr.r^s6  rcseZdZdedeffdd Zedddd dd ejd e ejd e ej de e de e de e de ej de e de e ejejfdeede eje e ejejfffddZZS)GraniteMoeSharedDecoderLayerr1rcst|j|_t||d|_|jdkrt||_t|j|j d|_ t|j|j d|_ |j |_ |j dkr:d|_dSt||_dS)N)r1rrrS)r5r6r7r self_attnrrblock_sparse_moerM rms_norm_epsinput_layernormpost_attention_layernormresidual_multiplierr9r0 shared_mlprrAr-r.r6s   "z%GraniteMoeSharedDecoderLayer.__init__rrrrNFrCrroutput_attentionsrroutput_router_logitsrrrDc  Ks|} ||}|jd|||||||| d| \}} | ||j}|} ||}||\} }|jdur7| }n| ||}~ | ||j}|f}|rP|| f7}|rW||f7}|S)a1 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. )rCrrrrrrrNr-)rrrrrr)r@rCrrrrrrrrrresidualself_attn_weightsmoe_hidden_statesroutputsr-r-r.rJs:'       z$GraniteMoeSharedDecoderLayer.forward)NNNFFNFN)r$r%r&rr+r6rr(rKrr)r rr]rr FloatTensorrJrLr-r-rAr.rsD     rcsFeZdZUeed<dZdZdgZdgZdZ dZ dZ fddZ Z S) GraniteMoeSharedPreTrainedModelr1modelTrrFcs4t|t|tr|jjjd|jjddSdS)Nr)rYstd) r5 _init_weights isinstancer`rQdatanormal_r1initializer_range)r@rrAr-r.r"s  z-GraniteMoeSharedPreTrainedModel._init_weights)r$r%r&rr*base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphrrLr-r-rAr.rs rcsDeZdZUejed<ddeffdd Zee ddZ Z S) GraniteMoeSharedRotaryEmbeddinginv_freqNr1cstt|drt|jtr|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultr F) persistent)r5r6hasattrrrdictgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr1r rope_init_fnattention_scalingregister_bufferr original_inv_freq)r@r1rur rAr-r.r6+s   z(GraniteMoeSharedRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t j |dd+|| dd}t j ||fdd }| |j}||j} Wdn1smwY|j|jd | j|jd fS) NrrErmpscpuF) device_typeenabledr2rF)rU)r rxrr^rVrurrstrr(autocastrrirrrrU) r@rrinv_freq_expandedposition_ids_expandedrfreqsembrrr-r-r.rJ<s0&z'GraniteMoeSharedRotaryEmbedding.forwardr) r$r%r&r(rKr*rr6no_gradrrJrLr-r-rAr.r (s r cs eZdZdeffdd Ze           ddeejdeej deejdee e e ej fd eej d eed eed eed eedeedeejde eeffddZ dde ej dfdej dej de d ef ddZedej dededejdej def ddZZS) GraniteMoeSharedModelr1cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_d|_j|_j|_j|_|j|j|_j|_j|_j|_|jdkr]tnd|_|dS)Ncsg|]}t|qSr-)r).0rr1r-r. Uz2GraniteMoeSharedModel.__init__..rFrope)r5r6 pad_token_id padding_idx vocab_sizer Embeddingr7 embed_tokens ModuleListrenum_hidden_layerslayersrMrnormgradient_checkpointingembedding_multiplierrrrr rope_thetaposition_embedding_typer  rotary_emb post_initr?rAr*r.r6Ns$  zGraniteMoeSharedModel.__init__N input_idsrrr inputs_embedsrroutput_hidden_statesr return_dictrrDc  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |du|duAr4td|jrC|jrC|rCt dd}|durL| |}||j }|r]|dur]t |jd}| dury|duri| nd} tj| | |jd|jd} |dur| d}|||| ||}|}d}|jdur|||}|rdnd}|rdnd}| rdnd}|jD],}|r||f7}|||||||| | |d }|d}|r||df7}| r||d f7}q||}|r||f7}| std d ||||fDSt|||||d S)Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fr*rrrur-)rrrrrrrrrEcss|] }|dur|VqdSrr-)r)vr-r-r. sz0GraniteMoeSharedModel.forward..)last_hidden_staterrC attentionsr)r1rr?ruse_return_dictrr7rrrr2r8r get_seq_lengthr(aranger^rur_update_causal_maskr;r5r6r]r)r@r=rrrr>rrr?rr@rrpast_seen_tokensrrCrall_hidden_statesall_self_attnsall_router_logits decoder_layer layer_outputsr-r-r.rJgs                 zGraniteMoeSharedModel.forwardFr input_tensorc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) Nflash_attention_2rflex_attentionrFsdpa)r>past_key_values_length is_trainingrrE)sequence_length target_lengthrUr batch_size)cudaxpunpu)r1ranyrr(rKrrGis_compileabler _ignore_causal_mask_sdparrUr^get_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrurfinfomin_unmask_unattended) r@rrPrrrrJusing_compilable_cacherUrVrWr min_dtyper-r-r.rIsT           z)GraniteMoeSharedModel._update_causal_maskrVrWrUrXc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerUrur)diagonalrArEr)rGr(rarbfullrutriurHrrcloner^rV masked_fill) rrVrWrUrrXrrre mask_length padding_maskr-r-r.r`s,  $ 6  zKGraniteMoeSharedModel._prepare_4d_causal_attention_mask_with_cache_position) NNNNNNNNNNN)F)r$r%r&rr6rrr(r)rKrr listrrr]rrJrI staticmethodr+rUr`rLr-r-rAr.r(Ls      p  Dr(r2 gate_logitsracs|dus t|ts dSt|tr#|djtjfdd|Ddd}tjjj|dd}tj||dd\}}tjj ||}|durStj | dd} tj |dd} nm|j \} } |j d| | } |dddddddf | | | ||fd||}tj| |ddtj|dd} |ddddddf | | | |j dfd|j d}tj||ddtj|dd} |jjdur|jjnd}|j dt|}t| dd|||j df| d}||S)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrcsg|]}|qSr-)rV)r) layer_gatecompute_devicer-r.r+pr,z,load_balancing_loss_func..rFrEr)rr]rur(rirrrzryone_hotrYrxr^rrrVrindexr+r)rqrarrrconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expertrXrVr4expert_attention_mask router_per_expert_attention_mask device_indexrank overall_lossr-rsr.load_balancing_loss_funcNsF    &rc seZdZdgZdeffdd Ze             ddeej deej d eej d ee e e ejfd eejd eej d eedeedeedeedeedeej de eej fde eeffddZZS)GraniteMoeSharedForCausalLMzlm_head.weightr1csXt|t||_|j|_tj|j|jdd|_|j |_ |j |_ |j |_ | dS)NFr3)r5r6r(rr0rr<r7lm_headrouter_aux_loss_coefrrarr<r?rAr-r.r6s   z$GraniteMoeSharedForCausalLM.__init__Nrr=rrrr>labelsrrr?rr@rlogits_to_keeprDcKs|dur|n|jj}| dur| n|jj} | dur| n|jj} | dur$| n|jj} |jd|||||||| | | | d |}|d}t| trKt| dn| }| |dd|ddf}||jj }d}|dury| }|j ||fd|jj i|}d}| rt| r|jn|d|j|j|}|dur||j||j7}| s|f|dd}| r|f|}|dur|f|S|St||||j|j|j|jdS) ax labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, GraniteMoeSharedForCausalLM >>> model = GraniteMoeSharedForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) r=rrrr>rrr?rr@rrr0rEr)lossaux_lossrrrCrErr-)r1rrr?rFrrr+slicerlogits_scalingrx loss_functionr0rrrarrrVrurrrCrE)r@r=rrrr>rrrr?rr@rrrrrC slice_indicesrrroutputr-r-r.rJsx(   z#GraniteMoeSharedForCausalLM.forward) NNNNNNNNNNNNr)r$r%r&_tied_weights_keysrr6rrr(r)rKrr rorrr+r]rrJrLr-r-rAr.rs\        r)rr(r)Nr)r)Nr2N)Gtypingrrrrr(torch.nn.functionalrrrg activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersr modeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrutils.deprecationrconfiguration_granitemoesharedr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerr$rrModuler0rMr`rqrrrrKr+rrxrrrrr r(r]rr__all__r-r-r-r.s           -0<  Wa$  V