o eiF@sddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z dd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZmZddlmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0ddl1m2Z2e*3e4Z5edGdddej6Z7Gdddej6Z8Gdddej6Z9Gdddej6Z:Gd d!d!ej6Z;Gd"d#d#ej6Zd)ej?d*e@d+ej?fd,d-ZA .dLd/ej6d0ej?d1ej?d2ej?d3ej?dBd4eBd5eBd6e%e'fd7d8ZCGd9d:d:ej6ZDGd;d<dd>e#ZFe(Gd?d@d@eFZG  A dMdBej?eHej?BdBdCe@dBd3ej?dBd+ej?e@BfdDdEZIGdFdGdGeFeZJGdHdIdIeeFZKgdJZLdS)N)Callable)OptionalN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hub)create_causal_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) JetMoeConfigRMSNormcsFeZdZd deddffdd Zdejdejfdd Zd d ZZ S) JetMoeRMSNormư>epsreturnNcs&ttt||_||_dS)z< JetMoeRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer% __class__h/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/jetmoe/modeling_jetmoe.pyr(2s  zJetMoeRMSNorm.__init__ hidden_statescCsJ|j}|tj}|djddd}|t||j}|j||S)NT)keepdim) dtypetor*float32powmeanrsqrtr-r,)r.r4 input_dtypevariancer2r2r3forward:s  zJetMoeRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler,shaper-)r.r2r2r3 extra_reprAszJetMoeRMSNorm.extra_repr)r$) __name__ __module__ __qualname__floatr(r*Tensorr@rC __classcell__r2r2r0r3r#0sr#c s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)JetMoeRotaryEmbeddinginv_freqNconfigcst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrKF) persistentoriginal_inv_freq) r'r(max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrLrope_parametersrMcompute_default_rope_parametersrattention_scalingregister_bufferclone)r.rLdevice rope_init_fnrKr0r2r3r(Hs   zJetMoeRotaryEmbedding.__init__rYz torch.deviceseq_lenr&z torch.TensorcCsZ|jd}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr5r8)rYr8) rTgetattrr/num_attention_headsr*arangeint64r9rG)rLrYr[basedimattention_factorrKr2r2r3rUXs &z5JetMoeRotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t |dd+|| dd}t j ||fdd }| |j}||j} Wdn1slwY|j|jd | j|jd fS) Nrr6r mpscpuF) device_typeenabledr5rdr^)rKrGexpandrBr9rY isinstancetypestrr transposer*catcosrVsinr8) r.x position_idsinv_freq_expandedposition_ids_expandedrhfreqsembrqrrr2r2r3r@vs0&zJetMoeRotaryEmbedding.forwardN)NNN)rDrErFr*rH__annotations__r!r( staticmethodrintrArGrUno_gradrr@rIr2r2r0r3rJEs&   rJcs6eZdZdedededdffdd Zdd ZZS) JetMoeParallelExperts num_experts input_size output_sizer&Ncs6ttt||||_||_||_||_ dS)a Initialize the JetMoeParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) r'r(rr)r*emptyr,rrr)r.rrrr0r2r3r(s  zJetMoeParallelExperts.__init__cCsP|j|dd}g}t|jD]}|t|||j|qtj|dd}|S)a Forward pass of the JetMoeParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rrj) splitrangerappendFlinearr,r*rp)r.inputs expert_size input_list output_listiresultsr2r2r3r@s  zJetMoeParallelExperts.forwardrDrErFr|r(r@rIr2r2r0r3r~sr~cs2eZdZdededeffdd ZddZZS)JetMoeTopKGatingrrtop_kcs2t||_||_||_tj||dd|_dS)a Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. FbiasN)r'r(rrrrLinearlayer)r.rrrr0r2r3r(s zJetMoeTopKGating.__init__cCs||}|j|jdd\}}tj|dd|}tj|d|j g|j |j d}| d|d}| d}|}|} | d\} } | j|jdd} |}|| } | | | ||fS)Nr rjrr8rYtrunc) rounding_mode)rrGtopkrr*softmaxtype_aszerossizerr8rYscatterlongsumtolistflattensortdiv)r.r4logits top_k_logits top_k_indices top_k_gatesrgatesr top_k_experts_index_sorted_experts batch_index batch_gatesr2r2r3r@szJetMoeTopKGating.forwardrr2r2r0r3rsrcs.eZdZdZdeffdd ZddZZS) JetMoeMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. rLcst|j|_|j|_t|j|_tj t |j|_ t |j|j|jd|_t |j|j|j|_t|j|j|jd|_dS)Nr5rrr)r'r(r/rintermediate_sizeractivation_function activationr*rr)rrr~num_local_experts input_linear output_linearrnum_experts_per_tokrouterr.rLr0r2r3r(s   zJetMoeMoE.__init__cCs|\}}}|d|}||\}}}}} ||} || |} | jddd} || d| d} || |} | |dddf} tj|||j f| j | j d}| d|| }| |||j }||j}|S)a Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. r6r5rjrr Nr)rreshaperrchunkrrr*rrr8rY index_addviewr)r. layer_inputbszlengthemb_sizerrrr router_logits expert_inputsr4chunked_hidden_statesexpert_outputsr layer_outputr2r2r3r@s    zJetMoeMoE.forward)rDrErF__doc__r!r(r@rIr2r2r0r3rsrcs>eZdZdZdeffdd ZddZddZd d ZZ S) JetMoeMoAz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. rLcst|j|_|j|_|j|j|_|j|_ t j t |j|_t|j|j|j|_t|j|j|j|_t|j|j|j d|_dS)Nr)r'r(rrr/r kv_channelsnum_key_value_headsrrr*rr)rrr~rrrrrr0r2r3r('s  zJetMoeMoA.__init__cCs|\}}}|d|}||\}}}}} ||||f} ||} || |} tj|||j|jf| j| j d} | d|| }| |||jd}|| | fS)z Map inputs to attention experts according to routing decision and compute query projection inside each experts. r6rr) rrrrr*rrr/r8rYrr)r.rrrrrrrrr topo_inforrrrr2r2r3map9s    z JetMoeMoA.mapcCs|\}}}}|d|}|\}}} } ||} || | } | | dddf} tj|||jf| j| jd} | d|| }| |||j}||j }|S)zu Compute output projection inside each attention experts and merge the outputs of different experts. r6Nrr) rrrr*rrr8rYrrr)r.rrrrkr/rrrrrrrrr2r2r3reducePs    zJetMoeMoA.reducecCstd)Nz-This module doesn't support call and forward.)NotImplementedError)r.rr2r2r3r@fszJetMoeMoA.forward) rDrErFrr!r(rrr@rIr2r2r0r3rs rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr6r5rj)rBr*rp)rsx1x2r2r2r3 rotate_halfjsrrotary_pos_embcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qrrqrr unsqueeze_dimq_embedk_embedr2r2r3apply_rotary_pos_embqs  rr4n_repr&cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r N)rBrkr)r4rbatchrslenr]r2r2r3 repeat_kvs 0rmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr5rr6)rdr8)ptrainingr )rnum_key_value_groupsr*matmulrorrrr:r9r8rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputr2r2r3eager_attention_forwards  rcseZdZdZddededBffdd Z    ddejdejdBd ej dBd e dBd ej dBd e ejejdBe ejdBff d dZ Z S)JetMoeAttentionzH Multi-headed attention from 'Attention Is All You Need' paper. NrL layer_idxcst||_||_d|_|durtd|jjdd|_ |j |_ |j |_ |j |j|_|j|_|j|_|j |_|jd|_t||_tjj|j|jddd |_dS) z Initialize the JetMoeAttention module. Args: config: Configuration object with model hyperparameters. layer_idx: Index of the layer in the model. TNzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r gr5Fr)r'r(rLr is_causallogger warning_oncer1rDrrrattention_dropoutrrkv_projection_sizer` num_headsr]rrexpertsr*rrr/kv_projr.rLrr0r2r3r(s$    zJetMoeAttention.__init__r4rposition_embeddingspast_key_valuescache_positionr&cKsp|jdd}g|d|jR}|j|\} } } ||jddd\} } | |dd} | |dd} | |dd} |\}}t| | ||\} } |durc|||d}| | | |j |\} } t |j jt}| d|jdd} | d|jdd} ||| | | |f|jsdn|j|jd|\}}|jg||jdR}|j|| }|jg|dR}||| fS)Nr6r5rjr )rrrqrr)rr)rBr]rrrrrrorupdaterr get_interfacerL_attn_implementationrrepeatrrrrr)r.r4rrrrr input_shape hidden_shape query_statesrrrrrqrr cache_kwargsattention_interfacerrr2r2r3r@sB     zJetMoeAttention.forwardry)NNNN)rDrErFrr!r|r(r*rH LongTensorr rAr@rIr2r2r0r3rs($rcseZdZddededBffdd Z      ddejdejdBd ejdBd e dBd e dBd ejdBd e ejejfdBde e dejfddZZS)JetMoeDecoderLayerNrLrcsDt|j|_t||_t|j|_t|j|_t|||_ dSry) r'r(r/rmlpr#input_layernormpost_attention_layernormrself_attentionrr0r2r3r( s    zJetMoeDecoderLayer.__init__Fr4rrtr use_cacherrrr&c Ks`|} ||}|jd|||||||d|\}} } | |}|} ||}||}| |}|S)N)r4rrtrr rrr2)r r r r ) r.r4rrtrr rrrresidualrr2r2r3r@s&     zJetMoeDecoderLayer.forwardry)NNNFNN)rDrErFr!r|r(r*rHrr boolrArrr@rIr2r2r0r3r s6   rcs~eZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eeddeed dgeeed dd Zefd d ZZS)JetMoePreTrainedModelrLmodelFrrTr5)indexr )rr4 attentionscsRt|t|trtj|jd|jjddSt|t t Br't |j dSdS)zInitialize the weights.r)r<stdN) r' _init_weightsrlr~initnormal_r,rLinitializer_rangerrzeros_r)r.rr0r2r3rHs  z#JetMoePreTrainedModel._init_weights)rDrErFr!rzbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrrrr_can_record_outputsr*r}rrIr2r2r0r3r6s"  rcseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd e dBd ej dBd eed efddZZS) JetMoeModelrLcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_j|_|dS)Ncsg|]}t|qSr2)r).0rrLr2r3 [z(JetMoeModel.__init__..)r%r'F)r'r( pad_token_id padding_idx vocab_sizer Embeddingr/ embed_tokens ModuleListrnum_hidden_layerslayersr# rms_norm_epsnormrJ rotary_embgradient_checkpointingr post_initrr0r'r3r(Ts   zJetMoeModel.__init__N input_idsrrtr inputs_embedsr rrr&c Ks|du|duAr td|r|durt|jd}|dur!||}|dur=|dur-|nd} tj| | |jd|jd}|durF| d}t |j|||||d} |} | | |} |j d|jj D]} | | f| | ||||d|} qb|| } t| |dS) Nz:You must specify exactly one of input_ids or inputs_embedsr'rr )rY)rLr8rrrrt)rrrr rrt)last_hidden_stater) ValueErrorr rLr.get_seq_lengthr*rarBrYrrr4r1r0r3r)r.r7rrtrr8r rrpast_seen_tokens causal_maskr4r decoder_layerr2r2r3r@esR      zJetMoeModel.forward)NNNNNNN)rDrErFr!r(rrrr*rrHr FloatTensorrrrrr@rIr2r2r0r3r%Rs>  r%r5 gate_logitsrcs|dus t|ts dSt|tr#|djtjfdd|Ddd}tjjj|dd}tj||dd\}}tjj ||}|durStj | dd} tj |dd} ng|j \} } |j d| | } |dddddddf | | | ||fd||}tj| |ddtj|dd} |ddddddf | | | |fd|}tj||ddtj|dd} t| | d}||S)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrcsg|]}|qSr2)r9)r& layer_gatecompute_devicer2r3r(r)z,load_balancing_loss_func..rjr6)rlrArYr*rprrrrone_hotr<rGrBrkrr9rr)r@rrrconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthr0expert_attention_mask router_per_expert_attention_mask overall_lossr2rBr3load_balancing_loss_funcs>    rPcseZdZddiZfddZee          ddejdBd ej dBd ejdBd e dBd ej dBd ejdBde dBdejdBde ej Bde dBdefddZZS)JetMoeForCausalLMzlm_head.weightzmodel.embed_tokens.weightcs`t|t||_|j|_|j|_tj|j|jdd|_ |j |_ |j |_ |j |_ |dS)NFr)r'r(r%rr, aux_loss_coefrrr/lm_headtie_word_embeddingsrrrr6rr0r2r3r(s   zJetMoeForCausalLM.__init__NrFr7rrtrr8labelsr rlogits_to_keepoutput_router_logitsr&c  Ks|jd|||||||| d| } | j} t| trt| dn| }|| dd|ddf}d}|durC|j||fd|jji| }d}| r`t | j |j |j |}|dur`||j ||j7}t|||| j| j| j| j dS)N)r7rrtrr8r rrWr,)lossaux_lossrrr4rrr2)rr9rlr|slicerS loss_functionrLr,rPrrrrRr9rYrrr4r)r.r7rrtrr8rUr rrVrWroutputsr4 slice_indicesrrXrYr2r2r3r@sX  zJetMoeForCausalLM.forward) NNNNNNNNrF)rDrErF_tied_weights_keysr(rrr*rrHr r?rr|rr@rIr2r2r0r3rQsL      rQc@s eZdZdS)JetMoeForSequenceClassificationN)rDrErFr2r2r2r3r_Hs r_)rQr%rr_)r )r)Nr5N)Mcollections.abcrtypingrr*rtorch.nnrrrr activationsr cache_utilsr r generationr integrationsr r masking_utilsrmodeling_layersrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrrconfiguration_jetmoer! get_loggerrDrModuler#rJr~rrrrrrHr|rrGrrrrr%rArPrQr___all__r2r2r2r3s           A-1:L  [+U RP