o ei_@sdZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l mZmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZddlmZddlmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3e"4e5Z6Gddde-Z7Gddde.Z8Gdddej9Z:Gdddej9Z;Gd d!d!ej9ZGd&d'd'e)Z?e Gd(d)d)e,Z@e Gd*d+d+e+ZAGd,d-d-e@eZBGd.d/d/ee@ZCgd0ZDdS)1zPyTorch JetMoe model.)CallableN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask) GenericForSequenceClassification)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)OutputRecordercapture_outputs)LlamaDecoderLayer) MixtralModelMixtralPreTrainedModelMixtralRMSNormMixtralRotaryEmbeddingapply_rotary_pos_embeager_attention_forwardload_balancing_loss_func) JetMoeConfigc@ eZdZdS) JetMoeRMSNormN__name__ __module__ __qualname__r*r*g/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/jetmoe/modular_jetmoe.pyr%4r%c@r$)JetMoeRotaryEmbeddingNr&r*r*r*r+r-8r,r-cs6eZdZdedededdffdd Zdd ZZS) JetMoeParallelExperts num_experts input_size output_sizereturnNcs6ttt||||_||_||_||_ dS)a Initialize the JetMoeParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) super__init__r Parametertorchemptyweightr/r0r1)selfr/r0r1 __class__r*r+r4=s  zJetMoeParallelExperts.__init__cCsP|j|dd}g}t|jD]}|t|||j|qtj|dd}|S)a Forward pass of the JetMoeParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rdim) splitranger/appendFlinearr8r6cat)r9inputs expert_size input_list output_listiresultsr*r*r+forwardTs  zJetMoeParallelExperts.forwardr'r(r)intr4rJ __classcell__r*r*r:r+r.<sr.cs2eZdZdededeffdd ZddZZS)JetMoeTopKGatingr0r/top_kcs2t||_||_||_tj||dd|_dS)a Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. FbiasN)r3r4r/r0rOrLinearlayer)r9r0r/rOr:r*r+r4js zJetMoeTopKGating.__init__cCs||}|j|jdd\}}tj|dd|}tj|d|j g|j |j d}| d|d}| d}|}|} | d\} } | j|jdd} |}|| } | | | ||fS)Nr"r<rdtypedevicetrunc) rounding_mode)rSfloattopkrOr6softmaxtype_aszerossizer/rUrVscatterlongsumtolistflattensortdiv)r9 hidden_stateslogits top_k_logits top_k_indices top_k_gatesr]gatesrE top_k_experts_index_sorted_experts batch_index batch_gatesr*r*r+rJ~szJetMoeTopKGating.forwardrKr*r*r:r+rNisrNcs.eZdZdZdeffdd ZddZZS) JetMoeMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. configcst|j|_|j|_t|j|_tj t |j|_ t |j|j|jd|_t |j|j|j|_t|j|j|jd|_dS)Nrr0r/rO)r3r4 hidden_sizer0intermediate_sizeractivation_function activationr6rr5r7rQr.num_local_experts input_linear output_linearrNnum_experts_per_tokrouterr9rrr:r*r+r4s   zJetMoeMoE.__init__cCs|\}}}|d|}||\}}}}} ||} || |} | jddd} || d| d} || |} | |dddf} tj|||j f| j | j d}| d|| }| |||j }||j}|S)a Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. rr<rr"NrT)r^reshaper|rychunkrwrzr6r]r0rUrV index_addviewrQ)r9 layer_inputbszlengthemb_sizermrorprE router_logits expert_inputsrfchunked_hidden_statesexpert_outputsr] layer_outputr*r*r+rJs    zJetMoeMoE.forward)r'r(r)__doc__r#r4rJrMr*r*r:r+rqsrqcs>eZdZdZdeffdd ZddZddZd d ZZ S) JetMoeMoAz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. rrcst|j|_|j|_|j|j|_|j|_ t j t |j|_t|j|j|j|_t|j|j|j|_t|j|j|j d|_dS)Nrs)r3r4rxr/rtr0 kv_channelsnum_key_value_headsr{rOr6rr5r7rQr.ryrzrNr|r}r:r*r+r4s  zJetMoeMoA.__init__cCs|\}}}|d|}||\}}}}} ||||f} ||} || |} tj|||j|jf| j| j d} | d|| }| |||jd}|| | fS)z Map inputs to attention experts according to routing decision and compute query projection inside each experts. r~rTr) r^rr|ryr6r]rOrtrUrVrr)r9rrrrrnrorprEr topo_inforrr]rr*r*r+maps    z JetMoeMoA.mapcCs|\}}}}|d|}|\}}} } ||} || | } | | dddf} tj|||jf| j| jd} | d|| }| |||j}||j }|S)zu Compute output projection inside each attention experts and merge the outputs of different experts. r~NrTr) r^rrzr6r]r0rUrVrrrQ)r9rrrrkrtrnrorprErrr]rr*r*r+reduces    zJetMoeMoA.reducecCstd)Nz-This module doesn't support call and forward.)NotImplementedError)r9rr*r*r+rJszJetMoeMoA.forward) r'r(r)rr#r4rrrJrMr*r*r:r+rs rcseZdZdZddededBffdd Z    ddejdejdBd ej dBd e dBd ej dBd e ejejdBe ejdBff d dZ Z S)JetMoeAttentionzH Multi-headed attention from 'Attention Is All You Need' paper. Nrr layer_idxcst||_||_d|_|durtd|jjdd|_ |j |_ |j |_ |j |j|_|j|_|j|_|j |_|jd|_t||_tjj|j|jddd |_dS) z Initialize the JetMoeAttention module. Args: config: Configuration object with model hyperparameters. layer_idx: Index of the layer in the model. TNzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r"grFrP)r3r4rrr is_causallogger warning_oncer;r'num_key_value_groupsr{rOattention_dropoutrrkv_projection_sizenum_attention_heads num_headshead_dimscalingrexpertsr6rrRrtkv_projr9rrrr:r*r+r4%s$    zJetMoeAttention.__init__rfattention_maskposition_embeddingspast_key_valuescache_positionr2cKsp|jdd}g|d|jR}|j|\} } } ||jddd\} } | |dd} | |dd} | |dd} |\}}t| | ||\} } |durc|||d}| | | |j |\} } t |j jt}| d|jdd} | d|jdd} ||| | | |f|jsdn|j|jd|\}}|jg||jdR}|j|| }|jg|dR}||| fS)Nr~rr<r")sincosr)dropoutr)shaperrrrrr transposerupdaterr get_interfacerr_attn_implementationr repeatrOtrainingrrr)r9rfrrrrkwargs input_shape hidden_shape query_statesrr key_states value_statesrr cache_kwargsattention_interface attn_output attn_weightsr*r*r+rJFsB     zJetMoeAttention.forwardN)NNNN)r'r(r)rr#rLr4r6Tensor LongTensorrtuplerJrMr*r*r:r+r s($rcseZdZddededBffdd Z      ddejdejdBd ejdBd e dBd e dBd ejdBd e ejejfdBde e dejfddZZS)JetMoeDecoderLayerNrrrcsDt||t|j|_t|||_t|j|_t||_ |` dSr) r3r4r%rtinput_layernormrself_attentionpost_attention_layernormrqmlp self_attnrr:r*r+r4|s     zJetMoeDecoderLayer.__init__Frfr position_idsr use_cacherrrr2c Ks`|} ||}|jd|||||||d|\}} } | |}|} ||}||}| |}|S)N)rfrrrrrrr*)rrrr) r9rfrrrrrrrresidualrmr*r*r+rJs&     zJetMoeDecoderLayer.forwardr)NNNFNN)r'r(r)r#rLr4r6rrrboolrrrrJrMr*r*r:r+r{s6   rc@sneZdZUeeddeeddgeeedddZee d<dZ dZ d gZ d gZ d Zd ZdZed d ZdS)JetMoePreTrainedModelr)indexr")rrf attentionsrrmodelFrrTcCsRt||t|trtj|jd|jjddSt|t t Br't |j dSdS)zInitialize the weights.r)meanstdN) r _init_weights isinstancer.initnormal_r8rrinitializer_rangerrqzeros_rQ)r9moduler*r*r+rs  z#JetMoePreTrainedModel._init_weightsN)r'r(r)rrrNr_can_record_outputsr#__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphr6no_gradrr*r*r*r+rs  rcseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd e dBd ej dBd eed efddZZS) JetMoeModelrrcsptj|_j|_tjj|j|_t fddt j D|_ j |_ tjjd|_dS)Ncsg|]}t|qSr*)r).0rrrr*r+ sz(JetMoeModel.__init__..)eps)r3r4 pad_token_id padding_idx vocab_sizer Embeddingrt embed_tokens ModuleListr?num_hidden_layerslayersrr% rms_norm_epsnormr}r:rr+r4s zJetMoeModel.__init__N input_idsrrr inputs_embedsrrrr2c Ks|du|duAr td|r|durt|jd}|dur!||}|dur=|dur-|nd} tj| | |jd|jd}|durF| d}t |j|||||d} |} | | |} |j d|jj D]} | | f| | ||||d|} qb|| } t| |dS) Nz:You must specify exactly one of input_ids or inputs_embedsrrr")rV)rrrrrrr)rrrrrr)last_hidden_stater) ValueErrorr rrrget_seq_lengthr6arangerrV unsqueezer rotary_embrrrr)r9rrrrrrrrpast_seen_tokens causal_maskrfr decoder_layerr*r*r+rJsR      zJetMoeModel.forward)NNNNNNN)r'r(r)r#r4rrrr6rrr FloatTensorrrrrrJrMr*r*r:r+rs>   rcseZdZddiZfddZee          ddejdBd ej dBd ejdBd e dBd ej dBd ejdBde dBdejdBde ej Bde dBdefddZZS)JetMoeForCausalLMzlm_head.weightzmodel.embed_tokens.weightcs`t|t||_|j|_|j|_tj|j|jdd|_ |j |_ |j |_ |j |_ |dS)NFrP)r3r4rrr aux_loss_coefrrRrtlm_headtie_word_embeddingsrxr/r{ post_initr}r:r*r+r4s   zJetMoeForCausalLM.__init__NrFrrrrrlabelsrrlogits_to_keepoutput_router_logitsr2c  Ks|jd|||||||| d| } | j} t| trt| dn| }|| dd|ddf}d}|durC|j||fd|jji| }d}| r`t | j |j |j |}|dur`||j ||j7}t|||| j| j| j| j dS)N)rrrrrrrrr)lossaux_lossrgrrfrrr*)rrrrLslicer loss_functionrrrr!rr/r{rtorVr rrfr)r9rrrrrrrrrrroutputsrf slice_indicesrgrrr*r*r+rJsX  zJetMoeForCausalLM.forward) NNNNNNNNrF)r'r(r)_tied_weights_keysr4rrr6rrrrrrLr rJrMr*r*r:r+rsL      rc@r$)JetMoeForSequenceClassificationNr&r*r*r*r+r_s r)rrrr)Ercollections.abcrr6rtorch.nnrrArr activationsr cache_utilsrr generationr masking_utilsr modeling_layersr modeling_outputsr rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrrllama.modeling_llamarmixtral.modeling_mixtralrrrrrr r!configuration_jetmoer# get_loggerr'rr%r-Moduler.rNrqrrrrrrr__all__r*r*r*r+sH           $ -1:L[+NP