o eiv@sdddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z dd lmZdd lmZmZmZdd lmZmZdd lmZdd lmZmZmZddlmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/Gdddej0Z1edGdddej0Z2Gdddej0Z3Gdddej0Z4Gdd d ej5Z6Gd!d"d"ej0Z7d#d$Z8ed%dCd&d'Z9d(ej:d)e;d*ej:fd+d,Z< -dDd.ej0d/ej:d0ej:d1ej:d2ej:dBd3e=d4e=d5e$e&fd6d7Z>ee9Gd8d9d9ej0Z?Gd:d;d;eZ@Gdd?d?eAZBe'Gd@dAdAeAeZCgdBZDdS)E)Callable)OptionalN)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)capture_outputs) AfmoeConfigc s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)AfmoeRotaryEmbeddinginv_freqNconfigcst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultr"F) persistentoriginal_inv_freq) super__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr#rope_parametersr$compute_default_rope_parametersrattention_scalingregister_bufferclone)selfr#device rope_init_fnr" __class__f/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/afmoe/modeling_afmoe.pyr)/s   zAfmoeRotaryEmbedding.__init__r3z torch.deviceseq_lenreturnz torch.TensorcCsZ|jd}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rdtyper3r?) r-getattr hidden_sizenum_attention_headstorcharangeint64tofloat)r#r3r9basedimattention_factorr"r7r7r8r.?s &z4AfmoeRotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t |dd+|| dd}t j ||fdd }| |j}||j} Wdn1slwY|j|jd | j|jd fS) NrrmpscpuF) device_typeenabledr=rJr>)r"rHexpandshaperGr3 isinstancetypestrr transposerDcatcosr/sinr?) r2x position_idsinv_freq_expandedposition_ids_expandedrOfreqsembrYrZr7r7r8forward]s0&zAfmoeRotaryEmbedding.forwardN)NNN)__name__ __module__ __qualname__rDTensor__annotations__r r) staticmethodrinttuplerHr.no_gradrra __classcell__r7r7r5r8r!,s&   r!RMSNormcs@eZdZd deddffdd ZdejfddZd d ZZ S) AfmoeRMSNormư>epsr:Ncs&ttt||_||_dS)z; AfmoeRMSNorm is equivalent to T5LayerNorm N)r(r)r ParameterrDonesweightvariance_epsilon)r2rBrpr5r7r8r)os  zAfmoeRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)Nr=rLT)keepdim) r?rGrDfloat32powmeanrsqrtrtrs)r2 hidden_states input_dtypevariancer7r7r8raws  zAfmoeRMSNorm.forwardcCst|jjd|jS)Nz, eps=)rjrsrSrt)r2r7r7r8 extra_repr~szAfmoeRMSNorm.extra_repr)ro) rcrdrerHr)rDrfrar}rlr7r7r5r8rnmsrncs&eZdZdfdd ZddZZS)AfmoeMLPNcs~t||_|j|_|dur|jn||_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) r(r)r#rBintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fn)r2r#rr5r7r8r)s zAfmoeMLP.__init__cCs$||||||}|Srb)rrrr)r2r[rr7r7r8ras zAfmoeMLP.forwardrb)rcrdrer)rarlr7r7r5r8r~s r~cs6eZdZdZfddZdejdejfddZZS)AfmoeTokenChoiceRouterz Token-choice top-K router for MoE routing. This router assigns each token to the top-K experts based on sigmoid scores, matching the released checkpoints. csBt||_|j|_|j|_|j|_tj|j |jdd|_ dSr) r(r)r#num_experts_per_toktop_k num_experts route_scalerrrBgater2r#r5r7r8r)s zAfmoeTokenChoiceRouter.__init__rz expert_biasc Cs|j\}}}|d|}t||tj}tj|||jdd\}}|j d|d}|j dddd}||}||j }||fS)NrLr)krJ)rJindexT)rJrug#B ;) rSviewrDsigmoidrrGrvtopkrgathersumr) r2rzr_ hidden_dimscoresselected_experts top_scores denominatorr7r7r8ras   zAfmoeTokenChoiceRouter.forward) rcrdre__doc__r)rDrfrarlr7r7r5r8rs rcsHeZdZdZdeffdd Zdejdejdejdejfd d ZZ S) AfmoeExpertsz Container holding the routed experts. This mirrors the Experts pattern used across other MoE models to ease checkpoint conversion. r#csBt|j|_|j|_t|jD] }|t||jdqdS)N)r) r(r)rrrrangeappendr~moe_intermediate_size)r2r#rr5r7r8r)s zAfmoeExperts.__init__rzrrouting_weightsr:cCsh|j\}}}|dkr||d|S|d|}|jd}tj|jd|jtjd|} |d} |d}tj | dd} | | } | | } || }| d| } t | } tj | dd\}}d}t ||D]!\}}|dkrqqh||}| ||}|||}|| ||<|}qh| tj|d|j}t |}| d|}|d||||||S)z Args: hidden_states: (batch, seq, hidden) selected_experts: (batch, seq, top_k) routing_weights: (batch, seq, top_k) rrLr@T)stable) return_counts)rS new_zerosrrDrEr3longrepeat_interleavereshapeargsort index_select zeros_likeunique_consecutiveziptolistrGrv unsqueezer? expand_as scatter_add_)r2rzrr batch_sizer9rhidden_states_flatr token_indicesexpert_indicessortingdispatched_tokensexpert_outputsunique_expertscountsstart expert_idcountend expert_input expert_outputweighted_outputs aggregatedscatter_indicesr7r7r8rasB           zAfmoeExperts.forward) rcrdrerr r)rDrfrarlr7r7r5r8rsrcs(eZdZdZfddZddZZS)AfmoeMoEz Mixture of Experts (MoE) module for AFMoE. This module implements a sparse MoE layer with both shared experts (always active) and routed experts (activated based on token-choice routing). csTt||_t||_t||j|j|_t ||_ t j t |jdd|_dS)NF) requires_grad)r(r)r#rrouterr~rnum_shared_expertsshared_expertsrexpertsrrqrDzerosrrrr5r7r8r)s   zAfmoeMoE.__init__c Csx|j\}}}|d|}|||j\}}||||jj}||||jj}|||||}||||} || S)NrL)rSrrrr#rrr) r2rzrr9rrrr shared_output routed_outputr7r7r8ras  zAfmoeMoE.forward)rcrdrerr)rarlr7r7r5r8rs rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..NrLr=rQ)rSrDrX)r[x1x2r7r7r8 rotate_half srrotary_pos_embcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )rr)qrrYrZ unsqueeze_dimq_embedk_embedr7r7r8apply_rotary_pos_embs  rrzn_repr:cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rSrRr)rzrbatchnum_key_value_headsslenr<r7r7r8 repeat_kv-s 0rmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr=rrL)rJr?)ptrainingr)rnum_key_value_groupsrDmatmulrWr functionalsoftmaxrvrGr?rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputr7r7r8eager_attention_forward9s  rcseZdZdZdedeffdd Z  ddejde ejejfd ejdBd e dBd ej dBd e e d e ejejffddZZS)AfmoeAttentionaJ Multi-headed attention module with optional sliding window and gating. This attention mechanism supports both full attention and sliding window attention, and includes Q/K normalization and gating of the output. It inherits from [`LlamaAttention`] to minimize the amount of custom logic we need to maintain. r# layer_idxcs2t||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_|j|dk|_|jrt|jnd|_t|j|jd|_t|j|jd|_t j|j|j|jdd|_dS)Nr<gTrsliding_attentionrpF)r(r)r#rrArBrCr<rrrattention_dropout is_causalrrattention_biasq_projk_projv_projo_proj layer_typesis_local_attentionsliding_windowrn rms_norm_epsq_normk_normrr2r#rr5r7r8r)\s2   zAfmoeAttention.__init__Nrzposition_embeddingsrpast_key_valuecache_positionrr:cKsN|jdd}g|d|jR}|||} |||} |||} ||} || dd} | | dd} | dd} |j rV|\} }t | | | |\} } |durid|i}| | | |j |\} } t|jjt}||| | | f||js}dn|j|j|jd|\}}|jg|dR}|t| }||}||fS)NrLrr=rr)rrrr)rSr<rrrrrrrWrrrupdaterr get_interfacer#_attn_implementationrrrrrrrDrr)r2rzrrrrr input_shape hidden_shape query_statesrr gate_statesrYrZ cache_kwargsattention_interfaceoutputrrr7r7r8ra{sF      zAfmoeAttention.forward)NN)rcrdrerr rir)rDrfrjr LongTensorrrrarlr7r7r5r8rRs($rcseZdZdZdedeffdd Z      ddejdejdBd ej dBd e dBd e dBd ej dBd e ejejfdBde edejfddZZS)AfmoeDecoderLayerz AFMoE decoder layer with dual normalization. This layer applies self-attention followed by either a dense MLP or MoE block, with dual normalization (pre and post) around each component. r#rcst|j|_||_t||d|_|j||_t|j|j d|_ t|j|j d|_ t|j|j d|_ t|j|j d|_ ||jk|_|jrMt||_dSt||_dS)N)r#rr)r(r)rBrr self_attnrattention_typernrinput_layernormpost_attention_layernormpre_mlp_layernormpost_mlp_layernormnum_dense_layers moe_enabledrmlpr~rr5r7r8r)s   zAfmoeDecoderLayer.__init__Nrzrr\r use_cacherrrr:c Ksr|} ||}|jd|||||||d|\}} ||}| |}|} ||}||}||}| |}|S)N)rzrr\rrrrr7)rrrrrr) r2rzrr\rrrrrresidualrr7r7r8ras*     zAfmoeDecoderLayer.forward)NNNNNN)rcrdrerr rir)rDrfrrboolrjrr FloatTensorrarlr7r7r5r8rs8  rcs`eZdZUdZeed<dZdgZdgZe e dZ gdZ dZ dZdZdZdZfd d ZZS) AfmoePreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r#modelrpast_key_values)rz attentions)rrrrrrnormrTcsFt|t|trt|jjdSt|tr!t|j dSdS)zInitialize the weightsN) r( _init_weightsrTrinitzeros_rrsrr)r2rr5r7r8r$s   z"AfmoePreTrainedModel._init_weights)rcrdrerr rgbase_model_prefix_no_split_modules_skip_keys_device_placementrr_can_record_outputs_keep_in_fp32_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendsupports_gradient_checkpointingr$rlr7r7r5r8rs  rcseZdZdZdeffdd Zeee       dde j dBde j dBde j dBd e j dBd e dBd e j dBd edBd eedeeBfddZZS) AfmoeModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`AfmoeDecoderLayer`] Args: config: AfmoeConfig r#cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr7)r).0rr#r7r8 *sz'AfmoeModel.__init__..rr3F)r(r) pad_token_id padding_idx vocab_sizer EmbeddingrB embed_tokens ModuleListrnum_hidden_layerslayersrnrr#r! rotary_embgradient_checkpointing post_initrr5r3r8r)#s   zAfmoeModel.__init__N input_idsr inputs_embedsr\r!rrrr:c KsD|du|duAr td|r|durt|jd}|dur!||}|dur=|dur-|nd} tj| | |jd|jd}|durF| d}t |} t se|j||||d} t d i| t d i| d} |} |jjrs| |jjd} || |} |jD]}|| f| |j||||| d |} q||| } t| |r|d Sdd S) Nz:You must specify exactly one of input_ids or inputs_embedsr3rr)r3)r#rArrr!)full_attentionrg?)rr\rrrr)last_hidden_stater!r7) ValueErrorr r#r9get_seq_lengthrDrErSr3rrTdictrr mup_enabledrBr=r<rr#r)r2r@rrAr\r!rrrpast_seen_tokenscausal_mask_mapping mask_kwargsrzr decoder_layerr7r7r8ra2sb          zAfmoeModel.forward)NNNNNNN)rcrdrerr r)rrrrDrrfrrrrrrjrrarlr7r7r5r8r1s@  r1cseZdZddiZddiZddgdgfiZfddZee dd e j d Bd e j d Bd e j d Bde d Bde j d Bde j d Bded Bde j d Bdee j BdeedefddZZS)AfmoeForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputrzlogitscs@t|t||_|j|_tj|j|jdd|_| dSr) r(r)r1r r7rrrBrMr?rr5r7r8r)s   zAfmoeForCausalLM.__init__Nrr@rr\r!rAlabelsrrlogits_to_keeprr:c  Ks|jd|||||||d| } | j} t| trt| dn| } || dd| ddf}d}|durB|jd|||jjd| }t ||| j | j | j dS)a Example: ```python >>> from transformers import AutoTokenizer, AfmoeForCausalLM >>> model = AfmoeForCausalLM.from_pretrained("meta-afmoe/Afmoe-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-afmoe/Afmoe-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)r@rr\r!rArrN)rOrPr7)lossrOr!rzr"r7) r rCrTrislicerM loss_functionr#r7rr!rzr")r2r@rr\r!rArPrrrQroutputsrz slice_indicesrOrRr7r7r8ras0  zAfmoeForCausalLM.forward) NNNNNNNNr)rcrdre_tied_weights_keys_tp_plan_pp_planr)rrrDrrfrrrrirrrrarlr7r7r5r8rL|sN     rL)rLr1r)r)r)Ecollections.abcrtypingrrDrrr% activationsr cache_utilsrr generationr integrationsr r r masking_utilsrrmodeling_layersrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_afmoer Moduler!rnr~rr:rrrrrfrirrHrrrrr1rL__all__r7r7r7r8sr          A>  [E'a I