o ei4@sddlZddlmZddlmZddlmZddlmZm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZddlmZmZddlmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*GdddeZ+GdddeZ,GdddeZ-Gddde Z.Gdddej/Z0Gdd d e"Z1Gd!d"d"e%Z2eGd#d$d$e#eZ3eGd%d&d&e'Z4Gd'd(d(e&Z5gd)Z6dS)*N)nn)initialization)ACT2FN)Cache DynamicCache)create_causal_mask)MoeCausalLMOutputWithPastMoeModelOutputWithPast)PreTrainedModel)Unpack)TransformersKwargsauto_docstring)can_return_tuplemerge_with_config_defaults)capture_outputs)GraniteRMSNormGraniteRotaryEmbedding)JetMoeParallelExpertsJetMoeTopKGating)LlamaAttentionLlamaPreTrainedModel)MixtralDecoderLayerMixtralForCausalLM MixtralModelload_balancing_loss_func)GraniteMoeConfigc@ eZdZdS)GraniteMoeRMSNormN__name__ __module__ __qualname__r%r%o/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/granitemoe/modular_granitemoe.pyr $r c@r)GraniteMoeRotaryEmbeddingNr!r%r%r%r&r((r'r(c@r)GraniteMoeParallelExpertsNr!r%r%r%r&r),r'r)c@r)GraniteMoeTopKGatingNr!r%r%r%r&r*0r'r*cs.eZdZdZdeffdd ZddZZS) GraniteMoeMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. configcslt|j|_|j|_t|j|_t|j |j|jd|_ t|j |j|j|_ t |j|j |j d|_dS)Nr) input_size num_expertstop_k)super__init__ hidden_sizer-intermediate_sizer hidden_act activationr)num_local_experts input_linear output_linearr*num_experts_per_tokrouterselfr, __class__r%r&r1=s   zGraniteMoeMoE.__init__cCs|\}}}|d|}||\}}}}}||} || |} | jddd} || d| d} || |} | |dddf} tj|||j f| j | j d} | d|| }| |||j }|S)Nr)dimrr)dtypedevice)sizereshaper:r7chunkr5r8torchzerosr-rArB index_addview)r< layer_inputbszlengthemb_size_ batch_index batch_gates expert_size expert_inputs hidden_stateschunked_hidden_statesexpert_outputsrG layer_outputr%r%r&forwardLs   zGraniteMoeMoE.forward)r"r#r$__doc__rr1rW __classcell__r%r%r=r&r+4sr+cs&eZdZdedeffdd ZZS)GraniteMoeAttentionr, layer_idxcst||||j|_dSN)r0r1attention_multiplierscalingr<r,r[r=r%r&r1`s zGraniteMoeAttention.__init__)r"r#r$rintr1rYr%r%r=r&rZ_srZc sveZdZdedeffdd Z    ddejdejdBdedBd ej dBd e ejejfdBd ejf d d Z Z S)GraniteMoeDecoderLayerr,r[csdt||t||d|_t||_t|j|jd|_ t|j|jd|_ |` t||_|j |_ dS)N)r,r[eps) r0r1rZ self_attnr+block_sparse_moer r2 rms_norm_epsinput_layernormpost_attention_layernormmlpresidual_multiplierr_r=r%r&r1fs   zGraniteMoeDecoderLayer.__init__NrSattention_maskpast_key_valuescache_positionposition_embeddingsreturnc Ksf|}||}|jd|||||d|\}}|||j}|}||}||}|||j}|S)N)rSrkrlrmrnr%)rgrdrjrhre) r<rSrkrlrmrnkwargsresidualrNr%r%r&rWps"    zGraniteMoeDecoderLayer.forward)NNNN) r"r#r$rr`r1rFTensorr LongTensortuplerWrYr%r%r=r&raes& rac@sFeZdZUeed<dZdZdgZdgZdZ dZ dZ e ddZd S) GraniteMoePreTrainedModelr,modelTrarlFcCs4t||t|trtj|jd|jjddSdS)Ng)meanstd) r _init_weights isinstancer)initnormal_weightr,initializer_range)r<moduler%r%r&rys  z'GraniteMoePreTrainedModel._init_weightsN)r"r#r$r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphrFno_gradryr%r%r%r&rus rucseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd e dBd ej dBd eed efddZZS)GraniteMoeModelr,csJttfddtjD|_tjj d|_ j |_ dS)Ncsg|]}t|qSr%)ra).0r[r,r%r& sz,GraniteMoeModel.__init__..rb) r0r1r ModuleListrangenum_hidden_layerslayersr r2rfnormembedding_multiplierr;r=rr&r1s  zGraniteMoeModel.__init__N input_idsrk position_idsrl inputs_embeds use_cachermrproc Ks|du|duAr td|r|durt|jd}|dur!||}|dur=|dur-|nd} tj| | |jd|jd}|durF| d}t |j|||||d} ||j }|} | | |} |j d|jjD]} | | f| | ||||d|} qg|| } t| |dS) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)rB)r,rrkrmrlr)rnrkrrlrrm)last_hidden_staterl) ValueErrorrr, embed_tokensget_seq_lengthrFarangeshaperB unsqueezerr rotary_embrrrr )r<rrkrrlrrrmrppast_seen_tokens causal_maskrSrn decoder_layerr%r%r&rWsT       zGraniteMoeModel.forward)NNNNNNN)r"r#r$rr1rrrrFrsrrr FloatTensorboolr r r rWrYr%r%r=r&rs>  rcseZdZdeffdd Zee         ddejdBdej dBdejdBd e dBd ej dBd ejdBd e dBd ejdBde ej BdeeBfddZZS)GraniteMoeForCausalLMr,cs"t|t||_|j|_dSr\)r0r1rrvlogits_scalingr;r=r%r&r1s   zGraniteMoeForCausalLM.__init__Nrrrkrrlrlabelsoutput_router_logitsrmlogits_to_keeproc  Ks|dur|n|jj}|jd||||||d| } | j} t| tr't| dn| } || dd| ddf}||jj}d}|durQ|j ||fd|jj i| }d}|rnt | j |j |j|}|durn||j||j7}t|||| j| j| j| j dS)al labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, GraniteMoeForCausalLM >>> model = GraniteMoeForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)rrkrrlrrm vocab_size)lossaux_losslogitsrlrS attentions router_logitsr%)r,rrvrrzr`slicelm_headr loss_functionrrrr.r9router_aux_loss_coeftorBr rlrSr)r<rrkrrlrrrrmrrpoutputsrS slice_indicesrrrr%r%r&rWsZ&  zGraniteMoeForCausalLM.forward) NNNNNNNNr)r"r#r$rr1rrrFrsrrrrrr`rtr rWrYr%r%r=r&rsD   r)rrru)7rFrrr{ activationsr cache_utilsrr masking_utilsrmodeling_outputsr r modeling_utilsr processing_utilsr utilsr r utils.genericrrutils.output_capturingrgranite.modeling_graniterrjetmoe.modeling_jetmoerrllama.modeling_llamarrmixtral.modeling_mixtralrrrrconfiguration_granitemoerr r(r)r*Moduler+rZrarurr__all__r%r%r%r&s<        +&J ^