o wi@sddlmZddlZddlmZddlmZddlmZddlm Z dd l m Z m Z m Z mZd d lmZe eZGd d d ejZGddde ZGdddeZGddde ZGddde ZgdZdS))OptionalN)nn)ACT2FN)Cache)logging)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigcs<eZdZdZdeffdd ZdejdejfddZZ S) GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configcsZt|j|_|j|_t|j|_tj |j|jddd|_ tj |j|jdd|_ dS)NrF)bias) super__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr __class__z/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyr-s  zGraniteMoeSharedMLP.__init__ hidden_statesreturncCs<||}|jddd}||d|d}||}|S)Nr)dimrr )rchunkrr)rr"chunked_hidden_statesr r r!forward6s  zGraniteMoeSharedMLP.forward) __name__ __module__ __qualname____doc__rrtorchTensorr( __classcell__r r rr!r$s rcseZdZdedeffdd Z        ddejdeejd eej d ee d ee d ee d eej dee dee ejejfde ej ee ej ej fffddZZS)GraniteMoeSharedDecoderLayerr layer_idxcs0t|||jdkrd|_dSt||_dS)Nr)rrrr shared_mlp)rrr1rr r!r?s"z%GraniteMoeSharedDecoderLayer.__init__NFr"attention_mask position_idspast_key_valueoutput_attentions use_cachecache_positionoutput_router_logitsposition_embeddingsr#c  Ks|} ||}|jd|||||||| d| \}} } | ||j}|} ||}||\}}|jdur8|}n|||}~| ||j}|f}|rQ|| f7}|rX|| f7}|r_||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r"r3r4r5r6r7r8r:Nr )input_layernorm self_attnresidual_multiplierpost_attention_layernormblock_sparse_moer2)rr"r3r4r5r6r7r8r9r:kwargsresidualself_attn_weightspresent_key_valuemoe_hidden_states router_logitsoutputsr r r!r(Cs>&        z$GraniteMoeSharedDecoderLayer.forward)NNNFFNFN)r)r*r+rintrr-r.r LongTensorrbooltuple FloatTensorr(r/r r rr!r0>s>   r0c@seZdZeZdgZdS)GraniteMoeSharedPreTrainedModelr0N)r)r*r+r config_class_no_split_modulesr r r r!rLs rLcs"eZdZdeffdd ZZS)GraniteMoeSharedModelrcs0ttfddtjD|_dS)Ncsg|]}t|qSr )r0).0r1rr r! sz2GraniteMoeSharedModel.__init__..)rrr ModuleListrangenum_hidden_layerslayersrrrQr!rs  zGraniteMoeSharedModel.__init__)r)r*r+rrr/r r rr!rOsrOcs(eZdZdgZdeffdd ZZS)GraniteMoeSharedForCausalLMzlm_head.weightrcs"t|t||_|dS)N)rrrOmodel post_initrrr r!rs   z$GraniteMoeSharedForCausalLM.__init__)r)r*r+_tied_weights_keysrrr/r r rr!rWsrW)rWrOrL)typingrr-r activationsr cache_utilsrutilsrgranitemoe.modeling_granitemoer r r r configuration_granitemoesharedr get_loggerr)loggerModulerr0rLrOrW__all__r r r r!s       Z