o i@sddlmZmZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZd d lmZmZmZmZd d lmZe eZGdddeddZGdddejZGdddeZGdddeZGdddeZGdddeZgdZ dS))Optional TypedDictN)nn)ACT2FN)Cache)Unpack)logging)deprecate_kwarg)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigc@s@eZdZUdZejed<ejed<eed<eed<ejed<dS)GraniteFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__int IntTensorr!r!q/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyr&s   rF)totalcs<eZdZdZdeffdd ZdejdejfddZZ S) GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configcsZt|j|_|j|_t|j|_tj |j|jddd|_ tj |j|jdd|_ dS)Nr F)bias) super__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr% __class__r!r"r(Hs  zGraniteMoeSharedMLP.__init__ hidden_statesreturncCs<||}|jddd}||d|d}||}|S)Nr )dimrr)r/chunkr-r0)r2r5chunked_hidden_statesr!r!r"forwardQs  zGraniteMoeSharedMLP.forward) rrrrrr(rTensorr; __classcell__r!r!r3r"r$?s r$cseZdZdedeffdd Zedddd dd ejd e ejd e ej de e de e de e de ej de e de e ejejfdeede eje e ejejfffddZZS)GraniteMoeSharedDecoderLayerr% layer_idxcs0t|||jdkrd|_dSt||_dS)Nr)r'r(r+r$ shared_mlp)r2r%r?r3r!r"r(Zs"z%GraniteMoeSharedDecoderLayer.__init__past_key_valuepast_key_valuesz4.58)new_nameversionNFr5attention_mask position_idsoutput_attentions use_cachecache_positionoutput_router_logitsposition_embeddingskwargsr6c  Ks|} ||}|jd|||||||| d| \}} | ||j}|} ||}||\} }|jdur7| }n| ||}~ | ||j}|f}|rP|| f7}|rW||f7}|S)a1 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. )r5rErFrBrGrHrIrKNr!)input_layernorm self_attnresidual_multiplierpost_attention_layernormblock_sparse_moer@)r2r5rErFrBrGrHrIrJrKrLresidualself_attn_weightsmoe_hidden_states router_logitsoutputsr!r!r"r;^s:'       z$GraniteMoeSharedDecoderLayer.forward)NNNFFNFN)rrrrrr(r rr<rrrbooltuplerr FloatTensorr;r=r!r!r3r"r>YsD     r>c@seZdZUeed<dgZdS)GraniteMoeSharedPreTrainedModelr%r>N)rrrrr_no_split_modulesr!r!r!r"rZs  rZcs"eZdZdeffdd ZZS)GraniteMoeSharedModelr%cs0ttfddtjD|_dS)Ncsg|]}t|qSr!)r>).0r?r%r!r" sz2GraniteMoeSharedModel.__init__..)r'r(r ModuleListrangenum_hidden_layerslayersr1r3r^r"r(s  zGraniteMoeSharedModel.__init__)rrrrr(r=r!r!r3r"r\sr\cs(eZdZdgZdeffdd ZZS)GraniteMoeSharedForCausalLMzlm_head.weightr%cs"t|t||_|dS)N)r'r(r\model post_initr1r3r!r"r(s   z$GraniteMoeSharedForCausalLM.__init__)rrr_tied_weights_keysrr(r=r!r!r3r"rdsrd)rdr\rZ)!typingrrrr activationsr cache_utilsrprocessing_utilsrutilsr utils.deprecationr granitemoe.modeling_granitemoer r rrconfiguration_granitemoesharedr get_loggerrloggerrModuler$r>rZr\rd__all__r!r!r!r"s$        X