o i\@sddlmZmZmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZmZdd lmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)Gdddej*Z+Gdddej*Z,dej-de.dej-fddZ/ d8dej*dej-dej-d ej-d!eej-d"e0d#e0d$ee!fd%d&Z1d'd(Z2d9d)d*Z3Gd+d,d,ej*Z4Gd-d.d.ej*Z5Gd/d0d0eZ6e"Gd1d2d2eZ7e"Gd3d4d4e7Z8e"Gd5d6d6e7e Z9gd7Z:dS):)CallableOptionalUnionN)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)deprecate_kwarg)check_model_inputs) Cohere2ConfigcsDeZdZUejed<ddeffdd Zee ddZ Z S) Cohere2RotaryEmbeddinginv_freqNconfigcstt|drt|jtr|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultrF) persistent)super__init__hasattr isinstancerdictgetr max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrr rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)selfrdevicer __class__i/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/cohere2/modeling_cohere2.pyr%.s   zCohere2RotaryEmbedding.__init__c Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}tj |dd*|| dd}tj |ddd }| |j }||j } Wdn1shwY|j|jd | j|jd fS) NrrmpscpuF) device_typeenableddimdtype)rfloatexpandshaper'r2r!strtorchautocast transposerepeat_interleavecosr.sintor@) r1x position_idsinv_freq_expandedposition_ids_expandedr:freqsembrIrJr5r5r6forward?s(&zCohere2RotaryEmbedding.forwardN) __name__ __module__ __qualname__rETensor__annotations__rr%no_gradrrR __classcell__r5r5r3r6r+s rcs&eZdZdfdd ZddZZS) Cohere2LayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)r$r%nn ParameterrEonesweightvariance_epsilon)r1 hidden_sizeepsbiasr3r5r6r%Ps  zCohere2LayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)Nr7T)keepdimr<) r@rKrEfloat32meanpowrsqrtrar`)r1 hidden_states input_dtypergvariancer5r5r6rRVs  zCohere2LayerNorm.forward)Nr\FrTrUrVr%rRrZr5r5r3r6r[Osr[rjn_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rCrBreshape)rjrnbatchnum_key_value_headsslenhead_dimr5r5r6 repeat_kv`s 0rumodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr<rr7)r>r@)ptrainingr)runum_key_value_groupsrEmatmulrGrCr] functionalsoftmaxrfrKr@r}r contiguous) rwrxryrzr{r|r}r~ key_states value_states attn_weights causal_mask attn_outputr5r5r6eager_attention_forwardls & rcCsB|ddddf}|ddddf}tj| |gddd}|S)N.r<rr7r=r)rEstackflatten)rLx1x2rot_xr5r5r6 rotate_halfsrc Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r?)r@rA unsqueezerrK) qkrIrJrM unsqueeze_dimr@q_embedk_embedr5r5r6apply_rotary_pos_embs  rcseZdZdZddedeeffdd Zeddd d   dd e j d e e j e j fd ee j dee dee j deede e j ee j ee e j ffddZZS)Cohere2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ |j |dkr4|jnd|_tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_dS)NrtgTsliding_attentionrd)r$r%rrgetattrrbnum_attention_headsrtrrrr|attention_dropout is_causal layer_typessliding_windowr]Linearattention_biasq_projk_projv_projo_projr1rrr3r5r6r%s*   zCohere2Attention.__init__past_key_valuepast_key_values4.58new_nameversionrjposition_embeddingsr{cache_positionr~rocKs2|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } |jdurGt| | | | \} } |dur\| | |d}| | | |j |\} } t }|j j dkrjt|j j }||| | | |f|jsvdn|j|j|jd|\}}|jg|dR}||}||fS)Nr7rr<)rJrIreagerrv)r}r|r)rCrtrviewrGrrrrupdaterrr_attn_implementationrrrr|rprr)r1rjrr{rrr~ input_shape hidden_shape query_statesrrrIrJ cache_kwargsattention_interfacerrr5r5r6rRs<       zCohere2Attention.forwardrS)NN)rTrUrV__doc__rrintr%rrErWtupler LongTensorrr rRrZr5r5r3r6rs* rcs$eZdZfddZddZZS) Cohere2MLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) r$r%rrbintermediate_sizer]r gate_projup_proj down_projr hidden_actact_fnr1rr3r5r6r%s zCohere2MLP.__init__cCs$||||||}|SrS)rrrr)r1rLrr5r5r6rRs zCohere2MLP.forwardrmr5r5r3r6rs  rcseZdZdedeffdd Zedddd dd ejd e ejejfd e ejde e de e de ej deede eje e ejejfffddZZS)Cohere2DecoderLayerrrcsLt|j|_t||d|_t||_t|j|jd|_ |j ||_ dS)N)rrrbrc) r$r%rbr self_attnrmlpr[layer_norm_epsinput_layernormrattention_typerr3r5r6r%s  zCohere2DecoderLayer.__init__rrrrNFrjrr{ use_cacherr~roc KsJ|}||}|jd||||||d|\} } ||} || | }|S)ar Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )rjrr{rrrNr5)rrr) r1rjrr{rrrr~residualhidden_states_attention_hidden_states_mlpr5r5r6rRs  zCohere2DecoderLayer.forward)NNFN)rTrUrVrrr%rrErWrrrboolrrr FloatTensorrRrZr5r5r3r6rs0  rc@sHeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)Cohere2PreTrainedModelrmodelTrr)rj attentionsN)rTrUrVrrXbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr5r5r5r6r=s  rcseZdZdeffdd Zee       ddeej deej deej dee d eej d ee d eej d eed efddZZS) Cohere2Modelrcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr5)r).0rrr5r6 Ysz)Cohere2Model.__init__..rrF)r$r% pad_token_id padding_idx vocab_sizer] Embeddingrb embed_tokens ModuleListrangenum_hidden_layerslayersr[rnormr rotary_embgradient_checkpointing post_initrr3rr6r%Rs   zCohere2Model.__init__N input_idsr{rMr inputs_embedsrrr~roc Ks&|du|duAr td|dur||}|r$|dur$|js$t|jd}|dur@|dur0|nd} tj| | |jd|j d}|durI| d}t |} t si|j|||||d} t d i| td i| d} |} || |} |jD]}|| f| | |j|||d|} qt|| } t| |d S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)r2)r input_embedsr{rrrM)full_attentionr)rr{rrr)last_hidden_staterr5) ValueErrorrrrrget_seq_lengthrEarangerCr2rr'r(r r rrrrr)r1rr{rMrrrrr~past_seen_tokenscausal_mask_mapping mask_kwargsrjr decoder_layerr5r5r6rRbsV        zCohere2Model.forward)NNNNNNN)rTrUrVrr%rrrrErrWrrrrrrrRrZr5r5r3r6rPs<  rcseZdZdgZddiZddgdgfiZfddZee           dd e e j d e e j d e e j d e e eee jfde e jde e j de ede ede ede e j de ee j fdeedefddZZS)Cohere2ForCausalLMzlm_head.weightlm_head colwise_reprjlogitscsPt|t||_|j|_tj|j|jdd|_|j |_ |j |_ | dSr) r$r%rrrr]rrbr logit_scaletie_word_embeddingsrrr3r5r6r%s   zCohere2ForCausalLM.__init__Nrrr{rMrrlabelsroutput_attentionsoutput_hidden_statesrlogits_to_keepr~roc  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||j }d}|dur]|j d|||jj d| }t ||| j | j| jdS)a~ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, Cohere2ForCausalLM >> model = Cohere2ForCausalLM.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rr{rMrrrrr r)rrr)lossrrrjrr5)rrr rrr'rslicerr loss_functionrrrrjr)r1rr{rMrrrrrr rr r~outputsrj slice_indicesrr r5r5r6rRs<'   zCohere2ForCausalLM.forward) NNNNNNNNNNr)rTrUrV_tied_weights_keys_tp_plan_pp_planr%rrrrErrWrrlistrrrrrrrRrZr5r5r3r6rsZ       r)rrr)rv)Nr);typingrrrrEtorch.nnr] activationsr cache_utilsrr generationr masking_utilsr r modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrutils.deprecationr utils.genericrconfiguration_cohere2rModulerr[rWrrurArrrrrrrrr__all__r5r5r5r6sd         $  I8R ]