o i_@sddlmZmZmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZmZddlmZddlmZm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'Gdddej(Z)Gdddej(Z*Gdddej(Z+dej,de-dej,fddZ. d9d ej(d!ej,d"ej,d#ej,d$eej,d%e/d&e/d'eefd(d)Z0d*d+Z1d:d,d-Z2Gd.d/d/ej(Z3Gd0d1d1eZ4e Gd2d3d3eZ5e Gd4d5d5e5Z6e Gd6d7d7e5e Z7gd8Z8dS);)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)deprecate_kwarg)check_model_inputs) CohereConfigcs&eZdZdfdd ZddZZS) CohereLayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeepsbias __class__g/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/cohere/modeling_cohere.pyr4s  zCohereLayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NT)keepdim) dtypetor!float32meanpowrsqrtr$r#)r% hidden_states input_dtyper3variancer+r+r,forward:s  zCohereLayerNorm.forward)NrF__name__ __module__ __qualname__rr9 __classcell__r+r+r)r,r3srcsDeZdZUejed<ddeffdd Zee ddZ Z S) CohereRotaryEmbeddinginv_freqNconfigcstt|drt|jtr|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultr@F) persistent)rrhasattr isinstancerBdictgetrCmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrAr rope_init_fnattention_scalingregister_bufferr@original_inv_freq)r%rAdevicer@r)r+r,rGs   zCohereRotaryEmbedding.__init__c Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}tj |dd*|| dd}tj |ddd }| |j }||j } Wdn1shwY|j|jd | j|jd fS) Nrr-rmpscpuF) device_typeenabledr/dimr0)r@floatexpandshaperHrRrDstrr!autocast transposerepeat_interleavecosrOsinr1r0) r%x position_idsinv_freq_expandedposition_ids_expandedrUfreqsembrarbr+r+r,r9Xs(&zCohereRotaryEmbedding.forwardN) r;r<r=r!Tensor__annotations__rrno_gradrr9r>r+r+r)r,r?Ds r?cs$eZdZfddZddZZS) CohereMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr() rrrAr&intermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnr%rAr)r+r,ris zCohereMLP.__init__cCs$||||||}|Sri)rtrvrrrs)r%rcrtr+r+r,r9ss zCohereMLP.forwardr:r+r+r)r,rmhs  rmr6n_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r\r[reshape)r6rxbatchnum_key_value_headsslenhead_dimr+r+r, repeat_kvxs 0rmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr/rr-)rXr0)ptrainingr)rnum_key_value_groupsr!matmulr_r\r functionalsoftmaxr2r1r0rr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputr+r+r,eager_attention_forwards & rcCsB|ddddf}|ddddf}tj| |gddd}|S)N.r/rr-rWr)r!stackflatten)rcx1x2rot_xr+r+r, rotate_halfsrc Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. rY)r0rZ unsqueezerr1) qkrarbrd unsqueeze_dimr0q_embedk_embedr+r+r,apply_rotary_pos_embs  rcseZdZdZddedeeffdd Zeddd d   dd e j d e e j e j fd ee j dee dee j deede e j ee j ffddZZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrA layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_|j|_|jrt|j|jf|jd|_t|j|jf|jd|_dSdS)Nr~gTror&r')rrrArgetattrr&num_attention_headsr~r|rrattention_dropout is_causalrrqattention_biasq_projk_projv_projo_proj use_qk_normrlayer_norm_epsq_normk_normr%rArr)r+r,rs:   zCohereAttention.__init__past_key_valuepast_key_values4.58new_nameversionr6position_embeddingsrcache_positionrrycKsJ|jdd}g|d|jR}|||} |||} |||} |jr6|| } || } | dd} | dd} | dd} |\} } t | | | | \} } |durj| | |d}| | | |j |\} } t }|jjdkrxt|jj}||| | | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)Nr-rr/)rbrareagerr)rr)r\r~rviewrrrrrr_rupdaterrrA_attn_implementationrrrrrzrr)r%r6rrrrr input_shape hidden_shape query_statesrrrarb cache_kwargsattention_interfacerrr+r+r,r9sD           zCohereAttention.forwardri)NN)r;r<r=__doc__rrintrrr!rjtupler LongTensorrr r9r>r+r+r)r,rs* rcseZdZdedeffdd Zedddd dd ejd e ejd e ej de e de e de ej de e ejejfdeede eje e ejejfffddZZS)CohereDecoderLayerrArcs@t|j|_t||d|_t||_t|j|jd|_ dS)N)rArr) rrr&r self_attnrmmlprrinput_layernormrr)r+r,rs  zCohereDecoderLayer.__init__rrrrNFr6rrd use_cacherrrryc KsL|} ||}|jd|||||||d|\} } ||} | | | }|S)ar Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r6rrdrrrrNr+)rrr) r%r6rrdrrrrrresidualhidden_states_attention_hidden_states_mlpr+r+r,r9$s   zCohereDecoderLayer.forward)NNNFNN)r;r<r=rrrrr!rjrrrboolrrr FloatTensorr9r>r+r+r)r,rs8   rc@sHeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)CoherePreTrainedModelrAmodelTrr)r6 attentionsN)r;r<r=rrkbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr+r+r+r,rVs  rcseZdZdeffdd Zee       ddeej deej deej dee d eej d eej d ee d eed efddZZS) CohereModelrAcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr+)r).0rrAr+r, rsz(CohereModel.__init__..rrF)rr pad_token_id padding_idx vocab_sizer Embeddingr& embed_tokens ModuleListrangenum_hidden_layerslayersrrnormr? rotary_embgradient_checkpointing post_initrwr)rr,rks   zCohereModel.__init__N input_idsrrdr inputs_embedsrrrryc Ks|du|duAr td|dur||}|r!|dur!t|jd}|dur=|dur-|nd} tj| | |jd|jd}|durF| d}t |j|||||d} |} | | |} |j d|jj D]} | | f| |||| d|} qb|| } t| |dS) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)rR)rA input_embedsrrrrd)rrdrrr)last_hidden_stater) ValueErrorrr rAget_seq_lengthr!aranger\rRrr rrrrr)r%rrrdrrrrrpast_seen_tokensrr6r decoder_layerr+r+r,r9{sP       zCohereModel.forward)NNNNNNN)r;r<r=rrrrrr!rrjrrrrrrr9r>r+r+r)r,ris<  rcseZdZdgZddiZddgdgfiZfddZee           dd e e j d e e j d e e j d e e eee jfde e jde e j de ede ede ede e j de ee j fdeedefddZZS)CohereForCausalLMzlm_head.weightlm_head colwise_repr6logitscsPt|t||_|j|_tj|j|jdd|_|j |_ |j |_ | dSrn) rrrrrrrqr&r logit_scaletie_word_embeddingsrrwr)r+r,rs   zCohereForCausalLM.__init__Nrrrrdrrlabelsroutput_attentionsoutput_hidden_statesrlogits_to_keeprryc  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||j }d}|dur]|j d|||jj d| }t ||| j | j| jdS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrdrrrrrr)rrr)lossrrr6rr+)rArrrrrHrslicerr loss_functionrrrr6r)r%rrrdrrrrrrrrroutputsr6 slice_indicesrrr+r+r,r9s<'   zCohereForCausalLM.forward) NNNNNNNNNNr)r;r<r=_tied_weights_keys_tp_plan_pp_planrrrrr!rrjrrlistrrrrrrr9r>r+r+r)r,rsZ       r)rrr)r)Nr)9typingrrrr!r activationsr cache_utilsrr generationr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrutils.deprecationr utils.genericrconfiguration_cohererModulerr?rmrjrrrZrrrrrrrr__all__r+r+r+r,sd          $  X:N ]