o eiy`@s ddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZdd lmZmZdd lmZdd lmZmZdd lmZmZdd lmZmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*Gdddej+Z,Gdddej+Z-dej.de/dej.fddZ0 d9dej+dej.d ej.d!ej.d"ej.dBd#e1d$e1d%ee!fd&d'Z2d(d)Z3d:d*d+Z4ee4Gd,d-d-ej+Z5Gd.d/d/ej+Z6Gd0d1d1eZ7e"Gd2d3d3eZ8e"Gd4d5d5e8Z9e"Gd6d7d7e8e Z:gd8Z;dS);)Callable)OptionalN)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernelized_func)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)capture_outputs) Cohere2Configc s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)Cohere2RotaryEmbeddinginv_freqNconfigcst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrF) persistentoriginal_inv_freq) super__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrrope_parametersrcompute_default_rope_parametersrattention_scalingregister_bufferclone)selfrdevice rope_init_fnr __class__j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/cohere2/modeling_cohere2.pyr$.s   zCohere2RotaryEmbedding.__init__r.z torch.deviceseq_lenreturnz torch.TensorcCsZ|jd}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rdtype)r.r:) r(getattr hidden_sizenum_attention_headstorcharangeint64tofloat)rr.r4basedimattention_factorrr2r2r3r)>s &z6Cohere2RotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}t|dd*|| dd}t j |ddd }| |j }||j } Wdn1sgwY|j|jd | j|jd fS) NrrmpscpuF) device_typeenabledr8rDr9)rrBexpandshape isinstancer.typestrr transposer>repeat_interleavecosr*sinrAr:) r-x position_idsinv_freq_expandedposition_ids_expandedrIfreqsembrSrTr2r2r3forward\s(&zCohere2RotaryEmbedding.forwardN)NNN)__name__ __module__ __qualname__r>Tensor__annotations__rr$ staticmethodrinttuplerBr)no_gradrr[ __classcell__r2r2r0r3r+s&   rcs&eZdZdfdd ZddZZS) Cohere2LayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)r#r$nn Parameterr>onesweightvariance_epsilon)r-r<epsbiasr0r2r3r$ms  zCohere2LayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NrFT)keepdimr8) r:rAr>float32meanpowrsqrtrmrl)r- hidden_states input_dtyperrvariancer2r2r3r[ss  zCohere2LayerNorm.forward)NrhFr]r^r_r$r[rfr2r2r0r3rglsrgrun_repr5cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rMrLreshape)rurybatchnum_key_value_headsslenr7r2r2r3 repeat_kv}s 0r~modulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr8rrF)rDr:)ptrainingr)r~num_key_value_groupsr>matmulrQri functionalsoftmaxrqrAr:rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputr2r2r3eager_attention_forwards  rcCsB|ddddf}|ddddf}tj| |gddd}|S)N.r8rrFrK)r>stackflatten)rUx1x2rot_xr2r2r3 rotate_halfsrcCsj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r9)r:rB unsqueezerrA)qkrSrT unsqueeze_dimr:q_embedk_embedr2r2r3apply_rotary_pos_embs  rcseZdZdZddededBffdd Z  ddejde ejejfd ejdBd e dBd ej dBd e e d e ejejdBe ejdBffddZZS)Cohere2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t |dr4|j|nd}|dkr=|jnd|_tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_dS)Nr7gT layer_typessliding_attentionro)r#r$rrr;r<r=r7r|rrattention_dropout is_causalhasattrrsliding_windowriLinearattention_biasq_projk_projv_projo_proj)r-rr layer_typer0r2r3r$s,   zCohere2Attention.__init__ruposition_embeddingsrpast_key_valuescache_positionrr5cKs&|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } |jdurGt| | | | \} } |dur\| | |d}| | | |j |\} } t |j jt}||| | | |f|jspdn|j|j|jd|\}}|jg|dR}||}||fS)NrFrr8)rTrSrr)rrr)rMr7rviewrQrrrrupdaterr get_interfacer_attn_implementationrrrrrzrr)r-rurrrrr input_shape hidden_shape query_statesrrrSrT cache_kwargsattention_interfacerrr2r2r3r[s<     zCohere2Attention.forwardr\)NN)r]r^r___doc__rrcr$r>r`rdr LongTensorrrr[rfr2r2r0r3rs(rcs$eZdZfddZddZZS) Cohere2MLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) r#r$rr<intermediate_sizerir gate_projup_proj down_projr hidden_actact_fnr-rr0r2r3r$s zCohere2MLP.__init__cCs$||||||}|Sr\)rrrr)r-rUrr2r2r3r[s zCohere2MLP.forwardrxr2r2r0r3rs  rcseZdZdedeffdd Z     ddejdeejejfdBd ejdBd e dBd e dBd ej dBd e e deejeejejfdBffddZZS)Cohere2DecoderLayerrrcsLt|j|_t||d|_t||_t|j|jd|_ |j ||_ dS)N)rrr<rn) r#r$r<r self_attnrmlprglayer_norm_epsinput_layernormrattention_type)r-rrr0r2r3r$!s  zCohere2DecoderLayer.__init__NFrurrr use_cacherrr5c KsJ|}||}|jd||||||d|\} } ||} || | }|S)ar Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )rurrrrrNr2)rrr) r-rurrrrrrresidualhidden_states_attention_hidden_states_mlpr2r2r3r[)s  zCohere2DecoderLayer.forward)NNNFN)r]r^r_rrcr$r>r`rdrboolrrr FloatTensorr[rfr2r2r0r3r s0  rc@sHeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)Cohere2PreTrainedModelrmodelTrr)ru attentionsN)r]r^r_rrabase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr2r2r2r3rWs  rcseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd e dBd ej dBd eed efddZZS) Cohere2Modelrcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_t|_d|_|dS)Ncsg|]}t|qSr2)r).0rrr2r3 ssz)Cohere2Model.__init__..rF)r#r$ pad_token_id padding_idx vocab_sizeri Embeddingr< embed_tokens ModuleListrangenum_hidden_layerslayersrgrnormr rotary_embgradient_checkpointing post_initrr0rr3r$ls   zCohere2Model.__init__N input_idsrrVr inputs_embedsrrrr5c Ks"|du|duAr td|dur||}|r!|dur!t|jd}|dur=|dur-|nd} tj| | |jd|jd}|durF| d}t |} t sf|j|||||d} t d i| t d i| d} |} || |} |jD]}|| f| |j| ||||d|} qq|| } t| |d S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)r.)rrrrrrV)full_attentionr)rrrrrrV)last_hidden_staterr2) ValueErrorrrrget_seq_lengthr>r?rMr.rrNdictr r rrrrr )r-rrrVrrrrrpast_seen_tokenscausal_mask_mapping mask_kwargsrur decoder_layerr2r2r3r[|sX        zCohere2Model.forward)NNNNNNN)r]r^r_rr$rrrr>rr`rrrrrr r[rfr2r2r0r3rjs>  rcseZdZddiZddiZddgdgfiZfddZee dd e j d Bd e j d Bd e j d Bde d Bde j d Bde j d Bded Bded Bded Bde j d Bdee j BdeedefddZZS)Cohere2ForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputrulogitscsPt|t||_|j|_tj|j|jdd|_|j |_ |j |_ | dSr) r#r$rrrrirr<r logit_scaletie_word_embeddingsrrr0r2r3r$s   zCohere2ForCausalLM.__init__NrrrrVrrlabelsroutput_attentionsoutput_hidden_statesrlogits_to_keeprr5c  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||j }d}|dur]|j d|||jj d| }t ||| j | j| jdS)a~ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, Cohere2ForCausalLM >> model = Cohere2ForCausalLM.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrVrrrrrr)rrr)lossrrrurr2)rrrrrrNrcslicerr loss_functionrrrrur)r-rrrVrrrrrrrr routputsru slice_indicesrr r2r2r3r[s<'   zCohere2ForCausalLM.forward) NNNNNNNNNNr)r]r^r__tied_weights_keys_tp_plan_pp_planr$rrr>rr`rrrrcrrrr[rfr2r2r0r3rsZ       r)rrr)r)r)<collections.abcrtypingrr>torch.nnri activationsr cache_utilsrr generationr integrationsr masking_utilsr r modeling_layersr modeling_outputsr rmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_cohere2rModulerrgr`rcr~rBrrrrrrrrr__all__r2r2r2r3sh          A  I7T ]