o eib@sddlmZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z dd l mZdd lmZdd lmZdd lmZdd lmZmZddlmZmZddlmZmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*Gdddej+Z,Gdddej+Z-Gdddej+Z.dej/de0dej/fdd Z1 !d;d"ej+d#ej/d$ej/d%ej/d&ej/dBd'e2d(e2d)ee!fd*d+Z3d,d-Z4dFcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeepsbias __class__h/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/cohere/modeling_cohere.pyr 5s  zCohereLayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NT)keepdim) dtypetor"float32meanpowrsqrtr%r$)r& hidden_states input_dtyper4variancer,r,r-forward;s  zCohereLayerNorm.forward)NrF__name__ __module__ __qualname__r r: __classcell__r,r,r*r-r4src s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)CohereRotaryEmbeddinginv_freqNconfigcst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrAF) persistentoriginal_inv_freq) rr max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrBrope_parametersrCcompute_default_rope_parametersrattention_scalingregister_bufferclone)r&rBdevice rope_init_fnrAr*r,r-r Hs   zCohereRotaryEmbedding.__init__rOz torch.deviceseq_lenreturnz torch.TensorcCsZ|jd}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr0r1)rOr1) rJgetattrr'num_attention_headsr"arangeint64r2float)rBrOrQbasedimattention_factorrAr,r,r-rKXs &z5CohereRotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}t|dd*|| dd}t j |ddd }| |j }||j } Wdn1sgwY|j|jd | j|jd fS) Nrr.rmpscpuF) device_typeenabledr0r\rU)rArZexpandshape isinstancerOtypestrr transposer"repeat_interleavecosrLsinr2r1) r&x position_idsinv_freq_expandedposition_ids_expandedr`freqsembrjrkr,r,r-r:vs(&zCohereRotaryEmbedding.forwardN)NNN)r<r=r>r"Tensor__annotations__rr staticmethodrinttuplerZrKno_gradrr:r?r,r,r*r-r@Es&   r@cs$eZdZfddZddZZS) CohereMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr)) rr rBr'intermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnr&rBr*r,r-r s zCohereMLP.__init__cCs$||||||}|Srr)rrr~r)r&rlrr,r,r-r:s zCohereMLP.forwardr;r,r,r*r-rys  ryr7n_reprRcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rdrcreshape)r7rbatchnum_key_value_headsslenrTr,r,r- repeat_kvs 0rmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr0rr.)r\r1)ptrainingr)rnum_key_value_groupsr"matmulrhr functionalsoftmaxr3r2r1rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputr,r,r-eager_attention_forwards  rcCsB|ddddf}|ddddf}tj| |gddd}|S)N.r0rr.rb)r"stackflatten)rlx1x2rot_xr,r,r- rotate_halfsrcCsj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. rU)r1rZ unsqueezerr2)qkrjrk unsqueeze_dimr1q_embedk_embedr,r,r-apply_rotary_pos_embs  rcseZdZdZddededBffdd Z  ddejde ejejfd ejdBd e dBd ej dBd e e d e ejejdBffddZZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrB layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_|j|_|jrt|j|jf|jd|_t|j|jf|jd|_dSdS)NrTgTr{r'r()rr rBrrVr'rWrTrrrattention_dropout is_causalrr}attention_biasq_projk_projv_projo_proj use_qk_normrlayer_norm_epsq_normk_normr&rBrr*r,r-r s:   zCohereAttention.__init__r7position_embeddingsrpast_key_valuescache_positionrrRcKs>|jdd}g|d|jR}|||} |||} |||} |jr6|| } || } | dd} | dd} | dd} |\} } t | | | | \} } |durj| | |d}| | | |j |\} } t |jjt}||| | | |f|js~dn|j|jd|\}}|jg|dR}||}||fS)Nr.rr0)rkrjrr)rr)rdrTrviewrrrrrrhrupdaterr get_interfacerB_attn_implementationrrrrrrr)r&r7rrrrr input_shape hidden_shape query_statesrrrjrk cache_kwargsattention_interfacerrr,r,r-r:sD         zCohereAttention.forwardrr)NN)r<r=r>__doc__rrvr r"rsrwr LongTensorrr r:r?r,r,r*r-rs(%rcseZdZdedeffdd Z      ddejdejdBd ejdBd e dBd e dBd ejdBd e ejejfdBde e de eje ejejfdBffddZZS)CohereDecoderLayerrBrcs@t|j|_t||d|_t||_t|j|jd|_ dS)N)rBrr) rr r'r self_attnrymlprrinput_layernormrr*r,r-r 8s  zCohereDecoderLayer.__init__NFr7rrmr use_cacherrrrRc KsL|} ||}|jd|||||||d|\} } ||} | | | }|S)ar Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r7rrmrrrrNr,)rrr) r&r7rrmrrrrrresidualhidden_states_attention_hidden_states_mlpr,r,r-r:?s   zCohereDecoderLayer.forward)NNNFNN)r<r=r>rrvr r"rsrrboolrwrr FloatTensorr:r?r,r,r*r-r7s6   rc@sHeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)CoherePreTrainedModelrBmodelTrr)r7 attentionsN)r<r=r>rrtbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr,r,r,r-rps  rcseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd ej dBd e dBd eed efddZZS) CohereModelrBcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr,)r).0rrBr,r- sz(CohereModel.__init__..rrF)rr pad_token_id padding_idx vocab_sizer Embeddingr' embed_tokens ModuleListrangenum_hidden_layerslayersrrnormr@ rotary_embgradient_checkpointing post_initrr*rr-r s   zCohereModel.__init__N input_idsrrmr inputs_embedsrrrrRc Ks|du|duAr td|dur||}|r!|dur!t|jd}|dur<|dur-|nd} tj|jd|jd| }|durE| d}t |j|||||d} |} |j | |d} |j d|jj D]} | | f| | ||||d|} qb|| } t| |d S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)rO)rBrrrrrm)rm)rrrmrrr)last_hidden_stater) ValueErrorrrrBget_seq_lengthr"rXrdrOrr rrrrr)r&rrrmrrrrrpast_seen_tokens causal_maskr7r decoder_layerr,r,r-r:sP      zCohereModel.forward)NNNNNNN)r<r=r>rr rrrr"rrsrrrrrrr:r?r,r,r*r-rs>  rcseZdZddiZddiZddgdgfiZfddZee dd e j d Bd e j d Bd e j d Bde d Bde j d Bde j d Bded Bded Bded Bde j d Bdee j BdeedefddZZS)CohereForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr7logitscsPt|t||_|j|_tj|j|jdd|_|j |_ |j |_ | dSrz) rr rrrrr}r'r logit_scaletie_word_embeddingsrrr*r,r-r s   zCohereForCausalLM.__init__Nrrrrmrrlabelsroutput_attentionsoutput_hidden_statesrlogits_to_keeprrRc  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||j }d}|dur]|j d|||jj d| }t ||| j | j| jdS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrmrrrrrr)rrr)lossrrr7rr,)rBrrrrrervslicerr loss_functionrrrr7r)r&rrrmrrrrrrrrroutputsr7 slice_indicesrrr,r,r-r:s<'   zCohereForCausalLM.forward) NNNNNNNNNNr)r<r=r>_tied_weights_keys_tp_plan_pp_planr rrr"rrsrrrrvrrrr:r?r,r,r*r-rsZ       r)rrr)r)r)<collections.abcrtypingrr"r activationsr cache_utilsrr generationr integrationsr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_cohererModulerr@ryrsrvrrZrrrrrrrr__all__r,r,r,r-sj            A  W9P ]