o wij@sddlmZmZmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZmZddlmZddlmZm Z m!Z!m"Z"ddl#m$Z$e"%e&Z'Gdddej(Z)Gdddej(Z*Gdddej(Z+dej,de-dej,fddZ. d8dej(dej,d ej,d!ej,d"eej,d#e/d$e/fd%d&Z0d'd(Z1d9d)d*Z2Gd+d,d,ej(Z3Gd-d.d.eZ4e Gd/d0d0eZ5e Gd1d2d2e5Z6Gd3d4d4eeZ7e Gd5d6d6e5e Z8gd7Z9dS):)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsauto_docstringcan_return_tuplelogging) CohereConfigcs&eZdZdfdd ZddZZS) CohereLayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeepsbias __class__g/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/cohere/modeling_cohere.pyr5s  zCohereLayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NT)keepdim) dtypetor float32meanpowrsqrtr#r")r$ hidden_states input_dtyper2variancer*r*r+forward;s  zCohereLayerNorm.forward)NrF__name__ __module__ __qualname__rr8 __classcell__r*r*r(r+r4srcs8eZdZddeffdd ZeeddZZ S)CohereRotaryEmbeddingNconfigcstt|dr|jdur|jd|jd|_nd|_|j|_|j|_||_ t |j|_ | |j |\}|_ |j d|dd|j|_dS)N rope_scaling rope_typetypedefaultinv_freqF) persistent)rrhasattrr@getrAmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr?r rope_init_fnattention_scalingregister_bufferrDoriginal_inv_freq)r$r?devicerDr(r*r+rFs   zCohereRotaryEmbedding.__init__c Cs|jddddf|jddd}|dddddf}t|jjtr2|jjdkr2|jjnd}tj |dd*|| dd}tj |ddd }| |j }||j } Wdn1shwY|j|jd | j|jd fS) Nrr,rmpscpuF) device_typeenabledr.dimr/)rDfloatexpandshape isinstancerOrBstrr autocast transposerepeat_interleavecosrLsinr0r/) r$x position_idsinv_freq_expandedposition_ids_expandedrRfreqsembr_r`r*r*r+r8Ws(&zCohereRotaryEmbedding.forwardN) r:r;r<rrr no_gradrr8r=r*r*r(r+r>Es r>cs$eZdZfddZddZZS) CohereMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr') rrr?r%intermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnr$r?r(r*r+rhs zCohereMLP.__init__cCs$||||||}|Srg)rprrrnro)r$rarpr*r*r+r8rs zCohereMLP.forwardr9r*r*r(r+rigs  rir5n_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rYrXreshape)r5rtbatchnum_key_value_headsslenhead_dimr*r*r+ repeat_kvws 0r{modulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr.rr,)rUr/)ptrainingr)r{num_key_value_groupsr matmulr]rYr functionalsoftmaxr1r0r/rr contiguous) r}r~rrrrrkwargs key_states value_states attn_weights causal_mask attn_outputr*r*r+eager_attention_forwards & rcCsB|ddddf}|ddddf}tj| |gddd}|S)N.r.rr,rTr)r stackflatten)rax1x2rot_xr*r*r+ rotate_halfsrc Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. rV)r/rW unsqueezerr0) qkr_r`rb unsqueeze_dimr/q_embedk_embedr*r*r+apply_rotary_pos_embs  rcseZdZdZddedeeffdd Z  ddej de ej ej fd eej d ee d eej d e ed e ej eej ee ej ffddZZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr? layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_|j|_|jrt|j|jf|jd|_t|j|jf|jd|_dSdS)NrzgTrkr%r&)rrr?rgetattrr%num_attention_headsrzrxrrattention_dropout is_causalrrmattention_biasq_projk_projv_projo_proj use_qk_normrlayer_norm_epsq_normk_normr$r?rr(r*r+rs:   zCohereAttention.__init__r5position_embeddingsrpast_key_valuecache_positionrrucKsJ|jdd}g|d|jR}|||} |||} |||} |jr6|| } || } | dd} | dd} | dd} |\} } t | | | | \} } |durj| | |d}| | | |j |\} } t }|jjdkrxt|jj}||| | | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)Nr,rr.)r`r_reagerr|)rr)rYrzrviewrrrrrr]rupdaterrr?_attn_implementationrrrrrvrr)r$r5rrrrr input_shape hidden_shape query_statesrrr_r` cache_kwargsattention_interfacerrr*r*r+r8sD           zCohereAttention.forwardrg)NN)r:r;r<__doc__rrintrr Tensortupler LongTensorrr r8r=r*r*r(r+rs(%rcseZdZdedeffdd Z       ddejdeejd eej d ee d ee d ee d eej dee ejejfde ede ejee ejejfffddZZS)CohereDecoderLayerr?rcs@t|j|_t||d|_t||_t|j|jd|_ dS)N)r?rr) rrr%r self_attnrimlprrinput_layernormrr(r*r+rs  zCohereDecoderLayer.__init__NFr5rrbroutput_attentions use_cacherrrruc  Ksb|} ||}|jd||||||||d| \} } ||} | | | }|f}|r/|| f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r5rrbrrrrrNr*)rrr)r$r5rrbrrrrrrresidualhidden_states_attentionself_attn_weightshidden_states_mlpoutputsr*r*r+r8"s(   zCohereDecoderLayer.forward)NNNFFNN)r:r;r<rrrr rrrrboolrrr FloatTensorr8r=r*r*r(r+rs<    rc@sLeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZdZddZdS)CoherePreTrainedModelmodelTrpast_key_valuescCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|t rQ|jj ddSdS)Nr|)r2stdg?)r?initializer_rangerZrrmr"datanormal_r'zero_ Embedding padding_idxrfill_)r$r}rr*r*r+ _init_weightsos     z#CoherePreTrainedModel._init_weightsN)r:r;r<r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_3_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_class_supports_quantized_cache_supports_static_cache_supports_attention_backendrr*r*r*r+r_s rcseZdZdeffdd ZddZddZee         dd e e j d e e j d e e j d e e d e e jde ede ede ede e j deedefddZZS) CohereModelr?cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr*)r).0rr?r*r+ sz(CohereModel.__init__..rrF)rr pad_token_idr vocab_sizerrr% embed_tokens ModuleListrangenum_hidden_layerslayersrrnormr> rotary_embgradient_checkpointing post_initrsr(rr+rs   zCohereModel.__init__cC|jSrgrr$r*r*r+get_input_embeddingsz CohereModel.get_input_embeddingscC ||_dSrgrr$rr*r*r+set_input_embeddings z CohereModel.set_input_embeddingsN input_idsrrbr inputs_embedsrroutput_hidden_statesrflash_attn_kwargsruc  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|du|duAr*td|jr9|jr9|r9tdd}t |t dt fsFtd|durO| |}|rX|durXt }| durt|durd|nd} tj| | |jd|jd} |dur}| d}t|j||| ||d} |} || |}|rd nd}|rd nd}|jd|jjD]&}|r|| f7}|| f| ||||| |d | }|d} |r||df7}q|| } |r|| f7}t| |r|nd||d S) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzBThe `past_key_values` should be either a `Cache` object or `None`.rr)rO)r? input_embedsrrrrbr*)rrbrrrrr)last_hidden_staterr5 attentions)r?rrr ValueErrorrrlogger warning_oncerZrBrrr get_seq_lengthr arangerYrOrr rrrrr)r$rrrbrrrrrrrpast_seen_tokensrr5rall_hidden_statesall_self_attns decoder_layer layer_outputsr*r*r+r8s            zCohereModel.forward) NNNNNNNNN)r:r;r<rrrrrrrr rrrrrrr rr8r=r*r*r(r+r}sL    rc@s eZdZdS)KwargsForCausalLMN)r:r;r<r*r*r*r+rs rcseZdZdgZddiZddgdgfiZfddZdd Zd d Zd d Z ddZ ddZ ddZ e e           d%deejdeejdeejdeeeeejfdeejdeejdeedeedeedeejd eeejfd!eed"efd#d$ZZS)&CohereForCausalLMzlm_head.weightlm_head colwise_repr5logitscsPt|t||_|j|_tj|j|jdd|_|j |_ |j |_ | dSrj) rrrrrrrmr%r logit_scaletie_word_embeddingsrrsr(r*r+rs   zCohereForCausalLM.__init__cCs|jjSrgrrrr*r*r+rsz&CohereForCausalLM.get_input_embeddingscCs ||j_dSrgrrr*r*r+rs z&CohereForCausalLM.set_input_embeddingscCrrgrrr*r*r+get_output_embeddingsrz'CohereForCausalLM.get_output_embeddingscCrrgr)r$new_embeddingsr*r*r+set_output_embeddingsrz'CohereForCausalLM.set_output_embeddingscCrrgr)r$decoderr*r*r+ set_decoderrzCohereForCausalLM.set_decodercCrrgrrr*r*r+ get_decoderrzCohereForCausalLM.get_decoderNrrrrbrrlabelsrrrrlogits_to_keeprruc  Ks|dur|n|jj}| dur| n|jj} |jd|||||||| | d | } | j}t| tr4t| dn| }||dd|ddf}||j }d}|dur]|j d|||jj d| }t ||| j | j| jdS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrbrrrrrr)rr!r)lossrrr5rr*)r?rrrrrZrslicerr loss_functionrrrr5r)r$rrrbrrr!rrrrr"rrr5 slice_indicesrr#r*r*r+r8!s<'   zCohereForCausalLM.forward) NNNNNNNNNNr)r:r;r<_tied_weights_keys_tp_plan_pp_planrrrrrrr rrrr rrrrlistrrrrrrr8r=r*r*r(r+rsf       r)rrr)r|)Nr):typingrrrr r activationsr cache_utilsrr generationr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrconfiguration_coherer get_loggerr:rModulerr>rirrr{rWrrrrrrrrr__all__r*r*r*r+s`         "  WE} o