o ei@sddlmZddlmZmZddlZddlmmZ ddlmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZmZddlm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.e)rddl/m0Z0m1Z1nd\Z0Z1edGdddej2Z3Gdddej2Z4Gdddej2Z5GdddZ6d d!Z7ed"dDd#d$Z8d%ej9d&e:d'ej9fd(d)Z; *dEd+ej2d,ej9d-ej9d.ej9d/ej9dBd0ed7d8Z?e0e1fZ@eAe@ZBGd9d:d:ej2ZCGd;d<dd>eZEe#Gd?d@d@eEZFe#GdAdBdBeEe ZGgdCZHdS)F)Callable)AnyOptionalN)nn)Cache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)is_causal_conv1d_availableis_torchdynamo_compiling)capture_outputs) Lfm2Config)causal_conv1d_fncausal_conv1d_updateNNRMSNormcsFeZdZd deddffdd Zdejdejfdd Zd d ZZ S) Lfm2RMSNormư>epsreturnNcs&ttt||_||_dS)z: Lfm2RMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer% __class__d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/lfm2/modeling_lfm2.pyr(3s  zLfm2RMSNorm.__init__ hidden_statescCsJ|j}|tj}|djddd}|t||j}|j||S)NT)keepdim) dtypetor*float32powmeanrsqrtr-r,)r.r4 input_dtypevariancer2r2r3forward;s  zLfm2RMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler,shaper-r.r2r2r3 extra_reprBszLfm2RMSNorm.extra_repr)r$) __name__ __module__ __qualname__floatr(r*Tensorr@rD __classcell__r2r2r0r3r#1sr#c s~eZdZUejed<ddeffdd Ze   ddedBde dde dBd e d e ffd d Z eed dZZS)Lfm2RotaryEmbeddinginv_freqNconfigcst|j|_|j|_||_|jjd|_|j}|jdkr$t |j}||j|\}|_ |j d|dd|j d| dddS)N rope_typedefaultrLF) persistentoriginal_inv_freq) r'r(max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrMrope_parametersrNcompute_default_rope_parametersrattention_scalingregister_bufferclone)r.rMdevice rope_init_fnrLr0r2r3r(Is   zLfm2RotaryEmbedding.__init__rZz torch.deviceseq_lenr&z torch.TensorcCsZ|jd}t|ddp|j|j}d}d|tjd|dtjdj|tjd|}||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr5r8rZr8) rUgetattrr/num_attention_headsr*arangeint64r9rH)rMrZr\basedimattention_factorrLr2r2r3rVYs &z3Lfm2RotaryEmbedding.compute_default_rope_parametersc Cs|jddddf|jddd|j}|dddddf}t|jjtr6|jjdkr6|jjnd}t |dd+|| dd}t j ||fdd }| |j}||j} Wdn1slwY|j|jd | j|jd fS) Nrr6rmpscpuF) device_typeenabledr5rfr_)rLrHexpandrBr9rZ isinstancetypestrr transposer*catcosrWsinr8) r.x position_idsinv_freq_expandedposition_ids_expandedrjfreqsembrsrtr2r2r3r@ws0&zLfm2RotaryEmbedding.forwardNNNN)rErFrGr*rI__annotations__rr( staticmethodrintrArHrVno_gradrr@rJr2r2r0r3rKFs&   rKcs*eZdZdeffdd ZddZZS)Lfm2MLPrMcst|j}|jr,td|d}|jdur,t|j|}|j||jd|j}tj|j |dd|_ tj|j |dd|_ tj||j dd|_ dS)Nr5rrFbias) r'r(intermediate_sizeblock_auto_adjust_ff_dimrblock_ffn_dim_multiplierblock_multiple_ofrLinearr/w1w3w2)r.rMrr0r2r3r(s  zLfm2MLP.__init__cCs |t||||Sr{)rFsilurr)r.rur2r2r3r@s zLfm2MLP.forward)rErFrGrr(r@rJr2r2r0r3rsrc @seZdZdZdZdZdZdZej dfde de dej dej eBdBfdd Z d!d ejd ejd e d eeefdBdeejejff ddZdejfddZd"d e dBde fddZdejd e dee e ffddZde fddZde fddZdd ZdS)#Lfm2HybridConvCachea Attention and conv cache for Lfm2. It stores the Key and Value states as a list of tensors, one for each layer. Attention layer cache shape: `[batch_size, num_heads, seq_len, head_dim]`. Conv layer cache shape: `[batch_size, hidden_size, L_cache-1]`. NFrMmax_batch_sizer8rZcCsg|_g|_||_|j|_|jd|_|j|_||_g|_|dur't |nd}t |j D](}t j |j|j|j|j|d}|j||jt g|jt gq.dS)Nfull_attention)r8rZ) key_cache value_cacher layer_typesindexfirst_attention_layer conv_L_cache_dtype conv_cacher*rZrangenum_hidden_layerszerosr/appendtensor)r.rMrr8rZ_ conv_stater2r2r3r(s* zLfm2HybridConvCache.__init__ key_states value_states layer_idx cache_kwargsr&cCsx|j|dkr||j|<||j|<ntj|j||gdd|j|<tj|j||gdd|j|<|j||j|fS)a Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. No additional arguments are used in `DynamicCache`. Return: A tuple containing the updated key and value states. rrl)rnumelrr*rr)r.rrrrr2r2r3updates   zLfm2HybridConvCache.updatebeam_idxcCstt|jD]O}|j|r:|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|rV|j|j}|j|d|||j|<qdS)zDReorders the cache for beam search, given the selected beam indices.rN) rlenrrrZ index_selectr9rr)r.rrrZr2r2r3 reorder_caches   z!Lfm2HybridConvCache.reorder_cachercCsL|j|dkr |jn|}t|j|ks|j|dkrdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rrr)rrrrrrBr.rr2r2r3get_seq_lengths z"Lfm2HybridConvCache.get_seq_lengthcache_positioncCs&d}|jd}|}||}||fS)aB Return a tuple (kv_length, kv_offset) corresponding to the length and offset that will be returned for the given layer at `layer_idx`. The masks are then prepared according to the given lengths (kv_length, kv_offset) and patterns (i.e. sliding_window, chunk_size), for each layer. r)rBr)r.rrfull_mask_kv_offset query_lengthpast_seen_tokens kv_lengthr2r2r3get_mask_sizess  z"Lfm2HybridConvCache.get_mask_sizes max_lengthcCs|dkr |t|}||krdStt|jD]+}|j|rF|j|dd|ddf|j|<|j|dd|ddf|j|<qdS)z"Crop the cache to the given lengthrN.)rabsrrrrr)r.ridxr2r2r3crop s ""zLfm2HybridConvCache.cropcCs t|jSr{)rrrCr2r2r3__len__s zLfm2HybridConvCache.__len__cCs&tt|jD] }|j|qdSr{)rrrzero_rr2r2r3resetszLfm2HybridConvCache.resetr{)r)rErFrG__doc__ris_compileablerrr*r:rrr8rZrpr(rIdictrrAr LongTensorrrrrrrr2r2r2r3rsF   # !   rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr6r5rl)rBr*rr)rux1x2r2r2r3 rotate_half!srrotary_pos_embcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrsrt unsqueeze_dimq_embedk_embedr2r2r3apply_rotary_pos_emb(s  rr4n_repr&cCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rBrmreshape)r4rbatchnum_key_value_headsslenr^r2r2r3 repeat_kvBs 0rmodulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |dur | |} tjj| dtjd |j } tjj | ||j d} t| | } | dd } | | fS)Nr5rr6)rfr8)ptrainingr)rnum_key_value_groupsr*matmulrqr functionalsoftmaxr:r9r8rr contiguous) rrrrrrrrrr attn_weights attn_outputr2r2r3eager_attention_forwardNs  rcseZdZdZdedeffdd Z  ddejde ejejfd ejdBd e dBd ej dBd e ejejdBff d dZ Z S) Lfm2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperrMrcst||_||_t|d|j|j|_|j|j|_ |jd|_ d|_ t j |j|j|jdd|_t j |j|j|jdd|_t j |j|j|jdd|_t j |j|j|jdd|_t|j|jd|_t|j|jd|_dS)Nr^gTFrr%)r'r(rMrrar/rbr^rrr is_causalrrq_projk_projv_projout_projr#norm_eps q_layernorm k_layernormr.rMrr0r2r3r(ks  zLfm2Attention.__init__Nr4position_embeddingsrpast_key_valuesrr&cKs|jdd}g|d|jR}|||j|dd} |||j|dd} ||j|dd} |\} } t | | | | \} } |dur]| | |d}| | | |j |\} } t |jjt}||| | | |fd|jd|\}}|jg|dR}||}||fS)Nr6rr5)rtrsrr)rr)rBr^rrviewrqrrrrrrr get_interfacerM_attn_implementationrrrrr)r.r4rrrrr input_shape hidden_shape query_statesrrrsrtrattention_interfacerroutputr2r2r3r@zs8    zLfm2Attention.forwardr!)rErFrGrrrr(r*rIrArrr@rJr2r2r0r3rgs$rcCsN|dur%|jddkr%|jddkr%|j}||dddddf|}|S)zm Tunes out the hidden states for padding tokens, see https://github.com/state-spaces/mamba/issues/66 Nrr)rBr8r9)r4rr8r2r2r3apply_mask_to_padding_statess$ rc seZdZdedeffdd Z   ddejdedBdej dBd ejdBfd d Z   ddejdedBdej dBd ejdBfd d Z   ddejdedBdej dBd ejdBfddZ Z S) Lfm2ShortConvrMrcst||_||_|j|_|j|_tj |j |j |j|j |j|jdd|_ tj |j d|j |jd|_ tj |j |j |jd|_dS)Nr) in_channels out_channels kernel_sizegroupsrpaddingrr)r'r(rMrrL_cache conv_biasrrConv1dr/convrin_projrrr0r2r3r(s zLfm2ShortConv.__init__Nrurrrc Cs t||}||dd}|jddd\}}}||}|jj|jjd|jjd} |durO|ddkrOt| d|j |j | |jj d} | d} n&|durktj||j|jddf} |j |j | t|| |jj dd} || } || dd} | S)Nr6rrrlrr5) activation)rrrqchunkrr,rsizer squeezerrrrrrpadrrBcopy_rrr) r.rurrrBCxBCBx conv_weightsconv_outryr2r2r3cuda_kernels_forwards* $  z"Lfm2ShortConv.cuda_kernels_forwardc Csz|jd}t||}||dd}|jddd\}}}||} |dur|ddkr|j|j} |d|jd}| j ddd} | j | j | j d| dddd|f<|j|j | tj| | j |jjdddddfdd} |jr}| |jj7} | d} n'|durtj| |j| jddf} |j|j | || d d|f} || } | dd} || } | S) Nrr6rrrlr)shiftsdimsr`.)rBrrrqrrrclamprrollr9rZr8rr*sumrr,rrrrrrr) r.rurrrseqlenr r r r rrrr2r2r3 slow_forwards.   $0   zLfm2ShortConv.slow_forwardr4cCs6trd|jjvrts|||||S|||||S)Ncuda)is_fast_path_availablerZrorrr)r.r4rrrr2r2r3r@szLfm2ShortConv.forwardr|)rErFrGrrr(r*rIrrrrr@rJr2r2r0r3rsT % )rcseZdZdedeffdd Z     ddejdeejejfdBdejdBd ej dBd e dBd ej dBd ejfd dZ Z S)Lfm2DecoderLayerrMrcslt|j|dk|_|jrt|||_nt|||_t||_ t |j |j d|_ t |j |j d|_dS)Nrr)r'r(ris_attention_layerr self_attnrrr feed_forwardr#r/r operator_normffn_normrr0r2r3r( s   zLfm2DecoderLayer.__init__Nr4rrrvrrr&c Ksl|}|jr|jd|||||||d|\}} n |j|||||d}||}||||}|S)N)r4rrrvrr)r4rrrr2)rrrrrr) r.r4rrrvrrrresidualrr2r2r3r@,s*   zLfm2DecoderLayer.forward)NNNNN) rErFrGrrr(r*rIrArrr@rJr2r2r0r3rs, rc@sHeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS) Lfm2PreTrainedModelrMmodelTrrF)r4 attentionsN)rErFrGrr}base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr2r2r2r3r!Ns  r!cseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd e dBd ej dBd eed efddZZS) Lfm2ModelrMcstj|_j|_tjj|j|_t fddt j D|_ t d|_d|_tjjd|_|dS)Ncsg|]}t|qSr2)r).0rrMr2r3 isz&Lfm2Model.__init__..r0Fr)r'r( pad_token_id padding_idx vocab_sizer Embeddingr/ embed_tokens ModuleListrrlayersrK rotary_embgradient_checkpointingr#rembedding_norm post_initr.rMr0r0r3r(bs   zLfm2Model.__init__N input_idsrrvr inputs_embeds use_cacherrr&c Ks6|du|duAr td|dur||}|r+|dur+|jd} t|j| |j|jd}|durG|dur7|nd} tj | | |jd|jd}|durP| d}t |j|||||d} |jddkrd|nd} |} |j | |d}|j d|jjD]}|jr| n| }|| f|||||d|} qx|| } t| |d S) Nz:You must specify exactly one of input_ids or inputs_embedsr)rMrr8rZr)rZ)rMr?rrrrv)rv)rrrvrr)last_hidden_stater) ValueErrorr6rBrrMr8rZrr*rcrr r9r8rrr;r)r.r>rrvrr?r@rr batch_sizer causal_masklinear_attentionr4r decoder_layer layer_maskr2r2r3r@rsZ      zLfm2Model.forward)NNNNNNN)rErFrGrr(rrrr*rrIr FloatTensorboolrrrr@rJr2r2r0r3r.`s>  r.cseZdZddiZddiZddgdgfiZfddZee dd e j d Bd e j d Bd e j d Bde d Bde j d Bde j d Bded Bde j d Bdee j BdeedefddZZS)Lfm2ForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr4logitscs@t|t||_|j|_tj|j|jdd|_| dS)NFr) r'r(r.r"r4rrr/rKr<r=r0r2r3r(s   zLfm2ForCausalLM.__init__Nrr>rrvrr?labelsr@rlogits_to_keeprr&c  Ks|jd|||||||d| } | j} t| trt| dn| } || dd| ddf}d}|durB|jd|||jjd| }t ||| j | j | j dS)a Example: ```python >>> from transformers import AutoTokenizer, Lfm2ForCausalLM >>> model = Lfm2ForCausalLM.from_pretrained("meta-lfm2/Lfm2-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-lfm2/Lfm2-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)r>rrvrr?r@rN)rMrNr4)lossrMrr4r#r2) r"rArnrslicerK loss_functionrMr4rrr4r#)r.r>rrvrr?rNr@rrOroutputsr4 slice_indicesrMrPr2r2r3r@s0  zLfm2ForCausalLM.forward) NNNNNNNNr)rErFrG_tied_weights_keys_tp_plan_pp_planr(rrr*rrIrrHrIrrrrr@rJr2r2r0r3rJsN      rJ)rJr.r!)r)r)Icollections.abcrtypingrrr*torch.nn.functionalrrr cache_utilsr generationr integrationsr r r masking_utilsr modeling_layersr modeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrrutils.import_utilsrrutils.output_capturingrconfiguration_lfm2r causal_conv1drr Moduler#rKrrrrrIrrrHrrrkernel_modulesallrrrr!r.rJ__all__r2r2r2r3s         A  < k/W K