o wi%@sdZddlZddlmZmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZddl m!Z!ddl"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)e'rddl*m+Z+m,Z,ddl-m.Z.nd\Z.Z,Z+e&rddl/m0Z0m1Z1nd\Z1Z0e2e.e,e0e1e+fZ3e$4e5Z6Gddde j7Z8dej9de:dej9fddZ;Gdd d eZ< !dBd"e j7d#ej9d$ej9d%ej9d&eej9d'e=d(e=fd)d*Z>Gd+d,d,e j7Z?Gd-d.d.e j7Z@Gd/d0d0e j7ZAGd1d2d2e j7ZBGd3d4d4e j7ZCGd5d6d6e j7ZDe#Gd7d8d8eZEe#Gd9d:d:eEZFGd;d<dGd?d@d@eEZHgdAZIdS)CzPyTorch Zamba model.N)AnyCallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)is_causal_conv1d_availableis_mamba_ssm_available) ZambaConfig)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNcs.eZdZdfdd ZddZddZZS) ZambaRMSNormư>cs&ttt||_||_dS)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__e/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/models/zamba/modeling_zamba.pyr&@s  zZambaRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NT)keepdim) dtypetor(float32powmeanrsqrtr+r*)r, hidden_states input_dtypevariancer1r1r2forwardHs  zZambaRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler*shaper+r,r1r1r2 extra_reprOszZambaRMSNorm.extra_repr)r$)__name__ __module__ __qualname__r&r?rC __classcell__r1r1r/r2r#?sr#r<n_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rAexpandreshape)r<rHbatchnum_key_value_headsslenhead_dimr1r1r2 repeat_kvTs 0rPc @seZdZdZejdfddZ ddejdejdede e e e fd e ejejff d d Zd ejfd dZdde ed efddZd e e eje ejffddZedde e e ejd dfddZdS)ZambaHybridDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. Nc s||_|j|_d|_|j|j|_|j|_|j|_ |j |_ g|_ g|_ g|_ i|_i|_i|_t|jD];}|j tj|j|j |dg7_ |j |j|j |jf}|j tj||dg7_ |j|dkro|j |q4fddt|jD|_fddt|jD|_dS)NFdevicer6hybridc g|] }tjggdqSrSr(tensor.0_ batch_sizerSr1r2  z4ZambaHybridDynamicCache.__init__..crUrVrXrZr]r1r2r_r`)r6layers_block_typehas_previous_state mamba_expandr-intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layers_modules _parameters_buffersrangenum_hidden_layersr(zerosappend key_cache value_cache)r,configr^r6rSi cache_shaper1r]r2r&ns8   z ZambaHybridDynamicCache.__init__ key_states value_states layer_idx cache_kwargsrIcCsz|j|jddkr||j|<||j|<ntj|j||gdd|j|<tj|j||gdd|j|<|j||j|fS)Nr4rr3dim)rtrArur(cat)r,ryrzr{r|r1r1r2updates   zZambaHybridDynamicCache.updatebeam_idxcCstt|jD]V}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<qdS)zDReorders the cache for beam search, given the selected beam indices.rN) rplenrtrS index_selectr7rurjrk)r,rr{rSr1r1r2 reorder_caches     z%ZambaHybridDynamicCache.reorder_cachercCs:||jvr |jdn|}t|j|krdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.r)rlrrtrA)r,r{r1r1r2get_seq_lengthsz&ZambaHybridDynamicCache.get_seq_lengthcCtdNz@ZambaHybridDynamicCache does not have a legacy cache equivalent.NotImplementedErrorrBr1r1r2to_legacy_cachez'ZambaHybridDynamicCache.to_legacy_cachepast_key_valuesr cCrrr)clsrr1r1r2from_legacy_cachesz)ZambaHybridDynamicCache.from_legacy_cacheN)r)rDrErF__doc__r(float16r&Tensorintrdictstrrr@r LongTensorrrr classmethod FloatTensorrr1r1r1r2rQ`s( % "(rQmodulequerykeyvalueattention_maskscalingdropoutc Kst||j}t||j} t||dd|} |dur3|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr3r rr4)r~r6)ptrainingr)rPnum_key_value_groupsr(matmul transposerAr functionalsoftmaxr8r7r6rr contiguous) rrrrrrrkwargsryrz attn_weights causal_mask attn_outputr1r1r2eager_attention_forwards & rcs|eZdZdZdedeffdd Z ddejdede ejd e e d e e d e eje eje e ejff d d ZZS)ZambaAttentionaA Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer and "Generating Long Sequences with Sparse Transformers". Adapted from transformers.models.mistral.modeling_mistral.MistralAttention: The input dimension here is attention_hidden_size = 2 * hidden_size, and head_dim = attention_hidden_size // num_heads. The extra factor of 2 comes from the input being the concatenation of original_hidden_states with the output of the previous (mamba) layer (see fig. 2 in https://huggingface.co/papers/2405.16712). Additionally, replaced attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim) with attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim/2) rvr{cst||_||_|j|_|j|_|j|j|_ |j |_ |jdd|_ d|_ |j |_ tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_dS)Nr3TFbias)r%r&rvr{attention_hidden_sizeattention_head_dimrOnum_attention_headsrMrmax_position_embeddingsr is_causalattention_dropoutrLinearq_projk_projv_projr-o_projr,rvr{r/r1r2r&s  zZambaAttention.__init__Nr<rpast_key_valuerrIcKs|jdd}g|d|jR}|||dd}|||dd} |||dd} |durB|| | |\} } t} |j j dkrPt |j j } | ||| | |f|j s\dn|j |jd|\} } | jg|dR} || } | | fS)Nr4rr3eagerr)rr)rArOrviewrrrrrrv_attn_implementationrrrrrKrr)r,r<r{rrr input_shape hidden_shape query_statesryrzattention_interfacerrr1r1r2r?s2    zZambaAttention.forwardr)rDrErFrrrr&r(rrrQrrr@r?rGr1r1r/r2rs" rcs^eZdZdZdeffdd Z ddejdefdd Z ddefd d Z ddefd d Z Z S)ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rvcst||_||_|j|_|j|_|j|_|j |j|_ |j |_ |j |_ |j |j |_|j|_|j|_tj|j |j |j|j|j |jdd|_|j|_t|j|_|j|_tj|j|j d|jd|_tt |j |j |jd|j|_!tt |j |j|j dd|j d|_"tt |j |j|_#tj$d|jdtj%ddddf}|&|j d'}tt(|)|j |jd|_*tt+|j |j|_,tj|j |j|jd|_-t.st/0ddSdS) Nr) in_channels out_channelsr kernel_sizegroupspaddingr3rg?r6r4apThe fast path is not available because on of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)1r%r&rvr{r-rerfrgrhrcrd mamba_dt_ranktime_step_rankrimamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1dhidden_mamba_act activationr actuse_mamba_kernelsuse_fast_kernelsrin_projr'r(rr x_proj_weightdt_proj_weight dt_proj_biasaranger8rJrlogrKA_logr)Dout_projis_fast_path_availablelogger warning_once)r,rvr{Ar/r1r2r&+sb  $ zZambaMambaMixer.__init__Nr< cache_paramsc Cs|j\}}}|duo|jo|dk}||dd}||dd|jddd\}} |d}| d} | ||j d|dd} |j j |j j d|j j d} |rnt |d|j|j| |j j|j}|d}nK|durt|dks||d}|durtj||j|jddf} |j|j| t|| |j j|jd}|durt|dks||d}|d|j |j|dd}|jdddddddf|dd} tj| |j|j|jgdd\} }}|j dddf| dd}t!|j"# }|j$dur|j$#nd}tj%|d|f|j&|j'd}|rtt(|j D]K}t)|j*|jdd|f||d df||d df||||dddf||dddf|j+|| |d df||d d d}tj,||fdd}q'nntj%|d|j|jf|j&|j'd}t(|j D]E}t-||||||||dd||dd|j+|#| |||d d d \}}tj,||fdd}tj,||dfdd}q|dur|dur|j*|j||.|dd}|S) Nrr3r4r}r)rrrR.T) dt_softplus)delta_softplusreturn_last_state)/rArbrrrchunksqueezerrKrirr*sizer!rjr{rr unsqueezer(allrrpadrhcopy_r rrsplitrrfrexprfloatremptyrSr6rprrkrrrr)r,r<rrr^seq_lenr\use_precomputed_statesprojected_statesgate conv_weightsrjssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statesr1r1r2cuda_kernels_forwardhs  $  *         z$ZambaMambaMixer.cuda_kernels_forwardc Cs|j\}}}|j}||dd}||dd|jddd\} } | d} | d} | ||j d|dd} t |t } | r+|j |j jd|kr+|jrZ|j |j } n|j |j } | | j} |jr|dkr|j|j jd|kr|j|j } tj| ddd} | dddddf| dddddf<| |j|j <tj| |jjdddddfdd} |jr| |jj7} || |d} n|durt|dks| |dd| jd dfd} tj | |j!| jddf} | |j|j <||| dd|f} |dur*t|dks*| |dd| jd dfd} nFtj"||j |j#|j$f| j|d} |durOt|dksO| |d} ||| dd|f} |durqt|dksq| |d} | d|j |j#|dd} |j%dddddddf| dd }tj&||j'|j$|j$gdd\}}}|j(dddf|dd |j)ddddddf}tj*|}t+|j,- }t+|ddddddddf|dddddddddf}|dddddddddf|dddddddddf-}|| dddddddddf-}g}t.|D]\}|dddddd|ddfdd| |dddddd|ddfdd} t/| dd||dddd|ddfd}|0|dddddddfq2tj1|dd}|| |j2ddddddf}||| }| r| |j |j <|3|dd|d|dd}|S) Nrr3r4r}r)shiftsdims.rRr)4rAr6rrrrrrrKri isinstancerQrkr{rcloner7rSrbrjr(rollsumrr*rrrrrrrrrhrrrrfrrrrrsoftplusrrrrprrsstackrr)r, input_statesrrr^rr\r6rr<r use_cacher conv_staterrrrrr discrete_A discrete_BdeltaB_urrw scan_outputrr1r1r2 slow_forwards      ( ( & &*  FH* X8&" zZambaMambaMixer.slow_forwardcCs@|jrtr d|jjjvrtd|j|||dS|j|||dS)NcudazFast Mamba kernels are not available. Make sure to they are installed and that the mamba module is on a CUDA device. lease run 'pip install causal-conv1d>=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r)rrrrStype ValueErrorr r)r,r<rrr1r1r2r?&szZambaMambaMixer.forwardr") rDrErFrrr&r(rrQr rr?rGr1r1r/r2rs > a]rcs$eZdZfddZddZZS)ZambaMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) r%r&rvr-rdrr gate_projup_proj down_projr hidden_actact_fnr,rvr/r1r2r&4s zZambaMLP.__init__cCs$||||||}|Sr)r!r#rr )r,xr!r1r1r2r?>s zZambaMLP.forward)rDrErFr&r?rGr1r1r/r2r3s  rcseZdZddedeeffdd Z    ddejdejded eejd ee d ee d ee d e e de ejee ejejfffddZZS)ZambaAttentionDecoderLayerNrvr{csHtt|||_t||_t|j|jd|_ t|j |jd|_ dS)Nr.) r%r&r self_attnr feed_forwardr#r rms_norm_epsinput_layernormr-pre_ff_layernormrr/r1r2r&Ds   z#ZambaAttentionDecoderLayer.__init__Fr<original_hidden_statesrroutput_attentionsrrrIc Ksjtj||gdd}||}|jd||||||d|\}} ||}||}|f} |r3| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r4r})r<r{rrr.rNr1)r( concatenater+r(r,r)) r,r<r-r{rrr.rrself_attn_weightsoutputsr1r1r2r?Ls$   z"ZambaAttentionDecoderLayer.forwardrNNFF)rDrErFrrrr&r(rrQboolrrr@rr?rGr1r1r/r2r&Cs2   r&cseZdZdedeffdd Z         ddejdeejdeed eejd eejd ee d ee d ee deej deejde ej ee ej ej fffddZZS)ZambaMambaDecoderLayerrvr{cs4tt||d|_t|j|jd|_||_dS)N)rvr{r') r%r&rmambar#r-r*r+r{rr/r1r2r&s  zZambaMambaDecoderLayer.__init__NFr<r-rrrr.rcache_positiontransformer_hidden_statesrIc Ksd|} | dur || n|}||}|j|||d}d} | |}|f}|r)|| f7}|r0||f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)r<rr)r+r5)r,r<r-r{rrrr.rr6r7rresidualr0r1r1r1r2r?s"   zZambaMambaDecoderLayer.forward) NNNNNFFNN)rDrErFrrr&r(rrrQr3rr@rr?rGr1r1r/r2r4sD     r4cseZdZdedejdeffdd Z        ddej d e ej d e e d e ej d e ej d e e de e de e de ejdeeje eejejfffddZZS)ZambaHybridLayer shared_transflinearr5cs t||_||_||_dSr)r%r&r:r; mamba_decoder)r,r:r;r5r/r1r2r&s  zZambaHybridLayer.__init__NFr<r-r{rrrr.rr6rIc Csp|j|||||||| d} | d} |r| d} || } |j|| ||||| d} |r6| d| f| dd} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )r-r{rrr.rr6rr)r7rrr.rr6r3N)r:r;r<) r,r<r-r{rrrr.rr6 layer_outputsr7r0r1r1r2r?s4   zZambaHybridLayer.forward)NNNNNFFN)rDrErFr&rrr4r&r(rrrrQr3rr@rr?rGr1r1r/r2r9s>    r9c seZdZeZdZdZddgZdZdZ dZ dZ dZ ddZ ee  dd eejd eeeeeeffd ed effdd ZZS)ZambaPreTrainedModelmodelTr&r4rFcCs|jj}t|tjtjfr%|jjjd|d|j dur#|j j dSdSt|tj rF|jjjd|d|j durD|jj|j  dSdSt|t rT|jjddSt|tr|jjjd|d|jjd}tj|j| ||jj|jj|jj}tt|jj|t|jjt|jjt|jjj|jj d}|tt!|  }|j"j#|tj$d|j%dtj&ddddf}|'|j(d)}|j*j#t|+|j|j,d|j-jddSdS) Nr)r:stdg?r)minrrr4).rvinitializer_ranger rrrr*datanormal_rzero_ Embedding padding_idxr#fill_rrrinituniform_rrcr-rir(rrandmathr time_step_max time_step_minclamptime_step_floorexpm1rrrrfr8rJrdrrrKrr)r,rr@ dt_init_stdrdtinv_dtrr1r1r2 _init_weightssD       $"z"ZambaPreTrainedModel._init_weightsN torch_dtype device_maphard_check_onlycheck_device_mapcs.tj|||||d}|s|jdkrd|_|S)z Overloads `PreTrainedModel._check_and_enable_flash_attn_2` so as to DISABLE Flash Attention 2 by default on Zamba models. Flash attention 2 is currently not supported in the HuggingFace implementation of Zamba v1. )rXrYflash_attention_2r)r%_check_and_enable_flash_attn_2r)rrvrVrWrXrYr/r1r2r[:s  z3ZambaPreTrainedModel._check_and_enable_flash_attn_2r2)rDrErFr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_class _is_statefulrUrrr(r6rrrrr3r[rGr1r1r/r2r>s4 r>cseZdZdZdeffdd ZddZddZe dd e e j d e e j d e e j d e e de e jde ede ede ede ede e j deeeffddZddZZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rvc st||j|_|j|_t|j|j|j|_t |}g}g}|j |_ t |j D]2}|j |dkr=| t||dq*|j |dkr\| tj|jj|jjdd| t||dq*t|}t|}g}g|_t|j D]6\}}|dkrd|dgd} g|jfd d | D|_| t|t|t|qo| t|qot||_|j|_t|j|jd |_d|_|dS) Nr5)r{rTFrzlayers..) z%shared_transf.self_attn.q_proj.weightz%shared_transf.self_attn.k_proj.weightz%shared_transf.self_attn.v_proj.weightz%shared_transf.self_attn.o_proj.weightz+shared_transf.feed_forward.gate_proj.weightz)shared_transf.feed_forward.up_proj.weightz+shared_transf.feed_forward.down_proj.weightz$shared_transf.input_layernorm.weightz%shared_transf.pre_ff_layernorm.weightcsg|]}|qSr1r1)r[r prefix_namer1r2r_~sz'ZambaModel.__init__..r')r%r& pad_token_idrG vocab_sizerrFr- embed_tokensr&rarprqrsr4rrviter_tied_weights_keys enumerater9next ModuleListlayersrr#r*final_layernormgradient_checkpointing post_init) r,rvblock mamba_layers linear_layersrwrqlayer_id layer_type tied_keysr/rgr2r&\s>     zZambaModel.__init__cC|jSrrkrBr1r1r2get_input_embeddingszZambaModel.get_input_embeddingscC ||_dSrr|r,rr1r1r2set_input_embeddings zZambaModel.set_input_embeddingsN input_idsr position_idsr inputs_embedsrr.output_hidden_states return_dictr6rIc  Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |du|duAr4td|jrC|jrC|rCt dd}|durL| |}|} t |} |r^|dur^t d| durmt j | jd| jd} |durv| d}|||| } |rdnd}|rdnd}t|jD]A\}}|r|| f7}|jr|jr||j| | ||| |||| }n || | ||| |||| d }|d} |r|ddur||df7}q|| } |r|| f7}|r|jsd |_t| |r|nd||d }| r|S|S) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.rrWrr1)r-r{rrrr.rr6T)last_hidden_staterr< attentions)rvr.rruse_return_dictrrsrrrrkr(r rrArSr_update_causal_maskrnrq_gradient_checkpointing_func__call__rrrbrto_tuple)r,rrrrrrr.rrr6r<r-rall_hidden_statesall_self_attnsr{layerr=outputr1r1r2r?s               zZambaModel.forwardc Csv|jjdkr|durd|vr|SdS|j|j}}t|j}|jd}|dd}tj||f|||d} |dkr@tj | dd} | tj ||d| ddk9} | ddddddf |jdddd} |dur| } |d kr|jd} | d d| fd|ddddddfd} | d d| f| || d d| f<|jjd kr|dur|jjd vrt| |} | S) NrZrrr4) fill_valuer6rS)diagonalrWrr3.sdpa)rxpunpu)rvrr6rSr(finforArAfulltriurrKrJr r~eq masked_fillrr_unmask_unattended) r,r input_tensorr6r6rS min_dtypesequence_length target_lengthr mask_length padding_maskr1r1r2rs0    *  4$   zZambaModel._update_causal_mask NNNNNNNNNN)rDrErFrrr&r}rrrr(rrrQrr3rr@rr?rrGr1r1r/r2reSsP/    nrecseZdZdeffdd ZddZddZdd Zd d Zd d Z ddZ e            d$de e jde e jde e jde ede e jde e jde ede ede ede ede e jdeee jfdeeeffdd Z      !d%d"d#ZZS)&ZambaForCausalLMrvcsPt|t||_dg|jj|_|j|_tj|j|jdd|_ | dS)Nzlm_head.weightFr) r%r&rer?rmrjrrr-lm_headrtr$r/r1r2r&&s   zZambaForCausalLM.__init__cC|jjSrr?rkrBr1r1r2r}0rz%ZambaForCausalLM.get_input_embeddingscC ||j_dSrrrr1r1r2r3 z%ZambaForCausalLM.set_input_embeddingscCr{rrrBr1r1r2get_output_embeddings6r~z&ZambaForCausalLM.get_output_embeddingscCrrr)r,new_embeddingsr1r1r2set_output_embeddings9rz&ZambaForCausalLM.set_output_embeddingscCrrr?)r,decoderr1r1r2 set_decoder<rzZambaForCausalLM.set_decodercCr{rrrBr1r1r2 get_decoder?r~zZambaForCausalLM.get_decoderNrrrrrrlabelsrr.rrr6logits_to_keeprIc  Ks|dur|n|jj}| dur| n|jj} | dur| n|jj} |j|||||||| | | d }|d}t| tr>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrrrrr.rr6rrrlosslogitsrr<r)rvr.rrr?r rslicer loss_functionrjrrr<r)r,rrrrrrrr.rrr6r loss_kwargsr1r< slice_indicesrrrr1r1r2r?Bs@( zZambaForCausalLM.forwardTc Ks|du} | s5|dus|d|jdkr"|dd|jd df}n!|jd|jdkr4|dd|f}nt|j|jd|j|jd}|durl|durl|dd}||dkd| sl|dd|jd df}|durw| rwd|i} nd|i} | |||||jj |d| S)Nr4rr)r6rSrr)rrrrrr6) rArQrvr6rSlongcumsum masked_fill_rrnum_logits_to_keep) r,rrrrr6rrr empty_past_kv model_inputsr1r1r2prepare_inputs_for_generations:     z.ZambaForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNNr)NNNNNT)rDrErFrr&r}rrrrrrrr(rrrQrr3rrr@rr?rrGr1r1r/r2r%sn        Tra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcseZdZfddZddZddZe          ddeej d eej d eej d ee e e ejfd eejd eej deedeedeedeede eeffddZZS)ZambaForSequenceClassificationcsJt||j|_t||_|jj|_tj|j|jdd|_ | dSr) r%r& num_labelsrer?rmrrr-scorertr$r/r1r2r&s    z'ZambaForSequenceClassification.__init__cCrrrrBr1r1r2r}rz3ZambaForSequenceClassification.get_input_embeddingscCrrrrr1r1r2rrz3ZambaForSequenceClassification.set_input_embeddingsNrrrrrrrr.rrrIc  CsB| dur| n|jj} |j|||||||| | d } | d} || } |dur+|jd}n|jd}|jjdur>|dkr>td|jjdurGd}n1|durl||jjk| jt j }t j |jd| jt j d}|| d}n d}t |jjd| t j || jd |f}d}|dur|| j}|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrrr.rrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r4rRz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rW regressionsingle_label_classificationmulti_label_classificationr)rvrr?rrArirr7rSr(int32rargmaxrrr0rD problem_typerr6rrr rrrrrrr<r)r,rrrrrrrr.rrtransformer_outputsr<rr^last_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrr1r1r2r?sx         "       z&ZambaForSequenceClassification.forwardr)rDrErFr&r}rrrr(rrrr listrr3r@rr?rGr1r1r/r2rsL      r)rrrer>)r)JrrLtypingrrrrr(torch.utils.checkpointrtorch.nnrrr activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrutils.import_utilsrrconfiguration_zambar&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1dr r!rr get_loggerrDrModuler#rrrPrQrrrrrr&r4r9r>rerr__all__r1r1r1r2s           e F?DHDR, p