o ei@sddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZmZdd lmZmZdd lmZmZdd lmZddlmZmZmZddlmZddl m!Z!m"Z"ddl#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-e.e/Z0Gddde%Z1GdddZ2Gddde$Z3Gdddej4Z5Gdd d e(Z6Gd!d"d"e*Z7Gd#d$d$ej4Z8Gd%d&d&eZ9Gd'd(d(eZ:e9e:d)Z;Gd*d+d+eZGd0d1d1ee<Z?gd2Z@dS)3)Callable)AnyN)nn)initialization)ACT2FN)lazy_load_kernel)create_causal_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)merge_with_config_defaults)OutputRecordercapture_outputs)LlamaAttention LlamaRMSNormeager_attention_forward) MistralMLP)MixtralExpertsMixtralForCausalLM) JambaConfigc@ eZdZdS) JambaRMSNormN__name__ __module__ __qualname__r&r&e/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/jamba/modular_jamba.pyr!-r!c @seZdZdZdZejdfddZddZdd Z dd ej d ej d e d e e efdBdeej ej ff ddZdejfddZdej d e dee e ffddZdd e dBde fddZdS) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNc s&||_|j|_d|_|j|j}|j}|j}g|_g|_g|_ t |j D]P}|j|dkrN|jt j |||dg7_|jt j |||dg7_q$|jt jggdg7_|jt jggdg7_|j |q$fddt |j D|_fddt |j D|_dS)NFmambadevicedtyper,c g|] }tjggdqSr.torchtensor.0_ batch_sizer,r&r' X z=HybridMambaAttentionDynamicCache.__init__..cr/r0r1r4r7r&r'r9Yr:)r-layers_block_typehas_previous_state mamba_expand hidden_size mamba_d_state mamba_d_conv conv_states ssm_statestransformer_layersrangenum_hidden_layersr2zerosr3append key_cache value_cache) selfconfigr8r-r,intermediate_sizessm_state_sizeconv_kernel_sizeir&r7r'__init__As,    z)HybridMambaAttentionDynamicCache.__init__cCs t|jSN)lenrH)rJr&r&r'__len__[s z(HybridMambaAttentionDynamicCache.__len__cCs|j||j|fSrQ)rHrIrJ layer_idxr&r&r' __getitem__^sz,HybridMambaAttentionDynamicCache.__getitem__ key_states value_statesrU cache_kwargsreturncCsz|j|jddkr||j|<||j|<ntj|j||gdd|j|<tj|j||gdd|j|<|j||j|fS)Nrrdim)rHshaperIr2cat)rJrWrXrUrYr&r&r'updateas   z'HybridMambaAttentionDynamicCache.updatebeam_idxcCs|dkrdtt|jD]X}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j |j}|j |d|||j |<q dSdS)zDReorders the cache for beam search, given the selected beam indices.rN) get_seq_lengthrDrRrHr, index_selecttorIrArB)rJrarUr,r&r&r' reorder_cachers      z.HybridMambaAttentionDynamicCache.reorder_cachecache_positioncCs$d}|jd}|||}||fS)zDReturn the length and offset of the cache, used to generate the maskr)r^rb)rJrfrU kv_offset query_length kv_lengthr&r&r'get_mask_sizess z/HybridMambaAttentionDynamicCache.get_mask_sizesrcCsN||jvr |jdn|}t|j|ks|j|jddkrdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rr[)rCrRrHr^rTr&r&r'rbs"z/HybridMambaAttentionDynamicCache.get_seq_lengthrQ)r)r#r$r%__doc__is_compileabler2float16rPrSrVTensorintdictstrrtupler` LongTensorrerjrbr&r&r&r'r)1s*   r)csveZdZdedeffdd Z   ddejdejdBdedBd ej dBd e e d e ejejdBff d d Z ZS)JambaAttentionrKrUcst||tj|j|j|jdd|_tj|j|j|jdd|_ tj|j|j|jdd|_ tj|j|j|jdd|_ dSNFbias) superrPrLinearr>num_attention_headshead_dimq_projnum_key_value_headsk_projv_projo_proj)rJrKrU __class__r&r'rPs  zJambaAttention.__init__N hidden_statesattention_maskpast_key_valuesrfkwargsrZcKs|jdd}g|d|jR}|||dd}|||dd} |||dd} |durF|| | |jd|i\} } t |j j t } | ||| | |f|jsZdn|j|jd|\} } | jg|dR} || } | | fS)Nr[rrrf)dropoutscaling)r^r|r}view transposerrr`rUr get_interfacerK_attn_implementationrtrainingattention_dropoutrreshape contiguousr)rJrrrrfr input_shape hidden_shape query_statesrWrXattention_interface attn_output attn_weightsr&r&r'forwards6  zJambaAttention.forwardNNN)r#r$r%rrprPr2ror)rtrrrsr __classcell__r&r&rr'rus$ rucseZdZdZdeffdd Z  ddejdedBdej dBfd d Z ddedBdej dBfd d Z  ddedBdej dBfd dZ Z S)JambaMambaMixeru Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) rKcst||_||_|j|_|j|_|j|_|j |j|_ |j |_ |j |_|j|_tj|j |j |j|j|j |jdd|_|j|_t|j|_tj|j|j d|jd|_tj|j |j |jddd|_tj|j |j dd|_td|jddddf}||j d}t t!||_"t t#|j |_$tj|j |j|jd|_%t&|j |j'd|_(t&|j|j'd|_)t&|j|j'd|_*t+d }t,|d da-t,|d da.t+d }t,|d da/t,|dda0t,|dda1t2t/t1t.t-t0fa3t3st45ddSdS)Nr) in_channels out_channelsrx kernel_sizegroupspaddingrrwFTr[epsz causal-conv1dcausal_conv1d_updatecausal_conv1d_fnz mamba-ssmselective_state_updatemamba_inner_fnselective_scan_fnaThe fast path is not available because on of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d.)6ryrPrKrUr>r?rMr@rNr=rL mamba_dt_ranktime_step_rankmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1d hidden_act activationractrzin_projx_projdt_projr2arangeexpandr ParameterlogA_logonesDout_projr! rms_norm_eps dt_layernorm b_layernorm c_layernormrgetattrrrrrrallis_fast_path_availablelogger warning_once)rJrKrUA causal_conv1d mamba_ssmrr&r'rPs\         zJambaMambaMixer.__init__Nr cache_paramsrc Cs|j\}}}|duo*|jo*|dko*|j|jjd|j|jjdko(|kn}||dd}|jddd\}} |durH||d}|j j |j j d|j j d} |rtt |d|j|j| |j j|j}|d}n'|durtj||j|jddf} |j|j| t|| |j j|jd}|dur||d}||dd} tj| |j|j|jgdd\} }}|| } ||}||}|jjj }t!t"|jjj |jj_ Wdn1swY|| dd}t! ||jj_ Wdn 1swYt#|j$% }|dur(|%nd}|rVt&|j|j|d|d||dddf|dddf|j'| d|dd d}n.t(||||dd|dd|j'%| |ddd \}}|dur|dur|j|j||)|dd}|S) Nrrrr\r[)r).rT) dt_softplus)delta_softplusreturn_last_state)*r^r<rArUrBrrchunk unsqueezerweightrsizersqueezerxrr functionalpadrNcopy_rrr2splitrrMrrrrdatano_grad zeros_likeexprfloatrrrr)rJrrrr8seq_lenr6use_precomputed_statesprojected_statesgate conv_weightsrAssm_parameters time_stepBCtime_proj_biasdiscrete_time_stepr scan_outputs ssm_statecontextualized_statesr&r&r'cuda_kernels_forward s  $                z$JambaMambaMixer.cuda_kernels_forwardc Cs|j\}}}|j}||dd}|jddd\} } |dur&| |d} t|t} | r|j|j jd|kr|j rD|j|j  } n|j|j } | | j } |jr|dkr|j|j jd|kr|j|j } tj| ddd} | dddddf| dddddf<| |j|j <tj| |jjdddddfdd} |jr| |jj7} ||  |d} n@tj| |j| jddf} | |j|j <||| dd|f} ntj||j|jf| j |d} ||| dd|f} |dur| |d} || dd}tj ||j!|j|jgdd\}}}|"|}|#|}|$|}|%|}tj&|dd}t'|j() }t'|ddddddf|dddddddf}|dddddddf|dddddddf)}|| dddddddf)}g}t*|D]D}|dddd|ddf| |dddd|ddf} t+| ||dd|ddfd}|,|dddddfqtj-|dd}|| |j.ddddf}||| }| r| |j|j <|/|dd}|S) Nrrr\rr[)shiftsdims.r+)0r^r-rrrr isinstancer)rBrUrclonerdr,r<rAr2rollsumrrrrxrrrrrNrFrLrMrrrrrrrsoftplusrrrrDmatmulrGstackrr)rJ input_statesrrr8rr6r-rrr use_cacher conv_staterrrrrr discrete_A discrete_BdeltaB_urrO scan_outputrr&r&r' slow_forwardtsx     ( (       :<$ <*  zJambaMambaMixer.slow_forwardcCsR|jjrtrd|jjjjvrtdd|j_|jjr"| |||S| |||S)NcudazFast Mamba kernels are not available. Make sure that they are installed and that the mamba module is on a CUDA device. Turning off the fast path `config.use_mamba_kernels=False` and falling back to the slow path.F) rKuse_mamba_kernelsrrrr,typerrrr)rJrrrr&r&r'rszJambaMambaMixer.forward)NN)r#r$r%rlrrPr2ror)rtrrrrr&r&rr'rs(D kXrc@r )JambaMLPNr"r&r&r&r'rr(rc@r ) JambaExpertsNr"r&r&r&r'rr(rcsDeZdZdZdeffdd ZddZdejdejfd d Z Z S) JambaSparseMoeBlocka This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accommodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. rKcsNt|j|_|j|_|j|_|j|_t j |j|jdd|_ t ||_ dSrv)ryrPr> hidden_dimrLffn_dim num_expertsnum_experts_per_toktop_krrzrouterrexpertsrJrKrr&r'rPs zJambaSparseMoeBlock.__init__cCs<tjjj|dtjd}tj||jdd\}}|||jfS)Nr[)r]r-r\) r2rrsoftmaxrtopkrrdr-)rJr router_logitsrouting_weights top_k_weights top_k_indexr&r&r'route_tokens_to_expertssz+JambaSparseMoeBlock.route_tokens_to_expertsrrZcCsR|j\}}}|d|}||}|||\}}||||}||||}|S)Nr[)r^rrrr r)rJrr8sequence_lengthrr rrr&r&r'rs   zJambaSparseMoeBlock.forward) r#r$r%rlrrPrr2rorrr&r&rr'rs   rcs~eZdZdedeffdd Z     ddejdejdBd ejdBd e dBd e dBd ejdBd e e dej fddZZS)JambaAttentionDecoderLayerrKrUcslt|jr |j|nd}t|||_|dkrtnt}|||_t|j |j d|_ t|j |j d|_ dS)Nrr) ryrPlayers_num_expertsru self_attnrr feed_forwardr!r>rinput_layernormpre_ff_layernormrJrKrUrffn_layer_classrr&r'rP s   z#JambaAttentionDecoderLayer.__init__NFrr position_idsrrrfrrZc Ks\|}||}|jd||||||d|\}} ||}|}||}||}||}|S)N)rrrrrrfr&)rrrr) rJrrrrrrfrresidualr6r&r&r'rs$     z"JambaAttentionDecoderLayer.forward)NNNFN)r#r$r%rrprPr2rortr)boolrr FloatTensorrrr&r&rr'r s0  rc sheZdZdedeffdd Z   ddejdejdBdejdBd e dBd e e d ej f d d Z ZS)JambaMambaDecoderLayerrKrUcsnt|jr |j|nd}t||d|_|dkrtnt}|||_t|j |j d|_ t|j |j d|_ dS)Nr)rKrUr) ryrPrrr*rrrr!r>rrrrrr&r'rP4s  zJambaMambaDecoderLayer.__init__NrrrrrrZcKsJ|}||}|j|||d}||}|}||}||}||}|S)N)rrr)rr*rr)rJrrrrrrr&r&r'r=s   zJambaMambaDecoderLayer.forwardr)r#r$r%rrprPr2rortr)rrrrrr&r&rr'r3s$ r) attentionr*csheZdZUeed<dZdZddgZdZdZ dZ dZ e e geeejddd Zefd d ZZS) JambaPreTrainedModelrKmodelTrrrr) layer_name)r attentionsr cst|t|tr5td|jddddf}||jd }t |j t |t |jdSt|trRt j|jd|jjdt j|jd|jjddSdS)Nrr[r)meanstd)ry _init_weightsrrr2rrMrrLrinitrrrones_rrnormal_ gate_up_projrKinitializer_range down_proj)rJmodulerrr&r'r'fs   z"JambaPreTrainedModel._init_weights)r#r$r%r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulrrrurrrz_can_record_outputsr2rr'rr&r&rr'r!Ws  r!cseZdZdeffdd Zeee       ddej dBdej dBdej dBde dBd ej dBd e dBd ej dBd eed efddZddZZS) JambaModelrKcst||j|_|j|_t|j|j|j|_g}t |j D]}t |j |}| |||dq t||_t|j|jd|_d|_|dS)N)rUrF)ryrP pad_token_id padding_idx vocab_sizer Embeddingr> embed_tokensrDrEALL_DECODER_LAYER_TYPESr;rG ModuleListlayersr!rfinal_layernormgradient_checkpointing post_init)rJrKdecoder_layersrO layer_classrr&r'rPus   zJambaModel.__init__N input_idsrrr inputs_embedsrrfrrZc Ks"|du|duAr td|dur||}|r)|dur)t|j|jd|j|jd}|durE|dur5|nd} tj | | |jd|jd}|durN| d}t |j|||||d} | ||} |} |j D]} t| trm| n| }| | f|||||d|} qd|| } |r|jsd|_t| |d S) Nz:You must specify exactly one of input_ids or inputs_embedsr)rKr8r-r,rr.)rKrGrrfrr)rrrrrfT)last_hidden_stater) ValueErrorr=r)rKr^r-r,rbr2rrr _update_mamba_maskr@rrrAr<r )rJrFrrrrGrrfrpast_seen_tokens causal_mask mamba_maskr decoder_layer layer_maskr&r&r'rs`      zJambaModel.forwardcCs6|}|dur |ddks|durt|dkrd}|S)zv No need for zeroing states when 1. Cached forward 2. Attending to all inputs Nrr)r2r)rJrrfrMr&r&r'rJs zJambaModel._update_mamba_mask)NNNNNNN)r#r$r%rrPrrrr2rtror)rrrrr rrJrr&r&rr'r8ss@  Cr8cseZdZdeffdd Z          ddejdBdejdBdejdBd edBd ej dBd ejdBd e dBd e dBdejdBde ejBde e deffdd ZZS)JambaForCausalLMrKcst||j|_dSrQ)ryrPrr rr&r'rPs  zJambaForCausalLM.__init__NrrFrrrrGlabelsroutput_router_logitsrflogits_to_keeprrZc s$tj|||||||| | f i| S)aj labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, JambaForCausalLM >>> model = JambaForCausalLM.from_pretrained("ai21labs/Jamba-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)ryr) rJrFrrrrGrQrrRrfrSrrr&r'rs$ zJambaForCausalLM.forward) NNNNNNNNNr)r#r$r%rrPr2rtror)rrrprrr rrr&r&rr'rPsJ     rPc@r )JambaForSequenceClassificationNr"r&r&r&r'rTr(rT)rPrTr8r!)Acollections.abcrtypingrr2rrr( activationsr integrationsr masking_utilsr modeling_layersr r modeling_outputsr r modeling_utilsrrprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrrllama.modeling_llamarrrmistral.modeling_mistralrmixtral.modeling_mixtralrrconfiguration_jambar get_loggerr#rr!r)ruModulerrrrrrr>r!r8rPrT__all__r&r&r&r'sL            _0 %( !g7