o ۾i@s ddlZddlZddlmZmZddlZddlZddlm m Z ddlm Z m Z ddl mZddlmZddlmZddlmZmZmZmZmZmZmZmZmZmZmZGdd d e jZ Gd d d ej!e jZ"Gd d d e"Z#Gddde jZ$Gddde jZ%dS)N)AnyLiteral)Tensornn)CheckpointWrapper)FullyShardedDataParallel)PretrainedConfig) AbsolutePositionalEncoding ConvModule FeedForwardMeanVarianceNormLayerMultiHeadedAttentionMultiSequentialNemoConvSubsamplingT5RelativeAttentionLogitBiasadaptive_enc_mask get_offset unfold_tensorc3seZdZdZ              d5dedededededededededededededed ed!ed"ed#ed$ed%ed&ed'ed(ed)ed*d+f2fd,d- Z +d6d.e j d/e j d0e j d1e j d2e d+Bd*e e j e j e j e j ff d3d4Z Z S)7ConformerEncoderLayeraConformerEncoder Layer module. for more details see conformer paper: https://arxiv.org/abs/2005.08100 This module implement the Conformer block layer. Args: d_model: int attention dim. ext_pw_out_channel: int if > 0, ext_pw_out_channel is a dim channel size for the last pointwise conv after swish activation. depthwise_seperable_out_channel: int if set different to 0, the number of depthwise_seperable_out_channel will be used as a channel_out of the second conv1d layer. otherwise, it equals to 0, the second conv1d layer is skipped. depthwise_multiplier: int number of input_dim channels duplication. this value will be used to compute the hidden channels of the Conv1D. n_head: int the number of heads for multihead attention module. d_ffn: int output size of the feed_forward blocks. ext_pw_kernel_size: int kernel size of the conv pointwise of the conformer. kernel_size: int kernel size. dropout_rate: float dropout rate. causal: bool, optional if set to True, convolution have no access to future frames. default False. batch_norm: bool, optional if set to True, apply batchnorm before activation in ConvModule layer of the conformer. default False activation: str, optional activation function name, one of ["relu", "swish", "sigmoid"], sigmoid activation is only used with "glu_in_fnn=True", default "relu". chunk_se: int, optional 0 for offline SE. 1 for streaming SE, where mean is computed by accumulated history until current chunk_se. 2 for streaming SE, where mean is computed by only the current chunk. default 0. chunk_size: int, optional chunk_size for cnn. default 18 conv_activation: str, optional activation function used in ConvModule part of the conformer, default "relu". conv_glu_type: str, optional activation function used for the glu inside the ConvModule part of the conformer. default: "sigmoid". bias_in_glu: bool, optional if set to True, use additive bias in the weight module before GLU. linear_glu_in_convm: bool, optional if set to True, use GLULinear module, otherwise, used GLUPointWiseConv module. default to False. attention_inner_dim: int, optional if equal to -1, attention dim for linears k/q/v is equal to d_model. otherwise attention_inner_dim is used. default -1. attention_glu_type: str, optional activation function for glu used in the multihead attention, default "swish". activation_checkpointing: str, optional a dictionary of {"module","interval","offload"}, where "module": str accept ["transformer", "attention"] to select which module should do activation checkpointing. "interval": int, default 1, interval of applying activation checkpointing, interval = 1 means that we apply checkpointing on every layer (if activation), otherwise, we apply it every x interval. "offload": bool, default False, if set to True, we offload activation to cpu and reload it during backward, otherwise, we recalculate activation in backward. default "". export: bool, optional if set to True, it removes the padding from convolutional layers and allow the onnx conversion for inference. default False. use_pt_scaled_dot_product_attention: bool, optional if set to True, use pytorch's scaled dot product attention implementation in training. attn_group_sizes: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), 1 = typical Multi-Head Attention, 1 < attn_group_sizes < attention_heads = Grouped-Query Attention attn_group_sizes = attention_heads = Multi-Query Attention r皙?FrelusigmoidTswishd_modelext_pw_out_channeldepthwise_seperable_out_channeldepthwise_multipliern_headd_ffnext_pw_kernel_size kernel_size dropout_ratecausal batch_norm activationchunk_se chunk_sizeconv_activation conv_glu_type bias_in_glulinear_glu_in_convmattention_inner_dimattention_glu_typeactivation_checkpointingexport#use_pt_scaled_dot_product_attentionattn_group_sizesreturnNcstt||| | |d|_t||| |||||d|_t||||||| | | | ||||||d|_t||| | |d|_t ||_ t ||_ dS)N)r"d_innerr*r-r2)r8 group_size)r7) super__init__r feed_forward_inr self_attnr convfeed_forward_outr LayerNormlayer_norm_att layer_norm)selfr"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9 __class__[/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/phi4mm_audio.pyr>sZ   zConformerEncoderLayer.__init__xpos_kpos_vmaskrelative_attention_biasc Csn|d||}||}||j|||||||d}|||}|d||}||}||||fS)aConformerEncoder forward. Args: x: input feature of shape (batch, max_time_in, size) pos_k: positional key embedding. pos_v: positional value embedding. mask: mask for x (batch, max_time_in) relative_attention_bias: bias added to attention logits w.r.t. relative positions (1, n_head, time1, time2) g?rO)r?rDr@rArBrE)rFrKrLrMrNrOnorm_xoutrIrIrJforwards    zConformerEncoderLayer.forward)rrrrrrrrrFFrrrrrTFrr r!FFrN)__name__ __module__ __qualname____doc__intfloatboolstrr>torchrtuplerS __classcell__rIrIrGrJr%sg     Src%sXeZdZdZ            d;d ed eeeBd eeeBdedededededededede ee fdBdede ee fdBde ddede ee fdBddf$fdd Z d ee jBdee jBfd!d"Zejde fd#d$Z  drdrhr/rerf num_headsrtrkrorurReLUupdaterembed ValueErrorr pos_embgetrelative_attention_bias_typerrelative_attention_bias_layerNotImplementedErrorr encoder_embedding)rFrdr/rerfrgrhrirjrkr*rlrmrnrorprtrudefault_nemo_conv_settingsirGrIrJr>Gsj           zTransformerEncoderBase.__init__ feature_lenscCs|jdkrb|jdddv}|jdd}|rP|rPt|tr(t||jnt ||j}||j}t|trF||dkd7<|S|dkrN|d7}|St|t rXt jntj}|||jSdS) ajfeature_lens: int return updated feature lens. This used to return a different lambda function for each case that computed the right thing. That does not work within Torchscript. If you really need this to be faster, create nn.Module()-s for all the cases and return one of them. Torchscript does support that. rarxrv)rvstridingstriding_conv1dr~FrN) rhror isinstancerr]ceilrklongmathrY)rFrsubsampling_causal_condr~ lens_changefeature_lens_remainder ceil_funcrIrIrJcompute_lens_changes,   z*TransformerEncoderBase.compute_lens_changecCsdS)z'Abstract forward method implementation.NrIrFrIrIrJrSszTransformerEncoderBase.forwardcCs|dur|j}|dur|j}t|tr@ttjdt|dd}||}t|ts,tdt|t|kr8td||}||fS|}|}||fS)z>If chunk size is a list, we will randomly select a chunk size.Nr)r)lowhighsizez5Since chunk_size is a list, left_chunk must be a listzBThe length of left_chunk must be the same as length of chunk_size.) r/rerlistrYr]randintlenr)rFr/rechunk_size_indexchunk_size_train_effleft_chunk_train_effrIrIrJ_chunk_size_selections,  z,TransformerEncoderBase._chunk_size_selectionrcCs0t|t}t|t}|}|r|j}|r|j}|SrT)rrr_checkpoint_wrapped_modulemodule)rFris_embed_using_act_chkptis_embed_fsdp_wrapped embed_classrIrIrJ_get_embed_classs  z'TransformerEncoderBase._get_embed_class input_tensormaskscCs2||j}t|ts J|||\}}||fSrT)rrrr)rFrrrrIrIrJ_forward_embeddings_cores z/TransformerEncoderBase._forward_embeddings_corecCs$d}d}|jdur||}||fSrT)rr)rFrrLrMrIrIrJ_position_embeddings z*TransformerEncoderBase._position_embeddingseq_len batch_sizec CsB|||\}}td||}t|||dd|ddg}|S)Nr) left_windowr)rnparanger unsqueezeexpand) rFrrr/rerrchunk_start_idxenc_streaming_maskrIrIrJ_streaming_masks z&TransformerEncoderBase._streaming_maskxs_pad chunk_size_nc left_chunk_nccCst||jd}|dkrtd|d|jd}||||j|j}|jr0|}|}|}| ||\}}|} | durI|durI|| @} n |durP|} n| } |durq|||||} |jre| } |durn|| @} n| } nd} | |\} }|dur|| || |fS|| || || fS)aForwarding the inputs through the top embedding layers Args: xs_pad: torch.Tensor input tensor masks: torch.Tensor input mask chunk_size_nc: (optional, default is None) chunk size for non-causal layers left_chunk_nc: (optional, default is None) # of left chunks for non-causal layers rrzFThe sequence length after time reduction is invalid: z. Your input feature is too short. Consider filtering out the very short sentence from data loaderN) rYrshaperrr/reis_cudacudarr)rFrrrrrrrrstreaming_maskhs_maskenc_streaming_mask_nc hs_mask_ncrLrMrIrIrJforward_embeddingssH%    z)TransformerEncoderBase.forward_embeddingscCst|j|jS)a!Returns offset used when retaining inputs for decoding. This is essentially, how many additional frames have to be added to the front-end CNN input to ensure it can produce a single output. So if the "padding" parameter is 0, typically offset will be > 0. )rrhrkrrIrIrJrosz!TransformerEncoderBase.get_offset)rrrarFrrbrNrbNrcrN)NN)rUrVrWrXrYrr\r[rZdictrrr>r]rrabcabstractmethodrSr^rrModulerrrrrrr_rIrIrGrJr`sM       T $           Vr`cOseZdZUdZeeed<ddddddd d d d d d dddd ddddd d dd d gdddd ddd ddf#dedeeeBdeeeBdedBdedededededede d e d!ed"e d#ed$ed%ed&ed'ed(ed)ed*ed+ed,e d-e d.ed/e d0edeed1ed2e ee fdBd3ed4e d5e ee fdBd6e d7d8e d9ed:e ee fdBd;dfNfdejd;ejdBfd?d@ZdAejdBejdCejdBd;ejfdDdEZejjdAejdFejd;eejejffdGdHZZS)IConformerEncoderarConformerEncoder module. see original paper for more details: https://arxiv.org/abs/2005.08100 Please set causal = True in streaming model Args: input_size: int input feature dimension. chunk_size: int, list(int) Number of frames for each chunk This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training Some examples for the 2 cases: chunk_size = 12 chunk_size = [6, 8, 12, 24] left_chunk: int, list(int) Number of chunks used for masking in streaming mode. This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training. When chunk_size is a list, left_chunk must be a list with same length. Some examples for the 2 cases: left_chunk = 6 left_chunk = [12, 9, 6, 3] num_lang: int This parameter is used to store the number of languages in the lang_dict, only used for multiseed/multilingual models. default None. attention_dim: int, optional attention dimension. default 256. attention_heads: int, optional the number of heads. default 4 linear_units: the number of units of position-wise feed forward. default 2048 num_block: number of Transformer layer. default 6 dropout_rate: float, optional dropout rate. default 0.1 input_layer: str, optional input layer type before Conformer, one of ["linear", "conv2d", "custom", "vgg2l", "embed"], default "conv2d" causal: bool, optional if set to True, convolution have no access to future frames. default False. batch_norm: bool, optional if set to True, apply batchnorm before activation in ConvModule layer of the conformer. default False cnn_out: int, optional the number of CNN channels before Conformer. default -1. cnn_layer_norm: bool, optional layer norm between Conformer and the first CNN. default False. ext_pw_out_channel: int, optional the number of channel for CNN before depthwise_seperable_CNN. If 0 then use linear. default 0. ext_pw_kernel_size: int, optional kernel size of N before depthwise_seperable_CNN. only work for ext_pw_out_channel > 0. default 1 depthwise_seperable_out_channel: int, optional the number of channel for depthwise_seperable_CNN. default 256. depthwise_multiplier: int, optional the number of multiplier for depthwise_seperable_CNN. default 1. chunk_se: int, optional 0 for offline SE. 1 for streaming SE, where mean is computed by accumulated history until current chunk_se. 2 for streaming SE, where mean is computed by only the current chunk. default 0. kernel_size: int, optional the number of kernels for depthwise_seperable_CNN. default 3. activation: str, optional FeedForward block activation. one of ["relu", "swish", "sigmoid"] default "relu". conv_activation: str, optional activation function used in ConvModule part of the conformer, default "relu". conv_glu_type: str, optional activation used use glu in depthwise_seperable_CNN, default "sigmoid" bias_in_glu: bool, optional if set to True, use additive bias in the weight module before GLU. default True linear_glu_in_convm: bool, optional if set to True, use GLULinear module, otherwise, used GLUPointWiseConv module. default to False. attention_glu_type: str only work for glu_in_attention !=0 default "swish". export: bool, optional if set to True, it removes the padding from convolutional layers and allow the onnx conversion for inference. default False. activation_checkpointing: str, optional a dictionarry of {"module","interval","offload"}, where "module": str accept ["transformer", "attention"] to select which module should do activation checkpointing. "interval": int, default 1, interval of applying activation checkpointing, interval = 1 means that we apply checkpointing on every layer (if activation), otherwise, we apply it every x interval. "offload": bool, default False, if set to True, we offload activation to cpu and reload it during backward, otherwise, we recalculate activation in backward. default "". extra_layer_output_idx: int the layer index to be exposed. relative_attention_bias_args: dict, optional use more efficient scalar bias-based relative multihead attention (Q*K^T + B) implemented in cmb.basics.embedding. [T5/ALiBi]RelativeAttentionLogitBias usage: relative_attention_bias_args={"type": t5/alibi} additional method-specific arguments can be provided (see transformer_base.py) time_reduction: int optional time reduction factor default 4 use_pt_scaled_dot_product_attention: whether to use pytorch scaled dot product attention in training. Default: False nemo_conv_settings: dict, optional A dictionary of settings for NeMo Subsampling. default: None usage: nemo_conv_settings= { "subsampling": dw_striding/striding/dw_striding_conv1d/striding_conv1d, "conv_channels": int, "subsampling_conv_chunking_factor": int, "is_causal": True/False } conv2d_extra_padding: str, optional Add extra padding in conv2d subsampling layers. Choices are (feat, feat_time, none, True) Default: none replication_pad_for_subsample_embedding: For batched-streaming decoding, use "replication" padding for the cache at start of utterance. Default: False attention_group_size: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), 1 = typical Multi-Head Attention, 1 < attention_group_size < attention_heads = Grouped-Query Attention attention_group_size = attention_heads = Multi-Query Attention extra_multi_layer_output_idxsNrrrrraTFrrrrrrr r!rcrdr/renum_langrfrg linear_units num_blocksr*rhr+r,rirjr#r(r$r%r.r)r-r0r1r2r3r5r7extra_layer_output_idxr6rmrkr8rorprq'replication_pad_for_subsample_embeddingrtrur:c''stj| || | || |d|"|#|&d||_||_|_|$|_|jdks-Jd|j|_t fddt |D|_ ||_ ||_ |j dtdd d dS) Nrb)r*rmrnrorprtrurrcsg|]L}tdidddd dddddd d d d d d d dddddddqS)r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r5r6r7r8r9rI)r.0_r-r6rfr5rtrgr,r2r+r.r/r0r1r%r$r*r7r(r#r)r3rr8rIrJ hsd     z-ConformerEncoder.__init__..dev_typerIF) persistent)r=r>rrr)rr num_heads_krrangeencodersrrregister_bufferr]zeros)'rFrdr/rerrfrgrrr*rhr+r,rirjr#r(r$r%r.r)r-r0r1r2r3r5r7rrr6rmrkr8rorprrtrurGrrJr>!sF) 6zConformerEncoder.__init__rcCs|jr||SdSrT)r)rFrrIrIrJinit_relative_attention_biass z-ConformerEncoder.init_relative_attention_biasrdevicerNc Cs|jd}|jd}||||j|j}||}|dur|S|d}|}tjd||d| dd| dk} | d} | |@} | S)Nrr)rr) rrr/retosumr]rrrr) rFrrrNmax_audio_lengthrrrpadding_lengthpad_maskrIrIrJcalculate_hs_masks$       z"ConformerEncoder.calculate_hs_maskrcCs||}|||\}}}}}d}|j\}} } d} | | kr{d}| | dkr-| | | } nd} | dkrEt|ddd| fdd} | |j}t|| }|durq|d}t|d| fdd}| d }t|| }|d }nd}| ||j|}| |}|jdko|du}|r|||||^}}nt|jD]\}}||||||d \}}}}q|r|jd}||d|}| dkr|ddd| ddf}||fS) zConformer Forward function Args: xs_pad: torch.Tensor input tensor masks: torch.Tensor post-embedding input lengths FiTrconstantNrrrP)rrrFpadrrrsqueezerrZr[rrrr enumeratereshape)rFrrrrLrMrunfoldedori_bzrD max_seq_lenchunk_pad_sizeinput_tensor_padsubsampled_pad_mask extra_padded_subsamlped_pad_mask masks_unfoldrO_simplified_pathrrlayer embed_dimrIrIrJrSst         zConformerEncoder.forward)rUrVrWrXrrY__annotations__rZr\r[rrrr>r]rrrrjitignorer^rSr_rIrIrGrJrys  &        !"#$%&'(j  rcseZdZdZ        dd ed ed ed ed edededeffdd Z ddej dej dBdedBde ej edBffddZ Z S) WindowQformerzWindow-level QformerrrrrbT window_size num_queriesrrfrgrr*normalize_beforec shttfddt|D|_ttd||_ r,tj ddnd|_ ||_ dS)Nc s$g|]}tjdddqS)rT)r"nheaddim_feedforwarddropoutr- batch_first norm_first)rTransformerDecoderLayerrrfrgr*rrrIrJrs z*WindowQformer.__init__..rg-q=)eps) r=r>r ModuleListrdecoders Parameterr]rqueriesrC after_normr) rFrrrrfrgrr*rrGr rJr>s   zWindowQformer.__init__N audio_embedrN embed_lenr:c Cs|dd}|jd|j}|dkrt|d|j|fdd}tj|ddddfd|jfd|jfd}|j\}}}||d|j|}|dd }||||jd}|j ||dd} |j D] } | | |d|d } qb|j durw| | } |dur||j}| ||d} | |fS) zforward decoderrrrrr.N)r)strider)tgtmemorytgt_mask memory_mask) transposerrrrunfoldview contiguousrrrr) rFrrNrpadding embed_chunkbszrslenqrrRrIrIrJrS(s0      zWindowQformer.forward)rrrrrrrbTrT) rUrVrWrXrYrZr[r>r]rr^rSr_rIrIrGrJrsJ &rc seZdZdZdededdffdd Zdejddfd d Z d ejddfd d Z  ddejdejdBde dejfddZ  ddejdejdBde dejfddZ ZS)AudioEmbeddingzImage embedding.configkwargsr:Nc st||_t|dr|jn|j}d}d|_t|jt rE|j dddkrE|j dd}|dus4Jt di||_ |d}|d}nt d|dusQJd ||_||_| d d |_| d d |_| dd r~| di}||d<tdi||_nd|_| dd r|jdusJd| di}d|j||dd td d} |r| |dD] } | |vsJdqtdi| |_nd|_| dd} | dkrt|||_ni| dkr7|} d} |js|jrd n|j|_t||j| g}td | D]}|tt| | gqtj||_t||j| g}td | D]}|tt| | gqtj||_ nt d| d|j!|_!d|_"d|_#dS)Nn_embdnamecascadesr$rfrdr!z(Remember to set values for audio_dim_outfreeze_audio_processorFdownsample_rater use_qformerqformer_configuse_conv_downsamplez6don't support use qformer and conv downsample togetherrorvrrwrrprojection_clslinearmlprzprojection_cls = z, not implementedrI)$r=r>r$hasattrr& hidden_size layer_idxraudio_processorrrrencoderr audio_dim_out audio_dim_inr*r+rqformerrrrrconv_dsLinearaudio_projectionlinear_downsample_raterextendGELU Sequentialaudio_projection_for_vision vocab_size input_embedsaudio_embed_sizes)rFr$r%r3r7encoder_confign_melsr-rorrr/dim_projectiondepthlayersrrGrIrJr>Xs                 zAudioEmbedding.__init__rCcC ||_dSrT)rC)rFrCrIrIrJset_audio_embeds zAudioEmbedding.set_audio_embedsrDcCrJrT)rD)rFrDrIrIrJset_audio_embed_sizesrLz$AudioEmbedding.set_audio_embed_sizesspeechaudio_attention_maskaudio_projection_modec CsB|jr t|||\}}Wdn1swYn|||\}}|jdur6|j|dd\}}|jdurM|durD|d}|j||d\}}|jdkr|\}}} ||j} | dkrqt |ddd|j| fdd}|d}| |||j| |j}|dkr| |} | S|dkr| |} | Std|d ) zl arguments: input_embeds: audio features (B, T, D) B: num audios in a sequence N)rNrrrrNvisionzaudio_projection_mode = z not implemented)r*r]no_gradr6r9r:rr=rrrrr<rAr) rFrCrOrPaudio_featuresrrbsrfeat_dimraudio_set_tensorrIrIrJget_audio_featuressJ          z!AudioEmbedding.get_audio_featuresrScCs |j|d||d}|dS)z arguments: audio_features: audio features (T, D) returns: audio_embeds: audio embeddings (num_audio_tokens, hidden_dim) r)rOrP)rWrr)rFrSrOrP audio_embedsrIrIrJrSs   zAudioEmbedding.forward)NrN)rUrVrWrXrrr>r]rrKrMr\rWrSr_rIrIrGrJr#Us6h 9r#)&rrtypingrrnumpyrr]torch.nn.functionalr functionalrr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperr2torch.distributed.fsdp.fully_sharded_data_parallelr transformersr'vllm.model_executor.models.phi4mm_utilsr r r r r rrrrrrrrABCr`rrr#rIrIrIrJs,   4Z} R