o پi@sddlZddlZddlmZmZddlZddlZddlm m Z ddlm Z m Z ddl mZddlmZddlmZddlmZmZmZmZmZmZmZmZmZmZmZdZGd d d e j Z!Gd d d ej"e j Z#Gd dde#Z$Gddde j Z%Gddde j Z&dS)N)LiteralOptional)Tensornn)CheckpointWrapper)FullyShardedDataParallel)PretrainedConfig) AbsolutePositionalEncoding ConvModule FeedForwardMeanVarianceNormLayerMultiHeadedAttentionMultiSequentialNemoConvSubsamplingT5RelativeAttentionLogitBiasadaptive_enc_mask get_offset unfold_tensoriK csneZdZdZ              ddeffdd Z ddeefddZZ S)ConformerEncoderLayeraConformerEncoder Layer module. for more details see conformer paper: https://arxiv.org/abs/2005.08100 This module implement the Conformer block layer. Args: d_model: int attention dim. ext_pw_out_channel: int if > 0, ext_pw_out_channel is a dim channel size for the last pointwise conv after swish activation. depthwise_seperable_out_channel: int if set different to 0, the number of depthwise_seperable_out_channel will be used as a channel_out of the second conv1d layer. otherwise, it equal to 0, the second conv1d layer is skipped. depthwise_multiplier: int number of input_dim channels duplication. this value will be used to compute the hidden channels of the Conv1D. n_head: int the number of heads for multihead attention module. d_ffn: int output size of the feed_forward blocks. ext_pw_kernel_size: int kernel size of the conv pointwise of the conformer. kernel_size: int kernel size. dropout_rate: float dropout rate. causal: bool, optional if set to True, convolution have no access to future frames. default False. batch_norm: bool, optional if set to True, apply batchnorm before activation in ConvModule layer of the conformer. default False activation: str, optional activation function name, one of ["relu", "swish", "sigmoid"], sigmoid activation is only used with "glu_in_fnn=True", default "relu". chunk_se: int, optional 0 for offline SE. 1 for streaming SE, where mean is computed by accumulated history until current chunk_se. 2 for streaming SE, where mean is computed by only the current chunk. default 0. chunk_size: int, optional chunk_size for cnn. default 18 conv_activation: str, optional activation function used in ConvModule part of the conformer, default "relu". conv_glu_type: str, optional activation function used for the glu inside the ConvModule part of the conformer. default: "sigmoid". bias_in_glu: bool, optional if set to True, use additive bias in the weight module before GLU. linear_glu_in_convm: bool, optional if set to True, use GLULinear module, otherwise, used GLUPointWiseConv module. default to False. attention_inner_dim: int, optional if equal to -1, attention dim for linears k/q/v is equal to d_model. otherwise attention_inner_dim is used. default -1. attention_glu_type: str, optional activation function for glu used in the multihead attention, default "swish". activation_checkpointing: str, optional a dictionarry of {"module","interval","offload"}, where "module": str accept ["transformer", "attention"] to select which module should do activation checkpointing. "interval": int, default 1, interval of applying activation checkpointing, interval = 1 means that we apply checkpointing on every layer (if activation), otherwise, we apply it every x interval. "offload": bool, default False, if set to True, we offload activation to cpu and reload it during backward, otherwise, we recalculate activation in backward. default "". export: bool, optional if set to True, it remove the padding from convolutional layers and allow the onnx conversion for inference. default False. use_pt_scaled_dot_product_attention: bool, optional if set to True, use pytorch's scaled dot product attention implementation in training. attn_group_sizes: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), 1 = typical Multi-Head Attention, 1 < attn_group_sizes < attention_heads = Grouped-Query Attention attn_group_sizes = attention_heads = Multi-Query Attention r皙?FrelusigmoidTswishattn_group_sizescstt||| | |d|_t||| |||||d|_t||||||| | | | ||||||d|_t||| | |d|_t ||_ t ||_ dS)N)d_modeld_inner dropout_rate activation bias_in_glu)#use_pt_scaled_dot_product_attention group_size)export) super__init__r feed_forward_inr self_attnr convfeed_forward_outr LayerNormlayer_norm_att layer_norm)selfr#ext_pw_out_channeldepthwise_seperable_out_channeldepthwise_multipliern_headd_ffnext_pw_kernel_size kernel_sizer%causal batch_normr&chunk_se chunk_sizeconv_activation conv_glu_typer'linear_glu_in_convmattention_inner_dimattention_glu_typeactivation_checkpointingr*r(r" __class__R/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/models/phi4mm_audio.pyr,sZ   zConformerEncoderLayer.__init__Nrelative_attention_biasc Csn|d||}||}||j|||||||d}|||}|d||}||}||||fS)aConformerEncoder forward. Args: x: torch.Tensor input feature of shape (batch, max_time_in, size) pos_k: torch.Tensor positional key embedding. mask: torch.Tensor mask for x (batch, max_time_in) relative_attention_bias: Optional[torch.Tensor] bias added to attention logits w.r.t. relative positions (1, n_head, time1, time2) g?rJ)r-r2r.r/r0r3)r4xpos_kpos_vmaskrJnorm_xoutrHrHrIforwards    zConformerEncoderLayer.forward)rrrrrrrrrFFrrrrrTFrr r!FFrN) __name__ __module__ __qualname____doc__intr,rrrR __classcell__rHrHrFrIr.sBgSrcseZdZdZ            d!d ed ffd d ZddZejddZ d"ddZ ddZ ddZ ddZ ddZd"ddZdd ZZS)#TransformerEncoderBaseaj The Base class for Transformer based encoders Please set causal = True in streaming model Args: input_size: int input feature dimension. chunk_size: int, list(int) Number of frames for each chunk This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training Some examples for the 2 cases: chunk_size = 12 chunk_size = [6, 8, 12, 24] left_chunk: int, list(int) Number of chunks used for masking in streaming mode. This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training. When chunk_size is a list, left_chunk must be a list with same length. Some examples for the 2 cases: left_chunk = 6 left_chunk = [12, 9, 6, 3] attention_dim: int, optional attention dimension. default 256. attention_heads: int, optional the number of heads. default 4 input_layer: str, optional input layer type before Conformer, one of ["linear", "conv2d", "custom", "vgg2l", "embed"], default "conv2d" cnn_out: int, optional the number of CNN channels before Conformer. default -1. cnn_layer_norm: bool, optional layer norm between Conformer and the first CNN. default False. time_reduction: int, optional time reduction factor default 4 dropout_rate: float, optional dropout rate. default 0.1 padding_idx: int, optional padding index for input_layer=embed default -1 relative_attention_bias_args: dict, optional use more efficient scalar bias-based relative multihead attention (Q*K^T + B) implemented in cmb.basics.embedding. [T5/ALiBi]RelativeAttentionLogitBias usage: relative_attention_bias_args={"type": t5/alibi} additional method-specific arguments can be provided (see transformer_base.py) positional_dropout_rate: float, optional dropout rate after positional encoding. default 0.0 nemo_conv_settings: dict, optional A dictionary of settings for NeMo Subsampling. default None conv2d_extra_padding: str, optional Add extra padding in conv2d subsampling layers. Choices are (feat, feat_time, none, True). if True or feat_time, the extra padding is added into non full supraframe utts in batch. Default: none attention_group_size: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), 1 = typical Multi-Head Attention, 1 < attention_group_size < attention_heads = Grouped-Query Attention attention_group_size = attention_heads = Multi-Query Attention rr nemo_convrFNnonerconv2d_extra_paddingfeat feat_timer]Tc s6t||_||_||_||_||_||_||_| |_ ||_ ||_ |jdkrSd|j ||ddt dd}|rJ||dD] }||vsIJdq?tdi||_ntd |t|| |_| rf| d nd|_|jd kr|j|jd kszJd t|j|j| dd| ddd|_ntt|j d|_dS)Nr[ dw_stridingrrF subsamplingsubsampling_factorfeat_infeat_out conv_channels subsampling_conv_chunking_factorr& is_causalrerfrg6{i} should be specified outside of the NeMo dictionaryzunknown input_layer: typet5r'attention_group_size must divide n_headt5_bias_max_distanceit5_bias_symmetric) max_distance symmetric input_sizerH)r+r,rt input_layerr? left_chunk attention_dim num_headsattention_group_sizetime_reductionnemo_conv_settingsencoder_embedding_configrReLUupdaterembed ValueErrorr pos_embgetrelative_attention_bias_typerrelative_attention_bias_layerNotImplementedErrorr encoder_embedding)r4rtr?rvrwattention_headsrucnn_outcnn_layer_normrzr% padding_idxrelative_attention_bias_argspositional_dropout_rater{r^ryr|default_nemo_conv_settingsirFrHrIr,Ssj           zTransformerEncoderBase.__init__cCs|jdkrb|jdddv}|jdd}|rP|rPt|tr(t||jnt ||j}||j}t|trF||dkd7<|S|dkrN|d7}|St|t rXt jntj}|||jSdS) ajfeature_lens: int return updated feature lens. This used to return a different lambda function for each case that computed the right thing. That does not work within Torchscript. If you really need this to be faster, create nn.Module()-s for all the cases and return one of them. Torchscript does support that. r[rdrb)rbstridingstriding_conv1drjFrN) rur{r isinstancertorchceilrzlongmathrX)r4 feature_lenssubsampling_causal_condrj lens_changefeature_lens_remainder ceil_funcrHrHrIcompute_lens_changes,   z*TransformerEncoderBase.compute_lens_changecCsdS)z'Abstract forward method implementation.NrHr4rHrHrIrRszTransformerEncoderBase.forwardcCs|dur|j}|dur|j}t|tr@ttjdt|dd}||}t|ts,tdt|t|kr8td||}||fS|}|}||fS)z>If chunk size is a list, we will randomly select a chunk size.Nr)r)lowhighsizez5Since chunk_size is a list, left_chunk must be a listzBThe length of left_chunk must be the same as length of chunk_size.) r?rvrlistrXrrandintlenr)r4r?rvchunk_size_indexchunk_size_train_effleft_chunk_train_effrHrHrI_chunk_size_selections,  z,TransformerEncoderBase._chunk_size_selectioncCs0t|t}t|t}|}|r|j}|r|j}|SrS)rrr_checkpoint_wrapped_modulemodule)r4ris_embed_using_act_chkptis_embed_fsdp_wrapped embed_classrHrHrI_get_embed_classs  z'TransformerEncoderBase._get_embed_classcCs2||j}t|ts J|||\}}||fSrS)rrrr)r4 input_tensormasksrrHrHrI_forward_embeddings_cores z/TransformerEncoderBase._forward_embeddings_corecCs$d}d}|jdur||}||fSrS)rr)r4rrMrNrHrHrI_position_embeddings z*TransformerEncoderBase._position_embeddingc CsB|||\}}td||}t|||dd|ddg}|S)Nr) left_windowr)rnparanger unsqueezeexpand) r4seq_len batch_sizer?rvrrchunk_start_idxenc_streaming_maskrHrHrI_streaming_masks z&TransformerEncoderBase._streaming_maskcCst||jd}|dkrtd|d|jd}||||j|j}|jr0|}|}|}| ||\}}|} | durI|durI|| @} n |durP|} n| } |durq|||||} |jre| } |durn|| @} n| } nd} | |\} }|dur|| || |fS|| || || fS)aForwarding the inputs through the top embedding layers Args: xs_pad: torch.Tensor input tensor masks: torch.Tensor input mask chunk_size_nc: (optional, default is None) chunk size for non-causal layers left_chunk_nc: (optional, default is None) # of left chunks for non-causal layers rrzEThe sequence length after time reduction is invalid: z. Your input feature is too short. Consider filtering out the very short sentence from data loaderN) rXrshaperrr?rvis_cudacudarr)r4xs_padr chunk_size_nc left_chunk_ncrrrrstreaming_maskhs_maskenc_streaming_mask_nc hs_mask_ncrMrNrHrHrIforward_embeddingssH    z)TransformerEncoderBase.forward_embeddingscCst|j|jS)a!Returns offset used when retaining inputs for decoding. This is essentially, how many additional frames have to be added to the front-end CNN input to ensure it can produce a single output. So if the "padding" parameter is 0, typically offset will be > 0. )rrurzrrHrHrIrVsz!TransformerEncoderBase.get_offset)rrr[rFrr\rNr\Nr]rN)NN)rTrUrVrWrr,rabcabstractmethodrRrrrrrrrrYrHrHrFrIrZ s:MT"    @rZc#seZdZUdZeeed<ddddddd d d d d d dddd ddddd d dd d gdddd ddd ddf#dedffdd ZddZ ddZ e j j ddZZS)ConformerEncoderaConformerEncoder module. see original paper for more details: https://arxiv.org/abs/2005.08100 Please set causal = True in streaming model Args: input_size: int input feature dimension. chunk_size: int, list(int) Number of frames for each chunk This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training Some examples for the 2 cases: chunk_size = 12 chunk_size = [6, 8, 12, 24] left_chunk: int, list(int) Number of chunks used for masking in streaming mode. This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training. When chunk_size is a list, left_chunk must be a list with same length. Some examples for the 2 cases: left_chunk = 6 left_chunk = [12, 9, 6, 3] left_chunk: int number of chunks used for masking in streaming mode. num_lang: int This parameter is used to store the number of languages in the lang_dict, only used for multiseed/multilingual models. default None. attention_dim: int, optional attention dimension. default 256. attention_heads: int, optional the number of heads. default 4 linear_units: the number of units of position-wise feed forward. default 2048 num_block: number of Transformer layer. default 6 dropout_rate: float, optional dropout rate. default 0.1 input_layer: str, optional input layer type before Conformer, one of ["linear", "conv2d", "custom", "vgg2l", "embed"], default "conv2d" causal: bool, optional if set to True, convolution have no access to future frames. default False. batch_norm: bool, optional if set to True, apply batchnorm before activation in ConvModule layer of the conformer. default False cnn_out: int, optional the number of CNN channels before Conformer. default -1. cnn_layer_norm: bool, optional layer norm between Conformer and the first CNN. default False. ext_pw_out_channel: int, optional the number of channel for CNN before depthwise_seperable_CNN. If 0 then use linear. default 0. ext_pw_kernel_size: int, optional kernel size of N before depthwise_seperable_CNN. only work for ext_pw_out_channel > 0. default 1 depthwise_seperable_out_channel: int, optional the number of channel for depthwise_seperable_CNN. default 256. depthwise_multiplier: int, optional the number of multiplier for depthwise_seperable_CNN. default 1. chunk_se: int, optional 0 for offline SE. 1 for streaming SE, where mean is computed by accumulated history until current chunk_se. 2 for streaming SE, where mean is computed by only the current chunk. default 0. kernel_size: int, optional the number of kernels for depthwise_seperable_CNN. default 3. activation: str, optional FeedForward block activation. one of ["relu", "swish", "sigmoid"] default "relu". conv_activation: str, optional activation function used in ConvModule part of the conformer, default "relu". conv_glu_type: str, optional activation used use glu in depthwise_seperable_CNN, default "sigmoid" bias_in_glu: bool, optional if set to True, use additive bias in the weight module before GLU. default True linear_glu_in_convm: bool, optional if set to True, use GLULinear module, otherwise, used GLUPointWiseConv module. default to False. attention_glu_type: str only work for glu_in_attention !=0 default "swish". export: bool, optional if set to True, it remove the padding from convolutional layers and allow the onnx conversion for inference. default False. activation_checkpointing: str, optional a dictionarry of {"module","interval","offload"}, where "module": str accept ["transformer", "attention"] to select which module should do activation checkpointing. "interval": int, default 1, interval of applying activation checkpointing, interval = 1 means that we apply checkpointing on every layer (if activation), otherwise, we apply it every x interval. "offload": bool, default False, if set to True, we offload activation to cpu and reload it during backward, otherwise, we recalculate activation in backward. default "". extra_layer_output_idx: int the layer index to be exposed. relative_attention_bias_args: dict, optional use more efficient scalar bias-based relative multihead attention (Q*K^T + B) implemented in cmb.basics.embedding. [T5/ALiBi]RelativeAttentionLogitBias usage: relative_attention_bias_args={"type": t5/alibi} additional method-specific arguments can be provided (see transformer_base.py) time_reduction: int optional time reduction factor default 4 use_pt_scaled_dot_product_attention: whether to use pytorch scaled dot product attention in training. Default: False nemo_conv_settings: dict, optional A dictionary of settings for NeMo Subsampling. default: None usage: nemo_conv_settings= { "subsampling": dw_striding/striding/dw_striding_conv1d/striding_conv1d, "conv_channels": int, "subsampling_conv_chunking_factor": int, "is_causal": True/False } conv2d_extra_padding: str, optional Add extra padding in conv2d subsampling layers. Choices are (feat, feat_time, none, True) Default: none replication_pad_for_subsample_embedding: For batched-streaming decoding, use "replication" padding for the cache at start of utterance. Default: False attention_group_size: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), 1 = typical Multi-Head Attention, 1 < attention_group_size < attention_heads = Grouped-Query Attention attention_group_size = attention_heads = Multi-Query Attention extra_multi_layer_output_idxsNrrrrr[TFrrrrrrr r!r]r^r_c''stj| || | || |d|"|#|&d||_||_|_|$|_|jdks-Jd|j|_t fddt |D|_ ||_ ||_ |j dtdd d dS) Nr\)r%rrr{r^ryr|rrocsg|]L}tdidddd dddddd d d d d d d dddddddqS)r#r5r6r7r8r9r:r;r%r<r=r&r>r?r@rAr'rBrDrEr*r(r"rH)r.0_r&rErwrDryrr=r'r<r>r?r@rAr7r6r%r*r:r5r;rB linear_unitsr(rHrI Qsd     z-ConformerEncoder.__init__..dev_typerHF) persistent)r+r, num_blocksnum_langr;'replication_pad_for_subsample_embeddingrx num_heads_krrangeencodersextra_layer_output_idxrregister_bufferrzeros)'r4rtr?rvrrwrrrr%rur<r=rrr5r:r6r7r>r;r&r@rAr'rBrDr*rrrErrzr(r{r^rryr|rFrrIr, sF) 6zConformerEncoder.__init__cCs|jr||SdSrS)r)r4rrHrHrIinit_relative_attention_biasts z-ConformerEncoder.init_relative_attention_biasc Cs|jd}|jd}||||j|j}||}|dur|S|d}|}tjd||d| dd| dk} | d} | |@} | S)Nrr)devicer) rrr?rvtosumrrrrr) r4rrrOmax_audio_lengthrrrpadding_lengthpad_maskrHrHrIcalculate_hs_maskxs$       z"ConformerEncoder.calculate_hs_maskcCs||}|||\}}}}}d}|j\}} } d} | | kr{d}| | dkr-| | | } nd} | dkrEt|ddd| fdd} | |j}t|| }|durq|d}t|d| fdd}| d }t|| }|d }nd}| ||j|}| |}|jdko|du}|r|||||^}}nt|jD]\}}||||||d \}}}}q|r|jd}||d|}| dkr|ddd| ddf}||fS) zConformer Forward function Args: xs_pad: torch.Tensor input tensor masks: torch.Tensor post-embedding input lengths FiTrconstantNrrrK)rrrFpadrrrsqueezerfloatboolrrrr enumeratereshape)r4rrrrMrNrunfoldedori_bzrD max_seq_lenchunk_pad_sizeinput_tensor_padsubsampled_pad_mask extra_padded_subsamlped_pad_mask masks_unfoldrJ_simplified_pathrrlayer embed_dimrHrHrIrRst         zConformerEncoder.forward)rTrUrVrWrrX__annotations__rr,rrrjitignorerRrYrHrHrFrIr`s\  ($jrcs^eZdZdZ        dd ed ed ed ed edededeffdd ZdddZZ S) WindowQformerzWindow-level Qformerrrrr\T window_size num_queriesrrwrrr%normalize_beforec shttfddt|D|_ttd||_ r,tj ddnd|_ ||_ dS)Nc s$g|]}tjdddqS)rT)r#nheaddim_feedforwarddropoutr& batch_first norm_first)rTransformerDecoderLayerrrwrr%rrrHrIrs z*WindowQformer.__init__..rg-q=)eps) r+r,r ModuleListrdecoders Parameterrrqueriesr1 after_normr) r4rrrrwrrr%rrFrrIr,s   zWindowQformer.__init__Nc Cs|dd}|jd|j}|dkrt|d|j|fdd}tj|ddddfd|jfd|jfd}|j\}}}||d|j|}|dd }||||jd}|j ||dd} |j D] } | | |d|d } qb|j durw| | } |dur||j}| ||d} | |fS) zforward decoderrrrrr.N)r;strider)tgtmemorytgt_mask memory_mask) transposerrrrunfoldview contiguousr rr r ) r4 audio_embedrO embed_lenpadding embed_chunkbszrslenqrrQrHrHrIrR s0      zWindowQformer.forward)rrrrrrr\TrS) rTrUrVrWrXrrr,rRrYrHrHrFrIrs8 "rc seZdZdZdeddffdd Zdejddfdd Zd ej ddfd d Z  ddejdej de dejfddZ  ddejdej de dejfddZZS)AudioEmbeddingzImage embedding.configreturnNc st||_t|dr|jn|j}d}d|_t|jt rE|j dddkrE|j dd}|dus4Jt di||_ |d}|d}nt d|dusQJd ||_||_| d d |_| d d |_| dd r~| di}||d<tdi||_nd|_| dd r|jdusJd| di}d|j||dd td d} |r| |dD] } | |vsJdqtdi| |_nd|_| dd} | dkrt|||_ni| dkr7|} d} |js|jrd n|j|_t||j| g}td | D]}|tt| | gqtj||_t||j| g}td | D]}|tt| | gqtj||_ nt d| d|j!|_!d|_"d|_#dS)Nn_embdnamecascadesrrwrtr!z(Remember to set values for audio_dim_outfreeze_audio_processorFdownsample_rater use_qformerqformer_configuse_conv_downsamplez6don't support use qformer and conv downsample togetherr{rbrrcrkrlprojection_clslinearmlprzprojection_cls = z, not implementedrH)$r+r,rhasattrr! hidden_size layer_idxraudio_processordictrrencoderr audio_dim_out audio_dim_inr%r&rqformerrr}r~rconv_dsLinearaudio_projectionlinear_downsample_raterextendGELU Sequentialaudio_projection_for_vision vocab_size input_embedsaudio_embed_sizes)r4rkwargsr.r3encoder_confign_melsr(r{rrr*dim_projectiondepthlayersrrFrHrIr,6s                 zAudioEmbedding.__init__r?cC ||_dSrS)r?)r4r?rHrHrIset_audio_embeds zAudioEmbedding.set_audio_embedsr@cCrGrS)r@)r4r@rHrHrIset_audio_embed_sizesrIz$AudioEmbedding.set_audio_embed_sizesspeechaudio_attention_maskaudio_projection_modec CsB|jr t|||\}}Wdn1swYn|||\}}|jdur6|j|dd\}}|jdurM|durD|d}|j||d\}}|jdkr|\}}} ||j} | dkrqt |ddd|j| fdd}|d}| |||j| |j}|dkr| |} | S|dkr| |} | Std|d ) zl arguments: input_embeds: audio features (B, T, D) B: num audios in a sequence N)rOrrrrKvisionzaudio_projection_mode = z not implemented)r%rno_gradr2r5r6rr9rrrrr8r=r) r4r?rLrMaudio_featuresrrbsrfeat_dimraudio_set_tensorrHrHrIget_audio_featuressJ          z!AudioEmbedding.get_audio_featuresrPcCs|j|||d}|S)z arguments: audio_features: audio features (num_audio_tokens, T, D) returns: audio_embeds: audio embeddings (num_audio_tokens, hidden_dim) )rLrM)rT)r4rPrLrM audio_embedsrHrHrIrRs  zAudioEmbedding.forward)NrK)rTrUrVrWrr,r FloatTensorrH LongTensorrJrstrrTrRrYrHrHrFrIr3s6h 9r)'rrtypingrrnumpyrrtorch.nn.functionalr functionalrr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperr2torch.distributed.fsdp.fully_sharded_data_parallelr transformersrsglang.srt.models.phi4mm_utilsr r r r r rrrrrr_AUDIO_PLACEHOLDER_TOKEN_IDModulerABCrZrrrrHrHrHrIs.   4]X M