o Ti@sdZddlmZmZmZddlZddlmZm Z m Z Gddde j Z Gddde j Z Gd d d e j ZGd d d e j ZGd dde j ZGddde j ZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZdS)z0Declares specification of the Transformer model.)OptionalTupleUnionN)attention_spec common_spec model_specc/@seZdZddejjdejjddddddddddddddddfdedede d e d ejd ed ejd e de de de de de de ede ede ede de e j de de de ede e de f.ddZdS)TransformerEncoderSpecTFN' num_layers num_headspre_norm no_final_norm activationnum_source_embeddingsembeddings_mergelayernorm_embeddingrelative_positionrelative_attention_biasffn_glurms_normmulti_query_attention num_heads_kvhead_dim rotary_dimrotary_interleaverotary_scaling_typerotary_scaling_factor rotary_basesliding_windowqk_normpre_post_layer_normc s| rdurdkrtdd| |_td||_||_td||_td||_ddt |D|_ d|_ sFsFt |_ |rQ|sQtjd |_|rZtjd |_|durgtd ||_ f d dt |D|_dS) a'Initializes a Transformer encoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of dimensions per attention head. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. sliding_window: Max sequence length to retain in KV Cache. qk_norm: Apply layer normalization to the query and key projections. pre_post_layer_norm: Add post layer norm for each pre norm layer. Nr 5Enabling multi_query_attention implies num_heads_kv=1int16int8cSsg|]}tqS)rEmbeddingsSpec.0_r%r%V/home/ubuntu/.local/lib/python3.10/site-packages/ctranslate2/specs/transformer_spec.py Ssz3TransformerEncoderSpec.__init__..Trint32cs.g|]}t d qS)) rrrrrrrrrrrr r!)TransformerEncoderLayerSpecr' rrrr!r rrrrrrrrr%r*r+`s$) ValueErrorrnpdtypetyper r rrrange embeddingsscale_embeddingsPositionEncoderSpecposition_encodingsr LayerNormSpec layer_normrrlayer)selfr r r rrrrrrrrrrrrrrrrrrr r!r%r/r*__init__ s4<" zTransformerEncoderSpec.__init__)__name__ __module__ __qualname__r ActivationRELUEmbeddingsMergeCONCATintboolrrRotaryScalingTypefloatr=r%r%r%r*r s     rcJ@s&eZdZdejjddddddddddddddddddddddddddddddddf"ded ed ed ejd ed ededededededededededededeededee j de de deded ed!ed"ed#ed$eed%eed&eed'eej d(eed)eed*ed+eefHd,d-Z ed.d/ZdS)0TransformerDecoderSpecTFr Nr rr r r rrwith_encoder_attentionrproject_in_outrralignment_layeralignment_headsrralibialibi_use_positive_positions scale_alibirrrrr original_max_position_embeddingsmax_position_embeddingsparallel_residualshared_layer_normr!rrrr quant_typequant_group_size quant_bitsr external_pre_post_encoder_layersc%%st|_r|s tdrtd|r"dur dkr tddtd||_||_td||_td| |_ td| |_ t |_ d|_tj|_||_||_||_durktd |_ sy sy|sy duryt|_|r|st j d |_|rt j d |_t |_ fd d t|D|_d |_|p|k|jd<|rt |_ t |_!| r| |jd<|"|jd<|!|jd<dSdS)a. Initializes a Transformer decoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. with_encoder_attention: Enable the encoder attention sublayers. no_final_norm: Disable the final layer norm in the pre-norm architecture. project_in_out: Add linear transformations after the embedding layer and before the final layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: Add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. sliding_window: Max sequence length to retain in KV Cache. quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) external_pre_post_encoder_layers: if the encoder attention pre and processing is done outside the attention. z/The GPT-J block expects a pre-norm architecturez-The GPT-J block does not have cross attentionNr r"r#r$Tr-r,csg|]C}tdidd d dd d dddd d d d d ddddddqS)rKrrrrrrrrrrRrSrTrUr!rrrr rYr%)TransformerDecoderLayerSpecr'rYrrrSrrRrTr!r rrrrrrrrrUrrKr%r*r+sX     z3TransformerDecoderSpec.__init__..Frquantization_typequantization_bitsquantization_group_size)"dict_configr0r1r2r3r r rrMrNrr&r5r6rOPTIONAL scale_outputsrOrPrQrr7r8r9r:r LinearSpec projectionr4r;start_from_zero_embedding project_in project_out)%r<r r r rrrKrrLrrrMrNrrrOrPrQrrrrrrRrSrTrUr!rrrrrVrWrXr rYr%r[r*r=usjX  0    zTransformerDecoderSpec.__init__cCs|jSN)r`r<r%r%r*configszTransformerDecoderSpec.config)r>r?r@rrArBrErFrrrGrH Quantizationr=propertyrjr%r%r%r*rIts      !"#$% 'rIc @sVeZdZ              ddeededeejd ed ed ef d d Z dS)r.FNTr r rrrrrr!cCstjd|||||||| | | | | d |_t||d|_|rFtj|d|_tj|d|_tj|d|_ tj|d|_ t |jdt |jddSdS)NT) self_attentionrrrrrrrrrrrr glurr,r:) rMultiHeadAttentionSpecrmFeedForwardSpecffnrr9input_layer_normpost_attention_layer_normpre_feedforward_layer_normpost_feedforward_layer_normdelattr)r<rrrrrrrrrrrrr r!r%r%r*r=!s< z$TransformerEncoderLayerSpec.__init__)FFFFNNNNTNr r FF) r>r?r@rrErFrrGrHr=r%r%r%r*r. s8     r.c@s>eZdZ                    d ddZdS) rZTFNr r rcCstjd||||||| | | | ||||d|_|r%tj||||||dud|_t||d|_| rL|r6t|_n t|_ t|_ t |jdt |jd|rtj|d|_ tj|d|_ |rn|rntj|d|_ tj|d|_ tj|d|_tj|d|_t |jdt |jddSdS)NT)rmrrrrrrrrrRrSrrrr F)rrrrr has_normrnr:r,)rrprm attentionrqrrrr9rUrsrtrw*external_post_encoder_attention_layer_norm)external_pre_encoder_attention_layer_normrurv)r<rKrrrrrrrrrrRrSrTrUr!rrrr rYr%r%r*r=Tsj         z$TransformerDecoderLayerSpec.__init__)TFFFFNTNr r rrFFFNNNFFr>r?r@r=r%r%r%r*rZSs,rZc@seZdZdddZdS)rqFcCs8tj|d|_t|_t|_|rt|_dSdS)Nr,)rr9r:rclinear_0linear_1linear_0_noact)r<rorr%r%r*r=s   zFeedForwardSpec.__init__N)FFr|r%r%r%r*rqsrqc@seZdZddZdS)r7cCs tj|_dSrh)rra encodingsrir%r%r*r=s zPositionEncoderSpec.__init__Nr|r%r%r%r*r7s r7c,eZdZdZddeeffdd ZZS)TransformerConfigz%Configuration for Transformer models.Nlayer_norm_epsilonc tjdd|i|dS)zInitializes the configuration for Transformer models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr%superr=r<rkwargs __class__r%r*r=zTransformerConfig.__init__rhr>r?r@__doc__rrHr= __classcell__r%r%rr*r rc seZdZdZdedeffdd Zedddej j dd d ej j dddddf d e eeeeffd ed ed ededej dedededej dededededefddZeddZeddZdd Zd!d"Zd#d$ZZS)%TransformerSpeczDescribes a Transformer model. The specification is invariant to hidden dimensions but requires to explicitly set the number of layers and attention heads. encoderdecodercsPt|ts tdt|tstdt||_||_|j d|jj dS)zInitializes a Transformer model specification. Args: encoder: The encoder specification. decoder: The decoder specification. 1encoder argument must be a TransformerEncoderSpec1decoder argument must be a TransformerDecoderSpecrN) isinstancer TypeErrorrIrr=rrr` add_attributer)r<rrrr%r*r=s   zTransformerSpec.__init__FTrJr r r with_relative_positionr rrrMrNrrrrrrrcCspt|ttfr |\}}n||}}t|||||| | | || | ||d }t|||||| || ||| ||d }|||S)aCreates a Transformer model specification. Args: num_layers: Number of encoder and decoder layers, or a 2-tuple if the number is different. num_heads: Number of attention heads. with_relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layer as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention. ) r rrrrrrrrrr) r rrrrrrMrNrrr)rlisttuplerrI)clsr r rr rrrMrNrrrrrrrnum_encoder_layersnum_decoder_layersrrr%r%r* from_configsD+   zTransformerSpec.from_configcCdS)Nrr%rir%r%r*name7zTransformerSpec.namecCr)Nr%rir%r%r*revision;rzTransformerSpec.revisioncCtSrh)rrir%r%r*get_default_config?z"TransformerSpec.get_default_configcCsdd|jjDS)NcSsg|]}|jjdqS)r)weightshape)r(specr%r%r*r+Csz>TransformerSpec.get_source_vocabulary_size..)rr5rir%r%r*get_source_vocabulary_sizeBsz*TransformerSpec.get_source_vocabulary_sizecC|jjjjdSNrrr5rrrir%r%r*get_target_vocabulary_sizeEz*TransformerSpec.get_target_vocabulary_size)r>r?r@rrrIr= classmethodrrArBrCrDrrErrFrrlrrrrrrr%r%rr*rsv      Q  rcr)TransformerDecoderModelConfigz-Configuration for Transformer decoder models.Nrc r)zInitializes the configuration for Transformer decoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr%rrrr%r*r=Lrz&TransformerDecoderModelConfig.__init__rhrr%r%rr*rIrrc@s>eZdZdZdeffdd Zedejj dddddddddddddd d d dddddddddddfd e d e d e dejde de de de de de de de de de e de de e jdedede de de d e d!e d"e d#e e d$e e d%e e d&e ejd'e e d(e e d)e f>d*d+Zed,d-Zed.d/Zd0d1Zd2d3ZZS)4TransformerDecoderModelSpecz3Describes a Transformer decoder model (e.g. GPT-2).rcsJt|ts tdt||_|jjD] \}}|j ||qdS)z|Initializes a Transformer decoder model specification. Args: decoder: The decoder specification. rN) rrIrrr=rrjitemsr`r)r<rkeyvaluerr%r*r=Ys  z$TransformerDecoderModelSpec.__init__TFNr r rr r r rrrrLrrrrOrPrQrrrrrrRrSrTrUr!rrrrrVrWrXr c !Cst||fid|d|d|ddd|d|d|d | d | d | d | d | d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|} || S) a! Creates a Transformer decoder model specification. Args: num_layers: Number of decoder layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. no_final_norm: Do not apply layer normalization after the last decoder block. project_in_out: Add a linear layer after the embedding layer and another one before the final output projection. with_relative_position: Enable relative position representations modules. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of head sliding_window: max sequence length to retain KV cache quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) r rrrKFrrLrrrrOrPrQrrrrrrRrSrTrUr!rrrrrVrWrXr )rI)!rr r r rrrrLrrrrOrPrQrrrrrrRrSrTrUr!rrrrrVrWrXr rr%r%r*rgsL      #z'TransformerDecoderModelSpec.from_configcCr)NrIr%rir%r%r*rrz TransformerDecoderModelSpec.namecCr)Nr%rir%r%r*rrz$TransformerDecoderModelSpec.revisioncCrrh)rrir%r%r*rrz.TransformerDecoderModelSpec.get_default_configcCrrrrir%r%r*get_vocabulary_sizerz/TransformerDecoderModelSpec.get_vocabulary_size)r>r?r@rrIr=rrrArBrErFrrrGrHrkrrlrrrrrr%r%rr*rVs      p  rcr)TransformerEncoderModelConfigz-Configuration for Transformer encoder models.Nrc r)zInitializes the configuration for Transformer encoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr%rrrr%r*r=rz&TransformerEncoderModelConfig.__init__rhrr%r%rr*rrrcsbeZdZdZdejjfdededejffdd Z e dd Z e d d Z d d Z ddZZS)TransformerEncoderModelSpecz2Describes a Transformer encoder model (e.g. BERT).Fr pooling_layerpooling_activationcs\t|ts tdt||_|jd|jj|r,t |_ t d||_dSdS)zInitializes a Transformer encoder model specification. Args: encoder: The encoder specification. pooling_layer: Add the pooling layer. pooling_activation: The activation to apply after the pooling layer. rrr$N)rrrrr=rr`rrrrc pooler_denser1r2r3pooler_activation)r<rrrrr%r*r=s   z$TransformerEncoderModelSpec.__init__cCr)Nrr%rir%r%r*rrz TransformerEncoderModelSpec.namecCr)Nr r%rir%r%r*rrz$TransformerEncoderModelSpec.revisioncCrrh)rrir%r%r*rrz.TransformerEncoderModelSpec.get_default_configcCs|jjdjjdSr)rr5rrrir%r%r*rsz/TransformerEncoderModelSpec.get_vocabulary_size)r>r?r@rrrATanhrrFr=rlrrrrrr%r%rr*rs"  r)rtypingrrrnumpyr1ctranslate2.specsrrr LayerSpecrrIr.rZrqr7SequenceToSequenceModelConfigrSequenceToSequenceModelSpecrLanguageModelConfigrLanguageModelSpecrrrr%r%r%r*s&j-3[