o ir@sdZddlmZmZmZddlZddlmZm Z m Z Gddde j Z Gddde j Z Gd d d e j ZGd d d e j ZGd dde j ZGddde j ZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZdS)z0Declares specification of the Transformer model.)OptionalTupleUnionN)attention_spec common_spec model_specc@sneZdZddejjdejjddddddf dedede de dejd ed ejd e d e d e de de de fddZ dS)TransformerEncoderSpecTF num_layers num_headspre_norm no_final_norm activationnum_source_embeddingsembeddings_mergelayernorm_embeddingrelative_positionrelative_attention_biasffn_glurms_normmulti_query_attentioncs|_td||_||_td||_td||_ddt|D|_ d|_ s6s6t |_ |rA|sAt jd|_|rJt jd|_fddt|D|_dS) aInitializes a Transformer encoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention. int16int8cSsg|]}tqS)rEmbeddingsSpec.0_rr_/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/ctranslate2/specs/transformer_spec.py 5sz3TransformerEncoderSpec.__init__..Trc s&g|]}tr dnddqS)r N)rrrr num_heads_kv)TransformerEncoderLayerSpecrrrrrrrrr?s N)rnpdtypetyper r rrrange embeddingsscale_embeddingsPositionEncoderSpecposition_encodingsr LayerNormSpec layer_normrlayer)selfr r r r rrrrrrrrrrr#r__init__ s$% zTransformerEncoderSpec.__init__N) __name__ __module__ __qualname__r ActivationRELUEmbeddingsMergeCONCATintboolr0rrrrr sN     rcF@seZdZdejjddddddddddddddddddddddddddddddf ded ed ed ejd ed ededededededededededededeededee j de de deded ed!ed"ed#ed$eed%eed&eed'eej d(eed)eefDd*d+Z ed,d-ZdS).TransformerDecoderSpecTFr N'rr r r rrwith_encoder_attentionr project_in_outrralignment_layeralignment_headsrralibialibi_use_positive_positions scale_alibi rotary_dimrotary_interleaverotary_scaling_typerotary_scaling_factor rotary_base original_max_position_embeddingsmax_position_embeddingsparallel_residualshared_layer_normpre_post_layer_normrr!head_dimsliding_window quant_typequant_group_size quant_bitsc##st|_r|s tdrtd|r"dur dkr tddr1dd|fvr1tdtd||_||_td||_td| |_ td| |_ t |_ d |_tj|_||_||_||_durztd |_ss|s durt|_|r|st j d |_|rt j d |_t |_ fd d t|D|_d|_|pƈ|k|jd<|rt |_ t |_!| dur| |jd<|"|jd<|!|jd<dSdS)a Initializes a Transformer decoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. with_encoder_attention: Enable the encoder attention sublayers. no_final_norm: Disable the final layer norm in the pre-norm architecture. project_in_out: Add linear transformations after the embedding layer and before the final layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: Add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. sliding_window: Max sequence length to retain in KV Cache. quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) z/The GPT-J block expects a pre-norm architecturez-The GPT-J block does not have cross attentionNr z5Enabling multi_query_attention implies num_heads_kv=1z>num_heads_kv=%d is not supported in the cross-attention layersrrTint32r csg|]=}tdiddddd d d dd d d d d d ddddqS)r=rrrrrDrErFrGrHrIrJrKrLrMr!rNrOr)TransformerDecoderLayerSpecrrrNrJr!rIrKrMrrrrHrDrErGrFrLrOr=rrrsP     z3TransformerDecoderSpec.__init__..Frquantization_typequantization_bitsquantization_group_size)"dict_config ValueErrorr$r%r&r r rr?r@rrr(r)rOPTIONAL scale_outputsrArBrCrOr*r+r,r-r LinearSpec projectionr'r.start_from_zero_embedding project_in project_out)#r/r r r rrr=r r>rrr?r@rrrArBrCrDrErFrGrHrIrJrKrLrMrr!rNrOrPrQrRrrUrr0LsvT  ,    zTransformerDecoderSpec.__init__cCs|jSN)rZr/rrrconfigszTransformerDecoderSpec.config)r1r2r3rr4r5r8r9rrRotaryScalingTypefloat Quantizationr0propertyrerrrrr:Ks      !"# 'r:c@s"eZdZ      dddZdS)r"FNcCs*tjd|||||d|_t||d|_dS)NT)self_attentionrrrr!rOglur)rMultiHeadAttentionSpecrjFeedForwardSpecffn)r/rrrrr!rOrrrr0s z$TransformerEncoderLayerSpec.__init__)FFFFNNr1r2r3r0rrrrr"sr"c@s:eZdZ                  d ddZdS) rTTFNr r<rcCstjd||||||| | | | |||d|_|rtj|||d|_t||d|_| rF|r0t|_n t|_ t|_ t |jdt |jd|rrtj|d|_ tj|d|_ tj|d|_ tj|d|_ t |jdt |jddSdS)NT)rjrrrrDrErFrGrHrIrJr!rNrO)rr!rOrkr-r )rrmrj attentionrnrorr,rLinput_layer_normpost_attention_layer_normdelattrpre_feedforward_layer_normpost_feedforward_layer_norm)r/r=rrrrrDrErFrGrHrIrJrKrLrMr!rNrOrrrr0 sX      z$TransformerDecoderLayerSpec.__init__)TFFFFNTNr r<rrFFFNNNrprrrrrT s(rTc@seZdZdddZdS)rnFcCs8tj|d|_t|_t|_|rt|_dSdS)Nr )rr,r-r^linear_0linear_1linear_0_noact)r/rlrrrrr0Ws   zFeedForwardSpec.__init__N)FFrprrrrrnVsrnc@seZdZddZdS)r*cCs tj|_dSrc)rr\ encodingsrdrrrr0`s zPositionEncoderSpec.__init__Nrprrrrr*_s r*c,eZdZdZddeeffdd ZZS)TransformerConfigz%Configuration for Transformer models.Nlayer_norm_epsilonc tjdd|i|dS)zInitializes the configuration for Transformer models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. r}Nrsuperr0r/r}kwargs __class__rrr0gzTransformerConfig.__init__rcr1r2r3__doc__rrgr0 __classcell__rrrrr|d r|c seZdZdZdedeffdd Zedddej j dd d ej j dddddf d e eeeeffd ed ed ededej dedededej dededededefddZeddZeddZdd Zd!d"Zd#d$ZZS)%TransformerSpeczDescribes a Transformer model. The specification is invariant to hidden dimensions but requires to explicitly set the number of layers and attention heads. encoderdecodercsPt|ts tdt|tstdt||_||_|j d|jj dS)zInitializes a Transformer model specification. Args: encoder: The encoder specification. decoder: The decoder specification. 1encoder argument must be a TransformerEncoderSpec1decoder argument must be a TransformerDecoderSpecrN) isinstancer TypeErrorr:rr0rrrZ add_attributer)r/rrrrrr0xs   zTransformerSpec.__init__FTr;r r r with_relative_positionr r rr?r@rrrrrrrcCspt|ttfr |\}}n||}}t|||||| | | || | ||d }t|||||| || ||| ||d }|||S)aCreates a Transformer model specification. Args: num_layers: Number of encoder and decoder layers, or a 2-tuple if the number is different. num_heads: Number of attention heads. with_relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layer as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention. ) r r rrrrrrrrr) r r rrrrr?r@rrr)rlisttuplerr:)clsr r rr r rr?r@rrrrrrrnum_encoder_layersnum_decoder_layersrrrrr from_configsD+   zTransformerSpec.from_configcCdS)NrrrdrrrnamezTransformerSpec.namecCr)NrrdrrrrevisionrzTransformerSpec.revisioncCtSrc)r|rdrrrget_default_configz"TransformerSpec.get_default_configcCsdd|jjDS)NcSsg|]}|jjdqS)r)weightshape)rspecrrrrsz>TransformerSpec.get_source_vocabulary_size..)rr(rdrrrget_source_vocabulary_sizesz*TransformerSpec.get_source_vocabulary_sizecC|jjjjdSNrrr(rrrdrrrget_target_vocabulary_sizez*TransformerSpec.get_target_vocabulary_size)r1r2r3rrr:r0 classmethodrr4r5r6r7rr8rr9rrirrrrrrrrrrrqsv      Q  rcr{)TransformerDecoderModelConfigz-Configuration for Transformer decoder models.Nr}c r~)zInitializes the configuration for Transformer decoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. r}Nrrrrrrr0rz&TransformerDecoderModelConfig.__init__rcrrrrrrrrc?s8eZdZdZdeffdd Zedejj dddddddddddddd d d ddddddddddfd e d e d e dejde de de de de de de de de de e de de e jdedede de de d e d!e d"e d#e e d$e e d%e e d&e ejd'e e d(e e frrrrArBrCrDrErFrGrHrIrJrKrLrMrr!rNrOrPrQrRc Cst||fid|d|d|ddd|d|d|d | d | d | d | d | d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|}||S)a! Creates a Transformer decoder model specification. Args: num_layers: Number of decoder layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. no_final_norm: Do not apply layer normalization after the last decoder block. project_in_out: Add a linear layer after the embedding layer and another one before the final output projection. with_relative_position: Enable relative position representations modules. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of head sliding_window: max sequence length to retain KV cache quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) r rrr=Fr r>rrrrArBrCrDrErFrGrHrIrJrKrLrMrr!rNrOrPrQrR)r:) rr r r rrr r>rrrrArBrCrDrErFrGrHrIrJrKrLrMrr!rNrOrPrQrRrrrrrs~K     "z'TransformerDecoderModelSpec.from_configcCr)Nr:rrdrrrr~rz TransformerDecoderModelSpec.namecCr)Nrrdrrrrrz$TransformerDecoderModelSpec.revisioncCrrc)rrdrrrrrz.TransformerDecoderModelSpec.get_default_configcCrrrrdrrrget_vocabulary_sizerz/TransformerDecoderModelSpec.get_vocabulary_size)r1r2r3rr:r0rrr4r5r8r9rrrfrgrhrrirrrrrrrrrrs      n  rcr{)TransformerEncoderModelConfigz-Configuration for Transformer encoder models.Nr}c r~)zInitializes the configuration for Transformer encoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. r}Nrrrrrrr0rz&TransformerEncoderModelConfig.__init__rcrrrrrrrrcsbeZdZdZdejjfdededejffdd Z e dd Z e d d Z d d Z ddZZS)TransformerEncoderModelSpecz2Describes a Transformer encoder model (e.g. BERT).Fr pooling_layerpooling_activationcs\t|ts tdt||_|jd|jj|r,t |_ t d||_dSdS)zInitializes a Transformer encoder model specification. Args: encoder: The encoder specification. pooling_layer: Add the pooling layer. pooling_activation: The activation to apply after the pooling layer. rrrN)rrrrr0rrZrrrr^ pooler_denser$r%r&pooler_activation)r/rrrrrrr0s   z$TransformerEncoderModelSpec.__init__cCr)Nrrrdrrrrrz TransformerEncoderModelSpec.namecCr)Nr rrdrrrrrz$TransformerEncoderModelSpec.revisioncCrrc)rrdrrrrrz.TransformerEncoderModelSpec.get_default_configcCs|jjdjjdSr)rr(rrrdrrrrsz/TransformerEncoderModelSpec.get_vocabulary_size)r1r2r3rrr4Tanhrr9r0rirrrrrrrrrrs"  r)rtypingrrrnumpyr$ctranslate2.specsrrr LayerSpecrr:r"rTrnr*SequenceToSequenceModelConfigr|SequenceToSequenceModelSpecrLanguageModelConfigrLanguageModelSpecrrrrrrrs&A-J