o i k/@sddlmZddlmZddlmZmZmZmZm Z m Z m Z ddl Z ddl mZmZddlmZmZddlmZmZddlmZdd lmZdd lmZmZdd lmZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m"Z"ddl+m,Z,ddl-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4m5Z5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEddlFmFZFdddddddd d d!dd!ddd"d"e dd#d$eGd%ee>ged$eGd%ee>geged$eGd%ee>ged$eGd%ee>geged&e2d8eeBd9e eCeEff dKdLZQd%ee>ged&e2d9eCfdMdNZRdOeCdPeAdQeJd9e eDe@ffdRdSZS dfdOeCdTeeAd9e"fdUdVZTd"e dWdXed+eGd6eJd7eKd9e>f dYdZZUd/eGd3eGd9e"fd[d\ZVdOe"dTe=dQeJd]eGd3eGd9e e.eegee0fff d^d_ZWdOe"d`e e=efdQeJd9e e?eeee/ge.fffdadbZXdced9ee>eeeeeege>ffdddeZYdS)g)partial)Path)AnyCallableListOptionalTupleUnioncastN) AlbertConfig AlbertEncoder) BertConfig BertEncoder)CuratedEncoderTCuratedTransformer)$convert_pretrained_model_for_encoder)PyTorchTransformerOutput) RobertaConfigRobertaEncoder)Doc)SimpleFrozenDict)ModelPyTorchWrapper_v2TorchScriptWrapper_v1get_torch_default_devicetorch2xpxp2torch)chain)r)PyTorchGradScaler) ArgsKwargsFloats2dInts1d)Tok2PiecesModelT)WrappedTransformerAndListenerreplace_listener_callbackreplace_listener_cfg_callback)TransformerModelOutput)remove_bos_eos) SpanExtractorModelTTorchTransformerInTTorchTransformerModelTTorchTransformerOutTTransformerBackpropTTransformerInTTransformerListenerModelTTransformerModelTTransformerOutT#WrappedTransformerAndListenerModelT)with_non_ws_tokensggelu_newii g-q=i F)attention_probs_dropout_probembedding_width hidden_acthidden_dropout_prob hidden_widthintermediate_widthlayer_norm_epsmax_position_embeddingsmodel_max_lengthnum_attention_headsnum_hidden_groupsnum_hidden_layers padding_idxtype_vocab_size torchscriptmixed_precisiongrad_scaler_configwrapped_listener vocab_size with_spans piece_encoderr8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIreturncCs`t|||| | ||||||| | | |d}|rt| |d}n t|}t||||d}t||||dS)aConstruct an ALBERT transformer model. vocab_size (int): Vocabulary size. with_spans (Callable): Callback that constructs a span generator model. piece_encoder (Model) The piece encoder to segment input tokens. attention_probs_dropout_prob (float): Dropout probabilty of the self-attention layers. embedding_width (int): Width of the embedding representations. hidden_act (str): Activation used by the point-wise feed-forward layers. hidden_dropout_prob (float): Dropout probabilty of the point-wise feed-forward and embedding layers. hidden_width (int): Width of the final representations. intermediate_width (int): Width of the intermediate projection layer in the point-wise feed-forward layer. layer_norm_eps (float): Epsilon for layer normalization. max_position_embeddings (int): Maximum length of position embeddings. model_max_length (int): Maximum length of model inputs. num_attention_heads (int): Number of self-attention heads. num_hidden_groups (int): Number of layer groups whose constituents share parameters. num_hidden_layers (int): Number of hidden layers. padding_idx (int): Index of the padding meta-token. type_vocab_size (int): Type vocabulary size. torchscript (bool): Set to `True` when loading TorchScript models, `False` otherwise. mixed_precision (bool): Use mixed-precision training. grad_scaler_config (dict): Configuration passed to the PyTorch gradient scaler. wrapped_listener (Optional[TransformerListenerModelT]): Optional listener to wrap. Only used when replacing listeners in downstream components. )r9r<r=rArBrCr8r;r:rJrEr?r@r>rDr@rDr<rGrHrKrL transformerrI)r _torchscript_encoderr _pytorch_encoder(build_transformer_or_listener_wrapper_v1)rJrKrLr8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIconfigrQencoderrWc/home/ubuntu/.local/lib/python3.10/site-packages/spacy_curated_transformers/models/architectures.py!build_albert_transformer_model_v15sDKrYg?gelu)r8r:r;r<r=r>r?r@rArCrDrErFrGrHrIcC^t|||| | |||||| | || d}|rt| | d}n t|}t||||d}t||||dS)ajConstruct a BERT transformer model. vocab_size (int): Vocabulary size. with_spans (Callable): Callback that constructs a span generator model. piece_encoder (Model) The piece encoder to segment input tokens. attention_probs_dropout_prob (float): Dropout probabilty of the self-attention layers. hidden_act (str): Activation used by the point-wise feed-forward layers. hidden_dropout_prob (float): Dropout probabilty of the point-wise feed-forward and embedding layers. hidden_width (int): Width of the final representations. intermediate_width (int): Width of the intermediate projection layer in the point-wise feed-forward layer. layer_norm_eps (float): Epsilon for layer normalization. max_position_embeddings (int): Maximum length of position embeddings. model_max_length (int): Maximum length of model inputs. num_attention_heads (int): Number of self-attention heads. num_hidden_layers (int): Number of hidden layers. padding_idx (int): Index of the padding meta-token. type_vocab_size (int): Type vocabulary size. torchscript (bool): Set to `True` when loading TorchScript models, `False` otherwise. mixed_precision (bool): Use mixed-precision training. grad_scaler_config (dict): Configuration passed to the PyTorch gradient scaler. wrapped_listener (Optional[TransformerListenerModelT]): Optional listener to wrap. Only used when replacing listeners in downstream components. r9r<r=rArCr8r;r:rJrEr?r@r>rDrNrOrP)r rRrrSrTrJrKrLr8r:r;r<r=r>r?r@rArCrDrErFrGrHrIrUrQrVrWrWrXbuild_bert_transformer_model_v1BEr^gh㈵>i)r8r:r;r<r=r>r?r@rArCrDrErGrFrHrIcCs^t|||| | |||||| | || d}|rt| | d}n t|}t||||d}t||||dS)aoConstruct a CamemBERT transformer model. vocab_size (int): Vocabulary size. with_spans (Callable): Callback that constructs a span generator model. piece_encoder (Model) The piece encoder to segment input tokens. attention_probs_dropout_prob (float): Dropout probabilty of the self-attention layers. hidden_act (str): Activation used by the point-wise feed-forward layers. hidden_dropout_prob (float): Dropout probabilty of the point-wise feed-forward and embedding layers. hidden_width (int): Width of the final representations. intermediate_width (int): Width of the intermediate projection layer in the point-wise feed-forward layer. layer_norm_eps (float): Epsilon for layer normalization. max_position_embeddings (int): Maximum length of position embeddings. model_max_length (int): Maximum length of model inputs. num_attention_heads (int): Number of self-attention heads. num_hidden_layers (int): Number of hidden layers. padding_idx (int): Index of the padding meta-token. type_vocab_size (int): Type vocabulary size. torchscript (bool): Set to `True` when loading TorchScript models, `False` otherwise. mixed_precision (bool): Use mixed-precision training. grad_scaler_config (dict): Configuration passed to the PyTorch gradient scaler. wrapped_listener (Optional[TransformerListenerModelT]): Optional listener to wrap. Only used when replacing listeners in downstream components. r\rNrOrPrrRrrSrT)rJrKrLr8r:r;r<r=r>r?r@rArCrDrErGrFrHrIrUrQrVrWrWrX$build_camembert_transformer_model_v1r_racCr[)amConstruct a RoBERTa transformer model. vocab_size (int): Vocabulary size. with_spans (Callable): Callback that constructs a span generator model. piece_encoder (Model) The piece encoder to segment input tokens. attention_probs_dropout_prob (float): Dropout probabilty of the self-attention layers. hidden_act (str): Activation used by the point-wise feed-forward layers. hidden_dropout_prob (float): Dropout probabilty of the point-wise feed-forward and embedding layers. hidden_width (int): Width of the final representations. intermediate_width (int): Width of the intermediate projection layer in the point-wise feed-forward layer. layer_norm_eps (float): Epsilon for layer normalization. max_position_embeddings (int): Maximum length of position embeddings. model_max_length (int): Maximum length of model inputs. num_attention_heads (int): Number of self-attention heads. num_hidden_layers (int): Number of hidden layers. padding_idx (int): Index of the padding meta-token. type_vocab_size (int): Type vocabulary size. torchscript (bool): Set to `True` when loading TorchScript models, `False` otherwise. mixed_precision (bool): Use mixed-precision training. grad_scaler_config (dict): Configuration passed to the PyTorch gradient scaler. wrapped_listener (Optional[TransformerListenerModelT]): Optional listener to wrap. Only used when replacing listeners in downstream components. r\rNrOrPr`r]rWrWrX"build_roberta_transformer_model_v1}r_rbcCr[)aqConstruct a XLM-RoBERTa transformer model. vocab_size (int): Vocabulary size. with_spans (Callable): Callback that constructs a span generator model. piece_encoder (Model) The piece encoder to segment input tokens. attention_probs_dropout_prob (float): Dropout probabilty of the self-attention layers. hidden_act (str): Activation used by the point-wise feed-forward layers. hidden_dropout_prob (float): Dropout probabilty of the point-wise feed-forward and embedding layers. hidden_width (int): Width of the final representations. intermediate_width (int): Width of the intermediate projection layer in the point-wise feed-forward layer. layer_norm_eps (float): Epsilon for layer normalization. max_position_embeddings (int): Maximum length of position embeddings. model_max_length (int): Maximum length of model inputs. num_attention_heads (int): Number of self-attention heads. num_hidden_layers (int): Number of hidden layers. padding_idx (int): Index of the padding meta-token. type_vocab_size (int): Type vocabulary size. torchscript (bool): Set to `True` when loading TorchScript models, `False` otherwise. mixed_precision (bool): Use mixed-precision training. grad_scaler_config (dict): Configuration passed to the PyTorch gradient scaler. wrapped_listener (Optional[TransformerListenerModelT]): Optional listener to wrap. Only used when replacing listeners in downstream components. r\rNrOrPr`r]rWrWrXbuild_xlmr_transformer_model_v1r_rcrQcCs$t|||d}|durt||S|S)N)rKrQrL)build_transformer_model_v1r%)rKrQrLrIthinc_transformerrWrWrXrTSs   rTc CsHtt|||tg}||d}tdtt||ttdd|didS)N)rLrQtransformer_model)replace_listenerreplace_listener_cfgnO)initlayersrefsattrsdims) r4rr)rtransformer_model_forwardtransformer_model_initr&r'get_dim)rKrQrLrkrlrWrWrXrdfs$  rdmodeldocsis_traincs*|jd||d\}fdd}||fS)Nr)rtcs |gSNrW)dYbackprop_layerrWrXbackpropsz+transformer_model_forward..backprop)rk)rrrsrtYryrWrwrXros roXcCs|jd|||S)Nr)rk initialize)rrr{rzrWrWrXrpsrp)rGrHrVc Csft|tri}d|vr||d<tt|tt|j|jdt|t di|d}| d|d|j d<|S)Nenabledmax_model_seq_lenrD)convert_inputsconvert_outputsrG grad_scalerriT_all_layer_outputsrW) isinstancerrrr_convert_inputs max_seq_lenrD_convert_outputsrset_dimrm)rVr<rGrHrrrWrWrXrSs$   rScCsttt||dtdS)Nr~)rr)rrrrrNrWrWrXrRsrRrc s|jtddD}||krtd|d|djt|f|}ttD]}|}|jd} |||d| f<q,t|}dt ffdd } t |fid } | | fS) Ncss|]}|jVqdSru)size.0xrWrWrX sz"_convert_inputs..zAAt least one sequence in the transformer's input has a length of z>, which is larger than the model's maximum sequence length of z tokensrd_inputscsfddDS)Ncsg|] }|jdqSr)alloc1fshaperopsrWrX szH_convert_inputs..convert_from_torch_backward..rW)rr{rrWrXconvert_from_torch_backwardsz4_convert_inputs..convert_from_torch_backwardargskwargs) rmax ValueErrorxpfulllenrangerrrr) rrr{rtrrDrXtispanspan_lenroutputrWrrXrs& rinputs_outputsc s|\}|j|jd}dd|D}|r!fddt|Dn fddt|DfddD}t|| d}dtttffd d }||fS) NrcSsg|]}|jdqSr)rrrWrWrXrsz$_convert_outputs..cs&g|]\fddjDqS)cs"g|] }|dddfqSrurWrrrrrWrXrs"/_convert_outputs... all_outputs)r model_outputsrrXrscs.g|]\}}jd|d|ddfgqS)Nr)rrrrrWrXrscsg|] }fdd|DqS)csg|] }ttt|dqS)r)r r r)rlayerrrWrXrrrWrrrWrXrs)outputslast_layer_onlyrvcsBddD}dd|D}t|t|ksJt|fd|idS)NcSsg|] }|D]}|qqSrWrWrinneryrWrWrXrszH_convert_outputs..convert_for_torch_backward..cSsg|] }|D]}t|qqSrW)rrrWrWrXr r grad_tensorsr)rr)rvYt_flatdYt_flat)YtrWrXconvert_for_torch_backwardsz4_convert_outputs..convert_for_torch_backward)rrm enumerater(rr ) rrrrt model_inputsall_layer_outputs input_lensrzrrrW)rrrrXrs"     rpathcsdfdd }|S)zConstruct a callback that initializes a supported transformer model with weights from a PyTorch checkpoint. path (Path): Path to the PyTorch checkpoint. NcsB|jdj}t}tj|d}t||}|||||S)Nr) map_location)shims_modelrtorchloadrload_state_dictto)rrr{rzrVdeviceparamsrrWrXrs    z0build_pytorch_checkpoint_loader_v1..loadNNrW)rrrWrrX"build_pytorch_checkpoint_loader_v1s  rr)Z functoolsrpathlibrtypingrrrrrr r r"curated_transformers.models.albertr r curated_transformers.models.bertr r/curated_transformers.models.curated_transformerrr#curated_transformers.models.hf_utilr"curated_transformers.models.outputr#curated_transformers.models.robertarr spacy.tokensr spacy.utilr thinc.apirrrrrr thinc.layersr thinc.modelthinc.shims.pytorch_grad_scalerr thinc.typesrr r!tokenization.typesr# listenersr%r&r'rr(remove_eos_bosr)typesr*r+r,r-r.r/r0r1r2r3r4intfloatstrbooldictrYr^rarbrcrTrdrorprSrRrrrrWrWrWrXsD $           0         z       s       s       s       k  "     !  !  )