o -i2 @sddlZddlmZddlZddlmZddlmZddlmZm Z m Z ddl m Z m Z mZddlmZddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZm Z dd l!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-Gdddej.Z/Gdddej.Z0e-ddGdddeZ1dee2e3ej4fde5e3de2ee2e3ej4fee2e3ej4fffddZ6Gd d!d!e1Z7e-ddGd"d#d#ej.e+Z8d$ej4d%ej4d&e9ddfd'd(Z:dS))N)Iterable)nn) RobertaConfig) ModelConfig PoolerConfig VllmConfig) BOSEOSFilterDispatchPoolerPooler)pooler_for_embed)AllPoolpooler_for_token_classifypooler_for_token_embed)VocabParallelEmbedding)DefaultModelLoader)default_weight_loader)TOKEN_TYPE_SHIFTBertEmbeddingModel BertModel_decode_token_type_ids_encode_token_type_ids)AutoWeightsLoader WeightsMapper maybe_prefix)IntermediateTensors) BertWithRopeJinaRobertaModel)SupportsCrossEncoding)default_pooling_typec sLeZdZdeffdd Z d dejdejdejdBdejfd d ZZS) RobertaEmbeddingconfigcst|j|_t|j|j|_|j|_t j |j |j|jd|_ t |j |j|_t j|j|jd|_|dt|j ddS)N) padding_idx)eps position_idsr)super__init__ hidden_sizesizer vocab_sizeword_embeddings pad_token_idr"r Embeddingmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsregister_buffertorcharange unsqueeze)selfr! __class___/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/model_executor/models/roberta.pyr&0s& zRobertaEmbedding.__init__N input_idsr$ inputs_embedsreturncCsHt|}|dur ||}||}||}|||}||}|SN)rr*r.r0r1)r7r<r$r=token_type_idsr.r0 embeddingsr:r:r;forwardFs     zRobertaEmbedding.forwardr?) __name__ __module__ __qualname__rr&r4TensorrB __classcell__r:r:r8r;r /sr cs8eZdZdZd fdd Zdejdejfdd ZZS) RobertaClassificationHeadz-Head for sentence-level classification tasks. model_configrcsFt|j}|j}tj|j|j|d|_tj|j|j|d|_ dS)Ndtype) r%r& hf_config head_dtyperLinearr'dense num_labelsout_proj)r7rIr!rMr8r:r;r&]s   z"RobertaClassificationHead.__init__xr>cCs"||}t|}||}|Sr?)rOr4tanhrQ)r7rRr:r:r;rBfs   z!RobertaClassificationHead.forward)rIr) rCrDrE__doc__r&r4rFrBrGr:r:r8r;rHZs rHCLS)seq_pooling_typec seZdZdZdddedeffddZ  dd ejd ejd e dBd ejdBd ejf ddZ ddeded e e BfddZ deeeejffddZZS)RobertaEmbeddingModelz?A model that uses Roberta to provide embedding functionalities.prefix vllm_configrZcs tj||d|jjj|_dS)Nr[rZ)r%r&rIrLr+r")r7r[rZr8r:r;r&rszRobertaEmbeddingModel.__init__Nr< positionsintermediate_tensorsr=r>cCs"t|||jd|j||||dS)Nr<r$r"r<r]r=r^)replace_roberta_positionsr"model)r7r<r]r^r=r:r:r;rBvs zRobertaEmbeddingModel.forwardcCsH|jj}t||d}t|dddkrtdi|dtiStdi|S)Nr\position_embedding_typeabsoluteembedding_classr:)rIrLdictgetattrrr r)r7r[rZrLkwargsr:r:r; _build_models  z"RobertaEmbeddingModel._build_modelweightscCsXt|}tdd|D}|rtddid}ntddid}t|dgd}|j||d S) Ncss|] \}}|dVqdS)roberta.N startswith).0name_r:r:r; s z5RobertaEmbeddingModel.load_weights..rkzmodel.)orig_to_new_prefixrXzlm_head.) skip_prefixesmapper)listanyrr load_weights)r7rj weights_listhas_roberta_prefixruloaderr:r:r;rxsz"RobertaEmbeddingModel.load_weights)NN)rX)rCrDrErTrstrr&r4rFrrBrrrirtuplerxrGr:r:r8r;rWns2  $ rW all_weightssecondary_weightsr>cs>t|\}}fddfdd|Dfdd|DfS)NcstfddDS)Nc3|]}|VqdSr?rl)rnfnr:r;rqsz=filter_secondary_weights..filtered..)rwr)rrr;filteredsz*filter_secondary_weights..filteredc3s$|] \}}|r||fVqdSr?r:rnrwrr:r;rqs"z+filter_secondary_weights..c3s$|] \}}|s||fVqdSr?r:rrr:r;rqs   ) itertoolstee)r~r all_weights1 all_weights2r:)rrr;filter_secondary_weightss  rcsbeZdZdZdddedeffddZded efd d Z d e e ee j fffd d ZZS)BgeM3EmbeddingModelzA model that extends RobertaEmbeddingModel with sparse embeddings. This class supports loading an additional sparse_linear.pt file to create sparse embeddings as described in https://arxiv.org/abs/2402.03216 rXrYr[rZcs|jjj|_j}|j|_|jj|_|jj|_tj|dddg|_dd|jD|_ fddt |j |jD|_ dS)Nr\zsparse_linear.zcolbert_linear.cSsg|]}|dqS)ptr:rnrZr:r:r; sz0BgeM3EmbeddingModel.__init__..cs(g|]\}}tjjjd||gdqS)N) model_or_pathrevisionrZallow_patterns_overrides)rSourcerIrb)rnfilenamerZr[r:r;rs) rIrLr'rM bos_token_id eos_token_idr%r&secondary_weight_prefixessecondary_weight_fileszipr)r7r[rZrIr8rr;r&s      zBgeM3EmbeddingModel.__init__ pooler_configr>c Csptj|jd|jd|_tj|j|j|jd|_tt|tt ||j|j tt |t |jt jd|j |jdS)NrrJ)pooling classifieract_fn)embed token_embedtoken_classify)rrNr'rM sparse_linearcolbert_linearr r rrrr r r4relur)r7rr:r:r; _build_poolers,  z!BgeM3EmbeddingModel._build_poolerr~cspt||j\}}t|t|}|D]\}tfdd|jDr5|}t|dt}|||qdS)Nc3rr?rlrror:r;rqs  z3BgeM3EmbeddingModel.load_weights.. weight_loader) rrr%rxrfnamed_parametersrwrgr)r7r~ secondaryrj params_dict loaded_weightparamrr8rr;rxs      z BgeM3EmbeddingModel.load_weights)rCrDrErTrr|r&rr rrr}r4rFrxrGr:r:r8r;rs (rc seZdZdZdZeddddddd d d d Zd ddedeffddZ de e ee j ffddZde j de j fddZ   d!de j dBde j dedBde j dBde j dBde j f dd ZZS)" RobertaForSequenceClassificationadA model that uses Roberta to provide embedding functionalities. This class encapsulates the BertModel and provides an interface for embedding operations and customized pooling functions. Attributes: roberta: An instance of BertModel used for forward operations. _pooler: An instance of Pooler used for pooling operations. Tzembeddings.LayerNormlayerzattention.self.qkv_projzattention.output.densezattention.output.LayerNormzintermediate.densez output.densezoutput.LayerNorm)emb_lnlayersz mixer.Wqkvzmixer.out_projnorm1zmlp.fc1zmlp.fc2norm2)orig_to_new_substrrXrYr[rZcsrt|jj}|jjj|_|j|_t|t|dt d|_ t |j|_ |jj }|dus.Jtj||j d|_dS)Nbert)r[rZre)r)r%r&rIrLr+r"rPrrr robertarHrrr for_seq_clspooler)r7r[rZr!rr8r:r;r&s     z)RobertaForSequenceClassification.__init__rjcCst|}|j||jdS)Nrt)rrxjina_to_vllm_mapper)r7rjr{r:r:r;rx2sz-RobertaForSequenceClassification.load_weightsr<r>cCs |j|Sr?)rembed_input_ids)r7r<r:r:r;r6s z0RobertaForSequenceClassification.embed_input_idsNr]r^r=r@cCsVt|||jd|dur"|jjjdt>ksJ|dusJt|||j||||dS)Nr_rr`)rar"rr!r)rr)r7r<r]r^r=r@r:r:r;rB9s  z(RobertaForSequenceClassification.forward)NNN)rCrDrErTis_pooling_modelrrrr|r&rr}r4rFrxrrrBrGr:r:r8r;rsB  rr<r$r"cCs||d7}dS)Nrr:r_r:r:r;raPs ra);rcollections.abcrr4r transformersr vllm.configrrr!vllm.model_executor.layers.poolerrr r )vllm.model_executor.layers.pooler.seqwiser )vllm.model_executor.layers.pooler.tokwiser r r3vllm.model_executor.layers.vocab_parallel_embeddingr/vllm.model_executor.model_loader.default_loaderr-vllm.model_executor.model_loader.weight_utilsrvllm.model_executor.models.bertrrrrr vllm.model_executor.models.utilsrrr vllm.sequencerbert_with_roperr interfacesrinterfaces_baserModuler rHrWr}r|rFrvrrrintrar:r:r:r;sR          +8& NL