o eiX@s dZddlZddlmZmZmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZmZddlmZd d lmZmZmZmZmZmZmZmZGd d d eZGd ddeZGdddeZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%gdZ&dS)zPyTorch CamemBERT model.N)BCEWithLogitsLossCrossEntropyLossMSELoss),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)Unpack)TransformersKwargsauto_docstring)can_return_tuple)RobertaForCausalLMRobertaForMaskedLMRobertaForMultipleChoiceRobertaForQuestionAnswering RobertaForSequenceClassificationRobertaForTokenClassification RobertaModelRobertaPreTrainedModelc@seZdZdZdS)CamembertPreTrainedModelrobertaN)__name__ __module__ __qualname__base_model_prefixr r m/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/camembert/modular_camembert.pyr,src@s eZdZdS)CamembertModelN)rrrr r r r!r"0sr"cseZdZdddZfddZee        ddejdBdej dBd ejdBd ejdBd ej dBd ej dBd ej dBdejdBde e de ej eBfddZZS)CamembertForMaskedLMz)roberta.embeddings.word_embeddings.weightz lm_head.bias)zlm_head.decoder.weightzlm_head.decoder.biasc"t||`t|dd|_dSNF)add_pooling_layersuper__init__ camembertr"rselfconfig __class__r r!r): zCamembertForMaskedLM.__init__N input_idsattention_masktoken_type_ids position_ids inputs_embedsencoder_hidden_statesencoder_attention_masklabelskwargsreturnc  Ks|j|f||||||dd| } | d} || } d} |dur7|| j}t}|| d|jj|d} t| | | j | j dS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)r2r3r4r5r6r7 return_dictrNlosslogits hidden_states attentions) rlm_headtodevicerviewr- vocab_sizerr@rA)r,r1r2r3r4r5r6r7r8r9outputssequence_outputprediction_scoresmasked_lm_lossloss_fctr r r!forward@s4   zCamembertForMaskedLM.forward)NNNNNNNN)rrr_tied_weights_keysr)rrtorch LongTensor FloatTensorr rtupleTensorrrL __classcell__r r r.r!r#4sH    r#ceZdZfddZee      ddejdBdejdBdejdBdejdBdejdBd ejdBd e e d e ej e Bfd d ZZS)"CamembertForSequenceClassificationcr$r%r'r+r.r r!r){r0z+CamembertForSequenceClassification.__init__Nr1r2r3r4r5r8r9r:c Ks6|j|f||||dd|}|d} || } d} |dur|| j}|jjdurN|jdkr4d|j_n|jdkrJ|jtj ksE|jtj krJd|j_nd|j_|jjdkrlt } |jdkrf| | | } n+| | |} n%|jjdkrt } | | d |j|d } n|jjdkrt} | | |} t| | |j|jd S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Tr2r3r4r5r;rN regressionsingle_label_classificationmulti_label_classificationr<r=)r classifierrCrDr- problem_type num_labelsdtyperNlongintrsqueezerrErr r@rA r,r1r2r3r4r5r8r9rGrHr?r>rKr r r!rLsN     "       z*CamembertForSequenceClassification.forwardNNNNNN)rrrr)rrrNrOrPr rrQrRr rLrSr r r.r!rUz6  rUcseZdZfddZee      ddejdBdejdBdejdBdejdBdejdBd ejdBd e e d e ej e Bfd d ZZS)CamembertForMultipleChoicecr$r%r'r+r.r r!r)r0z#CamembertForMultipleChoice.__init__Nr1r3r2r8r4r5r9r:cKs<|dur |jdn|jd}|dur|d|dnd} |dur*|d|dnd} |dur9|d|dnd} |durH|d|dnd} |dur[|d|d|dnd} |j| f| | | | dd|}|d}||}||}|d|}d}|dur||j}t}|||}t |||j |j dS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. NrWr<T)r4r3r2r5r;r=) shaperEsizerdropoutr[rCrDrr r@rA)r,r1r3r2r8r4r5r9 num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrG pooled_outputr?reshaped_logitsr>rKr r r!rLsF+     z"CamembertForMultipleChoice.forwardrc)rrrr)rrrNrOrPr rrQrRr rLrSr r r.r!rerdrecrT)CamembertForTokenClassificationcr$r%r'r+r.r r!r)&r0z(CamembertForTokenClassification.__init__Nr1r2r3r4r5r8r9r:c Ks|j|f||||dd|}|d} || } || } d} |dur9|| j}t} | | d|j|d} t| | |j |j dS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrVrNr<r=) rrir[rCrDrrEr]r r@rArbr r r!rL,s2   z'CamembertForTokenClassification.forwardrc)rrrr)rrrNrOrPr rrQrRr rLrSr r r.r!rr%rdrrcseZdZfddZee       ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd e e d e ej e Bfd dZZS)CamembertForQuestionAnsweringcr$r%r'r+r.r r!r)dr0z&CamembertForQuestionAnswering.__init__Nr1r2r3r4r5start_positions end_positionsr9r:cKs|j|f||||dd|} | d} || } | jddd\} } | d} | d} d}|dury|duryt|dkrF|d}t|dkrS|d}| d}|d|}|d|}t|d}|| |}|| |}||d }t || | | j | j d S) a[ token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) TrVrrWr<)dimN) ignore_indexr)r> start_logits end_logitsr@rA) r qa_outputssplitra contiguouslenrhclamprr r@rA)r,r1r2r3r4r5rtrur9rGrHr?rxry total_loss ignored_indexrK start_lossend_lossr r r!rLjsH          z%CamembertForQuestionAnswering.forward)NNNNNNN)rrrr)rrrNrOrPr rrQrRr rLrSr r r.r!rscs<   rsc seZdZfddZee            ddejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd e e ejdBde dBdej dBde ej Bde ede ej eBfddZZS)CamembertForCausalLMcr$r%r'r+r.r r!r)r0zCamembertForCausalLM.__init__Nrr1r2r3r4r5r6r7r8past_key_values use_cachecache_positionlogits_to_keepr9r:c  Ks|durd} |j|f||||||| | | dd | }|j}t| tr(t| dn| }||dd|ddf}d}|durL|jd|||jjd| }t |||j |j |j |j dS)aq token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, CamembertForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("almanach/camembert-base") >>> config = AutoConfig.from_pretrained("almanach/camembert-base") >>> config.is_decoder = True >>> model = CamembertForCausalLM.from_pretrained("almanach/camembert-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) r2r3r4r5r6r7rrrr;)r?r8rF)r>r?rr@rAcross_attentionsr )rlast_hidden_state isinstancer`slicerB loss_functionr-rFrrr@rAr)r,r1r2r3r4r5r6r7r8rrrrr9rGr@ slice_indicesr?r>r r r!rLs@1 zCamembertForCausalLM.forward) NNNNNNNNNNNr)rrrr)rrrNrOrPrQboolrRr`r rrrLrSr r r.r!rsZ       r)rr#rersrUrrr"r)'__doc__rNtorch.nnrrrmodeling_outputsrrrr r r r processing_utilsr utilsrr utils.genericrroberta.modeling_robertarrrrrrrrrr"r#rUrerrrsr__all__r r r r!s"$  ( FO\>J ]