o eij@sRdZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z mZddlmZdd lmZmZmZdd lmZdd lmZmZd d lmZeeZddZddZ d&ddZ!Gdddej"Z#ddZ$Gdddej"Z%eGdddeZ&eGddde&Z'eddGd d!d!e&eZ(ed"dGd#d$d$e&Z)gd%Z*dS)'zPyTorch CTRL model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)Cache DynamicCache)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)auto_docstringlogging) CTRLConfigcCs$dtdd|d|}||S)Nri')torchpow)posi d_model_size angle_ratesrd/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defn%srcCsttj|tjd|dtj|tjd|d|}t|dddddf}t|dddddf}tj||gdd}|S)Ndtyperrrdim) rrarangeint64to unsqueezesincoscat)positionrr angle_radssinescosines pos_encodingrrrpositional_encoding*sr.c Cst||dddd}|jd}|t|}|dur7|d|d}} ||| || d| fd7}|dur?||}tj|dd} t| |} | | fS) Nrrrrrgr )rmatmulpermuteshapenpsqrtsizesoftmax) qkvmaskattention_mask matmul_qkdkscaled_attention_logitsndnsattention_weightsoutputrrrscaled_dot_product_attention9s   rCcs:eZdZd fdd ZddZ     d ddZZS) MultiHeadAttentionNcsht||_||_||_t||j|_t|||_ t|||_ t|||_ t|||_ dSN) super__init__ num_headsr layer_idxintdepthrLinearWqWkWvdense)selfrrHrI __class__rrrGPs zMultiHeadAttention.__init__cCs"||d|j|j}|gdS)Nrrrrr)reshaperHrKr1)rQx batch_sizerrrsplit_into_heads^sz#MultiHeadAttention.split_into_headsFc Cs|jd} ||}||}||}||| }||| }||| }|dur7||||jd| i\}}t|||||} | dgd} | d} | | d|j }| |} | | fS)Nrcache_positionrTrr) r2rMrNrOrXupdaterIrCr1rUrrP)rQr9r8r7r: layer_pastr; use_cacheoutput_attentionsrYrWrBscaled_attentionattnoriginal_size_attentionrrrforwardbs       zMultiHeadAttention.forwardrENNFFN)__name__ __module__ __qualname__rGrXra __classcell__rrrRrrDOs rDcCs"tt||tt||SrE)r SequentialrLReLU)rdffrrrpoint_wise_feed_forward_networks"rjcs2eZdZdfdd Z     d ddZZS) EncoderLayer皙?Ncsbtt|||d|_t|||_tj|dd|_tj|dd|_ t ||_ t ||_ dS)NrIgư>eps) rFrGrDmulti_head_attentionrjffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)rQrrHriraterIrRrrrGs   zEncoderLayer.__init__Fc Cs|||}|j|||||||||d } | d} || } || } || } || } || } | | } | f| dd}|S)Nr[r;r\r]rYrr)rsrprvrtrqrw)rQrVr:r[r;r\r]rYnormed attn_outputs attn_outputout1out2 ffn_outputoutputsrrrras*     zEncoderLayer.forward)rlNrb)rcrdrerGrarfrrrRrrksrkcs*eZdZUeed<dZfddZZS)CTRLPreTrainedModelconfig transformercs<t|t|trt|jt|jj |j t j dSdSrE) rF _init_weights isinstance CTRLModelinitcopy_r-r.r n_positionsrrfloat)rQmodulerRrrrs  z!CTRLPreTrainedModel._init_weights)rcrdrer__annotations__base_model_prefixrrfrrrRrrs rcseZdZfddZddZddZe           ddejdBd e dBd ej dBd ejdBd ejdBd ej dBde dBde dBde dBde dBdej dBde ej eBfddZZS)rcstj|_j|_tjj|_ t j |_ t fddtjD|_tjjjd|_|jdtj|jtjdd|dS)Nc s&g|]}tjjjj|dqS)rm)rkn_embdn_headri resid_pdrop).0rrrr sz&CTRLModel.__init__..rnr-F) persistent)rFrGrrn_layer num_layersr Embedding vocab_sizewru embd_pdropdropout ModuleListrangehrrlayer_norm_epsilon layernormregister_bufferr.rrr post_initrQrrRrrrGs   zCTRLModel.__init__cCs|jSrEr)rQrrrget_input_embeddingsszCTRLModel.get_input_embeddingscCs ||_dSrEr)rQnew_embeddingsrrrset_input_embeddingss zCTRLModel.set_input_embeddingsN input_idspast_key_valuesr;token_type_ids position_ids inputs_embedsr\r]output_hidden_states return_dictrYreturnc  Ks |dur|n|jj}|dur|n|jj}| dur| n|jj} | dur$| n|jj} |dur4|dur4td|durP||||} |d| d}|j d}n|durb|dd} |j d}ntd|durm|j n|j }|r||dur|t |jd}|dur| nd}|durt j|| d|t j|d}|d}|dur|dkrtd||d}|d d }|j|jd }d |t |jj}|dur|d| d}||}|t|j9}nd}|dur||}| d}t t ||||d |}|t|j9}|j||_|j|ddf}|||}||}| r+d nd}|r2d nd}t|jD]&\}}| rE||f}|||||||| d}|d}|r^||d f7}q9||}| rm||f}| s}t dd||||fDSt!||||dS)a Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerrz5You have to specify either input_ids or inputs_embedsr)rdevicez$batch_size has to be defined and > 0rrrg?rrycss|] }|dur|VqdSrEr)rr9rrr hsz$CTRLModel.forward..)last_hidden_stater hidden_states attentions)"rr]r\ruse_return_dict ValueError%warn_if_padding_and_no_attention_maskr5viewr2rr get_seq_lengthrr"longr%r$rfinfominrr3r4rtriuonesr-r enumeraterrtupler )rQrrr;rrrr\r]rrrYkwargs input_shaperWr past_lengthtoken_type_embedsseq_lenr: pos_embedsrall_hidden_statesall_attentionsrrrrrrras$         "       zCTRLModel.forward NNNNNNNNNNN)rcrdrerGrrrr LongTensorr FloatTensorboolTensorrr rarfrrrRrrsR      rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introcseZdZddiZfddZe             ddejdBdedBd ej dBd ejdBd ejdBd ej dBd ejdBde dBde dBde dBde dBdej dBde ej Bde ej eBfddZ dfdd ZZS)CTRLLMHeadModelzlm_head.weightztransformer.w.weightcs8t|t||_tj|j|jdd|_| dS)NTbias) rFrGrrrrLrrlm_headrrrRrrrG}s   zCTRLLMHeadModel.__init__Nrrrr;rrrlabelsr\r]rrrYlogits_to_keeprc Ks| dur| n|jj} |j|||||||| | | | d }|d}t| tr)t| dn| }||dd|ddf}d}|durM|j||fd|jji|}| sc|f|dd}|dura|f|S|St |||j |j |j dS)ag labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N) rr;rrrr\r]rrrYrrr)losslogitsrrr) rrrrrJslicer loss_functionrr rrr)rQrrr;rrrrr\r]rrrYrrtransformer_outputsr slice_indicesrrrBrrrrasJ1zCTRLLMHeadModel.forwardFc s,tj|f|||d|}|dd|S)N)rr\is_first_iterationr)rFprepare_inputs_for_generationpop)rQrrr\rr model_inputsrRrrrs z-CTRLLMHeadModel.prepare_inputs_for_generation) NNNNNNNNNNNNr)NNF)rcrdre_tied_weights_keysrGrrrrrrrrJrr rarrfrrrRrrts`        [ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZfddZe           ddejdBdedBdejdBdejdBdejdBd ejdBd ejdBd e dBd e dBd e dBde dBde ej e BfddZ ZS)CTRLForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dS)NFr) rFrG num_labelsrrrrLr classifierrrrRrrrGs   z&CTRLForSequenceClassification.__init__Nrrr;rrrrr\r]rrrc  KsH| dur| n|jj} |j|||||||| | | d } | d}||}|dur0|jdd\}}n |jdd\}}|jjdurG|dkrGtd|jjdurPd}n1|duru||jjk|jt j }t j |jd|jt j d}|| d}n d}t |jjd |t j ||jd |f}d}|dur|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n,|||}n&|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```N) rr;rrrr\r]rrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rrrr)rrrrr2 pad_token_idrr$rrint32r"argmaxlogger warning_oncerSrc problem_typerrrrJrsqueezerrrr rr)rQrrr;rrrrr\r]rrrrrrrWsequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrBrrrrasv]       "      z%CTRLForSequenceClassification.forwardr)rcrdrerGrrrrrrrrr rarfrrrRrrsN       r)rrrrrE)+__doc__numpyr3rrtorch.nnrrrrr cache_utilsrr generationr modeling_outputsr r r modeling_utilsrutilsrrconfiguration_ctrlr get_loggerrcrrr.rCModulerDrjrkrrrr__all__rrrrsD       40 0y  3