o %ݫi@s@dZddlZddlmZddlmZeeZGdddeZdS)zThis lobe enables the integration of huggingface pretrained GPT2LMHeadModel model. Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html Authors * Pooneh Mousavi 2023 * Simone Alghisi 2023 N)HFTransformersInterface) get_loggercsleZdZdZ        d dfd d ZdejdejdejfddZ ddejdejfddZZ S)GPTaThis lobe enables the integration of HuggingFace pretrained GPT model. Source paper whisper: https://life-extension.github.io/2020/05/27/GPT%E6%8A%80%E6%9C%AF%E5%88%9D%E6%8E%A2/language-models.pdf Transformer from HuggingFace needs to be installed: https://huggingface.co/transformers/installation.html The model can be finetuned. It will download automatically the model from HuggingFace or use a local path. Arguments --------- source : str HuggingFace hub name: e.g "gpt2" save_path : str Path (dir) of the downloaded model. freeze : bool (default: False) If True, the model is frozen. If False, the model will be trained alongside with the rest of the pipeline. max_new_tokens : int Maximum count of new tokens allowed. min_length : int Minimum count of input tokens top_k : int Top results count to keep top_p : float Proportion of top results to keep num_beams : int Number of decoder beams eos_token_id : int Index of end-of-sentence token. early_stopping : int Whether to stop training early. Example ------- >>> model_hub = "gpt2" >>> save_path = "savedir" >>> model = GPT(model_hub, save_path) >>> tokens = torch.tensor([[1, 1]]) >>> tokens_type = torch.tensor([[1, 1]]) >>> attention_mask = torch.tensor([[1, 1]]) >>> outputs = model(tokens, tokens_type, attention_mask) F-?RTreturnNc stj|||dd||_||_||_||_||_| |_| |_|j |ddd|j r?t d|j |j D]} d| _q9dSdS)NT)source save_pathfreeze with_lm_headF)r pad_tokenuse_fastz!huggingface_GPT - GPT is frozen.)super__init__max_new_tokens min_lengthtop_ktop_p num_beamsearly_stopping eos_token_idload_tokenizerrloggerwarningmodeltrain parameters requires_grad) selfr r rrrrrrrrparam __class__i/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/huggingface_transformers/gpt.pyrBs$   z GPT.__init__ input_idstoken_type_idsattention_maskcCsFt|j |jj|||d}Wd|S1swY|S)aTakes an input a history of conversation and returns its corresponding reply. Arguments --------- input_ids : torch.Tensor A batch of input-id to transform to features. token_type_ids : torch.Tensor Token Type(Speaker) for each token in input_ids. attention_mask : torch.Tensor A batch of attention_mask. Returns ------- output : torch.Tensor Reply to conversation )r)r*N)torchset_grad_enabledrrforward)r"r(r)r*outputr&r&r'r-bs z GPT.forwardgreedycCst<|dkr#|jj|||d|j|j|j|j|jd|j |j d }n|jj|||j|j |d}Wd|SWd|S1sCwY|S)a Takes an input a history of conversation and returns its corresponding reply. Arguments --------- input_ids : torch.Tensor A batch of input-id which are dialogue context tokens token_type_ids : torch.Tensor attention_mask : torch.Tensor A batch of attention_mask. decoder_type : str It shows strategy for autoregressive decoding either beam search or greedy. Returns ------- hyp : torch.Tensor Conversation reply. beamTr) r(r)r* do_samplerrrrrnum_return_sequencesrr)r)rrr*N) r+no_gradrgeneraterrrrrrr)r"r(r)r* decoder_typehypr&r&r'r4s<   z GPT.generate)Frrrrr r T)r N)r/) __name__ __module__ __qualname____doc__rr+Tensorr-r4 __classcell__r&r&r$r'rs40   #r) r:r+=speechbrain.lobes.models.huggingface_transformers.huggingfacerspeechbrain.utils.loggerrr7rrr&r&r&r's