o yi-@sddlZddlZddlmZmZddlmZmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZmZddlmZmZerMddlmZmZmZmZndZZerWddlZded efd d Z d1d eeefd e e eejfd eeeffddZdedede d e eeffddZ!deded e eeeffddZ"    d2de edede de#de#de#d e eeefe effd d!Z$d3d"ed#e#d e ed$ffd%d&Z%d'eded dfd(d)Z& d1d*e eej'fd e e eejfd e eeffd+d,Z(Gd-d.d.eZ)Gd/d0d0e)Z*dS)4N)Counter defaultdict)AnyCallableDictListOptionalSetTupleUnion)Tensor) DataLoaderDataset)_TQDM_AVAILABLE_TRANSFORMERS_AVAILABLE)AutoModelForMaskedLM AutoTokenizerPreTrainedModelPreTrainedTokenizerBaseattention_maskreturncCsDd|dddf<|ddd}d|t|d|f<|S)a(Process attention mask to be zero for special [CLS] and [SEP] tokens as they're not included in a calculation for BERT score. Args: attention_mask: An attention mask to be returned, for example, by a `transformers` tokenizer. Return: A processed attention mask. rNg?)cumsumargmaxtorcharangesizelong)rsep_token_positionrh/home/ubuntu/.local/lib/python3.10/site-packages/torchmetrics/functional/text/helper_embedding_metric.py*_process_attention_mask_for_special_tokens"s r!batchdevicecCsjt|dd}|dddd|f|}|dddd|f|}|||d|S)z|Helper function that trims model inputs to the longest sequence within the batch and put the input on the proper device.r input_idsNr%r)intsummaxitemtoupdate)r"r#max_lenr%rrrr _input_data_collator4s r. model_output target_lencCs|t|j}||d|d<tj|tj||jd|jgdd}tj|d|d|jd|j}tj||gdd}||fS)zSHelper function that pads the model output and attention mask to the target length.)dtype)dimrr$)listshapercatzerosr2r+r#)r/rr0 zeros_shaper7rrr _output_data_collator@s "r9r%cCs(|d}||}||}|||fS)z=Sort tokenized sentence from the shortest to the longest one.r$)r(argsort)r%rsorted_indicesrrr _sort_data_according_lengthLs r<TFtext tokenizer max_length truncationsort_according_length own_tokenizerc Cs|s ||d||dd}nz|||}Wnty&}ztd|d}~ww|r>t|d|d\}} } || d} | | fS|d|dd} d} | | fS) aDefault text pre-processing function using `transformers` `AutoTokenizer` instance. Args: text: An iterable of sentences. tokenizer: Either `AutoTokenizer` instance from `transformers` package, or a user's own tokenizer. max_length: A maximum sequence length. truncation: An indication of whether tokenized sequences should be padded only to the length of the longest sequence. sort_according_length: An indication of whether tokenized sequences should be sorted from shortest to longest. This is appropriate to do for leveraging dynamic padding during embedding calculation and thereby to hasten inference. own_tokenizer: An indication of whether a non-default user's own tokenizer is used. Return: A dictionary of tokenized sentences including input_ids and attention_mask. Raises: BaseException: If a tokenization with a user's own tokenizer is not successful. r@pt)paddingr@rAreturn_tensorsz!Tokenization was not successful: Nr%rr&) BaseExceptionr<) r>r?r@rArBrCtokenized_dataer%rsorting_indices input_dictrrr _preprocess_textTs&    rL dataloaderverboseztqdm.auto.tqdmcCs|rtj|S|S)zHelper function returning either the dataloader itself when `verbose = False`, or it wraps the dataloader with `tqdm.auto.tqdm`, when `verbose = True` to display a progress bar during the embbeddings calculation.)tqdmauto)rMrNrrr _get_progress_barsrQoutputcCsb|jdd\}}t|jdkp|jd|kp|jd|k}|r/td|d|d|jd dS) z2Check if the shape of the user's own model output.Nr1rr$zVThe model output must be `Tensor` of a shape `[batch_size, seq_len, model_dim]` i.e. [z, z. , `model_dim`], but got .)r5len ValueError)rRr%bsseq_leninvalid_out_shaperrr _check_shape_of_model_outputs*rZmodel_name_or_pathcCs.t|}t|}|||||fS)agLoad HuggingFace `transformers`' tokenizer and model. This function also handle a device placement. Args: model_name_or_path: A name or a model path used to load `transformers` pretrained model. device: A device to be used for calculation. Return: Initialized `transformers`' tokenizer and model. )rfrom_pretrainedrevalr+)r[r#r?modelrrr _load_tokenizer_and_models   r_c@seZdZdZdeddfdeededede eeeege e ee fe e ee fee fffd ed ee eefd dfd d Zded e ee ffddZd efddZd e eeffddZd efddZede d efddZdS) TextDatasetzoPyTorch dataset class for storing tokenized sentences and other properties used for BERT score calculation.r=FNr>r?r@preprocess_text_fnidf tokens_idfrcCst||||}t|tr|\|_|_n||_|jdjd|_t||_||_i|_ |r8|dur1|n| |_ dSdS)aX Args: text: An iterable of sentences. tokenizer: `AutoTokenizer` instance from `transformers` package. max_length: A maximum sequence length. preprocess_text_fn: A function used for processing the input sentences. idf: An indication of whether calculate token inverse document frequencies to weight the model embeddings. tokens_idf: Inverse document frequencies (these should be calculated on reference sentences). r%r$N) isinstancetupler>rJr5r@rU num_sentencesrbrc_get_tokens_idf)selfr>r?r@rarbrc_textrrr __init__s   zTextDataset.__init__idxcsdjd|ddf}jd|ddf}||d}jr0tfdd|D}||d<|S)Nr%rr&csg|]}j|qSr)rc).0 input_idxrhrr sz+TextDataset.__getitem__.. input_ids_idf)r>rbrtensortolist)rhrkr%r inputs_dictrprrnr __getitem__s zTextDataset.__getitem__cCs|jSN)rfrnrrr __len__szTextDataset.__len__csRt}tjjdD]}||q tj}|fdd|D|S)zCalculate token inverse document frequences. Return: A python dictionary containing inverse document frequences for token ids. r%cs*i|]\}}|tjd|dqS)r$mathlogrf)rlrk occurrencernrr s*z/TextDataset._get_tokens_idf..)rmap_set_of_tokensr>r,r_get_tokens_idf_default_valueitems)rh token_countertokensrcrrnr rgs  zTextDataset._get_tokens_idfcCst|jddS)z9Helper function that ensures `defaultdict` to be pickled.r$rwrnrrr r~sz)TextDataset._get_tokens_idf_default_valuer%cCs t|S)zAReturn set of tokens from the `input_ids` :class:`~torch.Tensor`.)setrr)r%rrr r}s zTextDataset._set_of_tokens)__name__ __module__ __qualname____doc__rLrstrrr'rr rr r rboolfloatrjrtrvrgr~ staticmethodr r}rrrr r`s:4  & r`c @s@eZdZdZ  d dedededeeee fddf d d Z dS) TokenizedDatasetzHThe child class of `TextDataset` class used with already tokenized data.FNr%rrbrcrcCs~ttgdt||}|d|_t||_t|jd|_|jdj d|_ ||_ i|_ |r=|dur6|n| |_ dSdS)ah Args: input_ids: Input indexes attention_mask: Attention mask idf: An indication of whether calculate token inverse document frequencies to weight the model embeddings. tokens_idf: Inverse document frequencies (these should be calculated on reference sentences). )r%rrJrJr%r$N)dictzipr<poprJr.r>rUrfr5r@rbrcrg)rhr%rrbrcr>rrr rjs  zTokenizedDataset.__init__)FN) rrrrr rrrr'rrjrrrr rsrru)r=TTF)F)+rxos collectionsrrtypingrrrrrr r r rr torch.utils.datar rtorchmetrics.utilities.importsrr transformersrrrrrOr!rr#r.r'r9r<rrLrQrZPathLiker_r`rrrrr sn (    "   6    P