o ¹ÈÏiýã@sZdZddlmZmZddlmZddlmZddlm Z m Z m Z Gdd„deƒZ dgZ d S) z Processor class for MarkupLM. é)ÚOptionalÚUnioné)Ú TensorType)ÚProcessorMixin)Ú BatchEncodingÚPaddingStrategyÚTruncationStrategyc @sºeZdZdZdZdZdZ                   dded eee e fd eee e fd e e d e d e e de ede edededededede ee efdefdd„ZdS)ÚMarkupLMProcessoraJ Constructs a MarkupLM processor which combines a MarkupLM feature extractor and a MarkupLM tokenizer into a single processor. [`MarkupLMProcessor`] offers all the functionalities you need to prepare data for the model. It first uses [`MarkupLMFeatureExtractor`] to extract nodes and corresponding xpaths from one or more HTML strings. Next, these are provided to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`], which turns them into token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_tags_seq` and `xpath_subs_seq`. Args: feature_extractor (`MarkupLMFeatureExtractor`): An instance of [`MarkupLMFeatureExtractor`]. The feature extractor is a required input. tokenizer (`MarkupLMTokenizer` or `MarkupLMTokenizerFast`): An instance of [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`]. The tokenizer is a required input. parse_html (`bool`, *optional*, defaults to `True`): Whether or not to use `MarkupLMFeatureExtractor` to parse HTML strings into nodes and corresponding xpaths. ÚMarkupLMFeatureExtractor)ÚMarkupLMTokenizerÚMarkupLMTokenizerFastTNFrÚadd_special_tokensÚpaddingÚ truncationÚ max_lengthÚstrideÚpad_to_multiple_ofÚreturn_token_type_idsÚreturn_attention_maskÚreturn_overflowing_tokensÚreturn_special_tokens_maskÚreturn_offsets_mappingÚ return_lengthÚverboseÚreturn_tensorsÚreturncKsž|jr)|dur tdƒ‚|dus|dus|durtdƒ‚| |¡}|d}|d}n|dur1tdƒ‚|dus9|dur=tdƒ‚|durL|jrLt|tƒrL|g}|jdid|durW|n|“d |dur`|n8d“d|“d |“d |“d |“d |“d| “d| “d| “d| “d| “d|“d|“d|“d|“d|“d|“|¤Ž}|S“d|“d |“d |“d |“d |“d| “d| “d| “d| “d| “d|“d|“d|“d|“d|“d|“|¤Ž}|S)aÆ This method first forwards the `html_strings` argument to [`~MarkupLMFeatureExtractor.__call__`]. Next, it passes the `nodes` and `xpaths` along with the additional arguments to [`~MarkupLMTokenizer.__call__`] and returns the output. Optionally, one can also provide a `text` argument which is passed along as first sequence. Please refer to the docstring of the above two methods for more information. NzDMake sure to pass HTML strings in case `parse_html` is set to `True`zUPlease don't pass nodes, xpaths nor node labels in case `parse_html` is set to `True`ÚnodesÚxpathsz@You have passed HTML strings but `parse_html` is set to `False`.zIMake sure to pass nodes and xpaths in case `parse_html` is set to `False`ÚtextÚ text_pairÚ node_labelsrrrrrrrrrrrrrr©)Ú parse_htmlÚ ValueErrorÚfeature_extractorÚ isinstanceÚstrÚ tokenizer)ÚselfÚ html_stringsrrr!Ú questionsrrrrrrrrrrrrrrÚkwargsÚfeaturesÚencoded_inputsr"r"úd/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/markuplm/processing_markuplm.pyÚ__call__2sº!ÿ    €ÿþýüûúùø ÷ ö õ ô óòñðïíìþýüûúùø ÷ ö õ ô óòñðïízMarkupLMProcessor.__call__)NNNNNTFNNrNNNFFFFTN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Úfeature_extractor_classÚtokenizer_classr#Úboolrr'rr rÚintrrr0r"r"r"r/r snìù ø ÷ ö õ ô óòñðïîíìêr N)r4ÚtypingrrÚ file_utilsrÚprocessing_utilsrÚtokenization_utils_baserrr r Ú__all__r"r"r"r/Ús   i