o ¾e¦iNã@sZdZddlmZddlmZddlmZmZmZddl m Z e Gdd„deƒƒZ dgZ dS) z Processor class for MarkupLM. é)Ú TensorType)ÚProcessorMixin)Ú BatchEncodingÚPaddingStrategyÚTruncationStrategy)Úauto_docstringc sºeZdZdZ‡fdd„Ze                   ddedeeBeBd eeBe Bd e dBd e d e dBd edBdedBdedededededee BdBde fdd„ƒZ ‡ZS)ÚMarkupLMProcessorTcstƒ ||¡dS)N)ÚsuperÚ__init__)ÚselfÚfeature_extractorÚ tokenizer©Ú __class__©ún/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/markuplm/processing_markuplm.pyr szMarkupLMProcessor.__init__NFéÚadd_special_tokensÚpaddingÚ truncationÚ max_lengthÚstrideÚpad_to_multiple_ofÚreturn_token_type_idsÚreturn_attention_maskÚreturn_overflowing_tokensÚreturn_special_tokens_maskÚreturn_offsets_mappingÚ return_lengthÚverboseÚreturn_tensorsÚreturncKsž|jr)|dur tdƒ‚|dus|dus|durtdƒ‚| |¡}|d}|d}n|dur1tdƒ‚|dus9|dur=tdƒ‚|durL|jrLt|tƒrL|g}|jdid|durW|n|“d |dur`|n8d“d|“d |“d |“d |“d |“d| “d| “d| “d| “d| “d|“d|“d|“d|“d|“d|“|¤Ž}|S“d|“d |“d |“d |“d |“d| “d| “d| “d| “d| “d|“d|“d|“d|“d|“d|“|¤Ž}|S)a‡ html_strings (`str` or `list[str]`, *optional*): Raw HTML strings to parse and process. When `parse_html=True` (default), these strings are parsed to extract nodes and xpaths automatically. If provided, `nodes`, `xpaths`, and `node_labels` should not be provided. Required when `parse_html=True`. nodes (`list[list[str]]`, *optional*): Pre-extracted HTML nodes as a list of lists, where each inner list contains the text content of nodes for a single document. Required when `parse_html=False`. Should not be provided when `parse_html=True`. xpaths (`list[list[str]]`, *optional*): Pre-extracted XPath expressions corresponding to the nodes. Should be a list of lists with the same structure as `nodes`, where each XPath identifies the location of the corresponding node in the HTML tree. Required when `parse_html=False`. Should not be provided when `parse_html=True`. node_labels (`list[list[int]]`, *optional*): Labels for the nodes, typically used for training or fine-tuning tasks. Should be a list of lists with the same structure as `nodes`, where each label corresponds to a node. Optional and only used when `parse_html=False`. questions (`str` or `list[str]`, *optional*): Question strings for question-answering tasks. When provided, the tokenizer processes questions as the first sequence and nodes as the second sequence (text_pair). If a single string is provided, it is converted to a list to match the batch dimension of the parsed HTML. NzDMake sure to pass HTML strings in case `parse_html` is set to `True`zUPlease don't pass nodes, xpaths nor node labels in case `parse_html` is set to `True`ÚnodesÚxpathsz@You have passed HTML strings but `parse_html` is set to `False`.zIMake sure to pass nodes and xpaths in case `parse_html` is set to `False`ÚtextÚ text_pairÚ node_labelsrrrrrrrrrrrrrr r)Ú parse_htmlÚ ValueErrorr Ú isinstanceÚstrr )r Ú html_stringsr"r#r&Ú questionsrrrrrrrrrrrrrr ÚkwargsÚfeaturesÚencoded_inputsrrrÚ__call__sº.ÿ    €ÿþýüûúùø ÷ ö õ ô óòñðïíìþýüûúùø ÷ ö õ ô óòñðïízMarkupLMProcessor.__call__)NNNNNTFNNrNNNFFFFTN)Ú__name__Ú __module__Ú __qualname__r'r rÚboolr*rrÚintrrr0Ú __classcell__rrrrrsl ìù ø ÷ ö õ ô óòñðïîí ìêrN) Ú__doc__Ú file_utilsrÚprocessing_utilsrÚtokenization_utils_baserrrÚutilsrrÚ__all__rrrrÚs    d