o iV @sddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m"Z"erddl#Z$ddl#m%Z%ddl#m&Z'Gddde%Z(e )j*Z+e e,j-j-a.t.j-j-a/de+d0Z1GdddZ2dee3ej4fde3de3fddZ5de2de3d e3d!e6e7e3e8ffd"d#Z9d$e3d%e6e3fd&d'Z:d(e3de3d e3fd)d*Z;d(e3de3d!e6e7e3e8ffd+d,Zd-e3de2d1e3d.e7e3e3e6ffd2d3Z?de2de3d!e6e7e3e8fd.e3fd4d5Z@de2d!e6e7e3e8ffd6d7ZAde2de3d e3d!e6e7e3e8fd8e8f d9d:ZB   dKd;e3dee3d.e f d?d@ZCdAe3d.e8fdBdCZDdDdEZEdFefdGdHZFGdIdJdJe!ZGdS)LN)ArgumentParser Namespace)date)Path)AnyCallableOptionalUnion)CONFIG_MAPPING_NAMESMODEL_NAMES_MAPPING)FEATURE_EXTRACTOR_MAPPING_NAMES)IMAGE_PROCESSOR_MAPPING_NAMES)PROCESSOR_MAPPING_NAMES)TOKENIZER_MAPPING_NAMES)VIDEO_PROCESSOR_MAPPING_NAMES)is_libcst_available)BaseTransformersCLICommand)add_fast_image_processor) CSTVisitor)matchersc@sLeZdZdZddZdejddfddZdejfd d Zdej fd d Z dS) ClassFinderzC A visitor to find all classes in a python module. cCsg|_g|_d|_dSNF)classespublic_classes is_in_classselfr\/home/ubuntu/.local/lib/python3.10/site-packages/transformers/commands/add_new_model_like.py__init__/s zClassFinder.__init__nodereturnNcCs|j|jjd|_dS)zwRecord class names. We assume classes always only appear at top-level (i.e. no class definition in function or similar)TN)rappendnamevaluerrr"rrr visit_ClassDef4s zClassFinder.visit_ClassDefcCs d|_dSr)rr'rrr leave_ClassDef9s zClassFinder.leave_ClassDefcCstjtjtjtdgdgd}|js:t||r<|jdjdj j }|dkr>|jdj j }dd|D|_ dSdSdSdS) z:Record all public classes inside the `__all__` assignment.)target)targets)bodyr__all__cSsg|]}|jjqSr)r&).0elementrrr Esz9ClassFinder.visit_SimpleStatementLine..N) mSimpleStatementLineAssign AssignTargetNamermatchesr,r+r*r&elementsr)rr"!simple_top_level_assign_structureassigned_variabler7rrr visit_SimpleStatementLine<sz%ClassFinder.visit_SimpleStatementLine) __name__ __module__ __qualname____doc__r!cstClassDefr(r)r2r:rrrr r*s rz # coding=utf-8 # Copyright aN the HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. c@seZdZdZdefddZdS) ModelInfoszI Retrieve the basic information about an existing model classes. lowercase_namecCs|dddd|_|jtvr|jdd|jtvr%t|dt|j|_t|j|_|jdd|_|jt vrSt |j\|_ |_ |j dkrNdn|j |_ nd\|_ |_ t |jd\|_|_t |jd|_t |jd|_t |jd|_dS) N _-z is not a valid model nameConfigPreTrainedTokenizerFast)NN)lowerreplacerBr ValueErrorr paper_name config_classcamelcase_namertokenizer_classfast_tokenizer_classrgetimage_processor_classfast_image_processor_classrvideo_processor_classr feature_extractor_classrprocessor_class)rrBrrr r!cs&       zModelInfos.__init__N)r;r<r=r>strr!rrrr rA^srA file_name new_content add_aftercCst|ddd }|}Wdn1swY||d\}}||||}t|ddd}||WddS1sCwYdS)ac A utility to add some content inside a given file. Args: file_name (`str` or `os.PathLike`): The name of the file in which we want to insert some content. new_content (`str`): The content to add. add_after (`str`): The new content is added just after the first instance matching it. rutf-8encodingNrw)openreadsplitwrite)rXrYrZf old_contentbeforeafterrrr add_content_to_files  "rhold_model_infosnew_lowercase_namenew_model_paper_namefilenames_to_addc sddd|dddD}|j|j}fdd|dd D}g}|D] \}}td |rA|d \} } | | p<|f|d <q(|||fq(tt d d dd|d|dddtt d d dd|d|ddd|D]P\} }|r| dd} t t d d |  } | }Wd n1swYt dd|tj } | D]}tt d d | |||||dqqsd S)a Add a model to all the relevant mappings in the auto module. Args: old_model_infos (`ModelInfos`): The structure containing the class information of the old model. new_lowercase_name (`str`): The new lowercase model name. new_model_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. filenames_to_add (`list[tuple[str, bool]]`): A list of tuples of all potential filenames to add for a new model, along a boolean flag describing if we should add this file or not. For example, [(`modeling_xxx.px`, True), (`configuration_xxx.py`, True), (`tokenization_xxx.py`, False),...] rGcs|]}|VqdSNtitler.xrrr z-add_model_to_auto_mappings..rErDcs g|] \}}|d|fqS)autorJr.filenameto_addold_lowercase_namerr r0sz.add_model_to_auto_mappings..rNz2(?:tokenization)|(?:image_processing)_auto_fast.pymodelsruzconfiguration_auto.pyz ("z", "z Config"), zOCONFIG_MAPPING_NAMES = OrderedDict[str, str]( [ # Add configs here rYrZz"), zcMODEL_NAMES_MAPPING = OrderedDict[str, str]( [ # Add full (and cased) model names here _fast.py.pyz( {8,12}\(\s*"z",.*?\),\n)(?: {4,12}\(|\]))joinrJrbrBrNresearchr$rhTRANSFORMERS_PATHr`rafindallDOTALL)rirjrkrlnew_cased_nameold_cased_namecorrected_filenames_to_addfilery previous_fileprevious_to_addrxrdmatching_linesmatchrrzr add_model_to_auto_mappingssR"         rnew_paper_namerc Csd}tddtdd|}td|d|d}g}|D]}d |d |}d |vr1|d 7}||qd |}|||S)a4 Create a new doc file to fill for the new model. Args: new_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. public_classes (`list[str]`): A list of all the public classes that the model will have in the library. u ⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. --> z# ?rGz coding=utf-8 z