o i$1@sddlZddlmZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z ddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZmZddl m!Z!ddlm"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0dZ1d>de e2fddZ3Gddde"Z4GdddeZ5Gd d!d!eZ6e6j7d"d#d$ged%d%d&d'id(d)d)dd*d+d,ed-e d.e2d/e8d0e8d1e ef d2d3Z9ed4gd5Z:d?d7d8Z;d9d:ZdS)AN) namedtuple)Path)AnyCallableDictOptionalUnion)Model)util)Errors) BaseDefaultsLanguage) Morphologizer)DEFAULT_MORPH_MODEL)Scorer)POS)Doc MorphAnalysis)validate_examples)DummyTokenizerload_config_from_strregistry)Vocab) STOP_WORDS)SYNTAX_ITERATORS)TAG_BIGRAM_MAP)TAG_MAP) TAG_ORTH_MAPzU [nlp] [nlp.tokenizer] @tokenizers = "spacy.ja.JapaneseTokenizer" split_mode = null split_modecsfdd}|S)Ncst|jdS)Nr )JapaneseTokenizervocab)nlpr!J/home/ubuntu/.local/lib/python3.10/site-packages/spacy/lang/ja/__init__.pyjapanese_tokenizer_factory$sz4create_tokenizer..japanese_tokenizer_factoryr%)r r'r%r!r&create_tokenizer#s r(c@seZdZd#dedeeddfddZddZd edefd d Z d$d e fddZ ddZ ddZ deeeffddZifdeeefddfddZdefddZdeddfddZdeeefddfdd Zdeeefddfd!d"ZdS)%r"Nr#r returncCs0||_||_t|j|_|dup|dk |_dS)NA)r#r try_sudachi_import tokenizerneed_subtokens)selfr#r r%r%r&__init__+s zJapaneseTokenizer.__init__cCst|j|jffSN)r"r#r r.r%r%r& __reduce__2szJapaneseTokenizer.__reduce__textcCs6|j|}||}t||\}}|rt|nggd\}}}}} } } t| } t|j||d} d} tt| |D]W\}\}}|j |_ | rK| |_ d} nt |j |j |dt|kr^||dnd\|_ } |jrj|jn|j|_i}|jrx|j|d<|j|_|jrtdd|j|d<t|j||_q9|jr| | jd<| S) N)wordsspacesr Inflectionz[=|]_Reading sub_tokens)r,tokenize _get_dtokensget_dtokens_and_spacesziplistrr# enumeratetagtag_pos resolve_posorth_lenlemmasurfacelemma_infnormnorm_readingresubrmorphr- user_data)r.r3sudachipy_tokensdtokensr6r5tags inflectionslemmasnormsreadingssub_tokens_listdocnext_posidxtokendtokenrPr%r%r&__call__5s:     zJapaneseTokenizer.__call__Tneed_sub_tokenscs>|r||ndfddt|DfddtDS)Nc sg|]@\}}t|dkrt|ddd|ddDddd|ddD|||r?|ndqS)r-cSg|]}|dkr|qS*r%.0xxr%r%r& dz=JapaneseTokenizer._get_dtokens...N;cSrbrcr%rer%r%r&rheri)rFrH DetailedTokenjoinpart_of_speechdictionary_formnormalized_form reading_form)rfr\r])rYr%r&rhas  z2JapaneseTokenizer._get_dtokens..csTg|]&\}}|dks&|jr&|jdks&|djr&|djdkr|qS)r空白r)rHisspacerA)rfr\t)rSr%r&rhss )_get_sub_tokensr@)r.rRr`r%)rSrYr&r<]s   zJapaneseTokenizer._get_dtokenscCs|jsdSg}|D]S}||jjj}t|dkr|dq |jdkr/|||dgq ||jjj }t|t|krM||d}|||gq |||d||dgq |S)NrBF) r-splitr, SplitModer*rFappendr r<rv)r.rRrYr]sub_asub_brSr%r%r&ru}s&      z!JapaneseTokenizer._get_sub_tokenscCst|dt|S)NJapaneseTokenizer.score)rrscore_tokenization)r.examplesr%r%r&scores  r|cCs d|jiSNr r!r1r%r%r& _get_configs zJapaneseTokenizer._get_configconfigcCs|dd|_dSr)getr )r.rr%r%r& _set_configszJapaneseTokenizer._set_configc sdfddi}t|gS)NcfgcstSr0)srsly json_dumpsrr%r1r%r&sz,JapaneseTokenizer.to_bytes..)r to_bytes)r.kwargs serializersr%r1r&rs zJapaneseTokenizer.to_bytesdatac s.dfddi}t||gtj_S)Nrct|Sr0)rr json_loads)br1r%r&rz.JapaneseTokenizer.from_bytes..)r from_bytesr+r r,)r.rr deserializersr%r1r&rs zJapaneseTokenizer.from_bytespathc s,t|}dfddi}t||gdS)Nrcst|Sr0)r write_jsonrpr1r%r&rrz+JapaneseTokenizer.to_disk..)r ensure_pathto_diskr.rrrr%r1r&rs zJapaneseTokenizer.to_diskc s8t|}dfddi}t||gtj_S)Nrcrr0)rr read_jsonrr1r%r&rrz-JapaneseTokenizer.from_disk..)r r from_diskr+r r,rr%r1r&rs  zJapaneseTokenizer.from_diskr0)T)__name__ __module__ __qualname__rrstrr/r2rr_boolr<rurrrrrbytesrrrrrrr%r%r%r&r"*s( r"c@s(eZdZeeZeZeZ ddddZ dS)JapaneseDefaultsltrF) directionhas_case has_lettersN) rrrrDEFAULT_CONFIGrr stop_wordsrsyntax_iteratorswriting_systemr%r%r%r&rs rc@seZdZdZeZdS)JapanesejaN)rrrlangrDefaultsr%r%r%r&rsr morphologizerz token.morphz token.posTz@scorerszspacy.morphologizer_scorer.v1)model overwriteextendscorerg?)pos_acc morph_micro_fmorph_per_feat)assignsdefault_configdefault_score_weightsr$rnamerrrcCst|j|||||dS)N)rrr)rr#)r$rrrrrr%r%r&make_morphologizersrrl)rHrArJrGrKrMr:r*cCsjz(ddlm}m}|jjj|jjj|jjj|jjjd|}|j |d}|WSt y4t ddw)zSudachiPy is required for Japanese support, so check for it. It it's not available blow up and explain how to fix it. split_mode should be one of these values: "A", "B", "C", None->"A".r) dictionaryr,)Nr*rvC)modezJapanese support requires SudachiPy and SudachiDict-core (https://github.com/WorksApplications/SudachiPy). Install with `pip install sudachipy sudachidict_core` or install spaCy with `pip install spacy[ja]`.N) sudachipyrr, Tokenizerrxr*rvr Dictionarycreate ImportError)r rr,tokr%r%r&r+s$ r+cCst|tvrt|}||vr||dfS|r2||f}|tvr2t|\}}|dur.t|t|fS||fSt|tdfS)a2If necessary, add a field to the POS tag for UD mapping. Under Universal Dependencies, sometimes the same Unidic POS tag can be mapped differently depending on the literal token or its context in the sentence. This function returns resolved POSs for both token and next_token by tuple. N)rrrr)orthrAnext_tagorth_map tag_bigram current_posr[r%r%r&rDs    rDrrc Csdd|D}dd|d|kr"ttjj||dg}g}d}t|dkr2||fStdd|DdkrU|sCJt||d||ddg}dg}||fSt t ||D]t\}\}} |rgq\z ||d |} Wntyttjj||ddw| dkr|||| } | t| |d| | dd| d|| 7}| | | d|t|7}|dt|kr||dj d krd |d <|d7}q\|t|kr||d} | t| |d| | dd| d||fS) NcSsg|]}|jqSr%)rH)rfxr%r%r&rh"sz*get_dtokens_and_spaces..)r3r5rcSsg|]}|s|qSr%)rs)rfwordr%r%r&rh,riFr T)rmrw ValueErrorr E194formatrFrsrlr@r>indexryrH) rSr3gap_tagr5 text_dtokens text_spacestext_posirr^ word_startwr%r%r&r= sL"       "   r=r0)r*)rr)?rN collectionsrpathlibrtypingrrrrrr thinc.apir rr errorsr languager rpipelinerpipeline.morphologizerrrrsymbolsrtokensrrtrainingrrrrr#rrrrrtag_bigram_maprtag_mapr tag_orth_maprrrr(r"rrfactoryrrrlr+rDr=__all__r%r%r%r&sz                    " 5