o i@s|ddlZddlmZddlmZddlZddlZddlmZddl m Z ddl m Z ddl mZmZd d lmZmZmZmZmZmZmZed ed d ded ddded ddedddddedddddedddddedddd ded!d"d#d$d%dedd&d'd(dd)ed*d+d,d-df d.ed/ed0ed1ed2ed3ed4eed5ed6eed7efd8d9Zd:e d6ed;dfddd?ed d@dddAed dBdeddCdDdEded!d"d#d$d%deddFdGdHdfdIejdJedKedLeed5edMef dNdOZejdPddd>dQed d@dddAed dRdeddCdDdEded!d"d#d$d%deddFdGdHdfdIejdJedKedLeed5edMef dSdTZ dUdVZ!dS)WN)Path)Optional)msg)util)Language)convert_vectorsinit_nlp)ArgOpt import_codeinit_cliparse_config_overrides setup_gpushow_validation_errorvectors.z(The language of the nlp object to create)helpzVectors file in Word2Vec formatT)rexistszPipeline output directoryz--prunez-pz&Optional number of vectors to prune toz --truncatez-tzFOptional number of vectors to truncate to when reading in vectors filedefaultz--modez-mzVectors mode: default or floretz--namez-nz?Optional name for the word vectors, e.g. en_core_web_lg.vectorsFz --verbosez-Vz-VVz/Display more information for debugging purposesz--lexemes-jsonlz-jz+Location of JSONL-formatted attributes file)rhiddenORTHz--attrz-az?Optional token attribute to use for vectors, e.g. LOWER or NORMlang vectors_loc output_dirprunetruncatemodenameverbose jsonl_locattrc Cs|r tjtjtd|dt|} |dur!t| |t | |||||| dt dt | j j d| |t d|dS)zConvert word vectors for use with spaCy. Will export an nlp object that you can use in the [initialize] block of your config to initialize a model with vectors. z(Creating blank nlp object for language ''N)rrrrr"zSuccessfully converted z vectorszSaved nlp object with vectors to output directory. You can now use the path to it in your config as the 'vectors' setting in [initialize].)rloggersetLevelloggingDEBUGrinfoget_lang_classupdate_lexemesrgoodlenvocabrto_diskresolve) rrrrrrrr r!r"nlpr1K/home/ubuntu/.local/lib/python3.10/site-packages/spacy/cli/init_pipeline.pyinit_vectors_clis*   r3r0returncCs@t|}|D]}d|vrq|j|d}|jdi|qdS)Nsettingsorthr1)srsly read_jsonlr- set_attrs)r0r! lex_attrsattrslexemer1r1r2r*Bs r*)allow_extra_argsignore_unknown_options)context_settingsrzPath to config file)rr allow_dashz&Output directory for the prepared dataz--codez-czNPath to Python file with additional code (registered functions) to be importedz--gpu-idz-gzGPU ID or -1 for CPUctx config_path output_path code_pathuse_gpuc Cs|r tjtjt|j}t|t|t |tj ||d}Wdn1s,wYt ddt ||d}Wdn1sGwY| |t d|dS)N overridesF hint_fillrEzSaved initialized pipeline to )rr$r%r&r'rargsr rr load_configr r.rr+ rArBrCrDr rErGconfigr0r1r1r2init_pipeline_cliLs    rOlabels)r?zOutput directory for the labelsc Cs|r tjtj|s|jddt|j}t |t |t |tj ||d}Wdn1s6wYt ddt ||d}Wdn1sQwYt||dS)zGenerate JSON files for the labels in the data. This helps speed up the training process, since spaCy won't have to preprocess the data to extract the labels.T)parentsrFNFrHrJ)rr$r%r&r'rmkdirrrKr rrrLr _init_labelsrMr1r1r2init_labels_clihs    rTcCsj|jD]/\}}t|dddur)||d}t||jtd|d|qtd|dqdS)N label_dataz.jsonz!Saving label data for component 'z' to z#No label data found for component 'r#)pipelinegetattrr7 write_jsonrUrr+r()r0rCr component output_filer1r1r2rSsrS)"r&pathlibrtypingrr7typerwasabirrlanguagertraining.initializerr _utilr r r rrrrcommandstrintboolr3r*ContextrOrTrSr1r1r1r2s     $      *