o ॵi1V@sddlZddlZddlZddlZddlmZmZmZddlZddl m Z ddl Z ddl mZddlmZddlmZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZm Z dd l!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/ddl0m1Z1ddl2m3Z3m4Z4m5Z5ddl6m7Z7m8Z8m9Z9e)Z:ej;ej|d%|d0d1}t}tBjBC}tD|j*|D]}|jE|||d <|jFdd%}t :d?"||tG|j||j|j.||tH|j|j|j.|\}}t :d@"||||j dkrtj|jdA"|}tA|j|dddd dt$%dd|} tI|||dB}t&| d+}t'7|}|9|Wdn 1s wY|JdC|||JdD|||}|K|q|dur5|j dkr5|LtBjBC|}t :dE"|MdFdS)Gau Args: kwargs: train_data (int): wave list with kaldi style for training cv_data (int): wave list with kaldi style for cross validation trans_data (str): transcription list with kaldi style, merge train and cv checkpoint (str): basemodel checkpoint, if None, default to use base.pt in model path tensorboard_dir (str): path to save tensorboard results, create 'tensorboard_dir' in work_dir by default need_dump (bool): wether to dump data with mapping tokens or not checkpointNz train/base.pttensorboard_dir tensorboard train_dataz'please config train data in dict kwargscv_dataz$please config cv data in dict kwargs trans_dataz/please config transcription data in dict kwargs need_dumpFrT preprocessor speed_perturbspec_augshufflezdump_train.txtz dump_cv.txt pin_memoryprefetch) batch_sizerd num_workersprefetch_factorzthe number of model params: {}z.pt$z.yamlr)Loaderz'Training with random initialized paramsepochrr= start_epochlr optimizerdevicegpucuda:r.r/zStart training...)exist_ok log_interval weight_decay)rorwming?gư>g{Gz?)modefactorpatiencemin_lr thresholdzinit.pt grad_clip grad_accum world_sizer@ local_rank max_epochszEpoch {} TRAIN info lr {}z0Epoch {} EVAL info cv_loss {:.6f}, cv_acc {:.2f}z{}.pt)rlrocv_lossz epoch/cv_losszepoch/lrTotal time spent: {:.2f} hours @)Nr>r4r5rArYr6r3rZr\r]r^r@r_rGcopydeepcopyr%r rErF train_dataset cv_datasetrr= dataloaderworkers_per_gputrain_dataloader evaluation cv_dataloader build_modelr$r!rCwarningformatrresubrHrIload FullLoaderrnrrr train_devicetorchcuda is_availabletorparallelDistributedDataParallelrBrJrKrLrDrrAdam parameters lr_schedulerReduceLROnPlateaurdatetimenowrange set_epoch param_groupsrrdict add_scalarstepclose total_seconds) rMargsrNtrain_checkpoint train_confcv_confdump_train_file dump_cv_file num_params info_pathinfosfin lr_last_epoch device_namerRrSrTwriterru optim_confrpr final_epochsave_model_pathtraining_config max_epoch totaltimerlrorcv_acc info_dictrWrWrXr=ksX                                             zKWSNearfieldTrainer.traincheckpoint_pathreturnc sd|_|durtj|rtd||}nU|jdura|dd}tj|jd|d|_td|jt |j|jd|d }t d>i||_| |j }t ||j|j}td ||j}td |j|d ddur|d ddurtd|d } |d } n td|j} |j} td| td| t|j d} d| dd<d| dd<d| d<d| d<d| d<|dddur|d| dd<t| | | |j|jddd} t| d|ddd|j jjj|j jjd d!d"} |d#ddus Jd$|d#}|d%dd&}i}dhd'hd'di}|D]O}t||j|j\}}i||<|||d(<dd)d*|D||d+<fd,d-|Dfd.d-|Dt||D]\}}||ddurr|||<qaq&d}|D]\}}||d/|d07}q|td1||d2d3dkotj !}|d4d5}|j"d6kr|rd7|j#}t$||_%| |j |_&t'||j&|j&(|j%|_&i}|d8ddur|d8|d8<ntj)|}tj|jd9||d8<|d8|_*tj|j*st+|j*t,d:t-j-.}t/|j&| |j%||}t-j-.|}t,d;0|1d<t || | |d=}t2d>i|}t3|dS)?ab Args: checkpoint_path (str): evaluating with ckpt or default average ckpt kwargs: test_dir (str): local path for saving test results test_data (str): wave list with kaldi style trans_data (str): transcription list with kaldi style average_num (int): the NO. to do model averaging(checkpoint_path==None) batch_size (int): batch size during evaluating keywords (str): keyword string, split with ',' gpu (int): evaluating with cpu/gpu: -1 for cpu; >=0 for gpu Nz evaluating with specific model: average_numrvavg_z.ptz!default average model not exist: T) dst_modelsrc_pathval_bestavg_numz(average model convert to kaldi network: zevaluating with average model: test_datar^z/evaluating with specific data and transcriptionz5evaluating with cross validation data during trainingz test data: z trans data: r`i filter_conf max_lengthr min_lengthFrarbrcrg batch_confrdrerf)rgrdpersistent_workersrhrikeywordszat least one keyword is needed ,ztoken_idcss|] }dt|VqdS)z%s N)r1.0irWrWrX sz/KWSNearfieldTrainer.evaluate.. token_strcg|]}|qSrWaddr)keywords_strsetrWrX z0KWSNearfieldTrainer.evaluate..crrWrr)keywords_idxsetrWrXrr(z) zToken set is: rrrmrqcpurrstest_dirtest_zStart evaluating...rr)rrr^ score_filerW)4avg_checkpointr4r5rArCrr>r6r%rrrrGr r]r^rrr rErFrrrrstripreplacesplitrzipitemsrrrrrr test_device test_modelrrbasenamerrBrDrrrrrrprint) rMrrrNeval_checkpointr avg_kwargs model_cvt kaldi_cvtrr^ test_conf test_datasettest_dataloader keywords_str keywords_listkeywords_tokenkeywords_tokenmapkeywordstrsindexestxtidx token_printuse_cudartesting_config base_namerr det_kwargs det_resultsrW)rrrXevaluate-s                             zKWSNearfieldTrainer.evaluatecCs@tj|j|dd}t|trt|dr|jSt|tjr|SdS)z Instantiate a pytorch model and return. By default, we will create a model using config from configuration file. You can override this method in a subclass. T)cfg_dicttrainingr$N) r from_pretrainedr3r0r hasattrr$rModule)rMrGr$rWrWrXrs zKWSNearfieldTrainer.build_modelrc CsFtdddur dtjd<tdddurdtjd<tdddur'dtjd<tdddur4dtjd<tdddurAd tjd<ttjd|_ttjd|_ttjd|_tjd|_tjd|_|d kr|jd krt d |jt j |jd d n|d krt j |jd d t t d |j|j|j|j|jdS)NRANK0 LOCAL_RANK WORLD_SIZE1 MASTER_ADDR localhost MASTER_PORT29500rz'init dist on multiple gpus, this gpu {}zenv://)backend init_methodz-RANK {}/{}/{}, Master addr:{}, Master port:{})r4getenvenvironintr@rr master_addr master_portrCrDrdistinit_process_groupr*barrierrM train_nodesrWrWrXr?sB         zKWSNearfieldTrainer.init_distcCsT|dkr|jdkrtd|jtdSdS|dkr(ttdSdS)Nrz*destory dist on multiple gpus, this gpu {})rrCrDrrrdestroy_process_grouprrrWrWrX uninit_dists   zKWSNearfieldTrainer.uninit_dist)r)__name__ __module__ __qualname__rr1rrr:r=rfloatrrr rr?r! __classcell__rWrWrUrXr#%s0BC   !r#)>rrr4rtypingrrrrtorch.distributed distributedrrI tensorboardXrrrtorch.utils.datarmodelscope.metainfor modelscope.modelsr r Mmodelscope.msdatasets.dataset_cls.custom_datasets.audio.kws_nearfield_datasetr modelscope.trainers.baser modelscope.trainers.builderrmodelscope.utils.checkpointrrmodelscope.utils.configrmodelscope.utils.constantrrmodelscope.utils.devicermodelscope.utils.loggerrmodelscope.utils.torch_utilsrkws_utils.batch_utilsrrrkws_utils.det_utilsrkws_utils.file_utilsrrrkws_utils.model_utilsrr r!rCregister_module"speech_kws_fsmn_char_ctc_nearfieldr#rWrWrWrXs@