o :iK@s0ddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl Z ddl mZddlmZddlmZddlmZddlmZmZeZ        d?ded e ejefd e ejd e ejd e eeefd e ede ede ede ededdfddZ    d@ded e ejd e ejd e ejdedeeeff ddZ dAdeded ejdedeeeff ddZ!dedededed ejdeeeff d d!Z"dBd"ed#ede efd$d%Z#e $d&fd'ed(ed)e j$deee j%ffd*d+Z& dBd"ed,edefd-d.Z' dCd ed ejd ejd e ejde eeeff d/d0Z( 1dDd2eee j%fd3eee j%fd4e)d5e)d6e)deee j%ff d7d8Z*d eee j%fd9e ejefd ejddfd:d;Z+       dEd"edZ,dS)FN)Path)AnyDictListOptionalUnion) CutSampler)DistributedDataParallel) Optimizer) AttributeDict GradScalerfilenamemodel model_avg model_emaparams optimizer schedulerscalersamplerrankreturnc Cs| dkrdStd|t|tr|j}||dur!|nd|dur*|nd|dur3|nd|dur<|ndd} |durN|tj| d<|dur\|tj| d<|rq| D]\} } | | vslJ| | | <qbt | |dS)ahSave training information to a file. Args: filename: The checkpoint filename. model: The model to be saved. We only save its `state_dict()`. model_avg: The stored model averaged from the start of training. model_ema: The EMA version of model. params: User defined parameters, e.g., epoch, loss. optimizer: The optimizer to be saved. We only save its `state_dict()`. scheduler: The scheduler to be saved. We only save its `state_dict()`. scalar: The GradScaler to be saved. We only save its `state_dict()`. sampler: The sampler used in the labeled training dataset. We only save its `state_dict()`. rank: Used in DDP. We save checkpoint only for the node whose rank is 0. Returns: Return None. rNzSaving checkpoint to )rrr grad_scalerrrr) logginginfo isinstanceDDPmodule state_dicttotorchfloat32itemssave) r rrrrrrrrr checkpointkvr'0/home/ubuntu/LuxTTS/zipvoice/utils/checkpoint.pysave_checkpoint&s((   r)Fstrictc Cs6td|tj|ddd}|durctt|ddrPtd|}|d}| D]}d d |} | | ||<q0t |d ksHJ|j ||d ntd |j |d|d | d|dur~d |vr~td|j |d |d | d |durd|vrtd|j |d|d | d|S)NLoading checkpoint from cpuF map_location weights_onlyrmodule.Loading checkpoint saved by DDP{}.{}rrr*Loading checkpointrzLoading averaged modelrzLoading ema model)rrr loadnextiter startswithdebugrkeysformatpoplenload_state_dict) r rrrr*r$dst_state_dictsrc_state_dictkeysrc_keyr'r'r(load_checkpointks0         rCT extend_sizec Cstd|tj|ddd}|durztt|ddrItd|}|d}|D]}d d |}| |||<q0t |d ksHJn td |d}|d |d <|d } |d | d| ddf<| |d <|j ||ddSdS)Nr+r,Fr-rr0r1r2rrr4zspk_embed.weightz embed.weightr3) rrr r5r6r7r8rr:r;r<r=r>) r rDrr*r$r?r@rArB embed_weightr'r'r(!load_checkpoint_extend_vocab_sizes&     rF in_proj_key out_proj_keydimc CsDtd|tj|ddd}|dur tt|ddrFtdt}|d}|D] }| ||| d<q0t |dksEJn td |d}t |} | D]}||vrd |vr| |} tj | ddd|fd | ddd|fd | dd||d f| dd|d dfd | dd|d dfd gd d ||d d<| ||d d<d|vr| |} | ||dd<| ||dd<||vrd |vr| |} tj | | gdd ||d d<| ||d d<qWd|vr| |} tj | | gdd ||dd<| ||dd<qW|j|dddSdS)Nr+r,Fr-rr0r1rr4weight)rIz0.weightz1.weightbiasz0.biasz1.biasTr3)rrr r5r6r7r8dictr:r<lstripr=listcatreplacer>) r rGrHrIrr$r?r@rAr:rJrMr'r'r(-load_checkpoint_copy_proj_three_channel_alters^            rSout_dir iterationcstt|d}td}g}|D]}||}|s%td|q|t| d|fqt |dddd}d krJfd d |D}|Sfd d |D}|S) aFind all available checkpoints in a directory. The checkpoint filenames have the form: `checkpoint-xxx.pt` where xxx is a numerical value. Assume you have the following checkpoints in the folder `foo`: - checkpoint-1.pt - checkpoint-20.pt - checkpoint-300.pt - checkpoint-4000.pt Case 1 (Return all checkpoints):: find_checkpoints(out_dir='foo') Case 2 (Return checkpoints newer than checkpoint-20.pt, i.e., checkpoint-4000.pt, checkpoint-300.pt, and checkpoint-20.pt) find_checkpoints(out_dir='foo', iteration=20) Case 3 (Return checkpoints older than checkpoint-20.pt, i.e., checkpoint-20.pt, checkpoint-1.pt):: find_checkpoints(out_dir='foo', iteration=-20) Args: out_dir: The directory where to search for checkpoints. iteration: If it is 0, return all available checkpoints. If it is positive, return the checkpoints whose iteration number is greater than or equal to `iteration`. If it is negative, return the checkpoints whose iteration number is less than or equal to `-iteration`. Returns: Return a list of checkpoint filenames, sorted in descending order by the numerical value in the filename. z/checkpoint-[0-9]*.ptzcheckpoint-([0-9]+).ptzInvalid checkpoint filename TcSs|dS)Nrr')xr'r'r(sz"find_checkpoints..)reverserArcs g|] }|dkr|dqSrrVr'.0icrUr'r( s z$find_checkpoints..cs"g|] }|d kr|dqSrZr'r[r^r'r(r_"s") rPglobrecompilesearchrwarnappendintgroupsorted)rTrU checkpointspatterniter_checkpointscresultansr'r^r(find_checkpointss(  ror,filename_start filename_enddevicecCstj||dd}tj||dd}|d}|d}|||}|d}|||}||}|dks4J|||} d| } |d} |d} | } t| | d| | | d | S) aAverage model parameters over the range with given start model (excluded) and end model. Let start = batch_idx_train of model-start; end = batch_idx_train of model-end; interval = end - start. Then the average model over range from start (excluded) to end is (1) avg = (model_end * end - model_start * start) / interval. It can be written as (2) avg = model_end * weight_end + model_start * weight_start, where weight_end = end / interval, weight_start = -start / interval = 1 - weight_end. Since the terms `weight_end` and `weight_start` would be large if the model has been trained for lots of batches, which would cause overflow when multiplying the model parameters. To avoid this, we rewrite (2) as: (3) avg = (model_end + model_start * (weight_start / weight_end)) * weight_end The model index could be epoch number or iteration number. Args: filename_start: Checkpoint filename of the start model. We assume it is saved by :func:`save_checkpoint`. filename_end: Checkpoint filename of the end model. We assume it is saved by :func:`save_checkpoint`. device: Move checkpoints to this device before averaging. Fr-average_periodbatch_idx_trainrrVr?) state_dict_1 state_dict_2weight_1weight_2scaling_factor)r r5average_state_dict)rprqrrstate_dict_startstate_dict_endrsbatch_idx_train_startbatch_idx_train_endinterval weight_end weight_start model_end model_startavgr'r'r('average_checkpoints_with_averaged_model's0$  rtopkcCsx|dksJ||dkrdSt|}t|dkr"td|dSt||kr*dS||d}|D]}t|q2dS)aNRemove checkpoints from the given directory. We assume that checkpoint filename has the form `checkpoint-xxx.pt` where xxx is a number, representing the number of processed batches when saving that checkpoint. We sort checkpoints by filename and keep only the `topk` checkpoints with the highest `xxx`. Args: out_dir: The directory containing checkpoints to be removed. topk: Number of checkpoints to keep. rank: If using DDP for training, it is the rank of the current node. Use 0 if no DDP is used for training. rVrNzNo checkpoints found in )ror=rrdosremove)rTrrri to_removerlr'r'r(remove_checkpointsks    rcCsl|jd|jdd}|sJ|dt||||dd}|jdkr4gd}|D]}||||<q+|S)aLoad checkpoint from file. If params.start_epoch is larger than 1, it will load the checkpoint from `params.start_epoch - 1`. Apart from loading state dict for `model` and `optimizer` it also updates `best_train_epoch`, `best_train_loss`, `best_valid_epoch`, and `best_valid_loss` in `params`. Args: params: The return value of :func:`get_params`. model: The training model. Returns: Return a dict containing previously saved training info. zepoch-rV.ptz does not exist!T)rrrr*)best_train_epochbest_valid_epochrtbest_train_lossbest_valid_loss)exp_dir start_epochis_filerC)rrrrr saved_paramsr:r%r'r'r(resume_checkpoints rrurvrwrxryrzc Cst}|D]\}}|}||vrq|||<qt|} | D]"}||}t|rC||9}|||j||jd|7}||9}q!dS)zAverage two state_dict with given weights: state_dict_1 = (state_dict_1 * weight_1 + state_dict_2 * weight_2) * scaling_factor It is an in-place operation on state_dict_1 itself. )rrN) rNr"data_ptrrPvaluesr is_floating_pointrrr) rvrwrxryrzuniquedr%r& v_data_ptr uniqued_namesr'r'r(r{s   r{ model_curcCsH|j|j}d|}t|tr|j}|}|}t||||ddS)a]Update the averaged model: model_avg = model_cur * (average_period / batch_idx_train) + model_avg * ((batch_idx_train - average_period) / batch_idx_train) Args: params: User defined parameters, e.g., epoch, loss. model_cur: The current model. model_avg: The averaged model to be updated. rV)rvrwrxryN)rsrtrrrrr{)rrr weight_cur weight_avgcurrr'r'r(update_averaged_models   rglobal_batch_idxc CsDt|}|jddd|d|d} t| |||||||| d dS)aZSave training info after processing given number of batches. Args: out_dir: The directory to save the checkpoint. global_batch_idx: The number of batches processed so far from the very start of the training. The saved checkpoint will have the following filename: f'out_dir / checkpoint-{global_batch_idx}.pt' model: The neural network model whose `state_dict` will be saved in the checkpoint. model_avg: The stored model averaged from the start of training. params: A dict of training configurations to be saved. optimizer: The optimizer used in the training. Its `state_dict` will be saved. scheduler: The learning rate scheduler used in the training. Its `state_dict` will be saved. scaler: The scaler used for mix precision training. Its `state_dict` will be saved. sampler: The sampler used in the training dataset. rank: The rank ID used in DDP training of the current node. Set it to 0 if DDP is not used. T)parentsexist_okz checkpoint-r) r rrrrrrrrN)rmkdirr)) rTrrrrrrrrrr r'r'r(%save_checkpoint_with_global_batch_idxs+ r)NNNNNNNr)NNNF)T)r)N)ru)NNNNNNr)-r`rrrapathlibrtypingrrrrrr torch.nnnnlhotse.dataset.sampling.basertorch.nn.parallelr r torch.optimr zipvoice.utils.commonr r objectLRSchedulerTypeModulestrrfr)boolrCrFrSrorrTensorrrrfloatr{rrr'r'r'r(sP         G  )    ;B  G * 6       &