o i/$@sddlZddlZddlZddlZddlZddlmZmZm Z ddl m Z m Z m Z ddZGddde jZGdd d e jjZGd d d eZdS) N)Variablecudatraining) gather_grads gather_params scatter_gradsc Cstt}|D].}t|}|dur&|}||}|t||7<Wdn1s0wYqtddt |DS)zCalculate the norm of the array. Args: arr (numpy.ndarray) Returns: Float: Sum of the norm calculated from the given array. NcSsg|]}t|qS)float).0irr\/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/chainer_backend/rnn/training.py 'szsum_sqnorm..) collections defaultdictr rget_device_from_arrayraveldotintsumsix itervalues)arrsq_sumxdevsrrr sum_sqnorms   rc2eZdZdZd fdd ZddZddZZS) CustomUpdatera%Custom updater for chainer. Args: train_iter (iterator | dict[str, iterator]): Dataset iterator for the training dataset. It can also be a dictionary that maps strings to iterators. If this is just an iterator, then the iterator is registered by the name ``'main'``. optimizer (optimizer | dict[str, optimizer]): Optimizer to update parameters. It can also be a dictionary that maps strings to optimizers. If this is just an optimizer, then the optimizer is registered by the name ``'main'``. converter (espnet.asr.chainer_backend.asr.CustomConverter): Converter function to build input arrays. Each batch extracted by the main iterator and the ``device`` option are passed to this function. :func:`chainer.dataset.concat_examples` is used by default. device (int or dict): The destination device info to send variables. In the case of cpu or single gpu, `device=-1 or 0`, respectively. In the case of multi-gpu, `device={"main":0, "sub_1": 1, ...}`. accum_grad (int):The number of gradient accumulation. if set to 2, the network parameters will be updated once in twice, i.e. actual batchsize will be doubled. cs4tt|j||||dd|_||_d|_||_dS)N) converterdevicerT)superr__init__ forward_count accum_gradstartr!)self train_iter optimizerr r!r% __class__rr r#Cs  zCustomUpdater.__init__cCs|d}|d}|}|||j}|jr |jd|_|j||j}| | |j d7_ |j |jkr?dSd|_ t tdd|jdD}td|t|retd n||jdS) z'Main update routine for Custom Updater.mainFrNrcSg|]}|jqSrgradr prrr r fz-CustomUpdater.update_core.. grad norm={}&grad norm is nan. Do not update model.) get_iterator get_optimizernextr r!r&target cleargradsr%backwardunchain_backwardr$npsqrtrparamslogginginfoformatmathisnanwarningupdate)r'r(r)batchrloss grad_normrrr update_coreNs,      zCustomUpdater.update_corecC(||jdkr|jd7_dSdSNrrrIr$ iterationr'rrr rEo zCustomUpdater.updater__name__ __module__ __qualname____doc__r#rIrE __classcell__rrr*r r*s  !rcr) CustomParallelUpdateraCustom Parallel Updater for chainer. Defines the main update routine. Args: train_iter (iterator | dict[str, iterator]): Dataset iterator for the training dataset. It can also be a dictionary that maps strings to iterators. If this is just an iterator, then the iterator is registered by the name ``'main'``. optimizer (optimizer | dict[str, optimizer]): Optimizer to update parameters. It can also be a dictionary that maps strings to optimizers. If this is just an optimizer, then the optimizer is registered by the name ``'main'``. converter (espnet.asr.chainer_backend.asr.CustomConverter): Converter function to build input arrays. Each batch extracted by the main iterator and the ``device`` option are passed to this function. :func:`chainer.dataset.concat_examples` is used by default. device (torch.device): Device to which the training data is sent. Negative value indicates the host memory (CPU). accum_grad (int):The number of gradient accumulation. if set to 2, the network parameters will be updated once in twice, i.e. actual batchsize will be doubled. rcs:tt|j||||dddlm}||_d|_||_dS)N)r devicesr)nccl)r"rWr# cupy.cudarYr%r$)r' train_itersr)r rXr%rYr*rr r#s   zCustomParallelUpdater.__init__c Cs||dt|jd|d}|d}|||jd}|j ||j }| | tj j}|jdurbt|j }|j|jj|jj|j|jj|jjd|jt|j |~|jd7_|j|j krx WddSd|_ttdd|jdD}t d !|t"#|rt$d n|%|j &|jdurt'|j }|j(|jj|j|jjd|jWddSWddS1swYdS) z3Main Update routine of the custom parallel updater.)rENrr,NrcSr-rr.r0rrr r r2z5CustomParallelUpdater.update_core..Fr3r4)) setup_workers _send_messagerDevice_devicesr6r5r7r _masterr%r:r;StreamnullcommrreducedataptrsizerY NCCL_FLOATNCCL_SUMrr$r<r=rr8r>r?r@rArBrCrDrEr9rbcast) r'r)rFrrG null_streamggrHgprrr rIsX           -"z!CustomParallelUpdater.update_corecCrJrKrLrNrrr rErOzCustomParallelUpdater.updaterPrQrrr*r rWus  8rWc@s"eZdZdZdddZddZdS) CustomConverterz\Custom Converter. Args: subsampling_factor (int): The subsampling factor. rcCs ||_dSNsubsampling_factor)r'rqrrr r#s zCustomConverter.__init__cs|dkrtjntt|dksJ|d\}}jdkr%fdd|D}dd|D}fdd|D}j|jd}fd d|D}|||fS) aiPerform sabsampling. Args: batch (list): Batch that will be sabsampled. device (device): GPU device. Returns: chainer.Variable: xp.array that sabsampled from batch. xp.array: xp.array of the length of the mini-batches. chainer.Variable: xp.array that sabsampled from batch. rrcs$g|]}|ddjddfqSrorpr rrNrr r s$z,CustomConverter.__call__..cSsg|]}|jdqS)r)shapersrrr r sc g|] }tj|jdqSdtype)rarrayfloat32rsxprr r  rwcrurv)rryint32)r yr{rr r r})rcupyr<lenrqryr~)r'rFr!xsysilensr)r'r|r __call__s   zCustomConverter.__call__NrP)rRrSrTrUr#rrrrr rns  rn)rr?rBnumpyr<rchainerrrr7chainer.training.updaters.multiprocess_parallel_updaterrrrrStandardUpdaterrupdatersMultiprocessParallelUpdaterrWobjectrnrrrr sKd