o i(.@sdZddlZddlZddlZddlZddlZddlmZddlm Z ddlm Z ddl m Z ddlmZmZmZdd ZGd d d e jZGd d d e jjZGddde jZGdddeZdS)z7Class Declaration of Transformer's Training Subprocess.N)cuda) functions)training) extension) gather_grads gather_params scatter_gradsc Cstt}|D].}t|}|dur&|}||}|t||7<Wdn1s0wYqtddt |DS)zCalculate the norm of the array. Args: arr (numpy.ndarray) Returns: Float: Sum of the norm calculated from the given array. NcSsg|]}t|qS)float).0ir r d/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/chainer_backend/transformer/training.py 'szsum_sqnorm..) collections defaultdictr rget_device_from_arrayraveldotintsumsix itervalues)arrsq_sumxdevsr r r sum_sqnorms   rc2eZdZdZd fdd ZddZddZZS) CustomUpdatera%Custom updater for chainer. Args: train_iter (iterator | dict[str, iterator]): Dataset iterator for the training dataset. It can also be a dictionary that maps strings to iterators. If this is just an iterator, then the iterator is registered by the name ``'main'``. optimizer (optimizer | dict[str, optimizer]): Optimizer to update parameters. It can also be a dictionary that maps strings to optimizers. If this is just an optimizer, then the optimizer is registered by the name ``'main'``. converter (espnet.asr.chainer_backend.asr.CustomConverter): Converter function to build input arrays. Each batch extracted by the main iterator and the ``device`` option are passed to this function. :func:`chainer.dataset.concat_examples` is used by default. device (int or dict): The destination device info to send variables. In the case of cpu or single gpu, `device=-1 or 0`, respectively. In the case of multi-gpu, `device={"main":0, "sub_1": 1, ...}`. accum_grad (int):The number of gradient accumulation. if set to 2, the network parameters will be updated once in twice, i.e. actual batchsize will be doubled. cs>tt|j||||d||_d|_d|_||_tddS)zInitialize Custom Updater.) converterdevicerTz&using custom converter for transformerN) superr__init__ accum_grad forward_countstartr"loggingdebug)self train_iter optimizerr!r"r% __class__r r r$Cs zCustomUpdater.__init__cCs|d}|d}|}|||j}|jr |jd|_|j||j}| |j d7_ |j |jkr;dSd|_ t t dd|jdD}td|t|ratd n||jdS) z/Process main update routine for Custom Updater.mainFr NrcSg|]}|jqSr gradr pr r r rez-CustomUpdater.update_core.. grad norm={}&grad norm is nan. Do not update model.) get_iterator get_optimizernextr!r"r'target cleargradsr%backwardr&npsqrtrparamsr(infoformatmathisnanwarningupdate)r*r+r,batchrloss grad_normr r r update_coreOs*      zCustomUpdater.update_corecC(||jdkr|jd7_dSdS)zUpdate step for Custom Updater.rr NrJr& iterationr*r r r rFn zCustomUpdater.updater __name__ __module__ __qualname____doc__r$rJrF __classcell__r r r-r r*s  rcr) CustomParallelUpdateraCustom Parallel Updater for chainer. Defines the main update routine. Args: train_iter (iterator | dict[str, iterator]): Dataset iterator for the training dataset. It can also be a dictionary that maps strings to iterators. If this is just an iterator, then the iterator is registered by the name ``'main'``. optimizer (optimizer | dict[str, optimizer]): Optimizer to update parameters. It can also be a dictionary that maps strings to optimizers. If this is just an optimizer, then the optimizer is registered by the name ``'main'``. converter (espnet.asr.chainer_backend.asr.CustomConverter): Converter function to build input arrays. Each batch extracted by the main iterator and the ``device`` option are passed to this function. :func:`chainer.dataset.concat_examples` is used by default. device (torch.device): Device to which the training data is sent. Negative value indicates the host memory (CPU). accum_grad (int):The number of gradient accumulation. if set to 2, the network parameters will be updated once in twice, i.e. actual batchsize will be doubled. r csDddlm}tt|j||||d||_d|_||_tddS)z#Initialize custom parallel updater.r)nccl)r!devicesz-using custom parallel updater for transformerN) cupy.cudarXr#rWr$r%r&r(r))r* train_itersr,r!rYr%rXr-r r r$s  zCustomParallelUpdater.__init__c Cs||dt|jd|d}|d}|||jd}|j ||j }| tj j }|jdur^t|j }|j|jj|jj|j|jj|jjd|jt|j |~|jd7_|j|j krt WddSd|_ttdd|jdD}td |t!"|rt#d n|$|j %|jdurt&|j }|j'|jj|j|jjd|jWddSWddS1swYdS) z8Process main update routine for Custom Parallel Updater.)rFNrr/Nr cSr0r r1r3r r r rr5z5CustomParallelUpdater.update_core..Fr6r7)( setup_workers _send_messagerDevice_devicesr9r8r:r!_masterr%r=StreamnullcommrreducedataptrsizerX NCCL_FLOATNCCL_SUMrr&r>r?rr;r@r(rArBrCrDrErFr<rbcast) r*r,rGrrH null_streamggrIgpr r r rJsV           +"z!CustomParallelUpdater.update_corecCrK)z(Update step for Custom Parallel Updater.rr NrLrNr r r rFrOzCustomParallelUpdater.updaterPrQr r r-r rWus  6rWc@sLeZdZdZ     dddZddZd d Zd d Zd dZddZ dS) VaswaniRuleaTrainer extension to shift an optimizer attribute magically by Vaswani. Args: attr (str): Name of the attribute to shift. rate (float): Rate of the exponential shift. This value is multiplied to the attribute at each call. init (float): Initial value of the attribute. If it is ``None``, the extension extracts the attribute at the first call and uses it as the initial value. target (float): Target value of the attribute. If the attribute reaches this value, the shift stops. optimizer (~chainer.Optimizer): Target optimizer to adjust the attribute. If it is ``None``, the main optimizer of the updater is used. N?cCs@||_|d||_|d|_||_||_||_d|_d|_dS)z"Initialize Vaswani rule extension.grN)_attr_d_inv05_warmup_steps_inv15_init_target _optimizer_t _last_value)r*attrd warmup_stepsinitr;r,scaler r r r$s   zVaswaniRule.__init__cCsT||}|jdur|jd|j|_|jdur!|||jdS|||jdS)zInitialize Optimizer values.Nrp)_get_optimizerrursrtry _update_value)r*trainerr,r r r initializes   zVaswaniRule.initializecCsF|jd7_||}|jt|jd|j|j}|||dS)zForward extension.r rqN)rxrrsminrtr)r*rr,valuer r r __call__ s  zVaswaniRule.__call__cCs |d|j|_|d|j|_dS)zSerialize extension.rxryN)rxry)r* serializerr r r serializeszVaswaniRule.serializecCs|jp|jdS)zObtain optimizer from trainer.r/)rwupdaterr9)r*rr r r rszVaswaniRule._get_optimizercCst||j|||_dS)z!Update requested variable values.N)setattrrrry)r*r,rr r r rs zVaswaniRule._update_value)roNNNrp) rRrSrTrUr$rrrrrr r r r rns    rnc@s eZdZdZddZddZdS)CustomConverterz\Custom Converter. Args: subsampling_factor (int): The subsampling factor. cCsdS)zInitialize subsampling.Nr rNr r r r$*szCustomConverter.__init__cCsPt|dksJ|d\}}tj|ddj}tjdd|Dtjd}|||fS)aPerform subsampling. Args: batch (list): Batch that will be sabsampled. device (chainer.backend.Device): CPU or GPU device. Returns: chainer.Variable: xp.array that are padded and subsampled from batch. xp.array: xp.array of the length of the mini-batches. chainer.Variable: xp.array that are padded and subsampled from batch. r r)paddingcSsg|]}|jdqS)r)shape)r rr r r rAsz,CustomConverter.__call__..)dtype)lenF pad_sequencerer>arrayint32)r*rGr"xsysilensr r r r.s   zCustomConverter.__call__N)rRrSrTrUr$rr r r r r"s r)rUrr(rCnumpyr>rchainerrrrrchainer.trainingr7chainer.training.updaters.multiprocess_parallel_updaterrrrrStandardUpdaterrupdatersMultiprocessParallelUpdaterrW Extensionrnobjectrr r r r s     KdI