o }oi@sBddlZddlmZddlmZddlmZGdddejZ dS)N)Any)loggingc szeZdZdZdfdd Zdejdejdejj j d e d e ddf d d Z dejdejdejj j d e d e ddf d dZZS)GarbageCollectionCallbackaCallback for synchronized manual Garbage Collection. This is required for distributed training as all processes on different rank need to synchronize to garbage collect at the same time, without which one process might hog or straggle all the rest of the processes. Migration from NeMo 1.0: When mitrating from NeMo1, - gc_interval = 0 implied no GC, simply do not add this callback to the trainer - gc_interval > 0, this config is maps => gc_interval_train - env-var:NEMO_MANUAL_GC_IN_VALIDATION=0 or doesn't exist => Set gc_interval_val to a very high value that it does not practically run. - env-var:NEMO_MANUAL_GC_IN_VALIDATION=1 => Set gc_interval_val to the same value as gc_interval Moving from boolean flag (NEMO_MANUAL_GC_IN_VALIDATION) to integer is to allow user to set a specific value based on the size of the validation datasets. Note: This callback does not run gc at the start or the end of training or validation. returnNcsH|dksJd|dksJdt||_||_td|_dS)a_summary_ Args: gc_interval (int, mandatory): Number of global train steps at which garbage collection is done. gc_interval_val (int, mandatory): Number of global validation steps at which garbage collection is done. rz;gc_interval_train should be an integer value larger than 0.z9gc_interval_val should be an integer value larger than 0.N)super__init__gc_interval_traingc_interval_valgcdisablevalidation_global_step)selfrr  __class__g/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/pytorch/callbacks/garbage_collection.pyr)s  z"GarbageCollectionCallback.__init__trainer pl_moduleoutputsbatch batch_idxcCs2|j|jdkrtd|jtdSdS)Nrz1Running garbage collection at train global_step: ) global_steprrinfor collectr rrrrrrrron_train_batch_end;s z,GarbageCollectionCallback.on_train_batch_endcCs@|jd7_|j|jdkrtd|jtdSdS)Nrz/Running garbage collection at validation step: )r r rrr rrrrron_validation_batch_endGs  z1GarbageCollectionCallback.on_validation_batch_end)rN)__name__ __module__ __qualname____doc__rplTrainerLightningModule utilitiestypes STEP_OUTPUTrintrr __classcell__rrrrrs:  r) r typingrlightning.pytorchpytorchr" nemo.utilsrCallbackrrrrrs