o ̳i @slddlmZmZddlmZddlmZmZddl m Z ddZdej de d eee e fd d fd d Zd S))OptionalUnion)nn)checkpoint_wrapperCheckpointImpl) checkpointcCs|dkrt|tjtdddS|dkrE t|}|dks"td|tjddtj d7_ |r9tj |dkrCt|tjtdddS|St d ) NfullF)checkpoint_impl checkpoint_fn use_reentrantpreserve_rng_state selectiverzOSelective layer AC policy (every_x_layer) expects a positive integer, received _countz]Unknown AC type or AC config. Only selective op and selective layer ac implemented currently.) ptd_checkpoint_wrapperr NO_REENTRANTrint ValueErrorr__dict__ setdefaultrNotImplementedError)moduleac_modeac_style every_x_layerrR/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/training/activations.pyrs:  rmodelr ac_optionreturnNcCs6t|jD]\}}|dvrt|||}||j|<qdS)acUtility to setup activation checkpointing and wrap the model for checkpointing. Args: model (nn.Module): Model to setup activation checkpointing. ac_mode (str): Activation checkpointing mode. ['none', 'full', 'selective'] ac_option (Optional[Union[int, str]]): Activation checkpointing option. If ac_mode is "selective", ac_option can be an integer or a string representing the number of layers to checkpoint. If ac_mode is "selective" and ac_option is "op", then selective op ac is run. If ac_mode is "none" or "full", ac_option is ignored. )rr N) enumeratelayersr)rrrlayer_idtransformer_blockrrr(apply_selective_activation_checkpointingBs r$)typingrrtorchr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrrrtorch.utils.checkpointrModulestrrr$rrrrs  /