o }oi @sddlmZmZddlmZddlZddlZddl m Z ddl m Z ddl mZddlmZddlmZddlmZmZdd lmZdd lmZmZdd lmZdd lmZd Z ej!j"ej#dej$fddZ%ej!j"e ddej&ej'fddZ(          d+de)de)deej*dee)de)de+de)de)d e)d!ee,ej&e dej&ej-fd"d#Z.ej!j"ee d$dd%ddefd&ee/d'e/de)de)d(edej0f d)d*Z1dS),)CallableOptionalN)Callback)DistributedDataParallelConfig)OptimizerConfig) lightningMockDataModule)FluxModelParamsMegatronFluxModel)pretrain) default_logtensorboard_logger) bf16_mixed)TimingCallbackflux_12breturncCstddddddd}|S)zMock Datamodule InitializationiT)image_himage_wmicro_batch_sizeglobal_batch_sizeimage_precachedtext_precachedr) data_moduler_/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/recipes/flux_12b.pyflux_mock_datamodule#srnamecCstjtttdS)z Factory function to create a FLUX 12B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the FLUX 12B model. ) flux_params)runConfigr r rrrrmodel2s r$rF{tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps callbacksc CsVtjtj|||||dtjtdddd} tjtjdd| |ddd||t| d d d } | S) ab Configure the NeMo Lightning Trainer for FLUX 12B model. This function sets up the distributed training strategy and other training parameters. Args: tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. callbacks (Optional[list[run.Config[Callback]]]): List of callback configurations. Returns: run.Config[nl.Trainer]: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=flux_12b ... Python API usage: >>> trainer_config = trainer(num_nodes=2, num_gpus_per_node=8) >>> print(trainer_config) Note: For more information on distributed training strategies, refer to the NeMo documentation on multi-GPU and multi-node training. T)check_for_nan_in_gradgrad_reduce_in_fp32)tensor_model_parallel_sizepipeline_model_parallel_sizecontext_parallel_sizesequence_parallelpipeline_dtypegradient_accumulation_fusionddpgpur2 Fi) acceleratoraccumulate_grad_batchesr0deviceslimit_test_batcheslimit_val_batcheslog_every_n_stepsr/r-pluginsstrategyuse_distributed_samplerval_check_interval)r"r#nlMegatronStrategyrTrainerr) r'r(r)r*r+r,r-r.r/r0rEtrainerrrrrK>s>,rK)targetr defaultdirr fncCsntj|tt||ttgdtt||t|ddtjt j tjt ddddddtjt j dddd d }|S) a Create a pre-training recipe for FLUX 12B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory flux_12b Python API usage: >>> recipe = pretrain_recipe(name="flux_12b_pretrain", num_nodes=1) >>> print(recipe) Note: For more details on pre-training LLMs with NeMo, see the pre-training guide in the `examples/llm/pretrain/` directory. )r-r.r0r)rNr rg-C6?Tr)lrbf16use_distributed_optimizer weight_decay)config)resume_if_existsresume_ignore_no_checkpointresume_past_end)r$rKdatalogoptimresume) r"Partialr$rKr#rrr rrHMegatronOptimizerModuler AutoResume)rNr r-r.rOreciperrrpretrain_recipes8$  r`) rrNNrFrr%r&N)2typingrrlightning.pytorchpytorchplnemo_runr"torch$lightning.pytorch.callbacks.callbackrmegatron.core.distributedrmegatron.core.optimizerrnemorrH9nemo.collections.diffusion.data.diffusion_mock_datamoduler ,nemo.collections.diffusion.models.flux.modelr r nemo.collections.llm.apir (nemo.collections.llm.recipes.log.defaultr r6nemo.collections.llm.recipes.precision.mixed_precisionrnemo.utils.exp_managerrNAMEclifactory autoconvertLightningDataModulerr#LightningModuler$intdtypeboollistrJrKstrr\r`rrrrs              S