o }oiM @sddlmZddlmZddlZddlZddlm Z ddl m Z ddl m Z dZejjeddejejfd d Z dd ededejfddZejje ed   ddeeded ededejf ddZdS))OptionalN)pretrain)MockDataModule) llama3_8b llama3_8b_64k)namereturncCst}d|j_|S)a Factory function to create a Llama3 8B model configuration with 64k sequence length. Returns: run.Config[pl.LightningModule]: Configuration for the Llama3 8B model with 64k sequence length. Examples: CLI usage: $ nemo llm pretrain model=llama3_8b_64k ... Python API usage: >>> model_config = model() >>> print(model_config) )rmodelconfig seq_length) model_configr^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/llama3_8b_64k.pyr sr  num_nodesnum_gpus_per_nodec Cstjddtjddd||dS)a Configure the NeMo Lightning Trainer for Llama3 8B model with 64k sequence length. This function sets up the distributed training strategy optimized for long sequences. Args: num_nodes (int, optional): Number of compute nodes to use. Defaults to 4. num_gpus_per_node (int, optional): Number of GPUs per node. Defaults to 8. Returns: run.Config: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=llama3_8b_64k ... Python API usage: >>> trainer_config = trainer(num_nodes=4, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses significantly increased parallelism to handle the long sequence length efficiently. rNT)tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismrr)rtrainertorchbfloat16rrrrrr2sr)targetrdefaultdirrcCs@tj||||d}t|_t||d|_tjtdddd|_|S)ay Create a pre-training recipe for Llama3 8B model with 64k sequence length. This function sets up a complete configuration for pre-training, including model, trainer, and data settings optimized for 64k sequence length. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int, optional): Number of compute nodes to use. Defaults to 4. num_gpus_per_node (int, optional): Number of GPUs per node. Defaults to 8. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory llama3_8b_64k $ nemo llm pretrain --factory "llama3_8b_64k(num_nodes=4, name='my_64k_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="llama3_8b_64k_pretrain", num_nodes=4) >>> print(recipe) Note: This recipe is optimized for handling long sequences (64k) compared to the standard 8k version. It requires significant computational resources due to the extended sequence length. )rr!rrrr i)r global_batch_sizemicro_batch_size)rpretrain_reciper rrunConfigrdata)r!rrrreciperrrr%Ys #r%)rr)Nr rr)typingrlightning.pytorchpytorchplnemo_runr&rnemo.collections.llm.apir"nemo.collections.llm.gpt.data.mockrnemo.collections.llm.recipesrNAMEclifactoryr'LightningModuler intrstrPartialr%rrrrsF       '