o }oi @sddlmZddlmZddlZddlZddlm Z m Z ddl m Z ddl mZddlmZdZejjedd ejejfd d Z ddeded ejfddZejje ed   ddeedededed ejf ddZdS))OptionalN)finetunepretrain)MockDataModule)SquadDataModule) llama3_70bllama3_70b_16k)namereturncCst}d|j_|S)a Factory function to create a Llama3 70B model configuration with 16k sequence length. Returns: run.Config[pl.LightningModule]: Configuration for the Llama3 70B model with 16k sequence length. Examples: CLI usage: $ nemo llm pretrain model=llama3_70b_16k ... Python API usage: >>> model_config = model() >>> print(model_config) @)rmodelconfig seq_length) model_configr_/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/llama3_70b_16k.pyr sr  num_nodesnum_gpus_per_nodec Cstjddtjddd||dS)aD Configure the NeMo Lightning Trainer for Llama3 70B model with 16k sequence length. This function sets up the distributed training strategy optimized for the large 70B model with longer sequences. Args: num_nodes (int, optional): Number of compute nodes to use. Defaults to 4. num_gpus_per_node (int, optional): Number of GPUs per node. Defaults to 8. Returns: run.Config: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=llama3_70b_16k ... Python API usage: >>> trainer_config = trainer(num_nodes=4, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses extensive parallelism to handle the large model size and longer sequence length efficiently. rNT)tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismrr)rtrainertorchbfloat16rrrrrr3sr)targetr defaultdirr cCs@tj||||d}t|_t||d|_tjtdddd|_|S)aK Create a pre-training recipe for Llama3 70B model with 16k sequence length. This function sets up a complete configuration for pre-training, including model, trainer, and data settings optimized for 16k sequence length. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int, optional): Number of compute nodes to use. Defaults to 4. num_gpus_per_node (int, optional): Number of GPUs per node. Defaults to 8. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory llama3_70b_16k $ nemo llm pretrain --factory "llama3_70b_16k(num_nodes=4, name='my_70b_16k_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="llama3_70b_16k_pretrain", num_nodes=4) >>> print(recipe) Note: This recipe is optimized for the large 70B model with longer sequences (16k). It requires significant computational resources. )r r#rrr r i)rglobal_batch_sizemicro_batch_size)rpretrain_reciper rrunConfigrdata)r#r rrreciperrrr'Zs #r')rr)Nr"rr)typingrlightning.pytorchpytorchplnemo_runr(rnemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr#nemo.collections.llm.gpt.data.squadrnemo.collections.llm.recipesrNAMEclifactoryr)LightningModuler intrstrPartialr'rrrrsH       '