o }oi @sddlmZddlmZddlZddlZddlm Z m Z ddl m Z ddl mZddlmZddlmZdZejjed d ejejfd d Z ddeded ejfddZejje ed   ddeedededed ejf ddZdS))OptionalN)finetunepretrain)MockDataModule)SquadDataModule) mixtral_8x7b)TimingCallbackmixtral_8x7b_16k)namereturncCst}d|j_d|j_|S)a Factory function to create a Mixtral 8x7B model configuration with 16k sequence length. Returns: run.Config[pl.LightningModule]: Configuration for the Mixtral 8x7B model with 16k sequence length. Examples: CLI usage: $ nemo llm pretrain model=mixtral_8x7b_16k ... Python API usage: >>> model_config = model() >>> print(model_config) @)rmodelconfig seq_lengthmax_position_embeddings) model_configra/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/mixtral_8x7b_16k.pyr sr  num_nodesnum_gpus_per_nodec Cs(tjddtjdddd||ttgd S)a Configure the NeMo Lightning Trainer for Mixtral 8x7B model with 16k sequence length. This function sets up the distributed training strategy optimized for longer sequences. Args: num_nodes (int, optional): Number of compute nodes to use. Defaults to 4. num_gpus_per_node (int, optional): Number of GPUs per node. Defaults to 8. Returns: run.Config: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=mixtral_8x7b_16k ... Python API usage: >>> trainer_config = trainer(num_nodes=4, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses increased parallelism to handle the longer sequence length efficiently. rNT) tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismexpert_parallelismrr callbacks)rtrainertorchbfloat16runConfigrrrrrrr"5s r")targetr defaultdirr cCs@tj||||d}t|_t||d|_tjtdddd|_|S)a Create a pre-training recipe for Mixtral 8x7B model with 16k sequence length. This function sets up a complete configuration for pre-training, including model, trainer, and data settings optimized for 16k sequence length. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int, optional): Number of compute nodes to use. Defaults to 4. num_gpus_per_node (int, optional): Number of GPUs per node. Defaults to 8. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory mixtral_8x7b_16k $ nemo llm pretrain --factory "mixtral_8x7b_16k(num_nodes=4, name='my_16k_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="mixtral_8x7b_16k_pretrain", num_nodes=4) >>> print(recipe) )r r*rrr'r ir)rglobal_batch_sizemicro_batch_size)rpretrain_reciper r"r%r&rdata)r*r rrreciperrrr-^s r-)rr)Nr)rr)typingrlightning.pytorchpytorchplnemo_runr%r#nemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockr#nemo.collections.llm.gpt.data.squadrnemo.collections.llm.recipesrnemo.utils.exp_managerrNAMEclifactoryr&LightningModuler intr"strPartialr-rrrrsJ        )