o }oi[ @sddlmZddlmZddlZddlZddlm Z m Z ddl m Z ddl mZddlmZdZejjedd ejejfd d Z ddeded ejfddZejje ed   ddeedededed ejf ddZdS))OptionalN)finetunepretrain)MockDataModule) mixtral_8x7b)TimingCallbackmixtral_8x7b_64k)namereturncCst}d|j_|S)a Factory function to create a Mixtral 8x7B model configuration with 64k sequence length. Returns: run.Config[pl.LightningModule]: Configuration for the Mixtral 8x7B model with 64k sequence length. Examples: CLI usage: $ nemo llm pretrain model=mixtral_8x7b_64k ... Python API usage: >>> model_config = model() >>> print(model_config) )rmodelconfig seq_length) model_configra/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/mixtral_8x7b_64k.pyr sr  num_nodesnum_gpus_per_nodec Cs(tjddtjdddd||ttgd S)ao Configure the NeMo Lightning Trainer for Mixtral 8x7B model with 64k sequence length. This function sets up the distributed training strategy optimized for very long sequences. Args: num_nodes (int, optional): Number of compute nodes to use. Defaults to 8. num_gpus_per_node (int, optional): Number of GPUs per node. Defaults to 8. Returns: run.Config: Configuration for the NeMo Lightning Trainer. Examples: CLI usage: $ nemo llm pretrain trainer=mixtral_8x7b_64k ... Python API usage: >>> trainer_config = trainer(num_nodes=16, num_gpus_per_node=8) >>> print(trainer_config) Note: This configuration uses significantly increased parallelism to handle the long sequence length efficiently. It requires a substantial amount of computational resources. rNT) tensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelismexpert_parallelismrr callbacks)rtrainertorchbfloat16runConfigrrrrrrr 3s r )targetr defaultdirr cCs@tj||||d}t|_t||d|_tjtdddd|_|S)a Create a pre-training recipe for Mixtral 8x7B model with 64k sequence length. This function sets up a complete configuration for pre-training, including model, trainer, and data settings optimized for 64k sequence length. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int, optional): Number of compute nodes to use. Defaults to 16. num_gpus_per_node (int, optional): Number of GPUs per node. Defaults to 8. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory mixtral_8x7b_64k $ nemo llm pretrain --factory "mixtral_8x7b_64k(num_nodes=16, name='my_64k_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="mixtral_8x7b_64k_pretrain", num_nodes=16) >>> print(recipe) Note: This recipe is optimized for handling long sequences (64k) compared to the standard version. It requires extensive computational resources due to the model size and extended sequence length. )r r(rrr%r ir)rglobal_batch_sizemicro_batch_size)rpretrain_reciper r r#r$rdata)r(r rrreciperrrr+]s #r+)rr)Nr'rr)typingrlightning.pytorchpytorchplnemo_runr#r!nemo.collections.llm.apirr"nemo.collections.llm.gpt.data.mockrnemo.collections.llm.recipesrnemo.utils.exp_managerrNAMEclifactoryr$LightningModuler intr strPartialr+rrrrsH       *