o wi @sddlmZddlZddlmZddlmZm Z ddl m Z m Z m Z dZejjeddejefdd Zejjeddd ed edejejfddZejje eddddddddde f dejfddZejjeeddddddddddef dejfddZdS)N) TokenizerSpec)finetunepretrain) model_recipepretrain_recipe_creatertokenizer_recipe hyena_40b)namereturncCstS)a< Defines a factory function for creating a tokenizer configuration. This function is registered as a CLI factory with the specified name and returns a tokenizer configuration based on the `tokenizer_recipe`. Returns: run.Config[TokenizerSpec]: A configuration object for the tokenizer. )rr r c/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/hyena_40b.py tokenizers r F tp_comm_overlap seq_lengthcCs td||S)z Factory function to create a Striped-Hyena 40B model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the Striped-Hyena 40B model. 40b)r)rrr r r model+s r)targetr rc Ks td||||||||d| S)aZ Create a pre-training recipe for Striped-Hyena 40B model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. )dirmicro_batch_sizeglobal_batch_size num_nodesnum_gpus_per_nodetensor_parallel_sizecontext_parallel_size model_sizeNr r) rrrrrrrrfnkwargsr r r pretrain_recipe6s r#zhyena-40b-pretrainc Ks$td|||||||||| d | S) ) r resume_pathrrrrrrrr!Nr r ) rr%rrrrrrrr!r"r r r finetune_recipeas r&)Fr)lightning.pytorchpytorchplnemo_runrun1nemo.collections.common.tokenizers.tokenizer_specrnemo.collections.llm.apirr'nemo.collections.llm.recipes.hyena_baserrrNAMEclifactoryConfigr boolintLightningModulerPartialr#r&r r r r sJ    "  *