o }oi*@sbddlmZddlmZddlZddlZddlm Z ddl m Z ddl m Z mZddlmZmZmZddlmZddlmZd Zejjed d5d ed ejejfddZejje eddddddejdddddddddddddddddddd e fd!eed"ed ed#ed$ed%eej d&eed'ed(e!d)ed*ed+ed,ed-ed.e"d/ed0ed1ed2ed ej#f(d3d4Z$dS)6)OptionalN)pretrain)BERTMockDataModule) bert_model bert_trainer) default_logdefault_resumetensorboard_logger),distributed_fused_adam_with_cosine_annealing)TimingCallback bert_110mname huggingface bert_typereturncCs tt|dS)a Factory function to create a Bert-Base (110 million) model configuration. Returns: run.Config[pl.LightningModule]: Configuration for the BERT-Base (110 million) model. Examples: CLI usage: $ nemo llm pretrain model=bert_110m ... Python API usage: >>> model_config = model(bert_type="megatron") >>> print(model_config) )versionr)rNAMErrZ/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/bert_110m.pymodels r)targetrdefaultmegatronFi{z bf16-mixedg? iiigh㈵>g-C6?dirrtensor_parallelismpipeline_parallelismpipeline_parallelism_typevirtual_pipeline_parallelismcontext_parallelismsequence_parallelism num_nodesnum_gpus_per_node max_steps precisionaccumulate_grad_batchesgradient_clip_vallimit_test_batcheslimit_val_batcheslog_every_n_stepsval_check_intervalcCstj|t|dtdid|d|d|d|d|d|d| d | d | d | d | d |d|d|d|dttgtjt|||dt||t|ddt | |||||dt dS)a0 Create a pre-training recipe for BERT-base (110M) model. This function sets up a complete configuration for pre-training, including model, trainer, data, logging, optimization, and resumption settings. Args: dir (Optional[str]): Directory for saving logs and checkpoints. name (str): Name of the pre-training run. bert_type (str): Either "megatron" or "huggingface" type of BERT. tensor_parallelism (int): Degree of tensor model parallelism. pipeline_parallelism (int): Degree of pipeline model parallelism. pipeline_parallelism_type (Optional[torch.dtype]): Data type for pipeline parallelism. virtual_pipeline_parallelism (Optional[int]): Size of virtual pipeline parallelism. context_parallelism (int): Degree of context parallelism. sequence_parallelism (bool): Whether to use sequence parallelism. num_nodes (int): Number of compute nodes to use. num_gpus_per_node (int): Number of GPUs per node. max_steps (int): Maximum number of training steps. precision (str): Precision configuration, one of fp32, 16-mixed or bf16-mixed. accumulate_grad_batches (int): Number of steps per gradient accumulation. gradient_clip_val (float): Value for gradient clipping. limit_test_batches (int): Limit the number of test batches. limit_val_batches (int): Limit the number of validation batches. log_every_n_steps (int): Log every n steps. val_check_interval (int): Run validation every N steps. global_batch_size (int): Global batch size. micro_batch_size (int): Micro batch size. seq_length (int): Sequence length. warmup_steps (int): Number of warmup steps. constant_steps (int): Number of constant steps. min_lr (float): Minimum learning rate. max_lr (float): Maximum learning rate. fn (Callable): The pre-training function to use. Returns: run.Partial: Partial configuration for pre-training. Examples: CLI usage: $ nemo llm pretrain --factory bert_110m $ nemo llm pretrain --factory "bert_110m(num_nodes=1, name='my_bert_pretrain')" Python API usage: >>> recipe = pretrain_recipe(name="bert_pretrain", num_nodes=1) >>> print(recipe) rr!r"r#r$r%r&r'r(r)r*r+r-r.r/r0 callbacks) seq_lengthglobal_batch_sizemicro_batch_sizer )r rr )r* warmup_stepsconstant_stepsmin_lrmax_lr clip_grad)rtrainerdatalogoptimresumeNr) runPartialrrConfigr rrr r r)r rrr!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r3r4r2r5r6r7r8fnrrrpretrain_recipe2sjR      rC)r)%typingrlightning.pytorchpytorchplnemo_runr?torchnemo.collections.llm.apir#nemo.collections.llm.bert.data.mockr!nemo.collections.llm.recipes.bertrr(nemo.collections.llm.recipes.log.defaultrrr 'nemo.collections.llm.recipes.optim.adamr nemo.utils.exp_managerr rclifactorystrrALightningModulerbfloat16intdtypeboolfloatr@rCrrrrs            !