o }oi" @s@ddlZddlmZddlmZddlZddlZddl m Z ddl m Z ddlmZddlmZdeded ed ed e jf d d Zd ed e jfddZGdddejZGdddejZGdddejZGdddejZded dfddZ d(de jdededed ed df d!d"Z d#ej!fd$d%Z"eGd&d'd'e j#Z$dS))N) dataclass) lightning)llm)SentencePieceTokenizer)logging data_pathtokenizer_pathindex_mapping_dir seq_lengthreturnc Cs*tjt|d}tjtj|||ddd|dS)z/Single shard dataset tokenized by SentencePiece) model_path i)paths tokenizerr micro_batch_sizeglobal_batch_sizeseedr )runConfigrrPreTrainingDataModule)rrr r rrP/home/ubuntu/.local/lib/python3.10/site-packages/tests/collections/llm/common.py train_datasrc Cstjtjd|ddddddS)zSmall 145m modeli  i gZd;O?) rotary_baser num_layers hidden_sizeffn_hidden_sizenum_attention_headsinit_method_std)rrrLlama3Config8B)r rrrsmall_llama_cfg-sr$c@s8eZdZdZdefddZdejdejddfd d Z dS) StopBeforeEnda4Preemptively stop training at a given global step. Allows stopping training before reaching the max steps. Useful for testing checkpoint save and resume. Args: stop_on_step (int): Stop training when trainer.global_step reaches this value. Checked at the start of every step. stop_on_stepcCs ||_dSN)r&)selfr&rrr__init__Ds zStopBeforeEnd.__init__trainer pl_moduler NcCsP|j|jkr$td|jd|jdd|_|j|jdkr&d|_dSdSdS)Nz Global step z >= z, signaling Trainer to stop.Tr) global_stepr&rinfo should_stopval_check_intervallimit_val_batches)r(r*r+outputsbatch batch_idxrrron_train_batch_endGs  z StopBeforeEnd.on_train_batch_end) __name__ __module__ __qualname____doc__intr)plTrainerLightningModuler4rrrrr%;sr%c@s*eZdZdZdejdejddfddZdS)4AssertOptimizerParamGroupsHaveAtLeastTwoWeightDecaysaSanity test weight decay settings in optimizer param groups. Background: The Megatron/NeMo optimizer splits parameters into groups by whether or not the parameter should have weight decay applied. A typlical rule is that `bias` terms and `layer_norm` terms for example should not have weight decay applied. This callback checks for the existance of two distinct weight decay settings across optimizers and param groups related to a bug adddressed in https://github.com/NVIDIA/NeMo/pull/12123. r*r+r Nc Csni}t|jD]\}}t|jD]\}}d|d|}|d||<qqtt|dks5Jd|dS)Noptz;pg weight_decayzFAll weight decays in optimizer param groups should not be equal. Got: ) enumerate optimizers param_groupslensetvalues) r(r*r+ weight_decaysoioptimpi param_groupkeyrrron_train_start]szCAssertOptimizerParamGroupsHaveAtLeastTwoWeightDecays.on_train_start)r5r6r7r8r:r;r<rMrrrrr=Rs r=csHeZdZdZdeffdd ZddZdejdej d d fd d Z Z S) MCoreModelAttributeValidatorzOWalk through submodules and verify user-specified attributes like parallelisms. attr_dictct||_dSr'superr)rOr(rO __class__rrr)k  z%MCoreModelAttributeValidator.__init__c CsP|jD] \}}t||r%t||}||ks%Jd|d|d|dqdS)NzKey z for model (z) does not match z! from provided attribute mapping.)rOitemshasattrgetattr)r(targetkv model_valrrr _check_attrsos   z)MCoreModelAttributeValidator._check_attrsr*r+r Ncs.dtjjdtjjffdd }|j|dS)Nmoduler cst|dr |j|S)Nconfig)rXr^r`)r_r(rrwalk_fnxs  z.walk_fn)torchnnModulemodelwalk)r(r*r+rbrrarrMwsz+MCoreModelAttributeValidator.on_train_start) r5r6r7r8dictr)r^r:r;r<rM __classcell__rrrTrrNhs "rNcsJeZdZdZdeffdd Zdejdejddfd d Z dd dZ Z S)MiscAttributeValidatorzCPlace for any miscellaneous attribute assertions. Extend as needed.rOcrPr'rQrSrTrrr)rVzMiscAttributeValidator.__init__r*r+r NcCs|d|jvr:|jjjjddj}|j|jdks&Jd|jd|jd||jdksJd|ddt|D}tj }t |d |kscJd d |d t |dS) Nweightsz Weights not found in checkpoint z common.ptz"No 'common.pt' file in checkpoint z metadata.jsonz&No 'metadata.json' file in checkpoint cSsg|] }|dr|qS)z.distcpendswith).0shardrrr sz%verify_distcp_dir..z)Wrong number of .distcp files, Expected: z Found: ) ospathbasenamejoinisdirisfilelistdirrc distributedget_world_sizerD)ry ckpt_name weights_dirshards world_sizerrrverify_distcp_dirs $  rT model_ckptrkr/exp_dir dist_ckptsc CsHtj|d}t|}|jrtdd|DsJd|||j}|jdkrEt||ksDt||j|jksDJd|d|jdnt||ksSJd|d |D]L}tj||} d |vsiJd ||d rd |j vrd|d|vsJd|d||rtj | sJdt | qUtj | sJdqUdS)zEnsures that the provided checkpoint directory has - correct number of checkpoints - no more than top-k checkpoints - no unfinished checkpoints - a checkpoint for the last step - all checkpoints in the correct format checkpointscSsg|]}|dqS)-lastr{)r}crrrrsz#verify_ckpt_dir..z(No -last checkpoint found after trainingrz Expected z checkpoints or at most top z checkpoints besides '-last'z checkpointsz -unfinishedzIUnfinished checkpoint found. Something went wrong with saving checkpoint rstepzstep=r@zLast checkpoint z not for final step zCheckpoint is not correct typeN) rrrr save_lastany save_top_krDr|filenamerrr) rrkr/rrckpt_dirckptsexpected_countrryrrrverify_ckpt_dirs.  " & r precisioncsdtjddffdd }|S)Ntensorr cs|jksJdSr')dtype)rrrrverify_precisionsz1create_verify_precision..verify_precision)rcTensor)rrrrrcreate_verify_precisionsrc@sVeZdZUdZeed<dZeed<dZeed<dZeed<d Z eed <d Z eed <d S)Llama3ConfigCIir rrrri r r!i vocab_sizeN) r5r6r7r r9__annotations__rrr r!rrrrrrs     r)T)%r dataclassesrlightning.pytorchpytorchr:nemo_runrrcnemornlnemo.collectionsr"nemo.collections.common.tokenizersr nemo.utilsrstrr9rr GPTConfigr$Callbackr%r=rNrjrModelCheckpointboolrrrr#rrrrrsV        *