o }oi$>@sddlZdejd<ddlZddlZddlmZddlZddlm Z ddl Z ddl m Z ddlmZddlmZddlmZmZdd lmZd Zefd d ZdddZdejfddZddZdddZGddde jZ ddZ!e"dkrye!dSdS)Nz /tmp/hf_homeHF_HOME)Path)AutoModelForCausalLM) lightning)llm) JitConfig JitTransform)to_cpuz#/home/TestData/lite/hf_cache/squad/cs8fdd}tj|djd}|j|ddgdd|S) Ncsd|dd|ddd|ddd g}ttj|\}}t|d kr8|d jkr8|d jt|d krK|d jkrK|jt ||d d||dd d gt|d d gt|d S) Nz Context: contextz Question: questionz Answer: answerstextr)labels input_ids loss_mask) striplistmap text_to_idslenbos_idinserteos_idappenddict)exampleformatted_text context_ids answer_ids tokenizerP/home/ubuntu/.local/lib/python3.10/site-packages/tests/collections/llm/hf/sft.pyformatting_prompts_func$s z6make_squad_hf_dataset..formatting_prompts_funcz train[:10])split pad_token_idF)idtitler r r )batched batch_sizeremove_columns)rHFDatasetDataModulerr)r# data_pathr& datamoduler$r"r%make_squad_hf_dataset#s r2FcCsj|dkrtjjd|j|ddS|dkrtjj|j|ddS|dkr1tj||d|j|dd Std ) Nautozcuda:0) adapter_only)device checkpoint_ioddp)r6fsdp2r)data_parallel_sizetensor_parallel_sizer6zEncountered unknown strategy)pl strategiesSingleDeviceStrategymake_checkpoint_io DDPStrategynl FSDP2StrategyNotImplementedError)strategymodeldevices num_nodesr4r$r$r% make_strategy?s    rGreturncCs*tjddddddd}tjd|d|ddS)NTrreduced_train_loss) save_lastevery_n_train_stepsmonitor save_top_ksave_on_train_epoch_endsave_optim_on_train_end nemo2_sftF)namelog_diruse_datetime_versionckptwandb)r@ModelCheckpoint NeMoLogger) ckpt_folderrTr$r$r%loggerSs rYc Csd}d}d}td}t|D];\}}}|D]3}||} | rJtt| \} } } t| t| } } | |ks?| |krJ| |krJ| }| }tj ||}qqt |S)NrzD[a-z0-9_]+--reduced_train_loss=([\d\.]+)-epoch=(\d+)-step=(\d+)-last) recompileoswalkmatchrfloatgroupsintpathjoinr) base_dirlatest_checkpoint max_epochmax_steppatternrootdirs_dir_namer^lossepochstepr$r$r%get_latest_checkpointfs"   rpcshd}d}d}d}t|}|d}|sJd|t}g} |dD]} | j|r<| jdr<| | jq'|| jq'||v} t | d k} | rU| rUJd | r||vsbJd |t d d | Dfdd t dddD} t| t| ksJd| d| n| sJd| |sJd|||dsJdddg}|D]} |d| sJd| qt ddS)N> config.jsontokenizer.modeltokenizer_config.jsongeneration_config.jsonspecial_tokens_map.jsonzmodel.safetensorsmodel-zmodel.safetensors.index.json hf_weightszMissing hf_weights directory: * .safetensorsrzGBoth model.safetensors and sharded model files exist, which is invalid.zMissing index file: cSsg|] }t|ddqS)-r)rar').0fr$r$r% sz3verify_sft_checkpoint_structure..cs&g|]}d|dddddqS)rv05dz-of-rryr$)r{i shard_numbersr$r%r}srrz(Missing or extra shard files. Expected: z , Found: zMissing model file(s)zMissing files: z trainer.ptzMissing trainer.pt filez model.yamlzio.jsonr zMissing context file: z)Checkpoint structure verification passed.)rexistssetglobrQ startswithendswithraddrsortedrangeissubsetprint)rb has_io_bytesexpected_files model_filemodel_shard_prefix index_fileckpt_dirrw found_files found_shardsfilehas_model_filehas_sharded_filesexpected_shards context_filesr$rr%verify_sft_checkpoint_structure~sN     rcs`eZdZdZddededeffdd Zd d d d d eddfddZe ddZ dddZ Z S)!ValidateCheckpointRestoreCallbackzThis callback checks that the model weights and optimizer states are exactly restored from the checkpoint on the first training batch. T-C6? check_weightscheck_optimizer tolerancecs,t||_||_d|_d|_||_dSN)super__init__rrloaded_model_stateloaded_optimizer_statesr)selfrrr __class__r$r%rs  z*ValidateCheckpointRestoreCallback.__init__trainer pl.Trainer pl_modulezpl.LightningModule checkpointrHNcCs|d|_|d|_dS)z Save the loaded model and optimizer states so we can compare them to the actual states after resuming. state_dictoptimizer_statesNrr)rrrrr$r$r%on_load_checkpoints z4ValidateCheckpointRestoreCallback.on_load_checkpointcCs|jduo |jduSrr)rr$r$r%has_loaded_checkpointsz7ValidateCheckpointRestoreCallback.has_loaded_checkpointc Cs0|jr^|jdur^|jD]K\}}||jvrtd|d|j|}t|t|}}tj|||jdsXt j dd}|| d } t|d|d | d q td ntd |jr|jdur|j} t| t|jkrtd t| dt|jdt| |jD]k\} } | } | | krtd| dD]P\}}| d|}|D]A\}}||vrtd|d||}t|tjrtj||ddstd|d|dq||krtd|d|dqqqtdntd|js |j|dStdt ddS)z` Verify that the loaded model weights and optimizer state matches checkpoints'. Nz Parameter 'z%' not found in checkpoint state dict.)atol LOCAL_RANK0rz: Model parameter 'z(' does not match the checkpointed value .zmodel weights matchzDid not test model weightszNumber of optimizers (z!) does not match the checkpoint (z).zFMismatch in optimizer state keys between current state and checkpoint.statezKey 'z(' missing in the loaded optimizer state.h㈵>zOptimizer state for param_id=z, key='z ' does not match the checkpoint.z' differs from checkpoint.zoptim weights matchzdid not test optim weightszAll weights matchr)!rrrDnamed_parameters ValueErrorr torchallcloserr\environgetviewr_abscpumaxrrr optimizersrziprkeysitems isinstanceTensorr callbacksremovesysexit)rrlightning_modulebatch batch_idxrQ current_paramcheckpoint_paramrankdiffr optimizerloaded_opt_statecurrent_opt_stateparam_id param_stateloaded_param_state state_keycurrent_tensor_or_valloaded_tensor_or_valr$r$r%on_train_batch_startsl       z6ValidateCheckpointRestoreCallback.on_train_batch_start)TTr)rrrHN) __name__ __module__ __qualname____doc__boolr_rrrpropertyrr __classcell__r$r$rr%rs rcCsddl}|}|jdtdd|jdtdgdd |jd td d|jd td d|jd tddgd |jdtdd|jdtddgd |jdtdd|jddd|jdtdd|jdtdd|jddd|jddd|}d}|jdurddlm }d |j dd d}||j|d!|j d"|jd#}d}|jdkrdd$lm}||jd%}g}|jrtd&d'd(id(d)} t| g}|jr|tGd*d+d+tj} |jr| ntj} | |j |d,}t|j||j |jd(} |jrtjd&d(d-nd} |jt| du7_tj |j |j|j|j!| d d.dd |j"d(||d/d0}tj#j$|t%tj&|j |t'(tj)j*d1d2t+|j,| d3~~t-|j,}t.||dut/j0|d4d&d5}t1|d d6dksjJd7|ft1|d d8dks{Jd9|fdS):zMExample script to run SFT with a HF transformers-instantiated model on squad.rNz--modelzmeta-llama/Llama-3.2-1B)typedefaultz --strategyr3)r3r7r8)rrchoicesz --devicesrz --num-nodesz --acceleratorgpuz --grad-clipg?z--model-acceleratortez --max-stepsdz--fp8-autocast store_true)actionz--wandb-projectz --ckpt-folderz/tmp/nemo_automodel_sft/z--use-torch-jitz --auto-resume) WandbLoggerrk/_dev_strat_)projectrQ)TEConfig) fp8_autocastTdynamicF) use_torch torch_kwargs use_thundercseZdZfddZZS)z,main..ZeroInitHFAutoModelForCausalLMcs.tj|i|}|D]}|dq |S)Nr)rconfigure_model parametersfill_)rargskwargsansparamrr$r%configure_moduleNs  z=main..ZeroInitHFAutoModelForCausalLM.configure_module)rrrrrr$r$rr%ZeroInitHFAutoModelForCausalLMMsr) model_namemodel_accelerator)resume_if_existsresume_ignore_no_checkpointgbf16)rErF max_steps acceleratorrClog_every_n_stepslimit_val_batchesnum_sanity_val_stepsaccumulate_grad_batchesgradient_clip_valuse_distributed_samplerrYr precisionr)lr)rDdataroptimlogresumerw)output_loading_info missing_keyszNOT LOADABLE #1mismatched_keyszNOT LOADABLE #2)2argparseArgumentParser add_argumentstrrar_ parse_args wandb_projectlightning.pytorch.loggersrrcrDr'rErCr4nemo.lightning.pytorch.accelerate.transformer_enginerr use_torch_jitrr auto_resumerrrHFAutoModelForCausalLMrGrFr@ AutoResumerTrainerr grad_clipapifinetuner2configure_tokenizerfdlbuildadampytorch_adam_with_flat_lrrYrXrprrfrom_pretrainedr)rparserrrUrrDrrr jit_configr model_clsrCrrrbrr$r$r%main!s         "&r.__main__)F)#r\rrZrpathlibrfiddler&lightning.pytorchpytorchr;r transformersrnemorr@nemo.collectionsr nemo.lightning.pytorch.callbacksrr'nemo.lightning.pytorch.strategies.utilsr DATA_PATHr2rGrWrYrprCallbackrr.rr$r$r$r%s0          ?df