o wi~ @sdddlZddlZddlmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZed eZd edeje fddZdejdejedefddZd,ddZd-d ede fddZ! d-dejd ede fddZ"dedejede#fd d!Z$d"eede dBfd#d$Z% %d.d&ed'ed(ed)efd*d+Z&dS)/N)List)Callback)YamlSerializer) _serialize AutoTokenizerSquadDataModule)GPTModel)MegatronCommOverlapCallback)DEFAULT_NEMO_CACHE_HOME)logging NEMO_HOME model_namereturncCs6dtdddddg}td|tjt|dd S) a8 HuggingFace tokenizer. Args: model_name (str): corresponds to HuggingFace-AutoTokenizer's 'pretrained_model_name_or_path' input argument. For more details please refer to- huggingface.co/docs/transformers/v4.47.1/en/model_doc/auto#transformers.AutoTokenizer zE`AutoTokenizer` first searches for tokenizer files locally stored in .zK(from env var `NEMO_HOME`- can be changed using '-nh/--nemo_home' CLI arg).zbIf files are missing locally, `AutoTokenizer` will try downloading from HuggingFace. In this case-zkmake sure env vars 'TRANSFORMERS_OFFLINE':'0' and 'HF_TOKEN':'' are set in your sbatch script.zPBoth of these will be set automatically if you provide '-hf/--hf_token' CLI arg. T)pretrained_model_nameuse_fast)DEFAULT_NEMO_HOMEr warningjoinrunConfigr)rlog_msgrV/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/scripts/performance/utils.py hf_tokenizer"s rexecutormodelsourcecCsDddlm}ddlm}||}d|_d|_tj|||dd|dfS)a Downloads/Acceses checkpoint to be used for fine-tuning. `import_ckpt` first tries find the nemo checkpoint in /models/. For eg: for llama3 8b, the path will look like- /models/meta-llama/Meta-Llama-3-8B If missing, tries to downloads at the same location from HuggingFace and converts it nemo format. Args: source (str): HuggingFace URL. For eg- hf://meta-llama/Meta-Llama-3-70B rdeepcopy) import_ckptF)rr overwriteimport_ckpt_exp)copyr"nemo.collections.llmr#ntasks_per_nodenodesrPartial)rrr r"r#import_executorrrrimport_ckpt_experiment;s r-cCsh|du}dtjv}|r%|r%tjd|kr#td|dtjdd|S|r)|S|r0tjdStd)zd Get NEMO_HOME path. Checks for both nemo_home argument and NEMO_HOME environment variable. TrzUsing nemo_home (z) instead of NEMO_HOME ()zJNeither -nh/--nemo_home argument nor NEMO_HOME environment variable is set)osenvironr r ValueError) nemo_home arg_nemo_set env_nemo_setrrr get_nemo_homeOs  r5 seq_lengthc Csddlm}ddlm}ddlm}ddlm}|t|}|dd}|j ddd ||d } |||d d ||d | dddd } | |d| dd} t d| | ret dt| ddStd| d)a Prepare the SQuAD dataset for fine-tuning. Args: model_name (str): The name of the model seq_length (int): The sequence length to use for packing. Defaults to 2048. nemo_home: Optional path to NEMO home directory set via args.nemo_home r)Pathr)PackedSequenceSpecsrdatasetssquadT)parentsexist_ok)rr$)packed_sequence_sizeFi) dataset_rootr7global_batch_sizemicro_batch_sizepacked_sequence_specs tokenizerforce_redownload delete_rawseedpacked/--zPacked files should be in: z Files found:*z Packed dataset dir not found at z. Dataset download failedN)pathlibr8=nemo.collections.common.tokenizers.huggingface.auto_tokenizerr-nemo.collections.llm.gpt.data.packed_sequencer9#nemo.collections.llm.gpt.data.squadr r5mkdir prepare_datareplaceprintexistslistglobFileNotFoundError) rr7r2r8rr9r nemo_home_pathr@rD datamodule packed_dirrrrprepare_squad_datasetds2        r[cCs8ddlm}||}d|_d|_tjt|||d|dfS)zC Downloads and prepares the SQuAD dataset for fine-tuning. rr!r$)rr7r2prepare_squad_dataset_exp)r'r"r)r*rr+r[)rrr7r2r"dataset_executorrrr prepare_squad_dataset_experiments r^ hf_model_uri data_configcCsZtdtjtd}|dd}|jd}tj|dd||}tj|o,tj|S)z This method is used for fine-tuning. It checks if packed train data for a partiular sequence length exists locally. This is needed to set data flag (force_redownload=True) which avoids experiment crash in case files are missing. NEMO_DATASETS_CACHEr:rIrJz_metadata.jsonlr;rH) r/getenvpathrrrRr7rTisfile)r_r` datasets_dir model_dirmetadata_filenametrain_pack_metadata_filepathrrrisfile_train_pack_metadatas   ri callbackscCs,|rt|D] \}}|jtkr|SqdS)a nemo.lightning.Trainer has a list of callbacks defined. This method identifies index of MegatronCommOverlapCallback from the list defined in recipes in nemo.collections.llm.recipes. The index is needed to override ddp communication params N) enumerate __fn_or_cls__r )rjidxcallbackrrrget_comm_overlap_callback_idxs  roconfig_diff.txt base_recipe new_recipe output_dir file_namecCst|td}t|td}tj|jdd|jdddddd}d|}tdtj||t tj||d }| |Wd d S1sHwYd S) z4 Dump the config diff from the base recipe. )serializer_clsT)keependsrqrr)fromfiletofilelinetermzdumping config diff to wN) rrdifflib unified_diff splitlinesrrSr/rcopenwrite)rqrrrsrtbase_recipe_confignew_recipe_configdifffrrr!dump_config_diff_from_base_recipes      "r)N)r6N)rp)'r|r/typingrnemo_runr$lightning.pytorch.callbacks.callbackr nemo_run.core.serialization.yamlr%nemo_run.run.torchx_backend.packagingr.nemo.collections.common.tokenizers.huggingfacerrOr nemo.collections.llm.gpt.modelr &nemo.collections.llm.recipes.llama3_8br nemo.lightning.baser nemo.utilsr rbrstrrr SlurmExecutorr-r5intr[r^boolrirorrrrrsL            .