o }oiQ@sddlZddlmZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZed eZd ed ejefd dZdejdeje defddZdedeje d efddZdeed edBfddZdS)N)List)Callback) AutoTokenizer)SquadDataModule)GPTModel)MegatronCommOverlapCallback)DEFAULT_NEMO_CACHE_HOME)logging NEMO_HOME model_namereturncCs6dtdddddg}td|tjt|dd S) a8 HuggingFace tokenizer. Args: model_name (str): corresponds to HuggingFace-AutoTokenizer's 'pretrained_model_name_or_path' input argument. For more details please refer to- huggingface.co/docs/transformers/v4.47.1/en/model_doc/auto#transformers.AutoTokenizer zE`AutoTokenizer` first searches for tokenizer files locally stored in .zK(from env var `NEMO_HOME`- can be changed using '-nh/--nemo_home' CLI arg).zbIf files are missing locally, `AutoTokenizer` will try downloading from HuggingFace. In this case-zkmake sure env vars 'TRANSFORMERS_OFFLINE':'0' and 'HF_TOKEN':'' are set in your sbatch script.zPBoth of these will be set automatically if you provide '-hf/--hf_token' CLI arg. T)pretrained_model_nameuse_fast)DEFAULT_NEMO_HOMEr warningjoinrunConfigr)r log_msgrM/home/ubuntu/.local/lib/python3.10/site-packages/scripts/performance/utils.py hf_tokenizers rexecutormodelsourcecCsDddlm}ddlm}||}d|_d|_tj|||dd|dfS)a Downloads/Acceses checkpoint to be used for fine-tuning. `import_ckpt` first tries find the nemo checkpoint in /models/. For eg: for llama3 8b, the path will look like- /models/meta-llama/Meta-Llama-3-8B If missing, tries to downloads at the same location from HuggingFace and converts it nemo format. Args: source (str): HuggingFace URL. For eg- hf://meta-llama/Meta-Llama-3-70B r)deepcopy) import_ckptF)rr overwriteimport_ckpt_exp)copyrnemo.collections.llmrntasks_per_nodenodesrPartial)rrrrrimport_executorrrrimport_ckpt_experiment8s r( hf_model_uri data_configcCsZtdtjtd}|dd}|jd}tj|dd||}tj|o,tj|S)z This method is used for fine-tuning. It checks if packed train data for a partiular sequence length exists locally. This is needed to set data flag (force_redownload=True) which avoids experiment crash in case files are missing. NEMO_DATASETS_CACHEdatasets/z--z_metadata.jsonlsquadpacked) osgetenvpathrrreplace seq_lengthexistsisfile)r)r* datasets_dir model_dirmetadata_filenametrain_pack_metadata_filepathrrrisfile_train_pack_metadataLs   r; callbackscCs,|rt|D] \}}|jtkr|SqdS)a nemo.lightning.Trainer has a list of callbacks defined. This method identifies index of MegatronCommOverlapCallback from the list defined in recipes in nemo.collections.llm.recipes. The index is needed to override ddp communication params N) enumerate __fn_or_cls__r)r<idxcallbackrrrget_comm_overlap_callback_idx[s  rA)r0typingrnemo_runr$lightning.pytorch.callbacks.callbackr.nemo.collections.common.tokenizers.huggingfacer#nemo.collections.llm.gpt.data.squadrnemo.collections.llm.gpt.modelr&nemo.collections.llm.recipes.llama3_8brnemo.lightning.baser nemo.utilsr r1rstrrr SlurmExecutorr(boolr;intrArrrrs