o wi7@s(ddlZddlZddlmZddlmZmZmZddlm Z ddl Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZddlmZdd lmZdd lm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+erddl,m-Z-ddl.m/Z/GdddZ0 d4dedej1de j2defddZ3e j4dddfdedej1de j5d e6d!e6d"e7d#e8ee0ffd$d%Z9   &  d5dede0d'e:e;d(ee:e;d)e7d*e6d+ee6d,eed#edS)6N)Path) TYPE_CHECKINGAnyOptional) TrainerFn)CommonInferenceParams) MCoreEngine)AbstractModelInferenceWrapper)TextGenerationController) AttnBackend)MegatronModule)io)ADAPTER_META_FILENAMEckpt_to_context_subdir)ckpt_to_weights_subdir)PEFT)MegatronStrategy) RestoreConfig)logging)GPTModel)T5Modelc@sPeZdZdZdddZdddZdd Zed d Zed d Z eddZ dS)MCoreTokenizerWrappperz We need this wrapper since mcore generate uses methods/properties such as tokenizer.detokenize, tokenizer.tokenize, tokenizer.bos, tokenizer.pad, etc. to encode and decode prompts NcCs||_|j|_|p |j|_dSN) tokenizereod vocab_size)selfrrr`/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/inference/base.py__init__3szMCoreTokenizerWrappper.__init__FcCs.dt|jjjvr|j||S|j|S)a- Detokenizes a list of tokens into a string. Args: tokens (list): The list of tokens to detokenize. remove_special_tokens (bool, optional): Whether to remove special tokens. Defaults to False. Returns: str: The detokenized string. remove_special_tokens)inspect signaturer ids_to_text parameters)rtokensr rrr detokenize8s  z!MCoreTokenizerWrappper.detokenizecCs |j|S)z Tokenizes a prompt into a list of tokens. Args: prompt (str): The prompt to tokenize. Returns: list: The list of tokens. )r text_to_ids)rpromptrrrtokenizeHs zMCoreTokenizerWrappper.tokenizecC|jjS)z Gets the IDs of additional special tokens. Returns: list: The IDs of additional special tokens. )radditional_special_tokens_idsrrrrr+Tz4MCoreTokenizerWrappper.additional_special_tokens_idscCr*)z Gets the ID of the beginning of sequence token. Returns: int: The ID of the beginning of sequence token. )rbos_idr,rrrbos^r-zMCoreTokenizerWrappper.boscCr*)zs Gets the ID of the padding token. Returns: int: The ID of the padding token. )rpad_idr,rrrpadhr-zMCoreTokenizerWrappper.padr)F) __name__ __module__ __qualname____doc__rr&r)propertyr+r/r1rrrrr-s     rpathtrainermodelrcCst|jts Jd|jjdksJdddlm}|||t|ddt}rOt |d }t |}Wd n1s@wYt |d d dd }nt |d dd }|d uret d |||_||j_d|j_d |_|j|||_|jjd ur|jjjdd|d|j|s|tj|j_|jj|d||j_|j|j } t| t!r| |}|jj"#|} t$j%|| d} dd| &D} |jj'j(t|dd| d} |jj)| ddd Sd S)a Sets up the trainer and restores the model from the given checkpoint path. It does the following: - Defines a RestoreConfig to restore only model weights - Disables setting up optimizers in the Trainer - Calls strategy.setup_environment(), model.configure_model() and strategy.setup_megatron_parallel(trainer=trainer) - Finally loads the model weights Args: path (Path): The path to the checkpoint file. trainer (nl.Trainer): The trainer object. model (pl.LightningModule): The model object. tokenizer (Any): The tokenizer object to override the tokenizer in the model. Returns: None z8Only MegatronStrategy is supported for trainer.strategy.z3Context parallelism is not supported for inference.r)#set_modelopt_spec_if_exists_in_ckptF) is_savingrNmodel_ckpt_pathT)r7load_model_stateload_optim_statezOverriding model.tokenizer to: cSsdSrrrrrrsz2_setup_trainer_and_restore_model..)r8)metadatacSsi|] \}}d|vr||qS)z .adapter.r).0kvrrr sz4_setup_trainer_and_restore_model..)sharded_state_dict)strict)* isinstancestrategyrcontext_parallel_sizenemo.collections.llm.modeloptr;rrexistsopenjsonloadrrinforrestore_config_setup_optimizers ckpt_pathconnectr8launcherlaunchsetup_environment state_dictconfigure_modelrTESTINGstatefnsetup_megatron_parallelselective_restoremodel_transformrunwrapped_checkpoint_ioload_content_metadatar rGitems checkpoint_ioload_checkpointload_model_state_dict)r7r8r9rr;adapter_meta_pathfrBrRpeftsharded_sd_metadatarGadapter_sharded_state_dict adapter_staterrr _setup_trainer_and_restore_modelts\           rmii F params_dtype&inference_batch_times_seqlen_thresholdinference_max_seq_lengthenable_flash_decodereturnc Kstjt|dd}|r*|tjks|tjkr$tdd|j_ t j |j_ nt d||D]\}} t|j|r@t|j|| q.t d|dq.t|||d||||} | t|jt|jd d fS) a Sets up the model and tokenizer for inference. This function loads the model and tokenizer from the given checkpoint path, sets up the trainer, and returns the Megatron inference-wrapped model and tokenizer. Args: path (Path): The path to the checkpoint file. trainer (nl.Trainer): The trainer object. params_dtype (torch.dtype, optional): The data type of the model parameters. Defaults to torch.bfloat16. inference_batch_times_seqlen_threshold (int, optional): If batch-size times sequence-length is smaller than this threshold then we will not use pipelining, otherwise we will. inference_max_seq_length (int, optional): max_seq_length for inference. Required by MCoreEngine(>=0.12). Necessary for CUDA graphs. Defaults to 2560. enable_flash_decode (bool, optional): Whether to enable flash decode. Defaults to True. **kwargs: Additional keyword arguments to set in the model config. Returns: tuple[AbstractModelInferenceWrapper, MCoreTokenizerWrappper]: A tuple containing the inference-wrapped model and Mcore wrapped tokenizer. r9)r7subpathz0Enabling Flash Decode for in-framework inferenceTzZFlash Decode is not supported for params_dtype %s, defaulting to MCore's attention backendzConfig attribute zA not found in model.config, ignoring in setup_model_and_tokenizer)r7r8r9rN)r load_contextrtorchbfloat16float16rrQconfig flash_decoder flashattention_backendwarningrchasattrsetattrrmget_inference_wrapperrrgetattr) r7r8rnrorprqkwargsr9keyvalueinference_wrapped_modelrrrsetup_model_and_tokenizers*   rpromptsencoder_promptsadd_BOSmax_batch_size random_seedinference_paramsc Csbddlm}|dur|||d} nt||d} t| ||d} |p%tddd} | j|||| d } | S) a  Runs generate on the model with the given prompts. This function uses the loaded model, loaded tokenizer, and prompts to generate text. It returns a dictionary containing the generated text. Args: model (AbstractModelInferenceWrapper): The inference-wrapped model. tokenizer (MCoreTokenizerWrappper): The tokenizer. prompts (list[str]): The list of prompts to generate text for. encoder_prompts (Optional[list[str]], optional): The list of encoder prompts. Defaults to None. add_BOS (bool, optional): Whether to add the beginning of sequence token. Defaults to False. max_batch_size (int, optional): The maximum batch size. Defaults to 4. random_seed (Optional[int], optional): The random seed. Defaults to None. inference_params (Optional[CommonInferenceParams], optional): The inference parameters defined in Mcore's CommonInferenceParams. Defaults to None. Returns: dict: A dictionary containing the generated results. r)&EncoderDecoderTextGenerationControllerNrrtext_generation_controllerrrir:)num_tokens_to_generatetop_k)rrrcommon_inference_params)^megatron.core.inference.text_generation_controllers.encoder_decoder_text_generation_controllerrr rrgenerate) r9rrrrrrrrr mcore_enginerresultsrrrrs"  riT c Cs<t||||||d\}} t|| d} t| ||d} | || fS)a Sets up and returns a Megatron Core Engine for text generation inference. Args: path (Path): Path to the model checkpoint trainer (nl.Trainer): NeMo Lightning trainer instance params_dtype (torch.dtype): Data type for model parameters. Defaults to torch.bfloat16 inference_batch_times_seqlen_threshold (int): Batch size * sequence length threshold. Defaults to 1000 inference_max_seq_length (int): Maximum sequence length for inference. Defaults to 4096 enable_flash_decode (bool): Whether to enable flash attention decoding. Defaults to False max_batch_size (int): Maximum batch size for inference. Defaults to 32 random_seed (Optional[int]): Random seed for reproducibility. Defaults to None Returns: Tuple[MCoreEngine, AbstractModelInferenceWrapper, MCoreTokenizerWrapper]: - Configured Megatron Core Engine instance - Inference-wrapped model - Tokenizer wrapper )r7r8rnrorprqrr)rr r) r7r8rnrorprqrrr9rrrrrrsetup_mcore_engine3s  rr)NFrNN)?r!rOpathlibrtypingrrrlightning.pytorchpytorchpltorch.distributedru lightning.pytorch.trainer.statesr/megatron.core.inference.common_inference_paramsr,megatron.core.inference.engines.mcore_enginerQmegatron.core.inference.model_inference_wrappers.abstract_model_inference_wrapperr Nmegatron.core.inference.text_generation_controllers.text_generation_controllerr megatron.core.transformer.enumsr megatron.core.transformer.moduler nemo.lightning lightningnlr nemo.lightning.ckpt_utilsrrnemo.lightning.io.plr nemo.lightning.pytorch.callbacksr3nemo.lightning.pytorch.strategies.megatron_strategyr'nemo.lightning.pytorch.strategies.utilsr nemo.utilsr#nemo.collections.llm.gpt.model.baser nemo.collections.llm.t5.model.t5rrTrainerLightningModulermrvdtypeintbooltuplerliststrdictrrrrrrs                   H O  ?  ;