o wia@@s>ddlZddlZddlmZddlmZmZddlZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZddlmZddlmZddlmZdd lmZdd lmZmZmZmZm Z e j!d d Z"d Z#e"Z$zddl%m$Z$m&Z&ddl'm(Z(Wn e)ydZ#Ynwe*dZ+GdddZ,ddZ-GdddeZ.dS)N)Path)ListOptional)Template)StrictHandling)CommonInferenceParams)InferenceRequest) inference)ITritonDeployable)NEMO2broadcast_list cast_outputnemo_checkpoint_versionstr_ndarray2listcsfdd}|S)aA no-op decorator that returns the original function unchanged. Used as a fallback when pytriton's batch decorator is not available. Args: func: The function to decorate Returns: The original function without any modifications cs|i|S)z4 Wrapper method returning the func. )argskwargsfuncrc/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/deploy/nlp/megatronllm_deployable.pywrapper0sznoop_decorator..wrapperr)rrrrrnoop_decorator#s rT)batch first_value)TensorFNeMoc@s^eZdZdZe         ddededed ed ed ed ed eededefddZ dS)MegatronLLMDeployz A factory class for creating deployable instances of Megatron LLM models. This class provides a method to get the appropriate deployable instance based on the version of the NeMo checkpoint model used.  NFnemo_checkpoint_filepath num_devices num_nodestensor_model_parallel_sizepipeline_model_parallel_sizecontext_parallel_sizemax_batch_size random_seedenable_flash_decode legacy_ckptc Cs0t|tkrt|||||||||| d Std)a Returns the appropriate deployable instance for the given NeMo checkpoint. Args: nemo_checkpoint_filepath (str): Path to the .nemo checkpoint file. num_devices (int): Number of devices to use for deployment. num_nodes (int): Number of nodes to use for deployment. tensor_model_parallel_size (int): Size of the tensor model parallelism. pipeline_model_parallel_size (int): Size of the pipeline model parallelism. context_parallel_size (int): Size of the context parallelism. enable_flash_decode (bool): Whether to enable flash decode for inference. Returns: ITritonDeployable: An instance of a deployable class compatible with Triton inference server. rr r!r"r#r$r%r&r'r(z&Only NeMo 2.0 checkpoint is supported.)rr MegatronLLMDeployableNemo2 Exceptionr)rrrget_deployableKs  z MegatronLLMDeploy.get_deployable) rrrrrrNFF) __name__ __module__ __qualname____doc__ staticmethodstrintrboolr,rrrrrDsB  rcCs t|S)z Serializes dict to str )jsondumps)messagesrrr dict_to_strx r8c@s eZdZdZddddddddejddddddfd ed ed ed ed ededededejdededede ede de fddZ d6de ede e de efddZdd Zd7d!d"Zd#d$Zd%d&Zed'd(Zed)d*Zeed+dd,d-d.dd/d0d1d2 d3ejfd4d5ZdS)8r*a Triton inference server compatible deploy class for a .nemo model file Args: nemo_checkpoint_filepath (str): path for the nemo checkpoint. num_devices (int): number of GPUs. num_nodes (int): number of nodes. tensor_model_parallel_size (int): tensor parallelism. pipeline_parallelism_size (int): pipeline parallelism. context_parallel_size (int): context parallelism. params_dtype (torch.dtype): max input length. inference_batch_times_seqlen_threshold (int): squence threshold. inference_max_seq_length (int): max_seq_length for inference. Required by MCoreEngine (>=0.12). Defaults to 4096. max_batch_size (int): max batch size for inference. Defaults to 32. random_seed (Optional[int]): random seed for inference. Defaults to None. enable_flash_decode (bool): enable flash decode for inference. Defaults to False. NriirTFrr r!r"r#r$expert_model_parallel_sizeexpert_tensor_parallel_size params_dtype&inference_batch_times_seqlen_thresholdinference_max_seq_lengthr%r&r'r(c Cs||_tj|||||ddd|rtjndd }tjd|||tjdtjtjdddd}t j t ||| | | | | |d\|_ |_ |_dS)NF) r"r#r$r:r;sequence_parallelsetup_optimizersstore_optimizer_statesckpt_load_strictnessgpuz bf16-mixed) precisionr<pipeline_dtypeautocast_enabledgrad_reduce_in_fp32) acceleratordevicesr!strategyplugins)pathtrainerr<r=r>r%r&r')rnlMegatronStrategyrLOG_ALLTrainerMegatronMixedPrecisiontorchbfloat16r setup_mcore_enginer mcore_engineinference_wrapped_modelmcore_tokenizer)selfrr r!r"r#r$r:r;r<r=r>r%r&r'r(rJrMrrr__init__sF  z#MegatronLLMDeployableNemo2.__init__promptsinference_paramsreturncCs$|pt}|jj|d|d}t|S)a^ Generates text based on the provided input prompts. Args: prompts (List[str]): A list of input strings. inference_params (Optional[CommonInferenceParams]): Parameters for controlling the inference process. Returns: List[InferenceRequest]: A list containing the generated results. F)r[add_BOScommon_inference_params)rrVgeneratelist)rYr[r\resultsrrrr`s z#MegatronLLMDeployableNemo2.generatec Cs tjdtjdd}tjj|dd|dkr=tdgdd}tdgdd\}}}}}t|t|t|||d }| ||ndSq) zD Generate function for ranks other than the rank 0. TrcudadtypedevicersrcNdatarh) temperaturetop_ktop_pnum_tokens_to_generatereturn_log_probs) rSemptylong distributed broadcastr rr3floatr`) rYmessager[rkrlrmrn log_probsr\rrrgenerate_other_rankss z/MegatronLLMDeployableNemo2.generate_other_rankscCsPz|jjjj}|jjjj}t|}Wn tytdw|j|||d}|S)zy Load the chat template. Works when model's tokenizer has chat template (typically chat models). zThe tokenizer does not have chat template, if you would like to evaluate chat model ensure your model's tokenizer has a chat template)r7 bos_tokenadd_generation_prompt)rX tokenizer chat_templaterxrAttributeError ValueErrorrender)rYr7rytokenizer_chat_templaterxtemplaterendered_outputrrrapply_chat_templates    z.MegatronLLMDeployableNemo2.apply_chat_templatecCsH|jjjj}g}|D]}||vr|||ddq ||q |S)zV Removes eos token if it exists in the output, otherwise does nothing rr)rXrz eos_tokenappendrsplit)rYtextroutputtrrrremove_eos_tokens  z+MegatronLLMDeployableNemo2.remove_eos_tokencCs t|S)z& Convert str to dict. )r5loads)rYjson_strrrr str_to_dictr9z&MegatronLLMDeployableNemo2.str_to_dictcCstddtdtddtjddtddtjddtddtjddtd dtjddtd dtjddtd dtjddtd dtjddtd dtjddtddtjddtddtjddf }|S)Nr[nameshapere max_lengthT)rrreoptionalr%rlrmrkr&compute_logprobrn_top_logprobsecho)rbytesnpint_singlebool_)rYinputsrrrget_triton_input"s  z+MegatronLLMDeployableNemo2.get_triton_inputcCs*tddtdtddtjdtddtdfS)N sentencesrrrv top_logprobs)rrrrrYrrrget_triton_output3s  z,MegatronLLMDeployableNemo2.get_triton_outputrrlrmrkrrrrrc s^i}t|d}|dd}|dd}|dd}|dd }|d d }|d d } |d d} |dd } d| rOfdd|D}fdd|D}tjr|tjdkr|tjjtjdgtjddddt |ddt |||||gddt |t |t |||| d}  || } | rfdd| D}n fdd| D}|}dt|tji}|rg}| D]2}| rt|j|j}n t|j}t|dkr|dgq||q| r tdd|Dtfdd|D}||d<nt||d<| r-g}| D] }t|j}||qt|tj|d <|S)!Nr[rkg?rlrrmgrrFrrrrTcg|]}|qSr)r.0promptrrr Wz>MegatronLLMDeployableNemo2.triton_infer_fn..crr)rrrrrrXrrcrdrgri)rkrlrmrnrotop_n_logprobscs g|] }r |j|jn|qSr)rgenerated_textrr text_onlyrrrus csg|] }r |jn|qSr)rrrrrrwsrcss|]}t|VqdSN)lenrarrrrr sz=MegatronLLMDeployableNemo2.triton_infer_fn..cs(g|]}tj|dt|fddqS)r)constant_values)rpadrr)max_lenrrrs(rvr)rpoprSrris_initializedget_world_sizerstensorrqr rr3rtr`rr rbytes_prompt_log_probsgenerated_log_probscpudetachnumpyrrmaxarrayr8generated_top_n_logprobs)rYr output_inferr[rkrlrmrnrvrrrr\rb output_textsoutput_log_probsrlppaddedoutput_top_n_log_probstop_n_lpr)rrYrrtriton_infer_fn;s                   z*MegatronLLMDeployableNemo2.triton_infer_fnr)T)r-r.r/r0rSrTr2r3rerr4rZrrrr`rwrrrpropertyrrrrrndarrayrrrrrr*s      :      r*)/r5loggingpathlibrtypingrrrrrStorch.distributedwraptjinja2r+megatron.core.dist_checkpointing.validationr/megatron.core.inference.common_inference_paramsr)megatron.core.inference.inference_requestrnemo.lightning lightningrNnemo.collections.llmr nemo.deployr nemo.deploy.utilsr r r rr decoratorr use_pytritonrpytriton.decoratorsrpytriton.model_configrr+ getLoggerLOGGERrr8r*rrrrs<           4