o wi+@sddlZddlZddlmZddlmZmZddlmZeee fZ dddZ dee d ee d ee d e d e d e de de de dee de de fddZdS)N)Path)OptionalUnion)loggingreturncCs^tddd}g}dD] }||}||q |r(tdd|dStddS) z SLURM_, PMI_, PMIX_ Variables are needed to be unset for trtllm export to work on clusters. This method takes care of unsetting these env variables z+Unsetting all SLURM_, PMI_, PMIX_ VariablescSs\g}d|d}tj|dddd}|jd}|D]}|r+tj|d||q|S)Nz env | grep ^z | cut -d= -f1T)shellcapture_outputtext ) subprocessrunstdoutstripsplitosenvironpopappend)prefix unset_varscmdresult vars_to_unsetvarr]/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/deploy/base.pyunset_vars_with_prefix!s  z;unset_environment_variables..unset_vars_with_prefix)SLURM_PMI_PMIX_zUnset env variables: z, zNo env variables were unset.N)rinfoextendjoin)rall_unset_varsrrrrrunset_environment_variabless   r$nemo_checkpoint model_typetriton_model_repositorynum_gpustensor_parallelism_sizepipeline_parallelism_size max_input_lenmax_output_lenmax_batch_sizedtypeoutput_context_logitsoutput_generation_logitsc  Csddlm} |durd} t| jdddn|} |dur$|dur$td|dur2tj|s2td| | |dud }|durkzt d |j ||||||||| | | d W|St yj}zt d t |d}~ww|S) zp Exports the nemo checkpoint to trtllm and returns trt_llm_exporter that is used to deploy on PyTriton. r) TensorRTLLMNz/tmp/trt_llm_model_dir/T)parentsexist_okzThe provided model repository is not a valid TensorRT-LLM model directory. Please provide a --nemo_checkpoint or a TensorRT-LLM engine.zThe provided model repository is not a valid TensorRT-LLM model directory. Please provide a --nemo_checkpoint or a valid TensorRT-LLM engine.) model_dir load_modelzOExport operation will be started to export the nemo checkpoint to TensorRT-LLM.) nemo_checkpoint_pathr& gpus_per_noder)r*r+r,r-r.gather_context_logitsgather_generation_logitsz>An error has occurred during the model export. Error message: )nemo.export.tensorrt_llmr1rmkdir ValueErrorrpathisdirrr export Exception RuntimeErrorstr)r%r&r'r(r)r*r+r,r-r.r/r0r1 trt_llm_pathtrt_llm_exportererrorrrrget_trtllm_deployable:sN  rF)rN)rr pathlibrtypingrr nemo.utilsrrBAnyPathr$intboolrFrrrrs@    !