o wi5@sddlZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZmZmZddlmZddlmZmZmZed Zd gZGd d d eZdS) N)AnyListOptional) PeftModel)batch)Tensor) AutoModelAutoModelForCausalLM AutoTokenizer)ITritonDeployable)broadcast_list cast_outputstr_ndarray2listNeMotext-generationc@seZdZdZ         ddeedeedeed eed eed eef d d ZdddZ de de efddZ ddZ eddZeddZedejfddZdS)HuggingFaceLLMDeployayA Triton inference server compatible wrapper for HuggingFace models. This class provides a standardized interface for deploying HuggingFace models in Triton inference server. It supports various NLP tasks and handles model loading, inference, and deployment configurations. Args: hf_model_id_path (Optional[str]): Path to the HuggingFace model or model identifier. Can be a local path or a model ID from HuggingFace Hub. hf_peft_model_id_path (Optional[str]): Path to the PEFT model or model identifier. Can be a local path or a model ID from HuggingFace Hub. tokenizer_id_path (Optional[str]): Path to the tokenizer or tokenizer identifier. If None, will use the same path as hf_model_id_path. model (Optional[AutoModel]): Pre-loaded HuggingFace model. tokenizer (Optional[AutoTokenizer]): Pre-loaded HuggingFace tokenizer. tokenizer_padding (bool): Whether to enable padding in tokenizer. Defaults to True. tokenizer_truncation (bool): Whether to enable truncation in tokenizer. Defaults to True. tokenizer_padding_side (str): Which side to pad on ('left' or 'right'). Defaults to 'left'. task (str): HuggingFace task type (e.g., "text-generation"). Defaults to "text-generation". **hf_kwargs: Additional keyword arguments to pass to HuggingFace model loading. NTleftrhf_model_id_pathhf_peft_model_id_pathtokenizer_id_pathmodel tokenizertaskc Ks|dur |dur td|dur|durtd| tvs$Jd| ||_||_| |_||_||_ ||_ ||_ ||_ |durD||_ n||_ |durU|jdi| dSdS)Nz6hf_model_id_path or model parameters has to be passed.zahf_model_id_path will be ignored and the HuggingFace model set with model parameter will be used.zTask {0} is not a support task.) ValueErrorLOGGERwarningSUPPORTED_TASKSformatrrrrrtokenizer_paddingtokenizer_truncationtokenizer_padding_sider_load) selfrrrrrrr r!r hf_kwargsrrZ/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/deploy/nlp/hf_deployable.py__init__9s* zHuggingFaceLLMDeploy.__init__returncKs|jdus Jd|jdkr(tj|jfi||_|jdur't|j|j|_ntd|j|j t j|j | dd|j |j|jd|_|jjdurV|jj|j_dSdS)aV Load the HuggingFace pipeline with the specified model and task. This method initializes the HuggingFace AutoModel classes using the provided model configuration and task type. It handles the model and tokenizer loading process. Raises: AssertionError: If task is not specified. Nz/A task has to be given for the generation task.rzTask {0} is not supported.trust_remote_codeF)r(padding truncation padding_side)rr from_pretrainedrrrrrrcudar rpoprr r!r pad_token eos_token)r#r$rrr%r"`s$     zHuggingFaceLLMDeploy._loadkwargscKs|jstd|j|dd|j|jd}i||}|d|D]\}}t|r2| ||<q#t |jj di|}Wdn1sKwY| dd}|r~d|jj |d d d i}| d drp|d |d <| ddr||d|d<|S|jj |d d }|S)aGenerate text based on the provided input prompts. This method processes input prompts through the loaded pipeline and generates text according to the specified parameters. Args: **kwargs: Generation parameters including: - text_inputs: List of input prompts - max_length: Maximum number of tokens to generate - num_return_sequences: Number of sequences to generate per prompt - temperature: Sampling temperature - top_k: Number of highest probability tokens to consider - top_p: Cumulative probability threshold for token sampling - do_sample: Whether to use sampling - return_full_text: Whether to return full text or only generated part Returns: If output logits and output scores are False: List[str]: A list of generated texts, one for each input prompt. If output logits and output scores are True: Dict: A dictionary containing: - sentences: List of generated texts - logits: List of logits - scores: List of scores Raises: RuntimeError: If the pipeline is not initialized. zModel is not initialized text_inputspt)return_tensorsr)r*Nreturn_dict_in_generateF sentences sequencesT)skip_special_tokens output_logitslogits output_scoresscoresr)r RuntimeErrorrrr r.itemstorch is_tensorr-no_gradgenerateget batch_decode)r#r1inputskeyval generated_idsr5outputrrr%rBs6!          zHuggingFaceLLMDeploy.generatec Cs tjdtjdd}tjj|dd|dkrAtdgdd}tdgdd\}}}}}}d } |s0|r2d} |j|d||||||| d ndSq) zD Generate function for ranks other than the rank 0. Tr-dtypedevicersrcNdatarOF r2 do_sampletop_ktop_p temperaturemax_new_tokensr9r;r5)r?emptylong distributed broadcastr rB) r#messagepromptsrVrTrUnum_tokens_to_generater9r;r5rrr%generate_other_rankss0 z)HuggingFaceLLMDeploy.generate_other_rankscCstddtdtddtjddtddtjddtddtjddtd dtjddtd dtjddtd dtjddtddtjddtd dtjddtd dtjddf }|S)Nr]nameshaperL max_lengthT)rcrdrLoptionalmax_batch_sizerTrUrV random_seedr9r;)rbytesnpint_singlebool_)r#rErrr%get_triton_inputs  z%HuggingFaceLLMDeploy.get_triton_inputcCs,tddtdtddtjdtddtjdfS)Nr6r`rbr:r<)rrirjrl)r#rrr%get_triton_outputs z&HuggingFaceLLMDeploy.get_triton_outputrEc Ksi}zs