o }oi@slddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z ddl mZGdd d eZdS) )AnyDictListN)tensor_parallel)AbstractModelInferenceWrapper)InferenceWrapperConfig)InferenceParams)default_collatec seZdZdZdeffdd Z ddejdee fdd Z d e e e fd e d e d e e e ffddZd e e e fd ejfddZZS)LlavaInferenceWrappera3Constructor for the model inference wrapper The wrapper prepares the model for inference, provides the required input data, and runs the forward pass Args: model (NevaModel): The Neva model inference_wrapper_config (InferenceWrapperConfig): the config of inference wrapper inference_wrapper_configcst|||jjj|_dSN)super__init__modelmodule _img_seq_len)selfrr  __class__j/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/vlm/inference/llava_inference_wrapper.pyr&szLlavaInferenceWrapper.__init__Nprompts_tokens image_dictcCs|t|djdd}||dddd}|d}|d}tj|tj|jdd |}t |||j |_ |||d S) N pixel_valuesT) non_blockingriP)dtypedevice input_ids position_idsimages) r cudareshapesizetorcharangelongr unsqueeze expand_asrrinference_params)rrrmedia batch_size seq_lengthr!rrrprep_inference_input*s  z*LlavaInferenceWrapper.prep_inference_inputinference_inputcontext_start_positioncontext_end_positionreturncCsT|ddd||f}|ddd||f}|dk|jd|_|||ddS)Nr r!rrr"r)rimg_token_offset)rr0r1r2 tokens2use positions2userrrget_batch_for_context_windowCsz2LlavaInferenceWrapper.get_batch_for_context_windowcCsF|jdd|jd|}t|}|jj|dd|j7_|S)aUtility to carry out simple forward pass for TP or no model parallel models Runs a very simple forward pass for model. Used in the case of models without any parallelism or only tensor parallelism. Args: inference_input (List): A list containg the inputs for the neva model [input ids, position ids, media] Returns: torch.Tensor: The output logits of shape [batch_size, seq_len, padded_vocab_size] N)attention_maskr+r rr)rr+r(gather_from_tensor_model_parallel_regionsequence_len_offsetr%r4)rr0logitsrrr&forward_pass_without_pipeline_parallelTs  zs