o ۾i<@sdZddlZddlmZmZmZddlmZddlZddl m Z ddl m Z ddl mZddlmZddlmZmZmZmZdd lmZdd lmZmZmZmZdd lmZdd lm Z m!Z!m"Z"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6m7Z7m8Z8ddl9m:Z:ddl;mZ>ddl?m@Z@ddlAmBZBdZCdZDGddde,ZEGddde*eEZFGd d!d!e+eEZGejHeGeEeFd"Gd#d$d$e jIeeeZJdS)%zFInference-only Deepseek-OCR model compatible with HuggingFace weights.N)IterableMappingSequence)partial) BatchFeature) VllmConfig)BaseDummyOptions)MultiModalEmbeddings SupportsLoRASupportsMultiModal SupportsPP)MultiModelKeys)AutoWeightsLoader WeightsMapperinit_vllm_registered_model maybe_prefix)MULTIMODAL_REGISTRY)MultiModalDataDictMultiModalFieldConfigMultiModalKwargsItems NestedTensors)ImageEmbeddingItemsImageProcessorItems ImageSizeMultiModalDataItems)BaseDummyInputsBuilderBaseMultiModalProcessorBaseProcessingInfoPromptReplacement PromptUpdate)IntermediateTensors)cached_tokenizer_from_config)DeepseekVLV2Config) BASE_SIZE CROP_MODEDeepseekOCRProcessor) count_tiles)ImageEncoderViT)build_qwen2_decoder_as_encoder)DeepseekOCRImagePixelInputs) MlpProjectorc @sjeZdZddZdefddZdeeedBffdd Z d d d ed ede defddZ de fddZ dS)DeepseekOCR2ProcessingInfocCs |jtSN)ctx get_hf_configr"selfr5\/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/deepseek_ocr2.pyr2Es z(DeepseekOCR2ProcessingInfo.get_hf_configkwargscKs,ttttdd}|jjtfii||S)Nv2) image_size base_size crop_modestrategy)dict IMAGE_SIZEr#r$r1get_hf_processorr%)r4r7v2_processor_configr5r5r6r?Hs z+DeepseekOCR2ProcessingInfo.get_hf_processorreturnNcCsddiS)Nimager5r3r5r5r6get_supported_mm_limitsSsz2DeepseekOCR2ProcessingInfo.get_supported_mm_limitsT)cropping image_width image_heightrDcCst}t}d}d}tr#|dkr|dkrddg}nt||td}|\} } nd} } t|||} } t|||} }| | }| dksI| dkrR| | | |}nd}||dS)Nr-r()r9r)r>r#r$r'mathceil)r4rErFrDr9r: patch_sizedownsample_ratio crop_rationum_width_tilesnum_height_tileshwh2w2global_views_tokenslocal_views_tokensr5r5r6get_num_image_tokensVs&   z/DeepseekOCR2ProcessingInfo.get_num_image_tokenscCs(tdkrtdkrtdddStdddS)Nii)widthheighti)r>r#rr3r5r5r6!get_image_size_with_most_featuresws  z.get_replacement_deepseek_vl2rB)modalitytarget replacementr5)rer?rrr`r)r4rr}r hf_processorrr5rr6_get_prompt_updatessz3DeepseekOCR2MultiModalProcessor._get_prompt_updatesN)r[r\r]r_rr^rr{rrrrrrrr5r5r5r6ros8        ro)re dummy_inputsc sneZdZedddddddZeded ed ed Bfd d Zddde deffddZ de d e d BfddZ dejd ejfddZdejd ejd BfddZdejdejdejd efdd Zd!e d ejfd"d#Zde d ed Bfd$d%Z d4d&ejd'ejd(ed Bd)ejd Bde f d*d+Zd,ejd ejd Bfd-d.Zd/eeeejfd eefd0d1Zd efd2d3ZZ S)5DeepseekOCR2ForCausalLMz"language_model.model.embed_tokens.zlanguage_model.model.layers.zlanguage_model.model.norm.zlanguage_model.lm_head.)zmodel.embed_tokens.z model.layers.z model.norm.zlm_head.zmodel.)orig_to_new_prefixrirANcCs|drdStd)NrBr.z Only image modality is supported) startswith ValueError)clsrrr5r5r6get_placeholder_strs z+DeepseekOCR2ForCausalLM.get_placeholder_str)prefix vllm_configrc st|jj}|jj}||_||_|j|_|j|_|j|_|j}t |}|j t |_ | |d^tddddttjjddddd d gd d d d d |_t|_t|j|_|j|_|j|_|jj}dttj|tjd}|jdkrtt|||_ nt!d|jWdn1swY|"|t#||jt$|dd|_%Wdn1swY|j%j&|_&dS)NrB r-rWrHgư>)epsrGT)r i) depth embed_dimimg_size mlp_ratio norm_layer num_headsrKqkv_bias use_rel_posglobal_attn_indexes window_size out_chanslast_conv_outputr(dtype2Dz.Only 2D tile_tag is supported currently, got: language_model)r hf_configr)'super__init__ model_configrmultimodal_configconfig vision_configprojector_config text_configr!vocab _IMAGE_TOKENr_mark_tower_modelr)rrnn LayerNorm sam_modelr* qwen2_modelr, projectortile_tagglobal_view_posn_embedsqrttensorfloat32 Parameterrandnview_seperatorr_mark_language_modelrrrmake_empty_intermediate_tensors) r4rrrrrrzr embed_std __class__r5r6rsb      ! z DeepseekOCR2ForCausalLM.__init__r7cKs`|dd}|dd}|dd}|dust|dkr!dS|jj}td|||d|idS)Nrr~rrr:)typedatarr~resolve_bindings)poprsumitemrr9r+)r4r7rr~rr:r5r5r6_parse_and_validate_image_input>s   z7DeepseekOCR2ForCausalLM._parse_and_validate_image_input image_tensorcCs6||}||}||}|j\}}}|d|S)Nr)rrrshapeview)r4rglobal_features_1global_features_2features_hwrr5r5r6_encode_global_featuresSs     z/DeepseekOCR2ForCausalLM._encode_global_featurespatchescCsLt|dkr dS||}||}||}|j\}}}|d|S)Nrr)rrrrrrrr)r4rlocal_featuresrrrr5r5r6_encode_local_features]s     z.DeepseekOCR2ForCausalLM._encode_local_featuresrrr~c Csg}|dddfdk|dddfdkB}t||jddd}||}t|dD]@}||}||g} || } ||} | durZtj | | |j dddfgdd} ntj | |j dddfgdd} | | q/|S)Nrr(rr) rrrsplittolistrangerrrcatrappend) r4rrr~images_in_this_batchrrjdxr image_origlobal_featuresrcombinedr5r5r6_pixel_values_to_embeddingjs&(    z2DeepseekOCR2ForCausalLM._pixel_values_to_embedding image_inputcCs0|j}|j}|jjtjd}|j|||d}|S)Nr)rrr~)rrr~torlongr)r4rrrr~vision_featuresr5r5r6_process_image_inputsz,DeepseekOCR2ForCausalLM._process_image_inputcKs*|jdi|}|durdS||}|S)Nr5)rr)r4r7rvision_embeddingsr5r5r6embed_multimodals  z(DeepseekOCR2ForCausalLM.embed_multimodal input_ids positionsintermediate_tensors inputs_embedscKs"|durd}|j||||d}|S)N)r )r)r4rrrr r7 hidden_statesr5r5r6forwards zDeepseekOCR2ForCausalLM.forwardr cCs |j|Sr0)rcompute_logits)r4r r5r5r6r s z&DeepseekOCR2ForCausalLM.compute_logitsweightscCst|}|j||jd}|S)N)mapper)r load_weightshf_to_vllm_mapper)r4r loaderautoloaded_weightsr5r5r6rsz$DeepseekOCR2ForCausalLM.load_weightscCstjddddgdS)z< Get the module prefix in multimodal models rrrr)r connector tower_model)r from_string_fieldr3r5r5r6get_mm_mappings z&DeepseekOCR2ForCausalLM.get_mm_mapping)NN)!r[r\r]rr classmethodr_r`rrrr^r+rrTensorrrrrrr rr r r rtuplesetrr r __classcell__r5r5rr6rsl <         $r)K__doc__rIcollections.abcrrr functoolsrrtorch.nnr transformersr vllm.configrvllm.config.multimodalr%vllm.model_executor.models.interfacesr r r r )vllm.model_executor.models.module_mappingr vllm.model_executor.models.utilsrrrrvllm.multimodalrvllm.multimodal.inputsrrrrvllm.multimodal.parserrrrvllm.multimodal.processingrrrrr vllm.sequencer vllm.tokenizersr!,vllm.transformers_utils.configs.deepseek_vl2r"/vllm.transformers_utils.processors.deepseek_ocrr#r$r%*transformers_utils.processors.deepseek_ocrr' deepencoderr) deepencoder2r* deepseek_ocrr+ deepseek_vl2r,r>rr/rbroregister_processorModulerr5r5r5r6sP                9 N