o -i'@sUddlmZddlmZmZmZddlmZmZm Z m Z ddl m Z ddl mZddlmZddlmZe Gdd d Ze ed d dGd d d eZe ed d dGdddeZe ed d dGdddeZedZedZeeBeBeBZeed<ee GdddZdS))Mapping)AnyLiteral TypeAlias) ConfigDictFieldfield_validatormodel_validator) dataclass)config) safe_hash)AttentionBackendEnumc@s&eZdZUdZedddZeed<dS)BaseDummyOptionsz8Base options for generating dummy data during profiling.r)gecountN)__name__ __module__ __qualname____doc__rrint__annotations__rrS/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/config/multimodal.pyrs rforbid)extrac@sZeZdZUdZedddZedBed<edddZedBed<edddZ edBed<dS)VideoDummyOptionsz9Options for generating dummy video data during profiling.Nrgt num_frameswidthheight) rrrrrrrrr r!rrrrrs rc@sBeZdZUdZedddZedBed<edddZedBed<dS)ImageDummyOptionsz9Options for generating dummy image data during profiling.Nrrr r!) rrrrrr rrr!rrrrr"s r"c@s*eZdZUdZedddZedBed<dS)AudioDummyOptionsz9Options for generating dummy audio data during profiling.Nrrlength)rrrrrr$rrrrrrr#'s r#)weightsdata)shmlru DummyOptionsc @seZdZUdZeedZeeefe d< dZ e e d< eedZ eeeee ffe d< dZeeefdBe d< ed d d Zee d < d Zee d< edd d Zee d< dZe e d< dZee d< dZedBe d< dZe e d< dZe e d< eddddZedBe d< edddedeeeeeefBfdeeeffdd Z edddedeeBdBdedBfd!d"Z!e"d#dd$d%Z#defd&d'Z$d(edefd)d*Z%d(ede&dBfd+d,Z'd-e(eefdeeeffd.d/Z)d0d1Z*dS)2MultiModalConfigz+Controls the behavior of multimodal models.)default_factorylimit_per_promptFenable_mm_embedsmedia_io_kwargsNmm_processor_kwargsr)defaultrmm_processor_cache_gbr(mm_processor_cache_typemm_shm_cache_max_object_size_mbmm_encoder_onlyr%mm_encoder_tp_modemm_encoder_attn_backendinterleave_mm_stringsskip_mm_profilinggg?)r1rltvideo_pruning_ratebefore)modevaluereturncCs|D]@\}}t|trd|i}|dkrtdi|||<q|dkr-tdi|||<q|dkr;tdi|||<qtdi|||<q|S)Nrvideoimageaudior)items isinstancerrr"r#r)clsr?kvrrr_validate_limit_per_prompts z+MultiModalConfig._validate_limit_per_promptcCsRt|tr|dkrtd|dust|tr|St|ts#Jdt|S)NXFORMERSzwAttention backend 'XFORMERS' has been removed (See PR #29262 for details). Please select a supported attention backend.zDmm_encoder_attn_backend must be a string or an AttentionBackendEnum.)rEstrupper ValueErrorr )rFr?rrr!_validate_mm_encoder_attn_backends  z2MultiModalConfig._validate_mm_encoder_attn_backendaftercCs"|jdkr|jtjkrtd|S)Nr'z]'mm_shm_cache_max_object_size_mb' should only be set when 'mm_processor_cache_type' is 'shm'.)r3r5r*rMselfrrr_validate_multimodal_configs z,MultiModalConfig._validate_multimodal_configcCs8|jdur |jjnd|jg}tt|dd}|S)a WARNING: Whenever a new field is added to this config, ensure that it is included in the factors list if it affects the computation graph. Provide a hash that uniquely identifies all the configs that affect the structure of the computation graph from input ids/embeddings to the final hidden states, excluding anything before input ids/embeddings and after the final hidden states. NF)usedforsecurity)r8namer7r rKencode hexdigest)rQfactorshash_strrrr compute_hashs zMultiModalConfig.compute_hashmodalitycCs|j|}|dur dS|jS)z Get the maximum number of input items allowed per prompt for the given modality (backward compatible). Nr)r,getr)rQrZ limit_datarrrget_limit_per_prompts z%MultiModalConfig.get_limit_per_promptcCs |j|S)z Get the configurable dummy data options for a modality. Returns None if no options are configured for this modality. )r,r[)rQrZrrrget_dummy_optionss z"MultiModalConfig.get_dummy_optionsinference_kwargscCs|jpi}|t|BS)z Get the keyword arguments to pass to the multi-modal processor according to the extra arguments passed during inference. )r/dict)rQr_kwargsrrrmerge_mm_processor_kwargss  z*MultiModalConfig.merge_mm_processor_kwargscCs|jduo |jdkS)Nr)r<rPrrris_multimodal_pruning_enabledsz.MultiModalConfig.is_multimodal_pruning_enabled)+rrrrrr`r,rKr)rr-boolr.rr/objectr2floatr3 MMCacheTyper5rr6r7MMEncoderTPModer8r r9r:r<r classmethodrIrNr rRrYr]rr^rrbrcrrrrr*5sj  "                r*N)collections.abcrtypingrrrpydanticrrrr pydantic.dataclassesr vllm.config.utilsr vllm.utils.hashingr #vllm.v1.attention.backends.registryr rrr"r#rhrgr)rr*rrrrs,