o i$ @stddlZddlmZddlmZmZddlZddlmZm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZmZe eZeGd ddZdede ddfddZ!dedede fddZ" d(de#e$e$fde%ede&e$dBddfddZ' d)de#e$ej(fde#e$e fd e%ej(d!e ddf d"d#Z)d$e d%e de*fd&d'Z+dS)*N) defaultdict) dataclassfield) CacheConfig VllmConfig) init_logger) Attention)MultiModalEmbeddings)extract_layer_index)current_platform)MemorySnapshot format_gib)AttentionBackendAttentionMetadataBuilder)KVCacheGroupSpec KVCacheSpecc@s~eZdZUeeed<eeed<eed<e ed<e dddZ ee ed< dd e d Bd e fd dZ dde de fddZd S)AttentionGroupbackend layer_names kv_cache_speckv_cache_group_idcCsgSNrrrJ/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/v1/worker/utils.py szAttentionGroup.)default_factorymetadata_builderskernel_block_sizeNnum_metadata_builderscs<|dur j|njfddt|D_dS)Ncs"g|] }jjqSr)rget_builder_clsr).0_devicekv_cache_spec_builderself vllm_configrr /sz;AttentionGroup.create_metadata_builders..)rcopy_with_new_block_sizeranger)r&r'r$rrrr#rcreate_metadata_builders#s  z'AttentionGroup.create_metadata_buildersr ubatch_idreturncCst|j|ks J|j|Sr)lenr)r&r,rrrget_metadata_builder9s z#AttentionGroup.get_metadata_builderr)r)__name__ __module__ __qualname__typer__annotations__liststrrintrrrr+r/rrrrrs    r mm_embeddingsexpected_num_itemsr-cCs|t|tttjfsJdt|dt||ks&Jd|dt|dtdd|DsTsz2sanity_check_mm_encoder_outputs..z[Expected multimodal embeddings to be a sequence of 2D tensors, but got tensors with shapes cSsg|]}|jqSr)shaper=rrrr(Vsz3sanity_check_mm_encoder_outputs..N) isinstancer6tupletorchTensorr4r.all)r9r:rrrsanity_check_mm_encoder_outputs>s" rF init_snapshot cache_configc Cs\t|j|j}|j|kr,td|jdt|jdt|jd|jdt|d |S)z Calculate the amount of memory required by vLLM, then validate that the current amount of free memory is sufficient for that. zFree memory on device z (/z> GiB) on startup is less than desired GPU memory utilization (z, zT GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.)mathceil total_memorygpu_memory_utilization free_memory ValueErrordevice_r )rGrHrequested_memoryrrrrequest_memory\s     rRshared_kv_cache_layerskv_cache_groupsrunner_only_attn_layerscCs^i}|D] }|jD]}|||<q q|D]\}}||}|j||dur,||qdS)a Sets up KV cache sharing by reusing the allocated KV caches in `kv_caches` for layers that do not allocate its own KV cache, based on the mapping in `shared_kv_cache_layers`. Adds these layers to the corresponding KV cache group, which is needed to ensure that attention metadata is assigned later. Args: shared_kv_cache_layers: Layer pairings for cross-layer KV sharing. If an Attention layer `layer_name` is in the keys of this dict, it means this layer will perform attention using the keys and values from the KV cache of `shared_kv_cache_layers[layer_name]`. kv_cache_groups: The KV cache groups of the model. N)ritemsappendadd)rSrTrUlayer_to_kv_cache_groupkv_cache_group layer_nametarget_layer_nametgt_kv_cache_grouprrr(add_kv_sharing_layers_to_kv_cache_groupsss    r^r kv_cachesforward_contextrunner_kv_cachesnum_attn_modulec Cst|dksJtt}|D] }|t|||qt|D]'}||}t|dkrq!| D] \}}|g||_ qMdS)a{ Bind the allocated KV cache to both ModelRunner and forward context so that the KV cache can be used in the forward pass. This function: 1) Fills the ModelRunner's kv cache list (`runner_kv_caches`) with kv_caches. 2) Associates each attention layer in the `forward_context` with its corresponding KV cache in kv_caches. Args: kv_caches: The allocated kv_caches with layer names as keys. forward_context: The global forward context containing all Attention layers with layer names as keys. runner_kv_caches: The kv_cache declared by ModelRunner. rrN)r.rr6r rWsortedkeysr is_cuda_alikeis_xpuis_cpuNotImplementedErrorrVkv_cache) r_r`rarb index2namer[ layer_indexrrirrr bind_kv_caches*  rlr'num_input_tokenscCsb|jjjsdS|jj}|dkrdS||dksJ|jjr!|jjr#dS|jj}|dur-dS||vS)aCheck if the residual tensor is scattered for sequence parallelism. The residual tensor is scattered across tensor parallel ranks when sequence parallelism and tensor parallelism is enabled. This follows the same logic as SequenceParallelismPass.is_applicable_for_range(): - In full-graph compilation mode (no splitting ops or using inductor graph partition), SP is always applied - Otherwise, SP is only applied for specific shapes in compile_sizes FrrTN)compilation_config pass_config enable_spparallel_configtensor_parallel_size splitting_opsuse_inductor_graph_partition compile_sizes)r'rmtprurrris_residual_scattered_for_sps rwrr0),rJ collectionsr dataclassesrrrC vllm.configrr vllm.loggerr$vllm.model_executor.layers.attentionr%vllm.model_executor.models.interfacesr vllm.model_executor.models.utilsr vllm.platformsr vllm.utils.mem_utilsr r vllm.v1.attention.backendrrvllm.v1.kv_cache_interfacerrr1loggerrr8rFrRdictr7r6setr^rDrlboolrwrrrrsj      '    #   <