o TÃiã@s„ddlmZmZmZmZmZddlZddlmZddl m Z ddl m Z ddl mZdd lmZGd d „d e ƒZGd d „d eƒZdS)é)ÚAnyÚDictÚOptionalÚTupleÚTypeNé)ÚRaggedBatchWrapper)ÚDeepSpeedConfigModelé)Ú DSModuleBase)ÚDSModuleRegistryBase©ÚDSSelfAttentionConfigcsÒeZdZdZedeefdd„ƒZdede e e fddf‡fdd „ Z e defd d „ƒZe defd d „ƒZdeddfdd„Z   ddejdejdedeejdeejdeejdeejejffdd„Z‡ZS)ÚDSSelfAttentionBasea Base mixin for all attention modules. The interface represented by this module is broadly: output = attention(query_key_value, Optional[kv_cache], Optional[attention_mask], Optional[attention_bias]) ÚreturncCótS©Nr ©rrúl/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/inference/v2/modules/interfaces/attention_base.pyÚ config_classóz DSSelfAttentionBase.config_classÚconfigÚimplementation_configNcstƒ ||¡dSr)ÚsuperÚ__init__)Úselfrr©Ú __class__rrr szDSSelfAttentionBase.__init__cCótƒ‚)zS Return preferred granulatity for blocked KV-cache implementation. ©ÚNotImplementedError©rrrrÚ kv_block_size#sz!DSSelfAttentionBase.kv_block_sizecCr)zÛ Property to calculate blocking granularity for the query dimension. This has no impact on the KV-cache structure, but will affect the number of attention atoms associated with a batch. rr!rrrÚ q_block_size*sz DSSelfAttentionBase.q_block_sizeÚ ragged_batchcCsdS)z¥ Build the atoms for this module. This is not a strict requirement for the class, so this method is a no-op by default rather than abstract. Nr)rr$rrrÚ build_atoms3szDSSelfAttentionBase.build_atomsÚq_k_vÚkv_cacheÚbatchÚattention_maskÚattention_biasÚ inv_freqscCr)aÕ Parameters: q_k_v (torch.Tensor): Query, key, and value tensors. Expected shape is: [ batch, seq_len, 2 * self._config.n_heads_kv + self._config.n_heads_q, self._config.head_size ]. kv_cache (Optional[torch.Tensor]): Key and value cache tensor. Expected shape is [ 2, batch, kv_cache_len, self._config.n_heads_kv, self._config.head_size ]. If None, cache is disabled. The `kv_cache_len` dimension does not need to be contiguous (it should expand stride by `max_out_tokens`). batch (RaggedBatchWrapper): Ragged batch metadata. attention_mask (Optional[torch.Tensor]): Attention mask tensor. If None, masking is disabled. This will defer to the config in the case of conflicting information. This means if the config class is implying causal attention, the mask will be ignored. attention_bias (Optional[torch.Tensor]): Attention bias tensor. If None, bias is disabled. r)rr&r'r(r)r*r+rrrÚforward:szDSSelfAttentionBase.forward)NNN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú staticmethodrr rrrÚstrrrÚpropertyÚintr"r#rr%ÚtorchÚTensorrrr,Ú __classcell__rrrrrs8 " úÿþýüûúúrc@s0eZdZUiZeed<edeefdd„ƒZ dS)ÚDSSelfAttentionRegistryÚregistryrcCrr)rrrrrÚassociated_class_rz(DSSelfAttentionRegistry.associated_classN) r-r.r/r9rÚ__annotations__r1rr r:rrrrr8\s r8)Útypingrrrrrr5ÚraggedrÚdeepspeed.runtime.config_utilsr Ú ds_moduler Úmodule_registryr Úconfigsrrr8rrrrÚs     K