o پi@sdZddlmZddlZddlZddlmZmZmZm Z m Z ddl Z ddl m Z ddl mZmZddlmZmZmZmZmZmZmZddlmZdd lmZmZdd lmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&dd l'm(Z(m)Z)m*Z*m+Z+m,Z,erdd l-m.Z.m/Z/e*Z0e0oe(dZ1e2e3Z4gdZ5e)Z6e+Z7ddZ8d0ddZ9ddZ:ddZ;Gd d!d!e j j<Z=Gd"d#d#e=Z>Gd$d%d%e=Z?Gd&d'd'e?Z@Gd(d)d)e?ZAGd*d+d+e=ZBGd,d-d-e=ZCGd.d/d/e j<ZDdS)1zhAdapted from https://github.com/vllm-project/vllm/blob/v0.6.4.post1/vllm/model_executor/layers/linear.py) annotationsN) TYPE_CHECKINGDictListOptionalTuple)nn) ParameterUninitializedParameter)divideget_tensor_model_parallel_rank$get_tensor_model_parallel_world_size get_tp_groupsplit_tensor_along_last_dim tensor_model_parallel_all_gather tensor_model_parallel_all_reduce)use_symmetric_memory)get_attention_tp_groupis_allocation_symmetric)BasevLLMParameterBlockQuantScaleParameterPackedColumnParameterPackedvLLMParameterPerTensorScaleParameterRowvLLMParameter_ColumnvLLMParameter)pad_or_narrow_weight)get_bool_env_varis_cpuis_hipis_npuset_weight_attrs)QuantizationConfigQuantizeMethodBaseSGLANG_ROCM_DISABLE_LINEARQUANT)CompressedTensorsLinearMethodAWQMarlinLinearMethodAWQLinearMethodAWQLinearAscendMethodGPTQMarlinLinearMethodFp8LinearMethodBlockInt8LinearMethodMarlinLinearMethodQQQLinearMethodGPTQMarlin24LinearMethodTPUInt8LinearMethodGPTQLinearMethodFBGEMMFp8LinearMethodGPTQLinearAscendMethodModelOptFp8LinearMethodModelOptFp4LinearMethodIPEXAWQLinearMethodPetitNvFp4LinearMethodQuarkInt4Fp8LinearMethodcCs,t|dd}|dur||fS||||fS)Nmarlin_tile_size)getattr)param shard_size shard_offsetr8r=L/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/linear.pyadjust_marlin_shardPs r?r:r shard_offsetsDict[str, Tuple[int, int]]loaded_shard_idstrreturnTuple[int, int]c CsD|d\}}||\}}|jjd}|||}|||} | |fS)zDAdjust the quantization offsets and sizes for BitsAndBytes sharding.totalr)datashape) r:r@rBrF_ orig_offset orig_sizequantized_totalquantized_offsetquantized_sizer=r=r>adjust_bitsandbytes_4bit_shardXs     rOcCsldddd}t|tr||}n t|tstd|t|jdkr0|jddks,J|d}|||fS)aFor fused modules (QKV and MLP) we have an array of length N that holds 1 scale for each "logical" matrix. So the param is an array of length N. The loaded_weight corresponds to one of the shards on disk. Here, we slice the param based on the shard_id for loading. rqkvzUnknown Shard Id ) isinstancerCint ValueErrorlenrH)r: loaded_weightshard_idqkv_idxsr=r=r>adjust_scalar_to_fused_arraygs     r]c Csn||}|dd|dd}||kr5g}d}|D]\}}} || |} |||| f|| 7}q|S|S)Nr)sizeappend) r@rZdimactual_weight_sizetarget_weight_sizenew_shard_offsets new_offsetr[r<r;actual_shard_sizer=r=r>adjust_shard_offsets~s   rhcs6eZdZdZ    ddfdd ZdddZZS) LinearBaseaaBase linear layer. Args: input_size: input dimension of the linear layer. output_size: output dimension of the linear layer. bias: If true, add bias. skip_bias_add: If true, skip adding bias but instead return it. params_dtype: Data type for the parameters. quant_config: Quantization configure. FN input_sizerW output_size skip_bias_addbool params_dtypeOptional[torch.dtype] quant_configOptional[QuantizationConfig]prefixrCcslt||_||_||_|durt}||_||_|dur,ddl m }||_ dS|j ||d|_ dS)Nr)UnquantizedLinearMethodrs) super__init__rkrlrmtorchget_default_dtyperorq&sglang.srt.layers.quantization.unquantrt quant_methodget_quant_method)selfrkrlrmrorqrsrt __class__r=r>rws   zLinearBase.__init__x torch.TensorrDcCstN)NotImplementedError)r}rr=r=r>forwardszLinearBase.forwardFNNrj) rkrWrlrWrmrnrorprqrrrsrC)rrrDr)__name__ __module__ __qualname____doc__rwr __classcell__r=r=r~r>risricsLeZdZdZ     d!d"fdd Zd#ddZd$ddZd%dd ZZS)&ReplicatedLinearaReplicated linear layer. Args: input_size: input dimension of the linear layer. output_size: output dimension of the linear layer. bias: If true, add bias. skip_bias_add: If true, skip adding bias but instead return it. params_dtype: Data type for the parameters. quant_config: Quantization configure. prefix: The name of the layer in the state dict, including all parents (e.g. model.layers.0.qkv_proj) TFNrjrkrWrlbiasrnrmrorprqrrrsrCc stj||||||d|jdusJ|jj||j|jg|j|j|j|jd|rAtt j |j|jd|_ t |j d|jddS| dddS)Nru) weight_loaderdtyper output_dimrr)rvrwr{create_weightsrkrlrorr rxemptyrr!register_parameter)r}rkrlrrmrorqrsr~r=r>rws:   zReplicatedLinear.__init__r:r rZrcCst|jdkr |d}trI||kr3|ddkr3t||dr,|dd}nt|d|jtj ks?|jtj krI|j|jksIJd||ksSJ|j |dS)NrrPz are not all equalz:init para dtype and loaded weight dtype should be the same) rYrHreshape_is_npur`rxallcloserXrint8rGcopy_r}r:rZr=r=r>rs zReplicatedLinear.weight_loaderrrD+Tuple[torch.Tensor, Optional[torch.Tensor]]cCsF|js|jnd}|jdusJ|j|||}|jr|jnd}||fSr)rmrr{apply)r}rroutput output_biasr=r=r>rs zReplicatedLinear.forwardcCs4d|j}|d|j7}|d|jdu7}|S)N in_features=, output_features=, bias=)rkrlrr}sr=r=r> extra_reprs zReplicatedLinear.extra_repr)TFNNrj)rkrWrlrWrrnrmrnrorprqrrrsrCr:r rZr)rrrDrrDrC) rrrrrwrrrrr=r=r~r>rs - rcs`eZdZdZ           d)d*fdd Zd+d d!Zd+d"d#Zd$d%Zd,d'd(ZZ S)-ColumnParallelLinearaCLinear layer with column parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its second dimension as A = [A_1, ..., A_p]. Args: input_size: first dimension of matrix A. output_size: second dimension of matrix A. bias: If true, add bias. gather_output: If true, call all-gather on output and make Y available to all GPUs, otherwise, every GPU will have its output which is Y_i = XA_i skip_bias_add: This was added to enable performance optimizations where bias can be fused with other element-wise operations. we skip adding bias but instead return it. params_dtype: Data type for the parameters. quant_config: Quantization configure. output_sizes: list of output sizes packed into one output, like for QKV the list would be size 3. prefix: The name of the layer in the state dict, including all parents (e.g. model.layers.0.qkv_proj) TFNrjrkrWrlrrn gather_outputrmrorprqrr output_sizesOptional[List[int]]rsrCtp_rank Optional[int]tp_sizeuse_presharded_weightsskip_block_quant_checkc st|||||| ||_| |_| durt} durt| |_|_|jdus-Jt |j |_ |j g|_ t |drIfdd|jD|_ |durP|g}|jj||j|j |j|j |j| |jjjtvri|jn|jd|rttj|j |d|_t|jd|jddS|dddS) Nrcsg|]}t|qSr=)r .0rlrr=r> Os z1ColumnParallelLinear.__init__..layerinput_size_per_partitionoutput_partition_sizesrkrlrorrrrrr)rvrwrrr r rrr{r rloutput_size_per_partitionrhasattrrrrkrorrWEIGHT_LOADER_V2_SUPPORTEDweight_loader_v2rr rxzerosrr!r)r}rkrlrrrmrorqrrsrrrrr~rr>rw-sT    zColumnParallelLinear.__init__r:r rZrc Cst|dd}|j}t|dd}t|dd}|r||_|r,t|tr,|j|j|jdt|dd}|durc|sc|j|}|j |} t rYddl m } | ||d| |||j \}}n |j sc||| |}t|jdkro|d }|j|jkswJ||dS) Nris_gguf_weightFis_gguf_weight_typeruse_bitsandbytes_4bitr%narrow_padded_param_and_loaded_weightrP)r9rGitem weight_typerVr materializerHrr_is_cpu$sglang.srt.model_loader.weight_utilsrrnarrowrYrr) r}r:rZr param_datarrrr; start_idxrr=r=r>rrs>            z"ColumnParallelLinear.weight_loadercCt|jdkr|dksJ|d}t|tr%|j||j|jddSz |j||j|jdWdSt yA||YdSwNrrP)rr) rYrHnumelrrVrload_column_parallel_weightrr TypeErrorrr=r=r>rs$     z%ColumnParallelLinear.weight_loader_v2cCsZ|js|jnd}|jdusJ|j|||}|jrt|}n|}|jr'|jnd}||fSr)rmrr{rrr)r}input_routput_parallelrrr=r=r>rs zColumnParallelLinear.forwardrDcCTd|j}|d|j7}|d|jdu7}|d|j7}|d|j7}|S)Nrrr , tp_size=z, gather_output=)rkrrrrrr=r=r>r zColumnParallelLinear.extra_repr) TFFNNNrjNNFF)rkrWrlrWrrnrrnrmrnrorprqrrrrrsrCrrrrrrnrrnrr rrrrrwrrrrrr=r=r~r>rs$ E 1rcsfeZdZdZ         d(d)fdd Z d*d+dd Zd,d"d#Zd,d$d%Z d*d-d&d'ZZ S).MergedColumnParallelLineara(Packed linear layers with column parallelism. Similar to ColumnParallelLinear, but the weight matrix is concatenated along the output dimension. When the weight matrix is loaded, the different partitions are sharded separately. Args: input_size: input dimension of the linear layer. output_sizes: list of output dimensions of the linear layer. bias: If true, add bias. gather_output: If true, call all-gather on output and make the output available to all GPUs, otherwise, every GPU will have its own output. skip_bias_add: This was added to enable performance optimizations where bias can be fused with other element-wise operations. we skip adding bias but instead return it. params_dtype: Data type for the parameters. quant_config: Quantization configure. prefix: The name of the layer in the state dict, including all parents (e.g. model.layers.0.qkv_proj) TFNrjrkrWr List[int]rrnrrmrorprqrrrsrCrrrrc s||_| dur t} durt| |_|_tfdd|Ds%J| |_tj|t |||||||| | d ||_ dS)Nc3s|] }|dkVqdS)rNr=rrr=r> sz6MergedColumnParallelLinear.__init__..) rkrlrrrmrorqrsrrr) rr r rrallrrvrwsumrs) r}rkrrrrmrorqrsrrrr~rr>rws, z#MergedColumnParallelLinear.__init__r:r rZrrBcst|dd}t|dd}|r|j||||j|<dS|rQt|dd}|||j}|j|}||||}|j |t |j |j |<|j |dS|j} t|dd}t|dd} t|dd} |dur|dur| ryt| |d\} }| j|jksJ| |dSd} g} t|jD]\}}| || |f| |7} qt|dd}t|d d}trt| ||} | D]S\}}}||kr||j}||j}t|||\}}|rttdg|jfd d t|jD}|jdf|d <t||t|\}}||||}||||qdS|t |jksJ|durt|jd||j}|j||j}t|dd}||krN||j}||j}t|||\}}t|d d}|rc|j|}|j||}| |||} |j|}trdd lm}|| |d|||| o|j \} }nQ|s|j s||}||j|krt!||||}n6||||}n.| r|jd}||}| d||} n| rt| ||\} }nt|dd}|st"#d| j|jksJ| |dS)NrFrr is_metadataneeds_scalar_to_arrayr packed_dimrcs"i|] \}}t|||fqSr=)rC)rir`indexr=r> Zsz.rFrignore_warningzLoading a weight without `output_dim` attribute in MergedColumnParallelLinear, assume the weight is the same for all partitions.)$r9rGrrshard_weight_typer`rrrr[rarYdata_container shard_id_mapr]rH enumeraterrrh pack_factorr?list itertools accumulaterlrOrCrrrrrrloggerwarning)r}r:rZrBrrrr;rrrrcurrent_shard_offsetr@rrlrrr[r< orig_offsetsloaded_weight_shardrend_idxrr=rr>rs                              z(MergedColumnParallelLinear.weight_loaderrc Csd}g}t|jD]\}}||||f||7}q |D]*\}}} t|ttfr7|j|jkr7|j| |d\} }| |j|| } | || |qdS)a Handle special case for models where MLP layers are already fused on disk. In this case, we have no shard id. This function determmines the shard id by splitting these layers and then calls the weight loader using the shard id. An example of a model with these fused layers: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct rr;r<N) rrrarVrrrr adjust_shard_indexes_for_packingrr) r}r:rZrr@rrlr[r<r;rr=r=r>"_load_fused_module_from_checkpoints"     z=MergedColumnParallelLinear._load_fused_module_from_checkpointc Cs|jjj}|d|d}}t|ddrdn|}g}g}d}|jD]} | |d|} || |||| 7}q!tt||D]'\} \} } ||j | | } | |j }| |j }|j | | |||j |j |j dqAdS)z Handle block-wise scale loading for MergedColumnParallelLinear. Similar to QKVParallelLinear._load_qkv_block_scale, but for merged column layers. rrP format_ue8m0F)rZr[r<r;rrrN)r{rqweight_block_sizer9rrarziprrrload_merged_column_weightrr)r}r:rZrblock_nrIshard_block_sizesshard_block_offsetscurrent_block_offsetrlshard_block_sizer[shard_block_offsetrrank_shard_offsetrank_shard_sizer=r=r>_load_merged_block_scales:       z3MergedColumnParallelLinear._load_merged_block_scalec CsV|dur?t|tr|j|d|j|jddSt|tr#|||dSt|tt fvr7|j||j|jddS| ||dS|t |j ksHJt|tr|j jj}|d|d}}t|ddrcdn|}t|j d||d||j}|j ||d||j} nt|j d||j}|j ||j} |j|||| |j|j|jddS)Nr)rZr[rr)rZrrrPrF)rZr[r<r;rrr)rVrrrrrrtyperrrrYrr{rqrr9rr) r}r:rZrBr raw_block_nrIrr<r;r=r=r>rs\       z+MergedColumnParallelLinear.weight_loader_v2) TFFNNrjNNF)rkrWrrrrnrrnrmrnrorprqrrrsrCrrrrrrnr)r:r rZrrBrr:rrZr)r:rrZrrBr) rrrrrwrrrrrr=r=r~r>rs&)  ' $1rcs~eZdZdZ           d/d0fdd Zd1ddZd1dd Zd2d%d&Zd2d'd(Z d3d4d*d+Z d3d5d-d.Z Z S)6QKVParallelLineara1Linear layers for the attention's QKV transformation. Linear layers for the linear transformation of the query, key, and value vectors in the attention layer. The weight matrix is concatenated along the output dimension. The layer is parallelized along the head dimension. When the number of key/value heads is smaller than the number of query heads (e.g., multi-query/grouped-query attention), the key/value head may be replicated while the query heads are partitioned. Args: hidden_size: input hidden state size of the transformer. head_size: size of each attention head. total_num_heads: total number of attention query heads. total_num_kv_heads: total number of attention key/value heads. If None, assume total_num_kv_heads = total_num_heads. bias: If true, add bias. skip_bias_add: This was added to enable performance optimizations where bias can be fused with other element-wise operations. we skip adding bias but instead return it. params_dtype: Data type for the parameters. quant_config: Quantization configure. prefix: The name of the layer in the state dict, including all parents (e.g. model.layers.0.qkv_proj) NTFrj hidden_sizerW head_sizetotal_num_headstotal_num_kv_headsrrrnrmrorprqrrrsrCrrload_presharded_attn v_head_sizercsf||_||_| dur | n||_||_|dur|}||_| dur"t} | dur)t} | | |_|_t |j| |_ | |jkrGd|_ t | |j|_ n t |j| |_ d|_ |j |j|_ |j |j|_|j |j|_|j}|j |j|j |j|j |j| }|j |j| |j |j| |j |j| g|_| |_trdn|}tj|||d|||| | | |j|d dS)NrPF) rkrlrrrmrorqrsrrrr)rrr rrr r rrr num_heads num_kv_headsnum_kv_head_replicasq_proj_shard_sizekv_proj_shard_sizev_proj_shard_sizerr_disable_hip_linear_quantrvrw)r}rrrrrrmrorqrsrrrr rrkrlr~r=r>rw\sb      zQKVParallelLinear.__init__rBcCsHd|j|j|j|j|j|j|j|j|j|jd}||S)NrrSrTrUrFr rr r get)r}rBshard_offset_mappingr=r=r>_get_shard_offset_mappings   z+QKVParallelLinear._get_shard_offset_mappingcCs.|j|j|j|j|j|jd}||S)NrRr)r}rBshard_size_mappingr=r=r>_get_shard_size_mappings    z)QKVParallelLinear._get_shard_size_mappingr:rrZrcCsdd|j|jfd|j|j|j|jfd|j|j|j|j|jfg}|D]-\}}}t|ttfrB|j|jkrB|j ||d\}}|j sM| |j||}| |||q'dS)a Handle special case for models where QKV layers are already fused on disk. In this case, we have no shard id. This function determmines the shard id by splitting these layers and then calls the weight loader using the shard id. An example of a model with these fused layers: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct rSrrTrUrN) rrrr rVrrrrrrrr)r}r:rZr@r[r<r;rr=r=r>rs.      z4QKVParallelLinear._load_fused_module_from_checkpointc Cs|jjj\}}|j|j|}|j|j|}|j|j|}dd|fd||fd|||fg}|D]*\} } } ||j| | } || |} | | |}|j | |j | | ||j |j dq1dS)NrSrrTrUrZr r[r<r;rr)r{rqrrrrrrrrload_qkv_weightr rr)r}r:rZrrIq_sizek_sizev_sizer@r[r<r;rrrr=r=r>_load_qkv_block_scales0 z'QKVParallelLinear._load_qkv_block_scale Optional[str]c Cs |dur7t|tr|j|dddSt|ttfvr"|j|ddSt|tr/|||dS|||dS|dvs=J| |}| |}t|trt|j j j }|d|d}}t|ddrbdn|} || d| }|| d| }|j||j||||j|jddS) Nr)rZr[)rZrRrPrFr)rVrrrrrrrrrrr{rqrr9r rr) r}r:rZrBr<r;rrrIrr=r=r>rs<           z"QKVParallelLinear.weight_loader_v2r cCst|dd}t|dd}|r+|dur+dddd}|j|||||j|<dS|r]t|dd}|||j}|j|} ||| |}|j |t |j |j |<|j |dS|j} t|dd}t|d d} t|d d} |dur=|dur| rt| |d\} }| j|jksJ| |dSd d|j|jfd |j|j|j|jfd |j|j|j|j|jfg} t|dd}t|dd}trt| ||} | D]k\}}}||kr||j}||j}t|||\}}|r(d|j|jf|j|j|j|jf|j|j|j|j|jf|j|j|j|j|jdfd}t|||\}}|js3||||}||||qdS|dvsDJ|dur#|d krWd}|j|j}n&|d kri|j|j}|j|j}n|d kr}|j|j|j}|j|j}t|dd}||kr||j}||j}t|||\}}t|dd}|rd|j|jf|j|j|j|jf|j|j|j|j|jf|j|j|j|j|jdfd}t|||\}}| |||} |d kr|j}n|j|j}||} trddlm}|| |d| ||| o|j \} }nB|s"|js"||| |}n3| r<|jd}gd |}| d|||} n| rHt| ||\} }nt|dd}|sVt!"d| j|jksiJd| jd|j| |dS)NrFrrrPrQrRrrrrSrTrUrrrrrzwLoading a weight without `output_dim` attribute in QKVParallelLinear, assume the weight is the same for all partitions.param_data.shape= loaded_weight.shape=)#r9rGrrrr`rrrr[rarYrrr]rHrrrr rrhrr?rOrrr r r rrrrr)r}r:rZrBrridx_maprr;rrrrr@rrr[r<orig_qkv_offsetsrr shard_indexrr=r=r>r's2                                               zQKVParallelLinear.weight_loader) NTFNNrjNNFNF)rrWrrWrrWrrrrnrmrnrorprqrrrsrCrrrrrrnr rrrn)rBrCrr)r:rrZrrBr)r:r rZrrBr) rrrrrwrrrrrrrr=r=r~r>rBs, E  -! .rcsbeZdZdZ           d)d*fdd Zd+dd Zd,d"d#Zd-d$d%Zd.d'd(ZZ S)/RowParallelLinearaLinear layer with row parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its first dimension and X along its second dimension as: - - | A_1 | | . | A = | . | X = [X_1, ..., X_p] | . | | A_p | - - Arguments: input_size: first dimension of matrix A. output_size: second dimension of matrix A. bias: If true, add bias. Note that bias is not parallelized. input_is_parallel: If true, we assume that the input is already split across the GPUs and we do not split again. skip_bias_add: This was added to enable performance optimization where bias can be fused with other element-wise operations. We skip adding bias but instead return it. params_dtype: Data type for the parameters. quant_config: Quantization configure. TFNrjrkrWrlrrninput_is_parallelrmrorpreduce_resultsrqrrrsrCrrrruse_dp_attention_reducec strdn|}t|||||| ||_||_| |_| dur!t} | dur(t} | | |_|_ t ||j |_ |j dus=J| |_ |j j||j |jg|j|j|j|j jjtvrY|jn|jd|rwttj|j|d|_t|jd|jddS|dddS)N)rrrrkrlrorrrrr)rrvrwr%r&r'r r rrr rr{rrrlrkrorrrrrr rxrrr!r)r}rkrlrr%rmror&rqrsrrrr'r~r=r>rwsF  zRowParallelLinear.__init__r:r rZrc CsRt|dd}t|dd}t|dd}t|dd}|r||_|r@t|tr@t|j}|r6|||j||<|jt ||j d|j }|dur|s|j s|j|} |j | } trjddlm} | ||d| || \}}n| | } | |j|kr}t||| | }n||| | }t|jdkr|d }|j|jksJd |jd |j||dS) N input_dimrFrrrrrrPrr )r9rrrVr rrHrrtuplerrGrrrrrrrrYrr) r}r:rZr(rrr weight_shaperr;rrrr=r=r>rWsV            zRowParallelLinear.weight_loaderrcCrr) rYrHrrrVrload_row_parallel_weightrrrrr=r=r>rs$     z"RowParallelLinear.weight_loader_v2c Cs|jr|}nt||jd}||j}|jdusJ|jdks#|jr%dn|j}tt t d|jj |||d}Wdn1sEwY|j rc|jdkrc|sc|j r^t|}nt|}n|}|jrk|jnd}||fS)N)num_partitionsr)disabled)rrP)r%rrr contiguousr{rmrrrrrr&r'r all_reducer) r}rskip_all_reduceinput_parallelsplitted_inputbias_rrrr=r=r>rs(  zRowParallelLinear.forwardrDcCr)Nzinput_features=rrrz, reduce_results=)rrlrrr&rr=r=r>rrzRowParallelLinear.extra_repr) TTFNTNrjNNFF)rkrWrlrWrrnr%rnrmrnrorpr&rnrqrrrsrCrrrrrrnr'rnrr)Frrr=r=r~r>r$s$ = ? r$cs@eZdZdZ    dd fdd Zd!ddZd"ddZZS)#"MergedColumnParallelRepeatedLinearaMerged column parallel linear and repeated linear layer. TODO: quantization is not supported yet. Args: input_size: input dimension of the linear layer. column_output_sizes: output dimension of the column linear layers. repeated_output_sizes: output dimension of the repeated linear layers. skip_bias_add: If true, skip adding bias but instead return it. params_dtype: Data type for the parameters. quant_config: Quantization configure. FNrjrkrWcolumn_output_sizesrrepeated_output_sizesrmrnrorprqrrrsrCc st|t|}tj||||||dt|_t_t_fdd|D|_ j j j j j j jdjd|_dS)N)rkrlrmrorqrscsg|]}t|jqSr=)r r)rrr}r=r>rs z?MergedColumnParallelRepeatedLinear.__init__..Tr)rrvrwrYnum_column_parallelr rr rrr{rrkrlrorrs) r}rkr5r6rmrorqrsrlr~r7r>rws8    z+MergedColumnParallelRepeatedLinear.__init__rrrDcCs|j||Sr)r{r)r}rr=r=r>rsz*MergedColumnParallelRepeatedLinear.forwardr:r rZrBc Csb|j}t|jd|}|j|}|j|||}||jkr*|j|}||||}||dSr)rrrrGrr8rr) r}r:rZrBrr<r;rrr=r=r>rs   z0MergedColumnParallelRepeatedLinear.weight_loaderr)rkrWr5rr6rrmrnrorprqrrrsrC)rrrDrr:r rZrrBrWrDrrrrrrwrrrr=r=r~r>r4s 'r4cs6eZdZdZdfdd Zdd dZdddZZS)ColumnParallelBatchedLineara7Column parallel batched linear layer. TODO: quantization is not supported yet. Args: batch: batch dimension of the linear layer. input_size: input dimension of the linear layer. output_size: output dimension of the linear layer. dtype: Data type for the parameters. batchrWrkrlr torch.dtypecsRtt|_t|_tjtj |||j||ddd|_ t |j d|j dS)NrF) requires_gradr) rvrwr rr rrr rxrweightsetattrr)r}r<rkrlrr~r=r>rw-s z$ColumnParallelBatchedLinear.__init__inputrrDcCst||jddS)Nr^r_)rxbmmr? transpose)r}rAr=r=r>r9sz#ColumnParallelBatchedLinear.forwardr:r rZrBcCs8|jjd}|j|}|d||}|j||dS)Nr_r)r?rHrrrGr)r}r:rZrBr;rr=r=r>r<s  z)ColumnParallelBatchedLinear.weight_loader)r<rWrkrWrlrWrr=)rArrDrr9r:r=r=r~r>r;"s  r;)r:r r@rArBrCrDrE)Er __future__rrloggingtypingrrrrrrxrtorch.nn.parameterr r sglang.srt.distributedr r r rrrrsV  $ $   *_AoA]F