o پi% @sdZddlZddlmZmZmZmZddlZddlm Z ddl m Z de fddZ de fd d Zd ejd e d e deejejffddZe ddZ  d!dededeedeedee f ddZGdddZ d"dddddeedeedeeededef dd ZdS)#z!Utility methods for model layers.N)AnyCallableListOptional)Library)current_platformreturncCDt|dr|jSt|drtt|ddr|Stdt|)N world_sizesizeUnsupported group type: )hasattrr callablegetattrr ValueErrortypegroupr^/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/runtime/layers/utils.pyget_group_size rcCr )N rank_in_grouprankr )r rrrrrrrrrrget_group_rankrrtokens vocab_sizenum_seqscCsTtj||dftj|jd}|d|t||ddd|f}|dk}||fS)N)dtypedevicer)torchzeroslongr scatter_add_ ones_like)rrr bin_countsmaskrrrget_token_bin_counts_and_mask"sr(sglangFRAGMENTop_nameop_func mutates_args fake_impl target_libc CsFddl}|pt}z t|jdr|jjnd}t|j|r'tt|j||r'WdSWn ttfy3Ynwt|j drC|j j ||d}n ddl }|j j ||}z$|||| ||tsadnd|durq|||WdSWdSty} zd t| vrd t| vrn| WYd} ~ dSd} ~ wty} z| d} ~ ww) a `torch.library.custom_op` can have significant overhead because it needs to consider complicated dispatching logic. This function directly registers a custom op and dispatches it to the CUDA backend. See https://gist.github.com/youkaichao/ecbea9ec9fc79a45d2adce1784d7a9a5 for more details. By default, the custom op is registered to the vLLM library. If you want to register it to a different library, you can pass the library object to the `target_lib` argument. IMPORTANT: the lifetime of the operator is tied to the lifetime of the library object. If you want to bind the operator to a different library, make sure the library object is alive when the operator is used. Note: This function will silently skip registration if the operator with the same name is already registered to avoid RuntimeError in multi-engine scenarios (e.g., VERL framework). rNnamer) infer_schema)r-CUDA PrivateUse1zTried to register an operatorzmultiple times) torch.library sglang_libr mr0opsrAttributeError RuntimeErrorlibraryr1torch._custom_op.impl _custom_opimpldefineris_npu_register_fakestr) r+r,r-r.r/r!my_liblib_name schema_strerrorrrrdirect_register_custom_op6sH  rFc@sReZdZdededeefddZddZedefd d Z edefd d Z d S)CustomOpWrapperr+r,r-cKs"||_||_||_||_d|_dSN)r+r,r- extra_kwargs_impl)selfr+r,r-rIrrr__init__s  zCustomOpWrapper.__init__cOs|j|i|SrH) real_impl)rKargskwargsrrr__call__szCustomOpWrapper.__call__rcCsX|jdur)ttjj|jst|j|j|j|j dt tjj|j|_|jdus)J|jS)N)r+r,r-r.) rJr r!r7r)r+rFr,r-r.r)rKrrrrMs zCustomOpWrapper.real_implcsLdjvr jdSdjvsJtjjdfdd}|S)Nr. out_shapec sdurdSj|i|}|ztttr!|jWS|jWStt fy?t ddj ddw)Nz)Cannot find output argument at position `z` for custom operator `z` with signature `z`.) bindapply_defaultsr! empty_like isinstanceintrN arguments IndexErrorKeyErrorr9r+)rNrOboundrQrK signaturerrr.s( z,CustomOpWrapper.fake_impl..fake_impl)rIinspectr\r,)rKr.rr[rr.s    zCustomOpWrapper.fake_implN) __name__ __module__ __qualname__rArrrLrPpropertyrMr.rrrrrGs  rGT)r+r-eagerfnrbc  st}tddh}||ksJd||dv}dv}|r)|r)Jd|s1|s1dd<dtdtffdd } |durG| |S| S) a* A decorator to register a custom operator. Example usage: ```python # inplace operator, out_shape is None by default @register_custom_op(mutates_args=["x"]) def add_1_(x: torch.Tensor) -> None: x.add_(1) # operator with output, out_shape indicates the position of output @register_custom_op(mutates_args=["x"], out_shape=0) def add(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: return x.add_(y) ``` :param fn: The function to be registered as a custom operator. If None, return a decorator. :type fn: Callable :param op_name: The name of the operator. If None, use the function name :type op_name: Optional[str] :param mutates_args: A list of argument names that are mutated in-place. :type mutates_args: List[str] :param out_shape: The position (int for positional, str for keyword) of the output-shape tensor. It is used to generate a fake implementation for torch.compile compatibility. If the operator is inplace and has no output, set to None. :type out_shape: Optional[List[Union[int, str]]] :param fake_impl: A fake implementation for the operator. Only one of `out_shape` or `fake_impl` should be provided. :type fake_impl: Optional[Callable] :param eager: Whether to register the operator eagerly. If False, the registration will be deferred until the first call. If you met any issue with torch.compile, try to set eager=True. Currently, to avoid misuse, we set eager=True by default. :type eager: bool :return: The registered JIT custom operator, or a decorator. NOTE: the real register will occur at the first call of the function. :rtype: Callable rQr.zUnexpected extra kwargs: z:Only one of `out_shape` or `fake_impl` should be provided.Nr,rcs.tdp|j|p gd}r|jS|S)N)r+r,r-r)rGr^rM)r,wrapperrbrIr-r+rr decoratorsz%register_custom_op..decorator)setkeysr) rcr+r-rbrIextra_kwarg_keysexpected_kwarg_keys has_out_shape has_fake_implrfrrerregister_custom_ops& /    rm)NNrH)__doc__r]typingrrrrr!r4r'sglang.multimodal_gen.runtime.platformsrrVrrTensortupler(r5rArFrGboolrmrrrrs`      JB