o پi@spddlmZddlZddlZddlmZmZmZmZddl Z er&ddl m Z GdddZ Gdddej ZdS) ) annotationsN) TYPE_CHECKINGOptionalSetType) ScheduleBatchc@seZdZd(ddZed)d d Zejd*dd ZddZd+ddZd,ddZ d-ddZ d.ddZ d/d d!Z d.d"d#Z d0d%d&Zd'S)1BatchedPenalizerOrchestrator vocab_sizeintbatchr penalizersSet[Type['_BatchedPenalizer']]cs\|_t|_|j_fdd|D_d}jD] }|}||O}q|_dS)Ncsi|]}||qSr).0 Penalizerselfr_/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/sampling/penaltylib/orchestrator.py sz9BatchedPenalizerOrchestrator.__init__..F) r weakrefref _batch_refdevicer valuesprepare_if_required is_required)rr r r r penalizerpen_is_requiredrrr__init__s   z%BatchedPenalizerOrchestrator.__init__returnScheduleBatch | NonecC|SN)rrrrrr sz"BatchedPenalizerOrchestrator.batchvalueOptional[ScheduleBatch]cCs&|dur dd|_dSt||_dS)NcSsdSr"rrrrr&sz4BatchedPenalizerOrchestrator.batch..)rrr)rr#rrrr #scCs|jjSr")r reqsrrrrr&*z!BatchedPenalizerOrchestrator.reqs output_ids torch.TensorcCs |jD]}|j|dqdS)z Feed the output tokens to the penalizers. Args: output_ids (torch.Tensor): The output tokens. r(N)r rcumulate_output_tokens)rr(rrrrr+-sz3BatchedPenalizerOrchestrator.cumulate_output_tokenslogitscCs|jD]}||qdS)a Apply the penalizers to the logits. Note that it may apply the penalizers in-place. Args: logits (torch.Tensor): The logits to apply the penalizers to. Returns: torch.Tensor: The logits after applying the penalizers. N)r rapply)rr,rrrrr-7s z"BatchedPenalizerOrchestrator.apply keep_indicescCsj|jsdSt|dkr|dSd}|jD]}|}||O}|r+|j|dq|q||_dS)z Filter the penalizers based on the indices to keep in the batch. Args: keep_indices (torch.Tensor): Tensor of indices to keep in the batch. NrFr.)rlenreleaser rfilterteardown)rr.rrtmp_is_requiredrrrr2Es   z#BatchedPenalizerOrchestrator.filterNonecCs2|jD]}|q|jd|_d|_dS)zGRelease all penalizers and break references so GC can reclaim promptly.NF)r rr3clearrr)rrrrrr1_s    z$BatchedPenalizerOrchestrator.release'BatchedPenalizerOrchestrator'cCs|Sr"rrrrr __enter__isz&BatchedPenalizerOrchestrator.__enter__cCs |dSr")r1)rexc_typeexctbrrr__exit__ls z%BatchedPenalizerOrchestrator.__exit__theircCs>|js|jsdSd|_|jD] \}}|j||qdS)a Merge the penalizers of another orchestrator into this one. Note that this function **must** be called _before_ self.batch.reqs is updated (filtered). Each unprepared penalizers would have to be prepared (creating tensors, etc.) first before merging. This step requires the original batch.reqs, before it gets merged with other batch.reqs. Args: their (BatchedPenalizerOrchestrator): The orchestrator to merge into this one. NT)rr itemsmerge)rr=rtheir_penalizerrrrr?os z"BatchedPenalizerOrchestrator.mergeN)r r r rr r )rr )r#r$r(r)r,r)rr)r.r))rr5)rr7)r=r7)__name__ __module__ __qualname__rpropertyr setterr&r+r-r2r1r8r<r?rrrrr s       rc@seZdZdZd1ddZed2ddZd3d d Zd3d d ZddZ ddZ ddZ d4ddZ d5ddZ d6ddZd7d d!Zejd3d"d#Zejd$d%Zejd4d&d'Zejd5d(d)Zejd6d*d+Zejd7d,d-Zejd.d/Zd0S)8_BatchedPenalizerz4 An abstract class for a batched penalizer. orchestratorrcCst||_d|_dSNF)rr_orchestrator_ref _is_prepared)rrJrrrrs z_BatchedPenalizer.__init__rcCs|}|dur td|S)Nz7BatchedPenalizerOrchestrator has been garbage-collected)rL RuntimeError)rorchrrrrJs z_BatchedPenalizer.orchestratorboolcCs|jSr")rMrrrr is_preparedsz_BatchedPenalizer.is_preparedcCr!r") _is_requiredrrrrrr'z_BatchedPenalizer.is_requiredcCs|js |d|_dSdS)NT)rM_preparerrrrprepares z_BatchedPenalizer.preparecCs|r |dSdS)NTF)rRrTrrrrrsz%_BatchedPenalizer.prepare_if_requiredcCs|d|_dSrK) _teardownrMrrrrr3s z_BatchedPenalizer.teardownr(r)cC|jsdS|j|ddS)Nr*)rM_cumulate_output_tokensrr(rrrr+z(_BatchedPenalizer.cumulate_output_tokensr,cCrV)N)r,)rM_applyrr,rrrr-rYz_BatchedPenalizer.applyr.cCrV)Nr/)rM_filterrr.rrrr2rYz_BatchedPenalizer.filterr='_BatchedPenalizer'cCs.|js|jsdS||||dSr")rMrT_mergerr=rrrr?s z_BatchedPenalizer.mergecCdS)zD Check if the penalizer is required to be prepared. NrrrrrrRz_BatchedPenalizer._is_requiredcCra)zn Prepare the penalizer. Usually, this is where the penalizer initializes its tensors. NrrrrrrSz_BatchedPenalizer._preparecCra)z Cumulate the output tokens. Orchestrator will call this function to feed the output tokens to the penalizer. NrrXrrrrWrcz)_BatchedPenalizer._cumulate_output_tokenscCra)zq Apply the penalizer to the logits. Penalizers can modify the logits in-place if needed. Nrr[rrrrZrcz_BatchedPenalizer._applycCra)zn Filter the penalizer (tensors or underlying data) based on the indices to keep in the batch. Nrr]rrrr\rbz_BatchedPenalizer._filtercCra)z= Merge the penalizer with another penalizer. Nrr`rrrr_rbz_BatchedPenalizer._mergecCra)z) Teardown the penalizer. NrrrrrrUrbz_BatchedPenalizer._teardownN)rJr)rr)rrPrArBrC)r=r^)rDrErF__doc__rrGrJrQrrTrr3r+r-r2r?abcabstractmethodrRrSrWrZr\r_rUrrrrrIs8             rI) __future__rrertypingrrrrtorch"sglang.srt.managers.schedule_batchrrABCrIrrrrs  u