o پi@sddlmZddlmZmZmZmZmZddlZ ddl Z ddl m Z er*ddl mZee e jedfZddZdd ZGd d d eZdS) )abstractmethod) TYPE_CHECKINGListProtocolTypeUnionN)NDArrayzmx.arraycCs2zddlm}Wn tyYdSwt||jSNrF)mlx.corecore ImportError issubclassarray) array_typemxr]/home/ubuntu/.local/lib/python3.10/site-packages/outlines/processors/base_logits_processor.pyis_mlx_array_types   rcCs>zddl}Wn tyYdSwt||jjpt||jjSr )jaxlibr r xla_extension ArrayImpl isinstance)rrrrris_jax_array_types  rc@seZdZdZedeeedejdejfddZ e de de de fddZ e d e dejfd d Ze d ejd ede fddZdS)OutlinesLogitsProcessora  Base class for logits processors which normalizes types of logits: - ndarray (used by llama-cpp-python), converted to torch.Tensor - mlx.core.array (used by mlx-lm), converted to torch.Tensor - torch.Tensor (used by everything else) Normalization of types and conversion to torch.Tensor doesn't move memory, it just casts the type. Normalizing the types allows all logits processors inheriting from this class to implement a single method for all the business logit: `process_logits()` input_idslogitsreturncCsdS)aD input_ids and logits are always 2D tensors for handling a batch of sequences. - input_ids -> List[List[tokens]] - logits -> 2D_Tensor[logit floats] Important to keep in mind when designing universal logits processors - logits processors are only used once and never re-applied for a new sequence generator - Some models only pass output_ids, some models such as llamacpp and transformers prefix with input_ids - Some sampling methods, such as beam search, result in unstable sequence ordering in models like vLLM Nr)selfrrrrrprocess_logits/sz&OutlinesLogitsProcessor.process_logitscCs||}||}|jdd|jddksJt|jdkr(|||}nt|jdkr>||d|dd}||t|S)a Apply logits processor 1) Unify type - convert input_ids: either ndarray, mlx array, List[int], or Tensor -> List[List[int]] - convert logits: either ndarray, mlx array, or Tensor -> 2D float Tensor 2) Unify shape, ensure logits and input_ids are 2D 3) Call self.process_logits() to perform business logic 4) Cast logits back to original array library type Nr) _to_torchshapelenr unsqueezesqueeze _from_torchtype)rrr torch_logitsprocessed_logitsrrr__call__@s   z OutlinesLogitsProcessor.__call__ tensor_likecCst|tjr|St|tjrt|St|ttfrt|St t |r9ddl m }t tj||jddStt |rNddl}t |j|}|Stdt |d)z&Convert various types to torch.Tensor.rNF)copyzyLogitsProcessor must be called with either np.NDArray, torch.Tensor, list, or mlx.core.array typed logits. Logits type: ``)rtorchTensornpndarray from_numpylisttupletensorrr(r r from_dlpackrastypefloat32rjaxdlpack to_dlpack TypeError)r,rr: torch_tensorrrrr"as(       z!OutlinesLogitsProcessor._to_torchr6 target_typecCs|tjkr|S|tjkr|S|tkr|S|tkr(t|St |r;ddl m }| | St|rIddl}|j|Std|d)z2Convert torch.Tensor to the specified target type.rNz0Failed to convert torch tensors to target_type `r.)r/r0r1r2detachnumpyr4tolistr5rr r rfloatrr:r;r7r=)r6r?rr:rrrr's"       z#OutlinesLogitsProcessor._from_torchN)__name__ __module__ __qualname____doc__rrintr/r0rno_gradArrayr+ staticmethodr"rr'rrrrr!s,      r)abcrtypingrrrrrrAr1r/ numpy.typingrr r rr0rJrrrrrrrs