o i @sdZddlZddlmZddlmZmZddlmZddl m Z ddl m Z m Z mZmZmZmZddlmZdd lmZz ddlZdd lmZWn eyhZzed eed ed edZ[wwddZeGdddeZGdddeZdS)zMoondream vision service implementation. This module provides integration with the Moondream vision-language model for image analysis and description generation. N) dataclass)AsyncGeneratorOptional)logger)Image) ErrorFrameFrameUserImageRawFrameVisionFullResponseEndFrameVisionFullResponseStartFrameVisionTextFrame)VisionSettings) VisionService)AutoModelForCausalLMz Exception: zKIn order to use Moondream, you need to `pip install pipecat-ai[moondream]`.zMissing module(s): cCszddl}tjrtdtjfWSWn tyYnwtjr+tdtjfStj j r9tdtjfStdtjfS)aMDetect the appropriate device to run on. Detects available hardware acceleration and selects the best device and data type for optimal performance. Returns: tuple: A tuple containing (device, dtype) where device is a torch.device and dtype is the recommended torch data type for that device. rNxpucudampscpu) intel_extension_for_pytorchtorchr is_availabledevicefloat32 ImportErrorrfloat16backendsr)rrU/home/ubuntu/.local/lib/python3.10/site-packages/pipecat/services/moondream/vision.py detect_device(s     rc@seZdZdZdS)MoondreamSettingszkSettings for the Moondream vision service. Parameters: model: Moondream model identifier. N)__name__ __module__ __qualname____doc__rrrrrAsrcsfeZdZUdZeZeed<ddddddeedeeffd d Z d e d e e dffd dZ ZS)MoondreamServicezMoondream vision-language model service. Provides image analysis and description generation using the Moondream vision-language model. Supports various hardware acceleration options including CUDA, MPS, and Intel XPU. _settingsNz 2025-01-09F)modelrevisionuse_cpusettingsr&r)c  s|jdd}|dur|dd||_|dur||tjd d|i||s/t\}}ntd}tj }t dt j |jjd|d |i|d |_t d dS) amInitialize the Moondream service. Args: model: Hugging Face model identifier for the Moondream model. .. deprecated:: 0.0.105 Use ``settings=MoondreamService.Settings(model=...)`` instead. revision: Specific model revision to use. use_cpu: Whether to force CPU usage instead of hardware acceleration. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. **kwargs: Additional arguments passed to the parent VisionService. zvikhyatk/moondream2)r&Nr&r)rzLoading Moondream model...T)trust_remote_coder' device_mapdtypezLoaded Moondream modelr)Settings"_warn_init_param_moved_to_settingsr& apply_updatesuper__init__rrrrrdebugrfrom_pretrainedr%eval_model) selfr&r'r(r)kwargsdefault_settingsrr- __class__rrr2Us,      zMoondreamService.__init__framereturncsjs tdVdStdtjddtdttdtffdd }t |jj IdH}t Vt |d VtVdS) zsAnalyze an image and generate a description. Args: frame: The image frame to process. zMoondream model not availableNzAnalyzing image (bytes length: ) image_bytestextr=cs4tjj|}j|}j||d}|S)Nanswer)r frombytesformatsizer6 encode_imagequery)r?r@image image_embeds descriptionr<r7rrget_image_descriptions z:MoondreamService.run_vision..get_image_description)r@)r6rrr3lenrGbytesrstrasyncio to_threadr@r r r )r7r<rKrIrrJr run_visions    zMoondreamService.run_vision)r r!r"r#rr.__annotations__rrNr2r rrrQ __classcell__rrr:rr$Js "7r$) r#rO dataclassesrtypingrrlogururPILrpipecat.frames.framesrrr r r r pipecat.services.settingsr pipecat.services.vision_servicerr transformersrModuleNotFoundErroreerror Exceptionrrr$rrrrs,