o 3wit@s^ddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZddlmZddl mZddlZddlZddlmZdd lmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAd d ZBeB\ZCZDZEd eFfd dZGeGeDddZHdddZIeIdddZJddZKddZLddZMddZNddZOd d!ZPd"d#ZQd$d%ZRd&d'ZSd(d)ZTd*d+ZUd,d-ZVd.d/ZWd0d1ZXd2d3ZYd4d5ZZd6d7Z[d8d9Z\d:d;Z]dd?Z_d@dAZ`dBdCZadDdEZbdFdGZcdHdIZddJdKZedLdMZfdNdOZgdPdQZhdRdSZidTdUZjdVdWZkdXdYZldZd[Zmd\d]Zndd^d_Zod`daZpdbdcZqdddeZrdfdgZsdhdiZtdjdkZudldmZvdndoZwdpdqZxdrdsZydtduZzdvdwZ{dxdyZ|dzd{Z}d|d}Z~ee?e4e;e3goe Zd~dZddZddZGddde jZGddde jZGddde jZddZGdddZddZdd efddZddeFd efddZddZddZGdddeZddeFefddZded efddZe ddeded efddZddZdS)N)contextmanager)partial)Path)Union)mock)AcceleratorState)*check_cuda_fp8_capabilitycompare_versionsgatheris_aim_availableis_bnb_availableis_clearml_availableis_comet_ml_availableis_cuda_availableis_datasets_availableis_deepspeed_availableis_dvclive_availableis_fp8_availableis_fp16_availableis_habana_gaudi1is_hpu_availableis_import_timer_availableis_matplotlib_availableis_mlflow_availableis_mlu_availableis_mps_availableis_musa_availableis_npu_availableis_pandas_availableis_pippy_availableis_pytest_availableis_schedulefree_availableis_sdaa_availableis_swanlab_availableis_tensorboard_availableis_timm_availableis_torch_versionis_torch_xla_availableis_torchao_available*is_torchdata_stateful_dataloader_availableis_torchvision_availableis_trackio_availableis_transformer_engine_availableis_transformers_availableis_triton_availableis_wandb_availableis_xpu_available str_to_boolcCs tr dtjtjjfStrdtjtjjfStddr&ddtjjfStr0ddddfSt r=d tj tj jfSt rJd tj tj jfSt rWd tjtjjfStrdd tjtjjfStrqd tjtjjfStr~dtjtjjfSddddfS)Nxlacudaz2.0) min_versionmpscSdSNrr:r:r:Z/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/accelerate/test_utils/testing.pyZzget_backend..mlusdaamusanpuxpuhpucpucSr8r9r:r:r:r:r;r<hr=)r(torchr4 device_countmemory_allocatedrrr6current_allocated_memoryrr>r#r?rr@rrAr1rBrrCr:r:r:r; get_backendRs* rIreturncKs\ddg}|D]#\}}t|tr|r|d|q|dur+|d|d|q|S)z Wraps around `kwargs` to help simplify launching from `subprocess`. Example: ```python # returns ['accelerate', 'launch', '--num_processes=2', '--device_count=2'] get_launch_command(num_processes=2, device_count=2) ``` acceleratelaunchz--N=)items isinstanceboolappend)kwargscommandkvr:r:r;get_launch_commandns rVg?) num_processesmonitor_intervalFcCsVztj|}Wn ty|}Y|Swzt|}W|Sty*td|dw)NzIf set, z must be yes or no.)osenvironKeyErrorr2 ValueError)keydefaultvalue_valuer:r:r;parse_flag_from_envs   raRUN_SLOW)r^cCstd|S)z+Decorator that skips a test unconditionallyzTest was skipped)unittestskip test_caser:r:r;rdsrdcCttd|S)z Decorator marking a test as slow. Slow tests are skipped by default. Set the RUN_SLOW environment variable to a truthy value to run them. z test is slow)rc skipUnless_run_slow_testsrer:r:r;slowsrjcCsttdkd|S)zu Decorator marking a test that must be only ran on the CPU. These tests are skipped when a GPU is available. rDztest requires only a CPUrcrh torch_devicerer:r:r; require_cpurmcCttdkd|S)z Decorator marking a test that requires a hardware accelerator backend. These tests are skipped when there are no hardware accelerator available. rDtest requires a GPUrkrer:r:r;require_non_cpurqcCttot d|S)z Decorator marking a test that requires CUDA. These tests are skipped when there are no GPU available or when TorchXLA is available. rp)rcrhrr(rer:r:r; require_cudartcCs ttr t p td|S)z Decorator marking a test that requires CUDA or HPU. These tests are skipped when there are no GPU available or when TorchXLA is available. ztest requires a GPU or HPU)rcrhrr(rrer:r:r;require_cuda_or_hpu rvcCttd|S)zn Decorator marking a test that requires XPU. These tests are skipped when there are no XPU available. test requires a XPU)rcrhr1rer:r:r; require_xpurzcCs(tot }t}t|p|d|S)z Decorator marking a test that requires CUDA or XPU. These tests are skipped when there are no GPU available or when TorchXLA is available. ztest requires a CUDA GPU or XPU)rr(r1rcrh)rfcuda_condition xpu_conditionr:r:r;require_cuda_or_xpusr~cCro)zB Decorator marking a test that should be skipped for XPU. rBztest requires a non-XPUrkrer:r:r;require_non_xpurnrcCro)zB Decorator marking a test that should be skipped for HPU. rCztest requires a non-HPUrkrer:r:r;require_non_hpurnrcCrx)zj Decorator marking a test that requires FP16. These tests are skipped when FP16 is not supported. ztest requires FP16 support)rcrhrrer:r:r; require_fp16rcCs:t}tjr ts d}trtrd}t|d|S)zh Decorator marking a test that requires FP8. These tests are skipped when FP8 is not supported. Fztest requires FP8 support) rrEr4 is_availabler rrrcrh)rffp8_is_availabler:r:r; require_fp8s  rcCsttddd|S)N>=z2.5.0z$test requires FSDP2 (torch >= 2.5.0))rcrhr'rer:r:r; require_fsdp2srcCrx)zn Decorator marking a test that requires MLU. These tests are skipped when there are no MLU available. ztest require a MLU)rcrhrrer:r:r; require_mlur{rcCrx)zp Decorator marking a test that requires SDAA. These tests are skipped when there are no SDAA available. ztest require a SDAA)rcrhr#rer:r:r; require_sdaa r{rcCrx)zp Decorator marking a test that requires MUSA. These tests are skipped when there are no MUSA available. ztest require a MUSA)rcrhrrer:r:r; require_musar{rcCrx)zn Decorator marking a test that requires NPU. These tests are skipped when there are no NPU available. ztest require a NPU)rcrhrrer:r:r; require_npur{rcCrx)z Decorator marking a test that requires MPS backend. These tests are skipped when torch doesn't support `mps` backend. z0test requires a `mps` backend support in `torch`)rcrhrrer:r:r; require_mpsrrcCsttotd|S)zv Decorator marking a test that requires transformers and datasets. These tests are skipped when they are not. z$test requires the Hugging Face suite)rcrhr.rrer:r:r;require_huggingface_suite&s  rcCrx)zi Decorator marking a test that requires transformers. These tests are skipped when they are not. z&test requires the transformers library)rcrhr.rer:r:r;require_transformers0r{rcCrx)za Decorator marking a test that requires timm. These tests are skipped when they are not. ztest requires the timm library)rcrhr&rer:r:r; require_timm7r{rcCrx)zh Decorator marking a test that requires torchvision. These tests are skipped when they are not. z%test requires the torchvision library)rcrhr+rer:r:r;require_torchvision>r{rcCrx)zc Decorator marking a test that requires triton. These tests are skipped when they are not. z test requires the triton library)rcrhr/rer:r:r;require_tritonEr{rcCrx)zi Decorator marking a test that requires schedulefree. These tests are skipped when they are not. z&test requires the schedulefree library)rcrhr"rer:r:r;require_schedulefreeLr{rcCrx)zi Decorator marking a test that requires bitsandbytes. These tests are skipped when they are not. z&test requires the bitsandbytes library)rcrhr rer:r:r; require_bnbSr{rcCsttddd|S)zp Decorator marking a test that requires TPUs. These tests are skipped when there are no TPUs available. T) check_is_tpuztest requires TPUrcrhr(rer:r:r; require_tpuZsrcCstt d|S)z Decorator marking a test as requiring an environment without TorchXLA. These tests are skipped when TorchXLA is available. z%test requires an env without TorchXLArrer:r:r;require_non_torch_xlaarrrcCsttdko tdkd|S)z Decorator marking a test that requires a single device. These tests are skipped when there is no hardware accelerator available or number of devices is more than one. rDr7z)test requires a single device accelerator)rcrhrlrFrer:r:r;require_single_deviceis rcCttjdkd|S)z Decorator marking a test that requires CUDA on a single GPU. These tests are skipped when there are no GPU available or number of GPUs is more than one. r7rprcrhrEr4rFrer:r:r;require_single_gpusrurcCr)z Decorator marking a test that requires CUDA on a single XPU. These tests are skipped when there are no XPU available or number of xPUs is more than one. r7ryrcrhrErBrFrer:r:r;require_single_xpu{rurcCsttdkd|S)z Decorator marking a test that requires a multi-device setup. These tests are skipped on a machine without multiple devices. r7z,test requires multiple hardware accelerators)rcrhrFrer:r:r;require_multi_devicerrrcCttjdkd|S)z Decorator marking a test that requires a multi-GPU setup. These tests are skipped on a machine without multiple GPUs. r7ztest requires multiple GPUsrrer:r:r;require_multi_gpururcCr)z Decorator marking a test that requires a multi-XPU setup. These tests are skipped on a machine without multiple XPUs. r7ztest requires multiple XPUsrrer:r:r;require_multi_xpururcCs ttsto tdkd|S)z Decorator marking a test that requires a multi-GPU setup. These tests are skipped on a machine without multiple GPUs or XPUs. r7z#test requires multiple GPUs or XPUs)rcrhrr1rFrer:r:r;require_multi_gpu_or_xpurwrcCrx)z| Decorator marking a test that requires DeepSpeed installed. These tests are skipped when DeepSpeed isn't installed ztest requires DeepSpeed)rcrhrrer:r:r;require_deepspeedr{rcCs"ttddo tdddd|S)zn Decorator marking a test that requires TP installed. These tests are skipped when TP isn't installed rz2.3.0 transformersz4.52.0zGtest requires torch version >= 2.3.0 and transformers version >= 4.52.0)rcrhr'r rer:r:r; require_tps rcCs0|dur tt|dSttd|d||S)z Decorator marking that a test requires a particular torch version to be tested. These tests are skipped when an installed torch version is less than the required one. N)versionrztest requires torch version >= )rrequire_torch_min_versionrcrhr')rfrr:r:r;rs rcCrx)z Decorator marking a test that requires tensorboard installed. These tests are skipped when tensorboard isn't installed ztest requires Tensorboard)rcrhr%rer:r:r;require_tensorboardrrcCrx)zt Decorator marking a test that requires wandb installed. These tests are skipped when wandb isn't installed ztest requires wandb)rcrhr0rer:r:r; require_wandbr{rcCrx)zx Decorator marking a test that requires trackio installed. These tests are skipped when trackio isn't installed ztest requires trackio)rcrhr,rer:r:r;require_trackior{rcCrx)zz Decorator marking a test that requires comet_ml installed. These tests are skipped when comet_ml isn't installed ztest requires comet_ml)rcrhrrer:r:r;require_comet_mlr{rcCrx)zp Decorator marking a test that requires aim installed. These tests are skipped when aim isn't installed ztest requires aim)rcrhr rer:r:r; require_aimr{rcCrx)zx Decorator marking a test that requires clearml installed. These tests are skipped when clearml isn't installed ztest requires clearml)rcrhrrer:r:r;require_clearmlr{rcCrx)zx Decorator marking a test that requires dvclive installed. These tests are skipped when dvclive isn't installed ztest requires dvclive)rcrhrrer:r:r;require_dvcliver{rcCrx)zx Decorator marking a test that requires swanlab installed. These tests are skipped when swanlab isn't installed ztest requires swanlab)rcrhr$rer:r:r;require_swanlabr{rcCrx)zv Decorator marking a test that requires pandas installed. These tests are skipped when pandas isn't installed ztest requires pandas)rcrhrrer:r:r;require_pandasr{rcCrx)zv Decorator marking a test that requires mlflow installed. These tests are skipped when mlflow isn't installed ztest requires mlflow)rcrhrrer:r:r;require_mlflowr{rcCrs)z Decorator marking a test that requires pippy installed. These tests are skipped when pippy isn't installed It is also checked if the test is running on a Gaudi1 device which doesn't support pippy. ztest requires pippy)rcrhr rrer:r:r; require_pippyrurcCrx)z Decorator marking a test that requires tuna interpreter installed. These tests are skipped when tuna isn't installed ztest requires tuna interpreter)rcrhrrer:r:r;require_import_timerrrcCrx)z Decorator marking a test that requires transformers engine installed. These tests are skipped when transformers engine isn't installed z!test requires transformers engine)rcrhr-rer:r:r;require_transformer_enginerrcCrx)zx Decorator marking a test that requires torchao installed. These tests are skipped when torchao isn't installed ztest requires torchao)rcrhr)rer:r:r;require_torchaor{rcCrx)z Decorator marking a test that requires matplotlib installed. These tests are skipped when matplotlib isn't installed ztest requires matplotlib)rcrhrrer:r:r;require_matplotlib&rrcCrg)z Decorator marking that a test requires at least one tracking library installed. These tests are skipped when none are installed zYtest requires at least one tracker to be available and for `comet_ml` to not be installed)rcrh_atleast_one_tracker_availablerer:r:r;require_trackers4s rcCrx)z Decorator marking a test that requires torchdata.stateful_dataloader. These tests are skipped when torchdata with stateful_dataloader module isn't installed. z+test requires torchdata.stateful_dataloader)rcrhr*rer:r:r;%require_torchdata_stateful_dataloader?s rcCs"trddl}|jd|S|S)a Decorator marking a test with order(1). When pytest-order plugin is installed, tests marked with this decorator are garanteed to run first. This is especially useful in some test settings like on a Gaudi instance where a Gaudi device can only be used by a single process at a time. So we make sure all tests that run in a subprocess are launched first, to avoid device allocation conflicts. If pytest is not installed, test will be returned as is. rNr7)r!pytestmarkorder)rfrr:r:r; run_firstKs rc@s4eZdZdZdZeddZeddZddZd S) TempDirTestCasea A TestCase class that keeps a single `tempfile.TemporaryDirectory` open for the duration of the class, wipes its data at the start of a test, and then destroyes it at the end of the TestCase. Useful for when a class or API requires a single constant folder throughout it's use, such as Weights and Biases The temporary directory location will be stored in `self.tmpdir` TcCstt|_dS)zECreates a `tempfile.TemporaryDirectory` and stores it in `cls.tmpdir`N)rtempfilemkdtemptmpdirclsr:r:r; setUpClassjszTempDirTestCase.setUpClasscCs"tj|jrt|jdSdS)z1Remove `cls.tmpdir` after test suite has finishedN)rYpathexistsrshutilrmtreerr:r:r; tearDownClassoszTempDirTestCase.tearDownClasscCsD|jr|jdD]}|r|q |rt|q dSdS)zt|ttfr |n|g|_|jD] }|||jqdS)aQ Add custom mocks for tests that should be repeated on each test. Should be called during `MockingTestCase.setUp`, after `super().setUp()`. Args: mocks (`mock.Mock` or list of `mock.Mock`): Mocks that should be added to the `TestCase` after `TestCase.setUpClass` has been run N)rOtuplelistrstart addCleanupstop)rrmr:r:r; add_mockss  zMockingTestCase.add_mocksN) rrrrrrMockrrr:r:r:r;rs"rcCs`t}|d|j}t|}|d}t|jdD] }t |||s-dSq dS)NrFT) rclonetodevicer rDrangeshaperEequal)tensorstatetensorsir:r:r;are_the_same_tensorss  rc@seZdZddZdS) _RunOutputcCs||_||_||_dSN) returncodestdoutstderr)rrrrr:r:r;__init__s z_RunOutput.__init__N)rrrrr:r:r:r;rs rcs& |IdH}|r||ndSqr)readline)streamcallbackliner:r:r; _read_streams rc s|r tdd|tj|dg|ddR|tjjtjj|dIdH}ggd fdd tjtt|j fd d tt|j fd d g|d IdHt |IdHS)Nz Running:  rr7)stdinrrenvcs2|d}||st|||ddSdS)Nutf-8)file)decoderstriprQprint)rsinkpipelabel)quietr:r;tees  z_stream_subprocess..teec|tjddS)Nzstdout:r)sysrl)outr r:r;r<z$_stream_subprocess..cr )Nzstderr:r )r rr)errr r:r;r<r)timeout)r) rjoinasynciocreate_subprocess_exec subprocessPIPEwait create_taskrrrr)cmdrrrr echopr:)rrr r r;_stream_subprocesss.  rTrc Cst|D]\}}t|trt|||<qt}|t||||||d} d|} | j dkrCd| j } t d| d| j d| | S)N)rrrr rrr 'z' failed with returncode z, The combined stderr from workers follows: ) enumeraterOrstrrget_event_looprun_until_completerrrr RuntimeError) rrrrr rrcloopresultcmd_strrr:r:r;execute_subprocess_asyncs"     r+cCs*tjdd}tdd|dtj}t|S)z Returns an int value of worker's numerical id under `pytest-xdist`'s concurrent workers `pytest -n N` regime, or 0 if `-n 1` or `pytest-xdist` isn't being used. PYTEST_XDIST_WORKERgw0z^gwrr)rYrZgetresubMint)workerr:r:r;pytest_xdist_worker_idsr4cCsd}t}||S)z Returns a port number that can be fed to `torch.distributed.launch`'s `--master_port` argument. Under `pytest-xdist` it adds a delta number based on a worker id so that concurrent tests don't try to use the same port at once. ier:r:r; run_command s*     rA componentscGstttj}|j|S)z Get a path within the `accelerate` package's directory. Args: *components: Components of the path to join after the package directory. Returns: `Path`: The path to the requested file or directory. )rinspectgetfilerKparentjoinpath)rBaccelerate_package_dirr:r:r;path_in_accelerate_package7s rHexception_classmsgc csd}zdVd}Wn:tyD}z.t||s#Jd|dt||dur:|t|vs:Jd|dt|dWYd}~nd}~ww|rOtd|d dS) z Context manager to assert that the right `Exception` class was raised. If `msg` is provided, will check that the message is contained in the raised exception. FNTzExpected exception of type z but got zExpected message 'z' to be in exception but got 'r!z but ran without issue.) ExceptionrOtyper#AssertionError)rIrJwas_ranr@r:r:r;assert_exceptionFs"&rOc Os^t}tj}z#z |t_||i|Wnty"}z|d}~wwW|t_|S|t_w)z` Takes in a `func` with `args` and `kwargs` and returns the captured stdout as a string N)ioStringIOr rrKgetvalue)funcargsrRcaptured_outputoriginal_stdoutr@r:r:r;capture_call_outputYsrW)F)NN)NNNFF)NNrFT)FNr)rrCrPrYr/rrr rrc contextlibr functoolsrpathlibrtypingrrrErKrrutilsr r r r r rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2rIrlrFmemory_allocated_funcrrVDEFAULT_LAUNCH_COMMANDrarirdrjrmrqrtrvrzr~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrranyrrrrTestCaserrrrrrrr+r4r7rKr8r#rArHrPrOrWr:r:r:r;s      .              ! !  )