o ٷi@sddlZddlmZddlmZddlmZddlmZddl m Z ee Z dej jeBdee d ej jfd d Zdej jeBdee d ej jfd d Zdej jeBdee d ej jfddZdS)N)Optional) init_logger) ModelMixin)ParallelismBackend)ParallelismConfig transformerparallelism_configreturncCspt|tjjtfsJdt||dur|S|jtjkr$t ||dS|jtj kr0t ||dSt |jd)NJtransformer must be an instance of torch.nn.Module or ModelMixin, but got rrz backend is not supported yet) isinstancetorchnnModulertypebackendrNATIVE_DIFFUSER0maybe_enable_context_parallelism_for_transformerNATIVE_PYTORCH/maybe_enable_tensor_parallelism_for_transformer ValueErrorr r_/home/ubuntu/.local/lib/python3.10/site-packages/cache_dit/parallelism/transformers/dispatch.py(maybe_enable_parallelism_for_transformers$  rc Cst|tjjtfsJdt||dur|St|ts&Jdt||jtj ks8Jdtj d|j|j dusB|j durjddl m }|||}d|_||_td|jjd t|d |d|Std ) Nr Dparallelism_config must be an instance of ParallelismConfig but got zparallelism backend must be z , but got ) maybe_enable_context_parallelismTParallelize Transformer: , id:, zyNATIVE_DIFFUSER backend only support context parallelism now. Please set ulysses_size or ring_size in parallelism_config.)r r rrrrrrrr ulysses_size ring_sizecontext_parallelismr_is_parallelized_parallelism_configloggerinfo __class____name__idstrifyr)rrrrrrr+sH     rc Cst|tjjtfsJdt||dur|S|jtjks&Jd|jt|t s4Jdt||j dur>|j dusBJd|j duru|j dkruddl m}|||d}d|_||_td |jjd t|d |d|Std ) Nr zMparallelism_config.backend must be ParallelismBackend.NATIVE_PYTORCH but got rzqUlysses/Ring parallelism is not supported in Native_PyTorch backend. Please set it to None in parallelism_config.r)maybe_enable_tensor_parallelismr TrrrzdNATIVE_PYTORCH only supported tensor parallelism now. Please set tp_size > 1 for tensor parallelism.)r r rrrrrrrrr r!tp_sizetensor_parallelismr+r#r$r%r&r'r(r)r*r)rrr+rrrrXsN    r)r typingrcache_dit.loggerrdiffusers.models.modeling_utilsrcache_dit.parallelism.backendrcache_dit.parallelism.configrr(r%rrrrrrrrrs8         -