o ̳i( @sddlmZddlmZddlmZmZddlmZeedeedeeeeeeed Z dee effd d Z d S) )Dict) Replicate)ColwiseParallelRowwiseParallel) ParallelStyle) input_layouts)output_layouts) tok_embeddingsoutputzlayers.*.attn.q_projzlayers.*.attn.k_projzlayers.*.attn.v_projzlayers.*.attn.output_projzlayers.*.mlp.w1zlayers.*.mlp.w2zlayers.*.mlp.w3returncCstS)z Helper function to get the base tensor parallel plan for Llama3 model, which will also be shared with 3.1, 3.2, and 3.3 models Returns: Dict[str, Any]: The tensor parallel plan for Llama3 model. )BASE_LLAMA_TP_PLANr r X/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/models/llama3/_parallelism.pybase_llama_tp_plansrN) typingrtorch.distributed._tensorr!torch.distributed.tensor.parallelrr'torch.distributed.tensor.parallel.stylerr strrr r r rs