Directory listing for /maya3_transcribe/venv/lib/python3.10/site-packages/fairseq/examples/m2m_100/tokenizers/

• README.md
• seg_ja.sh
• seg_ko.sh
• thirdparty/
• tokenize_indic.py
• tokenize_thai.py
• tokenize_zh.py
• tokenizer_ar.sh