|
| 1 | +# *Metis*: A Foundation Speech Generation Model with Masked Generative Pre-training |
| 2 | + |
| 3 | +[](https://arxiv.org/pdf/2502.03128) |
| 4 | +[](../../../models/tts/metis/README.md) |
| 5 | +[](https://huggingface.co/amphion/metis) |
| 6 | +[](https://modelscope.cn/models/amphion/metis) |
| 7 | + |
| 8 | +<!-- [](https://huggingface.co/spaces/amphion/metis) --> |
| 9 | +<!-- [](https://modelscope.cn/studios/amphion/metis) --> |
| 10 | + |
| 11 | +## Overview |
| 12 | + |
| 13 | +We introduce ***Metis***, a foundation model for unified speech generation. |
| 14 | +Unlike previous task-specific or multi-task models, Metis follows a pre-training and fine-tuning paradigm. It is pre-trained on large-scale unlabeled speech data using masked generative modeling and then fine-tuned to adapt to diverse speech generation tasks. |
| 15 | +Specifically, (1) Metis utilizes two discrete speech representations: SSL tokens derived from speech self-supervised learning (SSL) features, and acoustic tokens directly quantized from waveforms. (2) Metis performs masked generative pre-training on SSL tokens, utilizing 300K hours of diverse speech data, without any additional condition. (3) Through fine-tuning with task-specific conditions, Metis achieves efficient adaptation to various speech generation tasks while supporting multimodal input, even when using limited data and trainable parameters. |
| 16 | +Experiments demonstrate that Metis can serve as a foundation model for unified speech generation: Metis outperforms state-of-the-art task-specific or multi-task systems |
| 17 | +across five speech generation tasks, including zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech, even with fewer than 20M trainable parameters or 300 times less training data. |
| 18 | +Audio samples are available at [demo page](https://metis-demo.github.io/). |
| 19 | + |
| 20 | + |
| 21 | +<div align="center"> |
| 22 | + <img src="../../../imgs/metis/pre-train.png" width="42%"> |
| 23 | + <img src="../../../imgs/metis/fine-tune.png" width="48%"> |
| 24 | +</div> |
| 25 | +<div align="center"> |
| 26 | + <p><i>Pre-training (left) and fine-tuning (right).</i></p> |
| 27 | +</div> |
| 28 | + |
| 29 | +## News |
| 30 | + |
| 31 | +- **2025/02/26**: We release ***Metis***, a foundation model for unified speech generation. The system supports zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech. |
| 32 | + |
| 33 | + |
| 34 | +<!-- ## Todo List |
| 35 | +
|
| 36 | +- [ ] Add inference code for lip2speech --> |
| 37 | + |
| 38 | + |
| 39 | +## Model Introduction |
| 40 | + |
| 41 | +Metis is fully compatible with MaskGCT and shares several key model components with it. These shared components are: |
| 42 | + |
| 43 | + |
| 44 | +| Model Name | Description | |
| 45 | +| --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | |
| 46 | +| [Semantic Codec](https://huggingface.co/amphion/MaskGCT/tree/main/semantic_codec) | Converting speech to semantic tokens. | |
| 47 | +| [Acoustic Codec](https://huggingface.co/amphion/MaskGCT/tree/main/acoustic_codec) | Converting speech to acoustic tokens and reconstructing waveform from acoustic tokens. | |
| 48 | +| [Semantic2Acoustic](https://huggingface.co/amphion/MaskGCT/tree/main/s2a_model) | Predicts acoustic tokens conditioned on semantic tokens. | |
| 49 | +<!-- | [MaskGCT-T2S](https://huggingface.co/amphion/MaskGCT/tree/main/t2s_model) | Predicting semantic tokens with text and prompt semantic tokens. | --> |
| 50 | + |
| 51 | +We open-source the pretrained model checkpoint of the first stage of Metis (with masked generative pre-training), as well as the fine-tuned models for speech enhancement (SE), target speaker extraction (TSE), voice conversion (VC), lip-to-speech (L2S), and the unified multi-task (Omni) model. |
| 52 | + |
| 53 | +For zero-shot text-to-speech, you can download the text2semantic model from MaskGCT, which is compatible with the Metis framework. |
| 54 | + |
| 55 | +| Model Name | Description | |
| 56 | +| --- | --- | |
| 57 | +| [Metis-Base](https://huggingface.co/amphion/metis/tree/main/metis_base) | The base model pre-trained with masked generative pre-training. | |
| 58 | +| [Metis-TSE](https://huggingface.co/amphion/metis/tree/main/metis_tse) | Fine-tuned model for target speaker extraction. Available in both full-scale and LoRA ($r = 32$) versions. | |
| 59 | +| [Metis-VC](https://huggingface.co/amphion/metis/tree/main/metis_vc) | Fine-tuned model for voice conversion. Available in full-scale version. | |
| 60 | +| [Metis-SE](https://huggingface.co/amphion/metis/tree/main/metis_se) | Fine-tuned model for speech enhancement. Available in both full-scale and LoRA ($r = 32$) versions. | |
| 61 | +| [Metis-L2S](https://huggingface.co/amphion/metis/tree/main/metis_l2s) | Fine-tuned model for lip-to-speech. Available in full-scale version. | |
| 62 | +| [Metis-TTS](https://huggingface.co/amphion/MaskGCT/tree/main/t2s_model) | Zero-shot text-to-speech model (as same as the first stage of MaskGCT). | |
| 63 | +| [Metis-Omni](https://huggingface.co/amphion/metis/tree/main/metis_omni) | Unified multi-task model supporting zero-shot TTS, VC, TSE, and SE. | |
| 64 | + |
| 65 | + |
| 66 | +## Usage |
| 67 | + |
| 68 | +To run this model, you need to follow the steps below: |
| 69 | + |
| 70 | +1. Clone the repository and install the environment. |
| 71 | +2. Run the Inference script. |
| 72 | + |
| 73 | +### Clone and Environment |
| 74 | + |
| 75 | +#### 1. Clone the repository |
| 76 | + |
| 77 | +```bash |
| 78 | +git clone https://github.com/open-mmlab/Amphion.git |
| 79 | +cd Amphion |
| 80 | +``` |
| 81 | +#### 2. Install the environment |
| 82 | + |
| 83 | +Before start installing, making sure you are under the `Amphion` directory. If not, use `cd` to enter. |
| 84 | + |
| 85 | +Since we use `phonemizer` to convert text to phoneme, you need to install `espeak-ng` first. More details can be found [here](https://bootphon.github.io/phonemizer/install.html). Choose the correct installation command according to your operating system: |
| 86 | + |
| 87 | +```bash |
| 88 | +# For Debian-like distribution (e.g. Ubuntu, Mint, etc.) |
| 89 | +sudo apt-get install espeak-ng |
| 90 | +# For RedHat-like distribution (e.g. CentOS, Fedora, etc.) |
| 91 | +sudo yum install espeak-ng |
| 92 | + |
| 93 | +# For Windows |
| 94 | +# Please visit https://github.com/espeak-ng/espeak-ng/releases to download .msi installer |
| 95 | +``` |
| 96 | + |
| 97 | +**The environment used for Metis is the same as the one used for MaskGCT.** |
| 98 | + |
| 99 | +Now, we are going to install the environment. It is recommended to use conda to configure: |
| 100 | + |
| 101 | +```bash |
| 102 | +conda create -n maskgct python=3.10 |
| 103 | +conda activate maskgct |
| 104 | + |
| 105 | +pip install -r models/tts/maskgct/requirements.txt |
| 106 | +``` |
| 107 | + |
| 108 | +### Inference |
| 109 | + |
| 110 | +#### 1. Inference Script |
| 111 | + |
| 112 | +```bash |
| 113 | +# Metis TSE |
| 114 | +python -m models.tts.metis.metis_infer_tse |
| 115 | + |
| 116 | +# Metis SE |
| 117 | +python -m models.tts.metis.metis_infer_se |
| 118 | + |
| 119 | +# Metis VC |
| 120 | +python -m models.tts.metis.metis_infer_vc |
| 121 | + |
| 122 | +# Metis Lip2Speech |
| 123 | +python -m models.tts.metis.metis_infer_l2s |
| 124 | +``` |
| 125 | + |
| 126 | +You can also use a similar framework for inference with MaskGCT: |
| 127 | + |
| 128 | +```bash |
| 129 | +# Metis TTS (MaskGCT) |
| 130 | +python -m models.tts.maskgct.maskgct_infer_tts |
| 131 | +``` |
| 132 | + |
| 133 | +You can also use only one model (Metis-Omni) to infer TTS, VC, TSE, and SE tasks. |
| 134 | + |
| 135 | +```bash |
| 136 | +# Metis Omni |
| 137 | +python -m models.tts.metis.metis_infer_omni |
| 138 | +``` |
| 139 | + |
| 140 | +Running this will automatically download the pretrained model from HuggingFace and start the inference process. We provide example audio files for inference. Please see the scripts for more details and parameter configurations. |
| 141 | + |
| 142 | + |
| 143 | +#### 2. Example Usaage |
| 144 | + |
| 145 | +Take Metis-TSE for example, the inference script first downloads the model checkpoints: |
| 146 | + |
| 147 | +```python |
| 148 | +# download base model, lora weights, and adapter weights |
| 149 | +base_ckpt_dir = snapshot_download( |
| 150 | + "amphion/metis", |
| 151 | + repo_type="model", |
| 152 | + local_dir="./models/tts/metis/ckpt", |
| 153 | + allow_patterns=["metis_base/model.safetensors"], |
| 154 | +) |
| 155 | +lora_ckpt_dir = snapshot_download( |
| 156 | + "amphion/metis", |
| 157 | + repo_type="model", |
| 158 | + local_dir="./models/tts/metis/ckpt", |
| 159 | + allow_patterns=["metis_tse/metis_tse_lora_32.safetensors"], |
| 160 | +) |
| 161 | +adapter_ckpt_dir = snapshot_download( |
| 162 | + "amphion/metis", |
| 163 | + repo_type="model", |
| 164 | + local_dir="./models/tts/metis/ckpt", |
| 165 | + allow_patterns=["metis_tse/metis_tse_lora_32_adapter.safetensors"], |
| 166 | +) |
| 167 | +``` |
| 168 | + |
| 169 | +Then, the script will load the model checkpoints and initialize the fine-tined Metis model: |
| 170 | + |
| 171 | +```python |
| 172 | +base_ckpt_path = os.path.join(base_ckpt_dir, "metis_base/model.safetensors") |
| 173 | +lora_ckpt_path = os.path.join( |
| 174 | + lora_ckpt_dir, "metis_tse/metis_tse_lora_32.safetensors" |
| 175 | +) |
| 176 | +adapter_ckpt_path = os.path.join( |
| 177 | + adapter_ckpt_dir, "metis_tse/metis_tse_lora_32_adapter.safetensors" |
| 178 | +) |
| 179 | + |
| 180 | +metis = Metis( |
| 181 | + base_ckpt_path=base_ckpt_path, |
| 182 | + lora_ckpt_path=lora_ckpt_path, |
| 183 | + adapter_ckpt_path=adapter_ckpt_path, |
| 184 | + cfg=metis_cfg, |
| 185 | + device=device, |
| 186 | + model_type="tse", |
| 187 | +) |
| 188 | +``` |
| 189 | + |
| 190 | +Finally, the script will generate the speech and save it to the `models/tts/metis/wav/tse/gen.wav` directory, you can change this in the script. |
| 191 | + |
| 192 | +```python |
| 193 | +prompt_speech_path = "./models/tts/metis/wav/tse/prompt.wav" |
| 194 | +source_speech_path = "./models/tts/metis/wav/tse/mix.wav" |
| 195 | + |
| 196 | +n_timesteps = 10 |
| 197 | +cfg = 0.0 |
| 198 | + |
| 199 | +gen_speech = metis( |
| 200 | + prompt_speech_path=prompt_speech_path, |
| 201 | + source_speech_path=source_speech_path, |
| 202 | + cfg=cfg, |
| 203 | + n_timesteps=n_timesteps, |
| 204 | + model_type="tse", |
| 205 | +) |
| 206 | + |
| 207 | +sf.write("./models/tts/metis/wav/tse/gen.wav", gen_speech, 24000) |
| 208 | +``` |
| 209 | + |
| 210 | +## Citations |
| 211 | + |
| 212 | +If you use Metis in your research, please cite the following paper: |
| 213 | + |
| 214 | +```bibtex |
| 215 | +@article{wang2025metis, |
| 216 | + title={Metis: A Foundation Speech Generation Model with Masked Generative Pre-training}, |
| 217 | + author={Wang, Yuancheng and Zheng, Jiachen and Zhang, Junan and Zhang, Xueyao and Liao, Huan and Wu, Zhizheng}, |
| 218 | + journal={arXiv preprint arXiv:2502.03128}, |
| 219 | + year={2025} |
| 220 | +} |
| 221 | +@inproceedings{wang2024maskgct, |
| 222 | + author={Wang, Yuancheng and Zhan, Haoyue and Liu, Liwei and Zeng, Ruihong and Guo, Haotian and Zheng, Jiachen and Zhang, Qiang and Zhang, Xueyao and Zhang, Shunsi and Wu, Zhizheng}, |
| 223 | + title={MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer}, |
| 224 | + booktitle = {{ICLR}}, |
| 225 | + publisher = {OpenReview.net}, |
| 226 | + year = {2025} |
| 227 | +} |
| 228 | +@article{amphion_v0.2, |
| 229 | + title = {Overview of the Amphion Toolkit (v0.2)}, |
| 230 | + author = {Jiaqi Li and Xueyao Zhang and Yuancheng Wang and Haorui He and Chaoren Wang and Li Wang and Huan Liao and Junyi Ao and Zeyu Xie and Yiqiao Huang and Junan Zhang and Zhizheng Wu}, |
| 231 | + year = {2025}, |
| 232 | + journal = {arXiv preprint arXiv:2501.15442}, |
| 233 | +} |
| 234 | +@inproceedings{amphion, |
| 235 | + author={Zhang, Xueyao and Xue, Liumeng and Gu, Yicheng and Wang, Yuancheng and Li, Jiaqi and He, Haorui and Wang, Chaoren and Song, Ting and Chen, Xi and Fang, Zihao and Chen, Haopeng and Zhang, Junan and Tang, Tze Ying and Zou, Lexiao and Wang, Mingxuan and Han, Jun and Chen, Kai and Li, Haizhou and Wu, Zhizheng}, |
| 236 | + title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit}, |
| 237 | + booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024}, |
| 238 | + year={2024} |
| 239 | +} |
| 240 | +``` |
0 commit comments