Skip to content

Commit d420444

Browse files
aleksandr-mokrovaleksandr-mokrov
authored
Add mistral-small-24b-instruct-2501 model to llm-chatbot (#3097)
CVS-174002
1 parent 3da1f12 commit d420444

File tree

3 files changed

+38
-75
lines changed

3 files changed

+38
-75
lines changed

‎notebooks/llm-chatbot/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ For more details, please refer to [model_card](https://huggingface.co/Qwen/Qwen2
6565
* **mistral-7B-Instruct-v0.3** - The Mistral-7B-Instruct-v0.3 state‑of‑the‑art large language model (LLM) useful on a variety of language understanding and generation tasks, it is an instruct fine-tuned version of the Mistral-7B-v0.3. You can find more details about model in the [model card](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3).
6666
>**Note**: run model with demo, you will need to accept license agreement.
6767
>You must be a registered user in 🤗 Hugging Face Hub. Please visit [HuggingFace model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), carefully read terms of usage and click accept button. You will need to use an access token for the code below to run. For more information on access tokens, refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).
68+
* **Mistral-Small-24B-Instruct-2501** - Mistral Small is a 24B parameter instruction-tuned language model from Mistral AI. More details about the model can be found in the [Mistral AI model card](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501).
6869
* **zephyr-7b-beta** - Zephyr is a series of language models that are trained to act as helpful assistants. Zephyr-7B-beta is the second model in the series, and is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) that was trained on on a mix of publicly available, synthetic datasets using [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290). You can find more details about model in [technical report](https://arxiv.org/abs/2310.16944) and [HuggingFace model card](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta).
6970
* **neural-chat-7b-v3-1** - Mistral-7b model fine-tuned using Intel Gaudi. The model fine-tuned on the open source dataset [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) and aligned with [Direct Preference Optimization (DPO) algorithm](https://arxiv.org/abs/2305.18290). More details can be found in [model card](https://huggingface.co/Intel/neural-chat-7b-v3-1) and [blog post](https://medium.com/@NeuralCompressor/the-practice-of-supervised-finetuning-and-direct-preference-optimization-on-habana-gaudi2-a1197d8a3cd3).
7071
* **notus-7b-v1** - Notus is a collection of fine-tuned models using [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290). and related [RLHF](https://huggingface.co/blog/rlhf) techniques. This model is the first version, fine-tuned with DPO over zephyr-7b-sft. Following a data-first approach, the only difference between Notus-7B-v1 and Zephyr-7B-beta is the preference dataset used for dDPO. Proposed approach for dataset creation helps to effectively fine-tune Notus-7b that surpasses Zephyr-7B-beta and Claude 2 on [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval/). More details about model can be found in [model card](https://huggingface.co/argilla/notus-7b-v1).

‎notebooks/llm-chatbot/llm-chatbot-generate-api.ipynb‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@
8888
"\"accelerate\" \\\n",
8989
"\"gradio>=4.19\" \\\n",
9090
"\"transformers==4.53.3\" \\\n",
91-
"\"huggingface-hub>=0.26.5\" \\\n",
91+
"\"huggingface-hub===0.35.3\" \\\n",
9292
"\"einops\" \"transformers_stream_generator\" \"tiktoken\" \"bitsandbytes\"\n",
9393
"\n",
9494
"if platform.system() == \"Darwin\":\n",
@@ -403,6 +403,22 @@
403403
" except OSError:\n",
404404
" notebook_login()\n",
405405
"```\n",
406+
"* **Mistral-Small-24B-Instruct-2501** - Mistral Small is a 24B parameter instruction-tuned language model from Mistral AI. More details about the model can be found in the [Mistral AI model card](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501).\n",
407+
">**Note**: run model with demo, you will need to accept license agreement. \n",
408+
">You must be a registered user in 🤗 Hugging Face Hub. Please visit [HuggingFace model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), carefully read terms of usage and click accept button. You will need to use an access token for the code below to run. For more information on access tokens, refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).\n",
409+
">You can login on Hugging Face Hub in notebook environment, using following code:\n",
410+
" \n",
411+
"```python\n",
412+
" ## login to huggingfacehub to get access to pretrained model \n",
413+
"\n",
414+
" from huggingface_hub import notebook_login, whoami\n",
415+
"\n",
416+
" try:\n",
417+
" whoami()\n",
418+
" print('Authorization token already provided')\n",
419+
" except OSError:\n",
420+
" notebook_login()\n",
421+
"```\n",
406422
"* **zephyr-7b-beta** - Zephyr is a series of language models that are trained to act as helpful assistants. Zephyr-7B-beta is the second model in the series, and is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) that was trained on on a mix of publicly available, synthetic datasets using [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290). You can find more details about model in [technical report](https://arxiv.org/abs/2310.16944) and [HuggingFace model card](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta).\n",
407423
"* **neural-chat-7b-v3-1** - Mistral-7b model fine-tuned using Intel Gaudi. The model fine-tuned on the open source dataset [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) and aligned with [Direct Preference Optimization (DPO) algorithm](https://arxiv.org/abs/2305.18290). More details can be found in [model card](https://huggingface.co/Intel/neural-chat-7b-v3-1) and [blog post](https://medium.com/@NeuralCompressor/the-practice-of-supervised-finetuning-and-direct-preference-optimization-on-habana-gaudi2-a1197d8a3cd3).\n",
408424
"* **notus-7b-v1** - Notus is a collection of fine-tuned models using [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290). and related [RLHF](https://huggingface.co/blog/rlhf) techniques. This model is the first version, fine-tuned with DPO over zephyr-7b-sft. Following a data-first approach, the only difference between Notus-7B-v1 and Zephyr-7B-beta is the preference dataset used for dDPO. Proposed approach for dataset creation helps to effectively fine-tune Notus-7b that surpasses Zephyr-7B-beta and Claude 2 on [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval/). More details about model can be found in [model card](https://huggingface.co/argilla/notus-7b-v1).\n",

‎notebooks/llm-chatbot/llm-chatbot.ipynb‎

Lines changed: 20 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@
8787
"\"datasets<4.0.0\" \\\n",
8888
"\"accelerate\" \\\n",
8989
"\"gradio>=4.19\" \\\n",
90-
"\"huggingface-hub>=0.26.5\" \\\n",
90+
"\"huggingface-hub==0.35.3\" \\\n",
9191
" \"einops\" \"transformers==4.53.3\" \"transformers_stream_generator\" \"tiktoken\" \"bitsandbytes\"\n",
9292
"\n",
9393
"if platform.system() == \"Darwin\":\n",
@@ -298,7 +298,22 @@
298298
" except OSError:\n",
299299
" notebook_login()\n",
300300
"```\n",
301+
"* **Mistral-Small-24B-Instruct-2501** - Mistral Small is a 24B parameter instruction-tuned language model from Mistral AI. More details about the model can be found in the [Mistral AI model card](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501).\n",
302+
">**Note**: run model with demo, you will need to accept license agreement. \n",
303+
">You must be a registered user in 🤗 Hugging Face Hub. Please visit [HuggingFace model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), carefully read terms of usage and click accept button. You will need to use an access token for the code below to run. For more information on access tokens, refer to [this section of the documentation](https://huggingface.co/docs/hub/security-tokens).\n",
304+
">You can login on Hugging Face Hub in notebook environment, using following code:\n",
305+
" \n",
306+
"```python\n",
307+
" ## login to huggingfacehub to get access to pretrained model \n",
308+
"\n",
309+
" from huggingface_hub import notebook_login, whoami\n",
301310
"\n",
311+
" try:\n",
312+
" whoami()\n",
313+
" print('Authorization token already provided')\n",
314+
" except OSError:\n",
315+
" notebook_login()\n",
316+
"```\n",
302317
"* **zephyr-7b-beta** - Zephyr is a series of language models that are trained to act as helpful assistants. Zephyr-7B-beta is the second model in the series, and is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) that was trained on on a mix of publicly available, synthetic datasets using [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290). You can find more details about model in [technical report](https://arxiv.org/abs/2310.16944) and [HuggingFace model card](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta).\n",
303318
"* **neural-chat-7b-v3-1** - Mistral-7b model fine-tuned using Intel Gaudi. The model fine-tuned on the open source dataset [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) and aligned with [Direct Preference Optimization (DPO) algorithm](https://arxiv.org/abs/2305.18290). More details can be found in [model card](https://huggingface.co/Intel/neural-chat-7b-v3-1) and [blog post](https://medium.com/@NeuralCompressor/the-practice-of-supervised-finetuning-and-direct-preference-optimization-on-habana-gaudi2-a1197d8a3cd3).\n",
304319
"* **notus-7b-v1** - Notus is a collection of fine-tuned models using [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290). and related [RLHF](https://huggingface.co/blog/rlhf) techniques. This model is the first version, fine-tuned with DPO over zephyr-7b-sft. Following a data-first approach, the only difference between Notus-7B-v1 and Zephyr-7B-beta is the preference dataset used for dDPO. Proposed approach for dataset creation helps to effectively fine-tune Notus-7b that surpasses Zephyr-7B-beta and Claude 2 on [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval/). More details about model can be found in [model card](https://huggingface.co/argilla/notus-7b-v1).\n",
@@ -624,7 +639,7 @@
624639
},
625640
{
626641
"cell_type": "code",
627-
"execution_count": 9,
642+
"execution_count": null,
628643
"id": "c4ef9112",
629644
"metadata": {
630645
"collapsed": false,
@@ -637,6 +652,9 @@
637652
"source": [
638653
"from pathlib import Path\n",
639654
"\n",
655+
"from llm_config import compression_configs\n",
656+
"\n",
657+
"\n",
640658
"pt_model_id = model_configuration[\"model_id\"]\n",
641659
"pt_model_name = model_id.value.split(\"-\")[0]\n",
642660
"fp16_model_dir = Path(model_id.value) / \"FP16\"\n",
@@ -672,78 +690,6 @@
672690
"\n",
673691
"\n",
674692
"def convert_to_int4():\n",
675-
" compression_configs = {\n",
676-
" \"zephyr-7b-beta\": {\n",
677-
" \"sym\": True,\n",
678-
" \"group_size\": 64,\n",
679-
" \"ratio\": 0.6,\n",
680-
" },\n",
681-
" \"mistral-7b\": {\n",
682-
" \"sym\": True,\n",
683-
" \"group_size\": 64,\n",
684-
" \"ratio\": 0.6,\n",
685-
" },\n",
686-
" \"minicpm-2b-dpo\": {\n",
687-
" \"sym\": True,\n",
688-
" \"group_size\": 64,\n",
689-
" \"ratio\": 0.6,\n",
690-
" },\n",
691-
" \"gemma-2b-it\": {\n",
692-
" \"sym\": True,\n",
693-
" \"group_size\": 64,\n",
694-
" \"ratio\": 0.6,\n",
695-
" },\n",
696-
" \"notus-7b-v1\": {\n",
697-
" \"sym\": True,\n",
698-
" \"group_size\": 64,\n",
699-
" \"ratio\": 0.6,\n",
700-
" },\n",
701-
" \"neural-chat-7b-v3-1\": {\n",
702-
" \"sym\": True,\n",
703-
" \"group_size\": 64,\n",
704-
" \"ratio\": 0.6,\n",
705-
" },\n",
706-
" \"llama-2-chat-7b\": {\n",
707-
" \"sym\": True,\n",
708-
" \"group_size\": 128,\n",
709-
" \"ratio\": 0.8,\n",
710-
" },\n",
711-
" \"llama-3-8b-instruct\": {\n",
712-
" \"sym\": True,\n",
713-
" \"group_size\": 128,\n",
714-
" \"ratio\": 0.8,\n",
715-
" },\n",
716-
" \"llama-3.1-8b-instruct\": {\n",
717-
" \"sym\": True,\n",
718-
" \"group_size\": 128,\n",
719-
" \"ratio\": 1.0,\n",
720-
" },\n",
721-
" \"gemma-7b-it\": {\n",
722-
" \"sym\": True,\n",
723-
" \"group_size\": 128,\n",
724-
" \"ratio\": 0.8,\n",
725-
" },\n",
726-
" \"chatglm2-6b\": {\n",
727-
" \"sym\": True,\n",
728-
" \"group_size\": 128,\n",
729-
" \"ratio\": 0.72,\n",
730-
" },\n",
731-
" \"qwen-7b-chat\": {\"sym\": True, \"group_size\": 128, \"ratio\": 0.6},\n",
732-
" \"red-pajama-3b-chat\": {\n",
733-
" \"sym\": False,\n",
734-
" \"group_size\": 128,\n",
735-
" \"ratio\": 0.5,\n",
736-
" },\n",
737-
" \"qwen2.5-7b-instruct\": {\"sym\": True, \"group_size\": 128, \"ratio\": 1.0},\n",
738-
" \"qwen2.5-3b-instruct\": {\"sym\": True, \"group_size\": 128, \"ratio\": 1.0},\n",
739-
" \"qwen2.5-14b-instruct\": {\"sym\": True, \"group_size\": 128, \"ratio\": 1.0},\n",
740-
" \"qwen2.5-1.5b-instruct\": {\"sym\": True, \"group_size\": 128, \"ratio\": 1.0},\n",
741-
" \"qwen2.5-0.5b-instruct\": {\"sym\": True, \"group_size\": 128, \"ratio\": 1.0},\n",
742-
" \"default\": {\n",
743-
" \"sym\": False,\n",
744-
" },\n",
745-
" }\n",
746-
"\n",
747693
" model_compression_params = compression_configs.get(model_id.value, compression_configs[\"default\"])\n",
748694
" if (int4_model_dir / \"openvino_model.xml\").exists():\n",
749695
" return\n",

0 commit comments

Comments
 (0)