Self-supervised Prompt Optimization

[fracapuano]

Cookbook Pull Request

Description

Prompt optimization using Metagpt's Self-supervised Prompt Optimization using Mistral models

Type of Change

What type of PR is it?

• New Cookbook
  • Notebook File
    • Does it work on google colab?
  • Markdown File
• Cookbook Update
  • Code Refactoring
  • Bug Fix
• README.md Update

---

• Other (please describe):

Cookbook Checklist:

• My code is easy to read and well structured.
• I've tagged the versions of any dependency required.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• My changes generate no new warnings or errors.

---

• My changes do not concern the cookbooks.

README.md Checklist

• I've added my cookbook to the table.

---

• My changes do not concern the README file.

Additional Context

• The currently available release of Metagpt's SPO is not really up to date, so using SPO requires some hacking
• Namely, one needs to run SPO from within the repo, modifying a few elements (mostly, config files)

[sophiamyang]

@fracapuano Thanks so much for the cookbook! Could you make sure it's runnable on Colab?

[fracapuano]

hey @sophiamyang 👋 --- did a little update, lmk what you think of it!

[sophiamyang]

Thanks @fracapuano ! I got this error somehow:

Writing MetaGPT/spo.py
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-17-1e3ffc3a4e45>](https://localhost:8080/#) in <cell line: 0>()
----> 1 get_ipython().run_cell_magic('writefile', 'MetaGPT/spo.py', '\nfrom metagpt.ext.spo.components.optimizer import PromptOptimizer\nfrom metagpt.ext.spo.utils.llm_client import SPO_LLM\n\n# Initialize LLM settings\nSPO_LLM.initialize(\n    # same temperature settings as metagpt\'s default!\n    optimize_kwargs={\n        "model": "mistral-large-latest", \n        "temperature": 0.6\n    },\n    evaluate_kwargs={\n        "model": "mistral-small-latest", \n        "temperature": 0.3\n    },\n    execute_kwargs={\n        "model": "mistral-small-latest", \n        "temperature": 0\n    }\n)\n\ntemplate_name = "job.yaml"  # change this for each prompt!\n\n# Create and run optimizer\noptimizer = PromptOptimizer(\n    optimized_path="workspace",  # Output directory\n    initial_round=1,  # Starting round\n    max_rounds=5,  # Maximum optimization rounds\n    template=template_name,  # Template file - Change this for each prompt!\n    name="Mistral-Prompt-Opt",  # Project name\n)\n\noptimizer.optimize()\n')

3 frames
<decorator-gen-98> in writefile(self, line, cell)

/usr/local/lib/python3.11/dist-packages/IPython/core/magics/osm.py in writefile(self, line, cell)
    854         with io.open(filename, mode, encoding='utf-8') as f:
    855             f.write(cell)

FileNotFoundError: [Errno 2] No such file or directory: 'MetaGPT/spo.py'

[jaccolor2]

Hi @fracapuano!

I've been trying to run your cookbook, but i ran into this error message:

Traceback (most recent call last):
  File "/mistralai/MetaGPT/metagpt/spo.py", line 2, in <module>
    from metagpt.ext.spo.components.optimizer import PromptOptimizer
ModuleNotFoundError: No module named 'metagpt'

Though i followed all the pip install and added a:

pip install metagpt

to make sure there wouldn't be any problem, but without success.

Would you have any insights on this?

[sophiamyang]

interesting. is it on pip? if so, we don't need to clone to whole repo and install from the Github repo @fracapuano?

[fracapuano]

hey @jaccolor2 👋 thank you so much for trying out my cookbook 🙏

• From what you say it looks like you haven't properly set up the notebook
• Did you (1) clone the repository and (2) %cd into it?

[fracapuano]

interesting. is it on pip? if so, we don't need to clone

@sophiamyang, it seems pypi's metagpt is not up to date with SPO. When I tried to implement this installing from pip rather than cloning the repo everything broke apart 🫠

[fracapuano]

Ciao @sophiamyang 👋

• d5ba0f7adds the report I have drafted in a markdown file. Lmk if you rather have it somewhere else than in this directory
• dba5b75 fixes the directory issue when running python spo.py 🫠 In general, I agree having to clone the repo is rather inconvenient, but I don't think there is a way around it as the repository seems to be outdated. Perhaps one could re-implement SPO into a mini-SPO directory, avoiding to rely on metagpt entirely & keeping this clean

[fracapuano]

Hey @sophiamyang @pandora-s-git 👋 Mind reviewing when you have a chance? 🙏