[Bug] `trust_remote_code` is not supported anymore.

[keviddles]

What happened?

I am trying to run the Optimizing prompts for a ReAct agent code from Getting Started III on the homepage.

It fails with "trust_remote_code is not supported anymore."

Steps to reproduce

Running:

import dspy
from dspy.datasets import HotPotQA

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

def search_wikipedia(query: str) -> list[str]:
    results = dspy.ColBERTv2(url="http://20.102.90.50:2017/wiki17_abstracts")(query, k=3)
    return [x["text"] for x in results]

trainset = [x.with_inputs('question') for x in HotPotQA(train_seed=2024, train_size=500).train]
react = dspy.ReAct("question -> answer", tools=[search_wikipedia])

tp = dspy.MIPROv2(metric=dspy.evaluate.answer_exact_match, auto="light", num_threads=24)
optimized_react = tp.compile(react, trainset=trainset)

Results in:

`trust_remote_code` is not supported anymore.
Please check that the Hugging Face dataset 'hotpot_qa' isn't based on a loading script and remove `trust_remote_code`.
If the dataset is based on a loading script, please ask the dataset author to remove it and convert it to a standard format like Parquet.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[17], line 10
      7     results = dspy.ColBERTv2(url="http://20.102.90.50:2017/wiki17_abstracts")(query, k=3)
      8     return [x["text"] for x in results]
---> 10 trainset = [x.with_inputs('question') for x in HotPotQA(train_seed=2024, train_size=500).train]
     11 react = dspy.ReAct("question -> answer", tools=[search_wikipedia])
     13 tp = dspy.MIPROv2(metric=dspy.evaluate.answer_exact_match, auto="light", num_threads=24)

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/dspy/datasets/hotpotqa.py:23](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/dspy/datasets/hotpotqa.py#line=22), in HotPotQA.__init__(self, only_hard_examples, keep_details, unofficial_dev, *args, **kwargs)
     16 assert only_hard_examples, (
     17     "Care must be taken when adding support for easy examples."
     18     "Dev must be all hard to match official dev, but training can be flexible."
     19 )
     21 from datasets import load_dataset
---> 23 hf_official_train = load_dataset("hotpot_qa", "fullwiki", split="train", trust_remote_code=True)
     24 hf_official_dev = load_dataset("hotpot_qa", "fullwiki", split="validation", trust_remote_code=True)
     26 official_train = []

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py:1392](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py#line=1391), in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, keep_in_memory, save_infos, revision, token, streaming, num_proc, storage_options, **config_kwargs)
   1387 verification_mode = VerificationMode(
   1388     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   1389 )
   1391 # Create a dataset builder
-> 1392 builder_instance = load_dataset_builder(
   1393     path=path,
   1394     name=name,
   1395     data_dir=data_dir,
   1396     data_files=data_files,
   1397     cache_dir=cache_dir,
   1398     features=features,
   1399     download_config=download_config,
   1400     download_mode=download_mode,
   1401     revision=revision,
   1402     token=token,
   1403     storage_options=storage_options,
   1404     **config_kwargs,
   1405 )
   1407 # Return iterable dataset in case of streaming
   1408 if streaming:

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py:1132](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py#line=1131), in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, storage_options, **config_kwargs)
   1130 if features is not None:
   1131     features = _fix_for_backward_compatible_features(features)
-> 1132 dataset_module = dataset_module_factory(
   1133     path,
   1134     revision=revision,
   1135     download_config=download_config,
   1136     download_mode=download_mode,
   1137     data_dir=data_dir,
   1138     data_files=data_files,
   1139     cache_dir=cache_dir,
   1140 )
   1141 # Get dataset builder class
   1142 builder_kwargs = dataset_module.builder_kwargs

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py:1031](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py#line=1030), in dataset_module_factory(path, revision, download_config, download_mode, data_dir, data_files, cache_dir, **download_kwargs)
   1026             if isinstance(e1, FileNotFoundError):
   1027                 raise FileNotFoundError(
   1028                     f"Couldn't find any data file at {relative_to_absolute_path(path)}. "
   1029                     f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1030                 ) from None
-> 1031             raise e1 from None
   1032 else:
   1033     raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.")

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py:989](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py#line=988), in dataset_module_factory(path, revision, download_config, download_mode, data_dir, data_files, cache_dir, **download_kwargs)
    981 try:
    982     api.hf_hub_download(
    983         repo_id=path,
    984         filename=filename,
   (...)    987         proxies=download_config.proxies,
    988     )
--> 989     raise RuntimeError(f"Dataset scripts are no longer supported, but found {filename}")
    990 except EntryNotFoundError:
    991     # Use the infos from the parquet export except in some cases:
    992     if data_dir or data_files or (revision and revision != "main"):

RuntimeError: Dataset scripts are no longer supported, but found hotpot_qa.py

DSPy version

2.6.27

[keviddles]

In fact, I'm seeing issues with the other examples in Getting Started III.

class RAG(dspy.Module):
    def __init__(self, num_docs=5):
        self.num_docs = num_docs
        self.respond = dspy.ChainOfThought("context, question -> response")

    def forward(self, question):
        context = search(question, k=self.num_docs)   # defined in tutorial linked below
        return self.respond(context=context, question=question)

tp = dspy.MIPROv2(metric=dspy.evaluate.SemanticF1(decompositional=True), auto="medium", num_threads=24)
optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[18], line 11
      8         return self.respond(context=context, question=question)
     10 tp = dspy.MIPROv2(metric=dspy.evaluate.SemanticF1(decompositional=True), auto="medium", num_threads=24)
---> 11 optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)

NameError: name 'trainset' is not defined

Should I be defining my own trainset here? If so, what format should it take?

import random
from typing import Literal

from datasets import load_dataset

import dspy
from dspy.datasets import DataLoader

# Load the Banking77 dataset.
CLASSES = load_dataset("PolyAI/banking77", split="train", trust_remote_code=True).features["label"].names
kwargs = {"fields": ("text", "label"), "input_keys": ("text",), "split": "train", "trust_remote_code": True}

# Load the first 2000 examples from the dataset, and assign a hint to each *training* example.
trainset = [
    dspy.Example(x, hint=CLASSES[x.label], label=CLASSES[x.label]).with_inputs("text", "hint")
    for x in DataLoader().from_huggingface(dataset_name="PolyAI/banking77", **kwargs)[:2000]
]
random.Random(0).shuffle(trainset)

`trust_remote_code` is not supported anymore.
Please check that the Hugging Face dataset 'PolyAI[/banking77](http://localhost:8898/banking77)' isn't based on a loading script and remove `trust_remote_code`.
If the dataset is based on a loading script, please ask the dataset author to remove it and convert it to a standard format like Parquet.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[19], line 10
      7 from dspy.datasets import DataLoader
      9 # Load the Banking77 dataset.
---> 10 CLASSES = load_dataset("PolyAI[/banking77](http://localhost:8898/banking77)", split="train", trust_remote_code=True).features["label"].names
     11 kwargs = {"fields": ("text", "label"), "input_keys": ("text",), "split": "train", "trust_remote_code": True}
     13 # Load the first 2000 examples from the dataset, and assign a hint to each *training* example.

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py:1392](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py#line=1391), in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, keep_in_memory, save_infos, revision, token, streaming, num_proc, storage_options, **config_kwargs)
   1387 verification_mode = VerificationMode(
   1388     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   1389 )
   1391 # Create a dataset builder
-> 1392 builder_instance = load_dataset_builder(
   1393     path=path,
   1394     name=name,
   1395     data_dir=data_dir,
   1396     data_files=data_files,
   1397     cache_dir=cache_dir,
   1398     features=features,
   1399     download_config=download_config,
   1400     download_mode=download_mode,
   1401     revision=revision,
   1402     token=token,
   1403     storage_options=storage_options,
   1404     **config_kwargs,
   1405 )
   1407 # Return iterable dataset in case of streaming
   1408 if streaming:

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py:1132](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py#line=1131), in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, storage_options, **config_kwargs)
   1130 if features is not None:
   1131     features = _fix_for_backward_compatible_features(features)
-> 1132 dataset_module = dataset_module_factory(
   1133     path,
   1134     revision=revision,
   1135     download_config=download_config,
   1136     download_mode=download_mode,
   1137     data_dir=data_dir,
   1138     data_files=data_files,
   1139     cache_dir=cache_dir,
   1140 )
   1141 # Get dataset builder class
   1142 builder_kwargs = dataset_module.builder_kwargs

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py:1031](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py#line=1030), in dataset_module_factory(path, revision, download_config, download_mode, data_dir, data_files, cache_dir, **download_kwargs)
   1026             if isinstance(e1, FileNotFoundError):
   1027                 raise FileNotFoundError(
   1028                     f"Couldn't find any data file at {relative_to_absolute_path(path)}. "
   1029                     f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1030                 ) from None
-> 1031             raise e1 from None
   1032 else:
   1033     raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_path(path)}.")

File [~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py:989](http://localhost:8898/lab/tree/notebooks/~/.cache/uv/archive-v0/KO8BfiDHTjQ5IO2ZclrNP/lib/python3.12/site-packages/datasets/load.py#line=988), in dataset_module_factory(path, revision, download_config, download_mode, data_dir, data_files, cache_dir, **download_kwargs)
    981 try:
    982     api.hf_hub_download(
    983         repo_id=path,
    984         filename=filename,
   (...)    987         proxies=download_config.proxies,
    988     )
--> 989     raise RuntimeError(f"Dataset scripts are no longer supported, but found {filename}")
    990 except EntryNotFoundError:
    991     # Use the infos from the parquet export except in some cases:
    992     if data_dir or data_files or (revision and revision != "main"):

RuntimeError: Dataset scripts are no longer supported, but found banking77.py

[keviddles]

Aha, it looks like it may be related to this: huggingface/datasets#7693

Downgrading to datasets==3.6.0 appears to resolve the issues with the ReAct code. (The RAG example is still missing the trainset, though.)

Does this mean the underlying datasets need to be converted to Parquet files in order to work with datasets>=4.x?

[lhoestq]

Hi ! I updated the hotpot_qa dataset to make it compatible with datasets>=4 here: https://huggingface.co/datasets/hotpotqa/hotpot_qa/discussions/9

it should be all good now :)

[jannichorst]

Same issue with conll2003 in https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/entity_extraction/index.ipynb . But can confirm that downgrading to datasets==3.6.0 does the trick.

[lhoestq]

I pinged the author of conll2003 on the PR to convert the dataset to parquet: https://huggingface.co/datasets/eriktks/conll2003/discussions/15