PudMedQA tasks require trust_remote_code=True argument, but argument doesnt work

[dhp-ks]

Error:

The repository for bigbio/pubmed_qa contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/bigbio/pubmed_qa. You can avoid this prompt in future by passing the argument trust_remote_code=True`.

Do you wish to run the custom code? [y/N] Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 402, in _wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 236, in simple_evaluate
task_dict = get_task_dict(tasks, task_manager)
File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/init.py", line 618, in get_task_dict
task_name_from_string_dict = task_manager.load_task_or_group(
File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/init.py", line 414, in load_task_or_group
collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/init.py", line 314, in _load_individual_task_or_group
return _load_task(task_config, task=name_or_config)
File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/init.py", line 280, in _load_task
task_object = ConfigurableTask(config=config)
File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/task.py", line 822, in init
self.download(self.config.dataset_kwargs)
File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/task.py", line 929, in download
self.dataset = datasets.load_dataset(
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2129, in load_dataset
builder_instance = load_dataset_builder(
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1849, in load_dataset_builder
dataset_module = dataset_module_factory(
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1731, in dataset_module_factory
raise e1 from None
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1681, in dataset_module_factory
).get_module()
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1331, in get_module
trust_remote_code = resolve_trust_remote_code(self.trust_remote_code, self.name)
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 138, in resolve_trust_remote_code
raise ValueError(
ValueError: The repository for bigbio/pubmed_qa contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/bigbio/pubmed_qa.
Please pass the argument trust_remote_code=True to allow custom code to be run.

my code
model_args = { "pretrained": model_name, "revision": checkpoint, "dtype": "float16", "parallelize": True, "trust_remote_code": True }

results = simple_evaluate( model="hf", model_args=model_args, tasks=tasks, batch_size=batch_size, confirm_run_unsafe_code=True, )

I am running this in the background over multiple revisions so I cannot sit and enter Y each time. Why doesn't doing what it says fix the issue?

confirm_run_unsafe_code was added in a recent commit as an attempt to fix this, but it clearly didnt work. 0.4.7 (latest release) still has this broken

[baberabb]

Hi! Have you tried --trust_remote_code? If you're using simple_evaluate as an entry point you can set the HF datasets environment variable like so
The confirm_run_unsafe_code is mainly for code tasks, to run code generated by the model. I should make that clearer.

[franciscoliu]

I am running into the same error even if I have passed --trust_remote_code=True.

[baberabb]

probably better to set datasets.config.HF_DATASETS_TRUST_REMOTE_CODE = True at the top of your script

[franciscoliu]

Thanks for the reply! I ended up solving the problem by adding --trust_remote_code=True in api/task.py file when it loads dataset in.

[owos]

Thanks for the reply! I ended up solving the problem by adding --trust_remote_code=True in api/task.py file when it loads dataset in.

please in which part of the file did you add this?

[franciscoliu]

Thanks for the reply! I ended up solving the problem by adding --trust_remote_code=True in api/task.py file when it loads dataset in.

please in which part of the file did you add this?

At here:
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L931
Just add --trust_remote_code=True