Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Add Dataset] SEEDBench 2 Plus #180

Merged
merged 1 commit into from
Aug 7, 2024
Merged

[Add Dataset] SEEDBench 2 Plus #180

merged 1 commit into from
Aug 7, 2024

Conversation

abzb1
Copy link
Contributor

@abzb1 abzb1 commented Aug 2, 2024

Add the SEEDBench 2 Plus dataset and minor change in dataset docs.

Key Changes:

  1. Add SEEDBench 2 Plus Dataset:
    Added the SEEDBench 2 Plus dataset proposed in the paper, which already implemented in VLMEvalKit.

  2. Rename Existing SEEDBench 2 Task Names:
    Changed the task names in the existing SEEDBench 2 dataset from hyphens (-) to underscores (_) to maintain consistency with other datasets. (seedbench-2 -> seenbench_2)

  3. Add Descriptions for seedbench 2 plus:
    Added descriptions for seedbench 2 plus

I have completed tests for SEEDBench 2 Plus benchmark on several models with this code change, but please leave a comment if further review is needed.

The data uploaded to the datasets repository was created with the following code:

data_json_fname = "SEED-Bench-2-plus-text-rich.json"
data = json.load(open(data_json_fname))

for i, d in enumerate(tqdm(data)):
    img = d["data_id"]
    img = Image.open(img)
    data[i]["image"] = img

data_in_list_dict = {k: [d[k] for d in data] for k in data[0]}

ds = Dataset.from_dict(data_in_list_dict)
ds.push_to_hub("doolayer/my-image-dataset")

If the datasets repository should be managed by lmms-lab, I would appreciate comments on the necessary steps to handle that aspect.

Thank you.


Before you open a pull-request, please check if a similar issue already exists or has been closed before.

When you open a pull-request, please be sure to include the following

  • A descriptive title: [xxx] XXXX
  • A detailed description

Thank you for your contributions!

@Luodian
Copy link
Contributor

Luodian commented Aug 7, 2024

Thanks for the integration, looks pretty good!

@Luodian Luodian merged commit 82abfa9 into EvolvingLMMs-Lab:main Aug 7, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants