Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MIEB] Change Flickr30k to test split #1449

Merged
merged 94 commits into from
Nov 15, 2024
Merged

[MIEB] Change Flickr30k to test split #1449

merged 94 commits into from
Nov 15, 2024

Conversation

Jamie-Stirling
Copy link

@Jamie-Stirling Jamie-Stirling commented Nov 14, 2024 •
edited by isaac-chung
Loading

Fixes #1434

Flickr30k was previously underperforming (CLIP) for Recall@5. Changing to the test split of only 5k images/25k captions moves performance to be much closer to CLIP benchmark.

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Jamie-Stirling and others added 30 commits September 6, 2024 16:20
@Jamie-Stirling
wip: start adding BLIP models
1e50557
@Jamie-Stirling
add other blip variants
8f8e05c
@Jamie-Stirling
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
3ea4d3e 
…nto mieb-blip

Merge changes from main
@Jamie-Stirling
wip: add blip2_models.py
be8b4bb
@Jamie-Stirling
make lint
b57a395
@Jamie-Stirling
wip: implement blip2 wrapper
236a94f
@Jamie-Stirling
feat: add blip2 models, still mismatched names
1f2f8c3
@Jamie-Stirling
fix: remove projections from image and text embeddings
8c64860
@Jamie-Stirling
make lint
20839ca
@Jamie-Stirling
wip: add coco BLIP2
ec47c69
@Jamie-Stirling
fix: BLIP2 better zero-shot classification without text_proj and visi…
e8f4ae1 
…on_proj
@Jamie-Stirling
tidy blip2
57bc3b8
@Jamie-Stirling
add imagenet-dog-15 dataset
4cbec1b
@Jamie-Stirling
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
6da26dc 
…nto mieb-imagenet
@Jamie-Stirling
tidy and lint
35be38d
@Jamie-Stirling
remove unused import
83d0f45
@Jamie-Stirling
add cluster_accuracy, ari and nmi to Image.ClusteringEvaluator
a309de5
@Jamie-Stirling
add imagenet-10 clustering task
02c1f81
@Jamie-Stirling
add SOPI2IRetrieval
cede108
@Jamie-Stirling
add results forclip on ImageNet10Clustering and ImageNetDog15Clustering
d226748
@Jamie-Stirling
Merge branch 'mieb-imagenet' of https://github.com/Jamie-Stirling/mteb …
2086904 
…into mieb-sop
@Jamie-Stirling
add SOPI2IRetrieval results for clip 32
c7c6893
@Jamie-Stirling
add results for clip vit 32/SOPI2IRetrieval
5bf66fd
@Jamie-Stirling
merge incoming change from mieb
3866647
@Jamie-Stirling
resolve conflict
b1759dc
@Jamie-Stirling
add RP2kI2IRetrieval dataset
a890171
@Jamie-Stirling
add RP2kI2IRetrieval results with clip-vit-base-patch32
d3396fa
@Jamie-Stirling
update image retrieval __init__.py
aa58556
@gowitheflow-1998
fix ImageTextPair dataloading for large datasets; more compositionali…
91ad565 
…ty evaluation datasets
@Jamie-Stirling
add RP2kI2IRetrieval and METI2IRetrieval
fc4d1ee
Jamie-Stirling and others added 28 commits October 20, 2024 20:34
@Jamie-Stirling
exclude AbsTaskAny2AnyMultiChoice from test_load_data
d0cddd4
@gowitheflow-1998
fix e5v&vista
bc8aa01
@gowitheflow-1998
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
7aa5bd2 
…nto mieb-run-tasks-fix
@Jamie-Stirling
remove duplicate corpus entries from BLINKIT2TRetreival dataset
6d8ed9b
@gowitheflow-1998
task type fix for running tasks
53065b8
@Jamie-Stirling
update BLINKIT2T metadata
3385584
@gowitheflow-1998
fix wrong meta
3b93440
@gowitheflow-1998
run mieb script
3998ccd
@Jamie-Stirling
split ROxford, RParis into easy, medium and hard
a3649a7
@Jamie-Stirling
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
59a19a1 
…nto mieb-datasets
@Jamie-Stirling
make lint
de9c573
@Jamie-Stirling
Merge branch 'mieb-run-tasks-fix' of https://github.com/embeddings-be…
4c345a7 
…nchmark/mteb into mieb-datasets
@Jamie-Stirling
resolve conflict with mieb
2f7e2d9
@Jamie-Stirling
add BLINK as multi choice tasks
50af768
@Jamie-Stirling
resolve conflicts with mieb
baf9674
@Jamie-Stirling
fix conflicts with mieb
e646368
@Jamie-Stirling
resolve conflict
016c0d0
@Jamie-Stirling
fix: license metadata in wrong format
27444fd
@Jamie-Stirling
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
3a3f812 
…nto mieb-datasets
@Jamie-Stirling
remove null examples from corpus of ROxford and RParis
1abce84
@Jamie-Stirling
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
a0e8caa 
…nto mieb-datasets
@Jamie-Stirling
fix: add/remove subtasks from BLINKIT2IMultiChoice and BLINKIT2TMulti…
ac51be8 
…Choice
@Jamie-Stirling
update blink metadata
dca764a
@Jamie-Stirling
add updated BLINK results
2d371c7
@Jamie-Stirling
Merge branch 'mieb' of https://github.com/embeddings-benchmark/mteb i…
220362d 
…nto mieb-datasets
@Jamie-Stirling
merge upstream mieb
7e69e16
@Jamie-Stirling
change Flickr30k to test split
460a2b8
@Jamie-Stirling
change flickr to test split
bada500
isaac-chung
isaac-chung approved these changes Nov 15, 2024
View reviewed changes
Copy link
Collaborator

@isaac-chung isaac-chung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I ran these locally and yes it's much closer now. cc @gowitheflow-1998

CLIP benchmark: 0.834 / 0.949
Ours: 0.889 / 0.914

@isaac-chung isaac-chung merged commit 7e0779a into embeddings-benchmark:mieb Nov 15, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@isaac-chung isaac-chung isaac-chung approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Jamie-Stirling @isaac-chung @gowitheflow-1998