Merge pull request #171 from neulab/jean-new-model_cambrian

[New Model] Adding Cambrian Model
EvolvingLMMs-Lab · Jul 30, 2024 · 19b96da · 19b96da
2 parents 04009b2 + 10f3de5
commit 19b96da
Show file tree

Hide file tree

Showing 3 changed files with 479 additions and 0 deletions.
diff --git a/M-MMMU/README.md b/M-MMMU/README.md
@@ -0,0 +1,58 @@
+# Multilingual MMMU
+Multilingual MMMU: Multilingual Massive Multi-task Multimodality Understanding
+
+## Instruction
+Multilingual MMMU is a benchmark that aims to evaluate the performance of multimodal foundation models in understanding multilingual multimodal data across multiple tasks. It provides a standardized evaluation framework and dataset for researchers to compare and analyze the general understandability of different multimodal models in handling various knowledge tasks in multilingual cases. 
+
+We are looking for high school or college exam questions that have both image and text in the question:
+- The question needs to contain both image and text.
+- The question needs to be highly precise, and the answer needs to be easy to verify, e.g., a multiple-choice answer, a number, a list of number, or a boolean variable, etc.
+- The topic covers Art, Business, Medical,  Humanities, Science and Engineering. See the full list of the category in the following.
+- You can either find questions from the Internet, or Textbook or come up with your own question (if you are an expert in that field). 
+
+## Category
+| Category                              	| Translation Count 	| Annotation Count 	|
+|---------------------------------------	|-------------------	|------------------	|
+| **Art and Design**                    	|                   	|                  	|
+| - Art                                 	|                   	|                  	|
+| - Art_Theory                          	|                   	|                  	|
+| - Design                              	|                   	|                  	|
+| - Music                               	|                   	|                  	|
+|                                       	|                   	|                  	|
+| **Business**                          	|                   	|                  	|
+| - Accounting                          	|                   	|                  	|
+| - Economics                           	|                   	|                  	|
+| - Finance                             	|                   	|                  	|
+| - Manage                              	|                   	|                  	|
+| - Marketing                           	|                   	|                  	|
+|                                       	|                   	|                  	|
+| **Health and Medicine**               	|                   	|                  	|
+| - Basic_Medical_Science               	|                   	|                  	|
+| - Clinical_Medicine                   	|                   	|                  	|
+| - Diagnostics_and_Laboratory_Medicine 	|                   	|                  	|
+| - Pharmacy                            	|                   	|                  	|
+| - Public_Health                       	|                   	|                  	|
+|                                       	|                   	|                  	|
+| **Humanities and Social Science**     	|                   	|                  	|
+| - History                             	|                   	|                  	|
+| - Literature                          	|                   	|                  	|
+| - Psychology                          	|                   	|                  	|
+| - Sociology                           	|                   	|                  	|
+|                                       	|                   	|                  	|
+| **Science**                           	|                   	|                  	|
+| - Biology                             	|                   	|                  	|
+| - Chemistry                           	|                   	|                  	|
+| - Geography                           	|                   	|                  	|
+| - Math                                	|                   	|                  	|
+| - Physics                             	|                   	|                  	|
+|                                       	|                   	|                  	|
+| **Tech and Engineering**              	|                   	|                  	|
+| - Agriculture                         	|                   	|                  	|
+| - Architecture_and_Engineering        	|                   	|                  	|
+| - Computer_Science                    	|                   	|                  	|
+| - Electronics                         	|                   	|                  	|
+| - Energy_and_Power                    	|                   	|                  	|
+| - Materials                           	|                   	|                  	|
+| - Mechanical_Engineering              	|                   	|                  	|
+
+## 
diff --git a/lmms_eval/models/__init__.py b/lmms_eval/models/__init__.py
@@ -12,6 +12,7 @@
 logger.add(sys.stdout, level="WARNING")
 
 AVAILABLE_MODELS = {
+    "cambrian1": "Cambrian1",
     "llava": "Llava",
     "qwen_vl": "Qwen_VL",
     "fuyu": "Fuyu",