Readme.md

🔥 Data-Centric ML方向

入门指南

注:⚡为基础必读,💎为基础选读,💡为进阶阅读

Data Centric ML Basic

数据侧基础：

Data-Centric ML是研究使用数据生成、选择与配比等方法实现大规模，高效以及提升模型表现的研究方向。

• ⚡ 🗞️ MIT DCAI 2024

课程难度不大，但是涵盖了很多 DCAI 的 Topic，包括但不限于：Data Selection/Data Cleaning/Distribution Shift/Data Curation 等，每节课都有一个配套的实验，完成难度也不大，完成后可以对该topic有一个比较 general 的认识

• ⚡ 🗞️ 华盛顿大学 DCAI

LLM and VLM Basic

Quick Start

• ⚡ 📄+🔧 多模态大模型入门文档
• ⚡ 📄 Survey

ImageLLMs

• ⚡ 💭 BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• ⚡ 💭 MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• ⚡ 💭 Visual Instruction Tuning
• ⚡ 💭 Improved Baselines with Visual Instruction Tuning

VideoLLMs

• ⚡ 💭 Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• ⚡ 💭 Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
• ⚡ 💭 Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
• ⚡ 💭 MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
• ⚡ 💭 InternVideo2: Scaling Foundation Models for Multimodal Video Understanding

可能有帮助的文档

Data-Centric VLMs（主要是数据的选择、增强(recaption)，比如重写caption）

• ⚡ 💭 Improved Baselines with Visual Instruction Tuning
• ⚡ 💭 Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
• ⚡ 💭 ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• ⚡ 💭 Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
• ⚡ 💭 MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
• ⚡ 💭 MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

见这些论文使用的数据，同时这三个论文的模型和数据量可以训练，其余模型训练不出来（Datasets）

Vision-Language Model Visual Information Extraction

1）主流的VLM普遍使用CLIP-ViT作为Vision Encoder，通过对比学习训练的视觉编码器无法提取细粒度的图像特征，单纯增加图像分辨率对输入窗口影响过大，因此不同VE的融合（比如DINO，SigLip等）与选择是必要的。

2）当前的高分辨率VLLM将图像编码为高分辨率的visual token，计算成本过大。如何选择特定的、问题相关的高分辨率视觉特征，设计灵活高效的注意力机制是必要的。

• ⚡ 📄+🔧 BRAVE: Broadening the visual encoding of vision-language models
• ⚡ 📄+🔧 Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
• ⚡ 📄+🔧 FlexAttention for Efficient High-Resolution Vision-Language Models
• ⚡ 📄+🔧 From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models

1. 不同vision encoder的视觉特征分析与组合

2. Question-aware的高分辨率视觉特征的提取与注入

Modality Alignment

1）在当前的X-to-T多模态理解模型中，通常使用简单的模块(如MLP)作为跨模态表示的桥梁，但这些方法往往因为简单的空间映射导致不同模态的表征之间产生偏差，从而影响模型的推理和生成。

2）设计更复杂且高效的跨模态对齐机制，尤其是在不同模态间的对齐机制上进行优化是必要的。

• ⚡ 📄+🔧 https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models?tab=readme-ov-file
• ⚡ 📄+🔧 https://arxiv.org/pdf/2408.05211

1. 创新性的训练范式

2. Modality Projector / Resampler的设计（拼接后输入或llm内部注入）

3. 跨模态间交互的可解释性

Data Centric LLM and VLM Algorithms

• ⚡ 💭 A Survey of Multimodal Large Language Model from A Data-centric Perspective

Data Processing

• ⚡ 💭 LLM-Enhanced Data Management
• ⚡ 💭 Scaling Laws for Data Filtering -- Data Curation cannot be Computed
• ⚡ 💭 LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Model
• ⚡ 💭 Improving CLIP Training with Language Rewrites

Data Evaluation

• ⚡ 💭 Awesome Data Evaluation

Data Generation

• ⚡ 💭 Comprehensive Exploration of Synthetic Data Generation: A Survey
• ⚡ 💭 On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey

Data Extraction(RAG)

• ⚡ 💭 Retrieval-Augmented Generation for AI-Generated Content: A Survey
• ⚡ 💭 GraphRAG
• ⚡ 💭 Raptor
• ⚡ 💭 QAEncoder: Towards Aligned Representation Learning in Question Answering System

Data Centric LLM and VLM Systems

1）Data Centric AI数据侧开发，主要关注Data Juicer，要求对于数据侧有比较详细的了解 2）Data Centric AI数据侧工具相关的科研 需要对于数据侧工具有一定的了解，阅读相关的论文 附录：数据评估相关论文

Image数据评估论文

• ⚡ 📄+🔧 CLIPScore: A Reference-free Evaluation Metric for Image Captioning

• ⚡ 📄+🔧 InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation

• ⚡ 📄+🔧 leverage large multimodal model

• ⚡ 📄+🔧 Video数据评估论文

• ⚡ 📄+🔧 EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching

• ⚡ 📄+🔧 Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation

Data-Centric Domain-Specific LLMs

MathLLMs

• ⚡ 📄+🔧 DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• ⚡ 📄+🔧 InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning
• ⚡ 📄+🔧 MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

Math Datasets：

• ⚡ 📄+🔧 DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• ⚡ 📄+🔧 InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

数学知识按照难度分为小学，初中，高中和大学。大学又有很多细分，比如数学分析，高等代数以及概率论与数理统计，最优化方法，数值分析等等目前观察到的现象是说，无论是否是Math的模型，在SFT之后高中和大学数学能力都会大幅度下降，说明数据集的构建不利于高中和大学数学能力的学习