Subscribe to the newsletter • Join the community
Welcome to our library of the best insights from the best papers on AI and machine learning. A new paper will be added every Friday.
Don't hesitate to open an issue and submit a paper that you found interesting and the 3 key takeaways.
- The paper introduces KOSMOS-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, follow instructions (i.e., zero-shot learning), and learn in context (i.e., few-shot learning).
- KOSMOS-1 was trained from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data, and achieved impressive performance on various tasks, such as language understanding, generation, perception-language tasks, and vision tasks.
- MLLMs benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. Additionally, the paper introduces a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.
Week #8: Big Little Transformer Decoder
- The paper presents a method to reduce the latency of autoregressive transformer models while preserving accuracy. The framework consists of a smaller model that is assisted in generation by a larger model.
- The framework is built around the observation that by correcting only the erroneous tokens of the smaller model using the prediction of the larger model, it is possible to preserve the accuracy of the latter. Based on this observation, a policy is created that decides when the smaller model needs help to generate.
- Only the smaller model is used as an autoregressive model, while the larger one is used to predict the tokens of the whole produced sequence, increasing the arithmetic intensity and thus reducing the overall latency of the generation.
- The paper presents a method to discover optimization algorithms for deep neural network training by formulating algorithm discovery as program search, leveraging efficient search techniques to explore an infinite and sparse program space.
- The method introduces program selection and simplification strategies to bridge the large generalization gap between proxy and target tasks.
- The discovered algorithm, Lion (EvoLved Sign Momentum), is more memory-efficient than Adam and achieves better performance on a variety of tasks, including image classification, vision-language contrastive learning, diffusion models, autoregressive, masked language modeling, and fine-tuning.
- The paper presents a neural network architecture called ControlNet that can control large image diffusion models (like Stable Diffusion) to learn task-specific input conditions.
- The ControlNet consists of a "trainable copy" and a "locked copy" of a large diffusion model, connected by a unique type of convolution layer called "zero convolution", which allows for end-to-end learning while preserving the generalization ability of the original model.
- ControlNet can be trained on small datasets (even <1k samples) and on personal devices, and can still achieve competitive results with models trained on large computation clusters with terabytes of GPU memory and thousands of GPU hours.
- The paper introduces Multimodal-CoT, a framework for incorporating vision signals in chain-of-thought (CoT) reasoning with 1B-models. The framework decouples rationale generation and answer inference into two stages, and incorporates vision features to help generate more effective rationales for more accurate answer inference.
- The authors compare Multimodal-CoT with GPT-3.5 on the ScienceQA benchmark and show that their approach surpasses GPT-3.5 by 16% accuracy. They also show that different vision features and backbone models affect the performance of the model, and that DETR (based on object detection) is the best performing vision feature.
- The authors perform a manual error analysis on randomly selected examples generated by Multimodal-CoT. They find that the majority of the errors are due to factual mistakes (failures in understanding maps and counting numbers in the images), commonsense mistakes (answering questions that require commonsense knowledge), and logical mistakes (contradictions in the reasoning chains).
Week #4: MusicLM: Generating Music From Text
- MusicLM is a text-conditioned generative model that can produce high-quality music. The model is trained on a synthetic dataset of audio pairs with matching melodies and different acoustics, as well as data pairs of people humming and singing. The text description is used as a conditioning signal to guide the music generation process.
- The model is able to generate music that follows the target melody contained in the input audio clip, while also being faithful to the text description. MusicLM is capable of generating long, coherent audio sequences that are semantically plausible and consistent with the text description. The model can also be used in "story mode," where the text description changes over time, leading to smooth transitions in the generated music.
- There are several risks associated with MusicLM and its use-case, such as the reflection of biases present in the training data and the potential misappropriation of creative content. The authors conducted a thorough study of memorization and found that only a small fraction of examples was memorized exactly.
- The ability of a Large Language Model (LLM) to produce a better quality response is closely related to the prompt used. For example, providing the model with an example or a chain of thoughts (CoT) as a prompt produces a higher quality response without training; this type of technique is usually called InContextLearning, since the model learns from the provided context instead of updating parameters.
- The chain-of-thought prompt is particularly useful in arithmetic reasoning, and it has been shown that for symbolic reasoning, the chain-of-thought prompt facilitates generalization of the out-of-distribution to longer sequences.
- A comparative analysis of the usefulness of chain reasoning versus model size was conducted, showing that larger models are better reasoners and can benefit more from the chain reasoning prompt than smaller models.
- Tuning and inference of LLMs are not trivial in terms of computational cost, so creating smaller models that can be used to solve specific tasks using LLMs as teachers can have several advantages.
- In this case, an LLM is used to produce chain reasoning that is then validated by comparing the final what answer from the LLM with that provided by dataset y. A new dataset {x, e, y} of explanations is then created from a smaller dataset {x, y} containing only question and answer; the new dataset of examples is used to train a smaller T5 3B model to produce the answer along with the chain of thought.
- The results show that the resulting model has comparable performance on the Common Sense Question Answering dataset of GPT3 using Zero-Shot-CoT.
- The Holistic Evaluation of Language Models (HELM) is a toolkit designed to improve the transparency of language models and better understand their capabilities, limitations, and risks.
- HELM uses a multi-metric approach to evaluate language models across a wide range of scenarios and metrics, including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.
- HELM conducts a large-scale evaluation of 30 prominent language models across 42 different scenarios, including 21 that have not previously been used in mainstream LM evaluation. The results of the evaluation and all raw model prompts and completions are made publicly available.