From 5d1cec8dab14aa55cde0217caa3c414997aefcbc Mon Sep 17 00:00:00 2001 From: Automated Date: Tue, 24 Dec 2024 09:08:30 +0000 Subject: [PATCH] Latest data: Tue Dec 24 09:08:30 UTC 2024 --- index.html | 594 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 352 insertions(+), 242 deletions(-) diff --git a/index.html b/index.html index cade7137..37cd33b4 100644 --- a/index.html +++ b/index.html @@ -19,7 +19,7 @@

Vincent's Arxiv FrontPage


-

Generated on 2024-12-23.


+

Generated on 2024-12-24.


This frontpage is made by scraping arxiv and by running a sentence-model that detects if the abstract describes a paper about a topic of interest. One cool feature: it all pretty much runs via Github Actions.


@@ -29,6 +29,182 @@

Vincent's Arx

New Datasets

+ +

2024-12-23

+ + +
+ + ANID: How Far Are We? Evaluating the Discrepancies Between AI-synthesized Images and Natural Images through Multimodal Guidance + +
+
+

+

In the rapidly evolving field of Artificial Intelligence Generated Content (AIGC), one of the key challenges is distinguishing AI-synthesized images from natural images.Despite the remarkable capabilities of advanced AI generative models in producing visually compelling images, significant discrepancies remain when these images are compared to natural ones.To systematically investigate and quantify these discrepancies, we introduce an AI-Natural Image Discrepancy Evaluation benchmark aimed at addressing the critical question: \textit{how far are AI-generated images (AIGIs) from truly realistic images?}We have constructed a large-scale multimodal dataset, the Distinguishing Natural and AI-generated Images (DNAI) dataset, which includes over 440,000 AIGI samples generated by 8 representative models using both unimodal and multimodal prompts, such as Text-to-Image (T2I), Image-to-Image (I2I), and Text \textit{vs.} 0.745Image-to-Image (TI2I).Our fine-grained assessment framework provides a comprehensive evaluation of the DNAI dataset across five key dimensions: naive visual feature quality, semantic alignment in multimodal generation, aesthetic appeal, downstream task applicability, and coordinated human validation.Extensive evaluation results highlight significant discrepancies across these dimensions, underscoring the necessity of aligning quantitative metrics with human judgment to achieve a holistic understanding of AI-generated image quality.Code is available at \href{https://github.com/ryliu68/ANID}{https://github.com/ryliu68/ANID}.

+

+

+ + link + +

+
+
+ + + +

2024-12-23

+ + +
+ + SCBench: A Sports Commentary Benchmark for Video LLMs + +
+
+

+

Recently, significant advances have been made in Video Large Language Models (Video LLMs) in both academia and industry.However, methods to evaluate and benchmark the performance of different Video LLMs, especially their fine-grained, temporal visual capabilities, remain very limited.On one hand, current benchmarks use relatively simple videos (e.g., subtitled movie clips) where the model can understand the entire video by processing just a few frames.On the other hand, their datasets lack diversity in task format, comprising only QA or multi-choice QA, which overlooks the models' capacity for generating in-depth and precise texts.Sports videos, which feature intricate visual information, sequential events, and emotionally charged commentary, present a critical challenge for Video LLMs, making sports commentary an ideal benchmarking task.Inspired by these challenges, we propose a novel task: sports video commentary generation, developed $\textbf{SCBench}$ for Video LLMs.To construct such a benchmark, we introduce (1) $\textbf{SCORES}$, a six-dimensional metric specifically designed for our task, upon which we propose a GPT-based evaluation method, and (2) $\textbf{CommentarySet}$, a dataset consisting of 5,775 annotated video clips and ground-truth labels tailored to our metric.Based on SCBench, we conduct comprehensive evaluations on multiple Video LLMs (e.g. VILA, Video-LLaVA, etc.) and chain-of-thought baseline methods.Our results found that InternVL-Chat-2 achieves the best performance with 5.44, surpassing the second-best by 1.04.Our work provides a fresh perspective for future research, aiming to enhance models' overall capabilities in complex visual understanding tasks.Our dataset will be released soon. 0.978

+

+

+ + link + +

+
+
+ + + +

2024-12-23

+ + +
+ + Chumor 2.0: Towards Benchmarking Chinese Humor Understanding + +
+
+

+

Existing humor datasets and evaluations predominantly focus on English, leaving limited resources for culturally nuanced humor in non-English languages like Chinese.To address this gap, we construct Chumor, the first Chinese humor explanation dataset that exceeds the size of existing humor datasets. 0.821Chumor is sourced from Ruo Zhi Ba, a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes.We test ten LLMs through direct and chain-of-thought prompting, revealing that Chumor poses significant challenges to existing LLMs, with their accuracy slightly above random and far below human.In addition, our analysis highlights that human-annotated humor explanations are significantly better than those generated by GPT-4o and ERNIE-4-turbo.We release Chumor at https://huggingface.co/datasets/dnaihao/Chumor, our project page is at https://dnaihao.github.io/Chumor-dataset/, our leaderboard is at https://huggingface.co/spaces/dnaihao/Chumor, and our codebase is at https://github.com/dnaihao/Chumor-dataset. 0.899

+

+

+ + link + +

+
+
+ + + +

2024-12-23

+ + +
+ + GauSim: Registering Elastic Objects into Digital World by Gaussian Simulator + +
+
+

+

In this work, we introduce GauSim, a novel neural network-based simulator designed to capture the dynamic behaviors of real-world elastic objects represented through Gaussian kernels.Unlike traditional methods that treat kernels as particles within particle-based simulations, we leverage continuum mechanics, modeling each kernel as a continuous piece of matter to account for realistic deformations without idealized assumptions.To improve computational efficiency and fidelity, we employ a hierarchical structure that organizes kernels into Center of Mass Systems (CMS) with explicit formulations, enabling a coarse-to-fine simulation approach.This structure significantly reduces computational overhead while preserving detailed dynamics.In addition, GauSim incorporates explicit physics constraints, such as mass and momentum conservation, ensuring interpretable results and robust, physically plausible simulations.To validate our approach, we present a new dataset, READY, containing multi-view videos of real-world elastic deformations. 0.705Experimental results demonstrate that GauSim achieves superior performance compared to existing physics-driven baselines, offering a practical and accurate solution for simulating complex dynamic behaviors.Code and model will be released.Project page: https://www.mmlab-ntu.com/project/gausim/index.html .

+

+

+ + link + +

+
+
+ + + +

2024-12-23

+ + +
+ + Cross-View Referring Multi-Object Tracking + +
+
+

+

Referring Multi-Object Tracking (RMOT) is an important topic in the current tracking field.Its task form is to guide the tracker to track objects that match the language description.Current research mainly focuses on referring multi-object tracking under single-view, which refers to a view sequence or multiple unrelated view sequences.However, in the single-view, some appearances of objects are easily invisible, resulting in incorrect matching of objects with the language description.In this work, we propose a new task, called Cross-view Referring Multi-Object Tracking (CRMOT).It introduces the cross-view to obtain the appearances of objects from multiple views, avoiding the problem of the invisible appearances of objects in RMOT task.CRMOT is a more challenging task of accurately tracking the objects that match the language description and maintaining the identity consistency of objects in each cross-view.To advance CRMOT task, we construct a cross-view referring multi-object tracking benchmark based on CAMPUS and DIVOTrack datasets, named CRTrack.Specifically, it provides 13 different scenes and 221 language descriptions.Furthermore, we propose an end-to-end cross-view referring multi-object tracking method, named CRTracker.Extensive experiments on the CRTrack benchmark verify the effectiveness of our method.The dataset and code are available at https://github.com/chen-si-jia/CRMOT. 0.852

+

+

+ + link + +

+
+
+ + + +

2024-12-23

+ + +
+ + Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoders + +
+
+

+

Recent 3D content generation pipelines commonly employ Variational Autoencoders (VAEs) to encode shapes into compact latent representations for diffusion-based generation.However, the widely adopted uniform point sampling strategy in Shape VAE training often leads to a significant loss of geometric details, limiting the quality of shape reconstruction and downstream generation tasks.We present Dora-VAE, a novel approach that enhances VAE reconstruction through our proposed sharp edge sampling strategy and a dual cross-attention mechanism.By identifying and prioritizing regions with high geometric complexity during training, our method significantly improves the preservation of fine-grained shape features.Such sampling strategy and the dual attention mechanism enable the VAE to focus on crucial geometric details that are typically missed by uniform sampling approaches.To systematically evaluate VAE reconstruction quality, we additionally propose Dora-bench, a benchmark that quantifies shape complexity through the density of sharp edges, introducing a new metric focused on reconstruction accuracy at these salient geometric features.Extensive experiments on the Dora-bench demonstrate that Dora-VAE achieves comparable reconstruction quality to the state-of-the-art dense XCube-VAE while requiring a latent space at least 8$\times$ smaller (1,280 vs. > 10,000 codes).We will release our code and benchmark dataset to facilitate future research in 3D shape modeling. 0.731

+

+

+ + link + +

+
+
+ + + +

2024-12-23

+ + +
+ + ChatGarment: Garment Estimation, Generation and Editing via Large Language Models + +
+
+

+

We introduce ChatGarment, a novel approach that leverages large vision-language models (VLMs) to automate the estimation, generation, and editing of 3D garments from images or text descriptions.Unlike previous methods that struggle in real-world scenarios or lack interactive editing capabilities, ChatGarment can estimate sewing patterns from in-the-wild images or sketches, generate them from text descriptions, and edit garments based on user instructions, all within an interactive dialogue.These sewing patterns can then be draped into 3D garments, which are easily animatable and simulatable.This is achieved by finetuning a VLM to directly generate a JSON file that includes both textual descriptions of garment types and styles, as well as continuous numerical attributes.This JSON file is then used to create sewing patterns through a programming parametric model.To support this, we refine the existing programming model, GarmentCode, by expanding its garment type coverage and simplifying its structure for efficient VLM fine-tuning.Additionally, we construct a large-scale dataset of image-to-sewing-pattern and text-to-sewing-pattern pairs through an automated data pipeline. 0.727Extensive evaluations demonstrate ChatGarment's ability to accurately reconstruct, generate, and edit garments from multimodal inputs, highlighting its potential to revolutionize workflows in fashion and gaming applications.Code and data will be available at https://chatgarment.github.io/. 0.812

+

+

+ + link + +

+
+
+ + + +

2024-12-23

+ + +
+ + FaceLift: Single Image to 3D Head with View Generation and GS-LRM + +
+
+

+

We present FaceLift, a feed-forward approach for rapid, high-quality, 360-degree head reconstruction from a single image.Our pipeline begins by employing a multi-view latent diffusion model that generates consistent side and back views of the head from a single facial input.These generated views then serve as input to a GS-LRM reconstructor, which produces a comprehensive 3D representation using Gaussian splats.To train our system, we develop a dataset of multi-view renderings using synthetic 3D human head as-sets. 0.757The diffusion-based multi-view generator is trained exclusively on synthetic head images, while the GS-LRM reconstructor undergoes initial training on Objaverse followed by fine-tuning on synthetic head data.FaceLift excels at preserving identity and maintaining view consistency across views.Despite being trained solely on synthetic data, FaceLift demonstrates remarkable generalization to real-world images.Through extensive qualitative and quantitative evaluations, we show that FaceLift outperforms state-of-the-art methods in 3D head reconstruction, highlighting its practical applicability and robust performance on real-world images.In addition to single image reconstruction, FaceLift supports video inputs for 4D novel view synthesis and seamlessly integrates with 2D reanimation techniques to enable 3D facial animation.Project page: https://weijielyu.github.io/FaceLift.

+

+

+ + link + +

+
+
+ +

2024-12-19

@@ -733,21 +909,26 @@

New Datasets

+ + +

Data Quality

+ + -

2024-12-12

+

2024-12-17

- OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages + Label Errors in the Tobacco3482 Dataset

-

We present OpenNER 1.0, a standardized collection of openly available named entity recognition (NER) datasets.OpenNER contains 34 datasets spanning 51 languages, annotated in varying named entity ontologies. 0.838We correct annotation format issues, standardize the original datasets into a uniform representation, map entity type names to be more consistent across corpora, and provide the collection in a structure that enables research in multilingual and multi-ontology NER. 0.731We provide baseline models using three pretrained multilingual language models to compare the performance of recent models and facilitate future research in NER.

+

Tobacco3482 is a widely used document classification benchmark dataset.However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. 0.665We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. 0.812We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. 0.747Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/.

- + link

@@ -756,20 +937,20 @@

New Datasets

-

2024-12-12

+

2024-12-16

- Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion + RepFace: Refining Closed-Set Noise with Progressive Label Correction for Face Recognition

-

Recovering the geometry and materials of objects from a single image is challenging due to its under-constrained nature.In this paper, we present Neural LightRig, a novel framework that boosts intrinsic estimation by leveraging auxiliary multi-lighting conditions from 2D diffusion priors.Specifically, 1) we first leverage illumination priors from large-scale diffusion models to build our multi-light diffusion model on a synthetic relighting dataset with dedicated designs.This diffusion model generates multiple consistent images, each illuminated by point light sources in different directions.2) By using these varied lighting images to reduce estimation uncertainty, we train a large G-buffer model with a U-Net backbone to accurately predict surface normals and materials.Extensive experiments validate that our approach significantly outperforms state-of-the-art methods, enabling accurate surface normal and PBR material estimation with vivid relighting effects.Code and dataset are available on our project page at https://projects.zxhezexin.com/neural-lightrig. 0.726

+

Face recognition has made remarkable strides, driven by the expanding scale of datasets, advancements in various backbone and discriminative losses.However, face recognition performance is heavily affected by the label noise, especially closed-set noise.While numerous studies have focused on handling label noise, addressing closed-set noise still poses challenges. 0.769This paper identifies this challenge as training isn't robust to noise at the early-stage training, and necessitating an appropriate learning strategy for samples with low confidence, which are often misclassified as closed-set noise in later training phases.To address these issues, we propose a new framework to stabilize the training at early stages and split the samples into clean, ambiguous and noisy groups which are devised with separate training strategies.Initially, we employ generated auxiliary closed-set noisy samples to enable the model to identify noisy data at the early stages of training.Subsequently, we introduce how samples are split into clean, ambiguous and noisy groups by their similarity to the positive and nearest negative centers.Then we perform label fusion for ambiguous samples by incorporating accumulated model predictions. 0.602Finally, we apply label smoothing within the closed set, adjusting the label to a point between the nearest negative class and the initially assigned label. 0.624Extensive experiments validate the effectiveness of our method on mainstream face datasets, achieving state-of-the-art results.The code will be released upon acceptance.

- + link

@@ -778,20 +959,20 @@

New Datasets

-

2024-12-12

+

2024-12-11

- RatBodyFormer: Rodent Body Surface from Keypoints + CAT: Class Aware Adaptive Thresholding for Semi-Supervised Domain Generalization

-

Rat behavior modeling goes to the heart of many scientific studies, yet the textureless body surface evades automatic analysis as it literally has no keypoints that detectors can find.The movement of the body surface, however, is a rich source of information for deciphering the rat behavior.We introduce two key contributions to automatically recover densely 3D sampled rat body surface points, passively.The first is RatDome, a novel multi-camera system for rat behavior capture, and a large-scale dataset captured with it that consists of pairs of 3D keypoints and 3D body surface points. 0.744The second is RatBodyFormer, a novel network to transform detected keypoints to 3D body surface points.RatBodyFormer is agnostic to the exact locations of the 3D body surface points in the training data and is trained with masked-learning.We experimentally validate our framework with a number of real-world experiments.Our results collectively serve as a novel foundation for automated rat behavior analysis and will likely have far-reaching implications for biomedical and neuroscientific research.

+

Domain Generalization (DG) seeks to transfer knowledge from multiple source domains to unseen target domains, even in the presence of domain shifts.Achieving effective generalization typically requires a large and diverse set of labeled source data to learn robust representations that can generalize to new, unseen domains.However, obtaining such high-quality labeled data is often costly and labor-intensive, limiting the practical applicability of DG.To address this, we investigate a more practical and challenging problem: semi-supervised domain generalization (SSDG) under a label-efficient paradigm.In this paper, we propose a novel method, CAT, which leverages semi-supervised learning with limited labeled data to achieve competitive generalization performance under domain shifts.Our method addresses key limitations of previous approaches, such as reliance on fixed thresholds and sensitivity to noisy pseudo-labels. 0.602CAT combines adaptive thresholding with noisy label refinement techniques, creating a straightforward yet highly effective solution for SSDG tasks.Specifically, our approach uses flexible thresholding to generate high-quality pseudo-labels with higher class diversity while refining noisy pseudo-labels to improve their reliability. 0.679Extensive experiments across multiple benchmark datasets demonstrate the superior performance of our method, highlighting its effectiveness in achieving robust generalization under domain shift.

- + link

@@ -800,20 +981,20 @@

New Datasets

-

2024-12-12

+

2024-12-10

- Hidden Biases of End-to-End Driving Datasets + Defending Against Neural Network Model Inversion Attacks via Data Poisoning

-

End-to-end driving systems have made rapid progress, but have so far not been applied to the challenging new CARLA Leaderboard 2.0.Further, while there is a large body of literature on end-to-end architectures and training strategies, the impact of the training dataset is often overlooked.In this work, we make a first attempt at end-to-end driving for Leaderboard 2.0.Instead of investigating architectures, we systematically analyze the training dataset, leading to new insights: (1) Expert style significantly affects downstream policy performance.(2) In complex data sets, the frames should not be weighted on the basis of simplistic criteria such as class frequencies.(3) Instead, estimating whether a frame changes the target labels compared to previous frames can reduce the size of the dataset without removing important information.By incorporating these findings, our model ranks first and second respectively on the map and sensors tracks of the 2024 CARLA Challenge, and sets a new state-of-the-art on the Bench2Drive test routes.Finally, we uncover a design flaw in the current evaluation metrics and propose a modification for future challenges.Our dataset, code, and pre-trained models are publicly available at https://github.com/autonomousvision/carla_garage. 0.716

+

Model inversion attacks pose a significant privacy threat to machine learning models by reconstructing sensitive data from their outputs.While various defenses have been proposed to counteract these attacks, they often come at the cost of the classifier's utility, thus creating a challenging trade-off between privacy protection and model utility.Moreover, most existing defenses require retraining the classifier for enhanced robustness, which is impractical for large-scale, well-established models.This paper introduces a novel defense mechanism to better balance privacy and utility, particularly against adversaries who employ a machine learning model (i.e., inversion model) to reconstruct private data.Drawing inspiration from data poisoning attacks, which can compromise the performance of machine learning models, we propose a strategy that leverages data poisoning to contaminate the training data of inversion models, thereby preventing model inversion attacks. Two defense methods are presented.The first, termed label-preserving poisoning attacks for all output vectors (LPA), involves subtle perturbations to all output vectors while preserving their labels.Our findings demonstrate that these minor perturbations, introduced through a data poisoning approach, significantly increase the difficulty of data reconstruction without compromising the utility of the classifier. 0.602Subsequently, we introduce a second method, label-flipping poisoning for partial output vectors (LFP), which selectively perturbs a small subset of output vectors and alters their labels during the process.Empirical results indicate that LPA is notably effective, outperforming the current state-of-the-art defenses.Our data poisoning-based defense provides a new retraining-free defense paradigm that preserves the victim classifier's utility.

- + link

@@ -822,20 +1003,20 @@

New Datasets

-

2024-12-12

+

2024-12-03

- Do Multimodal Large Language Models See Like Humans? + Class-wise Autoencoders Measure Classification Difficulty And Detect Label Mistakes

-

Multimodal Large Language Models (MLLMs) have achieved impressive results on various vision tasks, leveraging recent advancements in large language models.However, a critical question remains unaddressed: do MLLMs perceive visual information similarly to humans?Current benchmarks lack the ability to evaluate MLLMs from this perspective.To address this challenge, we introduce HVSBench, a large-scale benchmark designed to assess the alignment between MLLMs and the human visual system (HVS) on fundamental vision tasks that mirror human vision.HVSBench curated over 85K multimodal samples, spanning 13 categories and 5 fields in HVS, including Prominence, Subitizing, Prioritizing, Free-Viewing, and Searching. 0.744Extensive experiments demonstrate the effectiveness of our benchmark in providing a comprehensive evaluation of MLLMs.Specifically, we evaluate 13 MLLMs, revealing that even the best models show significant room for improvement, with most achieving only moderate results.Our experiments reveal that HVSBench presents a new and significant challenge for cutting-edge MLLMs.We believe that HVSBench will facilitate research on human-aligned and explainable MLLMs, marking a key step in understanding how MLLMs perceive and process visual information.

+

We introduce a new framework for analyzing classification datasets based on the ratios of reconstruction errors between autoencoders trained on individual classes.This analysis framework enables efficient characterization of datasets on the sample, class, and entire dataset levels.We define reconstruction error ratios (RERs) that probe classification difficulty and allow its decomposition into (1) finite sample size and (2) Bayes error and decision-boundary complexity.Through systematic study across 19 popular visual datasets, we find that our RER-based dataset difficulty probe strongly correlates with error rate for state-of-the-art (SOTA) classification models.By interpreting sample-level classification difficulty as a label mistakenness score, we further find that RERs achieve SOTA performance on mislabel detection tasks on hard datasets under symmetric and asymmetric label noise. 0.664Our code is publicly available at https://github.com/voxel51/reconstruction-error-ratios.

- + link

@@ -843,21 +1024,26 @@

New Datasets

+ + +

Benchmarks

+ + -

2024-12-12

+

2024-12-23

- Learning Camera Movement Control from Real-World Drone Videos + Can Stability be Detrimental? Better Generalization through Gradient Descent Instabilities

-

This study seeks to automate camera movement control for filming existing subjects into attractive videos, contrasting with the creation of non-existent content by directly generating the pixels.We select drone videos as our test case due to their rich and challenging motion patterns, distinctive viewing angles, and precise controls.Existing AI videography methods struggle with limited appearance diversity in simulation training, high costs of recording expert operations, and difficulties in designing heuristic-based goals to cover all scenarios.To avoid these issues, we propose a scalable method that involves collecting real-world training data to improve diversity, extracting camera trajectories automatically to minimize annotation costs, and training an effective architecture that does not rely on heuristics.Specifically, we collect 99k high-quality trajectories by running 3D reconstruction on online videos, connecting camera poses from consecutive frames to formulate 3D camera paths, and using Kalman filter to identify and remove low-quality data.Moreover, we introduce DVGFormer, an auto-regressive transformer that leverages the camera path and images from all past frames to predict camera movement in the next frame.We evaluate our system across 38 synthetic natural scenes and 7 real city 3D scans.We show that our system effectively learns to perform challenging camera movements such as navigating through obstacles, maintaining low altitude to increase perceived speed, and orbiting towers and buildings, which are very useful for recording high-quality videos.Data and code are available at dvgformer.github.io. 0.714

+

Traditional analyses of gradient descent optimization show that, when the largest eigenvalue of the loss Hessian - often referred to as the sharpness - is below a critical learning-rate threshold, then training is 'stable' and training loss decreases monotonically.Recent studies, however, have suggested that the majority of modern deep neural networks achieve good performance despite operating outside this stable regime.In this work, we demonstrate that such instabilities, induced by large learning rates, move model parameters toward flatter regions of the loss landscape.Our crucial insight lies in noting that, during these instabilities, the orientation of the Hessian eigenvectors rotate.This, we conjecture, allows the model to explore regions of the loss landscape that display more desirable geometrical properties for generalization, such as flatness.These rotations are a consequence of network depth, and we prove that for any network with depth > 1, unstable growth in parameters cause rotations in the principal components of the Hessian, which promote exploration of the parameter space away from unstable directions.Our empirical studies reveal an implicit regularization effect in gradient descent with large learning rates operating beyond the stability threshold.We find these lead to excellent generalization performance on modern benchmark datasets. 0.669

- + link

@@ -866,20 +1052,20 @@

New Datasets

-

2024-12-12

+

2024-12-23

- OmniDrag: Enabling Motion Control for Omnidirectional Image-to-Video Generation + Graph Neural Networks Are Evolutionary Algorithms

-

As virtual reality gains popularity, the demand for controllable creation of immersive and dynamic omnidirectional videos (ODVs) is increasing.While previous text-to-ODV generation methods achieve impressive results, they struggle with content inaccuracies and inconsistencies due to reliance solely on textual inputs.Although recent motion control techniques provide fine-grained control for video generation, directly applying these methods to ODVs often results in spatial distortion and unsatisfactory performance, especially with complex spherical motions.To tackle these challenges, we propose OmniDrag, the first approach enabling both scene- and object-level motion control for accurate, high-quality omnidirectional image-to-video generation.Building on pretrained video diffusion models, we introduce an omnidirectional control module, which is jointly fine-tuned with temporal attention layers to effectively handle complex spherical motion.In addition, we develop a novel spherical motion estimator that accurately extracts motion-control signals and allows users to perform drag-style ODV generation by simply drawing handle and target points.We also present a new dataset, named Move360, addressing the scarcity of ODV data with large scene and object motions. 0.834Experiments demonstrate the significant superiority of OmniDrag in achieving holistic scene-level and fine-grained object-level control for ODV generation.The project page is available at https://lwq20020127.github.io/OmniDrag.

+

In this paper, we reveal the intrinsic duality between graph neural networks (GNNs) and evolutionary algorithms (EAs), bridging two traditionally distinct fields.Building on this insight, we propose Graph Neural Evolution (GNE), a novel evolutionary algorithm that models individuals as nodes in a graph and leverages designed frequency-domain filters to balance global exploration and local exploitation.Through the use of these filters, GNE aggregates high-frequency (diversity-enhancing) and low-frequency (stability-promoting) information, transforming EAs into interpretable and tunable mechanisms in the frequency domain.Extensive experiments on benchmark functions demonstrate that GNE consistently outperforms state-of-the-art algorithms such as GA, DE, CMA-ES, SDAES, and RL-SHADE, excelling in complex landscapes, optimal solution shifts, and noisy environments.Its robustness, adaptability, and superior convergence highlight its practical and theoretical value. 0.639Beyond optimization, GNE establishes a conceptual and mathematical foundation linking EAs and GNNs, offering new perspectives for both fields.Its framework encourages the development of task-adaptive filters and hybrid approaches for EAs, while its insights can inspire advances in GNNs, such as improved global information propagation and mitigation of oversmoothing.GNE's versatility extends to solving challenges in machine learning, including hyperparameter tuning and neural architecture search, as well as real-world applications in engineering and operations research.By uniting the dynamics of EAs with the structural insights of GNNs, this work provides a foundation for interdisciplinary innovation, paving the way for scalable and interpretable solutions to complex optimization problems.

- + link

@@ -887,26 +1073,21 @@

New Datasets

- - -

Data Quality

- - -

2024-12-17

+

2024-12-23

- Label Errors in the Tobacco3482 Dataset + SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMAC

-

Tobacco3482 is a widely used document classification benchmark dataset.However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. 0.665We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. 0.812We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. 0.747Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/.

+

The availability of challenging simulation environments is pivotal for advancing the field of Multi-Agent Reinforcement Learning (MARL).In cooperative MARL settings, the StarCraft Multi-Agent Challenge (SMAC) has gained prominence as a benchmark for algorithms following centralized training with decentralized execution paradigm.However, with continual advancements in SMAC, many algorithms now exhibit near-optimal performance, complicating the evaluation of their true effectiveness. 0.658To alleviate this problem, in this work, we highlight a critical issue: the default opponent policy in these environments lacks sufficient diversity, leading MARL algorithms to overfit and exploit unintended vulnerabilities rather than learning robust strategies.To overcome these limitations, we propose SMAC-HARD, a novel benchmark designed to enhance training robustness and evaluation comprehensiveness.SMAC-HARD supports customizable opponent strategies, randomization of adversarial policies, and interfaces for MARL self-play, enabling agents to generalize to varying opponent behaviors and improve model stability.Furthermore, we introduce a black-box testing framework wherein agents are trained without exposure to the edited opponent scripts but are tested against these scripts to evaluate the policy coverage and adaptability of MARL algorithms.We conduct extensive evaluations of widely used and state-of-the-art algorithms on SMAC-HARD, revealing the substantial challenges posed by edited and mixed strategy opponents.Additionally, the black-box strategy tests illustrate the difficulty of transferring learned policies to unseen adversaries.We envision SMAC-HARD as a critical step toward benchmarking the next generation of MARL algorithms, fostering progress in self-play methods for multi-agent systems.Our code is available at https://github.com/devindeng94/smac-hard.

- + link

@@ -915,20 +1096,20 @@

Data Quality

-

2024-12-16

+

2024-12-23

- RepFace: Refining Closed-Set Noise with Progressive Label Correction for Face Recognition + Knowledge Editing through Chain-of-Thought

-

Face recognition has made remarkable strides, driven by the expanding scale of datasets, advancements in various backbone and discriminative losses.However, face recognition performance is heavily affected by the label noise, especially closed-set noise.While numerous studies have focused on handling label noise, addressing closed-set noise still poses challenges. 0.769This paper identifies this challenge as training isn't robust to noise at the early-stage training, and necessitating an appropriate learning strategy for samples with low confidence, which are often misclassified as closed-set noise in later training phases.To address these issues, we propose a new framework to stabilize the training at early stages and split the samples into clean, ambiguous and noisy groups which are devised with separate training strategies.Initially, we employ generated auxiliary closed-set noisy samples to enable the model to identify noisy data at the early stages of training.Subsequently, we introduce how samples are split into clean, ambiguous and noisy groups by their similarity to the positive and nearest negative centers.Then we perform label fusion for ambiguous samples by incorporating accumulated model predictions. 0.602Finally, we apply label smoothing within the closed set, adjusting the label to a point between the nearest negative class and the initially assigned label. 0.624Extensive experiments validate the effectiveness of our method on mainstream face datasets, achieving state-of-the-art results.The code will be released upon acceptance.

+

Large Language Models (LLMs) have demonstrated exceptional capabilities across a wide range of natural language processing (NLP) tasks.However, keeping these models up-to-date with evolving world knowledge remains a significant challenge due to the high costs of frequent retraining.To address this challenge, knowledge editing techniques have emerged to update LLMs with new information without rebuilding the model from scratch.Among these, the in-context editing paradigm stands out for its effectiveness in integrating new knowledge while preserving the model's original capabilities.Despite its potential, existing in-context knowledge editing methods are often task-specific, focusing primarily on multi-hop QA tasks using structured knowledge triples.Moreover, their reliance on few-shot prompting for task decomposition makes them unstable and less effective in generalizing across diverse tasks. In response to these limitations, we propose EditCoT, a novel knowledge editing framework that flexibly and efficiently updates LLMs across various tasks without retraining.EditCoT works by generating a chain-of-thought (CoT) for a given input and then iteratively refining this CoT process using a CoT editor based on updated knowledge.We evaluate EditCoT across a diverse range of benchmarks, covering multiple languages and tasks.The results demonstrate that our approach achieves state-of-the-art performance while offering superior generalization, effectiveness, and stability compared to existing methods, marking a significant advancement in the field of knowledge updating. 0.636Code and data are available at: https://github.com/bebr2/EditCoT.

- + link

@@ -937,20 +1118,20 @@

Data Quality

-

2024-12-11

+

2024-12-23

- CAT: Class Aware Adaptive Thresholding for Semi-Supervised Domain Generalization + RepoTransBench: A Real-World Benchmark for Repository-Level Code Translation

-

Domain Generalization (DG) seeks to transfer knowledge from multiple source domains to unseen target domains, even in the presence of domain shifts.Achieving effective generalization typically requires a large and diverse set of labeled source data to learn robust representations that can generalize to new, unseen domains.However, obtaining such high-quality labeled data is often costly and labor-intensive, limiting the practical applicability of DG.To address this, we investigate a more practical and challenging problem: semi-supervised domain generalization (SSDG) under a label-efficient paradigm.In this paper, we propose a novel method, CAT, which leverages semi-supervised learning with limited labeled data to achieve competitive generalization performance under domain shifts.Our method addresses key limitations of previous approaches, such as reliance on fixed thresholds and sensitivity to noisy pseudo-labels. 0.602CAT combines adaptive thresholding with noisy label refinement techniques, creating a straightforward yet highly effective solution for SSDG tasks.Specifically, our approach uses flexible thresholding to generate high-quality pseudo-labels with higher class diversity while refining noisy pseudo-labels to improve their reliability. 0.679Extensive experiments across multiple benchmark datasets demonstrate the superior performance of our method, highlighting its effectiveness in achieving robust generalization under domain shift.

+

Repository-level code translation refers to translating an entire code repository from one programming language to another while preserving the functionality of the source repository.Many benchmarks have been proposed to evaluate the performance of such code translators. 0.612However, previous benchmarks mostly provide fine-grained samples, focusing at either code snippet, function, or file-level code translation. 0.685Such benchmarks do not accurately reflect real-world demands, where entire repositories often need to be translated, involving longer code length and more complex functionalities.To address this gap, we propose a new benchmark, named RepoTransBench, which is a real-world repository-level code translation benchmark with an automatically executable test suite.We conduct experiments on RepoTransBench to evaluate the translation performance of 11 advanced LLMs.We find that the Success@1 score (test success in one attempt) of the best-performing LLM is only 7.33%.To further explore the potential of LLMs for repository-level code translation, we provide LLMs with error-related feedback to perform iterative debugging and observe an average 7.09% improvement on Success@1.However, even with this improvement, the Success@1 score of the best-performing LLM is only 21%, which may not meet the need for reliable automatic repository-level code translation.Finally, we conduct a detailed error analysis and highlight current LLMs' deficiencies in repository-level code translation, which could provide a reference for further improvements.

- + link

@@ -959,20 +1140,20 @@

Data Quality

-

2024-12-10

+

2024-12-23

- Defending Against Neural Network Model Inversion Attacks via Data Poisoning + Group Testing with General Correlation Using Hypergraphs

-

Model inversion attacks pose a significant privacy threat to machine learning models by reconstructing sensitive data from their outputs.While various defenses have been proposed to counteract these attacks, they often come at the cost of the classifier's utility, thus creating a challenging trade-off between privacy protection and model utility.Moreover, most existing defenses require retraining the classifier for enhanced robustness, which is impractical for large-scale, well-established models.This paper introduces a novel defense mechanism to better balance privacy and utility, particularly against adversaries who employ a machine learning model (i.e., inversion model) to reconstruct private data.Drawing inspiration from data poisoning attacks, which can compromise the performance of machine learning models, we propose a strategy that leverages data poisoning to contaminate the training data of inversion models, thereby preventing model inversion attacks. Two defense methods are presented.The first, termed label-preserving poisoning attacks for all output vectors (LPA), involves subtle perturbations to all output vectors while preserving their labels.Our findings demonstrate that these minor perturbations, introduced through a data poisoning approach, significantly increase the difficulty of data reconstruction without compromising the utility of the classifier. 0.602Subsequently, we introduce a second method, label-flipping poisoning for partial output vectors (LFP), which selectively perturbs a small subset of output vectors and alters their labels during the process.Empirical results indicate that LPA is notably effective, outperforming the current state-of-the-art defenses.Our data poisoning-based defense provides a new retraining-free defense paradigm that preserves the victim classifier's utility.

+

Group testing, a problem with diverse applications across multiple disciplines, traditionally assumes independence across nodes' states.Recent research, however, focuses on real-world scenarios that often involve correlations among nodes, challenging the simplifying assumptions made in existing models.In this work, we consider a comprehensive model for arbitrary statistical correlation among nodes' states.To capture and leverage these correlations effectively, we model the problem by hypergraphs, inspired by [GLS22], augmented by a probability mass function on the hyper-edges. Using this model, we first design a novel greedy adaptive algorithm capable of conducting informative tests and dynamically updating the distribution.Performance analysis provides upper bounds on the number of tests required, which depend solely on the entropy of the underlying probability distribution and the average number of infections. 0.625We demonstrate that the algorithm recovers or improves upon all previously known results for group testing settings with correlation.Additionally, we provide families of graphs where the algorithm is order-wise optimal and give examples where the algorithm or its analysis is not tight.We then generalize the proposed framework of group testing with general correlation in two directions, namely noisy group testing and semi-non-adaptive group testing.In both settings, we provide novel theoretical bounds on the number of tests required.

- + link

@@ -981,20 +1162,20 @@

Data Quality

-

2024-12-03

+

2024-12-23

- Class-wise Autoencoders Measure Classification Difficulty And Detect Label Mistakes + In Case You Missed It: ARC 'Challenge' Is Not That Challenging

-

We introduce a new framework for analyzing classification datasets based on the ratios of reconstruction errors between autoencoders trained on individual classes.This analysis framework enables efficient characterization of datasets on the sample, class, and entire dataset levels.We define reconstruction error ratios (RERs) that probe classification difficulty and allow its decomposition into (1) finite sample size and (2) Bayes error and decision-boundary complexity.Through systematic study across 19 popular visual datasets, we find that our RER-based dataset difficulty probe strongly correlates with error rate for state-of-the-art (SOTA) classification models.By interpreting sample-level classification difficulty as a label mistakenness score, we further find that RERs achieve SOTA performance on mislabel detection tasks on hard datasets under symmetric and asymmetric label noise. 0.664Our code is publicly available at https://github.com/voxel51/reconstruction-error-ratios.

+

ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity.Although some researchers have quietly shifted to a more appropriate scheme over the last year, the implications of this change have yet to be widely acknowledged.We highlight this overlooked shift, show how similar evaluation practices falsely imply reasoning deficits in other benchmarks, and demonstrate that fairer methods dramatically reduce performance gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). 0.605In doing so, we reveal how evaluation shapes perceived difficulty and offer guidelines to ensure that multiple-choice evaluations accurately reflect actual model capabilities.

- + link

@@ -1002,11 +1183,50 @@

Data Quality

- + +

2024-12-23

+ -

Benchmarks

+
+ + Large Motion Video Autoencoding with Cross-modal Video VAE + +
+
+

+

Learning a robust video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation.Directly applying image VAEs to individual frames in isolation can result in temporal inconsistencies and suboptimal compression rates due to a lack of temporal compression.Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance.In this paper, we present a novel and powerful video autoencoder capable of high-fidelity video encoding.First, we observe that entangling spatial and temporal compression by merely extending the image VAE to a 3D VAE can introduce motion blur and detail distortion artifacts.Thus, we propose temporal-aware spatial compression to better encode and decode the spatial information.Additionally, we integrate a lightweight motion compression model for further temporal compression.Second, we propose to leverage the textual information inherent in text-to-video datasets and incorporate text guidance into our model.This significantly enhances reconstruction quality, particularly in terms of detail preservation and temporal stability.Third, we further improve the versatility of our model through joint training on both images and videos, which not only enhances reconstruction quality but also enables the model to perform both image and video autoencoding.Extensive evaluations against strong recent baselines demonstrate the superior performance of our method. 0.825The project website can be found at~\href{https://yzxing87.github.io/vae/}{https://yzxing87.github.io/vae/}.

+

+

+ + link + +

+
+
+ + + +

2024-12-23

- + +
+ + Cross-View Referring Multi-Object Tracking + +
+
+

+

Referring Multi-Object Tracking (RMOT) is an important topic in the current tracking field.Its task form is to guide the tracker to track objects that match the language description.Current research mainly focuses on referring multi-object tracking under single-view, which refers to a view sequence or multiple unrelated view sequences.However, in the single-view, some appearances of objects are easily invisible, resulting in incorrect matching of objects with the language description.In this work, we propose a new task, called Cross-view Referring Multi-Object Tracking (CRMOT).It introduces the cross-view to obtain the appearances of objects from multiple views, avoiding the problem of the invisible appearances of objects in RMOT task.CRMOT is a more challenging task of accurately tracking the objects that match the language description and maintaining the identity consistency of objects in each cross-view.To advance CRMOT task, we construct a cross-view referring multi-object tracking benchmark based on CAMPUS and DIVOTrack datasets, named CRTrack.Specifically, it provides 13 different scenes and 221 language descriptions.Furthermore, we propose an end-to-end cross-view referring multi-object tracking method, named CRTracker.Extensive experiments on the CRTrack benchmark verify the effectiveness of our method. 0.788The dataset and code are available at https://github.com/chen-si-jia/CRMOT.

+

+

+ + link + +

+
+
+ +

2024-12-19

@@ -1402,28 +1622,6 @@

Benchmarks

- - -

2024-12-17

- - -
- - Boosting Test Performance with Importance Sampling--a Subpopulation Perspective - -
-
-

-

Despite empirical risk minimization (ERM) is widely applied in the machine learning community, its performance is limited on data with spurious correlation or subpopulation that is introduced by hidden attributes.Existing literature proposed techniques to maximize group-balanced or worst-group accuracy when such correlation presents, yet, at the cost of lower average accuracy. 0.658In addition, many existing works conduct surveys on different subpopulation methods without revealing the inherent connection between these methods, which could hinder the technology advancement in this area.In this paper, we identify important sampling as a simple yet powerful tool for solving the subpopulation problem.On the theory side, we provide a new systematic formulation of the subpopulation problem and explicitly identify the assumptions that are not clearly stated in the existing works.This helps to uncover the cause of the dropped average accuracy.We provide the first theoretical discussion on the connections of existing methods, revealing the core components that make them different.On the application side, we demonstrate a single estimator is enough to solve the subpopulation problem.In particular, we introduce the estimator in both attribute-known and -unknown scenarios in the subpopulation setup, offering flexibility in practical use cases.And empirically, we achieve state-of-the-art performance on commonly used benchmark datasets. 0.704

-

-

- - link - -

-
-
-

2024-12-17

@@ -1733,109 +1931,26 @@

Benchmarks

- -

2024-12-16

- - -
- - Witty: An Efficient Solver for Computing Minimum-Size Decision Trees - -
-
-

-

Decision trees are a classic model for summarizing and classifying data.To enhance interpretability and generalization properties, it has been proposed to favor small decision trees.Accordingly, in the minimum-size decision tree training problem (MSDT), the input is a set of training examples in $\mathbb{R}^d$ with class labels and we aim to find a decision tree that classifies all training examples correctly and has a minimum number of nodes.MSDT is NP-hard and therefore presumably not solvable in polynomial time.Nevertheless, Komusiewicz et al.[ICML '23] developed a promising algorithmic paradigm called witness trees which solves MSDT efficiently if the solution tree is small.In this work, we test this paradigm empirically.We provide an implementation, augment it with extensive heuristic improvements, and scrutinize it on standard benchmark instances. 0.695The augmentations achieve a mean 324-fold (median 84-fold) speedup over the naive implementation.Compared to the state of the art they achieve a mean 32-fold (median 7-fold) speedup over the dynamic programming based MurTree solver[Demirovi\'c et al., J. Mach.Learn.Res. '22] and a mean 61-fold (median 25-fold) speedup over SAT-based implementations [Janota and Morgado, SAT '20]. 0.663As a theoretical result we obtain an improved worst-case running-time bound for MSDT.

-

-

- - link - -

-
-
- - - -

2024-12-16

- - -
- - Memory-Reduced Meta-Learning with Guaranteed Convergence - -
-
-

-

The optimization-based meta-learning approach is gaining increased traction because of its unique ability to quickly adapt to a new task using only small amounts of data.However, existing optimization-based meta-learning approaches, such as MAML, ANIL and their variants, generally employ backpropagation for upper-level gradient estimation, which requires using historical lower-level parameters/gradients and thus increases computational and memory overhead in each iteration.In this paper, we propose a meta-learning algorithm that can avoid using historical parameters/gradients and significantly reduce memory costs in each iteration compared to existing optimization-based meta-learning approaches.In addition to memory reduction, we prove that our proposed algorithm converges sublinearly with the iteration number of upper-level optimization, and the convergence error decays sublinearly with the batch size of sampled tasks. 0.622In the specific case in terms of deterministic meta-learning, we also prove that our proposed algorithm converges to an exact solution.Moreover, we quantify that the computational complexity of the algorithm is on the order of $\mathcal{O}(\epsilon^{-1})$, which matches existing convergence results on meta-learning even without using any historical parameters/gradients.Experimental results on meta-learning benchmarks confirm the efficacy of our proposed algorithm. 0.682

-

-

- - link - -

-
-
- - - -

2024-12-16

- + -
- - Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentation - -
-
-

-

Domain Generalized Semantic Segmentation (DGSS) seeks to utilize source domain data exclusively to enhance the generalization of semantic segmentation across unknown target domains.Prevailing studies predominantly concentrate on feature normalization and domain randomization, these approaches exhibit significant limitations.Feature normalization-based methods tend to confuse semantic features in the process of constraining the feature space distribution, resulting in classification misjudgment.Domain randomization-based methods frequently incorporate domain-irrelevant noise due to the uncontrollability of style transformations, resulting in segmentation ambiguity.To address these challenges, we introduce a novel framework, named SCSD for Semantic Consistency prediction and Style Diversity generalization.It comprises three pivotal components: Firstly, a Semantic Query Booster is designed to enhance the semantic awareness and discrimination capabilities of object queries in the mask decoder, enabling cross-domain semantic consistency prediction.Secondly, we develop a Text-Driven Style Transform module that utilizes domain difference text embeddings to controllably guide the style transformation of image features, thereby increasing inter-domain style diversity.Lastly, to prevent the collapse of similar domain feature spaces, we introduce a Style Synergy Optimization mechanism that fortifies the separation of inter-domain features and the aggregation of intra-domain features by synergistically weighting style contrastive loss and style aggregation loss.Extensive experiments demonstrate that the proposed SCSD significantly outperforms existing state-of-theart methods. 0.605Notably, SCSD trained on GTAV achieved an average of 49.11 mIoU on the four unseen domain datasets, surpassing the previous state-of-the-art method by +4.08 mIoU. Code is available at https://github.com/nhw649/SCSD.

-

-

- - link - -

-
-
- - - -

2024-12-16

+

LLMs

- -
- - IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations - -
-
-

-

Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics.Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs, while still struggling with inherent ambiguities between lighting and material.On the other hand, learning-based approaches leverage rich material priors from existing 3D object datasets but face challenges with maintaining multi-view consistency.In this paper, we introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations.Our method achieves accurate and multi-view consistent estimation on surface normals and material properties.This is made possible through a novel cross-view, cross-domain attention module and an illumination-augmented, view-adaptive training strategy.Additionally, we introduce ARB-Objaverse, a new dataset that provides large-scale multi-view intrinsic data and renderings under diverse lighting conditions, supporting robust training.Extensive experiments demonstrate that IDArb outperforms state-of-the-art methods both qualitatively and quantitatively. 0.602Moreover, our approach facilitates a range of downstream tasks, including single-image relighting, photometric stereo, and 3D reconstruction, highlighting its broad applications in realistic 3D content creation.

-

-

- - link - -

-
-
- - + -

2024-12-16

+

2024-12-23

- No More Tuning: Prioritized Multi-Task Learning with Lagrangian Differential Multiplier Methods + Emerging Security Challenges of Large Language Models

-

Given the ubiquity of multi-task in practical systems, Multi-Task Learning (MTL) has found widespread application across diverse domains.In real-world scenarios, these tasks often have different priorities.For instance, In web search, relevance is often prioritized over other metrics, such as click-through rates or user engagement.Existing frameworks pay insufficient attention to the prioritization among different tasks, which typically adjust task-specific loss function weights to differentiate task priorities.However, this approach encounters challenges as the number of tasks grows, leading to exponential increases in hyper-parameter tuning complexity.Furthermore, the simultaneous optimization of multiple objectives can negatively impact the performance of high-priority tasks due to interference from lower-priority tasks. In this paper, we introduce a novel multi-task learning framework employing Lagrangian Differential Multiplier Methods for step-wise multi-task optimization.It is designed to boost the performance of high-priority tasks without interference from other tasks.Its primary advantage lies in its ability to automatically optimize multiple objectives without requiring balancing hyper-parameters for different tasks, thereby eliminating the need for manual tuning.Additionally, we provide theoretical analysis demonstrating that our method ensures optimization guarantees, enhancing the reliability of the process. 0.621We demonstrate its effectiveness through experiments on multiple public datasets and its application in Taobao search, a large-scale industrial search ranking system, resulting in significant improvements across various business metrics.

+

Large language models (LLMs) have achieved record adoption in a short period of time across many different sectors including high importance areas such as education [4] and healthcare 0.688[23].LLMs are open-ended models trained on diverse data without being tailored for specific downstream tasks, enabling broad applicability across various domains. 0.673They are commonly used for text generation, but also widely used to assist with code generation [3], and even analysis of security information, as Microsoft Security Copilot demonstrates [18].Traditional Machine Learning (ML) models are vulnerable to adversarial attacks [9].So the concerns on the potential security implications of such wide scale adoption of LLMs have led to the creation of this working group on the security of LLMs. 0.775During the Dagstuhl seminar on "Network Attack Detection and Defense - AI-Powered Threats and Responses", the working group discussions focused on the vulnerability of LLMs to adversarial attacks, rather than their potential use in generating malware or enabling cyberattacks. 0.699Although we note the potential threat represented by the latter, the role of the LLMs in such uses is mostly as an accelerator for development, similar to what it is in benign use. 0.78To make the analysis more specific, the working group employed ChatGPT as a concrete example of an LLM and addressed the following points, which also form the structure of this report: 1. How do LLMs differ in vulnerabilities from traditional ML models? 0.7262.What are the attack objectives in LLMs? 0.7273. How complex it is to assess the risks posed by the vulnerabilities of LLMs? 0.7434.What is the supply chain in LLMs, how data flow in and out of systems and what are the security implications? 0.752We conclude with an overview of open challenges and outlook.

- + link

@@ -1843,26 +1958,21 @@

Benchmarks

- - -

LLMs

- - -

2024-12-19

+

2024-12-23

- Nano-ESG: Extracting Corporate Sustainability Information from News Articles + Dynamic safety cases for frontier AI

-

Determining the sustainability impact of companies is a highly complex subject which has garnered more and more attention over the past few years.Today, investors largely rely on sustainability-ratings from established rating-providers in order to analyze how responsibly a company acts.However, those ratings have recently been criticized for being hard to understand and nearly impossible to reproduce. An independent way to find out about the sustainability practices of companies lies in the rich landscape of news article data.In this paper, we explore a different approach to identify key opportunities and challenges of companies in the sustainability domain.We present a novel dataset of more than 840,000 news articles which were gathered for major German companies between January 2023 and September 2024.By applying a mixture of Natural Language Processing techniques, we first identify relevant articles, before summarizing them and extracting their sustainability-related sentiment and aspect using Large Language Models (LLMs).Furthermore, we conduct an evaluation of the obtained data and determine that the LLM-produced answers are accurate. 0.636We release both datasets at https://github.com/Bailefan/Nano-ESG.

+

Frontier artificial intelligence (AI) systems present both benefits and risks to society.Safety cases - structured arguments supported by evidence - are one way to help ensure the safe development and deployment of these systems.Yet the evolving nature of AI capabilities, as well as changes in the operational environment and understanding of risk, necessitates mechanisms for continuously updating these safety cases.Typically, in other sectors, safety cases are produced pre-deployment and do not require frequent updates post-deployment, which can be a manual, costly process.This paper proposes a Dynamic Safety Case Management System (DSCMS) to support both the initial creation of a safety case and its systematic, semi-automated revision over time. 0.637Drawing on methods developed in the autonomous vehicles (AV) sector - state-of-the-art Checkable Safety Arguments (CSA) combined with Safety Performance Indicators (SPIs) recommended by UL 4600, a DSCMS helps developers maintain alignment between system safety claims and the latest system state.We demonstrate this approach on a safety case template for offensive cyber capabilities and suggest ways it can be integrated into governance structures for safety-critical decision-making.While the correctness of the initial safety argument remains paramount - particularly for high-severity risks - a DSCMS provides a framework for adapting to new insights and strengthening incident response. 0.613We outline challenges and further work towards development and implementation of this approach as part of continuous safety assurance of frontier AI systems.

- + link

@@ -1871,20 +1981,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Review-Then-Refine: A Dynamic Framework for Multi-Hop Question Answering with Temporal Adaptability + Tracking the Feature Dynamics in LLM Training: A Mechanistic Study

-

Retrieve-augmented generation (RAG) frameworks have emerged as a promising solution to multi-hop question answering(QA) tasks since it enables large language models (LLMs) to incorporate external knowledge and mitigate their inherent knowledge deficiencies.Despite this progress, existing RAG frameworks, which usually follows the retrieve-then-read paradigm, often struggle with multi-hop QA with temporal information since it has difficulty retrieving and synthesizing accurate time-related information.To address the challenge, this paper proposes a novel framework called review-then-refine, which aims to enhance LLM performance in multi-hop QA scenarios with temporal information. 0.669Our approach begins with a review phase, where decomposed sub-queries are dynamically rewritten with temporal information, allowing for subsequent adaptive retrieval and reasoning process.In addition, we implement adaptive retrieval mechanism to minimize unnecessary retrievals, thus reducing the potential for hallucinations.In the subsequent refine phase, the LLM synthesizes the retrieved information from each sub-query along with its internal knowledge to formulate a coherent answer. 0.642Extensive experimental results across multiple datasets demonstrate the effectiveness of our proposed framework, highlighting its potential to significantly improve multi-hop QA capabilities in LLMs. 0.707

+

Understanding training dynamics and feature evolution is crucial for the mechanistic interpretability of large language models (LLMs).Although sparse autoencoders (SAEs) have been used to identify features within LLMs, a clear picture of how these features evolve during training remains elusive.In this study, we: (1) introduce SAE-Track, a method to efficiently obtain a continual series of SAEs; (2) formulate the process of feature formation and conduct a mechanistic analysis; and (3) analyze and visualize feature drift during training.Our work provides new insights into the dynamics of features in LLMs, enhancing our understanding of training mechanisms and feature evolution. 0.693

- + link

@@ -1893,20 +2003,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search + SCBench: A Sports Commentary Benchmark for Video LLMs

-

In the realm of Text-Based Person Search (TBPS), mainstream methods aim to explore more efficient interaction frameworks between text descriptions and visual data.However, recent approaches encounter two principal challenges.Firstly, the widely used random-based Masked Language Modeling (MLM) considers all the words in the text equally during training.However, massive semantically vacuous words ('with', 'the', etc.) be masked fail to contribute efficient interaction in the cross-modal MLM and hampers the representation alignment.Secondly, manual descriptions in TBPS datasets are tedious and inevitably contain several inaccuracies.To address these issues, we introduce an Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM)Modeling and Text Enrichment Module (TEM). 0.603AGM dynamically masks semantically meaningful words by aggregating the attention weight derived from the text encoding process, thereby cross-modal MLM can capture information related to the masked word from text context and images and align their representations.Meanwhile, TEM alleviates low-quality representations caused by repetitive and erroneous text descriptions by replacing those semantically meaningful words with MLM's prediction.It not only enriches text descriptions but also prevents overfitting.Extensive experiments across three challenging benchmarks demonstrate the effectiveness of our AGA, achieving new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.

+

Recently, significant advances have been made in Video Large Language Models (Video LLMs) in both academia and industry. 0.629However, methods to evaluate and benchmark the performance of different Video LLMs, especially their fine-grained, temporal visual capabilities, remain very limited. 0.683On one hand, current benchmarks use relatively simple videos (e.g., subtitled movie clips) where the model can understand the entire video by processing just a few frames.On the other hand, their datasets lack diversity in task format, comprising only QA or multi-choice QA, which overlooks the models' capacity for generating in-depth and precise texts.Sports videos, which feature intricate visual information, sequential events, and emotionally charged commentary, present a critical challenge for Video LLMs, making sports commentary an ideal benchmarking task. 0.657Inspired by these challenges, we propose a novel task: sports video commentary generation, developed $\textbf{SCBench}$ for Video LLMs. 0.611To construct such a benchmark, we introduce (1) $\textbf{SCORES}$, a six-dimensional metric specifically designed for our task, upon which we propose a GPT-based evaluation method, and (2) $\textbf{CommentarySet}$, a dataset consisting of 5,775 annotated video clips and ground-truth labels tailored to our metric.Based on SCBench, we conduct comprehensive evaluations on multiple Video LLMs (e.g. VILA, Video-LLaVA, etc.) and chain-of-thought baseline methods. 0.632Our results found that InternVL-Chat-2 achieves the best performance with 5.44, surpassing the second-best by 1.04.Our work provides a fresh perspective for future research, aiming to enhance models' overall capabilities in complex visual understanding tasks.Our dataset will be released soon.

- + link

@@ -1915,20 +2025,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture + Detecting anxiety and depression in dialogues: a multi-label and explainable approach

-

Large language models (LLMs) demonstrate an impressive ability to utilise information within the context of their input sequences to appropriately respond to data unseen by the LLM during its training procedure. 0.684This ability is known as in-context learning (ICL).Humans and non-human animals demonstrate similar abilities, however their neural architectures differ substantially from LLMs. 0.705Despite this, a critical component within LLMs, the attention mechanism, resembles modern associative memory models, widely used in and influenced by the computational neuroscience community to model biological memory systems. 0.68Using this connection, we introduce an associative memory model capable of performing ICL.We use this as inspiration for a novel residual stream architecture which allows information to directly flow between attention heads.We test this architecture during training within a two-layer Transformer and show its ICL abilities manifest more quickly than without this modification.We then apply our architecture in small language models with 8 million parameters, focusing on attention head values, with results also indicating improved ICL performance at this larger and more naturalistic scale.

+

Anxiety and depression are the most common mental health issues worldwide, affecting a non-negligible part of the population.Accordingly, stakeholders, including governments' health systems, are developing new strategies to promote early detection and prevention from a holistic perspective (i.e., addressing several disorders simultaneously).In this work, an entirely novel system for the multi-label classification of anxiety and depression is proposed.The input data consists of dialogues from user interactions with an assistant chatbot.Another relevant contribution lies in using Large Language Models (LLMs) for feature extraction, provided the complexity and variability of language.The combination of LLMs, given their high capability for language understanding, and Machine Learning (ML) models, provided their contextual knowledge about the classification problem thanks to the labeled data, constitute a promising approach towards mental health assessment. 0.629To promote the solution's trustworthiness, reliability, and accountability, explainability descriptions of the model's decision are provided in a graphical dashboard.Experimental results on a real dataset attain 90 % accuracy, improving those in the prior literature.The ultimate objective is to contribute in an accessible and scalable way before formal treatment occurs in the healthcare systems.

- + link

@@ -1937,20 +2047,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Qwen2.5 Technical Report + Generating Completions for Fragmented Broca's Aphasic Sentences Using Large Language Models

-

In this report, we introduce Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. 0.621Compared to previous iterations, Qwen 2.5 has been significantly improved during both the pre-training and post-training stages.In terms of pre-training, we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens.This provides a strong foundation for common sense, expert knowledge, and reasoning capabilities.In terms of post-training, we implement intricate supervised finetuning with over 1 million samples, as well as multistage reinforcement learning.Post-training techniques enhance human preference, and notably improve long text generation, structural data analysis, and instruction following.To handle diverse and varied use cases effectively, we present Qwen2.5 LLM series in rich sizes. 0.689Open-weight offerings include base and instruction-tuned models, with quantized versions available.In addition, for hosted solutions, the proprietary models currently include two mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both available from Alibaba Cloud Model Studio.Qwen2.5 has demonstrated top-tier performance on a wide range of benchmarks evaluating language understanding, reasoning, mathematics, coding, human preference alignment, etc.Specifically, the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and proprietary models and demonstrates competitive performance to the state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5 times larger.Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness while performing competitively against GPT-4o-mini and GPT-4o respectively.Additionally, as the foundation, Qwen2.5 models have been instrumental in training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and multimodal models.

+

Broca's aphasia is a type of aphasia characterized by non-fluent, effortful and fragmented speech production with relatively good comprehension.Since traditional aphasia treatment methods are often time-consuming, labour-intensive, and do not reflect real-world conversations, applying natural language processing based approaches such as Large Language Models (LLMs) could potentially contribute to improving existing treatment approaches. 0.65To address this issue, we explore the use of sequence-to-sequence LLMs for completing fragmented Broca's aphasic sentences.We first generate synthetic Broca's aphasic data using a rule-based system designed to mirror the linguistic characteristics of Broca's aphasic speech.Using this synthetic data, we then fine-tune four pre-trained LLMs on the task of completing fragmented sentences. 0.629We evaluate our fine-tuned models on both synthetic and authentic Broca's aphasic data.We demonstrate LLMs' capability for reconstructing fragmented sentences, with the models showing improved performance with longer input utterances. 0.661Our result highlights the LLMs' potential in advancing communication aids for individuals with Broca's aphasia and possibly other clinical populations. 0.678

- + link

@@ -1959,20 +2069,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Adaptive Pruning for Large Language Models with Structural Importance Awareness + Large Language Model Safety: A Holistic Survey

-

The recent advancements in large language models (LLMs) have significantly improved language understanding and generation capabilities. 0.692However, it is difficult to deploy LLMs on resource-constrained edge devices due to their high computational and storage resource demands. 0.744To address this issue, we propose a novel LLM model pruning method, namely structurally-aware adaptive pruning (SAAP), to significantly reduce the computational and memory costs while maintaining model performance. 0.63We first define an adaptive importance fusion metric to evaluate the importance of all coupled structures in LLMs by considering their homoscedastic uncertainty. 0.645Then, we rank the importance of all modules to determine the specific layers that should be pruned to meet particular performance requirements.Furthermore, we develop a new group fine-tuning strategy to improve the inference efficiency of LLMs. 0.704Finally, we evaluate the proposed SAAP method on multiple LLMs across two common tasks, i.e., zero-shot classification and text generation. 0.678Experimental results show that our SAAP method outperforms several state-of-the-art baseline methods, achieving 2.17%, 2.37%, and 2.39% accuracy gains on LLaMA-7B, Vicuna-7B, andLLaMA-13B. Additionally, SAAP improves the token generation speed by 5%, showcasing its practical advantages in resource-constrained scenarios. 0.634

+

The rapid development and deployment of large language models (LLMs) have introduced a new frontier in artificial intelligence, marked by unprecedented capabilities in natural language understanding and generation. 0.687However, the increasing integration of these models into critical applications raises substantial safety concerns, necessitating a thorough examination of their potential risks and associated mitigation strategies. This survey provides a comprehensive overview of the current landscape of LLM safety, covering four major categories: value misalignment, robustness to adversarial attacks, misuse, and autonomous AI risks. 0.69In addition to the comprehensive review of the mitigation methodologies and evaluation resources on these four aspects, we further explore four topics related to LLM safety: the safety implications of LLM agents, the role of interpretability in enhancing LLM safety, the technology roadmaps proposed and abided by a list of AI companies and institutes for LLM safety, and AI governance aimed at LLM safety with discussions on international cooperation, policy proposals, and prospective regulatory directions. 0.71Our findings underscore the necessity for a proactive, multifaceted approach to LLM safety, emphasizing the integration of technical solutions, ethical considerations, and robust governance frameworks. 0.719This survey is intended to serve as a foundational resource for academy researchers, industry practitioners, and policymakers, offering insights into the challenges and opportunities associated with the safe integration of LLMs into society. 0.778Ultimately, it seeks to contribute to the safe and beneficial development of LLMs, aligning with the overarching goal of harnessing AI for societal advancement and well-being. 0.722A curated list of related papers has been publicly available at https://github.com/tjunlp-lab/Awesome-LLM-Safety-Papers. 0.698

- + link

@@ -1981,20 +2091,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Relaxed exception semantics for Arm-A (extended version) + RAGONITE: Iterative Retrieval on Induced Databases and Verbalized RDF for Conversational QA over KGs with RAG

-

To manage exceptions, software relies on a key architectural guarantee, precision: that exceptions appear to execute between instructions.However, this definition, dating back over 60 years, fundamentally assumes a sequential programmers model.Modern architectures such as Arm-A with programmer-observable relaxed behaviour make such a naive definition inadequate, and it is unclear exactly what guarantees programmers have on exception entry and exit. In this paper, we clarify the concepts needed to discuss exceptions in the relaxed-memory setting -- a key aspect of precisely specifying the architectural interface between hardware and software.We explore the basic relaxed behaviour across exception boundaries, and the semantics of external aborts, using Arm-A as a representative modern architecture. 0.601We identify an important problem, present yet unexplored for decades: pinning down what it means for exceptions to be precise in a relaxed setting.We describe key phenomena that any definition should account for.We develop an axiomatic model for Arm-A precise exceptions, tooling for axiomatic model execution, and a library of tests.Finally we explore the relaxed semantics of software-generated interrupts, as used in sophisticated programming patterns, and sketch how they too could be modelled.

+

Conversational question answering (ConvQA) is a convenient means of searching over RDF knowledge graphs (KGs), where a prevalent approach is to translate natural language questions to SPARQL queries.However, SPARQL has certain shortcomings: (i) it is brittle for complex intents and conversational questions, and (ii) it is not suitable for more abstract needs. 0.604Instead, we propose a novel two-pronged system where we fuse: (i) SQL-query results over a database automatically derived from the KG, and (ii) text-search results over verbalizations of KG facts.Our pipeline supports iterative retrieval: when the results of any branch are found to be unsatisfactory, the system can automatically opt for further rounds.We put everything together in a retrieval augmented generation (RAG) setup, where an LLM generates a coherent response from accumulated search results. 0.677We demonstrate the superiority of our proposed system over several baselines on a knowledge graph of BMW automobiles.

- + link

@@ -2003,20 +2113,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Language Models as Continuous Self-Evolving Data Engineers + Knowledge Editing through Chain-of-Thought

-

Large Language Models (LLMs) have demonstrated remarkable capabilities on various tasks, while the further evolvement is limited to the lack of high-quality training data. 0.686In addition, traditional training approaches rely too much on expert-labeled data, setting an upper limit on the performance of LLMs. 0.674To address this issue, we propose a novel paradigm that enables LLMs to train itself by autonomously generating, cleaning, reviewing, and annotating data with preference information, named LANCE. 0.737Our approach demonstrates that LLMs can serve as continuous self-evolving data engineers, significantly reducing the time and cost of the post-training data construction process. 0.708Through iterative fine-tuning on different variants of the Qwen2, we validate the effectiveness of LANCE across various tasks, showing that it can continuously improve model performance and maintain high-quality data generation.Across eight benchmark dimensions, LANCE resulted in an average score enhancement of 3.36 for Qwen2-7B and 2.70 for Qwen2-7B-Instruct.This training paradigm with autonomous data construction not only reduces the reliance on human experts or external models but also ensures that the data aligns with human values and preferences, paving the way for the development of future superintelligent systems that can exceed human capabilities.

+

Large Language Models (LLMs) have demonstrated exceptional capabilities across a wide range of natural language processing (NLP) tasks. 0.668However, keeping these models up-to-date with evolving world knowledge remains a significant challenge due to the high costs of frequent retraining.To address this challenge, knowledge editing techniques have emerged to update LLMs with new information without rebuilding the model from scratch. 0.711Among these, the in-context editing paradigm stands out for its effectiveness in integrating new knowledge while preserving the model's original capabilities.Despite its potential, existing in-context knowledge editing methods are often task-specific, focusing primarily on multi-hop QA tasks using structured knowledge triples.Moreover, their reliance on few-shot prompting for task decomposition makes them unstable and less effective in generalizing across diverse tasks. In response to these limitations, we propose EditCoT, a novel knowledge editing framework that flexibly and efficiently updates LLMs across various tasks without retraining. 0.681EditCoT works by generating a chain-of-thought (CoT) for a given input and then iteratively refining this CoT process using a CoT editor based on updated knowledge.We evaluate EditCoT across a diverse range of benchmarks, covering multiple languages and tasks.The results demonstrate that our approach achieves state-of-the-art performance while offering superior generalization, effectiveness, and stability compared to existing methods, marking a significant advancement in the field of knowledge updating.Code and data are available at: https://github.com/bebr2/EditCoT.

- + link

@@ -2025,20 +2135,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Measuring DNA Microswimmer Locomotion in Complex Flow Environments + Chumor 2.0: Towards Benchmarking Chinese Humor Understanding

-

Microswimmers are sub-millimeter swimming microrobots that show potential as a platform for controllable locomotion in applications including targeted cargo delivery and minimally invasive surgery. 0.605To be viable for these target applications, microswimmers will eventually need to be able to navigate in environments with dynamic fluid flows and forces. 0.634Experimental studies with microswimmers towards this goal are currently rare because of the difficulty isolating intentional microswimmer motion from environment-induced motion. 0.609In this work, we present a method for measuring microswimmer locomotion within a complex flow environment using fiducial microspheres.By tracking the particle motion of ferromagnetic and non-magnetic polystyrene fiducial microspheres, we capture the effect of fluid flow and field gradients on microswimmer trajectories.We then determine the field-driven translation of these microswimmers relative to fluid flow and demonstrate the effectiveness of this method by illustrating the motion of multiple microswimmers through different flows.

+

Existing humor datasets and evaluations predominantly focus on English, leaving limited resources for culturally nuanced humor in non-English languages like Chinese.To address this gap, we construct Chumor, the first Chinese humor explanation dataset that exceeds the size of existing humor datasets.Chumor is sourced from Ruo Zhi Ba, a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes.We test ten LLMs through direct and chain-of-thought prompting, revealing that Chumor poses significant challenges to existing LLMs, with their accuracy slightly above random and far below human. 0.744In addition, our analysis highlights that human-annotated humor explanations are significantly better than those generated by GPT-4o and ERNIE-4-turbo.We release Chumor at https://huggingface.co/datasets/dnaihao/Chumor, our project page is at https://dnaihao.github.io/Chumor-dataset/, our leaderboard is at https://huggingface.co/spaces/dnaihao/Chumor, and our codebase is at https://github.com/dnaihao/Chumor-dataset.

- + link

@@ -2047,20 +2157,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM + Reasoning to Attend: Try to Understand How Token Works

-

Text-to-video models have made remarkable advancements through optimization on high-quality text-video pairs, where the textual prompts play a pivotal role in determining quality of output videos.However, achieving the desired output often entails multiple revisions and iterative inference to refine user-provided prompts.Current automatic methods for refining prompts encounter challenges such as Modality-Inconsistency, Cost-Discrepancy, and Model-Unaware when applied to text-to-video diffusion models.To address these problem, we introduce an LLM-based prompt adaptation framework, termed as Prompt-A-Video, which excels in crafting Video-Centric, Labor-Free and Preference-Aligned prompts tailored to specific video diffusion model.Our approach involves a meticulously crafted two-stage optimization and alignment system.Initially, we conduct a reward-guided prompt evolution pipeline to automatically create optimal prompts pool and leverage them for supervised fine-tuning (SFT) of the LLM. 0.632Then multi-dimensional rewards are employed to generate pairwise data for the SFT model, followed by the direct preference optimization (DPO) algorithm to further facilitate preference alignment.Through extensive experimentation and comparative analyses, we validate the effectiveness of Prompt-A-Video across diverse generation models, highlighting its potential to push the boundaries of video generation.

+

Current Large Multimodal Models (LMMs) empowered visual grounding typically rely on $\texttt{}$ token as a text prompt to jointly optimize the vision-language model (e.g., LLaVA) and the downstream task-specified model (\eg, SAM).However, we observe that little research has looked into how it works.In this work, we first visualize the similarity maps, which are obtained by computing the semantic similarity between the $\texttt{}$ token and the image token embeddings derived from the last hidden layer in both the LLaVA encoder and SAM decoder.Intriguingly, we have found that a striking consistency holds in terms of activation responses in the similarity map,which reveals that what $\texttt{}$ token contributes to is the semantic similarity within image-text pairs.Specifically, $\texttt{}$ token, a placeholder expanded in text vocabulary, extensively queries among individual tokenized image patches to match the semantics of an object from text to the paired image while the Large Language Models (LLMs) are being fine-tuned. 0.602Upon the above findings, we present READ, which facilitates LMMs' resilient $\textbf{REA}$soning capability of where to atten$\textbf{D}$ under the guidance of highly activated points borrowed from similarity maps.Remarkably, READ features an intuitive design, Similarity as Points module (SasP), which can be seamlessly applied to $\texttt{}$-like paradigms in a plug-and-play fashion.Also, extensive experiments have been conducted on the ReasonSeg and RefCOCO(+/g) datasets.To validate whether READ suffers from catastrophic forgetting of previous skills after fine-tuning, we further assess its generation ability on an augmented FP-RefCOCO(+/g) dataset.All codes and models are publicly available at https://github.com/rui-qian/READ.

- + link

@@ -2069,20 +2179,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Rethinking Uncertainty Estimation in Natural Language Generation + YuLan-Mini: An Open Data-efficient Language Model

-

Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generated text. 0.687To this end, reliable uncertainty estimation is essential.Since current LLMs generate text autoregressively through a stochastic process, the same prompt can lead to varying outputs. 0.724Consequently, leading uncertainty estimation methods generate and analyze multiple output sequences to determine the LLM's uncertainty.However, generating output sequences is computationally expensive, making these methods impractical at scale.In this work, we inspect the theoretical foundations of the leading methods and explore new directions to enhance their computational efficiency.Building on the framework of proper scoring rules, we find that the negative log-likelihood of the most likely output sequence constitutes a theoretically grounded uncertainty measure.To approximate this alternative measure, we propose G-NLL, which has the advantage of being obtained using only a single output sequence generated by greedy decoding.This makes uncertainty estimation more efficient and straightforward, while preserving theoretical rigor.Empirical results demonstrate that G-NLL achieves state-of-the-art performance across various LLMs and tasks. 0.675Our work lays the foundation for efficient and reliable uncertainty estimation in natural language generation, challenging the necessity of more computationally involved methods currently leading the field.

+

Effective pre-training of large language models (LLMs) has been challenging due to the immense resource demands and the complexity of the technical processes involved. 0.67This paper presents a detailed technical report on YuLan-Mini, a highly capable base model with 2.42B parameters that achieves top-tier performance among models of similar parameter scale.Our pre-training approach focuses on enhancing training efficacy through three key technical contributions: an elaborate data pipeline combines data cleaning with data schedule strategies, a robust optimization method to mitigate training instability, and an effective annealing approach that incorporates targeted data selection and long context training.Remarkably, YuLan-Mini, trained on 1.08T tokens, achieves performance comparable to industry-leading models that require significantly more data.To facilitate reproduction, we release the full details of the data composition for each training phase.Project details can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.

- + link

@@ -2091,20 +2201,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying + RepoTransBench: A Real-World Benchmark for Repository-Level Code Translation

-

Studies have underscored how, regardless of the recent breakthrough and swift advances in AI research, even state-of-the-art Large Language models (LLMs) continue to struggle when performing logical and mathematical reasoning. 0.635The results seem to suggest that LLMs still work as (highly advanced) data pattern identifiers, scoring poorly when attempting to generalise and solve reasoning problems the models have never previously seen or that are not close to samples presented in their training data. 0.733To address this compelling concern, this paper makes use of the notion of critical questions from the literature on argumentation theory, focusing in particular on Toulmin's model of argumentation.We show that employing these critical questions can improve the reasoning capabilities of LLMs. 0.774By probing the rationale behind the models' reasoning process, the LLM can assess whether some logical mistake is occurring and correct it before providing the final reply to the user prompt. 0.721The underlying idea is drawn from the gold standard of any valid argumentative procedure: the conclusion is valid if it is entailed by accepted premises.Or, to paraphrase such Aristotelian principle in a real-world approximation, characterised by incomplete information and presumptive logic, the conclusion is valid if not proved otherwise.This approach successfully steers the models' output through a reasoning pipeline, resulting in better performance against the baseline and its Chain-of-Thought (CoT) implementation.To this end, an extensive evaluation of the proposed approach on the MT-Bench Reasoning and Math tasks across a range of LLMs is provided. 0.617

+

Repository-level code translation refers to translating an entire code repository from one programming language to another while preserving the functionality of the source repository.Many benchmarks have been proposed to evaluate the performance of such code translators.However, previous benchmarks mostly provide fine-grained samples, focusing at either code snippet, function, or file-level code translation.Such benchmarks do not accurately reflect real-world demands, where entire repositories often need to be translated, involving longer code length and more complex functionalities.To address this gap, we propose a new benchmark, named RepoTransBench, which is a real-world repository-level code translation benchmark with an automatically executable test suite.We conduct experiments on RepoTransBench to evaluate the translation performance of 11 advanced LLMs. 0.734We find that the Success@1 score (test success in one attempt) of the best-performing LLM is only 7.33%. 0.705To further explore the potential of LLMs for repository-level code translation, we provide LLMs with error-related feedback to perform iterative debugging and observe an average 7.09% improvement on Success@1. 0.731However, even with this improvement, the Success@1 score of the best-performing LLM is only 21%, which may not meet the need for reliable automatic repository-level code translation. 0.715Finally, we conduct a detailed error analysis and highlight current LLMs' deficiencies in repository-level code translation, which could provide a reference for further improvements. 0.734

- + link

@@ -2113,20 +2223,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages + Deliberation in Latent Space via Differentiable Cache Augmentation

-

Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants, yet they are often designed for general purpose programming tasks and perform poorly for more specialized domains such as high performance computing. 0.709Creating specialized models and tools for these domains is crucial towards gaining the benefits of LLMs in areas such as HPC. 0.723While previous work has explored HPC-specific models, LLMs still struggle to generate parallel code and it is not at all clear what hurdles are still holding back these LLMs and what must be done to overcome them. 0.734In this work, we conduct an in-depth study along the many axes of fine-tuning a specialized HPC LLM in order to better understand the challenges. 0.668Based on our findings we fine-tune and evaluate a specialized HPC LLM that is shown to be the best performing open-source code LLM for parallel code generation to date. 0.684

+

Techniques enabling large language models (LLMs) to "think more" by generating and attending to intermediate reasoning steps have shown promise in solving complex problems. 0.672However, the standard approaches generate sequences of discrete tokens immediately before responding, and so they can incur significant latency costs and be challenging to optimize.In this work, we demonstrate that a frozen LLM can be augmented with an offline coprocessor that operates on the model's key-value (kv) cache. 0.714This coprocessor augments the cache with a set of latent embeddings designed to improve the fidelity of subsequent decoding.We train this coprocessor using the language modeling loss from the decoder on standard pretraining data, while keeping the decoder itself frozen.This approach enables the model to learn, in an end-to-end differentiable fashion, how to distill additional computation into its kv-cache.Because the decoder remains unchanged, the coprocessor can operate offline and asynchronously, and the language model can function normally if the coprocessor is unavailable or if a given cache is deemed not to require extra computation.We show experimentally that when a cache is augmented, the decoder achieves lower perplexity on numerous subsequent tokens.Furthermore, even without any task-specific training, our experiments demonstrate that cache augmentation consistently reduces perplexity and improves performance across a range of reasoning-intensive tasks.

- + link

@@ -2135,20 +2245,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- LlamaFusion: Adapting Pretrained Language Models for Multimodal Generation + ADC: Enhancing Function Calling Via Adversarial Datasets and Code Line-Level Feedback

-

We present LlamaFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities, enabling them to understand and generate both text and images in arbitrary sequences. 0.604LlamaFusion leverages existing Llama-3's weights for processing texts autoregressively while introducing additional and parallel transformer modules for processing images with diffusion. 0.609During training, the data from each modality is routed to its dedicated modules: modality-specific feedforward layers, query-key-value projections, and normalization layers process each modality independently, while the shared self-attention layers allow interactions across text and image features.By freezing the text-specific modules and only training the image-specific modules, LlamaFusion preserves the language capabilities of text-only LLMs while developing strong visual understanding and generation abilities. 0.735Compared to methods that pretrain multimodal generative models from scratch, our experiments demonstrate that, LlamaFusion improves image understanding by 20% and image generation by 3.6% using only 50% of the FLOPs while maintaining Llama-3's language capabilities. 0.618We also demonstrate that this framework can adapt existing vision-language models with multimodal generation ability.Overall, this framework not only leverages existing computational investments in text-only LLMs but also enables the parallel development of language and vision capabilities, presenting a promising direction for efficient multimodal model development. 0.651

+

Large Language Models (LLMs) have made significant strides in Natural Language Processing and coding, yet they struggle with robustness and accuracy in complex function calls. 0.631To tackle these challenges, this paper introduces ADC, an innovative approach that enhances LLMs' ability to follow function formats and match complex parameters. 0.647ADC utilizes a high-quality code fine-tuning dataset with line-level execution feedback, providing granular process supervision that fosters strong logical reasoning and adherence to function formats.It also employs an adversarial dataset generation process to improve parameter matching.The staged training methodology capitalizes on both enriched code datasets and refined adversarial datasets, leading to marked improvements in function calling capabilities on the Berkeley Function-Calling Leaderboard (BFCL) Benchmark.The innovation of ADC lies in its strategic combination of process supervision, adversarial refinement, and incremental learning, setting a new standard for LLM proficiency in complex function calling. 0.608

- + link

@@ -2157,20 +2267,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- Face the Facts! Evaluating RAG-based Fact-checking Pipelines in Realistic Settings + In Case You Missed It: ARC 'Challenge' Is Not That Challenging

-

Natural Language Processing and Generation systems have recently shown the potential to complement and streamline the costly and time-consuming job of professional fact-checkers.In this work, we lift several constraints of current state-of-the-art pipelines for automated fact-checking based on the Retrieval-Augmented Generation (RAG) paradigm.Our goal is to benchmark, under more realistic scenarios, RAG-based methods for the generation of verdicts - i.e., short texts discussing the veracity of a claim - evaluating them on stylistically complex claims and heterogeneous, yet reliable, knowledge bases.Our findings show a complex landscape, where, for example, LLM-based retrievers outperform other retrieval techniques, though they still struggle with heterogeneous knowledge bases; larger models excel in verdict faithfulness, while smaller models provide better context adherence, with human evaluations favouring zero-shot and one-shot approaches for informativeness, and fine-tuned models for emotional alignment. 0.626

+

ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. 0.671Although some researchers have quietly shifted to a more appropriate scheme over the last year, the implications of this change have yet to be widely acknowledged.We highlight this overlooked shift, show how similar evaluation practices falsely imply reasoning deficits in other benchmarks, and demonstrate that fairer methods dramatically reduce performance gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA).In doing so, we reveal how evaluation shapes perceived difficulty and offer guidelines to ensure that multiple-choice evaluations accurately reflect actual model capabilities.

- + link

@@ -2179,20 +2289,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark + ResearchTown: Simulator of Human Research Community

-

Multiple-choice question (MCQ) datasets like Massive Multitask Language Understanding (MMLU) are widely used to evaluate the commonsense, understanding, and problem-solving abilities of large language models (LLMs). 0.611However, the open-source nature of these benchmarks and the broad sources of training data for LLMs have inevitably led to benchmark contamination, resulting in unreliable evaluation results. 0.642To alleviate this issue, we propose a contamination-free and more challenging MCQ benchmark called MMLU-CF.This benchmark reassesses LLMs' understanding of world knowledge by averting both unintentional and malicious data leakage. 0.669To avoid unintentional data leakage, we source data from a broader domain and design three decontamination rules.To prevent malicious data leakage, we divide the benchmark into validation and test sets with similar difficulty and subject distributions.The test set remains closed-source to ensure reliable results, while the validation set is publicly available to promote transparency and facilitate independent verification.Our evaluation of mainstream LLMs reveals that the powerful GPT-4o achieves merely a 5-shot score of 73.4% and a 0-shot score of 71.9% on the test set, which indicates the effectiveness of our approach in creating a more rigorous and contamination-free evaluation standard. 0.664The GitHub repository is available at https://github.com/microsoft/MMLU-CF and the dataset refers to https://huggingface.co/datasets/microsoft/MMLU-CF.

+

Large Language Models (LLMs) have demonstrated remarkable potential in scientific domains, yet a fundamental question remains unanswered: Can we simulate human research communities with LLMs? 0.673Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights.In this work, we propose ResearchTown, a multi-agent framework for research community simulation.Within this framework, the human research community is simplified and modeled as an agent-data graph, where researchers and papers are represented as agent-type and data-type nodes, respectively, and connected based on their collaboration relationships.We also introduce TextGNN, a text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as special forms of a unified message-passing process on the agent-data graph.To evaluate the quality of the research simulation, we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity.Our experiments reveal three key findings: (1) ResearchTown can provide a realistic simulation of collaborative research activities, including paper writing and review writing; (2) ResearchTown can maintain robust simulation with multiple researchers and diverse papers; (3) ResearchTown can generate interdisciplinary research ideas that potentially inspire novel research directions.

- + link

@@ -2201,20 +2311,20 @@

LLMs

-

2024-12-19

+

2024-12-23

- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks + Memory makes computation universal, remember?

-

This paper introduces LongBench v2, a benchmark designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks. 0.684LongBench v2 consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words, across six major task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured data understanding.To ensure the breadth and the practicality, we collect data from nearly 100 highly educated individuals with diverse professional backgrounds.We employ both automated and manual review processes to maintain high quality and difficulty, resulting in human experts achieving only 53.7% accuracy under a 15-minute time constraint.Our evaluation reveals that the best-performing model, when directly answers the questions, achieves only 50.1% accuracy.In contrast, the o1-preview model, which includes longer reasoning, achieves 57.7%, surpassing the human baseline by 4%.These results highlight the importance of enhanced reasoning ability and scaling inference-time compute to tackle the long-context challenges in LongBench v2.The project is available at https://longbench2.github.io.

+

Recent breakthroughs in AI capability have been attributed to increasingly sophisticated architectures and alignment techniques, but a simpler principle may explain these advances: memory makes computation universal.Memory enables universal computation through two fundamental capabilities: recursive state maintenance and reliable history access. 0.601We formally prove these requirements are both necessary and sufficient for universal computation.This principle manifests across scales, from cellular computation to neural networks to language models.Complex behavior emerges not from sophisticated processing units but from maintaining and accessing state across time.We demonstrate how parallel systems like neural networks achieve universal computation despite limitations in their basic units by maintaining state across iterations.This theoretical framework reveals a universal pattern: computational advances consistently emerge from enhanced abilities to maintain and access state rather than from more complex basic operations.Our analysis unifies understanding of computation across biological systems, artificial intelligence, and human cognition, reminding us that humanity's own computational capabilities have evolved in step with our technical ability to remember through oral traditions, writing, and now computing.

- + link