New submissions for Wed, 28 Jun 23

[e-tornike]

Keyword: contrastive

Contrastive Meta-Learning for Few-shot Node Classification

Authors: Song Wang, Zhen Tan, Huan Liu, Jundong Li
Arxiv: https://arxiv.org/abs/2306.15154
TLDR: Few-shot node classification, which aims to predict labels for nodes on graphs with only limited labeled nodes as references, is of great significance in real-world graph mining tasks. Particularly, in this paper, we refer to the task of classifying nodes in classes with a few labeled nodes. To tackle such a label shortage issue, existing works generally leverage the meta-learning framework, which utilizes a number of episodes to extract transferable knowledge from classes with abundant labeled nodes and generalizes the
Repo: None

FBA-Net: Foreground and Background Aware Contrastive Learning for Semi-Supervised Atrium Segmentation

Authors: Yunsung Chung, Chanho Lim, Chao Huang, Nassir Marrouche, Jihun Hamm
Arxiv: https://arxiv.org/abs/2306.15189
TLDR: Medical image segmentation of gadolinium enhancement magnetic resonance imaging (GE MRI) is an important task in clinical applications. However, manual annotation is time-consuming and requires specialized expertise. Semi-supervised segmentation methods that leverage both labeled and unlabeled data have shown promise, with contrastive learning emerging as a particularly effective approach. In this paper, we propose a contrastive training strategy of foreground and background representations for semi-supersupervised 3D medical image segmentsation (
Repo: None

Internal Contrastive Learning for Generalized Out-of-distribution Fault Diagnosis (GOOFD) Framework

Authors: Xingyue Wang, Hanrong Zhang, Ke Ma, Shuting Tao, Peng Peng, Hongwei Wang
Arxiv: https://arxiv.org/abs/2306.15266
TLDR: Fault diagnosis is essential in industrial processes for monitoring the conditions of important machines. With the ever-increasing complexity of working conditions and demand for safety during production and operation, different diagnosis methods are required, and more importantly, an integrated fault diagnosis system that can cope with multiple tasks is highly desired. However, the diagnosis subtasks are often studied separately, and the currently available methods still need improvement for such a generalized system. To address this issue, we propose the Generalized Out-of
Repo: None

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?

Authors: Xinzhe Li, Ming Liu, Shang Gao
Arxiv: https://arxiv.org/abs/2306.15268
TLDR: For Pretrained Language Models (PLMs), their susceptibility to noise has recently been linked to subword segmentation. However, it is unclear which aspects of segmentation affect their understanding. This study assesses the robustness of PLMs against various disrupted segmentation caused by noise. An evaluation framework for subword sectionation, named Contrastive Lexical Semantic (CoLeS) probe, is proposed. It provides a systematic categorization of segmentations under noise and evaluation protocols by
Repo: None

Mean Field Theory in Deep Metric Learning

Authors: Takuya Furusawa
Arxiv: https://arxiv.org/abs/2306.15368
TLDR: In this paper, we explore the application of mean field theory, a technique from statistical physics, to deep metric learning and address the high training complexity commonly associated with conventional metric learning loss functions. By adapting mean field field theory for deep metric Learning, we develop an approach to design classification-based loss functions from pair-based ones, which can be considered complementary to the proxy-based approach. Applying the mean field theories to two pair-less loss functions, we derive two new loss
Repo: None

Dental CLAIRES: Contrastive LAnguage Image REtrieval Search for Dental Research

Authors: Tanjida Kabir, Luyao Chen, Muhammad F Walji, Luca Giancardo, Xiaoqian Jiang, Shayan Shams
Arxiv: https://arxiv.org/abs/2306.15651
TLDR: Learning about diagnostic features and related clinical information from dental radiographs is important for dental research. However, the lack of expert-annotated data and convenient search tools poses challenges. Our primary objective is to design a search tool that uses a user's query for oral-related research. The proposed framework, Contrastive LAnguage Image REtrieval Search, utilizes periapical radiographs and associated clinical details such as periodontal diagnosis, demographic information to retrieve the best-
Repo: None

Keyword: data augmentation

TranssionADD: A multi-frame reinforcement based sequence tagging model for audio deepfake detection

Authors: Jie Liu, Zhiba Su, Hui Huang, Caiyan Wan, Quanxiu Wang, Jiangli Hong, Benlai Tang, Fengjie Zhu
Arxiv: https://arxiv.org/abs/2306.15212
TLDR: Thanks to recent advancements in end-to-end speech modeling technology, it has become increasingly feasible to imitate and clone a user`s voice. This leads to a significant challenge in differentiating between authentic and fabricated audio segments. To address the issue of user voice abuse and misuse, the second Audio Deepfake Detection Challenge (ADD 2023) aims to detect and analyze deepfake speech utterances. Specifically, Track 2, named the Manipulation Region Location (RL), aims to pinpoint the
Repo: None

Multi-perspective Information Fusion Res2Net with RandomSpecmix for Fake Speech Detection

Authors: Shunbo Dong, Jun Xue, Cunhang Fan, Kang Zhu, Yujie Chen, Zhao Lv
Arxiv: https://arxiv.org/abs/2306.15389
TLDR: In this paper, we propose the multi-perspective information fusion (MPIF) Res2Net with random Specmix for fake speech detection (FSD). The main purpose of this system is to improve the model's ability to learn precise forgery information for FSD task in low-quality scenarios. The task of random Spe cmix, a data augmentation, is to increase the generalization ability of the model and enhance the model’s ability to locate discrim
Repo: None

Using Large Language Models to Provide Explanatory Feedback to Human Tutors

Authors: Jionghao Lin, Danielle R. Thomas, Feifei Han, Shivang Gupta, Wei Tan, Ngoc Dang Nguyen, Kenneth R. Koedinger
Arxiv: https://arxiv.org/abs/2306.15498
TLDR: Research demonstrates learners engaging in the process of producing explanations to support their reasoning, can have a positive impact on learning. However, providing learners real-time explanatory feedback often presents challenges related to classification accuracy, particularly in domain-specific environments, containing situationally complex and nuanced responses. We present two approaches for supplying tutors real-Time feedback within an online lesson on how to give students effective praise. This work-in-progress demonstrates considerable accuracy in binary classification for corrective feedback of effective,
Repo: None

On the Usefulness of Synthetic Tabular Data Generation

Authors: Dionysis Manousakas, Sergül Aydöre
Arxiv: https://arxiv.org/abs/2306.15636
TLDR: Despite recent advances in synthetic data generation, the scientific community still lacks a unified consensus on its usefulness. It is commonly believed that synthetic data can be used for both data exchange and boosting machine learning (ML) training. Privacy-preserving synthetic data has been shown to accelerate data exchange for downstream tasks, but there is not enough evidence to show how or why synthetic data is useful for ML training. In this study, we benchmarked ML performance using synthetic tabular data for four use cases
Repo: None

Enhancing Representation Learning on High-Dimensional, Small-Size Tabular Data: A Divide and Conquer Method with Ensembled VAEs

Authors: Navindu Leelarathna, Andrei Margeloiu, Mateja Jamnik, Nikola Simidjievski
Arxiv: https://arxiv.org/abs/2306.15661
TLDR: Variational Autoencoders and their many variants have displayed impressive ability to perform dimensionality reduction, often achieving state-of-the-art performance. Many current methods however, struggle to learn good representations in High Dimensional, Low Sample Size (HDLSS) tasks, which is an inherently challenging setting. We address this challenge by using an ensemble of lightweight VAEs to learn posteriors over subsets of the feature-space, which get aggregated into a joint posterior in
Repo: None

Keyword: knowledge discovery

PRISMA-DFLLM: An Extension of PRISMA for Systematic Literature Reviews using Domain-specific Finetuned Large Language Models

Authors: Teo Susnjak
Arxiv: https://arxiv.org/abs/2306.14905
TLDR: With the proliferation of open-sourced Large Language Models (LLMs) and efficient finetuning techniques, we are on the cusp of the emergence of numerous domain-specific LLMs that have been finetuned for expertise across specialized fields and applications for which the current general-purpose LLMs are unsuitable. In academia, this technology has the potential to revolutionize the way we conduct systematic literature reviews (SLRs), access knowledge and generate new insights. This paper proposes an
Repo: None

A novel structured argumentation framework for improved explainability of classification tasks

Authors: Lucas Rizzo, Luca Longo
Arxiv: https://arxiv.org/abs/2306.15500
TLDR: This paper presents a novel framework for structured argumentation, named extend argumentative decision graph ($xADG$). It is an extension of argumentsative decision graphs built upon Dung's abstract argumentation graphs. It is the first paper to describe the structure of argumentative argumentative models, and the first to describe their structure and structure in terms of their size and predictive capacity. The framework allows for arguments to use boolean logic operators and multiple premises (supports) within their internal structure
Repo: None

Keyword: multi-task

Detect Depression from Social Networks with Sentiment Knowledge Sharing

Authors: Yan Shi, Yao Tian, Chengwei Tong, Chunyan Zhu, Qianqian Li, Mengzhu Zhang, Wei Zhao, Yong Liao, Pengyuan Zhou
Arxiv: https://arxiv.org/abs/2306.14903
TLDR: Social network plays an important role in propagating people's viewpoints, emotions, thoughts, and fears. Notably, following lockdown periods during the COVID-19 pandemic, the issue of depression has garnered increasing attention, with a significant portion of individuals resorting to social networks as an outlet for expressing emotions. Using deep learning techniques to discern potential signs of depression from social network messages facilitates the early identification of mental health conditions. Current efforts in detecting depression through social networks typically rely solely on analyzing
Repo: None

Precursor-of-Anomaly Detection for Irregular Time Series

Authors: Sheo Yon Jhin, Jaehoon Lee, Noseong Park
Arxiv: https://arxiv.org/abs/2306.15489
TLDR: Anomaly detection is an important field that aims to identify unexpected patterns or data points, and it is closely related to many real-world problems, particularly to applications in finance, manufacturing, cyber security, and so on. While anomaly detection has been studied extensively in various fields, detecting future anomalies before they occur remains an unexplored territory. In this paper, we present a novel type of anomaly detection, called \emph{\textbf{P}recursor-of-\text
Repo: None

DCID: Deep Canonical Information Decomposition

Authors: Alexander Rakowski, Christoph Lippert
Arxiv: https://arxiv.org/abs/2306.15619
TLDR: We consider the problem of identifying the signal shared between two one-dimensional target variables, in the presence of additional multivariate observations. Canonical Correlation Analysis (CCA)-based methods have traditionally been used to identify shared variables, however, they were designed for multivariate targets and only offer trivial solutions for univariate cases. In the context of Multi-Task Learning (MTL), various models were postulated to learn features that are sparse and shared across multiple tasks. However, these
Repo: https://github.com/alexrakowski/dcid

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

Authors: Chiori Hori, Puyuan Peng, David Harwath, Xinyu Liu, Kei Ota, Siddarth Jain, Radu Corcodel, Devesh Jha, Diego Romeres, Jonathan Le Roux
Arxiv: https://arxiv.org/abs/2306.15644
TLDR: To realize human-robot collaboration, robots need to execute actions for new tasks according to human instructions given to them by human experts with prior knowledge. Human experts can share their knowledge of how to perform a task with a robot through multi-modal instructions in their demonstrations, showing a sequence of short-horizon steps to achieve a long-horizons goal. This paper introduces a method for robot action sequence generation from instruction videos using (1) an audio-visual Transformer that
Repo: None

Keyword: plagiarism

Testing of Detection Tools for AI-Generated Text

Authors: Debora Weber-Wulff (University of Applied Sciences HTW Berlin, Germany), Alla Anohina-Naumeca (Riga Technical University, Latvia), Sonja Bjelobaba (Uppsala University, Sweden), Tomáš Foltýnek (Masaryk University, Czechia), Jean Guerrero-Dib (Universidad de Monterrey, Mexico), Olumide Popoola (Queen Mary's University, UK), Petr Šigut (Masaryk University, Czechia), Lorna Waddington (University of Leeds, UK)
Arxiv: https://arxiv.org/abs/2306.15666
TLDR: Recent advances in generative pre-trained machine translation and large language models have emphasised the potential risks of unfair use of artificial intelligence (AI) generated content in an academic environment and intensified efforts in searching for solutions to detect such content. The paper examines the general functionality of detection tools for artificial intelligence generated text and evaluates them based on accuracy and error type analysis. Specifically, the study seeks to answer research questions about whether existing detection tools can reliably differentiate between human-written text and ChatG
Repo: None

Keyword: robustness

Efficient High-Resolution Template Matching with Vector Quantized Nearest Neighbour Fields

Authors: Ankit Gupta, Ida-Maria Sintorn
Arxiv: https://arxiv.org/abs/2306.15010
TLDR: Template matching is a fundamental problem in computer vision and has applications in various fields, such as object detection, image registration, and object tracking. The current state-of-the-art methods rely on nearest-neighbour (NN) matching in which the query feature space is converted to NN space by representing each query pixel with its NN in the template pixels. The NN-based methods have been shown to perform better in occlusions, changes in appearance, illumination
Repo: None

Structured Dialogue Discourse Parsing

Authors: Ta-Chung Chi, Alexander I. Rudnicky
Arxiv: https://arxiv.org/abs/2306.15103
TLDR: Dialogue discourse parsing aims to uncover the internal structure of a multi-participant conversation by finding all the discourse~\emph{links} and corresponding~\eph{relations}. Previous work either treats this task as a series of independent multiple-choice problems, in which the link existence and relations are decoded separately, or the encoding is restricted to only local interaction, ignoring the holistic structural information. In contrast, we propose a principled method that improves upon previous work from two perspectives
Repo: None

Improvise, Adapt, Overcome: Dynamic Resiliency Against Unknown Attack Vectors in Microgrid Cybersecurity Games

Authors: Suman Rath, Tapadhir Das, Shamik Sengupta
Arxiv: https://arxiv.org/abs/2306.15106
TLDR: Cyber-physical microgrids are vulnerable to rootkit attacks that manipulate system dynamics to create instabilities in the network. Rootkits tend to hide their access level within microgrid system components to launch sudden attacks that prey on the slow response time of defenders to manipulate system trajectory. This problem can be formulated as a multi-stage, non-cooperative, zero-sum game with the attacker and the defender modeled as opposing players. To solve the game, this paper proposes a
Repo: None

Efficient and Accurate Scene Text Detection with Low-Rank Approximation Network

Authors: Yuchen Su
Arxiv: https://arxiv.org/abs/2306.15142
TLDR: Recently, regression-based methods, which predict parameter curves for localizing texts, are popular in scene text detection. However, these methods struggle to balance concise structure and fast post-processing, and the existing parameter curves are still not ideal for modeling arbitrary-shaped texts, leading to a challenge in balancing speed and accuracy. To tackle these challenges, we firstly propose a dual matching scheme for positive samples, which accelerates inference speed through sparse matching scheme and accelerates model convergence through dense
Repo: None

DSRM: Boost Textual Adversarial Training with Distribution Shift Risk Minimization

Authors: Songyang Gao, Shihan Dou, Yan Liu, Xiao Wang, Qi Zhang, Zhongyu Wei, Jin Ma, Ying Shan
Arxiv: https://arxiv.org/abs/2306.15164
TLDR: Adversarial training is one of the best-performing methods in improving the robustness of deep language models. However, robust models come at the cost of high time consumption, as they require multi-step gradient ascents or word substitutions to obtain adversarial samples. In addition, these generated samples are deficient in grammatical quality and semantic consistency, which impairs the effectiveness of adversarial training. To address these problems, we introduce a novel, effective procedure for estimating the probability distribution
Repo: None

Exploiting Inferential Structure in Neural Processes

Authors: Dharmesh Tailor, Mohammad Emtiyaz Khan, Eric Nalisnick
Arxiv: https://arxiv.org/abs/2306.15169
TLDR: Neural Processes (NPs) are appealing due to their ability to perform fast adaptation based on a context set. This set is encoded by a latent variable, which is often assumed to follow a simple distribution. However, in real-word settings, the context set may be drawn from richer distributions having multiple modes, heavy tails, etc. In this work, we provide a framework that allows NPs' latent variable to be given a rich prior defined by a graphical model. These
Repo: None

TranssionADD: A multi-frame reinforcement based sequence tagging model for audio deepfake detection

Authors: Jie Liu, Zhiba Su, Hui Huang, Caiyan Wan, Quanxiu Wang, Jiangli Hong, Benlai Tang, Fengjie Zhu
Arxiv: https://arxiv.org/abs/2306.15212
TLDR: Thanks to recent advancements in end-to-end speech modeling technology, it has become increasingly feasible to imitate and clone a user`s voice. This leads to a significant challenge in differentiating between authentic and fabricated audio segments. To address the issue of user voice abuse and misuse, the second Audio Deepfake Detection Challenge (ADD 2023) aims to detect and analyze deepfake speech utterances. Specifically, Track 2, named the Manipulation Region Location (RL), aims to pinpoint the
Repo: None

[Re] Double Sampling Randomized Smoothing

Authors: Aryan Gupta, Sarthak Gupta, Abhay Kumar, Harsh Dugar
Arxiv: https://arxiv.org/abs/2306.15221
TLDR: This paper is a contribution to the reproducibility challenge in the field of machine learning, specifically addressing the issue of certifying the robustness of neural networks (NNs) against adversarial perturbations. The proposed Double Sampling Randomized Smoothing (DSRS) framework overcomes the limitations of existing methods by using an additional smoothing distribution to improve the robustity certification. The paper provides a clear manifestation of DSRS for a generalized family of Gaussian smoothing and
Repo: None

Learning to Rank in Generative Retrieval

Authors: Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
Arxiv: https://arxiv.org/abs/2306.15222
TLDR: Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target. This paradigm leverages powerful generation models and represents a new paradigm distinct from traditional learning-to-rank methods. However, despite its rapid development, current generative retrieval methods are still limited. They typically rely on a heuristic function to transform predicted identifiers into a passage rank list, which creates a gap between the learning objective of generative retrieve and the desired passage ranking target
Repo: None

Errorless Robust JPEG Steganography Using Steganographic Polar Codes

Authors: Jimin Zhang, Xianfeng Zhao, Xiaolei He
Arxiv: https://arxiv.org/abs/2306.15246
TLDR: Recently, a robust steganographic algorithm that achieves errorless robustness against JPEG recompression is proposed. The method evaluates the behavior of DCT coefficients after recompression using the local JPEG encoder to select robust coefficients and sets the other coefficients as wet cost. Combining the lattice embedding scheme, the method is errorless by construction. However, the authors only concern with the success rate under theoretical embedding, while the failure rate of the implementation with practical steganography codes
Repo: None

A Survey on Out-of-Distribution Evaluation of Neural NLP Models

Authors: Xinzhe Li, Ming Liu, Shang Gao, Wray Buntine
Arxiv: https://arxiv.org/abs/2306.15261
TLDR: Adversarial robustness, domain generalization and dataset biases are three active lines of research contributing to out-of-distribution (OOD) evaluation on neural NLP models. However, a comprehensive, integrated discussion of the three research lines is still lacking in the literature. In this survey, we 1) compare the three lines of work under a unifying definition; 2) summarize the data-generating processes and evaluation protocols for each line of research; and 3) emphasize the
Repo: None

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?

Authors: Xinzhe Li, Ming Liu, Shang Gao
Arxiv: https://arxiv.org/abs/2306.15268
TLDR: For Pretrained Language Models (PLMs), their susceptibility to noise has recently been linked to subword segmentation. However, it is unclear which aspects of segmentation affect their understanding. This study assesses the robustness of PLMs against various disrupted segmentation caused by noise. An evaluation framework for subword sectionation, named Contrastive Lexical Semantic (CoLeS) probe, is proposed. It provides a systematic categorization of segmentations under noise and evaluation protocols by
Repo: None

Shoggoth: Towards Efficient Edge-Cloud Collaborative Real-Time Video Inference via Adaptive Online Learning

Authors: Liang Wang, Kai Lu, Nan Zhang, Xiaoyang Qu, Jianzong Wang, Jiguang Wan, Guokuan Li, Jing Xiao
Arxiv: https://arxiv.org/abs/2306.15333
TLDR: This paper proposes Shoggoth, an efficient edge-cloud collaborative architecture, for boosting inference performance on real-time video of changing scenes. ShoggOTH uses online knowledge distillation to improve the accuracy of models suffering from data drift and offloads the labeling process to the cloud, alleviating constrained resources of edge devices. At the edge, we design adaptive training using small batches to adapt models under limited computing power, and adaptive sampling of training frames for robustness and reducing bandwidth. The
Repo: None

Robust domain decomposition methods for high-contrast multiscale problems on irregular domains with virtual element discretizations

Authors: Juan G. Calvo, Juan Galvis
Arxiv: https://arxiv.org/abs/2306.15424
TLDR: Our research focuses on the development of domain decomposition preconditioners tailored for second-order elliptic partial differential equations. Our approach addresses two major challenges simultaneously: i) effectively handling coefficients with high-contrast and multiscale properties, and ii) accommodating irregular domains in the original problem, the coarse mesh, and the subdomain partition. The robustness of our preconditionsers is crucial for real-world applications, such as the efficient and accurate modeling of subsur
Repo: None

Advancing Adversarial Training by Injecting Booster Signal

Authors: Hong Joo Lee, Youngjoon Yu, Yong Man Ro
Arxiv: https://arxiv.org/abs/2306.15451
TLDR: Recent works have demonstrated that deep neural networks (DNNs) are highly vulnerable to adversarial attacks. To defend against adversarial Attacks, many defense strategies have been proposed, among which adversarial training has been demonstrated to be the most effective strategy. However, it has been known that adversarial learning sometimes hurts natural accuracy. Then, many works focus on optimizing model parameters to handle the problem. Different from the previous approaches, in this paper, we propose a new approach to improve the
Repo: None

Robust Proxy: Improving Adversarial Robustness by Robust Proxy Learning

Authors: Hong Joo Lee, Yong Man Ro
Arxiv: https://arxiv.org/abs/2306.15457
TLDR: Recently, it has been widely known that deep neural networks are highly vulnerable and easily broken by adversarial attacks. To mitigate the adversarial vulnerability, many defense algorithms have been proposed. Recently, to improve adversarial robustness, many works try to enhance feature representation by imposing more direct supervision on the discriminative feature. However, existing approaches lack an understanding of learning adversarially robust feature representation. In this paper, we propose a novel training framework called Robust Proxy Learning. In
Repo: None

Cooperation or Competition: Avoiding Player Domination for Multi-Target Robustness via Adaptive Budgets

Authors: Yimu Wang, Dinghuai Zhang, Yihan Wu, Heng Huang, Hongyang Zhang
Arxiv: https://arxiv.org/abs/2306.15482
TLDR: Despite incredible advances, deep learning has been shown to be susceptible to adversarial attacks. Numerous approaches have been proposed to train robust networks both empirically and certifiably. However, most of them defend against only a single type of attack, while recent work takes steps forward in defending against multiple attacks. In this paper, to understand multi-target robustness, we view this problem as a bargaining game in which different players (adversaries) negotiate to reach an agreement on a joint
Repo: None

Simple Steps to Success: Axiomatics of Distance-Based Algorithmic Recourse

Authors: Jenny Hamer, Jake Valladares, Vignesh Viswanathan, Yair Zick
Arxiv: https://arxiv.org/abs/2306.15557
TLDR: We propose a novel data-driven framework for algorithmic recourse that offers users interventions to change their predicted outcome. Existing approaches to compute recourse find a set of points that satisfy some desiderata -- e.g. an intervention in the underlying causal graph, or minimizing a cost function. Satisfying these criteria, however, requires extensive knowledge of the underlying model structure, often an unrealistic amount of information in several domains. We propose a data-data-driven, computationally efficient approach
Repo: None

See Through the Fog: Curriculum Learning with Progressive Occlusion in Medical Imaging

Authors: Pradeep Singh, Kishore Babu Nampalle, Uppala Vivek Narayan, Balasubramanian Raman
Arxiv: https://arxiv.org/abs/2306.15574
TLDR: In recent years, deep learning models have revolutionized medical image interpretation, offering substantial improvements in diagnostic accuracy. However, these models often struggle with challenging images where critical features are partially or fully occluded, which is a common scenario in clinical practice. In this paper, we propose a novel curriculum learning-based approach to train deep learning model to handle occluding medical images effectively. Our method progressively introduces occlusion, starting from clear, unobstructed images and gradually moving to images
Repo: None

Learning Nonautonomous Systems via Dynamic Mode Decomposition

Authors: Hannah Lu, Daniel M. Tartakovsky
Arxiv: https://arxiv.org/abs/2306.15618
TLDR: We present a data-driven learning approach for unknown nonautonomous dynamical systems with time-dependent inputs based on dynamic mode decomposition (DMD). To circumvent the difficulty of approximating the time-time-dependent Koopman operators fornonautonomous systems, a modified system derived from local parameterization of the external time-independent inputs. The modified system comprises a sequence of local parametric systems, which can be well approximated by a parametric surrogate model using our previously
Repo: None

SparseOptimizer: Sparsify Language Models through Moreau-Yosida Regularization and Accelerate through Compiler Co-design

Authors: Fu-Ming Guo
Arxiv: https://arxiv.org/abs/2306.15656
TLDR: This paper introduces SparseOptimizer, a novel deep learning optimizer that exploits Moreau-Yosida regularization to naturally induce sparsity in large language models such as BERT, ALBERT and GPT. Key to the design of SparseVictimizer is an embedded shrinkage operator, which imparts sparsity directly within the optimization process. This operator, backed by a sound theoretical framework, includes an analytical solution, thereby reinforcing the optimizer's robustness
Repo: None

Keyword: simplification

Revisiting Tropical Polynomial Division: Theory, Algorithms and Application to Neural Networks

Authors: Ioannis Kordonis, Petros Maragos
Arxiv: https://arxiv.org/abs/2306.15157
TLDR: Tropical geometry has recently found several applications in the analysis of neural networks with piecewise linear activation functions. This paper presents a new look at the problem of tropical polynomial division and its application to the simplification of neural nets. We analyze tropical poolesomials with real coefficients, extending earlier ideas and methods developed for polynomials and integer coefficients. We first prove the existence of a unique quotient-remainder pair and characterize the quotient in terms of
Repo: None

Frex: dependently-typed algebraic simplification

Authors: Guillaume Allais, Edwin Brady, Nathan Corbyn, Ohad Kammar, Jeremy Yallop
Arxiv: https://arxiv.org/abs/2306.15375
TLDR: We present an extensible, mathematically-structured algebraic simplification library design. We structure the library using universal algebraic concepts: a free algebra -- fral -- and a free extension -- frex -- of an algebra by a set of variables. The library's dependently-typed API guarantees simplification modules, even user-defined ones, are terminating, sound, and complete with respect to a well-specified class of equations. Completeness offers intangible benefits in
Repo: None

Keyword: summarization

FeedbackMap: a tool for making sense of open-ended survey responses

Authors: Doug Beeferman, Nabeel Gillani
Arxiv: https://arxiv.org/abs/2306.15112
TLDR: Analyzing open-ended survey responses is a crucial yet challenging task for social scientists, non-profit organizations, and educational institutions, as they often face the trade-off between obtaining rich data and the burden of reading and coding textual responses. This demo introduces FeedbackMap, a web-based tool that uses natural language processing techniques to facilitate the analysis of open-ending survey responses. FeedbackMap lets researchers generate summaries at multiple levels, identify interesting response examples, and visualize the response space
Repo: None

Extending Context Window of Large Language Models via Positional Interpolation

Authors: Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian
Arxiv: https://arxiv.org/abs/2306.15595
TLDR: We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaLAMA 7B to 65B. Meanwhile, the extended model by PositionInterpolation preserves quality relatively well
Repo: None

On the Usefulness of Synthetic Tabular Data Generation

Authors: Dionysis Manousakas, Sergül Aydöre
Arxiv: https://arxiv.org/abs/2306.15636
TLDR: Despite recent advances in synthetic data generation, the scientific community still lacks a unified consensus on its usefulness. It is commonly believed that synthetic data can be used for both data exchange and boosting machine learning (ML) training. Privacy-preserving synthetic data has been shown to accelerate data exchange for downstream tasks, but there is not enough evidence to show how or why synthetic data is useful for ML training. In this study, we benchmarked ML performance using synthetic tabular data for four use cases
Repo: None

Keyword: text generation

Semi-Supervised Image Captioning with CLIP

Authors: Chuanyang Jin
Arxiv: https://arxiv.org/abs/2306.15111
TLDR: Image captioning, a fundamental task in vision-language understanding, seeks to generate accurate natural language descriptions for provided images. The CLIP model, with its rich semantic features learned from a large corpus of image-text pairs, is well-suited for this task. In this paper, we present a two-stage semi-supervised image captioning approach that exploits the potential of CLIP encoding. Our model comprises a CLIP visual encoder, a mapping network, and a
Repo: None

Learning to Rank in Generative Retrieval

Authors: Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
Arxiv: https://arxiv.org/abs/2306.15222
TLDR: Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target. This paradigm leverages powerful generation models and represents a new paradigm distinct from traditional learning-to-rank methods. However, despite its rapid development, current generative retrieval methods are still limited. They typically rely on a heuristic function to transform predicted identifiers into a passage rank list, which creates a gap between the learning objective of generative retrieve and the desired passage ranking target
Repo: None