khot greater than 1

[yujianll]

Hi, thanks for sharing the code!

I'm using the SubsetOperator to sample k elements from n elements (k < n). However, I found khot > 1 for some scores. I wonder if this is the expected output. In the paper, it says each value in the k-hot vector should be within [0, 1].

[yujianll]

As a follow up question, is there any methods that can give a relaxed khot vector within [0, 1]? I want to represent the unselected elements as 1 - khot but fail to do so if khot is greater than 1.

[sangmichaelxie]

Sorry this is late - How much greater than 1 is it? It might just be a numerical issue that could be resolved by renormalizing.

[SiyuanHuangSJTU]

I've encountered the same problem.

Here I offer an example of the problem:

import torch
import numpy as np

class SubsetOperator(torch.nn.Module):
    def __init__(self, k, tau=1.0):
        super(SubsetOperator, self).__init__()
        self.k = k
        self.tau = tau

    def forward(self, scores):
        # m = torch.distributions.gumbel.Gumbel(torch.zeros_like(scores), torch.ones_like(scores))
        # g = m.sample()
        g = torch.tensor([[-0.1926, -1.8388, -0.7433, -0.0096, -0.6953,  1.1131, -0.4599,  3.3375,
          0.5950,  0.0287, -0.5226,  0.5701,  0.0701,  1.9500,  0.2947, -0.1015,
         -0.7057,  1.9853,  2.0632, -0.0271]])
        scores = scores + g 

        khot = torch.zeros_like(scores)
        onehot_approx = torch.zeros_like(scores)
        for i in range(self.k):
            khot_mask = torch.max(1.0 - onehot_approx, torch.tensor([EPSILON]))
            scores = scores + torch.log(khot_mask)
            onehot_approx = torch.nn.functional.softmax(scores / self.tau, dim=-1)
            khot = khot + onehot_approx

        return khot

# Constants
EPSILON = np.finfo(np.float32).tiny

k = 10 # Number of top passages to select
tau = 3.  # Temperature parameter

subset_op = SubsetOperator(k=k, tau=tau)

# score = torch.randn(size=(1,20))
score = torch.tensor([[ 1.4577,  0.1808, -0.4349,  0.5752,  0.4115, -0.1067, -2.4714,  0.2549,
          0.4630,  0.7541, -0.0901,  1.8299,  0.5339, -0.5185,  0.6371, -0.4329,
          1.0309, -1.8988,  0.9963,  1.7575]])

from matplotlib import pyplot as plt
res = subset_op(score).detach().numpy().flatten()
print(res)
plt.plot(res)

which outputs: [0.5602655 0.2231781 0.26037002 0.45167285 0.34610078 0.5175879 0.14770216 ****1.107019**** 0.5258558 0.483119 0.3118482 0.78725654 0.4570909 0.58937955 0.5058449 0.31969813 0.4190858 0.38890892 0.9528999 0.64511645]

In fact, for me, the statement that "the numerical range is between 0 and 1" is problematic. I don't believe that the algorithm provided in this paper (https://arxiv.org/abs/1901.10517) can ensure that the numerical range is within 0 and 1. This is because the soft k-hot mask does not fully meet the requirement to completely mask the selected elements. I wonder if my understanding is incorrect?

[SiyuanHuangSJTU]

I recently noticed that on page 4 of your paper, you explicitly mentioned that the output might exceed 1, e.g., “The output on this input is [1.05, 0.95]”. So the assumption in Algorithm 2 regarding outputs being strictly within the range [0, 1] might not always hold ahhh.

I’ve been trying to adapt your algorithm for use in attention mechanisms, but I’ve encountered cases where the scores significantly exceed 1 (and sometimes even go beyond 2). This behavior has had a substantial negative impact on my model's performance in my specific use case. Do you happen to know of any algorithms for continuous subset sampling that ensure outputs are strictly constrained within the [0, 1] range? I'd greatly appreciate any insights or suggestions!
、、