something about data format

[wkywwds]

If I want to replace the data format in "example/vipseg" with my own (as shown below), need I give the masked objects（with red colour）different ids, that is, label them with different colors? And need i do this for each picture in my own dataset?

[hkchengrex]

The masks should be read by PIL as index masks. See https://github.com/hkchengrex/XMem/blob/main/docs/PALETTE.md

[wkywwds]

The masks should be read by PIL as index masks. See https://github.com/hkchengrex/XMem/blob/main/docs/PALETTE.md

If I want to replace the data format in "example/vipseg" with my own (as shown below), need I give the masked objects（with red colour）different ids, that is, label them with different colors? And need i do this for each picture in my own dataset?

OK.But The objects with the red mask are the same class of objects, but I hope each of them separated by DEVA, which means I want to mark them in different colors. In this case, do I have to give each object a different ID in the corresponding json file

[hkchengrex]

In this mode, we don't care about the class. It doesn't matter that they are in the same class. If you want to track them independently, label them independently.

On a side note, this type of data seems very out-of-domain.

[wkywwds]

Fine,it is the CT of ceramic matrix composites(CMCs),which is applied in high-temperature components of aerospace spacecraft.And again, I want to confirm if in the data set required by DEVA, only a part of the pictures（not all） divided by my own model are needed, and each object is numbered in the corresponding json file, that is, different objects are marked with different colors.

[hkchengrex]

Yes.

[wkywwds]

Hi,where to change the "Max allocated memory"

[wkywwds]

Is the json file in the example necessary？I found that the database even without the json file, the program can still run.

[hkchengrex]

The json files allow users to propagate segment information (e.g., object classes) to the output. It is not strictly necessary.

Max allocated memory is just reporting the maximum amount of GPU memory allocated by PyTorch.

[wkywwds]

Why isn't it ideal to run my dataset with DEVA

The mask is this

[hkchengrex]

I think your mask is an RGB image and not an index mask as mentioned above.

[wkywwds]

If so,how to change the RGB mask into index mask

[hkchengrex]

The masks should be read by PIL as index masks. See https://github.com/hkchengrex/XMem/blob/main/docs/PALETTE.md

Please see this reply from above.
Simply put, the underlying data structure should be a single-channel integer mask. You can verify this by reading the image with PIL and converting it to a numpy array.

[wkywwds]

Is it to change colour mask into palette mask? But after transforming into palette mask by PIL, it seems that the result is still not satisfactory

[hkchengrex]

The conversion is wrong. You would need to find the unique colors in the image and remap the pixels.

The following is a response from Claude 3 Sonnet.

To find the unique colors in an RGB image and remap the corresponding pixels to an index mask using PIL (Python Imaging Library) and NumPy, you can follow these steps:

1. Load the image using PIL and convert it to a NumPy array.
2. Reshape the NumPy array to a 2D array, where each row represents a pixel and each column represents the R, G, and B values.
3. Use NumPy's unique function to find the unique color combinations in the reshaped array.
4. Create a mapping from each unique color combination to a unique index.
5. Use NumPy's searchsorted function to find the index of each pixel's color combination in the unique color combinations array.
6. Create a new NumPy array with the same shape as the original image, but with the indices from the previous step as the values.

Here's the code to implement this:

from PIL import Image
import numpy as np

# Load the image
image = Image.open('image.jpg')

# Convert the image to a NumPy array
image_array = np.array(image)

# Reshape the array to a 2D array (pixels x RGB)
reshaped_array = image_array.reshape(-1, 3)

# Find unique color combinations
unique_colors = np.unique(reshaped_array, axis=0)

# Create a mapping from unique colors to indices
color_to_index = {tuple(color): index for index, color in enumerate(unique_colors)}

# Find the index of each pixel's color combination
indices = np.array([color_to_index[tuple(color)] for color in reshaped_array])

# Reshape the indices back to the original image shape
index_mask = indices.reshape(image_array.shape[0], image_array.shape[1])

In this code:

• image_array is the NumPy array representation of the image.
• reshaped_array is a 2D array where each row represents a pixel and each column represents the R, G, and B values.
• unique_colors is a 2D array containing the unique color combinations in the image.
• color_to_index is a dictionary that maps each unique color combination to a unique index.
• indices is a 1D array containing the index of each pixel's color combination in the unique_colors array.
• index_mask is a 2D array with the same shape as the original image, but with the indices from indices as the values.

After running this code, index_mask will contain the index mask, where each pixel value corresponds to the index of its color combination in the unique_colors array.

Note that this approach assumes that the image has a limited number of unique colors. For images with a large number of unique colors (e.g., high-resolution photographs), this method may not be efficient, and you might need to use alternative techniques, such as color quantization or clustering.

[wkywwds]

I wrote the related program based on the above code but the effect is not very good. What goes wrong this time.

[hkchengrex]

Two observations:

1. The cropped version is working better
2. The input and output colors don't match

It seems to me that your mask input (or conversion) is still buggy.

[wkywwds]

After trying, it is found that even if the mask is RGB, the program can run normally, and the fewer the number of targets, the better the prediction effect. May I ask whether there will be a discount for the prediction effect of DEVA for a larger number of targets

[hkchengrex]

Thank you for the update. It is possible that having many targets degrades the output (due to increased noise in memory matching), especially in out-of-domain cases like yours.

[wkywwds]

Is there a solution for this case

Thank you for the update. It is possible that having many targets degrades the output (due to increased noise in memory matching), especially in out-of-domain cases like yours.

[hkchengrex]

1. Can you provide all the output frames up to the point of failure so that we can have a closer look?
2. What if you supply fewer objects but the image remains uncropped?

[wkywwds]

The file size exceeds the limit and needs to be sent separately

…

------------------ 原始邮件 ------------------ 发件人: "hkchengrex/Tracking-Anything-with-DEVA" ***@***.***>; 发送时间: 2024年5月12日(星期天) 凌晨2:06 ***@***.***>; ***@***.******@***.***>; 主题: Re: [hkchengrex/Tracking-Anything-with-DEVA] something about data format (Issue #85) Can you provide all the output frames up to the point of failure so that we can have a closer look? What if you supply fewer objects but the image remains uncropped? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[wkywwds]

The file size exceeds the limit and needs to be sent separately

…

------------------ 原始邮件 ------------------ 发件人: "hkchengrex/Tracking-Anything-with-DEVA" ***@***.***>; 发送时间: 2024年5月12日(星期天) 凌晨2:06 ***@***.***>; ***@***.******@***.***>; 主题: Re: [hkchengrex/Tracking-Anything-with-DEVA] something about data format (Issue #85) Can you provide all the output frames up to the point of failure so that we can have a closer look? What if you supply fewer objects but the image remains uncropped? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[wkywwds]

Hello, have you received the file I sent? Have you found the reason why DEVA can't efficiently identify my dataset

[hkchengrex]

No, I haven't received anything.

[wkywwds]

No, I haven't received anything.
Unet is used first, and then the connected domain is filtered according to the area threshold to form a mask.Only seven images have been tested.
IMG.zip

[hkchengrex]

Sorry I cannot check it right now. I'll get back at this later.

[wkywwds]

Can you check the "IMG.zip" file? If not, I'll send it to you by email

[wkywwds]

Have you found the cause of the problem when deva tracking multiple targets?

[hkchengrex]

I haven't had the time to test it yet.