Data loader issue

[aplusc98]

hey,

I'm using my own dataset which follows the folder structure as given in the readme. I have 89 classes and want to split 64 10 15 as the train validation and evaluation split . Each class is balanced and has 20 samples.
I made the following changes:
data = dataset.FolderDatasetLoader(num_of_gpus=1, batch_size=args.batch_size, image_height=200, image_width=300,
image_channels=3,
train_val_test_split=(64/89, 10/89, 15/89),
samples_per_iter=1, num_workers=4,
data_path="datasets/city_names", name="city_names",
indexes_of_folders_indicating_class=-2, reset_stored_filepaths=False,
num_samples_per_class=args.samples_per_class,
num_classes_per_set=args.classes_per_set, label_as_int=False)
and getting the following error
Mapped data paths can't be found, remapping paths..
Get images from /home/user/MatchingNetworks/datasets/city_names
Traceback (most recent call last):
File "/home/cvprisi19/MatchingNetworks/data.py", line 98, in load_datapaths
data_image_paths = self.load_dict(data_path_file)
File "/home/cvprisi19/MatchingNetworks/data.py", line 115, in load_dict
with open(name, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'datasets/city_names.pkl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_one_shot_learning_matching_network.py", line 20, in
num_classes_per_set=args.classes_per_set, label_as_int=False)
File "/home/cvprisi19/MatchingNetworks/data.py", line 453, in init
samples_per_iter, num_workers, reverse_channels, seed, labels_as_int=label_as_int)
File "/home/cvprisi19/MatchingNetworks/data.py", line 347, in init
reset_stored_filepaths=False, data_path=data_path, labels_as_int=labels_as_int)
File "/home/cvprisi19/MatchingNetworks/data.py", line 466, in get_dataset
reverse_channels=reverse_channels)
File "/home/cvprisi19/MatchingNetworks/data.py", line 439, in init
indexes_of_folders_indicating_class=indexes_of_folders_indicating_class)
File "/home/cvprisi19/MatchingNetworks/data.py", line 41, in init
self.x_train, self.x_val, self.x_test = self.load_dataset()
File "/home/cvprisi19/MatchingNetworks/data.py", line 71, in load_dataset
data_image_paths, index_to_label_name_dict_file, label_to_index = self.load_datapaths()
File "/home/cvprisi19/MatchingNetworks/data.py", line 104, in load_datapaths
data_image_paths, code_to_label_name, label_name_to_code = self.get_data_paths()
File "/home/cvprisi19/MatchingNetworks/data.py", line 145, in get_data_paths
label = self.get_label_from_path(filepath)
File "/home/cvprisi19/MatchingNetworks/data.py", line 179, in get_label_from_path
label = "_".join([label_bits[idx] for idx in self.indexes_of_folders_indicating_class])
TypeError: 'int' object is not iterable

even though the path contains the dataset and the images ...the pickle file itself is not being generated.

[AntreasAntoniou]

I see. Let's diagnose this together.

1. Do you have write access to the folder? If yes, move to question 2.
2. Search for .pkl files in your whole framework directory and let me know if you can find the currently missing pkl file.

[aplusc98]

1. Yes I have read/write access to the folder.
2. The pkl files are not being generated that's the main issue. I think the program is not able to locate the dataset . I'm guessing it has something to do with this argument below
   indexes_of_folders_indicating_class=-2,
   does this seem right for the dataset structure that I have?

[aplusc98]

I see. Let's diagnose this together.

1. Do you have write access to the folder? If yes, move to question 2.
2. Search for .pkl files in your whole framework directory and let me know if you can find the currently missing pkl file.

,
I kind of solved the data loader issue, now i'm able to togenerate the pkl files and load data but after the training starts, I'm getting this error,

0%| | 0/200 [00:00<?, ?it/s]
Traceback (most recent call last): | 0/1000 [00:00<?, ?it/s]
File "train_one_shot_learning_matching_network.py", line 69, in
sess=sess)
File "/home/cvprisi19/MatchingNetworks/experiment_builder.py", line 82, in run_training_epoch
self.learning_rate: self.current_learning_rate})
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1128, in _run
str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1, 32, 300, 200, 3) for Tensor 'support_set_images:0', which has shape '(1, 32, 5, 1, 200, 300, 3)'

any insight would be useful

[Dennis-Chu]

I also encounter the data loader issues, no PKL file is generated while running the 'train_one_shot_learning_matching_network' as instructured.

[Dennis-Chu]

Can any one share the solution to make this source work?

[AntreasAntoniou]

Hello everyone, This implementation is now quite old. Expect a full update both of the code and the data providers within the next month.

…

On Sat, 4 Apr 2020 at 08:49, Dennis-Chu ***@***.***> wrote: Can any one share the solution to make this source work? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSK4NTRWF7RB2S45ZGTDC3RK3RA3ANCNFSM4H6HQXBA> .

[lwzhaojun]

I see. Let's diagnose this together.

1. Do you have write access to the folder? If yes, move to question 2.
2. Search for .pkl files in your whole framework directory and let me know if you can find the currently missing pkl file.

,
I kind of solved the data loader issue, now i'm able to togenerate the pkl files and load data but after the training starts, I'm getting this error,

0%| | 0/200 [00:00<?, ?it/s]
Traceback (most recent call last): | 0/1000 [00:00<?, ?it/s]
File "train_one_shot_learning_matching_network.py", line 69, in
sess=sess)
File "/home/cvprisi19/MatchingNetworks/experiment_builder.py", line 82, in run_training_epoch
self.learning_rate: self.current_learning_rate})
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1128, in _run
str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1, 32, 300, 200, 3) for Tensor 'support_set_images:0', which has shape '(1, 32, 5, 1, 200, 300, 3)'

any insight would be useful

请问你是如何解决你的第一个问题，也就是 data loader issue,

[lwzhaojun]

I also encounter the data loader issues, no PKL file is generated while running the 'train_one_shot_learning_matching_network' as instructured.

你好，请问你解决了这个问题了吗，无法生成pkl文件。。

[Hongjiew]

I see. Let's diagnose this together.

1. Do you have write access to the folder? If yes, move to question 2.
2. Search for .pkl files in your whole framework directory and let me know if you can find the currently missing pkl file.

,
I kind of solved the data loader issue, now i'm able to togenerate the pkl files and load data but after the training starts, I'm getting this error,

0%| | 0/200 [00:00<?, ?it/s]
Traceback (most recent call last): | 0/1000 [00:00<?, ?it/s]
File "train_one_shot_learning_matching_network.py", line 69, in
sess=sess)
File "/home/cvprisi19/MatchingNetworks/experiment_builder.py", line 82, in run_training_epoch
self.learning_rate: self.current_learning_rate})
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1128, in _run
str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1, 32, 300, 200, 3) for Tensor 'support_set_images:0', which has shape '(1, 32, 5, 1, 200, 300, 3)'

any insight would be useful

Would you like to share your method to solve this data loader problem? I ran the code successfully on the Linux based server, but failed on my personal computer, which is based on Windows 10.