MiniRocket with MMAP and Dataloaders #909
Unanswered
smagmaster
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi there,
First of I want to thank the tsai library it has been a tremendous help in every way, although I need some help now...
So I used the MiniRocketClassifier() model to classify movements of an EMG data. The first dataset worked perfectly fine but now I have a bigger dataset and I always get Memory errors all over the place.
So I have read the tutorial with using mmaps to not allocate to much memory on the RAM but for some reason it is not working properly. Here is what I have done so far:
`
X_train = np.load(f'{save}/X_train.npy', mmap_mode='r') ...
Convert numpy arrays to torch tensors
X_train_tensor = torch.tensor(X_train, dtype=torch.float32)
y_train_tensor = torch.tensor(y_train, dtype=torch.long)
X_test_tensor = torch.tensor(X_test, dtype=torch.float32)
y_test_tensor = torch.tensor(y_test, dtype=torch.long)
Create datasets and dataloaders
train_dataset = TensorDataset(X_train_tensor, y_train_tensor)
test_dataset = TensorDataset(X_test_tensor, y_test_tensor)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
`
and then trained the model. Before I saved and loaded the variables with 'r' and 'c' but it did not make a difference. I have tried using batches as well to reduce the memory allocation but this is just bad for the minirocket model in general because mini-rocket does not work that way.
My Question here would be:
How can I use the mmaps and dataloader correctly to train a large dataset in tsai minirocket?
Any help is appreciated.
Thank you so much in advance and have a good day!
Best regards,
A desperate student
Beta Was this translation helpful? Give feedback.
All reactions