Memory leak with rgb render

[vedantdave97]

Hello,
I am facing issues with memory leak when I'm setting rgb_render as True in this line for getting rgb_images in observation

ManiSkill-ViTac2024/envs/peg_insertion.py

Line 743 in ef0e008

render_rgb: bool = False,

I am pretty sure that I have undone all the changes that I made with the code.
I have attached the log files for both where you can see the gpu memory rise in the rgb case. I am not sure if this should be the case.

If this is supposed to happen, please let me know.
gpu_memory_log_with_rgb.txt
gpu_memory_log_without_rgb.txt

[callmeray]

hi, we cannot open the log files. are you training or evaluating?

[vedantdave97]

I don't know why its not opening. I also tried incognito and it works. These are the details of both the files. They however only have the GPU memory usage that I generated from a simple bash script. If you need anything else, please let me know.

Without rgb_render
Timestamp, GPU Memory (MB)
2024-03-12 16:34:15, 5918
2024-03-12 16:34:20, 5927
2024-03-12 16:34:25, 5905
2024-03-12 16:34:30, 5915
2024-03-12 16:34:35, 5907
2024-03-12 16:34:40, 5906
2024-03-12 16:34:45, 5905
2024-03-12 16:34:50, 5909
2024-03-12 16:34:55, 5908
2024-03-12 16:35:00, 5906
2024-03-12 16:35:05, 5889
2024-03-12 16:35:10, 5895
2024-03-12 16:35:15, 5919
2024-03-12 16:35:20, 5909
2024-03-12 16:35:25, 5907
2024-03-12 16:35:30, 5906
2024-03-12 16:35:35, 5919
2024-03-12 16:35:40, 5906
2024-03-12 16:35:45, 5901
2024-03-12 16:35:50, 5909
2024-03-12 16:35:55, 5917
2024-03-12 16:36:00, 5917
2024-03-12 16:36:05, 5923
2024-03-12 16:36:10, 5926
2024-03-12 16:36:15, 5917
2024-03-12 16:36:20, 5908
2024-03-12 16:36:25, 5918
2024-03-12 16:36:30, 5921
2024-03-12 16:36:35, 5927
2024-03-12 16:36:40, 5929
2024-03-12 16:36:45, 5936
2024-03-12 16:36:50, 5923
2024-03-12 16:36:55, 5919
2024-03-12 16:37:00, 5919
2024-03-12 16:37:05, 5919
2024-03-12 16:37:10, 5919
2024-03-12 16:37:15, 5919
2024-03-12 16:37:20, 5919
2024-03-12 16:37:25, 5919
2024-03-12 16:37:30, 5919
2024-03-12 16:37:35, 5919
2024-03-12 16:37:40, 5919
2024-03-12 16:37:45, 5689
2024-03-12 16:37:50, 5689
2024-03-12 16:37:55, 5689
2024-03-12 16:38:00, 5689
2024-03-12 16:38:05, 5689
2024-03-12 16:38:10, 5689
2024-03-12 16:38:15, 5689
2024-03-12 16:38:20, 5691
2024-03-12 16:38:25, 5691
2024-03-12 16:38:30, 5705
2024-03-12 16:38:35, 5696
2024-03-12 16:38:40, 5694
2024-03-12 16:38:45, 5688
2024-03-12 16:38:51, 5713
2024-03-12 16:38:56, 5703
2024-03-12 16:39:01, 5710
2024-03-12 16:39:06, 5701
2024-03-12 16:39:11, 5694
2024-03-12 16:39:16, 5694
2024-03-12 16:39:21, 5694
2024-03-12 16:39:26, 5669
2024-03-12 16:39:31, 5669
2024-03-12 16:39:36, 5669
2024-03-12 16:39:41, 5669
2024-03-12 16:39:46, 5669
2024-03-12 16:39:51, 5701
2024-03-12 16:39:56, 5721
2024-03-12 16:40:01, 5721
2024-03-12 16:40:06, 5690
2024-03-12 16:40:11, 5690
2024-03-12 16:40:16, 5690
2024-03-12 16:40:21, 5690
2024-03-12 16:40:26, 5690
2024-03-12 16:40:31, 5690
2024-03-12 16:40:36, 5690
2024-03-12 16:40:41, 5690
2024-03-12 16:40:46, 5690
2024-03-12 16:40:51, 5690
2024-03-12 16:40:56, 5690
2024-03-12 16:41:01, 5690
2024-03-12 16:41:06, 5690
2024-03-12 16:41:11, 5692
2024-03-12 16:41:16, 5699
2024-03-12 16:41:21, 5721
2024-03-12 16:41:26, 5729
2024-03-12 16:41:31, 5729
2024-03-12 16:41:36, 5740
2024-03-12 16:41:41, 5758
2024-03-12 16:41:46, 534

With rgb_render
Timestamp, GPU Memory (MB)
2024-03-12 16:42:09, 544
2024-03-12 16:42:14, 5000
2024-03-12 16:42:19, 6301
2024-03-12 16:42:24, 6557
2024-03-12 16:42:29, 6547
2024-03-12 16:42:34, 7086
2024-03-12 16:42:39, 7084
2024-03-12 16:42:44, 7071
2024-03-12 16:42:49, 7072
2024-03-12 16:42:54, 7088
2024-03-12 16:42:59, 7588
2024-03-12 16:43:04, 7588
2024-03-12 16:43:09, 7590
2024-03-12 16:43:14, 7590
2024-03-12 16:43:19, 7594
2024-03-12 16:43:24, 8107
2024-03-12 16:43:29, 8111
2024-03-12 16:43:34, 8111
2024-03-12 16:43:39, 8113
2024-03-12 16:43:44, 8115
2024-03-12 16:43:49, 8627
2024-03-12 16:43:54, 8631
2024-03-12 16:43:59, 8633
2024-03-12 16:44:04, 8635
2024-03-12 16:44:09, 8637
2024-03-12 16:44:15, 9149
2024-03-12 16:44:20, 9153
2024-03-12 16:44:25, 9117
2024-03-12 16:44:30, 9121
2024-03-12 16:44:35, 9115
2024-03-12 16:44:40, 9373
2024-03-12 16:44:45, 9633
2024-03-12 16:44:50, 9633
2024-03-12 16:44:55, 9639
2024-03-12 16:45:00, 9639
2024-03-12 16:45:05, 9897
2024-03-12 16:45:10, 10155
2024-03-12 16:45:15, 10157
2024-03-12 16:45:20, 10159
2024-03-12 16:45:25, 10161
2024-03-12 16:45:30, 10419
2024-03-12 16:45:35, 10675
2024-03-12 16:45:40, 10677
2024-03-12 16:45:45, 10680
2024-03-12 16:45:50, 10684
2024-03-12 16:45:55, 10942
2024-03-12 16:46:00, 11198
2024-03-12 16:46:05, 11199
2024-03-12 16:46:10, 11201
2024-03-12 16:46:15, 11203
2024-03-12 16:46:20, 11205
2024-03-12 16:46:25, 11717
2024-03-12 16:46:30, 11721
2024-03-12 16:46:35, 11721
2024-03-12 16:46:40, 11723
2024-03-12 16:46:45, 11725
2024-03-12 16:46:50, 12237
2024-03-12 16:46:55, 12240
2024-03-12 16:47:00, 12240
2024-03-12 16:47:05, 12242
2024-03-12 16:47:10, 12244
2024-03-12 16:47:15, 12503
2024-03-12 16:47:20, 12761
2024-03-12 16:47:25, 12761
2024-03-12 16:47:30, 12763
2024-03-12 16:47:35, 12765
2024-03-12 16:47:40, 13023
2024-03-12 16:47:45, 13281
2024-03-12 16:47:50, 13283
2024-03-12 16:47:55, 13283
2024-03-12 16:48:00, 13285
2024-03-12 16:48:05, 13543
2024-03-12 16:48:10, 13545
2024-03-12 16:48:15, 13803
2024-03-12 16:48:20, 13805
2024-03-12 16:48:25, 13807
2024-03-12 16:48:30, 14063
2024-03-12 16:48:35, 14321
2024-03-12 16:48:40, 14323
2024-03-12 16:48:45, 14325
2024-03-12 16:48:50, 14329
2024-03-12 16:48:55, 14331
2024-03-12 16:49:00, 14587
2024-03-12 16:49:05, 14845
2024-03-12 16:49:10, 14847
2024-03-12 16:49:15, 14849
2024-03-12 16:49:20, 14851
2024-03-12 16:49:25, 15064
2024-03-12 16:49:30, 15323
2024-03-12 16:49:35, 15323
2024-03-12 16:49:40, 15325
2024-03-12 16:49:45, 15327
2024-03-12 16:49:50, 15585
2024-03-12 16:49:55, 15843
2024-03-12 16:50:00, 15843
2024-03-12 16:50:05, 15806
2024-03-12 16:50:10, 15808
2024-03-12 16:50:15, 16067
2024-03-12 16:50:20, 16325
2024-03-12 16:50:26, 16327
2024-03-12 16:50:31, 16327
2024-03-12 16:50:36, 16329
2024-03-12 16:50:41, 16331
2024-03-12 16:50:46, 16589
2024-03-12 16:50:51, 16997
2024-03-12 16:50:56, 17127
2024-03-12 16:51:01, 17385
2024-03-12 16:51:06, 17387
2024-03-12 16:51:11, 17645
2024-03-12 16:51:16, 17647
2024-03-12 16:51:21, 17905
2024-03-12 16:51:26, 17907
2024-03-12 16:51:31, 17909
2024-03-12 16:51:36, 18166
2024-03-12 16:51:41, 18168
2024-03-12 16:51:46, 18426
2024-03-12 16:51:51, 18428
2024-03-12 16:51:56, 18686
2024-03-12 16:52:01, 18688
2024-03-12 16:52:06, 18946
2024-03-12 16:52:11, 18948
2024-03-12 16:52:16, 18948
2024-03-12 16:52:21, 18955
2024-03-12 16:52:26, 18954
2024-03-12 16:52:31, 18956
2024-03-12 16:52:36, 19212
2024-03-12 16:52:41, 19470
2024-03-12 16:52:46, 19474
2024-03-12 16:52:51, 19476
2024-03-12 16:52:56, 19479
2024-03-12 16:53:01, 19480
2024-03-12 16:53:06, 19737
2024-03-12 16:53:11, 19995
2024-03-12 16:53:16, 19996
2024-03-12 16:53:21, 19999
2024-03-12 16:53:26, 20001

These are during training.

[vedantdave97]

Hello. Can you please see this issue. We are waiting for it and the deadline for the competition is approaching.

[callmeray]

Hi, we're working on it. We also observe memory increasing, but the increasing speed on our machine is much slower than that in your log. You can first try reducing the buffer size.