RuntimeError::CUDA_ERROR_OUT_OF_MEMORY ; Could not free memory:CUDA_ERROR_INVALID_VALUE

[PegaXuX]

Thanks for your work!

I use server with TITANGPU 12GB and can use neus2 through python bindings (vscode ssh terminal
but when using rustdesk to run gui mode i met the error below
when the remote desktop is on, i can't even run it without gui (both ssh terminal and rustdesk
close rustdesk and everything is ok
how can i use gui mode on server

RuntimeError: /share/program/jjy/NeuS2/dependencies/neus2_tcnn/include/tiny-cuda-nn/gpu_memory.h:556 cuMemCreate(&m_handles.back(), n_bytes_to_allocate, &prop, 0) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: /share/program/jjy/NeuS2/dependencies/neus2_tcnn/include/tiny-cuda-nn/gpu_memory.h:462 cuMemRelease(handle) failed with error CUDA_ERROR_INVALID_VALUE

[NyxAither]

I have a similar issue, though I'm not doing anything with remote desktop - just running locally on my RTX3060 6GB. My images are 1920x1200 grayscale .jpgs, and I can run instant-ngp with 400 images on my PC. With NeuS2 (currently using the neuspp branch) I have been able to very rarely run 100-400 images but it only runs consistently with <10 images. Usually I get the error below.

I haven't discovered any obvious pattern, other than setting aabb_scale to 16 seems to give the greatest chance of success. I assume the developer will say I don't have enough VRAM, but it's very odd that it does work sometimes on 100+ images. I've also watched my GPU utilization closely while I do all this, and haven't learned anything interesting. Basically anytime I get NeuS2 to run successfully it uses my entire GPU, regardless of how many images I've loaded. The amount of memory NeuS2 claims to be using at the top of the GUI does change as you'd expect though.

Thank you!

13:22:36 ERROR Uncaught exception: C:\Users\xxxxx\NeuS2\dependencies\neus2_tcnn\include\tiny-cuda-nn/gpu_memory.h:564 cuMemSetAccess(m_base_address + m_size, n_bytes_to_allocate, &access_desc, 1) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: C:\Users\xxxxx\NeuS2\dependencies\neus2_tcnn\include\tiny-cuda-nn/gpu_memory.h:465 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE