Use pudb inside of an apptainer / Docker container

[j93hahn]

Some context: I am using tmux, and have my TERM env variable set to tmux-256color.

Now, I want to use pudb inside of an apptainer environment. If I do something like pudb script.py, the editor looks normal and I have no problems.

But if I use either pudb.set_trace() or breakpoint() inside of the script itself and run it, the editor turns into something like this:

I guess I am okay with using pudb and then manually setting the breakpoint inside pudb to the line I want, but this would get very tedious if I have a large codebase that spans dozens of files, and would have to traverse all those files just by using the "n" and "s" keys inside pudb's editor. I only get this issue when I am inside of an apptainer, that is itself inside of a tmux environment on a GPU, which I remotely access via SSH. If I just use a standard conda environment, no problem but conda is a terrible solution for my use case.

Are there any ideas or insights on how to fix this? Thanks!

[mvanderkamp-cmb]

[j93hahn]

It's both issues!

[mvanderkamp]

Whoops sorry posted that from the wrong account! Here's the one I normally use...

Anyway, this will probably be something with the environment. How is your python script invoked when you're using pudb.set_trace()?

[j93hahn]

I am currently just calling it as python MY_SCRIPT.py and have pudb.set_trace() in the code.
Here is a list of all the environment variables inside of the apptainer:

SLURM_CLUSTER_NAME: beehive
NV_LIBNPP_VERSION: 12.2.3.2-1
SLURM_CPUS_ON_NODE: 2
SINGULARITY_ENVIRONMENT: /.singularity.d/env/91-environment.sh
NCCL_VERSION: 2.20.3-1
SLURM_UMASK: 0022
SLURM_JOB_CPUS_PER_NODE: 2
SLURM_GPUS_ON_NODE: 1
PWD: /share/data/2pals/jjahn/exp/gaussian-splatting/test
SLURM_GTIDS: 0
LOGNAME: jjahn
XDG_SESSION_TYPE: tty
NV_CUDNN_PACKAGE: libcudnn9-cuda-12=9.0.0.312-1
CONDA_PREFIX: /conda_tmp/mc3
SLURM_JOB_PARTITION: whc-own-gpu
NVIDIA_DRIVER_CAPABILITIES: compute,utility
NV_NVPROF_DEV_PACKAGE: cuda-nvprof-12-3=12.3.101-1
NV_LIBNPP_PACKAGE: libnpp-12-3=12.2.3.2-1
NV_LIBNCCL_DEV_PACKAGE_NAME: libnccl-dev
SLURM_JOB_NUM_NODES: 1
SRUN_DEBUG: 3
SLURM_STEPID: 0
TZ: America/Chicago

APPTAINER_APPNAME:
SLURM_PTY_WIN_ROW: 24
HOME: /home-nfs/jjahn
LANG: en_US.UTF-8
APPTAINER_COMMAND: shell
SLURM_PROCID: 0
CUDA_VERSION: 12.3.2
SINGULARITY_CONTAINER: /share/data/2pals/jjahn/apptainers/3dgs.sif
NV_LIBCUBLAS_PACKAGE: libcublas-12-3=12.3.4.1-1
NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE: cuda-nsight-compute-12-3=12.3.2-1
CONDA_PROMPT_MODIFIER: (base)
TMPDIR: /tmp
LC_TERMINAL: iTerm2
SLURM_NTASKS: 1
SLURM_TOPOLOGY_ADDR: priv-g4
SSH_CONNECTION: 10.150.24.61 55912 128.135.221.190 22
NV_LIBNPP_DEV_PACKAGE: libnpp-dev-12-3=12.2.3.2-1
PIP_CACHE_DIR: /share/data/2pals/jjahn/pip_cache
NV_LIBCUBLAS_PACKAGE_NAME: libcublas-12-3
NV_LIBNPP_DEV_VERSION: 12.2.3.2-1
SLURM_TOPOLOGY_ADDR_PATTERN: node
CUDA_VISIBLE_DEVICES: 0
SLURM_SRUN_COMM_HOST: 128.135.221.190
XDG_SESSION_CLASS: user
APPTAINER_CONTAINER: /share/data/2pals/jjahn/apptainers/3dgs.sif
SLURM_SCRIPT_CONTEXT: prolog_task
SLURM_WORKING_CLUSTER: beehive:beehive:6817:9728:109
TERM: tmux-256color
NV_LIBCUSPARSE_DEV_VERSION: 12.2.0.103-1
_CE_CONDA:
SLURM_PTY_WIN_COL: 80
TORCH_USE_CUDA_DSA: 1
USER: jjahn
TMUX_PANE: %0
LIBRARY_PATH: /usr/local/cuda/lib64/stubs
SLURM_NODELIST: priv-g4
NV_CUDNN_VERSION: 9.0.0.312
CONDA_SHLVL: 2
SLURM_SRUN_COMM_PORT: 35337
LC_TERMINAL_VERSION: 3.4.23
SLURM_STEP_ID: 0
SLURM_JOB_ACCOUNT: jjahn
SLURM_PRIO_PROCESS: 0
SLURM_NPROCS: 1
SHLVL: 4
SLURM_NNODES: 1

NV_CUDA_COMPAT_PACKAGE: cuda-compat-12-3
APPTAINER_BIND: /share
CONDA_PYTHON_EXE: /conda_tmp/mc3/bin/python
NV_LIBNCCL_PACKAGE: libnccl2=2.20.3-1+cuda12.3
LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
XDG_RUNTIME_DIR: /run/user/1397
SLURM_JOB_ID: 34696
SLURM_NODEID: 0
PYTHONBREAKPOINT: pudb.set_trace
PS1: (base) Apptainer>
SLURM_STEP_NUM_NODES: 1
SSH_CLIENT: 10.150.24.61 55912 22
CONDA_DEFAULT_ENV: base
NV_CUDA_NSIGHT_COMPUTE_VERSION: 12.3.2-1
NV_NVPROF_VERSION: 12.3.101-1
SLURM_STEP_TASKS_PER_NODE: 1
SLURM_CONF: /etc/slurm/slurm.conf
PATH: /conda_tmp/mc3/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SLURM_JOB_NAME: bash
NV_LIBNCCL_PACKAGE_NAME: libnccl2
NV_LIBNCCL_PACKAGE_VERSION: 2.20.3-1
DBUS_SESSION_BUS_ADDRESS: unix:path=/run/user/1397/bus
SSH_TTY: /dev/pts/16
CONDA_PREFIX_1: /share/data/2pals/jjahn/mc3
SLURM_STEP_LAUNCHER_PORT: 35337
SLURM_JOB_GID: 2002
DEBIAN_FRONTEND: noninteractive
OLDPWD: /share/data/2pals/jjahn/exp/gaussian-splatting
SLURM_JOB_NODELIST: priv-g4
TERM_PROGRAM: tmux
_: /conda_tmp/mc3/bin/python
LC_CTYPE: C.UTF-8

I have TERM and PYTHONBREAKPOINT configured, but maybe there is another environment flag I am missing. Thanks!

[mvanderkamp]

🤔 So pudb script.py works but python script.py with pudb.set_trace() doesn't?

Can you check if there's a mismatch between the pudb versions- the one being executed by your shell and the one being imported by python?

pudb --version
pip list | grep pudb

[j93hahn]

Yes, they are the same versions for me!

[mvanderkamp]

Sorry I've been away from this for a while. What about python -m pudb --version?