Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

使用paddle3D进行Centerpoint模型在KITTI数据集上的训练报错 #462

Open
Derbins opened this issue Mar 27, 2024 · 4 comments
Open

使用paddle3D进行Centerpoint模型在KITTI数据集上的训练报错 #462

Derbins opened this issue Mar 27, 2024 · 4 comments
Assignees
@wangna11BD

Comments

@Derbins
Copy link

Derbins commented Mar 27, 2024

环境：按照教程依次安装paddle、centerpoint、无错误，本地服务器单卡3090
问题：官方给的训练代码是8张3090,我这只有一张3090所以我输入的训练代码是python -m paddle.distributed.launch --gpus 0 tools/train.py --config configs/centerpoint/centerpoint_pillars_016voxel_kitti.yml --num_workers 4 --save_interval 5
原代码是python -m paddle.distributed.launch --gpus 0,1,2,3,4,5,6,7 tools/train.py --config configs/centerpoint/centerpoint_pillars_016voxel_kitti.yml --num_workers 4 --save_interval 5
报如下错误：

(paddle_env) root@control:/home/test/xbin/project/Paddle3D# python -m paddle.distributed.launch --gpus 0 tools/train.py --config configs/centerpoint/centerpoint_pillars_016voxel_kitti.yml --num_workers 4 --save_interval 5
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('mpl_toolkits').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
LAUNCH INFO 2024-03-27 15:28:00,434 ----------- Configuration ----------------------
LAUNCH INFO 2024-03-27 15:28:00,435 devices: 0
LAUNCH INFO 2024-03-27 15:28:00,435 elastic_level: -1
LAUNCH INFO 2024-03-27 15:28:00,435 elastic_timeout: 30
LAUNCH INFO 2024-03-27 15:28:00,435 gloo_port: 6767
LAUNCH INFO 2024-03-27 15:28:00,435 host: None
LAUNCH INFO 2024-03-27 15:28:00,435 ips: None
LAUNCH INFO 2024-03-27 15:28:00,435 job_id: default
LAUNCH INFO 2024-03-27 15:28:00,435 legacy: False
LAUNCH INFO 2024-03-27 15:28:00,435 log_dir: log
LAUNCH INFO 2024-03-27 15:28:00,435 log_level: INFO
LAUNCH INFO 2024-03-27 15:28:00,435 master: None
LAUNCH INFO 2024-03-27 15:28:00,435 max_restart: 3
LAUNCH INFO 2024-03-27 15:28:00,435 nnodes: 1
LAUNCH INFO 2024-03-27 15:28:00,435 nproc_per_node: None
LAUNCH INFO 2024-03-27 15:28:00,435 rank: -1
LAUNCH INFO 2024-03-27 15:28:00,435 run_mode: collective
LAUNCH INFO 2024-03-27 15:28:00,435 server_num: None
LAUNCH INFO 2024-03-27 15:28:00,435 servers:
LAUNCH INFO 2024-03-27 15:28:00,435 start_port: 6070
LAUNCH INFO 2024-03-27 15:28:00,435 trainer_num: None
LAUNCH INFO 2024-03-27 15:28:00,435 trainers:
LAUNCH INFO 2024-03-27 15:28:00,435 training_script: tools/train.py
LAUNCH INFO 2024-03-27 15:28:00,435 training_script_args: ['--config', 'configs/centerpoint/centerpoint_pillars_016voxel_kitti.yml', '--num_workers', '4', '--save_interval', '5']
LAUNCH INFO 2024-03-27 15:28:00,435 with_gloo: 1
LAUNCH INFO 2024-03-27 15:28:00,435 --------------------------------------------------
LAUNCH INFO 2024-03-27 15:28:00,436 Job: default, mode collective, replicas 1[1:1], elastic False
LAUNCH INFO 2024-03-27 15:28:00,437 Run Pod: lymytw, replicas 1, status ready
LAUNCH INFO 2024-03-27 15:28:00,458 Watching Pod: lymytw, replicas 1, status running
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('mpl_toolkits').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
Traceback (most recent call last):
File "tools/train.py", line 266, in
main(args)
File "tools/train.py", line 165, in main
logger = Logger(output=args.save_dir)
TypeError: init() got an unexpected keyword argument 'output'
LAUNCH INFO 2024-03-27 15:28:03,463 Pod failed
LAUNCH ERROR 2024-03-27 15:28:03,464 Container failed !!!
Container rank 0 status failed cmd ['/home/test/anaconda3/envs/paddle_env/bin/python', '-u', 'tools/train.py', '--config', 'configs/centerpoint/centerpoint_pillars_016voxel_kitti.yml', '--num_workers', '4', '--save_interval', '5'] code 1 log log/workerlog.0
env {'SHELL': '/bin/bash', 'COLORTERM': 'truecolor', 'TERM_PROGRAM_VERSION': '1.87.2', 'CONDA_EXE': '/home/test/anaconda3/bin/conda', '_CE_M': '', 'LANGUAGE': 'zh_CN:zh', 'LC_ADDRESS': 'zh_CN.UTF-8', 'LC_NAME': 'zh_CN.UTF-8', 'LC_MONETARY': 'zh_CN.UTF-8', 'PWD': '/home/test/xbin/project/Paddle3D', 'LOGNAME': 'root', 'XDG_SESSION_TYPE': 'tty', 'CONDA_PREFIX': '/home/test/anaconda3/envs/paddle_env', 'VSCODE_GIT_ASKPASS_NODE': '/root/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/server/node', 'MOTD_SHOWN': 'pam', 'HOME': '/root', 'LC_PAPER': 'zh_CN.UTF-8', 'LANG': 'zh_CN.UTF-8', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36:', 'SSL_CERT_DIR': '/usr/lib/ssl/certs', 'CONDA_PROMPT_MODIFIER': '(paddle_env) ', 'GIT_ASKPASS': '/root/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/server/extensions/git/dist/askpass.sh', 'SSH_CONNECTION': '10.10.97.10 57540 10.10.97.170 22', 'VSCODE_GIT_ASKPASS_EXTRA_ARGS': '', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'TERM': 'xterm-256color', 'LC_IDENTIFICATION': 'zh_CN.UTF-8', 'CE_CONDA': '', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'LIBVIRT_DEFAULT_URI': 'qemu:///system', 'USER': 'root', 'VSCODE_GIT_IPC_HANDLE': '/run/user/0/vscode-git-ee71e6b990.sock', 'CONDA_SHLVL': '2', 'DISPLAY': 'localhost:10.0', 'SHLVL': '2', 'LC_TELEPHONE': 'zh_CN.UTF-8', 'LC_MEASUREMENT': 'zh_CN.UTF-8', 'XDG_SESSION_ID': '1450', 'CONDA_PYTHON_EXE': '/home/test/anaconda3/bin/python', 'XDG_RUNTIME_DIR': '/run/user/0', 'SSL_CERT_FILE': '/usr/lib/ssl/certs/ca-certificates.crt', 'SSH_CLIENT': '10.10.97.10 57540 22', 'CONDA_DEFAULT_ENV': 'paddle_env', 'LC_TIME': 'zh_CN.UTF-8', 'VSCODE_GIT_ASKPASS_MAIN': '/root/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/server/extensions/git/dist/askpass-main.js', 'XDG_DATA_DIRS': '/usr/local/share:/usr/share:/var/lib/snapd/desktop', 'BROWSER': '/root/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/server/bin/helpers/browser.sh', 'PATH': '/home/test/anaconda3/envs/paddle_env/bin:/root/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/server/bin/remote-cli:/home/test/anaconda3/bin:/home/test/anaconda3/condabin:/home/test/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/usr/local/cuda/bin:/home/test/anaconda3/bin:/home/test/anaconda3/condabin:/home/test/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/usr/local/cuda/bin:/home/test/anaconda3/bin:/root/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/server/bin/remote-cli:/home/test/anaconda3/bin:/home/test/anaconda3/condabin:/home/test/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/usr/local/cuda/bin:/home/test/anaconda3/bin:/home/test/anaconda3/condabin:/home/test/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/0/bus', 'CONDA_PREFIX_1': '/home/test/anaconda3', 'LC_NUMERIC': 'zh_CN.UTF-8', 'TERM_PROGRAM': 'vscode', 'VSCODE_IPC_HOOK_CLI': '/run/user/0/vscode-ipc-8ba4616d-1325-4d99-8331-45315ed73acd.sock', '': '/home/test/anaconda3/envs/paddle_env/bin/python', 'OLDPWD': '/home/test/xbin/project/Paddle3D/datasets/KITTI', 'CUSTOM_DEVICE_ROOT': '', 'OMP_NUM_THREADS': '1', 'QT_QPA_PLATFORM_PLUGIN_PATH': '/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/cv2/qt/plugins', 'QT_QPA_FONTDIR': '/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/cv2/qt/fonts', 'LD_LIBRARY_PATH': '/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/cv2/../../lib64:', 'POD_NAME': 'lymytw', 'PADDLE_MASTER': '172.17.0.1:50515', 'PADDLE_GLOBAL_SIZE': '1', 'PADDLE_LOCAL_SIZE': '1', 'PADDLE_GLOBAL_RANK': '0', 'PADDLE_LOCAL_RANK': '0', 'PADDLE_NNODES': '1', 'PADDLE_TRAINER_ENDPOINTS': '172.17.0.1:50516', 'PADDLE_CURRENT_ENDPOINT': '172.17.0.1:50516', 'PADDLE_TRAINER_ID': '0', 'PADDLE_TRAINERS_NUM': '1', 'PADDLE_RANK_IN_NODE': '0', 'FLAGS_selected_gpus': '0'}
LAUNCH INFO 2024-03-27 15:28:03,464 ------------------------- ERROR LOG DETAIL -------------------------
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('mpl_toolkits').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/home/test/anaconda3/envs/paddle_env/lib/python3.8/site-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google').
Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
Traceback (most recent call last):
File "tools/train.py", line 266, in
main(args)
File "tools/train.py", line 165, in main
logger = Logger(output=args.save_dir)
TypeError: init() got an unexpected keyword argument 'output'
LAUNCH INFO 2024-03-27 15:28:03,464 Exit code 1

求解答感谢～
@chenjian88888
Copy link

单张显卡不要-m paddle.distributed.launch --gpus 0,1,2这些试试

@yaobaishen
Copy link

我直接用的Apllo 基于 Paddle3D 的工程：https://github.com/ApolloAuto/apollo-model-centerpoint
也是基于Paddle3D的，这样可以单卡训练：

python tools/train.py --config 
configs/centerpoint/centerpoint_pillars_016voxel_kitti.yml 
 --log_interval 10 --save_interval 10

@Derbins
Copy link
Author

Derbins commented May 27, 2024 via email

@yaobaishen
Copy link

@Derbins 麻烦问下你试过自定义的kitti format dataset的训练吗？我用kitti dataset训练没问题，但是自定义dataset训练会报错，#466

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wangna11BD wangna11BD

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yaobaishen @wangna11BD @Derbins @chenjian88888