Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix package clobbering #52

Open
jakirkham opened this issue Nov 6, 2024 · 2 comments
Open

Fix package clobbering #52

jakirkham opened this issue Nov 6, 2024 · 2 comments
Assignees

Comments

@jakirkham
Copy link
Member

jakirkham commented Nov 6, 2024

Some packages appear to install content (like license files) in the $PREFIX without naming them or placing them in a way to avoid clobbering between packages (for example: conda-forge/cuda-nvvp-feedstock#14 (comment) )

We should work to remove any clobbering issues between CUDA Toolkit packages

When it comes to license files specifically, we already make sure to include these in all packages with about/license_file (for example). So there is no need to duplicate this effort. So with license files specifically, we should not copy them to the $PREFIX, which should fix that particular kind of clobbering

More generally we may just want to adapt scripts to only copy directories

If we need to copy over any files, we likely will need to do so in a clearly specified way

Other clobbering issues can be handled on a case by case basis

@jakirkham
Copy link
Member Author

jakirkham commented Nov 6, 2024

To help identify which packages have clobbering issues, tried creating an environment with the cuda package and clobbering warnings enabled

Here is the full output:

$ CONDA_PATH_CONFLICT="warn" conda create -n cuda cuda
Channels:
 - conda-forge
Platform: linux-aarch64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/cuda

  added / updated specs:
    - cuda


The following NEW packages will be INSTALLED:

  _openmp_mutex      conda-forge/linux-aarch64::_openmp_mutex-4.5-2_gnu 
  alsa-lib           conda-forge/linux-aarch64::alsa-lib-1.2.12-h68df207_0 
  attr               conda-forge/linux-aarch64::attr-2.5.1-h4e544f5_1 
  binutils           conda-forge/linux-aarch64::binutils-2.43-hf1166c9_2 
  binutils_impl_lin~ conda-forge/linux-aarch64::binutils_impl_linux-aarch64-2.43-h4c662bb_2 
  binutils_linux-aa~ conda-forge/linux-aarch64::binutils_linux-aarch64-2.43-hf1166c9_2 
  bzip2              conda-forge/linux-aarch64::bzip2-1.0.8-h68df207_7 
  c-compiler         conda-forge/linux-aarch64::c-compiler-1.8.0-h6561dab_1 
  ca-certificates    conda-forge/linux-aarch64::ca-certificates-2024.8.30-hcefe29a_0 
  cuda               conda-forge/noarch::cuda-12.6.2-ha804496_0 
  cuda-cccl_linux-a~ conda-forge/noarch::cuda-cccl_linux-aarch64-12.6.77-h579c4fd_0 
  cuda-command-line~ conda-forge/linux-aarch64::cuda-command-line-tools-12.6.2-h8af1aa0_0 
  cuda-compiler      conda-forge/noarch::cuda-compiler-12.6.2-hbad6d8a_0 
  cuda-crt-dev_linu~ conda-forge/noarch::cuda-crt-dev_linux-aarch64-12.6.77-h579c4fd_1 
  cuda-crt-tools     conda-forge/linux-aarch64::cuda-crt-tools-12.6.77-h579c4fd_1 
  cuda-cudart        conda-forge/linux-aarch64::cuda-cudart-12.6.77-h3ae8b8a_0 
  cuda-cudart-dev    conda-forge/linux-aarch64::cuda-cudart-dev-12.6.77-h3ae8b8a_0 
  cuda-cudart-dev_l~ conda-forge/noarch::cuda-cudart-dev_linux-aarch64-12.6.77-h3ae8b8a_0 
  cuda-cudart-static conda-forge/linux-aarch64::cuda-cudart-static-12.6.77-h3ae8b8a_0 
  cuda-cudart-stati~ conda-forge/noarch::cuda-cudart-static_linux-aarch64-12.6.77-h3ae8b8a_0 
  cuda-cudart_linux~ conda-forge/noarch::cuda-cudart_linux-aarch64-12.6.77-h3ae8b8a_0 
  cuda-cuobjdump     conda-forge/linux-aarch64::cuda-cuobjdump-12.6.77-h05609ea_0 
  cuda-cupti         conda-forge/linux-aarch64::cuda-cupti-12.6.80-h5101a13_0 
  cuda-cupti-dev     conda-forge/linux-aarch64::cuda-cupti-dev-12.6.80-h3ae8b8a_0 
  cuda-cuxxfilt      conda-forge/linux-aarch64::cuda-cuxxfilt-12.6.77-h05609ea_1 
  cuda-driver-dev    conda-forge/linux-aarch64::cuda-driver-dev-12.6.77-h3ae8b8a_0 
  cuda-driver-dev_l~ conda-forge/noarch::cuda-driver-dev_linux-aarch64-12.6.77-h3ae8b8a_0 
  cuda-gdb           conda-forge/linux-aarch64::cuda-gdb-12.6.77-h704f0fb_1 
  cuda-libraries     conda-forge/linux-aarch64::cuda-libraries-12.6.2-h8af1aa0_0 
  cuda-libraries-dev conda-forge/linux-aarch64::cuda-libraries-dev-12.6.2-h8af1aa0_0 
  cuda-nvcc          conda-forge/linux-aarch64::cuda-nvcc-12.6.77-ha346c71_2 
  cuda-nvcc-dev_lin~ conda-forge/noarch::cuda-nvcc-dev_linux-aarch64-12.6.77-h4310d6a_1 
  cuda-nvcc-impl     conda-forge/linux-aarch64::cuda-nvcc-impl-12.6.77-h614329b_1 
  cuda-nvcc-tools    conda-forge/linux-aarch64::cuda-nvcc-tools-12.6.77-h614329b_1 
  cuda-nvcc_linux-a~ conda-forge/linux-aarch64::cuda-nvcc_linux-aarch64-12.6.77-hf9514e2_2 
  cuda-nvdisasm      conda-forge/linux-aarch64::cuda-nvdisasm-12.6.77-h5101a13_1 
  cuda-nvml-dev      conda-forge/linux-aarch64::cuda-nvml-dev-12.6.77-h5101a13_1 
  cuda-nvprune       conda-forge/linux-aarch64::cuda-nvprune-12.6.77-h05609ea_1 
  cuda-nvrtc         conda-forge/linux-aarch64::cuda-nvrtc-12.6.77-h5101a13_0 
  cuda-nvrtc-dev     conda-forge/linux-aarch64::cuda-nvrtc-dev-12.6.77-h3ae8b8a_0 
  cuda-nvtx          conda-forge/linux-aarch64::cuda-nvtx-12.6.77-h5101a13_0 
  cuda-nvvm-dev_lin~ conda-forge/noarch::cuda-nvvm-dev_linux-aarch64-12.6.77-h579c4fd_1 
  cuda-nvvm-impl     conda-forge/linux-aarch64::cuda-nvvm-impl-12.6.77-h614329b_1 
  cuda-nvvm-tools    conda-forge/linux-aarch64::cuda-nvvm-tools-12.6.77-h614329b_1 
  cuda-profiler-api  conda-forge/linux-aarch64::cuda-profiler-api-12.6.77-h16bee8c_0 
  cuda-runtime       conda-forge/noarch::cuda-runtime-12.6.2-ha804496_0 
  cuda-sanitizer-api conda-forge/linux-aarch64::cuda-sanitizer-api-12.6.77-h4177691_0 
  cuda-toolkit       conda-forge/noarch::cuda-toolkit-12.6.2-ha804496_0 
  cuda-tools         conda-forge/linux-aarch64::cuda-tools-12.6.2-h8af1aa0_0 
  cuda-version       conda-forge/noarch::cuda-version-12.6-h7480c83_3 
  cuda-visual-tools  conda-forge/linux-aarch64::cuda-visual-tools-12.6.2-h8af1aa0_0 
  cxx-compiler       conda-forge/linux-aarch64::cxx-compiler-1.8.0-heb6c788_1 
  dbus               conda-forge/linux-aarch64::dbus-1.13.6-h12b9eeb_3 
  expat              conda-forge/linux-aarch64::expat-2.6.3-h5ad3122_0 
  font-ttf-dejavu-s~ conda-forge/noarch::font-ttf-dejavu-sans-mono-2.37-hab24e00_0 
  font-ttf-inconsol~ conda-forge/noarch::font-ttf-inconsolata-3.000-h77eed37_0 
  font-ttf-source-c~ conda-forge/noarch::font-ttf-source-code-pro-2.038-h77eed37_0 
  font-ttf-ubuntu    conda-forge/noarch::font-ttf-ubuntu-0.83-h77eed37_3 
  fontconfig         conda-forge/linux-aarch64::fontconfig-2.15.0-h8dda3cd_1 
  fonts-conda-ecosy~ conda-forge/noarch::fonts-conda-ecosystem-1-0 
  fonts-conda-forge  conda-forge/noarch::fonts-conda-forge-1-0 
  freetype           conda-forge/linux-aarch64::freetype-2.12.1-hf0a5ef3_2 
  gcc                conda-forge/linux-aarch64::gcc-13.3.0-h8a56e6e_1 
  gcc_impl_linux-aa~ conda-forge/linux-aarch64::gcc_impl_linux-aarch64-13.3.0-hcdea9b6_1 
  gcc_linux-aarch64  conda-forge/linux-aarch64::gcc_linux-aarch64-13.3.0-h1cd514b_5 
  gds-tools          conda-forge/linux-aarch64::gds-tools-1.11.1.6-h3ae8b8a_1 
  gettext            conda-forge/linux-aarch64::gettext-0.22.5-h0a1ffab_3 
  gettext-tools      conda-forge/linux-aarch64::gettext-tools-0.22.5-h0a1ffab_3 
  gmp                conda-forge/linux-aarch64::gmp-6.3.0-h0a1ffab_2 
  gxx                conda-forge/linux-aarch64::gxx-13.3.0-h8a56e6e_1 
  gxx_impl_linux-aa~ conda-forge/linux-aarch64::gxx_impl_linux-aarch64-13.3.0-h1211b58_1 
  gxx_linux-aarch64  conda-forge/linux-aarch64::gxx_linux-aarch64-13.3.0-h2864abd_5 
  icu                conda-forge/linux-aarch64::icu-75.1-hf9b3779_0 
  kernel-headers_li~ conda-forge/noarch::kernel-headers_linux-aarch64-4.18.0-h05a177a_18 
  keyutils           conda-forge/linux-aarch64::keyutils-1.6.1-h4e544f5_0 
  krb5               conda-forge/linux-aarch64::krb5-1.21.3-h50a48e9_0 
  ld_impl_linux-aar~ conda-forge/linux-aarch64::ld_impl_linux-aarch64-2.43-h80caac9_2 
  libasprintf        conda-forge/linux-aarch64::libasprintf-0.22.5-h87f4aca_3 
  libasprintf-devel  conda-forge/linux-aarch64::libasprintf-devel-0.22.5-h87f4aca_3 
  libcap             conda-forge/linux-aarch64::libcap-2.69-h883460d_0 
  libcublas          conda-forge/linux-aarch64::libcublas-12.6.3.3-h5101a13_1 
  libcublas-dev      conda-forge/linux-aarch64::libcublas-dev-12.6.3.3-h3ae8b8a_1 
  libcufft           conda-forge/linux-aarch64::libcufft-11.3.0.4-h5101a13_0 
  libcufft-dev       conda-forge/linux-aarch64::libcufft-dev-11.3.0.4-h3ae8b8a_0 
  libcufile          conda-forge/linux-aarch64::libcufile-1.11.1.6-h71dc0f0_1 
  libcufile-dev      conda-forge/linux-aarch64::libcufile-dev-1.11.1.6-h3ae8b8a_1 
  libcurand          conda-forge/linux-aarch64::libcurand-10.3.7.77-h5101a13_0 
  libcurand-dev      conda-forge/linux-aarch64::libcurand-dev-10.3.7.77-h3ae8b8a_0 
  libcusolver        conda-forge/linux-aarch64::libcusolver-11.7.1.2-h5101a13_0 
  libcusolver-dev    conda-forge/linux-aarch64::libcusolver-dev-11.7.1.2-h3ae8b8a_0 
  libcusparse        conda-forge/linux-aarch64::libcusparse-12.5.4.2-h5101a13_0 
  libcusparse-dev    conda-forge/linux-aarch64::libcusparse-dev-12.5.4.2-h3ae8b8a_0 
  libedit            conda-forge/linux-aarch64::libedit-3.1.20191231-he28a2e2_2 
  libexpat           conda-forge/linux-aarch64::libexpat-2.6.3-h5ad3122_0 
  libffi             conda-forge/linux-aarch64::libffi-3.4.2-h3557bc0_5 
  libgcc             conda-forge/linux-aarch64::libgcc-14.2.0-he277a41_1 
  libgcc-devel_linu~ conda-forge/noarch::libgcc-devel_linux-aarch64-13.3.0-h0c07274_101 
  libgcc-ng          conda-forge/linux-aarch64::libgcc-ng-14.2.0-he9431aa_1 
  libgcrypt          conda-forge/linux-aarch64::libgcrypt-1.11.0-h68df207_1 
  libgettextpo       conda-forge/linux-aarch64::libgettextpo-0.22.5-h0a1ffab_3 
  libgettextpo-devel conda-forge/linux-aarch64::libgettextpo-devel-0.22.5-h0a1ffab_3 
  libglib            conda-forge/linux-aarch64::libglib-2.82.2-hc486b8e_0 
  libgomp            conda-forge/linux-aarch64::libgomp-14.2.0-he277a41_1 
  libgpg-error       conda-forge/linux-aarch64::libgpg-error-1.50-hb13efb6_0 
  libiconv           conda-forge/linux-aarch64::libiconv-1.17-h31becfc_2 
  libnl              conda-forge/linux-aarch64::libnl-3.10.0-h68df207_0 
  libnpp             conda-forge/linux-aarch64::libnpp-12.3.1.54-h3ae8b8a_0 
  libnpp-dev         conda-forge/linux-aarch64::libnpp-dev-12.3.1.54-h3ae8b8a_0 
  libnvfatbin        conda-forge/linux-aarch64::libnvfatbin-12.6.77-h5101a13_0 
  libnvfatbin-dev    conda-forge/linux-aarch64::libnvfatbin-dev-12.6.77-h3ae8b8a_0 
  libnvjitlink       conda-forge/linux-aarch64::libnvjitlink-12.6.77-h5101a13_1 
  libnvjitlink-dev   conda-forge/linux-aarch64::libnvjitlink-dev-12.6.77-h3ae8b8a_1 
  libnvjpeg          conda-forge/linux-aarch64::libnvjpeg-12.3.3.54-h3ae8b8a_0 
  libnvjpeg-dev      conda-forge/linux-aarch64::libnvjpeg-dev-12.3.3.54-h579c4fd_0 
  libpng             conda-forge/linux-aarch64::libpng-1.6.44-hc4a20ef_0 
  libsanitizer       conda-forge/linux-aarch64::libsanitizer-13.3.0-ha58e236_1 
  libsqlite          conda-forge/linux-aarch64::libsqlite-3.47.0-hc4a20ef_1 
  libstdcxx          conda-forge/linux-aarch64::libstdcxx-14.2.0-h3f4de04_1 
  libstdcxx-devel_l~ conda-forge/noarch::libstdcxx-devel_linux-aarch64-13.3.0-h0c07274_101 
  libstdcxx-ng       conda-forge/linux-aarch64::libstdcxx-ng-14.2.0-hf1166c9_1 
  libsystemd0        conda-forge/linux-aarch64::libsystemd0-256.7-hd54d049_1 
  libudev1           conda-forge/linux-aarch64::libudev1-256.7-h86ecc28_1 
  libuuid            conda-forge/linux-aarch64::libuuid-2.38.1-hb4cce97_0 
  libxcb             conda-forge/linux-aarch64::libxcb-1.17.0-h262b8f6_0 
  libxkbcommon       conda-forge/linux-aarch64::libxkbcommon-1.7.0-h46f2afe_1 
  libxkbfile         conda-forge/linux-aarch64::libxkbfile-1.1.0-h4e544f5_1 
  libxml2            conda-forge/linux-aarch64::libxml2-2.13.4-hf4efe5d_2 
  libzlib            conda-forge/linux-aarch64::libzlib-1.3.1-h86ecc28_2 
  lz4-c              conda-forge/linux-aarch64::lz4-c-1.9.4-hd600fc2_0 
  ncurses            conda-forge/linux-aarch64::ncurses-6.5-hcccb83c_1 
  nsight-compute     conda-forge/linux-aarch64::nsight-compute-2024.3.2.3-h223ec0a_0 
  nspr               conda-forge/linux-aarch64::nspr-4.36-h5ad3122_0 
  nss                conda-forge/linux-aarch64::nss-3.106-hcffee33_0 
  openssl            conda-forge/linux-aarch64::openssl-3.3.2-h86ecc28_0 
  pcre2              conda-forge/linux-aarch64::pcre2-10.44-h070dd5b_2 
  pthread-stubs      conda-forge/linux-aarch64::pthread-stubs-0.4-h86ecc28_1002 
  rdma-core          conda-forge/linux-aarch64::rdma-core-54.0-h1d056c8_1 
  sysroot_linux-aar~ conda-forge/noarch::sysroot_linux-aarch64-2.17-h5b4a56d_18 
  tzdata             conda-forge/noarch::tzdata-2024b-hc8b5060_0 
  wayland            conda-forge/linux-aarch64::wayland-1.23.1-h698ed42_0 
  xcb-util           conda-forge/linux-aarch64::xcb-util-0.4.1-h5c728e9_2 
  xcb-util-cursor    conda-forge/linux-aarch64::xcb-util-cursor-0.1.5-h86ecc28_0 
  xcb-util-image     conda-forge/linux-aarch64::xcb-util-image-0.4.0-h5c728e9_2 
  xcb-util-keysyms   conda-forge/linux-aarch64::xcb-util-keysyms-0.4.1-h5c728e9_0 
  xcb-util-renderut~ conda-forge/linux-aarch64::xcb-util-renderutil-0.3.10-h5c728e9_0 
  xcb-util-wm        conda-forge/linux-aarch64::xcb-util-wm-0.4.2-h5c728e9_0 
  xkeyboard-config   conda-forge/linux-aarch64::xkeyboard-config-2.43-h86ecc28_0 
  xorg-libice        conda-forge/linux-aarch64::xorg-libice-1.1.1-h57736b2_1 
  xorg-libsm         conda-forge/linux-aarch64::xorg-libsm-1.2.4-hbac51e1_1 
  xorg-libx11        conda-forge/linux-aarch64::xorg-libx11-1.8.9-he755bbd_2 
  xorg-libxau        conda-forge/linux-aarch64::xorg-libxau-1.0.11-h86ecc28_1 
  xorg-libxcomposite conda-forge/linux-aarch64::xorg-libxcomposite-0.4.6-h86ecc28_2 
  xorg-libxdamage    conda-forge/linux-aarch64::xorg-libxdamage-1.1.6-h86ecc28_0 
  xorg-libxdmcp      conda-forge/linux-aarch64::xorg-libxdmcp-1.1.5-h57736b2_0 
  xorg-libxext       conda-forge/linux-aarch64::xorg-libxext-1.3.6-h57736b2_0 
  xorg-libxfixes     conda-forge/linux-aarch64::xorg-libxfixes-6.0.1-h57736b2_0 
  xorg-libxi         conda-forge/linux-aarch64::xorg-libxi-1.8.2-h57736b2_0 
  xorg-libxrandr     conda-forge/linux-aarch64::xorg-libxrandr-1.5.4-h86ecc28_0 
  xorg-libxrender    conda-forge/linux-aarch64::xorg-libxrender-0.9.11-h57736b2_1 
  xorg-libxtst       conda-forge/linux-aarch64::xorg-libxtst-1.2.5-h57736b2_3 
  xorg-xorgproto     conda-forge/linux-aarch64::xorg-xorgproto-2024.1-h86ecc28_1 
  xz                 conda-forge/linux-aarch64::xz-5.2.6-h9cdd2b7_0 
  zstd               conda-forge/linux-aarch64::zstd-1.5.6-h02f22dd_0 


Proceed ([y]/n)? 


Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: / 
ClobberWarning: This transaction has incompatible packages due to a shared path.
  packages: conda-forge/linux-aarch64::cuda-cuxxfilt-12.6.77-h05609ea_1, conda-forge/linux-aarch64::cuda-nvprune-12.6.77-h05609ea_1, conda-forge/linux-aarch64::cuda-sanitizer-api-12.6.77-h4177691_0, conda-forge/linux-aarch64::cuda-cuobjdump-12.6.77-h05609ea_0
  path: 'LICENSE'



done
Executing transaction: | 
ClobberWarning: Conda was asked to clobber an existing path.
  source path: /opt/conda/pkgs/cuda-nvprune-12.6.77-h05609ea_1/LICENSE
  target path: /opt/conda/envs/cuda/LICENSE




ClobberWarning: Conda was asked to clobber an existing path.
  source path: /opt/conda/pkgs/cuda-sanitizer-api-12.6.77-h4177691_0/LICENSE
  target path: /opt/conda/envs/cuda/LICENSE



/ 
ClobberWarning: Conda was asked to clobber an existing path.
  source path: /opt/conda/pkgs/cuda-cuobjdump-12.6.77-h05609ea_0/LICENSE
  target path: /opt/conda/envs/cuda/LICENSE



done
#
# To activate this environment, use
#
#     $ conda activate cuda
#
# To deactivate an active environment, use
#
#     $ conda deactivate

The following packages were flagged as having clobbering issues. All with the license file:

@jakirkham
Copy link
Member Author

libcublas has logic to place all other files in targets, which is what it uses for its own LICENSE file

So simply doing that consistently across feedstocks would be helpful

Maybe this is a good argument for consolidating CUDA Toolkit build scripts into one commonly used script: conda-forge/cf-nvidia-tools-feedstock#1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants