Skip to content

Commit

Permalink
Update CONTRIBUTING for pyproject-only builds (#14653)
Browse files Browse the repository at this point in the history 
Now that we no longer have setup.py, builds must be handled by a front-end like `pip install`.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #14653
  • Loading branch information
@vyasr
vyasr authored Dec 20, 2023
1 parent 57fb798 commit 96a1b00
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 102 deletions.
109 changes: 12 additions & 97 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -151,116 +151,31 @@ cd $CUDF_HOME
./build.sh --help
```

### Build, install and test cuDF libraries for contributors
#### Building for development

The general workflow is provided below. Please also see the last section about
[code formatting](#code-formatting).

#### `libcudf` (C++)

- If you're only interested in building the library (and not the unit tests):

```bash
cd $CUDF_HOME
./build.sh libcudf
```

- If, in addition, you want to build tests:
To build Python packages for development purposes, add the `--pydevelop` flag.
To build C++ tests, you can also request that build.sh build the `test` target.
To build all libraries and tests, with Python packages in development mode, simply run

```bash
./build.sh libcudf tests
./build.sh --pydevelop libcudf libcudf_kafka cudf dask_cudf cudf_kafka custreamz
```

- To run the tests:
To run the C++ tests, run

```bash
make test
ctest --test-dir ${CUDF_HOME}/cpp/build # libcudf
ctest --test-dir ${CUDF_HOME}/cpp/libcudf_kafka/build # libcudf_kafka
```

#### `cudf` (Python)

- First, build the `libcudf` C++ library following the steps above

- To build and install in edit/develop `cudf` Python package:
```bash
cd $CUDF_HOME/python/cudf
python setup.py build_ext --inplace
python setup.py develop
```
To run python tests, run

- To run `cudf` tests:
```bash
cd $CUDF_HOME/python
pytest -v cudf/cudf/tests
```

#### `dask-cudf` (Python)

- First, build the `libcudf` C++ and `cudf` Python libraries following the steps above

- To install the `dask-cudf` Python package in editable/develop mode:
```bash
cd $CUDF_HOME/python/dask_cudf
python setup.py build_ext --inplace
python setup.py develop
```

- To run `dask_cudf` tests:
```bash
cd $CUDF_HOME/python
pytest -v dask_cudf
```

#### `libcudf_kafka` (C++)

- If you're only interested in building the library (and not the unit tests):

```bash
cd $CUDF_HOME
./build.sh libcudf_kafka
```

- If, in addition, you want to build tests:

```bash
./build.sh libcudf_kafka tests
```

- To run the tests:

```bash
make test
```

#### `cudf-kafka` (Python)

- First, build the `libcudf` and `libcudf_kafka` libraries following the steps above

- To install the `cudf-kafka` Python package in editable/develop mode:

```bash
cd $CUDF_HOME/python/cudf_kafka
python setup.py build_ext --inplace
python setup.py develop
```

#### `custreamz` (Python)

- First, build `libcudf`, `libcudf_kafka`, and `cudf_kafka` following the steps above

- To install the `custreamz` Python package in editable/develop mode:

```bash
cd $CUDF_HOME/python/custreamz
python setup.py build_ext --inplace
python setup.py develop
```

- To run `custreamz` tests :

```bash
cd $CUDF_HOME/python
pytest -v custreamz
pytest -v ${CUDF_HOME}/python/cudf/cudf/tests
pytest -v ${CUDF_HOME}/python/dask_cudf/dask_cudf/ # There are tests in both tests/ and io/tests/
pytest -v ${CUDF_HOME}/python/custreamz/custreamz/tests
```

#### `cudf` (Java):
Expand Down
15 changes: 10 additions & 5 deletions build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ ARGS=$*
# script, and that this script resides in the repo dir!
REPODIR=$(cd $(dirname $0); pwd)

VALIDARGS="clean libcudf cudf cudfjar dask_cudf benchmarks tests libcudf_kafka cudf_kafka custreamz -v -g -n -l --allgpuarch --disable_nvtx --opensource_nvcomp --show_depr_warn --ptds -h --build_metrics --incl_cache_stats"
VALIDARGS="clean libcudf cudf cudfjar dask_cudf benchmarks tests libcudf_kafka cudf_kafka custreamz -v -g -n --pydevelop -l --allgpuarch --disable_nvtx --opensource_nvcomp --show_depr_warn --ptds -h --build_metrics --incl_cache_stats"
HELP="$0 [clean] [libcudf] [cudf] [cudfjar] [dask_cudf] [benchmarks] [tests] [libcudf_kafka] [cudf_kafka] [custreamz] [-v] [-g] [-n] [-h] [--cmake-args=\\\"<args>\\\"]
clean - remove all existing build artifacts and configuration (start
over)
Expand All @@ -33,6 +33,7 @@ HELP="$0 [clean] [libcudf] [cudf] [cudfjar] [dask_cudf] [benchmarks] [tests] [li
-v - verbose build mode
-g - build for debug
-n - no install step (does not affect Python)
--pydevelop - Install Python packages in editable mode
--allgpuarch - build for all supported GPU architectures
--disable_nvtx - disable inserting NVTX profiling ranges
--opensource_nvcomp - disable use of proprietary nvcomp extensions
Expand Down Expand Up @@ -69,6 +70,7 @@ BUILD_PER_THREAD_DEFAULT_STREAM=OFF
BUILD_REPORT_METRICS=OFF
BUILD_REPORT_INCL_CACHE_STATS=OFF
USE_PROPRIETARY_NVCOMP=ON
PYTHON_ARGS_FOR_INSTALL="-m pip install --no-build-isolation --no-deps"

# Set defaults for vars that may not have been defined externally
# FIXME: if INSTALL_PREFIX is not set, check PREFIX, then check
Expand Down Expand Up @@ -228,6 +230,9 @@ fi
if hasArg --incl_cache_stats; then
BUILD_REPORT_INCL_CACHE_STATS=ON
fi
if hasArg --pydevelop; then
PYTHON_ARGS_FOR_INSTALL="${PYTHON_ARGS_FOR_INSTALL} -e"
fi

# Append `-DFIND_CUDF_CPP=ON` to EXTRA_CMAKE_ARGS unless a user specified the option.
if [[ "${EXTRA_CMAKE_ARGS}" != *"DFIND_CUDF_CPP"* ]]; then
Expand Down Expand Up @@ -333,15 +338,15 @@ if buildAll || hasArg cudf; then

cd ${REPODIR}/python/cudf
SKBUILD_CMAKE_ARGS="-DCMAKE_PREFIX_PATH=${INSTALL_PREFIX};-DCMAKE_LIBRARY_PATH=${LIBCUDF_BUILD_DIR};-DCMAKE_CUDA_ARCHITECTURES=${CUDF_CMAKE_CUDA_ARCHITECTURES};${EXTRA_CMAKE_ARGS}" \
python -m pip install --no-build-isolation --no-deps .
python ${PYTHON_ARGS_FOR_INSTALL} .
fi


# Build and install the dask_cudf Python package
if buildAll || hasArg dask_cudf; then

cd ${REPODIR}/python/dask_cudf
python -m pip install --no-build-isolation --no-deps .
python ${PYTHON_ARGS_FOR_INSTALL} .
fi

if hasArg cudfjar; then
Expand Down Expand Up @@ -369,11 +374,11 @@ fi
if hasArg cudf_kafka; then
cd ${REPODIR}/python/cudf_kafka
SKBUILD_CMAKE_ARGS="-DCMAKE_PREFIX_PATH=${INSTALL_PREFIX};-DCMAKE_LIBRARY_PATH=${LIBCUDF_BUILD_DIR};${EXTRA_CMAKE_ARGS}"
python -m pip install --no-build-isolation --no-deps .
python ${PYTHON_ARGS_FOR_INSTALL} .
fi

# build custreamz Python package
if hasArg custreamz; then
cd ${REPODIR}/python/custreamz
python -m pip install --no-build-isolation --no-deps .
python ${PYTHON_ARGS_FOR_INSTALL} .
fi

0 comments on commit 96a1b00

Please sign in to comment.