![](\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_merlin_getting-started-movielens-03-training-with-hugectr/nvidia_logo.png\"){kind=logo}
\n",
- "\n",
- "# Getting Started MovieLens: Training with HugeCTR\n",
- "\n",
- "This notebook is created using the latest stable [merlin-hugectr](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-hugectr/tags) container.\n",
- "\n",
- "## Overview\n",
- "\n",
- "In this notebook, we want to provide an overview what HugeCTR framework is, its features and benefits. We will use HugeCTR to train a basic neural network architecture.\n",
- "\n",
- "Learning Objectives:\n",
- "* Adopt NVTabular workflow to provide input files to HugeCTR\n",
- "* Define HugeCTR neural network architecture\n",
- "* Train a deep learning model with HugeCTR"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "16956c69",
- "metadata": {},
- "source": [
- "### Why use HugeCTR?\n",
- "\n",
- "HugeCTR is a GPU-accelerated recommender framework designed to distribute training across multiple GPUs and nodes and estimate Click-Through Rates (CTRs).\n", - "\n", - "HugeCTR offers multiple advantages to train deep learning recommender systems:\n", - "1. **Speed**: HugeCTR is a highly efficient framework written in C++. We experienced up to 10x speed up. HugeCTR on a NVIDIA DGX A100 system proved to be the fastest commercially available solution for training the architecture Deep Learning Recommender Model (DLRM) developed by Facebook.\n", - "2. **Scale**: HugeCTR supports model parallel scaling. It distributes the large embedding tables over multiple GPUs or multiple nodes. \n", - "3. **Easy-to-use**: Easy-to-use Python API similar to Keras. Examples for popular deep learning recommender systems architectures (Wide&Deep, DLRM, DCN, DeepFM) are available." - ] - }, - { - "cell_type": "markdown", - "id": "a7e099b5", - "metadata": {}, - "source": [ - "### Other Features of HugeCTR\n", - "\n", - "HugeCTR is designed to scale deep learning models for recommender systems. It provides a list of other important features:\n", - "* Proficiency in oversubscribing models to train embedding tables with single nodes that don’t fit within the GPU or CPU memory (only required embeddings are prefetched from a parameter server per batch)\n", - "* Asynchronous and multithreaded data pipelines\n", - "* A highly optimized data loader.\n", - "* Supported data formats such as parquet and binary\n", - "* Integration with Triton Inference Server for deployment to production" - ] - }, - { - "cell_type": "markdown", - "id": "e8c0c88f", - "metadata": {}, - "source": [ - "### Getting Started" - ] - }, - { - "cell_type": "markdown", - "id": "065096f1", - "metadata": {}, - "source": [ - "In this example, we will train a neural network with HugeCTR. We will use preprocessed datasets generated via NVTabular in `02-ETL-with-NVTabular` notebook." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "add3372c", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", - " from .autonotebook import tqdm as notebook_tqdm\n" - ] - } - ], - "source": [ - "# External dependencies\n", - "import os\n", - "import nvtabular as nvt" - ] - }, - { - "cell_type": "markdown", - "id": "ac22a3ba", - "metadata": {}, - "source": [ - "We define our base directory, containing the data." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "b81cacce", - "metadata": {}, - "outputs": [], - "source": [ - "# path to preprocessed data\n", - "INPUT_DATA_DIR = os.environ.get(\n", - " \"INPUT_DATA_DIR\", os.path.expanduser(\"/workspace/nvt-examples/movielens/data/\")\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "6750ce01", - "metadata": {}, - "source": [ - "Let's load our saved workflow from the `02-ETL-with-NVTabular` notebook." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "5ab5923c", - "metadata": {}, - "outputs": [], - "source": [ - "workflow = nvt.Workflow.load(os.path.join(INPUT_DATA_DIR, \"workflow\"))" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "aa5405c1", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'userId': dtype('int64'),\n", - " 'movieId': dtype('int64'),\n", - " 'genres': dtype('int64'),\n", - " 'rating': dtype('int8')}" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "workflow.output_dtypes" - ] - }, - { - "cell_type": "markdown", - "id": "94bef620", - "metadata": {}, - "source": [ - "Note: We do not have numerical output columns" - ] - }, - { - "cell_type": "markdown", - "id": "9e8ad562", - "metadata": {}, - "source": [ - "Let's clear existing directory should it exist from previous runs and create the output folders." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "0c678d48", - "metadata": {}, - "outputs": [], - "source": [ - "MODEL_DIR = os.path.join(INPUT_DATA_DIR, \"model/movielens_hugectr/\")\n", - "!rm -rf {MODEL_DIR}\n", - "!mkdir -p {MODEL_DIR}\"1\"" - ] - }, - { - "cell_type": "markdown", - "id": "435c7e86", - "metadata": {}, - "source": [ - "## Scaling Accelerated training with HugeCTR" - ] - }, - { - "cell_type": "markdown", - "id": "5b76f6ea", - "metadata": {}, - "source": [ - "HugeCTR is a deep learning framework dedicated to recommendation systems. It is written in CUDA C++. As HugeCTR optimizes the training in CUDA++, we need to define the training pipeline and model architecture and execute it via the commandline. We will use the Python API, which is similar to Keras models.\n", - "\n", - "For more information on HugeCTR please consult the [HugeCTR repository](https://github.com/NVIDIA-Merlin/HugeCTR)." - ] - }, - { - "cell_type": "markdown", - "id": "0624ca30", - "metadata": {}, - "source": [ - "## Let's define our model\n", - "\n", - "Let's define our model. We will write the model to `./train_hugeCTR.py` and execute it afterwards." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "baaf3563", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Overwriting train_hugeCTR.py\n" - ] - } - ], - "source": [ - "%%writefile train_hugeCTR.py\n", - "\n", - "# External dependencies\n", - "import os\n", - "import nvtabular as nvt\n", - "from nvtabular.ops import get_embedding_sizes\n", - "import hugectr\n", - "from mpi4py import MPI \n", - "\n", - "# path to preprocessed data\n", - "INPUT_DATA_DIR = os.environ.get(\n", - " \"INPUT_DATA_DIR\", os.path.expanduser(\"/workspace/nvt-examples/movielens/data/\")\n", - ")\n", - "\n", - "MODEL_DIR = os.path.join(INPUT_DATA_DIR, \"model/movielens_hugectr/\")\n", - "\n", - "workflow = nvt.Workflow.load(os.path.join(INPUT_DATA_DIR, \"workflow\"))\n", - "\n", - "embeddings = get_embedding_sizes(workflow)\n", - "\n", - "solver = hugectr.CreateSolver(\n", - " vvgpu=[[0]],\n", - " batchsize=2048,\n", - " batchsize_eval=2048,\n", - " max_eval_batches=160,\n", - " i64_input_key=True,\n", - " use_mixed_precision=False,\n", - " repeat_dataset=True,\n", - ")\n", - "optimizer = hugectr.CreateOptimizer(optimizer_type=hugectr.Optimizer_t.Adam)\n", - "reader = hugectr.DataReaderParams(\n", - " data_reader_type=hugectr.DataReaderType_t.Parquet,\n", - " source=[INPUT_DATA_DIR + \"train/_file_list.txt\"],\n", - " eval_source=INPUT_DATA_DIR + \"valid/_file_list.txt\",\n", - " check_type=hugectr.Check_t.Non,\n", - " slot_size_array=[162542, 56586, 21],\n", - ")\n", - "\n", - "\n", - "model = hugectr.Model(solver, reader, optimizer)\n", - "\n", - "model.add(\n", - " hugectr.Input(\n", - " label_dim=1,\n", - " label_name=\"label\",\n", - " dense_dim=0,\n", - " dense_name=\"dense\",\n", - " data_reader_sparse_param_array=[\n", - " hugectr.DataReaderSparseParam(\"data1\", nnz_per_slot=10, is_fixed_length=False, slot_num=3)\n", - " ],\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.SparseEmbedding(\n", - " embedding_type=hugectr.Embedding_t.LocalizedSlotSparseEmbeddingHash,\n", - " workspace_size_per_gpu_in_mb=200,\n", - " embedding_vec_size=16,\n", - " combiner=\"sum\",\n", - " sparse_embedding_name=\"sparse_embedding1\",\n", - " bottom_name=\"data1\",\n", - " optimizer=optimizer,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.Reshape,\n", - " bottom_names=[\"sparse_embedding1\"],\n", - " top_names=[\"reshape1\"],\n", - " leading_dim=48,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"reshape1\"],\n", - " top_names=[\"fc1\"],\n", - " num_output=128,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.ReLU,\n", - " bottom_names=[\"fc1\"],\n", - " top_names=[\"relu1\"],\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"relu1\"],\n", - " top_names=[\"fc2\"],\n", - " num_output=128,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.ReLU,\n", - " bottom_names=[\"fc2\"],\n", - " top_names=[\"relu2\"],\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"relu2\"],\n", - " top_names=[\"fc3\"],\n", - " num_output=1,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.BinaryCrossEntropyLoss,\n", - " bottom_names=[\"fc3\", \"label\"],\n", - " top_names=[\"loss\"],\n", - " )\n", - ")\n", - "\n", - "model.compile()\n", - "model.summary()\n", - "model.fit(max_iter=2000, display=100, eval_interval=200, snapshot=1900)\n", - "model.graph_to_json(graph_config_file=MODEL_DIR + \"1/movielens.json\")" - ] - }, - { - "cell_type": "markdown", - "id": "e177dee5", - "metadata": {}, - "source": [ - "Now please run the script we outputted above in the terminal using the following command:\n", - "\n", - "```python train_hugeCTR.py```" - ] - }, - { - "cell_type": "markdown", - "id": "2ae972f8", - "metadata": {}, - "source": [ - "After training terminates, we can see that multiple `.model` files and folders are generated." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "9a6a869f", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "0_opt_sparse_1900.model _dense_1900.model _opt_dense_1900.model\r\n", - "\r\n", - "0_sparse_1900.model:\r\n", - "emb_vector key slot_id\r\n" - ] - } - ], - "source": [ - "ls *.model" - ] - }, - { - "cell_type": "markdown", - "id": "2d7dace8", - "metadata": {}, - "source": [ - "Let's move these files into the `movielens_hugectr` folder. When we start the Triton Inference Server, we will be able to point it to that directory and ask it to load our model using the files we move there below." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "b085ceb4", - "metadata": {}, - "outputs": [], - "source": [ - "!mv *.model {MODEL_DIR}" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.10" - }, - "merlin": { - "containers": [ - "nvcr.io/nvidia/merlin/merlin-hugectr:latest" - ] - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/examples/getting-started-movielens/03-Training-with-TF.ipynb b/examples/getting-started-movielens/03-Training-with-TF.ipynb index aa385e25b..9f92725d6 100644 --- a/examples/getting-started-movielens/03-Training-with-TF.ipynb +++ b/examples/getting-started-movielens/03-Training-with-TF.ipynb @@ -69,7 +69,18 @@ "cell_type": "code", "execution_count": 2, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-05-09 22:46:08.000436: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX\n", + "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", + "/usr/local/lib/python3.8/dist-packages/merlin/dtypes/mappings/torch.py:43: UserWarning: PyTorch dtype mappings did not load successfully due to an error: No module named 'torch'\n", + " warn(f\"PyTorch dtype mappings did not load successfully due to an error: {exc.msg}\")\n" + ] + } + ], "source": [ "# External dependencies\n", "import os\n", @@ -104,39 +115,49 @@ "execution_count": 4, "metadata": {}, "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11.\n" + ] + }, { "name": "stderr", "output_type": "stream", "text": [ - "2023-01-20 11:26:39.373230: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX\n", - "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n" + "2023-05-09 22:46:09.294968: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2023-05-09 22:46:09.295315: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2023-05-09 22:46:09.295465: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11.\n" + "[INFO]: sparse_operation_kit is imported\n", + "WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.base has been moved to tensorflow.python.trackable.base. The old module will be deleted in version 2.11.\n", + "[SOK INFO] Import /usr/local/lib/python3.8/dist-packages/merlin_sok-1.1.4-py3.8-linux-x86_64.egg/sparse_operation_kit/lib/libsok_experiment.so\n", + "[SOK INFO] Import /usr/local/lib/python3.8/dist-packages/merlin_sok-1.1.4-py3.8-linux-x86_64.egg/sparse_operation_kit/lib/libsok_experiment.so\n", + "[SOK INFO] Initialize finished, communication tool: horovod\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ - "2023-01-20 11:26:40.514976: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:40.515413: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:40.515595: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:40.727125: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX\n", + "2023-05-09 22:46:10.932696: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX\n", "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", - "2023-01-20 11:26:40.728141: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:40.728351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:40.728508: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:41.475009: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:41.475231: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:41.475394: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", - "2023-01-20 11:26:41.475508: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0\n", - "2023-01-20 11:26:41.475570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1637] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 24576 MB memory: -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:08:00.0, compute capability: 7.5\n", - "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + "2023-05-09 22:46:10.933669: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2023-05-09 22:46:10.933873: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2023-05-09 22:46:10.934029: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2023-05-09 22:46:11.056253: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2023-05-09 22:46:11.056472: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2023-05-09 22:46:11.056632: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", + "2023-05-09 22:46:11.056905: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n", + "2023-05-09 22:46:11.056916: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0\n", + "2023-05-09 22:46:11.056987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1621] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 24576 MB memory: -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:08:00.0, compute capability: 7.5\n", + "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] } @@ -275,17 +296,27 @@ "execution_count": 8, "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-05-09 22:46:16.890983: I tensorflow/compiler/xla/service/service.cc:173] XLA service 0x11d82730 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n", + "2023-05-09 22:46:16.891012: I tensorflow/compiler/xla/service/service.cc:181] StreamExecutor device (0): Quadro RTX 8000, Compute Capability 7.5\n", + "2023-05-09 22:46:16.904028: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n", + "2023-05-09 22:46:17.039510: I tensorflow/compiler/jit/xla_compilation_cache.cc:480] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n" + ] + }, { "name": "stdout", "output_type": "stream", "text": [ - "1221/1221 [==============================] - 9s 6ms/step - loss: 0.6609 - auc: 0.5281 - regularization_loss: 0.0000e+00 - loss_batch: 0.6609 - val_loss: 0.6588 - val_auc: 0.5626 - val_regularization_loss: 0.0000e+00 - val_loss_batch: 0.6537\n" + "1221/1221 [==============================] - 55s 42ms/step - loss: 0.6611 - auc: 0.5247 - regularization_loss: 0.0000e+00 - loss_batch: 0.6611 - val_loss: 0.6596 - val_auc: 0.5521 - val_regularization_loss: 0.0000e+00 - val_loss_batch: 0.6610\n" ] }, { "data": { "text/plain": [ - "
![](\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_merlin_getting-started-movielens-04-triton-inference-with-hugectr/nvidia_logo.png\"){kind=logo}
\n",
- "\n",
- "# Serve Recommendations from the HugeCTR Model\n",
- "\n",
- "This notebook is created using the latest stable [merlin-hugectr](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-hugectr/tags) container. \n",
- "\n",
- "## Overview\n",
- "\n",
- "In this notebook, we will show how we do inference with our trained deep learning recommender model using Triton Inference Server. In this example, we deploy the NVTabular workflow and HugeCTR model with Triton Inference Server. We deploy them as an ensemble. For each request, Triton Inference Server will feed the input data through the NVTabular workflow and its output through the HugeCR model."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e0157e1c",
- "metadata": {},
- "source": [
- "## Getting Started"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "71304a10",
- "metadata": {},
- "source": [
- "We need to write configuration files with the stored model weights and model configuration.\n",
- "\n",
- "Let us first move all of our model files to a directory that we will be able to access from the scripts that we will generate."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "c2efc44e",
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import json\n",
- "\n",
- "# path to preprocessed data\n",
- "INPUT_DATA_DIR = os.environ.get(\n",
- " \"INPUT_DATA_DIR\", os.path.expanduser(\"/workspace/nvt-examples/movielens/data/\")\n",
- ")\n",
- "\n",
- "# path to saved model\n",
- "MODEL_DIR = os.path.join(INPUT_DATA_DIR, \"model/movielens_hugectr\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "6a9fbb6d",
- "metadata": {
- "tags": [
- "flake8-noqa-cell"
- ]
- },
- "outputs": [],
- "source": [
- "file_to_write = \"\"\"\n",
- "name: \"movielens_hugectr\"\n",
- "backend: \"hugectr\"\n",
- "max_batch_size: 64\n",
- "input [\n",
- " {\n",
- " name: \"DES\"\n",
- " data_type: TYPE_FP32\n",
- " dims: [ -1 ]\n",
- " },\n",
- " {\n",
- " name: \"CATCOLUMN\"\n",
- " data_type: TYPE_INT64\n",
- " dims: [ -1 ]\n",
- " },\n",
- " {\n",
- " name: \"ROWINDEX\"\n",
- " data_type: TYPE_INT32\n",
- " dims: [ -1 ]\n",
- " }\n",
- "]\n",
- "output [\n",
- " {\n",
- " name: \"OUTPUT0\"\n",
- " data_type: TYPE_FP32\n",
- " dims: [ -1 ]\n",
- " }\n",
- "]\n",
- "instance_group [\n",
- " {\n",
- " count: 1\n",
- " kind : KIND_GPU\n",
- " gpus:[0]\n",
- " }\n",
- "]\n",
- "\n",
- "parameters [\n",
- " {\n",
- " key: \"config\"\n",
- " value: { string_value: \"$MODEL_DIR/1/movielens.json\" }\n",
- " },\n",
- " {\n",
- " key: \"gpucache\"\n",
- " value: { string_value: \"true\" }\n",
- " },\n",
- " {\n",
- " key: \"hit_rate_threshold\"\n",
- " value: { string_value: \"0.8\" }\n",
- " },\n",
- " {\n",
- " key: \"gpucacheper\"\n",
- " value: { string_value: \"0.5\" }\n",
- " },\n",
- " {\n",
- " key: \"label_dim\"\n",
- " value: { string_value: \"1\" }\n",
- " },\n",
- " {\n",
- " key: \"slots\"\n",
- " value: { string_value: \"3\" }\n",
- " },\n",
- " {\n",
- " key: \"cat_feature_num\"\n",
- " value: { string_value: \"4\" }\n",
- " },\n",
- " {\n",
- " key: \"des_feature_num\"\n",
- " value: { string_value: \"0\" }\n",
- " },\n",
- " {\n",
- " key: \"max_nnz\"\n",
- " value: { string_value: \"2\" }\n",
- " },\n",
- " {\n",
- " key: \"embedding_vector_size\"\n",
- " value: { string_value: \"16\" }\n",
- " },\n",
- " {\n",
- " key: \"embeddingkey_long_type\"\n",
- " value: { string_value: \"true\" }\n",
- " }\n",
- "]\n",
- "\"\"\"\n",
- "\n",
- "with open(os.path.join(MODEL_DIR, \"config.pbtxt\"), \"w\", encoding=\"utf-8\") as f:\n",
- " f.write(file_to_write.replace(\"$MODEL_DIR\", MODEL_DIR))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "0a23cb52",
- "metadata": {
- "tags": [
- "flake8-noqa-cell"
- ]
- },
- "outputs": [],
- "source": [
- "config = json.dumps(\n",
- "{\n",
- " \"supportlonglong\": True,\n",
- " \"models\": [\n",
- " {\n",
- " \"model\": \"movielens_hugectr\",\n",
- " \"sparse_files\": [f\"{MODEL_DIR}/0_sparse_1900.model\"],\n",
- " \"dense_file\": f\"{MODEL_DIR}/_dense_1900.model\",\n",
- " \"network_file\": f\"{MODEL_DIR}/1/movielens.json\",\n",
- " \"num_of_worker_buffer_in_pool\": \"1\",\n",
- " \"num_of_refresher_buffer_in_pool\": \"1\",\n",
- " \"cache_refresh_percentage_per_iteration\": \"0.2\",\n",
- " \"deployed_device_list\": [\"0\"],\n",
- " \"max_batch_size\": \"64\",\n",
- " \"default_value_for_each_table\": [\"0.0\",\"0.0\"],\n",
- " \"hit_rate_threshold\": \"0.9\",\n",
- " \"gpucacheper\": \"0.5\",\n",
- " \"maxnum_catfeature_query_per_table_per_sample\": [\"162542\", \"56632\",\"12\"],\n",
- " \"embedding_vecsize_per_table\": [\"16\",\"16\",\"16\"],\n",
- " \"gpucache\":\"true\"\n",
- " }\n",
- " ] \n",
- "})\n",
- "\n",
- "config = json.loads(config)\n",
- "with open(os.path.join(MODEL_DIR, \"ps.json\"), \"w\", encoding=\"utf-8\") as f:\n",
- " json.dump(config, f)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5eb3627f",
- "metadata": {},
- "source": [
- "Let's import required libraries."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "f5b54092",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
- " from .autonotebook import tqdm as notebook_tqdm\n"
- ]
- }
- ],
- "source": [
- "import tritonclient.grpc as httpclient\n",
- "import cudf\n",
- "import numpy as np"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4e4592a9",
- "metadata": {},
- "source": [
- "### Load Models on Triton Server"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c6f50e9e",
- "metadata": {},
- "source": [
- "In the running docker container, you can start triton server with the command below:"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bc8aa849",
- "metadata": {},
- "source": [
- "```\n",
- "tritonserver --model-repository=![](\"https://developer.download.nvidia.com//notebooks/dlsw-notebooks/merlin_merlin_scaling-criteo-03-training-with-hugectr/nvidia_logo.png\"){kind=logo}
\n",
- "\n",
- "# Scaling Criteo: Training with HugeCTR\n",
- "\n",
- "This notebook is created using the latest stable [merlin-hugectr](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-hugectr/tags) container.\n",
- "\n",
- "## Overview\n",
- "\n",
- "HugeCTR is an open-source framework to accelerate the training of CTR estimation models on NVIDIA GPUs. It is written in CUDA C++ and highly exploits GPU-accelerated libraries such as cuBLAS, cuDNN, and NCCL.\n", - "HugeCTR offers multiple advantages to train deep learning recommender systems:\n", - "\n", - "1. **Speed**: HugeCTR is a highly efficient framework written in C++. We experienced up to 10x speed up. HugeCTR on a NVIDIA DGX A100 system proved to be the fastest commercially available solution for training the architecture Deep Learning Recommender Model (DLRM) developed by Facebook.\n", - "2. **Scale**: HugeCTR supports model parallel scaling. It distributes the large embedding tables over multiple GPUs or multiple nodes. \n", - "3. **Easy-to-use**: Easy-to-use Python API similar to Keras. Examples for popular deep learning recommender systems architectures (Wide&Deep, DLRM, DCN, DeepFM) are available.\n", - "\n", - "HugeCTR is able to train recommender system models with larger-than-memory embedding tables by leveraging a parameter server. \n", - "\n", - "You can find more information about HugeCTR from the [GitHub repository](https://github.com/NVIDIA-Merlin/HugeCTR).\n", - "\n", - "### Learning objectives\n", - "\n", - "In this notebook, we learn how to to use HugeCTR for training recommender system models\n", - "\n", - "- Use **HugeCTR** to define a recommender system model\n", - "- Train Facebook's [Deep Learning Recommendation Model](https://arxiv.org/pdf/1906.00091.pdf) with HugeCTR" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Training with HugeCTR" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As HugeCTR optimizes the training in CUDA++, we need to define the training pipeline and model architecture and execute it via the commandline. We will use the Python API, which is similar to Keras models." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you are not familiar with HugeCTR's Python API and parameters, you can read more in its GitHub repository:\n", - "- [HugeCTR User Guide](https://nvidia-merlin.github.io/HugeCTR/master/hugectr_user_guide.html)\n", - "- [HugeCTR Python API](https://nvidia-merlin.github.io/HugeCTR/master/api/python_interface.html)\n", - "- [HugeCTR example architectures](https://github.com/NVIDIA/HugeCTR/tree/main/samples)" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "\n", - "BASE_DIR = os.environ.get(\"BASE_DIR\", \"/raid/data/criteo\")\n", - "OUTPUT_DATA_DIR = os.environ.get(\"OUTPUT_DATA_DIR\", BASE_DIR + \"/test_dask/output\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First, we clean the output directory." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "0" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "os.system(\"rm -rf \" + os.path.join(OUTPUT_DATA_DIR, \"criteo_hugectr/\"))\n", - "os.system(\"mkdir -p \" + os.path.join(OUTPUT_DATA_DIR, \"criteo_hugectr/1/\"))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We write the code to a `./model.py` file and execute it. The code creates snapshots, which we will use for inference in the next notebook. We use `graph_to_json` to convert the model to a JSON configuration, required for the inference." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [], - "source": [ - "file_to_write = f\"\"\"\n", - "import hugectr\n", - "from mpi4py import MPI # noqa\n", - "\n", - "# HugeCTR\n", - "solver = hugectr.CreateSolver(\n", - " vvgpu=[[0]],\n", - " max_eval_batches=100,\n", - " batchsize_eval=2720,\n", - " batchsize=2720,\n", - " i64_input_key=True,\n", - " use_mixed_precision=False,\n", - " repeat_dataset=True,\n", - ")\n", - "optimizer = hugectr.CreateOptimizer(optimizer_type=hugectr.Optimizer_t.SGD)\n", - "reader = hugectr.DataReaderParams(\n", - " data_reader_type=hugectr.DataReaderType_t.Parquet,\n", - " source=[\"{os.path.join(OUTPUT_DATA_DIR, \"train/_file_list.txt\")}\"],\n", - " eval_source=\"{os.path.join(OUTPUT_DATA_DIR, \"valid/_file_list.txt\")}\",\n", - " check_type=hugectr.Check_t.Non,\n", - " slot_size_array=[\n", - " 10000000,\n", - " 10000000,\n", - " 3014529,\n", - " 400781,\n", - " 11,\n", - " 2209,\n", - " 11869,\n", - " 148,\n", - " 4,\n", - " 977,\n", - " 15,\n", - " 38713,\n", - " 10000000,\n", - " 10000000,\n", - " 10000000,\n", - " 584616,\n", - " 12883,\n", - " 109,\n", - " 37,\n", - " 17177,\n", - " 7425,\n", - " 20266,\n", - " 4,\n", - " 7085,\n", - " 1535,\n", - " 64,\n", - " ],\n", - ")\n", - "model = hugectr.Model(solver, reader, optimizer)\n", - "model.add(\n", - " hugectr.Input(\n", - " label_dim=1,\n", - " label_name=\"label\",\n", - " dense_dim=13,\n", - " dense_name=\"dense\",\n", - " data_reader_sparse_param_array=[hugectr.DataReaderSparseParam(\"data1\", 1, False, 26)],\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.SparseEmbedding(\n", - " embedding_type=hugectr.Embedding_t.LocalizedSlotSparseEmbeddingHash,\n", - " workspace_size_per_gpu_in_mb=6000,\n", - " embedding_vec_size=128,\n", - " combiner=\"sum\",\n", - " sparse_embedding_name=\"sparse_embedding1\",\n", - " bottom_name=\"data1\",\n", - " optimizer=optimizer,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"dense\"],\n", - " top_names=[\"fc1\"],\n", - " num_output=512,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc1\"], top_names=[\"relu1\"])\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"relu1\"],\n", - " top_names=[\"fc2\"],\n", - " num_output=256,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc2\"], top_names=[\"relu2\"])\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"relu2\"],\n", - " top_names=[\"fc3\"],\n", - " num_output=128,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc3\"], top_names=[\"relu3\"])\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.Interaction,\n", - " bottom_names=[\"relu3\", \"sparse_embedding1\"],\n", - " top_names=[\"interaction1\"],\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"interaction1\"],\n", - " top_names=[\"fc4\"],\n", - " num_output=1024,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc4\"], top_names=[\"relu4\"])\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"relu4\"],\n", - " top_names=[\"fc5\"],\n", - " num_output=1024,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc5\"], top_names=[\"relu5\"])\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"relu5\"],\n", - " top_names=[\"fc6\"],\n", - " num_output=512,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc6\"], top_names=[\"relu6\"])\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"relu6\"],\n", - " top_names=[\"fc7\"],\n", - " num_output=256,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(layer_type=hugectr.Layer_t.ReLU, bottom_names=[\"fc7\"], top_names=[\"relu7\"])\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.InnerProduct,\n", - " bottom_names=[\"relu7\"],\n", - " top_names=[\"fc8\"],\n", - " num_output=1,\n", - " )\n", - ")\n", - "model.add(\n", - " hugectr.DenseLayer(\n", - " layer_type=hugectr.Layer_t.BinaryCrossEntropyLoss,\n", - " bottom_names=[\"fc8\", \"label\"],\n", - " top_names=[\"loss\"],\n", - " )\n", - ")\n", - "\n", - "MAX_ITER = 10000\n", - "EVAL_INTERVAL = 3200\n", - "model.compile()\n", - "model.summary()\n", - "model.fit(max_iter=MAX_ITER, eval_interval=EVAL_INTERVAL, display=1000, snapshot=3200)\n", - "model.graph_to_json(graph_config_file=\"{os.path.join(OUTPUT_DATA_DIR, \"criteo_hugectr/1/\", \"criteo.json\")}\")\n", - "\"\"\"\n", - "with open('./model.py', 'w', encoding='utf-8') as fi:\n", - " fi.write(file_to_write)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can execute the training process." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HugeCTR Version: 4.1\n", - "====================================================Model Init=====================================================\n", - "[HCTR][13:56:03.374][WARNING][RK0][main]: The model name is not specified when creating the solver.\n", - "[HCTR][13:56:03.374][INFO][RK0][main]: Global seed is 2831956451\n", - "[HCTR][13:56:03.376][INFO][RK0][main]: Device to NUMA mapping:\n", - " GPU 0 -> node 0\n", - "[HCTR][13:56:06.490][WARNING][RK0][main]: Peer-to-peer access cannot be fully enabled.\n", - "[HCTR][13:56:06.490][INFO][RK0][main]: Start all2all warmup\n", - "[HCTR][13:56:06.490][INFO][RK0][main]: End all2all warmup\n", - "[HCTR][13:56:06.491][INFO][RK0][main]: Using All-reduce algorithm: NCCL\n", - "[HCTR][13:56:06.493][INFO][RK0][main]: Device 0: Tesla V100-SXM2-32GB-LS\n", - "[HCTR][13:56:06.493][INFO][RK0][main]: num of DataReader workers for train: 1\n", - "[HCTR][13:56:06.493][INFO][RK0][main]: num of DataReader workers for eval: 1\n", - "[HCTR][13:56:06.540][INFO][RK0][main]: Vocabulary size: 54120457\n", - "[HCTR][13:56:06.541][INFO][RK0][main]: max_vocabulary_size_per_gpu_=12288000\n", - "[HCTR][13:56:06.562][INFO][RK0][main]: Graph analysis to resolve tensor dependency\n", - "===================================================Model Compile===================================================\n", - "[HCTR][13:56:24.055][INFO][RK0][main]: gpu0 start to init embedding\n", - "[HCTR][13:56:24.107][INFO][RK0][main]: gpu0 init embedding done\n", - "[HCTR][13:56:24.111][INFO][RK0][main]: Starting AUC NCCL warm-up\n", - "[HCTR][13:56:24.113][INFO][RK0][main]: Warm-up done\n", - "===================================================Model Summary===================================================\n", - "run_time: 125.9122965335846\n" - ] - } - ], - "source": [ - "import time\n", - "\n", - "start = time.time()\n", - "os.system('python model.py')\n", - "end = time.time() - start\n", - "print(f\"run_time: {end}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We trained the model and created snapshots." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.10" - }, - "merlin": { - "containers": [ - "nvcr.io/nvidia/merlin/merlin-hugectr:latest" - ] - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/examples/scaling-criteo/04-Triton-Inference-with-HugeCTR.ipynb b/examples/scaling-criteo/04-Triton-Inference-with-HugeCTR.ipynb deleted file mode 100644 index 0d875274e..000000000 --- a/examples/scaling-criteo/04-Triton-Inference-with-HugeCTR.ipynb +++ /dev/null @@ -1,639 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "# Copyright 2021 NVIDIA Corporation. All Rights Reserved.\n", - "#\n", - "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", - "# you may not use this file except in compliance with the License.\n", - "# You may obtain a copy of the License at\n", - "#\n", - "# http://www.apache.org/licenses/LICENSE-2.0\n", - "#\n", - "# Unless required by applicable law or agreed to in writing, software\n", - "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", - "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", - "# See the License for the specific language governing permissions and\n", - "# limitations under the License.\n", - "# ==============================================================================\n", - "\n", - "# Each user is responsible for checking the content of datasets and the\n", - "# applicable licenses and determining if suitable for the intended use." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "
![](\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_merlin_scaling-criteo-04-triton-inference-with-hugectr/nvidia_logo.png\"){kind=logo}
\n",
- "\n",
- "# Scaling Criteo: Triton Inference with HugeCTR\n",
- "\n",
- "This notebook is created using the latest stable [merlin-hugectr](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-hugectr/tags) container. \n",
- "\n",
- "## Overview\n",
- "\n",
- "The last step is to deploy the ETL workflow and saved model to production. In the production setting, we want to transform the input data as during training (ETL). We need to apply the same mean/std for continuous features and use the same categorical mapping to convert the categories to continuous integer before we use the deep learning model for a prediction. Therefore, we deploy the NVTabular workflow with the HugeCTR model as an ensemble model to Triton Inference. The ensemble model guarantees that the same transformation are applied to the raw inputs.\n",
- "\n",
- "
\n",
- "\n",
- "### Learning objectives\n",
- "\n",
- "In this notebook, we learn how to deploy our models to production:\n",
- "\n",
- "- Use **NVTabular** to generate config and model files for Triton Inference Server\n",
- "- Deploy an ensemble of NVTabular workflow and HugeCTR model\n",
- "- Send example request to Triton Inference Server"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Deploying Ensemble to Triton Inference Server\n",
- "\n",
- "First, we need to generate the Triton Inference Server configurations and save the models in the correct format. In the previous notebooks [02-ETL-with-NVTabular](./02-ETL-with-NVTabular.ipynb) and [03-Training-with-HugeCTR](./03-Training-with-HugeCTR.ipynb) we saved the NVTabular workflow and HugeCTR model to disk. We will load them."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "After training terminates, we can see that two `.model` files are generated. We need to move them inside a temporary folder, like `criteo_hugectr/1`. Let's create these folders."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!\n",
- " warnings.warn(\"urllib3 ({}) or chardet ({}) doesn't match a supported \"\n"
- ]
- }
- ],
- "source": [
- "import os\n",
- "import glob\n",
- "import json\n",
- "\n",
- "import numpy as np\n",
- "import nvtabular as nvt\n",
- "import tritonclient.grpc as grpcclient\n",
- "\n",
- "from merlin.core.dispatch import get_lib\n",
- "from merlin.systems.triton import convert_df_to_triton_input\n",
- "from nvtabular.inference.triton import export_hugectr_ensemble\n",
- "\n",
- "BASE_DIR = os.environ.get(\"BASE_DIR\", \"/raid/data/criteo\")\n",
- "OUTPUT_DATA_DIR = os.environ.get(\"OUTPUT_DATA_DIR\", BASE_DIR + \"/test_dask/output\")\n",
- "original_data_path = os.environ.get(\"INPUT_FOLDER\", BASE_DIR + \"/converted/criteo\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now we move our saved `.model` files inside 1 folder. We use only the last snapshot after `9600` iterations."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "os.system(\"mv *9600.model \" + os.path.join(OUTPUT_DATA_DIR, \"criteo_hugectr/1/\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We need to load the NVTabular workflow first"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "workflow = nvt.Workflow.load(os.path.join(OUTPUT_DATA_DIR, \"workflow\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's clear the directory"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "os.system(\"rm -rf \" + os.path.join(OUTPUT_DATA_DIR, \"model_inference\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### Export artifacts\n",
- "\n",
- "Now, we can save our models for use later during the inference stage. To do so we use export_hugectr_ensemble method below. With this method, we can generate the `config.pbtxt` files automatically for each model.\n", - "The script below creates an ensemble triton server model where\n", - "- workflow is the the nvtabular workflow used in preprocessing,\n", - "- hugectr_model_path is the HugeCTR model that should be served. This path includes the model files.\n", - "- name is the base name of the various triton models.\n", - "- output_path is the path where is model will be saved to.\n", - "- cats are the categorical column names\n", - "- conts are the continuous column names" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [], - "source": [ - "hugectr_params = dict()\n", - "# Config File in the final directory for serving\n", - "hugectr_params[\"config\"] = os.path.join(OUTPUT_DATA_DIR, \"model_inference\", \"criteo/1/criteo.json\")\n", - "hugectr_params[\"slots\"] = 26\n", - "hugectr_params[\"max_nnz\"] = 1\n", - "hugectr_params[\"embedding_vector_size\"] = 128\n", - "hugectr_params[\"n_outputs\"] = 1\n", - "export_hugectr_ensemble(\n", - " workflow=workflow,\n", - " # Current directory with model weights and config file\n", - " hugectr_model_path=os.path.join(OUTPUT_DATA_DIR, \"criteo_hugectr/1/\"),\n", - " hugectr_params=hugectr_params,\n", - " name=\"criteo\",\n", - " # Base directory for serving\n", - " output_path=os.path.join(OUTPUT_DATA_DIR, \"model_inference\"),\n", - " label_columns=[\"label\"],\n", - " cats=[\"C\" + str(x) for x in range(1, 27)],\n", - " conts=[\"I\" + str(x) for x in range(1, 14)],\n", - " max_batch_size=64,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can take a look at the generated files." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[01;34m/tmp/test_merlin_criteo_hugectr/output/criteo//model_inference\u001b[00m\r\n", - "├── \u001b[01;34mcriteo\u001b[00m\r\n", - "│ ├── \u001b[01;34m1\u001b[00m\r\n", - "│ │ ├── \u001b[01;34m0_sparse_9600.model\u001b[00m\r\n", - "│ │ │ ├── emb_vector\r\n", - "│ │ │ ├── key\r\n", - "│ │ │ └── slot_id\r\n", - "│ │ ├── _dense_9600.model\r\n", - "│ │ ├── _opt_dense_9600.model\r\n", - "│ │ └── criteo.json\r\n", - "│ └── config.pbtxt\r\n", - "├── \u001b[01;34mcriteo_ens\u001b[00m\r\n", - "│ ├── \u001b[01;34m1\u001b[00m\r\n", - "│ └── config.pbtxt\r\n", - "├── \u001b[01;34mcriteo_nvt\u001b[00m\r\n", - "│ ├── \u001b[01;34m1\u001b[00m\r\n", - "│ │ ├── \u001b[01;34m__pycache__\u001b[00m\r\n", - "│ │ │ └── model.cpython-38.pyc\r\n", - "│ │ ├── model.py\r\n", - "│ │ └── \u001b[01;34mworkflow\u001b[00m\r\n", - "│ │ ├── \u001b[01;34mcategories\u001b[00m\r\n", - "│ │ │ ├── unique.C1.parquet\r\n", - "│ │ │ ├── unique.C10.parquet\r\n", - "│ │ │ ├── unique.C11.parquet\r\n", - "│ │ │ ├── unique.C12.parquet\r\n", - "│ │ │ ├── unique.C13.parquet\r\n", - "│ │ │ ├── unique.C14.parquet\r\n", - "│ │ │ ├── unique.C15.parquet\r\n", - "│ │ │ ├── unique.C16.parquet\r\n", - "│ │ │ ├── unique.C17.parquet\r\n", - "│ │ │ ├── unique.C18.parquet\r\n", - "│ │ │ ├── unique.C19.parquet\r\n", - "│ │ │ ├── unique.C2.parquet\r\n", - "│ │ │ ├── unique.C20.parquet\r\n", - "│ │ │ ├── unique.C21.parquet\r\n", - "│ │ │ ├── unique.C22.parquet\r\n", - "│ │ │ ├── unique.C23.parquet\r\n", - "│ │ │ ├── unique.C24.parquet\r\n", - "│ │ │ ├── unique.C25.parquet\r\n", - "│ │ │ ├── unique.C26.parquet\r\n", - "│ │ │ ├── unique.C3.parquet\r\n", - "│ │ │ ├── unique.C4.parquet\r\n", - "│ │ │ ├── unique.C5.parquet\r\n", - "│ │ │ ├── unique.C6.parquet\r\n", - "│ │ │ ├── unique.C7.parquet\r\n", - "│ │ │ ├── unique.C8.parquet\r\n", - "│ │ │ └── unique.C9.parquet\r\n", - "│ │ ├── metadata.json\r\n", - "│ │ └── workflow.pkl\r\n", - "│ └── config.pbtxt\r\n", - "└── ps.json\r\n", - "\r\n", - "10 directories, 40 files\r\n" - ] - } - ], - "source": [ - "!tree $OUTPUT_DATA_DIR/model_inference" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We need to write a configuration file with the stored model weights and model configuration." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": { - "tags": [ - "flake8-noqa-cell" - ] - }, - "outputs": [], - "source": [ - "config = json.dumps(\n", - "{\n", - " \"supportlonglong\": \"true\",\n", - " \"models\": [\n", - " {\n", - " \"model\": \"criteo\",\n", - " \"sparse_files\": [os.path.join(OUTPUT_DATA_DIR, \"model_inference\", \"criteo/1/0_sparse_9600.model\")],\n", - " \"dense_file\": os.path.join(OUTPUT_DATA_DIR, \"model_inference\", \"criteo/1/_dense_9600.model\"),\n", - " \"network_file\": os.path.join(OUTPUT_DATA_DIR, \"model_inference\", \"criteo/1/criteo.json\"),\n", - " \"max_batch_size\": \"64\",\n", - " \"gpucache\": \"true\",\n", - " \"hit_rate_threshold\": \"0.9\",\n", - " \"gpucacheper\": \"0.5\",\n", - " \"num_of_worker_buffer_in_pool\": \"4\",\n", - " \"num_of_refresher_buffer_in_pool\": \"1\",\n", - " \"cache_refresh_percentage_per_iteration\": 0.2,\n", - " \"deployed_device_list\": [\"0\"],\n", - " \"default_value_for_each_table\": [\"0.0\", \"0.0\"],\n", - " \"maxnum_catfeature_query_per_table_per_sample\": [2, 26],\n", - " \"embedding_vecsize_per_table\": [16 for x in range(26)],\n", - " }\n", - " ],\n", - "}\n", - ")\n", - "\n", - "config = json.loads(config)\n", - "with open(os.path.join(OUTPUT_DATA_DIR, \"model_inference\", \"ps.json\"), \"w\", encoding=\"utf-8\") as f:\n", - " json.dump(config, f)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Start Triton Inference Server\n", - "\n", - "After we export the ensemble, we are ready to start the Triton Inference Server. The server is installed in the merlin-tensorflow-container. If you are not using one of our containers, then ensure it is installed in your environment. For more information, see the Triton Inference Server [documentation](https://github.com/triton-inference-server/server/blob/r22.03/README.md#documentation). \n", - "\n", - "You can start the server by running the following command:\n", - "\n", - "```shell\n", - "tritonserver --model-repository=
\n",
- "\n",
- "\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 ... \\\n",
- "70000 2714039 29401 11464 1122 9355 2 6370 1010 37 1865651 ... \n",
- "70001 3514299 27259 8072 395 9361 1 544 862 11 3292987 ... \n",
- "70002 1304577 5287 7367 2033 2899 2 712 640 36 6415968 ... \n",
- "\n",
- " I4 I5 I6 I7 I8 I9 I10 \\\n",
- "70000 0.208215 0.952671 0.955872 0.944922 0.139380 0.994092 0.056103 \n",
- "70001 0.171709 0.759526 0.795019 0.716366 0.134964 0.516737 0.065577 \n",
- "70002 0.880028 0.347701 0.207892 0.753950 0.371013 0.759502 0.201477 \n",
- "\n",
- " I11 I12 I13 \n",
- "70000 0.547473 0.709442 0.930728 \n",
- "70001 0.129782 0.471361 0.386101 \n",
- "70002 0.192447 0.085893 0.957961 \n",
- "\n",
- "[3 rows x 39 columns]"
- ]
- },
- "execution_count": 30,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "df_lib = get_lib()\n",
- "input_cols = workflow.input_schema.column_names\n",
- "# read in data for request\n",
- "data = df_lib.read_parquet(\n",
- " os.path.join(sorted(glob.glob(original_data_path + \"/*.parquet\"))[-1]),\n",
- " columns=input_cols\n",
- ")\n",
- "batch = data[:3]\n",
- "batch = batch[[x for x in batch.columns if x not in ['label']]]\n",
- "batch"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We generate a Triton Inference Server request object. \n",
- "\n",
- "Currently, `NA` and `None` values are not supported for `int32` columns. As a workaround, we will `NA` values with `0`. The output of the HugeCTR model is called `OUTPUT0`. For the same reason of dropping the target column, we need to remove it from the input schema, as well."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 31,
- "metadata": {},
- "outputs": [],
- "source": [
- "input_schema = workflow.input_schema.remove_col('label')\n",
- "inputs = convert_df_to_triton_input(\n",
- " input_schema, \n",
- " batch.fillna(0), \n",
- " grpcclient.InferInput\n",
- ")\n",
- "output_cols = ['OUTPUT0']\n",
- "outputs = [\n",
- " grpcclient.InferRequestedOutput(col)\n",
- " for col in output_cols\n",
- "]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We send the request to Triton Inference Server."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 29,
- "metadata": {},
- "outputs": [],
- "source": [
- "# send request to tritonserver\n",
- "with grpcclient.InferenceServerClient(\"localhost:8001\") as client:\n",
- " response = client.infer(\"criteo_ens\", inputs, request_id=\"1\", outputs=outputs)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We print out the predictions. The outputs are the probability scores, predicted by our model, how likely the ad will be clicked."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 35,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "OUTPUT0 [0.52164096 0.50390565 0.4957397 ] (3,)\n"
- ]
- }
- ],
- "source": [
- "for col in output_cols:\n",
- " print(col, response[col], response[col].shape)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Summary\n",
- "\n",
- "In this example, we deployed a recommender system pipeline as an ensemble. First, NVTabular created features and afterwards, HugeCTR predicted the processed data. This process ensures that the training and production environments use the same feature engineering.\n",
- "\n",
- "## Next steps\n",
- "\n",
- "There is more detailed information in the [API documentation](https://nvidia-merlin.github.io/HugeCTR/main/hugectr_user_guide.html) and [more examples](https://nvidia-merlin.github.io/HugeCTR/main/notebooks/index.html) in the [HugeCTR repository](https://github.com/NVIDIA-Merlin/HugeCTR)."
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.10"
- },
- "merlin": {
- "containers": [
- "nvcr.io/nvidia/merlin/merlin-hugectr:latest"
- ]
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/examples/scaling-criteo/README.md b/examples/scaling-criteo/README.md
index feed55e04..26e3ca28a 100644
--- a/examples/scaling-criteo/README.md
+++ b/examples/scaling-criteo/README.md
@@ -6,8 +6,7 @@ We demonstrate how to scale NVTabular, as well as:
- Use multiple GPUs and nodes with NVTabular for feature engineering.
- Train recommender system models with the Merlin Models for TensorFlow.
-- Train recommender system models with HugeCTR using multiple GPUs.
-- Inference with the Triton Inference Server and Merlin Models for TensorFlow or HugeCTR.
+- Inference with the Triton Inference Server and Merlin Models for TensorFlow.
Our recommendation is to use our latest stable [Merlin containers](https://catalog.ngc.nvidia.com/containers?filters=&orderBy=dateModifiedDESC&query=merlin) for the examples. Each notebook provides the required container.
@@ -18,9 +17,3 @@ Training and Deployment with **TensorFlow**:
- [Feature Engineering with NVTabular](02-ETL-with-NVTabular.ipynb)
- [Training with TensorFlow](03-Training-with-Merlin-Models-TensorFlow.ipynb)
- [Deploy the TensorFlow Model with Triton Inference Server](04-Triton-Inference-with-Merlin-Models-TensorFlow.ipynb)
-
-Training and Deployment with **HugeCTR**:
-- [Download and Convert](01-Download-Convert.ipynb)
-- [Feature Engineering with NVTabular](02-ETL-with-NVTabular.ipynb)
-- [Training with HugeCTR](03-Training-with-HugeCTR.ipynb)
-- [Deploy the HugeCTR Model with Triton Inference Server](04-Triton-Inference-with-HugeCTR.ipynb)
| \n", - " | C1 | \n", - "C2 | \n", - "C3 | \n", - "C4 | \n", - "C5 | \n", - "C6 | \n", - "C7 | \n", - "C8 | \n", - "C9 | \n", - "C10 | \n", - "... | \n", - "I4 | \n", - "I5 | \n", - "I6 | \n", - "I7 | \n", - "I8 | \n", - "I9 | \n", - "I10 | \n", - "I11 | \n", - "I12 | \n", - "I13 | \n", - "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 70000 | \n", - "2714039 | \n", - "29401 | \n", - "11464 | \n", - "1122 | \n", - "9355 | \n", - "2 | \n", - "6370 | \n", - "1010 | \n", - "37 | \n", - "1865651 | \n", - "... | \n", - "0.208215 | \n", - "0.952671 | \n", - "0.955872 | \n", - "0.944922 | \n", - "0.139380 | \n", - "0.994092 | \n", - "0.056103 | \n", - "0.547473 | \n", - "0.709442 | \n", - "0.930728 | \n", - "
| 70001 | \n", - "3514299 | \n", - "27259 | \n", - "8072 | \n", - "395 | \n", - "9361 | \n", - "1 | \n", - "544 | \n", - "862 | \n", - "11 | \n", - "3292987 | \n", - "... | \n", - "0.171709 | \n", - "0.759526 | \n", - "0.795019 | \n", - "0.716366 | \n", - "0.134964 | \n", - "0.516737 | \n", - "0.065577 | \n", - "0.129782 | \n", - "0.471361 | \n", - "0.386101 | \n", - "
| 70002 | \n", - "1304577 | \n", - "5287 | \n", - "7367 | \n", - "2033 | \n", - "2899 | \n", - "2 | \n", - "712 | \n", - "640 | \n", - "36 | \n", - "6415968 | \n", - "... | \n", - "0.880028 | \n", - "0.347701 | \n", - "0.207892 | \n", - "0.753950 | \n", - "0.371013 | \n", - "0.759502 | \n", - "0.201477 | \n", - "0.192447 | \n", - "0.085893 | \n", - "0.957961 | \n", - "
3 rows × 39 columns
\n", - "