Merge pull request #1 from tizhou86/master

Init commit for gpu sharing device.

.gitignore

@@ -0,0 +1,131 @@
+# OSX leaves these everywhere on SMB shares
+._*
+
+# OSX trash
+.DS_Store
+
+# Eclipse files
+.classpath
+.project
+.settings/**
+
+# Files generated by JetBrains IDEs, e.g. IntelliJ IDEA
+.idea/
+*.iml
+
+# Vscode files
+.vscode
+
+# This is where the result of the go build goes
+/output*/
+/_output*/
+/_output
+
+# Emacs save files
+*~
+\#*\#
+.\#*
+
+# Vim-related files
+[._]*.s[a-w][a-z]
+[._]s[a-w][a-z]
+*.un~
+Session.vim
+.netrwhist
+
+# cscope-related files
+cscope.*
+
+# Go test binaries
+*.test
+/hack/.test-cmd-auth
+
+# JUnit test output from ginkgo e2e tests
+/junit*.xml
+
+# Mercurial files
+**/.hg
+**/.hg*
+
+# Vagrant
+.vagrant
+network_closure.sh
+
+# Local cluster env variables
+/cluster/env.sh
+
+# Compiled binaries in third_party
+/third_party/pkg
+
+# Also ignore etcd installed by hack/install-etcd.sh
+/third_party/etcd*
+/default.etcd
+
+# User cluster configs
+.kubeconfig
+
+.tags*
+
+# Version file for dockerized build
+.dockerized-kube-version-defs
+
+# Web UI
+/www/master/node_modules/
+/www/master/npm-debug.log
+/www/master/shared/config/development.json
+
+# Karma output
+/www/test_out
+
+# precommit temporary directories created by ./hack/verify-generated-docs.sh and ./hack/lib/util.sh
+/_tmp/
+/doc_tmp/
+
+# Test artifacts produced by Jenkins jobs
+/_artifacts/
+
+# Go dependencies installed on Jenkins
+/_gopath/
+
+# Config directories created by gcloud and gsutil on Jenkins
+/.config/gcloud*/
+/.gsutil/
+
+# CoreOS stuff
+/cluster/libvirt-coreos/coreos_*.img
+
+# Juju Stuff
+/cluster/juju/charms/*
+/cluster/juju/bundles/local.yaml
+
+# Downloaded Kubernetes binary release
+/kubernetes/
+
+# direnv .envrc files
+.envrc
+
+# Downloaded kubernetes binary release tar ball
+kubernetes.tar.gz
+
+# generated files in any directory
+# TODO(thockin): uncomment this when we stop committing the generated files.
+#zz_generated.*
+#zz_generated.openapi.go
+
+# make-related metadata
+/.make/
+# Just in time generated data in the source, should never be commited
+/test/e2e/generated/bindata.go
+
+# This file used by some vendor repos (e.g. github.com/go-openapi/...) to store secret variables and should not be ignored
+!\.drone\.sec
+
+/bazel-*
+*.pyc
+
+# e2e log files
+*.log
+
+# test coverage file
+coverage.txt
+

LICENSE

@@ -1,3 +1,4 @@
+
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/

Makefile

@@ -0,0 +1,63 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+.PHONY: all build builder test
+.DEFAULT_GOAL := all
+
+##### Global variables #####
+
+DOCKER   ?= docker
+REGISTRY ?= volcanosh
+VERSION  ?= 1.0.0
+
+##### Public rules #####
+
+all: ubuntu16.04 centos7
+
+push:
+	$(DOCKER) push "$(REGISTRY)/k8s-device-plugin:$(VERSION)-ubuntu16.04"
+	$(DOCKER) push "$(REGISTRY)/k8s-device-plugin:$(VERSION)-centos7"
+
+push-short:
+	$(DOCKER) tag "$(REGISTRY)/k8s-device-plugin:$(VERSION)-ubuntu16.04" "$(REGISTRY)/k8s-device-plugin:$(VERSION)"
+	$(DOCKER) push "$(REGISTRY)/k8s-device-plugin:$(VERSION)"
+
+push-latest:
+	$(DOCKER) tag "$(REGISTRY)/k8s-device-plugin:$(VERSION)-ubuntu16.04" "$(REGISTRY)/k8s-device-plugin:latest"
+	$(DOCKER) push "$(REGISTRY)/k8s-device-plugin:latest"
+
+ubuntu16.04:
+	$(DOCKER) build --pull \
+		--tag $(REGISTRY)/k8s-device-plugin:$(VERSION)-ubuntu16.04 \
+		--file docker/amd64/Dockerfile.ubuntu16.04 .
+
+centos7:
+	$(DOCKER) build --pull \
+		--tag $(REGISTRY)/k8s-device-plugin:$(VERSION)-centos7 \
+		--file docker/amd64/Dockerfile.centos7 .
+
+include Makefile.def
+
+BIN_DIR=_output/bin
+RELEASE_DIR=_output/release
+REL_OSARCH=linux/amd64
+
+init:
+	mkdir -p ${BIN_DIR}
+	mkdir -p ${RELEASE_DIR}
+
+gen_bin: init
+	go get github.com/mitchellh/gox
+	CGO_ENABLED=1 gox -osarch=${REL_OSARCH} -ldflags ${LD_FLAGS} -output ${BIN_DIR}/${REL_OSARCH}/k8s-device-plugin ./

Makefile.def

@@ -0,0 +1,11 @@
+
+# If tag not explicitly set in users default to the git sha.
+TAG ?= $(shell git rev-parse --verify HEAD)
+GitSHA=`git rev-parse HEAD`
+Date=`date "+%Y-%m-%d %H:%M:%S"`
+RELEASE_VER=latest
+LD_FLAGS=" \
+    -X '${REPO_PATH}/pkg/version.GitSHA=${GitSHA}' \
+    -X '${REPO_PATH}/pkg/version.Built=${Date}'   \
+    -X '${REPO_PATH}/pkg/version.Version=${RELEASE_VER}'"
+

README.md

@@ -1,2 +1,161 @@
-# volcano-devices
-Device plugins for Volcano, e.g. GPU
+# Volcano device plugin for Kubernetes
+
+**Note**:
+This is based on [Nvidia Device Plugin](https://github.com/NVIDIA/k8s-device-plugin) to support soft isolation of GPU card.
+And collaborate with volcano, it is possible to enable GPU sharing.
+
+## Table of Contents
+
+- [About](#about)
+- [Prerequisites](#prerequisites)
+- [Quick Start](#quick-start)
+  - [Preparing your GPU Nodes](#preparing-your-gpu-nodes)
+  - [Enabling GPU Support in Kubernetes](#enabling-gpu-support-in-kubernetes)
+  - [Running GPU Sharing Jobs](#running-gpu-jobs)
+- [Docs](#docs)
+- [Issues and Contributing](#issues-and-contributing)
+
+
+## About
+
+The Volcano device plugin for Kubernetes is a Daemonset that allows you to automatically:
+- Expose the number of GPUs on each node of your cluster
+- Keep track of the health of your GPUs
+- Run GPU enabled containers in your Kubernetes cluster.
+
+This repository contains Volcano's official implementation of the [Kubernetes device plugin](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md).
+
+## Prerequisites
+
+The list of prerequisites for running the Volcano device plugin is described below:
+* NVIDIA drivers ~= 384.81
+* nvidia-docker version > 2.0 (see how to [install](https://github.com/NVIDIA/nvidia-docker) and it's [prerequisites](https://github.com/nvidia/nvidia-docker/wiki/Installation-\(version-2.0\)#prerequisites))
+* docker configured with nvidia as the [default runtime](https://github.com/NVIDIA/nvidia-docker/wiki/Advanced-topics#default-runtime).
+* Kubernetes version >= 1.10
+
+## Quick Start
+
+### Preparing your GPU Nodes
+
+The following steps need to be executed on all your GPU nodes.
+This README assumes that the NVIDIA drivers and nvidia-docker have been installed.
+
+Note that you need to install the nvidia-docker2 package and not the nvidia-container-toolkit.
+This is because the new `--gpus` options hasn't reached kubernetes yet. Example:
+```bash
+# Add the package repositories
+$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
+$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
+$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
+
+$ sudo apt-get update && sudo apt-get install -y nvidia-docker2
+$ sudo systemctl restart docker
+```
+
+You will need to enable the nvidia runtime as your default runtime on your node.
+We will be editing the docker daemon config file which is usually present at `/etc/docker/daemon.json`:
+```json
+{
+    "default-runtime": "nvidia",
+    "runtimes": {
+        "nvidia": {
+            "path": "/usr/bin/nvidia-container-runtime",
+            "runtimeArgs": []
+        }
+    }
+}
+```
+> *if `runtimes` is not already present, head to the install page of [nvidia-docker](https://github.com/NVIDIA/nvidia-docker)*
+
+### Enabling GPU Support in Kubernetes
+
+Once you have enabled this option on *all* the GPU nodes you wish to use,
+you can then enable GPU support in your cluster by deploying the following Daemonset:
+
+```shell
+$ kubectl create -f volcano-device-plugin.yml
+```
+
+### Running GPU Sharing Jobs
+
+NVIDIA GPUs can now be shared via container level resource requirements using the resource name volcano.sh/gpu-memory:
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: gpu-pod1
+spec:
+  containers:
+    - name: cuda-container
+      image: nvidia/cuda:9.0-devel
+      resources:
+        limits:
+          volcano.sh/gpu-memory: 1024 # requesting 1024MB GPU memory
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: gpu-pod2
+spec:
+  containers:
+    - name: cuda-container
+      image: nvidia/cuda:9.0-devel
+      resources:
+        limits:
+          volcano.sh/gpu-memory: 1024 # requesting 1024MB GPU memory
+```
+
+> **WARNING:** *if you don't request GPUs when using the device plugin with NVIDIA images all
+> the GPUs on the machine will be exposed inside your container.*
+
+## Docs
+
+Please note that:
+- the device plugin feature is beta as of Kubernetes v1.11.
+- the Volcano device plugin is alpha and is missing
+    - More comprehensive GPU health checking features
+    - GPU cleanup features
+    - GPU hard isolation
+    - ...
+
+The next sections are focused on building the device plugin and running it.
+
+### With Docker
+
+#### Build
+```shell
+$ make ubuntu16.04.
+```
+
+#### Run locally
+```shell
+$ docker run --security-opt=no-new-privileges --cap-drop=ALL --network=none -it -v /var/lib/kubelet/device-plugins:/var/lib/kubelet/device-plugins nvidia/k8s-device-plugin:{version}
+```
+
+#### Deploy as DaemonSet:
+```shell
+$ kubectl create -f nvidia-device-plugin.yml
+```
+
+# Issues and Contributing
+[Checkout the Contributing document!](CONTRIBUTING.md)
+
+* You can report a bug by [filing a new issue](https://github.com/volcano-sh/k8s-device-plugin/issues/new)
+* You can contribute by opening a [pull request](https://help.github.com/articles/using-pull-requests/)
+
+## Versioning
+
+The version exactly matches with [Volcano](https://github.com/volcano-sh/volcano).
+
+## Upgrading Kubernetes with the device plugin
+
+Upgrading Kubernetes when you have a device plugin deployed doesn't require you to do any,
+particular changes to your workflow.
+The API is versioned and is pretty stable (though it is not guaranteed to be non breaking),
+upgrading kubernetes won't require you to deploy a different version of the device plugin and you will
+see GPUs re-registering themselves after you node comes back online.
+
+
+Upgrading the device plugin is a more complex task. It is recommended to drain GPU tasks as
+we cannot guarantee that GPU tasks will survive a rolling upgrade.
+However we make best efforts to preserve GPU tasks during an upgrade.