add feedstock for facebookresearch/faiss

.travis.yml

@@ -29,12 +29,26 @@ branches:
 
 matrix:
   include:
-    - env: CONFIG=linux_noarch UPLOAD_PACKAGES=True DOCKER_IMAGE=ibmcom/powerai:1.7.0-dev-ubuntu18.04-py36-ppc64le
+    - env: CONFIG=linux_ppc64le_python3.6 UPLOAD_PACKAGES=True DOCKER_IMAGE=ibmcom/powerai:1.7.0-dev-ubuntu18.04-py36-ppc64le
       language: generic
       os: linux
       arch: ppc64le
 
+    - env: CONFIG=linux_ppc64le_python3.7 UPLOAD_PACKAGES=True DOCKER_IMAGE=ibmcom/powerai:1.7.0-dev-ubuntu18.04-py37-ppc64le
+      language: generic
+      os: linux
+      arch: ppc64le
+
+    - env: CONFIG=linux_python3.6 UPLOAD_PACKAGES=True DOCKER_IMAGE=ibmcom/powerai:1.7.0-dev-ubuntu18.04-py36-x86_64
+      language: generic
+      os: linux
+      arch: x86_64
+
+    - env: CONFIG=linux_python3.7 UPLOAD_PACKAGES=True DOCKER_IMAGE=ibmcom/powerai:1.7.0-dev-ubuntu18.04-py37-x86_64
+      language: generic
+      os: linux
+      arch: x86_64
+
 script:
-  -  cd conda-recipes/tensorflow-datasets-feedstock
-  -  "travis_wait 60 sleep 3600 &"
-  -  ./ci_support/run_docker_build.sh 
+  - cd conda-recipes/faiss-gpu-feedstock
+  - travis_wait 60 ./ci_support/run_docker_build.sh

conda-recipes/faiss-gpu-feedstock/.ci_support/linux_ppc64le_python3.6.yaml

@@ -0,0 +1,16 @@
+c_compiler:
+- gcc
+c_compiler_version:
+- '8'
+channel_sources:
+- defaults
+channel_targets:
+- powerai main
+docker_image:
+- ibmcom/powerai:1.7.0-dev-ubuntu18.04-py36-ppc64le
+pin_run_as_build:
+  python:
+    min_pin: x.x
+    max_pin: x.x
+python:
+- '3.6'

conda-recipes/faiss-gpu-feedstock/.ci_support/linux_ppc64le_python3.7.yaml

@@ -0,0 +1,16 @@
+c_compiler:
+- gcc
+c_compiler_version:
+- '8'
+channel_sources:
+- defaults
+channel_targets:
+- powerai main
+docker_image:
+- ibmcom/powerai:1.7.0-dev-ubuntu18.04-py37-ppc64le
+pin_run_as_build:
+  python:
+    min_pin: x.x
+    max_pin: x.x
+python:
+- '3.7'

conda-recipes/faiss-gpu-feedstock/.ci_support/linux_python3.6.yaml

@@ -0,0 +1,16 @@
+c_compiler:
+- gcc
+c_compiler_version:
+- '8'
+channel_sources:
+- defaults
+channel_targets:
+- powerai main
+docker_image:
+- ibmcom/powerai:1.7.0-dev-ubuntu18.04-py36-x86_64
+pin_run_as_build:
+  python:
+    min_pin: x.x
+    max_pin: x.x
+python:
+- '3.6'

conda-recipes/faiss-gpu-feedstock/.ci_support/linux_python3.7.yaml

@@ -0,0 +1,16 @@
+c_compiler:
+- gcc
+c_compiler_version:
+- '8'
+channel_sources:
+- defaults
+channel_targets:
+- powerai main
+docker_image:
+- ibmcom/powerai:1.7.0-dev-ubuntu18.04-py37-x86_64
+pin_run_as_build:
+  python:
+    min_pin: x.x
+    max_pin: x.x
+python:
+- '3.7'

conda-recipes/faiss-gpu-feedstock/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Facebook, Inc. and its affiliates.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

conda-recipes/faiss-gpu-feedstock/README-recipe.md

@@ -0,0 +1,3 @@
+conda recipe adapted from the recipe in the upstream faiss repo:
+
+https://github.com/facebookresearch/faiss/tree/master/conda/faiss-gpu

conda-recipes/faiss-gpu-feedstock/README.md

@@ -0,0 +1,91 @@
+# Faiss 
+
+Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy. Some of the most useful algorithms are implemented on the GPU. It is developed by [Facebook AI Research](https://research.fb.com/category/facebook-ai-research-fair/).
+
+## NEWS
+
+*NEW: version 1.6.1 (2019-11-29) bugfix.*
+
+*NEW: version 1.6.0 (2019-10-15) code structure reorg, support for codec interface.*
+
+*NEW: version 1.5.3 (2019-06-24) fix performance regression in IndexIVF.*
+
+*NEW: version 1.5.2 (2019-05-27) the license was relaxed to MIT from BSD+Patents. Read LICENSE for details.*
+
+*NEW: version 1.5.0 (2018-12-19) GPU binary flat index and binary HNSW index*
+
+*NEW: version 1.4.0 (2018-08-30) no more crashes in pure Python code*
+
+*NEW: version 1.3.0 (2018-07-12) support for binary indexes*
+
+*NEW: latest commit (2018-02-22) supports on-disk storage of inverted indexes, see demos/demo_ondisk_ivf.py*
+
+*NEW: latest commit (2018-01-09) includes an implementation of the HNSW indexing method, see benchs/bench_hnsw.py*
+
+*NEW: there is now a Facebook public discussion group for Faiss users at https://www.facebook.com/groups/faissusers/*
+
+*NEW: on 2017-07-30, the license on Faiss was relaxed to BSD from CC-BY-NC. Read LICENSE for details.*
+
+## Introduction
+
+Faiss contains several methods for similarity search. It assumes that the instances are represented as vectors and are identified by an integer, and that the vectors can be compared with L2 (Euclidean) distances or dot products. Vectors that are similar to a query vector are those that have the lowest L2 distance or the highest dot product with the query vector. It also supports cosine similarity, since this is a dot product on normalized vectors.
+
+Most of the methods, like those based on binary vectors and compact quantization codes, solely use a compressed representation of the vectors and do not require to keep the original vectors. This generally comes at the cost of a less precise search but these methods can scale to billions of vectors in main memory on a single server. 
+
+The GPU implementation can accept input from either CPU or GPU memory. On a server with GPUs, the GPU indexes can be used a drop-in replacement for the CPU indexes (e.g., replace `IndexFlatL2` with `GpuIndexFlatL2`) and copies to/from GPU memory are handled automatically. Results will be faster however if both input and output remain resident on the GPU. Both single and multi-GPU usage is supported.
+
+## Building 
+
+The library is mostly implemented in C++, with optional GPU support provided via CUDA, and an optional Python interface. The CPU version requires a BLAS library. It compiles with a Makefile and can be packaged in a docker image. See [INSTALL.md](INSTALL.md) for details.
+
+## How Faiss works
+
+Faiss is built around an index type that stores a set of vectors, and provides a function to search in them with L2 and/or dot product vector comparison. Some index types are simple baselines, such as exact search. Most of the available indexing structures correspond to various trade-offs with respect to
+
+- search time
+- search quality
+- memory used per index vector 
+- training time
+- need for external data for unsupervised training
+
+The optional GPU implementation provides what is likely (as of March 2017) the fastest exact and approximate (compressed-domain) nearest neighbor search implementation for high-dimensional vectors, fastest Lloyd's k-means, and fastest small k-selection algorithm known. [The implementation is detailed here](https://arxiv.org/abs/1702.08734).
+
+## Full documentation of Faiss
+
+The following are entry points for documentation: 
+
+- the full documentation, including a [tutorial](https://github.com/facebookresearch/faiss/wiki/Getting-started), a [FAQ](https://github.com/facebookresearch/faiss/wiki/FAQ) and a [troubleshooting section](https://github.com/facebookresearch/faiss/wiki/Troubleshooting) can be found on the [wiki page](http://github.com/facebookresearch/faiss/wiki)
+- the [doxygen documentation](http://rawgithub.com/facebookresearch/faiss/master/docs/html/annotated.html) gives per-class information
+- to reproduce results from our research papers, [Polysemous codes](https://arxiv.org/abs/1609.01882) and [Billion-scale similarity search with GPUs](https://arxiv.org/abs/1702.08734), refer to the [benchmarks README](benchs/README.md). For [
+Link and code: Fast indexing with graphs and compact regression codes](https://arxiv.org/abs/1804.09996), see the [link_and_code README](benchs/link_and_code)
+
+## Authors
+
+The main authors of Faiss are:
+- [Hervé Jégou](https://github.com/jegou) initiated the Faiss project and wrote its first implementation
+- [Matthijs Douze](https://github.com/mdouze) implemented most of the CPU Faiss
+- [Jeff Johnson](https://github.com/wickedfoo) implemented all of the GPU Faiss
+- [Lucas Hosseini](https://github.com/beauby) implemented the binary indexes
+
+## Reference
+
+Reference to cite when you use Faiss in a research paper:
+
+```
+@article{JDH17,
+  title={Billion-scale similarity search with GPUs},
+  author={Johnson, Jeff and Douze, Matthijs and J{\'e}gou, Herv{\'e}},
+  journal={arXiv preprint arXiv:1702.08734},
+  year={2017}
+}
+```
+
+## Join the Faiss community
+
+For public discussion of Faiss or for questions, there is a Facebook public discussion group at https://www.facebook.com/groups/faissusers/
+
+We monitor the [issues page](http://github.com/facebookresearch/faiss/issues) of the repository. You can report bugs, ask questions, etc.
+
+## License
+
+Faiss is MIT-licensed.

conda-recipes/faiss-gpu-feedstock/ci_support/build_steps.sh

@@ -0,0 +1,51 @@
+# (C) Copyright IBM Corp. 2018, 2019. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#!/usr/bin/env bash
+
+set -xeuo pipefail
+export PYTHONUNBUFFERED=1
+export FEEDSTOCK_ROOT=/home/conda/feedstock_root
+export RECIPE_ROOT=/home/conda/recipe_root
+export CI_SUPPORT=/home/conda/feedstock_root/.ci_support
+export CONFIG_FILE="${CI_SUPPORT}/${CONFIG}.yaml"
+export PATH=/opt/anaconda/bin:$PATH
+
+cat >~/.condarc <<CONDARC
+
+conda-build:
+ root-dir: /home/conda/feedstock_root/build_artifacts
+
+CONDARC
+
+conda install --yes --quiet conda-forge-ci-setup=2 conda-build -c conda-forge
+
+# set up the condarc
+setup_conda_rc "${FEEDSTOCK_ROOT}" "${RECIPE_ROOT}" "${CONFIG_FILE}"
+
+# make the build number clobber
+make_build_number "${FEEDSTOCK_ROOT}" "${RECIPE_ROOT}" "${CONFIG_FILE}"
+
+conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
+conda config --set channel_priority strict
+export IBM_POWERAI_LICENSE_ACCEPT=yes
+
+conda build "${RECIPE_ROOT}" -m "${CI_SUPPORT}/${CONFIG}.yaml" \
+    --clobber-file "${CI_SUPPORT}/clobber_${CONFIG}.yaml"
+
+if [[ "${UPLOAD_PACKAGES}" != "False" ]]; then
+    upload_package "${FEEDSTOCK_ROOT}" "${RECIPE_ROOT}" "${CONFIG_FILE}"
+fi
+
+touch "/home/conda/feedstock_root/build_artifacts/conda-forge-build-done-${CONFIG}"

conda-recipes/faiss-gpu-feedstock/ci_support/run_docker_build.sh

@@ -0,0 +1,67 @@
+# (C) Copyright IBM Corp. 2018, 2020. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#!/usr/bin/env bash
+
+THISDIR="$( cd "$( dirname "$0" )" >/dev/null && pwd )"
+PROVIDER_DIR="$(basename $THISDIR)"
+
+FEEDSTOCK_ROOT=$(cd "$(dirname "$0")/.."; pwd;)
+RECIPE_ROOT=$FEEDSTOCK_ROOT/recipe
+
+docker info
+
+# In order for the conda-build process in the container to write to the mounted
+# volumes, we need to run with the same id as the host machine, which is
+# normally the owner of the mounted volumes, or at least has write permission
+export HOST_USER_ID=$(id -u)
+# Check if docker-machine is being used (normally on OSX) and get the uid from
+# the VM
+if hash docker-machine 2> /dev/null && docker-machine active > /dev/null; then
+    HOST_USER_ID=$(docker-machine ssh $(docker-machine active) id -u)
+fi
+
+ARTIFACTS="$FEEDSTOCK_ROOT/build_artifacts"
+mkdir -p "$ARTIFACTS"
+DONE_CANARY="$ARTIFACTS/conda-forge-build-done-${CONFIG}"
+rm -f "$DONE_CANARY"
+
+# Enable running in interactive mode ONLY if attached to a tty,
+# so "-t" is made conditional
+test -t 1 && USE_TTY="-t"
+DOCKER_RUN_ARGS=" -i ${USE_TTY} "
+
+if [ -z "${DOCKER_IMAGE}" ]; then
+  echo "WARNING: DOCKER_IMAGE variable not set. Falling back to condaforge/linux-anvil-ppc64le"
+  DOCKER_IMAGE="condaforge/linux-anvil-ppc64le"
+fi
+
+docker run ${DOCKER_RUN_ARGS} \
+                        -v "${RECIPE_ROOT}":/home/conda/recipe_root:ro,z \
+                        -v "${FEEDSTOCK_ROOT}":/home/conda/feedstock_root:rw,z \
+                        -e CONFIG \
+                        -e BINSTAR_TOKEN \
+                        -e HOST_USER_ID \
+                        -e UPLOAD_PACKAGES \
+                        -e CI \
+                        -a stdin -a stdout -a stderr -u root \
+                        $DOCKER_IMAGE \
+                        bash \
+                        /home/conda/feedstock_root/${PROVIDER_DIR}/build_steps.sh
+
+# double-check that the build got to the end
+# see https://github.com/conda-forge/conda-smithy/pull/337
+# for a possible fix
+
+test -f "$DONE_CANARY"

conda-recipes/faiss-gpu-feedstock/recipe/0001-add-tolerance-for-ppc64le.patch

@@ -0,0 +1,28 @@
+From 65775f8725760e4bc7eb9b6b8ab8f9bdd6a26ed2 Mon Sep 17 00:00:00 2001
+From: "Brian W. Hart" <[email protected]>
+Date: Mon, 16 Mar 2020 20:21:31 +0000
+Subject: [PATCH] add tolerance for ppc64le
+
+numpy's CPU info dictionary does not have a 'flags' field on
+ppc64le. Tolerate a lookup failure when checking for instruction
+set to allow faiss to run on ppc64le.
+---
+ python/faiss.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/python/faiss.py b/python/faiss.py
+index 8973731..163e2cb 100644
+--- a/python/faiss.py
++++ b/python/faiss.py
+@@ -28,7 +28,7 @@ def instruction_set():
+             return "default"
+     elif platform.system() == "Linux":
+         import numpy.distutils.cpuinfo
+-        if "avx2" in numpy.distutils.cpuinfo.cpu.info[0]['flags']:
++        if "avx2" in numpy.distutils.cpuinfo.cpu.info[0].get('flags', ""):
+             return "AVX2"
+         else:
+             return "default"
+-- 
+1.8.3.1
+

conda-recipes/faiss-gpu-feedstock/recipe/0001-faiss-update-CUDA-include-path-for-WML-CE-cudatoolki.patch

@@ -0,0 +1,29 @@
+From f58b6330a636433ca8d80e9395dcdb29cbef09ba Mon Sep 17 00:00:00 2001
+From: "Brian W. Hart" <[email protected]>
+Date: Tue, 17 Mar 2020 19:10:52 +0000
+Subject: [PATCH] faiss: update CUDA include path for WML CE cudatoolkit
+ packages
+
+For building against WML CE cudatoolkit packages, we can always
+expect to find the CUDA headers in the top-level include/
+directory of the build environment.
+---
+ makefile.inc.in | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/makefile.inc.in b/makefile.inc.in
+index 19d8511..fde9eb7 100644
+--- a/makefile.inc.in
++++ b/makefile.inc.in
+@@ -16,7 +16,7 @@ SWIGFLAGS    = -DSWIGWORDSIZE64
+ NVCC         = @NVCC@
+ CUDA_ROOT    = @CUDA_PREFIX@
+ CUDA_ARCH    = @CUDA_ARCH@
+-NVCCFLAGS    = -I $(CUDA_ROOT)/targets/x86_64-linux/include/ \
++NVCCFLAGS    = -I $(CUDA_ROOT)/include/ \
+ -Xcompiler -fPIC \
+ -Xcudafe --diag_suppress=unrecognized_attribute \
+ $(CUDA_ARCH) \
+-- 
+1.8.3.1
+