v0.14.0

SMQTK v0.14.0 Release Notes

Notable updates with this release:

• Simplification and vectorization of a few algorithm APIs.
• New algorithm implementations and updates to existing ones.
• Beginning to use docker-compose configuration to define the build
  configurations of various images, beginning with an image to provide FAISS as
  a TPL dependency.
• Renamed DescriptorIndex to DescriptorSet to reduce confusion on
  implied functionality.

Notable fixes with this release:

• Fixed issue with smqtk.utils.parallel.parallel_map to not hang on
  keyboard interrupts.

Updates / New Features

Algorithms

• Classifier

  • Overhauled interface API to have the abstract method be a many-to-many
    iterator instead of the previous one-to-one signature.

  • Updated implementations and usages of this interface throughout SMQTK.

  • Added wrapper for scikit-learn LogisticRegression classifier.

• DescriptorGenerator

  • Overhauled interface API to have the abstract method be a many-to-many
    iterator instead of the previous one-to-one signature.

  • Updated colordescriptor implementation for interface API update.

  • Updated caffe implementation for interface API update.

  • Updated KWCNN implementation for interface API update.

• NearestNeighborsIndex

  • FAISS

    • Exposed nprobe parameter for when using IVF type indices to be
      utilized at query time.

• RelevancyIndex

  • Added NoIndexError exception for when attempting to perform ranking
    before an index is built.

  • Added SupervisedClassifierRelevancyIndex to enable using any available
    supervised classifier implementation to satisfy the RelevancyIndex API.

Compute Functions

• Updated smqtk.compute_functions.compute_many_descriptors to utilize new
  DescriptorGenerator API.

Docker

• Started use of docker-compose YAML file to organize image building.

• Added FAISS TPL image to be copied from by utilizing images.

• IQR "Playground"

  • Fixed compute test scripts to use updated DescriptorGenerator API.

Documentation

• Updated docs/algorithminterfaces.rst to reflect the new
  DescriptorGenerator API.

• Updated docs/algorithmmodels.rst to reflect the new DescriptorGenerator
  API.

• Updated the docs/examples/caffe_build_index.rst example to use the new
  DescriptorGenerator API.

• Updated the docs/examples/simple_feature_computation.rst example to use
  the new DescriptorGenerator API.

IQR

• Remove forcing of relevancy scores in refine when a result element is
  contained in the positive or negative exemplar or adjudication sets. This is
  because a user of an IqrSession instance can determine this intersection
  optionally outside of the class, so this forcing of the values is a loss of
  information.

• Added accessor functions to specific segments of the relevancy result
  predictions: positively adjudicated, negatively adjudicated and
  not-adjudicated elements.

Misc.

• Cleaned up various test warnings.

Representation

• AxisAlignedBoundingBox

  • Added intersection method.

• Data Element

  • Added PostgreSQL implementation.

• DataSet

  • Added PostgreSQL implementation, storing data representation natively in
    the database.

• DetectionElement

  • Added individual component accessors.

• Renamed "DescriptorIndex" to "DescriptorSet" in order to better represent
  what the structure and API represents. "Index" can carry the connotation that
  more is happening within the structure than actually is.

Tests

• Updated colordescriptor DescriptorGenerator tests to "skip" when deemed not
  available so that the tests are not just hidden when the optional
  dependencies are not present.

• Updated dummy classes used in classifier service unit tests to match the new
  DescriptorGenerator API.

• Update IQR service unit tests stub class for the new DescriptorGenerator API
  and iteration properties.

• Updated various class unit tests to make use of new configuration test helper
  function.

• Added a skip mark to ContextualReadWriteLock class unit tests which
  currently fail non-deterministically. This class is currently not used within
  SMQTK and a user-warning is now emitted when an attempted construction of
  this class occurs.

Tools / Scripts

• Updated the smqtk.bin.classifyFiles tool to use the new
  DescriptorGenerator API.

• Updated the smqtk.bin.computeDescriptor tool to use the new
  DescriptorGenerator API.

• Updated the smqtk.bin.iqr_app_model_generation tool to use the new
  DescriptorGenerator API.

• Updated some old MEMEX scripts to use the new DescriptorGenerator API.

Utils

• Added additional description capability to ProgressReporter.

• Added a return of self in the ContentTypeValidator.raise_valid_element()
  method.

• Added helper function for testing Configurable mixing instance functionality.

• Promoted service proxy helper class from IQR service server to a general web
  utility.

• Update random character generator to use random.SystemRandom which, at
  least for Posix systems, uses a source suitable for cryptographic purposes.

• Expanded debug logging enabling options in runApplication tool.

• Added --use-simple-cors option to the runApplication tool to enable
  CORS for all domains on all routes.

Web

• Added endpoints IQR headless service for expanded getter methods added to
  IqrSession class.

• Changed IQR web service endpoint to retrieve nearest-neighbors to a GET
  method instead of the previous POST method, as the previous method did not
  make sense for the request being made.

• Fixed usage of DescriptorGenerator instances in the classifier service for
  the API update.

• Updated smqtk.web.descriptor_service to use the new DescriptorGenerator
  API.

• Updated smqtk.web.iqr_service to use the new DescriptorGenerator API.

• Updated smqtk.web.nearestneighbor_service to use the new
  DescriptorGenerator API.

Fixes

Algorithms

• DescriptorGenerator

  • Caffe

    • Fix configuration overrides to correctly handle configuration from JSON.

    • Coerce unicode arguments to Net constructor to strings (or bytes in
      python 3).

    • Fixed numpy load call to explicitly allow loading pickled components due
      to a parameter default change in numpy version 1.16.3.

• HashIndex

  • SkLearnBallTreeHashIndex

    • Fixed numpy load call to explicitly allow loading pickled components due
      to a parameter default change in numpy version 1.16.3.

• ImageMatrixObjectDetector

  • Add abstractmethod decorator to intermediate implementation of
    get_config method.

Documentation

• Add missing reference to v0.13.0 change notes.

Tests

• Fixed PostgreSQL KeyValueStore implementation unit test that became
  non-deterministic in Python 3+.

Utilities

• Fixed issue with ProgressReporter when reporting before the first interval
  period.

• Fixed issue with smqtk.utils.parallel.parallel_map function where it
  could hang during threading-mode when a keyboard interrupt occurred.

• Fixed incorrectly calling the module-level debug logging function to use the
  locally passed logger, cleaning up a duplicate logging issue.

Web

• Classifier Service

  • Fix configuration of CaffeDescriptorGenerator.

• IQR Service

  • Fix configuration of CaffeDescriptorGenerator.

v0.13.0

SMQTK v0.13.0 Release Notes

This release incorporates updates and fixes performed on the VIGILANT project
and approved for public release (case number 88ABW-2019-5287).
Some of the major updates and fixes in this include:

• Object detection algorithm interface and supporting DetectionElement
  interface and implementations.
• Revised plugin implementation accessor via the mixin class instead what used
  to be manually implemented side-car functions for every interface. Also moved
  some configuration specific functions out of the plugin utility module and
  into a configuration utility submodule, where the Configurable mixin
  class has also moved to.
• Moves unit tests out of the installed SMQTK package and into a dedicated
  sub-directory in the repository.

Updates / New Features

Algorithms

• Added ImageReader algorithm interface

  • Added matrix reading short-cut if DataElement instance provided has a
    matrix attribute/property.

  • Added PIL (pillow) implementation with tests.

  • Added GDAL implementation with tests.

• Descriptor Generators

  • Change CaffeDescriptorGenerator constructor to take DataElement
    instances rather than URIs.

• HashIndex

  • SkLearnBallTreeHashIndex

    • Fixed numpy load call to explicitly allow loading pickled components due
      to a parameter default change in numpy version 1.16.3.

• Object Detection

  • Added initial abstract interface.

  • Added "ImageMatrixObjectDetector" interface for object detectors that
    specifically operate on image data and standardizes the use of an
    "ImageReader" algorithm to provide the pixel matrix as input.

• Nearest Neighbors

  • FAISS

    • Gracefully handle addition of duplicated descriptors to avoid making
      index unusable due to an unexpected external failure.

    • Make use of new get_many method of key-value stores to improve
      runtime performance.

    • Make use of new get_many_vectors classmethod of DescriptorElement to
      improve runtime performance.

  • LSH Hash Functor

    • Use ProgressReporter in itq to avoid bugs from deprecated
      report_progress function

Compute Functions

• Add compute_transformed_descriptors function to compute_functions.py for
  conducting searches with augmented copies of an image

Misc.

• Updated numpy version in requirements.txt to current versions. Also split
  versioning between python 2 and 3 due to split availability.

• Resolve python static analysis warnings and errors.

Representation

• Added AxisAlignedBoundingBox class for describing N-dimensional euclidean spatial
  regions.

• Added DetectionElement interface, and in-memory implementation, with
  associated unit tests.

• Added DetectionElementFactory class for factory construction of
  DetectionElement instances.

• Add use of smqtk.utils.configuration.cls_conf_from_config_dict and
  smqtk.utils.configuration.cls_conf_to_config_dict to appropriate
  methods in factory classes.

• Add get_many method to KeyValueStore interface class and provide an
  optimized implementation of it for the PostgresKeyValueStore
  implementation class.

• Add get_many_vectors classmethod for efficiently retrieving vectors from
  several descriptor elements at once

• Add efficient implementation of _get_many_vectors for Postgres descriptor
  elements.

• Updated MemoryKeyValueStore.add_many to use dict.update method
  instead of manually updating keys.

• Removed unnecessary method override in DataFileElement.

• Added MatrixDataElement representation that stores a numpy.ndarray
  instance internally, generating bytes on-the-fly when requested.

• DataMemoryElement now raises a TypeError if a non-bytes-line object is
  passed during construction or setting of bytes. Configuration mixin hooks
  have been updated to convert to and from strings for JSON-compliant
  dictionary input and output. Fixed various usages of DataMemoryElement to
  actually pass bytes.

Tests

• Moved tests out of main package tree.

• Added use of pytest-runner in setup.py, removing run_tests.sh
  script. New method of running tests is python setup.py test.

Utilities

• Added to Pluggable interface the get_impls method, replacing the
  separate get_*_impls functions defined for each interface type. Removed
  previous get_*_impls functions from algorithm and representation
  interfaces, adjusting tests and utilities as appropriate.

• Renamed smqtk.utils.configurable to smqtk.utils.configuration.
  Ramifications fixed throughout the codebase. Added documentation to
  doc-strings.

• Added cls_conf_from_config_dict and cls_conf_to_config_dict
  intermediate helper functions to smqtk.utils.configuration for the
  from_config_dict and to_config_dict sub-problems, respectively.
  This was motivated by duplicated functionality in element factory class
  from_config and get_config methods.

• Moved some helper functions from smqtk.utils.pluginto
  smqtk.utils.configuration as those functions more specifically had to do
  with configuration dictionary construction and manipulation. Ramifications
  fixed throughout the codebase.

• Updated smqtk.utils.plugin.get_plugins signature and return. Now more
  simply takes the interface class (previously referred to as the base-class)
  instead of the original first two positional, string arguments as they could
  be easily introspected from the interface class object. Ramifications fixed
  throughout the codebase.

• Added ContentTypeValidator interface for algorithms that operate on raw
  DataElement instances, providing methods for validating reported content
  types against a sub-class defined set of "valid" types. Applied to
  DescriptorGenerator interface.

• Replace usage of smqtk.utils.bin_utils.report_progress with the
  ProgressReporter class throughout package.

• Removed bundled "jsmin" in favor of using pip installed package.

• Moved merge_dict out of smqtk/utils/__init__.py and into its own
  module.

• Created combinatorics utils module, moved ncr function to here.

• Renamed various utility modules that included _utils in their name to not
  include _utils for the sake of reducing redundancy.

• Removed FileModificationMonitor utility class due to having no current
  use anywhere as well as its tests non-deterministically failing (issues
  with timing and probably lack of sufficient use of mock, time to fix not
  worth its lack of use). The watchdog python package should be used
  instead.

• Added entry-point extension method of plugin discovery.

• Added warning to smqtk.utils.file.safe_file_write when used on Windows
  platforms.

Fixes

Algorithms

• Nearest Neighbors

  • FAISS

    • Fix issue with storing and retrieving index IDs as numpy types by casting
      to python native integers due to an incompatibility with some
      KeyValueStore implementations (specificially an issue with the PostgreSQL
      implementation).

Misc.

• Removed some unused imports.

Representation

• Fixed bug with ClassificationElement.max_label where an exception would
  be raised if there was no label with associated confidence greater than 0.

• Fix some postgres test comparisons due to not being able to byte case
  Binary instances in python 3. Instead using the getquoted conversion for
  the sake of actual/expected comparisons.

Tests

• Moved --cov options from pytest.ini file into the runner script. This
  fixes debugger break-pointing in some IDEs (e.g. PyCharm).

• Fix various minor testing errors.

Utilities

• Fix ZeroDivisionError in smqtk.utils.bin_utils.report_progress. Also
  added deprecation warning to this function.

v0.12.0

SMQTK v0.12.0 Release Notes

This minor release includes minor fixes and known dependency version updates.

Fixes

Docker

• Fix issue with IQR playground image where matplotlib was attempting to use
  the TkAgg backend by default by adding a matplotlibrc file to specify the
  use of the Agg backend.

Misc

• Update requirements versions for: Flask, Flask-Cors, Pillow

• Update Travis-CI configuration to assume less default values.

Web

• IQR Service

  • Broaden base64 parsing error catch. Specific message of the error changed
    with python 3.7.

v0.11.0

SMQTK v0.11.0 Release Notes

This minor release includes a number of security and stability fixes for
algorithms and the IQR demo web application.

Updates / New Features since v0.10.0

Documentation

• Updated IQR Demo Application documentation RST file and images to
  reflect the current state of SMQTK and that process.

Fixes since v0.10.0

Algorithms

• Classifiers

  • SVM

    • Fixed broken large model saving in Python 2, creating
      parity with Python 3.

• Nearest-Neighbors

  • FAISS

    • Fixed use of strings for compatibility with Python 2.
    • Fixed broken large model saving in Python 2, creating
      parity with Python 3.

  • FLANN

    • Fixed broken large model saving in Python 2, creating
      parity with Python 3.

  • Hash Index

    • Scikit-Learn BallTree

      • Fix save_model and load_model methods for additional
        compatibility with scikit-learn version 0.20.0.

  • LSH

    • Fix issue with update and remove methods when constructed with
      a key-value store structure that use the frozenset type.

    • Fix issue with on-the-fly linear hash index build which was
      previously not correctly setting a set of integers.

Descriptor Generator Plugins

• Fix issue with CaffeDescriptorGenerator where the GPU would not be
  appropriately used on a separate thread/process after initialization occurs on
  the main (or some other) thread.

Docker

• IQR Playground

  • Updated README for better instruction on creating the docker image
    first.

• Caffe image

  • Resolved an issue with upgrading pip for a newer version of matplotlib.

Documentation

• Removed module mocking in sphinx conf.py as it has been shown to be
  brittle to changes in the source code. If we isolate and document a
  use-case where the mocking becomes relevant again we can bring it back.

Misc.

• Update requests and flask package version in requirements.txt and
  devops/docker/smqtk_wrapper_python/requirements.txt files due to
  GitHub security alert.

• Updated package versions for packages in the requirements.docs.txt
  requirements file.

Utilities

• Fixed broken large file writing in Python 2, creating parity
  with Python 3.

• Fixed iqr_app_model_generation.py script for the current state of
  SMQTK functionality.

• Fixed double logging issue in python/smqtk/bin/classifyFiles.py
  tool.

Web

• IQR Search Demo App

  • Fixed input element autocomplete property value being set
    from disabled" to the correct value of "off".

  • Fix CSRF vulnerability in demo web application front-end.

  • Fixed sample configuration files for the current state of
    associated tools.

v0.10.0

SMQTK v0.10.0 Release Notes

This minor release represents the merger of public release request 88ABW-2018-3703. This large updated adds a number of functionality improvements and API changes, docker image improvements and expansions (see the new classifier service), FAISS algorithm wrapper improvements, NearestNeighborIndex update and removal support, a switch to py.test testing framework, generalized classification probability adjustment function, code clean-up, bug fixes and more.

Updates / New Features since v0.9.0

Algorithms

• Classifier

  • Added ClassifierCollection support class. This assists with aggregating multiple SMQTK classifier implementations and applying one or more of those classifiers to input descriptors.

  • Split contents of the __init__.py file into multiple component files. This file was growing too large with the multiple abstract classes and a new utility class.

  • Changed classify abstract method to raise a ValueError instead of a RuntimeError upon being given an empty DescriptorElement.

  • Updated SupervisedClassifier abstract interface to use the template pattern with the train method. Now, implementing classes need to define _train. The train method is not abstract anymore and calls the _train method after the input data consolidation.

  • Update API of classifier to support use of generic extra training parameters.

  • Updated libSVM classifier algorithm to weight classes based on the geometric mean of class counts divided by specific class count to more properly handle weighting even if there is class imbalance.

• Hash Index

  • Made to be its own interface descending from SmqtkAlgorithm instead of NearestNeighborsIndex. While the functionality of a NN-Index and a HashIndex are very similar, all method interfaces are different in terms of the types they accept and return and the HashIndex implementation redefined and documented them to the point where there was no shared functionality.

  • Switched to using the template method for abstract methods.

  • Add update and remove methods to abstract interface. Implemented new interface methods in all subclasses.

  • Added model concurrency protection to implementations.

• Nearest-Neighbors

  • Switched to using the template method for abstract methods.

  • Add update and remove methods to abstract interface. Implemented new interface methods in all subclasses.

  • Fix imports in FAISS wrapper module.

  • Added model concurrency protection to implementations.

  • FAISS

    • Add model persistence via optionally provided DataElement.

    • Fixed use of strings for python 2/3 compatibility.

    • Changed default factory string to "IVF1,Flat".

    • Added initial GPU support to wrapper. Currently only supports one GPU with explicit GPU ID specification.

Representations

• Descriptor Index

  • Added __contains__ method to abstract class to call the has method. This should usually be more efficient than scanning the iteration of the index which is what was happening before. For some implementations, at worst, the runtime for checking for inclusion will be the same (some implementations may have to iterate).

• Descriptor Element

  • Interface

    • Hash value for an element is now only composed of UID value. This is an initial step in deprecating the use of the type-string property on descriptor elements.

    • Equality check between elements now just vector equality.

    • Added base implementation of __getstate__ and __setstate__. Updated implementations to handle this as well as be backward compatible with their previous serialization formats.

    • Added a return of self to vector setting method for easier in-line setting after construction.

  • PostgreSQL

    • Updated to use PsqlConnectionHelper class.

• KeyValueStore

  • Added remove and remove_many abstract methods to the interface. Added implementations to current subclasses.

  • Added __getitem__ implementation.

Docker

• Caffe

  • Updated docker images for CPU or GPU execution.

  • Updated Caffe version built to 1.0.0.

• Added Classifier service docker images for CPU or GPU execution.

  • Inherits from the Caffe docker images.

  • Uses MSRA's ResNet-50 deep learning models.

• IQR Playground

  • Updated configuration files.

  • Now only runs IQR RESTful service and IQR GUI web app (removed nearest-neighbors service).

  • Simplified source image mount point to /images.

  • Updated run_container.*.sh helper scripts.

  • Change deep-learning model used from AlexNet to MSRA's RestNet-50 model.

• Versioning changes to, by default, encode date built instead of arbitrary separate versioning compared to SMQTK's versioning.

• Classifier and IQR docker images now use the local SMQTK checkout on the host system instead of cloning from the internet.

IQR module

• Added serialization load/save methods to the IqrSession class.

Scripts

• generate_image_transform

  • Added stride parameter to image tile cropping feature to allow for more than just discrete, abutting tile cropping.

• runApplication

  • Add ability to get more than individual app description from providing the -l option. Now includes the title portion of each web app's doc-string.

• Added smqtk-make-train-test-sets

  • Create train/test splits from the output of the compute_many_descriptors tool, usually for training and testing a classifier.

Testing

• Remove use of nose-exclude since there are now actual tests in the web sub-module.

• Switch to using pytest as the test running instead of nose. Nose is now in "maintenance mode" and recommends a move to a different testing framework. Pytest is a popular a new powerful testing framework alternative with a healthy ecosystem of extensions.

• Travis CI

  • Removed use of Miniconda installation since it wasn't being utilized in special way.

• Added more tests for Flask-based web services.

Utilities module

• Added mimetype utilities sub-module.

• Added a web utilities module.

  • Added common function for making response Flask JSON instances.

• Added an iter_validation utility submodule.

• Plugin utilities

  • Updated plugin discovery function to be more descriptive as to why a module or class was ignored. This helps debugging and understanding why an implementation for an interface is not available at runtime.

• PostgreSQL

  • Added locking to table creation upsert call.

• Added probability utils submodule and initial probability adjustment function.

Web

• Added new classifier service for managing multiple SMQTK classifier instances via a RESTful interface as well as describe arbitrary new data with the stored classifiers. This service also has the ability to take in saved IQR session states and train a new binary classifier from it.

  • Able to query the service with arbitrary data to be described and classified by one or more managed classifiers.

  • Able to get and set serializations of classifier models for archival.

  • Added example directory of show how to run and to interact with the classifier service via curl.

  • Optionally take a new parameter on the classify endpoint to adjust the precision/recall balance of results.

• IQR Search Dispatcher (GUI web app)

  • Refactored to use RESTful IQR service.

  • Added GUI and JS to load an IQR state from file.

  • Update sample JSON configuration file at python/smqtk/web/search_app/sample_configs/config.IqrSearchApp.json.

  • Added /is_ready endpoint for determining that the service is alive.

• IQR service

  • Added ability to an IQR state serialization into a session.

  • Added sample JSON configuration file to python/smqtk/web/search_app/sample_configs/config.IqrRestService.json.

  • Added /is_ready endpoint for determining that the service is alive.

  • Move class out of the __init__.py file and into its own dedicated file.

  • Make IQR state getter endpoint return a JSON containing the base64 of the state instead of directly returning the serialization bytes.

  • Added endpoints to update, remove from and query against the global nearest-neighbors index.

Fixes since v0.9.0

Algorithms

• Nearest-Neighbor Index

  • LSH

    • Fix bug where it was reporting the size of the nested descriptor index as the size of the neighbor index when the actual index state is defined by the hash-to-uids key-value mapping.

Representations

• DataElement

  • Fixed bug where write_temp() would fail if the content_type() was unknown (i.e. when it returned None).

• Descriptor Index

  • PostgreSQL

    • Fix bug where an instance would create a table even though the create_table parameter was set to false.

• Descriptor Elements

  • PostgreSQL implementation

    • Fix set_vector method to be able to take in sequences that are not explicitly numpy arrays.

• KeyValue

  • PostgreSQL

    • Fix bug where an instance would create a table even though the create_table parameter was set to false.

Scripts

• classifier_model_validation

  • Fixed confidence interval plotting.

  • Fixed confusion matrix plot value range to the [0,1] range which causes the matrix colors to have meaning across plots.

Setup.py

• Add smqtk- to some scripts with camel-case names in order to cause them to be successfully removed upon uninstallation of the SMQTK package.

Tests

• Fixed ambiguous ordering check in libsvm-hik implementation of Re...

v0.9.0

SMQTK v0.9.0 Release Notes

This minor release represents an update to supporting python 3 versions
as well as adding connection pooling support to the PostgreSQL helper
class.

Updates / New Features since v0.8.1

General

• Added support for Python 3.
• Made some optimizations to the Postgres database access.

Travis CI

• Removed use of Miniconda installation since it wasn't being utilized in
  special way.

Fixes since v0.8.1

Tests

• Fixed ambiguous ordering check in libsvm-hik implementation of
  RelevancyIndex algorithm.

v0.8.1

SMQTK v0.8.1 Release Notes

This patch release addresses a bug with PostgreSQL implementations incorrectly
calling a helper class.

Fixes since v0.8.0

Descriptor Index Plugins

• Fix bug in PostgreSQL plugin where the helper class was not being called
  appropriately.

Utilities

• Fix bug in PostgreSQL connection helper where the connection object was
  being called upon when it may not have been initialized.

v0.8.0

SMQTK v0.8.0 Release Notes

This minor release represents the merger of a public release request that added a
Girder-based implementation of the DataElement interface. We also optimized the
use of the PostgreSQL DescriptorIndex implementation to use named cursors for
large queries.

Updates / New Features since v0.7.0

Data Structures

• Revise GirderDataElement to use girder_client python module and added the
  the use of girder authentication token values in lieu of username/password
  for communication authorization.

• Add the optional use of named cursors in PostgreSQL implementation of the
  DescriptorIndex interface. Assists with large selects such that the server
  only sends batches of results at a time instead of the whole result pool.

• Added PostgreSQL implementation of the KeyValueStore interface.

Girder

• Initial SMQTK Girder plugin to support image descriptor processing via
  girder-worker.

• Initial SMQTK Girder plugin implementing a resource and UI for SMQTK nearest
  neighbors and IQR.

Fixes since v0.7.0

Data Structures

• Added locking to PostgreSQL DescriptorElement table creation to fix race
  condition when multiple elements tried to create the same table at the same
  time.

• Fix unconditional import of optional girder_client dependency.

Dependencies

• Pinned Pillow version requirement to 4.0.0 due to a large-image conversion
  issue that appeared in 4.1.x. This issue may have been resolved in newer
  versions of Pillow.

Scripts

• Various fixes to IQR model generation process due to changes made to
  algorithm input parameters (i.e. taking DataElement instances instead of
  filepaths).

• Fixes build_iqr_models.sh to follow symlinks when compiling input image
  file list.

Tests

• Fix missing abstract function override in KeyValueStore test stub.

• Fix test girder_client.HttpError import issue.

v0.7.0

SMQTK v0.7.0 Release Notes

This minor release incorporates various fixes and enhancements to
representation and algorithms interfaces and implementations.

A new docker image has been added to wrap the IQR web interface and headless
services. This image can either be used as a push-button image ingestion and
IQR interface container, or as a fully feature environment to play around with
SMQTK, Caffe deep-learning-based content description and IQR.

A major departure has happened for some representation structures, like
DataElements, as they are no longer considered hashable and now have interfaces
reflecting their mutability. Representation structures, by their nature of
having arbitrary backends, may be modifiable my external agents interacting in
a separate manner with the backend being used. This has also opened up the
ability to provide algorithm implementations with DataElement instances instead
of filepaths for desired byte content and many implementations have
transitioned over to using this pattern. There is nothing fundamentally wrong
with requesting file-path input, however it is restricting as to where
configuration files or data models may come from.

Updates / New Features since v0.6.2

Algorithms

• Descriptor Generators

  • Added KWCNN DescriptorGenerator plugin

Build System

• Added setup.py script in support of installation by pip. Updated
  CMake code to install python components via this scripts.

• Added SMQTK_BUILD_FLANN and SMQTK_BUILD_LIBSVM to CMake for
  optionally building libSVM and Flann (both default ON).

Classifier Interface

• Added default ClassificationElementFactory that uses the in-memory
  back-end.

Compute Functions

• Added minibatch kmeans based descriptor clustering function with CLI
  interface.

Descriptor Elements

• Revised implementation of in-memory representation, doing away with
  global cache.

• Added optimization to Postgres backend for a slightly faster has_vector
  implementation.

Descriptor Generator

• Removed lingering assumption of pyflann module presence in
  colordescriptor.py.

Devops::Ansible

• Added initial Ansible roles for SMQTK and Caffe dependency.

Devops::Docker

• Revised default IQR service configuration file to take into account
  recently added session expiration support. Defaults were used before,
  but now it needs to be specifically enabled as by default expiration is
  not enabled.

• Added IQR / playground docker container setup. Includes:

  • CPU + NVIDIA GPU capable docker file.
  • Optional input image tiling.
  • Optional startup of RESTfule NN and IQR services.

Documentation

• Updated build and installation documentation.

• Added missing utility script documentation hooks.

• Standardized utility script definition of argument parser generation
  function for documentation use.

Girder

• Added initial simple Girder plugin to link to an external IQR webapp
  instance.

Misc.

• Added algo/rep/iqr imports to top level __init__.py to make basic
  functionality available without special imports.

Representation

• Data Elements

  • Added plugin for Girder-hosted data elements

  • Added from_uri member function as well as global function to handle
    instance construction or selection via URI string specification.

  • Postgres data element will now automatically create its configured table
    if it doesn't exist and authentication and sufficient privileges.

• Descriptor Element

  • Postgres descriptor element will now automatically create its configured
    table if it doesn't exist and authentication and sufficient privileges.

• Descriptor Index

  • Postgres descriptor index will now automatically create its configured
    table if it doesn't exist and authentication and sufficient privileges.

Scripts

• Add script to conveniently make Ball-tree hash index model given an
  existing hash2uuids.pickle model file required for the
  LSHNearestNeighborsIndex implementation.

• compute_many_descriptor.py batch size parameter now defaulted to 0
  instead of 256.

• Add script to cluster an index of descriptors via mini-batch kmeans
  (scikit-learn).

• Added script wraping the use of the mini-batch kmeans descriptor clustering
  function.

• Added scripts and notebooks for retrieving MEMEX-specific data from
  ElasticSearch.

• Moved-command line scripts to the smqtk.bin sub-module in order to use
  setuptool support for cross-platform executable generation.

• classifier_kfold_validation utility now only uses
  MemoryClassificationElement instead of letting it be configurable.

• Added script for finding nearest neighbors of a set of UUIDs given a
  nearest neighbors index.

• Added script to add GirderDataElements to a data set

Utilities

• Started a module containing URL-base utility functions, initially adding a
  url-join function similar in capability to os.path.join.

• Added fixed tile cropping to image transform tool.

• Added utility functions to detect mimetypes of files via file-magic or
  tika optional dependencies.

Web

• Updated/Rearchitected IqrSearchApp (now IqrSearchDispatcher) to be able to
  spawn multiple IQR configurations during runtime in addition to any
  configured in the input configuration JSON file. This allows external
  applications to manage configuration storage and generation.

• Added directory for Girder plugins and added an initial one that, given
  a folder with the correct metadata attached, can initialize an IQR
  instance based on that configuration, and then link to IQR web interface
  (uses existing/updated IqrSearch web app).

• Added ability to automatically login via a valid Girder token and parent
  Girder URL for token/user verification. This primarily allows restricted
  external IQR instance creation and automatic login from Girder redirects.

• Mongo session information block at bottom IQR app page now only shows up
  when running server in debug mode.

• Added document showing complete use case with IQR RESTful webservice using
  the IQR docker image with LEEDS Butterfly data. Includes expected results
  users should be able to replicate.

Fixes since v0.6.2

Documentation

• Fixed issues caused by moving scripts out of ./bin/ to
  ./python/smqtk/bin.

Scripts

• Fix logging bug in compute_many_descriptors.py when file path has unicode
  in it.

• Removed final loop progress report from compute_many_descriptors.py as it
  did not report valid statistics.

• Fixed deprecated import of flask-basicauth module.

• Fixed DescriptorFileElement cache-file save location directory when
  configured to use subdirectories. Now no longer creates directories to
  store only a single file. Previous file-element roots are not compatible
  with this change and need to be re-ingested.

• Fixed IQR web app url prefix check

Metrics

• Fixed cosine distance function to return angular distance.

Utilities

• SmqtkObject logger class accessor name changed to not conflict with
  flask.Flask logger instance attribute.

Web

• Fixed Flow upload browse button to not only allow directory selection
  on Chrome.

v0.6.2

SMQTK v0.6.1 Release Notes

This is a patch release with a bug fix for Caffe descriptor generation
introduced in v0.6.0.

Fixes since v0.6.0

Descriptor Generation

• Fixed bug in Caffe wrapper image array loading where loaded arrays were
  not in the correctly associated with data identifiers.