Apache DataSketches 4.2.0

• added generic KLL quantile sketch

Apache Release 4.1.0

• This is a minor release that primarily fixed a problem where the Java KLL sketches could not read KLL images produced by C++.
• In addition, a number of code improvements to fix issues found by SpotBugs and CodeQL.
• Documentation improvements both internal as well as Javadocs.

Apache Release 4.0.0

Major new features and enhancements

- Quantile Sketches
    - The major APIs for all the quantile sketches now derive from interfaces common to all the quantile sketches. This makes it much easier for the user to move from one quantile sketch to another with only very minor API changes.
    - All the quantile sketches now have a "SortedView", which is iterable and makes analysis of the quantile distribution even easier.
- HLL Sketches
    - Major speed performance improvements for HLL union/merge operations.
    - Major improvements to the HLL Javadocs.
- Theta Sketches
    - The Theta sketch has been enhanced with an optional compress operation that makes the serialized theta sketch smaller.
- TestNG has been updated to version 7.5.1 (works with Java 8), which includes the Zip Slip Vulnerability fix.

Apache Release 3.3.0

This release removes support for allocating off-heap memory in Java9 and Java10

Only LTS Java versions are now fully supported and since the Java 9 and Java 10 no
longer receive security patches, this release removes off-heap memory support
for these versions.

This applies to Datasketches-Memory when allocating Memory-Mapped files,
Direct or direct ByteBuffers in these runtime versions.
Support for allocating Memory in both heap and non-heap remains unaffected
for Java 8, Java 11, Java 12 and Java 13.

Apache Release 3.2.0

• This release extends the quantiles KLL sketch to include both doubles and float types as well as full updatable operation off-heap in contiguous memory.

• This release also adds two new methods to Theta Union based on the discussion on Druid issue # 12261:

  • getCurrentBytes() This behaves similarly to the Theta Sketch method by the same name.
  • getMaxUnionBytes() This behaves similarly to the static Sketches method by the same name, but is non-static and returns the maximum stored bytes for this union given its nominal entries configuration.

Apache Release 3.1.0

This is primarily a maintenance release.
Highlights:

• Fixed "Mikhail's Issue #368". This was a specific Theta set-operation corner case bug.
  • In addition, Developed a standard model for all Theta, Tuple set operation corner cases. Implemented across all corner cases for Java (and C++).
• Improvements in clarity of Javadocs, primarily in Theta, Tuple sketches.
• Fixed some parameter leakage cases related to LGTM warnings.
• Improved variable naming to enhance code clarity.
• Fixed warnings discovered by latest SpotBugs.
• Significant improvements and cleanup of ArrayOfDoubles Tuple Sketch code.

Apache Release 3.0.0

This release of DataSketches-Java 3.0.0 depends on DataSketches-Memory 2.0.0 and is compatible with JDK8 through JDK13. If you wish to compile this release from source please refer to the README of this release.

There are some minor API changes as part of this release

• The MurmurHash3v2 class has been relocated to be part of the DataSketches-Memory component.
• Some of the API changes in the new Memory component are reflected in this component where they are exposed as part of the public API here. For example, the API of the MemoryRequestServer of Memory has changed slightly.
• Because this is a major release a number of deprecated methods have been removed.

Apache Release 2.0.0

Introducing the Relative Error Quantiles Sketch

See website documentation.

Improved input null and empty handling policies for Theta / Tuple sketches

Exceptions are now being thrown where there could be accidental destruction of data. Where a null is harmless, it is ignored. However, receiving an empty sketch is considered valid, and the result follows the set operation logic.

Operation	Null Policy	Empty Policy
Union	Ignored	Ignored
Intersection	Exception	Empty result
AnotB, first Argument, A	Exception	Empty result
AnotB, subsequent Argument, B	Ignored	Ignored

Improved Set Operation handling

In Theta Sketch and Tuple Sketch (Generic), the APIs for Union, AnotB and Intersection now allow for both stateless and stateful operation.

• see ExamplesTest.java for full example code

• Pseudocode model for stateless operation.
  Two set operation arguments and the result are accomplished in one call:
  result = setOp(Sketch A, Sketch B);
  where setOp = union, intersect, or aNotB

• Pseudocode model for stateful operation. Any number of arguments can be presented sequentially:
  void setOp(Sketch 1);
void setOp(Sketch 2);
...
void setOp(Sketch N);
result = getResult();
  Where setOp = union or intersect
  Because the AnotB operation is asymmetric, the method names are a little different:
  void setA(Sketch 1); //for the first argument
  void notB(Sketch 2); //for all subsequent arguments
  ...
void notB(Sketch N);
result = getResult();

Tuple Sketch (Generic) set operations now allow combining with Theta Sketch

• Pseudocode model for stateless operation:
  result = setOp(TupleSketch A, ThetaSketch B);

• Pseudocode model for stateful operation:
  void setOp(TupleSketch 1);
void setOp(ThetaSketch 1);
void setOp(TupleSketch 2);
...
void setOp(ThetaSketch N);
result = getResult();

Package changes

• The tuple package was too large and confusing. The ArrayOfDoubles classes, generic base classes and example generic classes were all mingled together. These have now been split up into the following hierarchy:
  • tuple/
    • adouble/
    • aninteger/
    • arrayofdoubles
    • strings

Lots of internal refactoring, code cleanup, and documentation improvements

Release Process

• Replaced Travis-CI with GitHub Actions
• Added Maven toolchains.xml in anticipation of multiple JDK releases
• Cleaned up Release Zip files to exclude unneeded hidden files.

Apache Release 1.3.0-incubating

• This release contains a critical fix for a recently discovered bug identified by Druid Issue #9736.

Apache Release 1.2.0-incubating

• Significant Merge speed improvements to HLL
• Fixed issue #292

This Release was directly derived from 1.2.0-incubating-RC2. Please see notes there.

Shortly before this was released a bug (issue #296) was discovered. PR #297 fixes this issue but it wasn't in time for this release. This is now fixed in master and will be part of the next release.

This bug only affects unit test code on Windows platforms. It has no effect on run-time production code.