[client-v2] Support Dynamic and JSON Data Types #2130

chernser · 2025-02-04T22:18:02Z

Summary

This PR adds support for Variant, Dynamic and JSON for reading and writing. Here is details what is supported
All this is try only for RowBinary format. Native will need another implementation because it has another structure.

DataType	POJO	Binary Reader	JDBC
Variant	Read/Write	Yes	Yes
Dynamic	Read/Write	Yes	Yes
JSON	No	Yes	Yes/No

JSON Support

JSON in RowBinary is encoded as map of path-value (for example "a.b" = 1). Reader returns a flat map for this.
It is convenient when path is known and hardcoded. But inconvenient for parsing. It should be possible to get Map-of-Map structure. However this is most heavy-weight solution.
Writing JSON from POJO is not done in this PR. It can be done in two ways:

in-house solution - tedious work with reflection
plugin GSON/Jackson libraries to make Map-of-Map structure so can be easily transformed into a flat map.

PS: There are some test fixes.

Closes #1925
Closes #1833
Closes #1926

Checklist

Delete items not relevant to your PR:

Closes [client-v2] Dynamic Data Type Support #1925, [client-v2] JSON Support #1833,[client-v2] Variant Type Support #1926
Unit and integration tests covering the common scenarios were added
A human-readable description of the changes was provided to include in CHANGELOG
For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

chernser · 2025-02-12T18:47:14Z

clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseDataType.java

-    Enum16(String.class, true, true, false, 2, 0, 0, 0, 0, false),
-    FixedString(String.class, true, true, false, 0, 0, 0, 0, 0, false, "BINARY"),
-    Int8(Byte.class, false, true, true, 1, 3, 0, 0, 0, false, "BYTE", "INT1", "INT1 SIGNED", "TINYINT",
+    Bool(Boolean.class, false, false, true, 1, 1, 0, 0, 0, false,0x2D, "BOOLEAN"),


added bin tag according to the https://clickhouse.com/docs/en/sql-reference/data-types/data-types-binary-encoding

chernser · 2025-02-12T18:47:51Z

clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseDataType.java

+    Int128(BigInteger.class, false, true, true, 16, 39, 0, 0, 0, false, 0x0B),
+    UInt128(BigInteger.class, false, true, false, 16, 39, 0, 0, 0, false, 0x05),
+    Int256(BigInteger.class, false, true, true, 32, 77, 0, 0, 0, false, 0x0C),
+    UInt256(BigInteger.class, false, true, false, 32, 78, 0, 0, 0, false, 0x06),
    Decimal(BigDecimal.class, true, false, true, 0, 76, 0, 0, 76, false, "DEC", "FIXED", "NUMERIC"),


Virtual type - no bin tag

chernser · 2025-02-12T18:50:31Z

clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseDataType.java

        return map;
    }

    private static Set<Class<?>> setOf(Class<?>... args) {
        return Collections.unmodifiableSet(new HashSet<>(Arrays.stream(args).collect(Collectors.toList())));
    }

+    public static final byte INTERVAL_BIN_TAG = 0x22;


following tags are not actual types or require special processing.

chernser · 2025-02-12T18:51:38Z

clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseDataType.java

@@ -258,6 +333,21 @@ private static Set<Class<?>> setOf(Class<?>... args) {

        allAliases = Collections.unmodifiableSet(set);
        name2type = Collections.unmodifiableMap(map);
+


Creating lookup maps for types and interval kinds

chernser · 2025-02-12T18:52:51Z

clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseEnum.java

@@ -51,7 +51,7 @@ protected ClickHouseEnum(Collection<String> params) {
        }
    }

-    protected ClickHouseEnum(String[] names, int[] values) {
+    public ClickHouseEnum(String[] names, int[] values) {


no need to protect the constructor of such simple class.

chernser · 2025-02-12T18:55:43Z

client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java

@@ -641,6 +658,38 @@ public synchronized <T> List<T> asList() {
        }
    }

+    public static class EnumValue extends Number {


Have to create the class to overcome problem:

when dynamic column then each enum value may have different set of columns

when enum column is requested as String we need string constant but normally it is stored in column definition but it will be dynamic.

chernser · 2025-02-12T18:57:08Z

client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java

@@ -956,4 +1005,81 @@ public byte[] allocate(int size) {
            return new byte[size];
        }
    }
+
+    private static final Set<Byte> DECIMAL_TAGS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(


decimal has special handling because of arguments. This set just simplifies if statements.

chernser · 2025-02-12T18:57:37Z

client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java

+            ClickHouseDataType.Decimal256.getBinTag()
+    )));
+
+    private ClickHouseColumn readDynamicData() throws IOException {


reads metadata for dynamic value

chernser · 2025-02-12T18:59:23Z

client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java

-    private static void serializeArrayData(OutputStream stream, Object value, ClickHouseColumn column) throws IOException {
+    private static final Map<Class<?>, ClickHouseColumn> PREDEFINED_TYPE_COLUMNS = getPredefinedTypeColumnsMap();
+
+    private static Map<Class<?>, ClickHouseColumn> getPredefinedTypeColumnsMap() {


We use Column object to properly serialize/deserialize values, so I've created predefined list to avoid creation all the time.
But it can be more lightweight - will redesign in the future.

chernser · 2025-02-12T19:07:45Z

client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java

-                    if (val == null) {
-                        writeNull(stream);
-                        continue;
+    public static ClickHouseColumn valueToColumnForDynamicType(Object value) {


we need to crate column definition for each value so serializer can use it like a normal column serialization.

sonarqubecloud · 2025-02-13T06:55:09Z

Quality Gate failed

Failed conditions
41.7% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

mzitnik · 2025-02-13T08:08:12Z

clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseEnum.java

@@ -51,7 +51,7 @@ protected ClickHouseEnum(Collection<String> params) {
        }
    }

-    protected ClickHouseEnum(String[] names, int[] values) {
+    public ClickHouseEnum(String[] names, int[] values) {


will it be more efficient to use a map here i see a lot scan operations that we have

Current approach is more memory efficient. We can introduce map and measure performance.

mzitnik · 2025-02-13T08:10:15Z

client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatSerializer.java

@@ -207,6 +207,8 @@ public static boolean writeValuePreamble(OutputStream out, boolean defaultsSuppo
                    return false;//And we're done
                } else if (dataType == ClickHouseDataType.Array) {//If the column is an array
                    SerializerUtils.writeNonNull(out);//Then we send nonNull
+                } else if (dataType == ClickHouseDataType.Dynamic) {
+                    // do nothing


if we can add an explanation why we are not doing here anything

mzitnik · 2025-02-13T08:13:55Z

client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java

+                case Enum8: {
+                    byte enum8Val = (byte) readUnsignedByte();
+                    String name = actualColumn.getEnumConstants().nameNullable(enum8Val);
+                    return (T) new EnumValue(name == null ? "<unknown>" : name, enum8Val);


"" maybe can define as const

"" will be a blank space and it will be hard to detect error. <unknown> will attract attention.

mshustov

Writing JSON from POJO is not done in this PR. It can be done in two ways:

+1 to use Gson or jackson

JSON + JDBC

What is our plan for JSON type usage via JDBC?

mshustov · 2025-02-13T08:57:06Z

.github/workflows/build.yml

@@ -268,7 +268,7 @@ jobs:
    needs: compile
    strategy:
      matrix:
-        clickhouse: ["23.8", "24.3", "24.6", "latest"]
+        clickhouse: ["23.8", "24.3", "24.8", "latest"]


nit: 23.8 can be excluded as not supported https://github.com/ClickHouse/ClickHouse/blob/master/SECURITY.md

I've thought about it. Will do.

mshustov · 2025-02-13T09:04:09Z

client-v2/src/test/java/com/clickhouse/client/insert/InsertTests.java

@@ -638,8 +640,57 @@ public static Object[][] testAppCompressionDataProvider() {
        };
    }

+    @Test(groups = { "integration" }, enabled = true)
+    public void testPOJOWithDynamicType() throws Exception {
+        if (isVersionMatch("(,24.8]")) {


should it be a proper comparison with a semver version ( greater or equal 24.8)? as done below https://github.com/ClickHouse/clickhouse-java/pull/2130/files#diff-89c11757b9243105d9931d9b667e953b7beabe6ca45ab2f333e905bb7003f8a9R944

This expression does the same - it is from old tests and I just reuse it. We will unify such things later when we have one client and can have a utility method.

mshustov · 2025-02-13T09:10:06Z

client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java

+        private Object field;
+    }
+
+    private void testDynamicWith(String withWhat, Object[] values, String[] expectedStrValues) throws Exception {


do we need separate json type tests?

I think no - JSON uses the same code as Dynamic for values. Paths are just string additions to each value.
The rest of logic is tested in InsertTests/QueryTests thru POJO.

chernser added 15 commits February 3, 2025 14:32

added basic dynamic support for rowbinary reader

f0645b1

implemented reading in rowbinary format for json and dynamic

96102c9

Merge branch 'main' into v2_support_dynamic_and_json

dc82be7

fixed tests for ch versions

4f8de9d

implemented dynamic basic types

2ae4d27

Merge branch 'main' into v2_support_dynamic_and_json

f18ce46

implemented interval and partially decimal

af28fca

almost all types are working

d2b6d59

fixed array handling when Dynamic

9c3a6ed

fixed maps and decimals

beae9f4

Fixed dynamic primitives. Dates left

35f0b31

fixed enums for different releases

c6335d7

Merge branch 'main' into v2_support_dynamic_and_json

629c0be

fixed arrays

dbee722

enabled JDBC tests for variant and dynamic

455b367

chernser commented Feb 12, 2025

View reviewed changes

fixed dates, added test for POJO read

de27d86

chernser marked this pull request as ready for review February 13, 2025 01:11

chernser added 3 commits February 12, 2025 18:00

Merge branch 'main' into v2_support_dynamic_and_json

48a0b09

fixed some tests

e969608

fixed more tests

7741b2f

chernser added 3 commits February 12, 2025 21:22

Merge branch 'main' into v2_support_dynamic_and_json

170674e

fixed proper settings for cloud

b93815a

one more test fix

1b81ce5

mshustov requested review from mzitnik and Paultagoras February 13, 2025 08:28

mzitnik reviewed Feb 13, 2025

View reviewed changes

mshustov reviewed Feb 13, 2025

View reviewed changes

chernser requested a review from mzitnik February 13, 2025 18:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[client-v2] Support Dynamic and JSON Data Types #2130

[client-v2] Support Dynamic and JSON Data Types #2130

chernser commented Feb 4, 2025 •

edited

Loading

chernser Feb 12, 2025

chernser Feb 12, 2025

chernser Feb 12, 2025

chernser Feb 12, 2025

chernser Feb 12, 2025

chernser Feb 12, 2025

chernser Feb 12, 2025

chernser Feb 12, 2025

chernser Feb 12, 2025

chernser Feb 12, 2025

sonarqubecloud bot commented Feb 13, 2025

mzitnik Feb 13, 2025

chernser Feb 13, 2025

mzitnik Feb 13, 2025

mzitnik Feb 13, 2025

chernser Feb 13, 2025

mshustov left a comment

mshustov Feb 13, 2025

chernser Feb 13, 2025

mshustov Feb 13, 2025

chernser Feb 13, 2025

mshustov Feb 13, 2025

chernser Feb 13, 2025

		@@ -258,6 +333,21 @@ private static Set<Class<?>> setOf(Class<?>... args) {

		allAliases = Collections.unmodifiableSet(set);
		name2type = Collections.unmodifiableMap(map);

[client-v2] Support Dynamic and JSON Data Types #2130

Are you sure you want to change the base?

[client-v2] Support Dynamic and JSON Data Types #2130

Conversation

chernser commented Feb 4, 2025 • edited Loading

Summary

JSON Support

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 13, 2025

Quality Gate failed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mshustov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chernser commented Feb 4, 2025 •

edited

Loading