BREAKING CHANGE(server): support "parent & child" EdgeLabel type #2662

VGalaxies · 2024-09-08T08:22:20Z

HugeGraph supports the parent-child edge feature, meaning that an Edgelabel can have a subordinate type. Using the bank transfer graph as an example, transfers may include person-to-person transfers (person-to-person), person-to-company transfers (person-to-company), and company-to-company transfers (company-to-company). These three different types of transfers share a common operation, transfer.

In actual business scenarios, it is often necessary to retrieve all transfer edges with a single query. Currently, competitors can only manually split the transfer edge types, perform multiple queries, and then aggregate the results. With HugeGraph's parent-child edge feature, it is possible to query the corresponding person-to-person transfers, person-to-company transfers, and other sub-edge types, and also conveniently and efficiently retrieve all transfer-related edges at once.

PS: The parent-child edge feature for the cassandra and scylladb backends has been temporarily disabled through the store feature.

Code formatting will be done separately in the next PR.

Related to:

Design doc: https://hugegraph.feishu.cn/wiki/ZgLBwGAusiQFACkuIADcjSJtnof

Client(Toolchain) need adapter it ASAP (BREAKING CHANGE), see apache/incubator-hugegraph-toolchain#624

Test: hugegraph/hugegraph-test#17

codecov · 2024-09-08T11:58:31Z

Codecov Report

Attention: Patch coverage is 59.67153% with 221 lines in your changes missing coverage. Please review.

Project coverage is 47.78%. Comparing base (a529c02) to head (eea7356).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
...in/java/org/apache/hugegraph/schema/EdgeLabel.java	52.43%	35 Missing and 4 partials ⚠️
.../hugegraph/backend/store/AbstractBackendStore.java	0.00%	30 Missing ⚠️
...che/hugegraph/schema/builder/EdgeLabelBuilder.java	71.76%	14 Missing and 10 partials ⚠️
...ugegraph/backend/serializer/TableSerializerV2.java	62.00%	18 Missing and 1 partial ⚠️
.../org/apache/hugegraph/api/schema/EdgeLabelAPI.java	7.14%	11 Missing and 2 partials ⚠️
.../apache/hugegraph/backend/tx/GraphTransaction.java	80.59%	5 Missing and 8 partials ⚠️
.../apache/hugegraph/io/GraphSONSchemaSerializer.java	56.66%	9 Missing and 4 partials ⚠️
...he/hugegraph/backend/cache/CachedBackendStore.java	0.00%	12 Missing ⚠️
...hugegraph/backend/serializer/BinarySerializer.java	38.88%	9 Missing and 2 partials ⚠️
...gegraph/backend/store/memory/InMemoryDBTables.java	65.62%	8 Missing and 3 partials ⚠️
... and 10 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2662      +/-   ##
============================================
+ Coverage     47.60%   47.78%   +0.18%     
- Complexity      820      821       +1     
============================================
  Files           718      719       +1     
  Lines         58469    58914     +445     
  Branches       7496     7595      +99     
============================================
+ Hits          27835    28154     +319     
- Misses        27854    27946      +92     
- Partials       2780     2814      +34

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

VGalaxies · 2024-09-12T04:48:09Z

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/schema/EdgeLabel.java

+        return this.links.stream().anyMatch(pair -> {
+            Id sourceLabel = pair.getLeft();
+            Id targetLabel = pair.getRight();
+            if (dir.equals(Directions.IN)) {
+                return targetLabel.equals(label);
+            } else if (dir.equals(Directions.OUT)) {
+                return sourceLabel.equals(label);
+            } else if (dir.equals(Directions.BOTH)) {
+                return targetLabel.equals(label) || sourceLabel.equals(label);
+            }
+            return false;
+        });


mark, see #2408

VGalaxies · 2024-09-12T04:49:53Z

...rver/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransaction.java

+        EdgeLabel label = edge.schemaLabel();
+        if (label.hasFather()) {
+            for (Id id : graph().edgeLabel(label.fatherId()).indexLabels()) {
+                this.updateIndex(id, edge, removed);
+            }
+        }


missing in internal version

VGalaxies · 2024-09-12T04:50:22Z

hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/schema/EdgeLabelAPI.java

@@ -183,10 +185,16 @@ private static class JsonEdgeLabel implements Checkable {
        public long id;
        @JsonProperty("name")
        public String name;
+        @JsonProperty("edge_label_type")


edgelabel_type or edge_label_type

keep edgelabel_type

VGalaxies · 2024-09-12T04:50:40Z

...ver/hugegraph-core/src/main/java/org/apache/hugegraph/traversal/optimize/HugeVertexStep.java

too many diffs...

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

This reverts commit e44aaf3.

VGalaxies · 2024-09-21T07:19:40Z

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

+        if (query instanceof ConditionQuery && !query.paging()) {
+            // TODO: support: paging + parent label
+            boolean supportIn = this.storeFeatures().supportsQueryWithInCondition();
+            // consider multi labels + properties, see org.apache.hugegraph.core.EdgeCoreTest.testQueryInEdgesOfVertexByLabels
+            Stream<ConditionQuery> flattenedQueries = ConditionQueryFlatten.flatten((ConditionQuery) query, supportIn).stream();
+
+            Stream<Iterator<HugeEdge>> edgeIterators = flattenedQueries.map(cq -> {
+                Id label = cq.condition(HugeKeys.LABEL);
+                if (this.storeFeatures().supportsFatherAndSubEdgeLabel() &&
+                    label != null &&
+                    graph().edgeLabel(label).isFather() &&
+                    cq.condition(HugeKeys.SUB_LABEL) == null &&
+                    cq.condition(HugeKeys.OWNER_VERTEX) != null &&
+                    cq.condition(HugeKeys.DIRECTION) != null &&
+                    matchEdgeSortKeys(cq, false, this.graph())) {
+                    // g.V("V.id").outE("parentLabel").has("sortKey","value")
+                    return parentElQueryWithSortKeys(
+                            graph().edgeLabel(label), graph().edgeLabels(), cq);
+                } else {
+                    return queryEdgesFromBackendInternal(cq);
+                }
+            });
+
+            return edgeIterators.reduce(ExtendableIterator::concat).orElse(Collections.emptyIterator());
+        }
+
+        return queryEdgesFromBackendInternal(query);
+    }
+
+    private Iterator<HugeEdge> queryEdgesFromBackendInternal(Query query) {
+        assert query.resultType().isEdge();
+


TBD.

because of #1737

VGalaxies · 2024-09-21T07:20:01Z

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

+        if (cq.condition(HugeKeys.LABEL) != null && cq.resultType().isEdge()) {
+            if (cq.conditions().size() == 1) {
+                // g.E().hasLabel(xxx)
+                return true;
+            }
+            if (cq.optimized() == OptimizedType.INDEX) {
+                // g.E().hasLabel(xxx).has(yyy)
+                // consider OptimizedType.INDEX_FILTER occurred in org.apache.hugegraph.core.EdgeCoreTest.testQueryCount
+                try {
+                    this.indexTx.asyncRemoveIndexLeft(cq, elem);
+                } catch (Throwable e) {
+                    LOG.warn("Failed to remove left index for query '{}', " +
+                             "element '{}'", cq, elem, e);
+                }
+                return true;
+            }
+        }
+


VGalaxies · 2024-09-21T07:28:08Z

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

+                if (this.storeFeatures().supportsFatherAndSubEdgeLabel() && byLabel && query.resultType().isEdge()) {
+                    // for memory backend
+                    EdgeLabel edgeLabel = graph().edgeLabel(label);
+                    if (edgeLabel.hasFather()) {
+                        query.resetConditions();
+                        query.eq(HugeKeys.LABEL, edgeLabel.fatherId());
+                        query.eq(HugeKeys.SUB_LABEL, edgeLabel.id());
+                    }
+                }


javeme · 2024-09-22T02:37:43Z

@VGalaxies Nice feature, can you add design documentation?

…dgeExistenceGet

…st#testEdgeExistenceGet" This reverts commit 81fcf27.

VGalaxies · 2024-10-09T15:12:15Z

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

+                // fix org.apache.hugegraph.api.traverser.EdgeExistenceAPITest#testEdgeExistenceGet
+                // add sub label if only the sub label is missing
+                ConditionQuery finalQuery = query;
+                if (this.storeFeatures().supportsFatherAndSubEdgeLabel() &&
+                    query.condition(HugeKeys.SUB_LABEL) == null &&
+                    Arrays.stream(EdgeId.KEYS)
+                          .filter(key -> !Objects.equals(key, HugeKeys.SUB_LABEL))
+                          .allMatch(key -> finalQuery.condition(key) != null)) {
+                    EdgeLabel el = this.graph().edgeLabel(label);
+                    if (!el.isFather()) {
+                        query.eq(HugeKeys.SUB_LABEL, el.id());
+                    }
+                }


VGalaxies · 2024-10-09T15:13:52Z

Some specialized adaptations have been made for parent-child edges in existing tests, as seen in the aforementioned comments. Currently, paging + parent label queries are not supported.

VGalaxies added 2 commits September 8, 2024 16:17

setup

0e202d4

add tests and examples

042ed6a

VGalaxies added 4 commits September 8, 2024 21:36

fix FakeObjects.java

22fb285

fix TaskCoreTest.java

7d8f2f7

more fix

8a9d676

g.V("V.id").outE("parentLabel").has("sortKey","value")

9afdcb0

VGalaxies commented Sep 12, 2024

View reviewed changes

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java Outdated Show resolved Hide resolved

VGalaxies commented Sep 12, 2024

View reviewed changes

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java Outdated Show resolved Hide resolved

VGalaxies commented Sep 12, 2024

View reviewed changes

VGalaxies added 2 commits September 19, 2024 20:03

fix

499cca7

fix

ea96dce

VGalaxies commented Sep 19, 2024

View reviewed changes

...ph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java Outdated Show resolved Hide resolved

VGalaxies added 12 commits September 19, 2024 21:39

fix hstore

58e66ce

fix page

b31f5db

add TODO comments

b48e3cb

fix memory backend

06ec600

try fix mysql

a07ddf7

try fix cassandra

e44aaf3

Revert "try fix cassandra"

a67be44

This reverts commit e44aaf3.

try fix cassandra

7371937

fixup

bd640f7

fixup

b0a60f2

try fix scylladb

a88680c

improve CJK comments

45fab36

VGalaxies requested a review from imbajin September 21, 2024 07:17

VGalaxies marked this pull request as ready for review September 21, 2024 07:18

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. feature New feature labels Sep 21, 2024

VGalaxies commented Sep 21, 2024

View reviewed changes

imbajin added this to the 1.5.0 milestone Sep 21, 2024

fix for client

19485e5

imbajin changed the title ~~feat(server): support father and sub edge label~~ BREAKING CHANGE(server): support "parent & child" EdgeLabel type Sep 24, 2024

Thespica mentioned this pull request Sep 25, 2024

[Feature] support "parent & child" EdgeLabel type apache/incubator-hugegraph-toolchain#623

Closed

3 tasks

minor improve for source + target label builder

87e76ff

imbajin mentioned this pull request Oct 2, 2024

feat(client): support "parent & child" EdgeLabel type apache/incubator-hugegraph-toolchain#624

Merged

11 tasks

VGalaxies added 6 commits October 3, 2024 18:59

write sub label id for range cond

a876ec0

fixup

9242757

try fix org.apache.hugegraph.api.traverser.EdgeExistenceAPITest#testE…

81fcf27

…dgeExistenceGet

fixup

263b6a6

Revert "try fix org.apache.hugegraph.api.traverser.EdgeExistenceAPITe…

14c99e2

…st#testEdgeExistenceGet" This reverts commit 81fcf27.

mark deprecated for source & target label builder

eea7356

VGalaxies commented Oct 9, 2024

View reviewed changes

coderzc approved these changes Oct 9, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Oct 9, 2024

imbajin approved these changes Oct 9, 2024

View reviewed changes

imbajin merged commit f6f3708 into master Oct 9, 2024
18 checks passed

imbajin deleted the father-sub-edge branch October 9, 2024 16:15

VGalaxies mentioned this pull request Oct 16, 2024

fix(server): serialize source and target label for non-father edge label #2682

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BREAKING CHANGE(server): support "parent & child" EdgeLabel type #2662

BREAKING CHANGE(server): support "parent & child" EdgeLabel type #2662

VGalaxies commented Sep 8, 2024 •

edited

Loading

codecov bot commented Sep 8, 2024 •

edited

Loading

VGalaxies Sep 12, 2024

VGalaxies Sep 12, 2024

VGalaxies Sep 12, 2024

VGalaxies Oct 9, 2024

VGalaxies Sep 12, 2024

VGalaxies Sep 21, 2024 •

edited

Loading

VGalaxies Sep 21, 2024

VGalaxies Sep 21, 2024

javeme commented Sep 22, 2024

VGalaxies Oct 9, 2024

VGalaxies commented Oct 9, 2024

BREAKING CHANGE(server): support "parent & child" EdgeLabel type #2662

BREAKING CHANGE(server): support "parent & child" EdgeLabel type #2662

Conversation

VGalaxies commented Sep 8, 2024 • edited Loading

codecov bot commented Sep 8, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VGalaxies Sep 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme commented Sep 22, 2024

Choose a reason for hiding this comment

VGalaxies commented Oct 9, 2024

VGalaxies commented Sep 8, 2024 •

edited

Loading

codecov bot commented Sep 8, 2024 •

edited

Loading

VGalaxies Sep 21, 2024 •

edited

Loading