feat(server): support MemoryManagement for graph query framework #2649

Pengzna · 2024-08-25T08:20:14Z

As title.

Users can use following new options to configure memory management:

# The memory management switch in HugeGraph. Options: off-heap, on-heap, disable.
memory.mode=off-heap
# The maximum memory capacity that can be managed for all queries in HugeGraph.
memory.max_capacity=1073741824
# The maximum memory capacity that can be managed for a query in HugeGraph.
memory.one_query_max_capacity=104857600
# The alignment used for round memory size.
memory.alignment=8

Detailed docs:

codecov · 2024-08-25T09:34:54Z

Codecov Report

Attention: Patch coverage is 0% with 1046 lines in your changes missing coverage. Please review.

Project coverage is 1.19%. Comparing base (21855c6) to head (b77346b).

Files with missing lines	Patch %	Lines
...e/hugegraph/memory/consumer/factory/IdFactory.java	0.00%	102 Missing ⚠️
...ache/hugegraph/memory/pool/AbstractMemoryPool.java	0.00%	99 Missing ⚠️
...gegraph/memory/consumer/impl/id/EdgeIdOffHeap.java	0.00%	94 Missing ⚠️
...hugegraph/memory/pool/impl/OperatorMemoryPool.java	0.00%	93 Missing ⚠️
...ava/org/apache/hugegraph/memory/MemoryManager.java	0.00%	88 Missing ⚠️
...he/hugegraph/memory/pool/impl/QueryMemoryPool.java	0.00%	70 Missing ⚠️
...gegraph/memory/consumer/impl/id/UuidIdOffHeap.java	0.00%	47 Missing ⚠️
...he/hugegraph/memory/pool/impl/MemoryPoolStats.java	0.00%	47 Missing ⚠️
...che/hugegraph/memory/pool/impl/TaskMemoryPool.java	0.00%	38 Missing ⚠️
...gegraph/memory/consumer/impl/id/LongIdOffHeap.java	0.00%	37 Missing ⚠️
... and 19 more

❗ There is a different number of reports uploaded between BASE (21855c6) and HEAD (b77346b). Click for more details.

HEAD has 6 uploads less than BASE

Flag BASE (21855c6) HEAD (b77346b)

7 1

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #2649       +/-   ##
============================================
- Coverage     47.70%   1.19%   -46.52%     
+ Complexity      821       8      -813     
============================================
  Files           720     723        +3     
  Lines         58989   57517     -1472     
  Branches       7607    7214      -393     
============================================
- Hits          28143     685    -27458     
- Misses        28031   56781    +28750     
+ Partials       2815      51     -2764

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

...ver/hugegraph-core/src/main/java/org/apache/hugegraph/memory/allocator/IMemoryAllocator.java

...rver/hugegraph-core/src/main/java/org/apache/hugegraph/memory/pool/impl/MemoryPoolStats.java

...rver/hugegraph-core/src/main/java/org/apache/hugegraph/memory/pool/impl/QueryMemoryPool.java

...er/hugegraph-core/src/main/java/org/apache/hugegraph/memory/allocator/AbstractAllocator.java

hugegraph-server/hugegraph-dist/src/assembly/static/conf/graphs/hugegraph.properties

javeme

How to re-point a Java variable to an off-heap object? we can set its address to an off-heap memory:

But! There is a risk in using off-heap objects, that is, off-heap objects refer to on-heap objects, such as vertex label, which may be released by schema cache.
We have two ways to solve this risk:

One way is to pin the referenced in-heap object when an off-heap object is created, such as saving it in a hashset collection;
The other way is to refresh the in-heap object when taking an off-heap object (such as re-get it from the schema cache by label id).

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/memory/MemoryManager.java

...server/hugegraph-core/src/main/java/org/apache/hugegraph/memory/pool/AbstractMemoryPool.java

…anagement

This reverts commit ea9a459.

This reverts commit 0e70e44.

Pengzna · 2024-10-29T07:14:17Z

@javeme @imbajin @zyxxoo @MingzhenHan PTAL~

imbajin

Thanks & merge it first~

VGalaxies · 2024-11-04T11:38:27Z

hugegraph-commons/hugegraph-dist/scripts/dependency/known-dependencies.txt

why change this file?

why change this file?

maybe influenced by Fury dependencies🤔?

why change this file?

maybe influenced by Fury dependencies🤔?

Yes, and I ran wrong regenerate_known_dependencies.sh which is in hugegraph-dist/scripts/dependency, resulting in a duplicate known-dependencies.txt under hugegraph-dist/scripts/dependency.

Actually, only regenerate_known_dependencies.sh in /install-dist/scripts/dependency is enough and correct. regenerate_known_dependencies.sh in hugegraph-dist/scripts/dependency seems useless and may cause misunderstanding for programmers.

@imbajin maybe could entirely remove hugegraph-commons/hugegraph-dist?

This reverts commit 7c86e84.

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/id/EdgeId.java

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/structure/HugeProperty.java

chaokunyang · 2024-11-08T05:18:45Z

...ver/hugegraph-core/src/main/java/org/apache/hugegraph/memory/util/FurySerializationUtil.java

+
+public class FurySerializationUtil {
+
+    public static final Fury FURY = Fury.builder().withLanguage(Language.JAVA)


Seems Fury is used to serialize/deserialize object to/from offheap, serialized payload size may be not important here. Maybe we can disable number compress here by FuryBuilder#withNumerCompressed(false). If we have lots of number for serialization, disable this option will have much better performance

This Fury is not thread safe, please use ThreadSafeFury instead

Thanks for your suggestions! All these will be improved in next pr

Pengzna added 2 commits August 25, 2024 16:04

feat: framework for memoryManagement

7a652be

wip: memory pool and manager framework

ccc918b

javeme reviewed Sep 22, 2024

View reviewed changes

Pengzna added 6 commits October 8, 2024 21:30

wip: memory allocation

b266358

wip: memory reclaim

ff10c3a

rename class

26b0cfe

fix review

f6aeace

remove useless allocator

72e5bf3

netty allocator

48f4817

VGalaxies reviewed Oct 10, 2024

View reviewed changes

hugegraph-server/hugegraph-dist/src/assembly/static/conf/graphs/hugegraph.properties Outdated Show resolved Hide resolved

revert config.properties

d906d04

javeme reviewed Oct 13, 2024

View reviewed changes

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/memory/MemoryManager.java Outdated Show resolved Hide resolved

...server/hugegraph-core/src/main/java/org/apache/hugegraph/memory/pool/AbstractMemoryPool.java Outdated Show resolved Hide resolved

Pengzna added 17 commits October 22, 2024 22:52

fix and improvement for allocation and deallocation

b308be0

Merge remote-tracking branch 'refs/remotes/base/master' into memory/m…

09367a1

…anagement

move monitor

ea9a459

Revert "move monitor"

f552fd2

This reverts commit ea9a459.

improve memory arbitration

8a2c65c

suspend query when arbitration & kill query when OOM

c37f869

fix review

5904909

fury test

0e70e44

offHeap magic

5d71541

Revert "fury test"

f73f0ab

This reverts commit 0e70e44.

offHeap magic util

871015e

complete adoption for all id

8344443

complete property adoption

d9cf408

release ByteBuf off heap memory block

aaeacb5

complete allocate memory test and fix bug

ef0d629

fix some bugs: arbitration & suspend

54d1fd8

complete OOM UT and fix bugs

bfe75c0

Pengzna added 3 commits October 29, 2024 13:41

fix deallocate netty memory block bug & add complexId test

f34e233

complete all ut

879390b

fix all bugs

a96e9ee

Pengzna marked this pull request as ready for review October 29, 2024 07:13

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. feature New feature labels Oct 29, 2024

Pengzna added 9 commits October 29, 2024 15:22

dependency

7c86e84

add comments

5af2cb9

add private constructor for singleton

5e47bb0

add memory management config

b77346b

improve robustness

d00a8df

remove duplicate

fecc909

improve condition usage

31f1feb

improve log

d4035bd

fix memory conservation bug

d87388b

imbajin previously approved these changes Nov 3, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Nov 3, 2024

VGalaxies reviewed Nov 4, 2024

View reviewed changes

Pengzna added 2 commits November 4, 2024 19:48

Revert "dependency"

d25396d

This reverts commit 7c86e84.

revert duplicate known-dependencies.txt under huge-common

b334c61

Pengzna dismissed imbajin’s stale review via b334c61 November 4, 2024 11:50

Pengzna requested review from imbajin and VGalaxies November 4, 2024 11:53

VGalaxies reviewed Nov 4, 2024

View reviewed changes

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/id/EdgeId.java Show resolved Hide resolved

hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/structure/HugeProperty.java Show resolved Hide resolved

VGalaxies approved these changes Nov 5, 2024

View reviewed changes

imbajin approved these changes Nov 5, 2024

View reviewed changes

imbajin changed the title ~~feat(server): Introduce MemoryManagement for HugeGraph Query Framework~~ feat(server): support MemoryManagement for graph query framework Nov 5, 2024

imbajin merged commit 35e5a8c into apache:master Nov 5, 2024
16 of 17 checks passed

chaokunyang reviewed Nov 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(server): support MemoryManagement for graph query framework #2649

feat(server): support MemoryManagement for graph query framework #2649

Pengzna commented Aug 25, 2024 •

edited by imbajin

Loading

codecov bot commented Aug 25, 2024 •

edited

Loading

javeme left a comment •

edited by imbajin

Loading

Pengzna commented Oct 29, 2024

imbajin left a comment

VGalaxies Nov 4, 2024

imbajin Nov 4, 2024

Pengzna Nov 4, 2024

Pengzna Nov 4, 2024

VGalaxies Nov 4, 2024

chaokunyang Nov 8, 2024 •

edited

Loading

chaokunyang Nov 8, 2024

Pengzna Nov 8, 2024


		public class FurySerializationUtil {

		public static final Fury FURY = Fury.builder().withLanguage(Language.JAVA)

feat(server): support MemoryManagement for graph query framework #2649

feat(server): support MemoryManagement for graph query framework #2649

Conversation

Pengzna commented Aug 25, 2024 • edited by imbajin Loading

codecov bot commented Aug 25, 2024 • edited Loading

Codecov Report

javeme left a comment • edited by imbajin Loading

Choose a reason for hiding this comment

Pengzna commented Oct 29, 2024

imbajin left a comment

Choose a reason for hiding this comment

VGalaxies Nov 4, 2024

Choose a reason for hiding this comment

imbajin Nov 4, 2024

Choose a reason for hiding this comment

Pengzna Nov 4, 2024

Choose a reason for hiding this comment

Pengzna Nov 4, 2024

Choose a reason for hiding this comment

VGalaxies Nov 4, 2024

Choose a reason for hiding this comment

chaokunyang Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

chaokunyang Nov 8, 2024

Choose a reason for hiding this comment

Pengzna Nov 8, 2024

Choose a reason for hiding this comment

Pengzna commented Aug 25, 2024 •

edited by imbajin

Loading

codecov bot commented Aug 25, 2024 •

edited

Loading

javeme left a comment •

edited by imbajin

Loading

chaokunyang Nov 8, 2024 •

edited

Loading