Migrate SHM #1798

yellowhatter · 2025-02-26T14:37:07Z

migrate from shared-memory crate to own SHM api wrapper
suppress SHM persistency on Windows and BSD
optimize some SHM object sizes
get rid of separate SegmentLock struct - use advisory locking within Segment
improve SHM error handling
query real SHM segment size upon creation and opening (the sizes will differ depending on system page size)
expose real segment size in all our segment abstractions (Segment and ArrayInShm)
utilize real segment sizes in PosixShmProviderBackend to expose more occupied memory to user

- some good refactoring

github-actions · 2025-02-26T14:37:20Z

PR missing one of the required labels: {'bug', 'dependencies', 'enhancement', 'internal', 'breaking-change', 'documentation', 'new feature'}

…s (and always had) some issues to fix

- disable some shm tests only for macos

- fix clippy - take new elem_count() behavior into account (elem_count may be bigger then the one requested upon segment creation)

…der.rs

yellowhatter · 2025-02-27T18:32:03Z

lints: base64ct issue
macos: flacky test

…pfs, so need other bsd-specific mechanism)

- add comments

YuanYuYuan

Hi @yellowhatter, besides the comments, I have one more question.

We added advisory-lock as a new dependency, but it has been inactive for five years. Should we be concerned about this?

YuanYuYuan · 2025-02-28T09:27:58Z

commons/zenoh-shm/tests/shm.rs

@@ -0,0 +1,99 @@
+//
+// Copyright (c) 2023 ZettaScale Technology


2023 -> 2025

commons/zenoh-shm/Cargo.toml

commons/zenoh-shm/tests/shm.rs

YuanYuYuan · 2025-02-28T15:49:39Z

commons/zenoh-shm/tests/shm.rs

+}
+
+#[test]
+fn create_and_open_amd_reopen() {


commons/zenoh-shm/tests/shm.rs

commons/zenoh-shm/src/posix_shm/cleanup.rs

commons/zenoh-shm/src/shm/unix.rs

commons/zenoh-shm/src/shm/mod.rs

commons/zenoh-shm/src/shm/unix.rs

commons/zenoh-shm/src/shm/windows.rs

Dev/shm persistency on mac

codecov · 2025-03-05T13:16:37Z

Codecov Report

Attention: Patch coverage is 89.39394% with 21 lines in your changes missing coverage. Please review.

Project coverage is 70.40%. Comparing base (cac67d9) to head (e5a856b).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
commons/zenoh-shm/src/shm/unix.rs	89.16%	13 Missing ⚠️
commons/zenoh-shm/src/posix_shm/segment.rs	72.22%	5 Missing ⚠️
...mplementations/posix/posix_shm_provider_backend.rs	85.71%	1 Missing ⚠️
commons/zenoh-shm/src/posix_shm/array.rs	92.30%	1 Missing ⚠️
commons/zenoh-shm/src/shm/mod.rs	93.75%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1798      +/-   ##
==========================================
- Coverage   70.41%   70.40%   -0.02%     
==========================================
  Files         358      359       +1     
  Lines       64586    64617      +31     
==========================================
+ Hits        45476    45491      +15     
- Misses      19110    19126      +16

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- fix segment collision from different runs

commons/zenoh-shm/build.rs

commons/zenoh-shm/src/shm/mod.rs

commons/zenoh-shm/src/shm/unix.rs

yellowhatter · 2025-03-06T08:13:31Z

Hi @yellowhatter, besides the comments, I have one more question.

We added advisory-lock as a new dependency, but it has been inactive for five years. Should we be concerned about this?

We are already using advisory-lock crate in our workspace (unixpipe link)
This crate just wraps a very simple system AP (flock())

I think there is no problem to use it while it satisfies our needs

yellowhatter added 4 commits February 24, 2025 17:45

WIP on SHM migration

be11cad

posix implementation

77ba35d

tests for shm api abstraction

e68a389

- move advisory locking functionality to shm segment class

59b1cf1

- some good refactoring

yellowhatter self-assigned this Feb 26, 2025

yellowhatter added enhancement Existing things could work better dependencies Pull requests that update a dependency file release Part of the next release internal Changes not included in the changelog labels Feb 26, 2025

yellowhatter added 20 commits February 26, 2025 17:45

fix lints

2bfd409

fix workspace clippy

b1e441b

handle flock errors more precisely

d57c37e

use advisory-lock-based persistency suppression only on linux, mac ha…

02711ec

…s (and always had) some issues to fix

Update unix.rs

0e3b383

Update unix.rs

8df8035

Update unix.rs

394bde5

Merge commit '1964079188113dd72bf1ee9df90543c75e476338'

2ed1a6b

remove unsupported index for shm segment

86a78b2

Update shm.rs

bdd7a15

rough windows support for shm

8b4123d

Update unix.rs

e159a7d

- format fix

00d7641

- disable some shm tests only for macos

add missing win_sys

8514f07

fix errors on win

92c66c7

Update unix.rs and windows.rs

0e7572b

- get rid of some unwrap

0b7eb95

- fix clippy - take new elem_count() behavior into account (elem_count may be bigger then the one requested upon segment creation)

Update shm.rs

3fd05e3

Update shm.rs

2a453c9

Update array.rs and shm.rs

fe6899a

yellowhatter added 3 commits February 27, 2025 18:10

Update posix_shm_provider_backend.rs, windows.rs, and posix_shm_provi…

0e05278

…der.rs

fix allocator test

64753a2

Merge commit '2311334ebb82378e341d350a08d0667169c7f211'

0d0de98

yellowhatter requested a review from YuanYuYuan February 27, 2025 18:31

yellowhatter marked this pull request as ready for review February 27, 2025 18:31

yellowhatter added 5 commits February 28, 2025 15:04

WIP on mac persistency problem (flock() doesn't always work on mac tm…

18daf0c

…pfs, so need other bsd-specific mechanism)

Make BSD and non-BSD implementation for shared memory cleanup

586ac85

Add config aliases to properly detect BSD systems

d765bd5

- fix CI

a663b5a

- add comments

Update unix.rs

721319d

YuanYuYuan requested changes Feb 28, 2025

View reviewed changes

yellowhatter and others added 8 commits March 5, 2025 11:44

Apply advisory locking on non-tmpfs for BSD

73089b2

fix issues on mac

9fa8221

Update unix.rs

278687a

Update unix.rs

355170c

Update windows.rs

fabfa0f

Update unix.rs and windows.rs

68febcf

Merge pull request #25 from ZettaScaleLabs/dev/shm_persistency_on_mac

1a560a9

Dev/shm persistency on mac

Merge commit 'ac19c9213f438921335e90ee3bed95cd099d4ebc'

df7add7

yellowhatter added 3 commits March 5, 2025 16:30

- review fixes

071bf6d

- fix segment collision from different runs

Update shm.rs

71fbcd4

Update shm.rs

833b107

yellowhatter requested a review from YuanYuYuan March 5, 2025 14:45

YuanYuYuan requested changes Mar 6, 2025

View reviewed changes

commons/zenoh-shm/build.rs Outdated Show resolved Hide resolved

commons/zenoh-shm/src/shm/mod.rs Outdated Show resolved Hide resolved

commons/zenoh-shm/src/shm/unix.rs Outdated Show resolved Hide resolved

yellowhatter added 2 commits March 6, 2025 10:02

Merge commit 'cac67d97805379ac45a093fc72466b21f2d9639c'

18425ab

Review fixes

e5a856b

YuanYuYuan merged commit e76243b into eclipse-zenoh:main Mar 6, 2025
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate SHM #1798

Migrate SHM #1798

yellowhatter commented Feb 26, 2025 •

edited

Loading

github-actions bot commented Feb 26, 2025

yellowhatter commented Feb 27, 2025

YuanYuYuan left a comment

YuanYuYuan Feb 28, 2025

yellowhatter Mar 5, 2025

YuanYuYuan Feb 28, 2025

yellowhatter Mar 6, 2025

codecov bot commented Mar 5, 2025 •

edited

Loading

yellowhatter commented Mar 6, 2025

		@@ -0,0 +1,99 @@
		//
		// Copyright (c) 2023 ZettaScale Technology

Migrate SHM #1798

Migrate SHM #1798

Conversation

yellowhatter commented Feb 26, 2025 • edited Loading

github-actions bot commented Feb 26, 2025

yellowhatter commented Feb 27, 2025

YuanYuYuan left a comment

Choose a reason for hiding this comment

YuanYuYuan Feb 28, 2025

Choose a reason for hiding this comment

yellowhatter Mar 5, 2025

Choose a reason for hiding this comment

YuanYuYuan Feb 28, 2025

Choose a reason for hiding this comment

yellowhatter Mar 6, 2025

Choose a reason for hiding this comment

codecov bot commented Mar 5, 2025 • edited Loading

Codecov Report

yellowhatter commented Mar 6, 2025

yellowhatter commented Feb 26, 2025 •

edited

Loading

codecov bot commented Mar 5, 2025 •

edited

Loading