feat: reintroduce and deprecate named sharding, make autosharding default for all other cases #1751

adklempner · 2023-12-06T07:36:15Z

Problem

Named sharding was removed completely and the APIs were changed. If we release a new package, this would break any existing projects that use js-waku. Named sharding should be deprecated, but the constructors/API should be maintained such that any existing projects have time to migrate to static or autosharding.

Solution

Introduce a new SDK function, createNode, which only supports static/autosharding.
For all existing SDK functions, only enforce static or autosharding if shardInfo is set in ProtocolCreateOptions.
Deprecate the pubsubTopics field in ProtocolCreateOptions.
Add tests to ensure named sharding still works.
For all protocol and encoder/decoder constructors, use explicit pubsub topic if set (named sharding behavior). Otherwise, set pubsub topics based on shard info.

Notes

Resolves feat: make autosharding default node behavior #1749
Depends on feat: add support for autosharded pubsub topics #1723

github-actions · 2023-12-22T02:10:49Z

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
Waku core	82.02 KB (+0.08% 🔺)	1.7 s (+0.08% 🔺)	449 ms (+46.52% 🔺)	2.1 s
Waku Simple Light Node	261.54 KB (+0.09% 🔺)	5.3 s (+0.09% 🔺)	643 ms (+7.36% 🔺)	5.9 s
ECIES encryption	31.97 KB (+0.1% 🔺)	640 ms (+0.1% 🔺)	227 ms (-12.39% 🔽)	866 ms
Symmetric encryption	31.96 KB (+0.1% 🔺)	640 ms (+0.1% 🔺)	126 ms (-22.01% 🔽)	765 ms
DNS discovery	123.17 KB (0%)	2.5 s (0%)	478 ms (-2.64% 🔽)	3 s
Privacy preserving protocols	119.34 KB (+0.04% 🔺)	2.4 s (+0.04% 🔺)	504 ms (+34.38% 🔺)	2.9 s
Light protocols	79.56 KB (+0.1% 🔺)	1.6 s (+0.1% 🔺)	291 ms (+32.67% 🔺)	1.9 s
History retrieval protocols	78.45 KB (+0.09% 🔺)	1.6 s (+0.09% 🔺)	171 ms (-6.35% 🔽)	1.8 s
Deterministic Message Hashing	5.92 KB (0%)	119 ms (0%)	76 ms (+14.24% 🔺)	194 ms

danisharora099 · 2024-01-02T11:47:58Z

packages/interfaces/src/filter.ts

@@ -25,7 +25,7 @@ export interface IFilterSubscription {
 export type IFilter = IReceiver &
  IBaseProtocol & {
    createSubscription(
-      pubsubTopicShardInfo?: SingleShardInfo,
+      pubsubTopicShardInfo?: SingleShardInfo | string,


let's maintain using PubsubTopic as the type instead of string

fixed in latest commit

danisharora099 · 2024-01-02T11:48:25Z

packages/interfaces/src/message.ts

+   */
+  pubsubTopic?: PubsubTopic;
  pubsubTopicShardInfo?: SingleShardInfo;


not a fan of this redundancy :(

@danisharora099 This is temporary and a deprecated option. I kept it to allow autosharding support without introducing breaking changes for any existing users.

Another option would be to just keep pubsubTopic and make its type pubsubTopic?: PubsubTopic | SingleShardInfo, but I'm not sure how to communicate that we're deprecating part of a type. I'm open to other suggestions.

I think deprecation period is a good way to go so that we will fully remove the flag in the upcoming update.
In which case I think we should bump minor version and release 0.1.*.

danisharora099 · 2024-01-02T11:49:10Z

packages/message-encryption/src/symmetric.ts

@@ -79,6 +80,8 @@ class Encoder implements IEncoder {
 }

 export interface EncoderOptions extends BaseEncoderOptions {
+  /** Indicates if autosharding is enabled */
+  autosharding?: boolean;


not sure i understand the relevance of having this flag on EncoderOptions

these shouldn't be here; removed from here and ecies.ts in latest commit

weboko · 2024-01-02T23:18:10Z

packages/sdk/src/create.ts

+/**
+ * Create a Waku node configured to use autosharding or static sharding.
+ */
+export async function createNode(


To me it seems that we should avoid introduction of a new create function and change the createLightNode.

Eventually we will change createLightNode, but for now we don't want to introduce breaking changes as it was an existing function. If createLightNode is used with options.shardInfo set, then it behaves almost identical to this new function, with the only difference being that shardInfo is optional. If that's satisfactory, then I'm happy to just remove this new function.

I think that is enough - but we should make sure to depict it in an example and document it.

weboko · 2024-01-02T23:20:28Z

packages/core/src/lib/light_push/index.ts

@@ -49,7 +49,8 @@ class LightPush extends BaseProtocol implements ILightPush {

  constructor(libp2p: Libp2p, options?: ProtocolCreateOptions) {
    super(LightPushCodec, libp2p.components);
-    this.pubsubTopics = this.initializePubsubTopic(options?.shardInfo);
+    this.pubsubTopics =


nit: seems repetitive, let's include into this.initializePubsubTopic

I did it this way because we will be removing options.pubsubTopics in the future. This way, we just need to remove the ?? expression in this spot while keeping the function signature and behavior of initializePubsubTopics the same

weboko · 2024-01-02T23:22:37Z

packages/core/src/lib/filter/index.ts

-    const pubsubTopic = pubsubTopicShardInfo
-      ? singleShardInfoToPubsubTopic(pubsubTopicShardInfo)
-      : DefaultPubsubTopic;
+    const pubsubTopic =


wondering is this.initializePubsubTopic can be used here or adjusted to be used

this.initializePubsubTopic is used to set the pubsub topics for the Filter object, while this const pubsubTopic is used to get/set the pubsub topic for the Subscription object.

updated ternary expression to be less verbose in latest commit

weboko · 2024-01-08T18:36:28Z

packages/interfaces/src/protocols.ts

+   * @deprecated
+   * Waku will stop supporting named sharding. Only static sharding and autosharding will be supported moving forward.
+   */
+  pubsubTopics?: PubsubTopic[];


@adklempner can you, please, create a task with description of what should be removed to completely deprecate parameters/functionality? So that we address it in a skip release or something

I opened #1775

weboko · 2024-01-08T18:46:15Z

packages/core/src/lib/store/index.ts

@@ -79,7 +79,8 @@ class Store extends BaseProtocol implements IStore {

  constructor(libp2p: Libp2p, options?: ProtocolCreateOptions) {
    super(StoreCodec, libp2p.components);
-    this.pubsubTopics = this.initializePubsubTopic(options?.shardInfo);
+    this.pubsubTopics =


It is duplicated - let's make this.initializePubsubTopic(options.pubsubTopics, options.shardInfo) or something.

ok, I think I figured out a good approach: have initializePubsubTopic accept ProtocolCreateOptions and have the deprecated logic in there. When it comes time to remove named sharding, we'll just need to update the function instead of each instance where it's called. Updated in latest commit

adklempner force-pushed the adklempner/default-autosharding branch 2 times, most recently from 7fc587f to 2ab556e Compare December 8, 2023 09:43

adklempner force-pushed the adklempner/default-autosharding branch from 2ab556e to 656de73 Compare December 22, 2023 02:07

adklempner force-pushed the adklempner/default-autosharding branch 2 times, most recently from 54dfdc3 to 3c40f20 Compare December 22, 2023 20:58

adklempner marked this pull request as ready for review December 22, 2023 21:10

adklempner requested a review from a team as a code owner December 22, 2023 21:10

chair28980 requested review from danisharora099 and weboko December 26, 2023 19:42

adklempner force-pushed the adklempner/default-autosharding branch from 3c40f20 to 2d12040 Compare December 27, 2023 18:38

danisharora099 reviewed Jan 2, 2024

View reviewed changes

adklempner force-pushed the adklempner/default-autosharding branch 2 times, most recently from e7fe3ff to eb98b3d Compare January 2, 2024 20:14

weboko reviewed Jan 2, 2024

View reviewed changes

adklempner force-pushed the adklempner/default-autosharding branch from eb98b3d to 9811938 Compare January 3, 2024 03:39

weboko reviewed Jan 8, 2024

View reviewed changes

weboko approved these changes Jan 8, 2024

View reviewed changes

adklempner mentioned this pull request Jan 8, 2024

feat: remove support for named sharding (setting pubsub topic manually) #1775

Open

adklempner force-pushed the adklempner/default-autosharding branch from 9811938 to 960f38f Compare January 8, 2024 19:42

adklempner mentioned this pull request Jan 9, 2024

feat: setup node to use autosharding in rln-js waku-org/examples.waku.org#296

Closed

reintroduce and deprecate named sharding

df09fc6

adklempner force-pushed the adklempner/default-autosharding branch from 960f38f to df09fc6 Compare January 9, 2024 18:00

danisharora099 approved these changes Jan 10, 2024

View reviewed changes

danisharora099 merged commit 69406bf into master Jan 10, 2024
9 of 11 checks passed

danisharora099 deleted the adklempner/default-autosharding branch January 10, 2024 07:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: reintroduce and deprecate named sharding, make autosharding default for all other cases #1751

feat: reintroduce and deprecate named sharding, make autosharding default for all other cases #1751

adklempner commented Dec 6, 2023 •

edited

Loading

github-actions bot commented Dec 22, 2023 •

edited

Loading

danisharora099 Jan 2, 2024

adklempner Jan 2, 2024

danisharora099 Jan 2, 2024

adklempner Jan 2, 2024

weboko Jan 2, 2024

danisharora099 Jan 2, 2024

adklempner Jan 2, 2024

weboko Jan 2, 2024

adklempner Jan 3, 2024

weboko Jan 8, 2024

weboko Jan 2, 2024

adklempner Jan 3, 2024

weboko Jan 2, 2024

adklempner Jan 3, 2024

adklempner Jan 3, 2024

weboko Jan 8, 2024

adklempner Jan 8, 2024

weboko Jan 8, 2024

adklempner Jan 8, 2024

feat: reintroduce and deprecate named sharding, make autosharding default for all other cases #1751

feat: reintroduce and deprecate named sharding, make autosharding default for all other cases #1751

Conversation

adklempner commented Dec 6, 2023 • edited Loading

Problem

Solution

Notes

github-actions bot commented Dec 22, 2023 • edited Loading

size-limit report 📦

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adklempner commented Dec 6, 2023 •

edited

Loading

github-actions bot commented Dec 22, 2023 •

edited

Loading