Refactor create community #2305

leblowl · 2024-02-19T17:19:34Z

This PR migrates the Community model to the backend so that the backend is now the source of truth for Community data. In order to accomplish this, for existing communities (existing Quiet install), we move (migrate) the data from the frontend to the backend and then make sure all operations that previously affected the frontend Community state are done on the backend. For new communities (new Quiet install), some of the logic for creating/joining has been moved to the backend and the event-flow is simplified. Whenever the Community model is updated on the backend, the backend now emits a COMMUNITY_UPDATED event so that the frontend can synchronize its Community state.

Solution for: #2329

Pull Request Checklist

I have linked this PR to a related GitHub issue.
I have added a description of the change (and Github issue number, if any) to the root CHANGELOG.md.

(Optional) Mobile checklist

Please ensure you completed the following checks if you did any changes to the mobile package:

I have run e2e tests for mobile
I have updated base screenshots for visual regression tests

leblowl · 2024-03-08T04:12:19Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

+    } else {
+      this.logger('Nothing to migrate')
+    }
+  }


Since we are moving the Community model to the backend, new communities created after this code is released will have the Community model in the backend, however existing communities that upgrade Quiet need to have their community data migrated - that's what this function is for.

leblowl · 2024-03-08T04:14:28Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

-    await this.localDbService.put(LocalDBKeys.PSK, pskBase64)
-    this.logger('Generated Libp2p PSK')
-    this.serverIoProvider.io.emit(SocketActionTypes.LIBP2P_PSK_STORED, { psk: pskBase64 })
-  }


We store the psk in the Community object, so there is no more need for the PSK key in LevelDB. We also return the entire Community object from createCommunity so there is no more need for the LIBP2P_PSK_STORED event.

leblowl · 2024-03-08T04:17:15Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

+    } catch (e) {
+      this.logger.error('Failed to register owner certificate')
+      return
+    }


We register the owner certificate in createCommunity since that's how it was done before in sagas. We previously called REGISTER_OWNER_CERTIFICATE and then when that completed, we called CREATED_COMMUNITY. Now we do it all together. However, we may want to decouple the owner cert from the community in future work.

leblowl · 2024-03-08T04:21:26Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

-  public async launchCommunity(payload: InitCommunityPayload) {
-    this.logger('Launching community: peers:', payload.peers)
-    this.communityState = ServiceState.LAUNCHING
+  public async joinCommunity(payload: InitCommunityPayload): Promise<Community | undefined> {


joinCommunity is a new function so that we can focus on validation and creating a Community object, while launchCommunity is still used but it's only for launching an already-validated community.

leblowl · 2024-03-08T04:24:01Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

-      emitError(this.serverIoProvider.io, payload)
-    })
-  }
-


Not needed anymore since we call to the registration service directly and await on its return

leblowl · 2024-03-08T04:25:37Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

+            ...community,
+            ownerOrbitDbIdentity: meta.ownerOrbitDbIdentity,
+          })
+        }


Update Community object on backend when setting CommunityMetadata. This is the first time that ownerOrbitDbIdentity is available. In the future, we can do this all when creating the community and remove this SET_COMMUNITY_METADATA event.

leblowl · 2024-03-08T04:26:22Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

-      async (args: RegisterOwnerCertificatePayload) => {
-        await this.registrationService.registerOwnerCertificate(args)
-      }
-    )


No longer required - we do this in createCommunity

leblowl · 2024-03-08T04:27:33Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

@@ -609,8 +724,8 @@ export class ConnectionsManagerService extends EventEmitter implements OnModuleI
    this.storageService.on(StorageEvents.MESSAGE_MEDIA_UPDATED, (payload: FileMetadata) => {
      this.serverIoProvider.io.emit(SocketActionTypes.MESSAGE_MEDIA_UPDATED, payload)
    })
-    this.storageService.on(StorageEvents.UPDATE_PEERS_LIST, (payload: StorePeerListPayload) => {
-      this.serverIoProvider.io.emit(SocketActionTypes.PEER_LIST, payload)


Instead of sending an event for a single piece of data (e.g. peerList), we send a general event (COMMUNITY_UPDATED) anytime the backend updates the Community model so that the frontend can replicate the state.

leblowl · 2024-03-08T04:28:15Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

+        await this.localDbService.setCommunity(updatedCommunity)
+
+        this.serverIoProvider.io.emit(SocketActionTypes.COMMUNITY_UPDATED, updatedCommunity)
+      }


When another peer receives the CommunityMetadata from OrbitDB, they update their Community object in LevelDB and then notify the frontend via COMMUNITY_UPDATED.

leblowl · 2024-03-08T04:30:31Z

packages/backend/src/nest/local-db/local-db.types.ts

+  COMMUNITIES = 'communities',
+  // ID of current community
+  CURRENT_COMMUNITY_ID = 'currentCommunityId',
+  // Record of peer details
  PEERS = 'peers',


Backend LevelDB state mirrors the way things have been organized in Redux

leblowl · 2024-03-08T04:31:13Z

packages/backend/src/nest/registration/registration.service.ts

-        message: ErrorMessages.REGISTRATION_FAILED,
-        community: payload.communityId,
-      })
+      throw new Error('Failed to register owner certificate')


Removing events since we can call this directly and await it

leblowl · 2024-03-08T04:36:17Z

packages/state-manager/src/sagas/app/loadMigrationData/loadMigrationData.saga.ts

+  }
+
+  yield* apply(socket, socket.emit, applyEmitParams(SocketActionTypes.LOAD_MIGRATION_DATA, data))
+}


Saga for sending frontend data to the backend in order to migrate existing communities when upgrading Quiet.

packages/state-manager/src/sagas/communities/communities.slice.ts

leblowl · 2024-03-08T04:44:43Z

packages/state-manager/src/sagas/communities/createCommunity/createCommunity.saga.ts

+  // community creation from CSR/certificate creation and create the community
+  // first and then add the owner's CSR and issue their certificate.
+  yield* put(identityActions.saveUserCsr())
+}


I renamed savedOwnerCertificate.saga.ts to createCommunity.saga.ts and modified it a bit. We generate the owner's certificate when creating the community now and just call updateCommunityData with the returned Community object.

leblowl · 2024-03-08T04:46:26Z

packages/state-manager/src/sagas/communities/createNetwork/createNetwork.saga.ts

-  if (psk) {
-    console.log('create network saga: saving PSK')
-    yield* put(communitiesActions.savePSK(psk))
-  }


No need for this anymore, otherwise just reorganizing things a bit.

leblowl · 2024-03-08T04:47:49Z

...ages/state-manager/src/sagas/communities/saveCommunityMetadata/saveCommunityMetadata.saga.ts

-      ownerCertificate: action.payload.ownerCertificate,
-    })
-  )
-}


Now when CommunityMetadata is received from OrbitDB, we update the Community on the backend and then emit a COMMUNITY_UPDATED event. This isn't needed anymore.

leblowl · 2024-03-08T04:48:23Z

packages/state-manager/src/sagas/communities/updateCommunity/updateCommunity.saga.ts

-  }
-
-  yield* put(communitiesActions.updateCommunityData(payload))
-}


Now when CommunityMetadata is received from OrbitDB, we update the Community on the backend and then emit a COMMUNITY_UPDATED event. This isn't needed anymore.

leblowl · 2024-03-08T04:51:06Z

.../state-manager/src/sagas/communities/updateCommunityMetadata/updateCommunityMetadata.saga.ts

-
-  if (meta) {
-    yield* put(communitiesActions.saveCommunityMetadata(meta))
-  }


This is also replaced by a COMMUNITY_UPDATED event. Going forward, we can remove this saga entirely.

leblowl · 2024-03-08T04:51:49Z

packages/state-manager/src/sagas/identity/savedOwnerCertificate/savedOwnerCertificate.saga.ts

-  }
-
-  yield* apply(socket, socket.emit, applyEmitParams(SocketActionTypes.CREATE_COMMUNITY, payload))
-}


This was renamed to createCommunity.saga.ts

leblowl · 2024-03-08T04:53:22Z

packages/state-manager/src/sagas/socket/startConnection/startConnection.saga.ts

-      // all the backend services up and running and then save the
-      // CSR, register the owner's certificate and set community
-      // metadata.
-      emit(identityActions.saveUserCsr())


Since we are using an acknowledgement for the CREATE_COMMUNITY event, this was moved into createCommunitySaga

leblowl · 2024-03-08T04:55:06Z

packages/types/src/community.ts

@@ -11,8 +11,6 @@ export interface Community {
  rootCa?: string
  peerList?: string[]
  onionAddress?: string
-  privateKey?: string
-  port?: number


Apparently unused

leblowl · 2024-03-08T04:55:20Z

packages/types/src/community.ts

  peerId: PeerId
  hiddenService: HiddenService
-  certs?: Certificates


Also apparently unused

EmiM · 2024-03-12T17:11:41Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

+
+    // Only require migration data for existing communities. We can tell because
+    // they are using the deprecated COMMUNITY key in LevelDB. This is related
+    // to a specific migration. Perhaps we want a more general purpose migration


That's definitely a good idea especially that leveldb will be storing more data from now on. We can get inspiration from Django migrations files or any other migration mechanism.

Yea I was thinking about something like that

EmiM · 2024-03-12T17:24:28Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

    await this.tor.destroyHiddenService(hiddenService.onionAddress.split('.')[0])
+
+    // TODO: Do we want to create the PeerId here? It doesn't necessarily have


I suppose it was added here because it's needed to create full peer's p2p address in order for peer to join the network. So it's kind of related. But we don't have to do that here.

EmiM · 2024-03-13T12:06:23Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

+        const communityName = getCertFieldValue(rootCaCert, CertFieldsTypes.commonName)
+
+        if (!communityName) {
+          this.logger.error(`Could not retrieve ${CertFieldsTypes.commonName} from CommunityMetadata.rootCa`)


If rootCa doesn't have commonName there may be something wrong with rootCa. Are we sure that we still want to update community metadata in this case?

This code runs after community metadata has been updated. I think I mostly copied it from sagas. I think we can definitely improve things. My feeling is we should probably validate the rootCA somewhere when it's first created and then trust it from then on.

EmiM · 2024-03-13T12:15:04Z

packages/backend/src/nest/connections-manager/connections-manager.service.ts

-      async (payload: CommunityMetadata, callback: (response?: CommunityMetadata) => void) => {
-        const meta = await this.storageService?.updateCommunityMetadata(payload)
+      async (payload: CommunityMetadata, callback: (response: CommunityMetadata | undefined) => void) => {
+        const meta = await this.storageService.updateCommunityMetadata(payload)


I see that you added throwing exception in updateCommunityMetadata. Maybe we should catch it here?

Where is the exception? I think I'm missing it.

packages/state-manager/src/sagas/communities/communities.slice.ts

EmiM · 2024-03-13T12:51:57Z

packages/desktop/src/rtl-tests/generalChannel.create.test.tsx

-
-jest.setTimeout(20_000)
-
-describe('General channel', () => {


Why is it deleted?

I think this is fully tested in community.create and community.join tests

…ate-community

… into refactor-create-community

leblowl force-pushed the refactor-create-community branch 6 times, most recently from 156fe39 to 5d86954 Compare February 23, 2024 23:11

leblowl force-pushed the refactor-create-community branch 14 times, most recently from a8db4ee to 5bd37ad Compare March 8, 2024 01:04

leblowl commented Mar 8, 2024

View reviewed changes

packages/state-manager/src/sagas/communities/communities.slice.ts Outdated Show resolved Hide resolved

leblowl commented Mar 8, 2024

View reviewed changes

packages/types/src/community.ts

peerId: PeerId

hiddenService: HiddenService

certs?: Certificates

Copy link

Collaborator Author

leblowl Mar 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also apparently unused

leblowl force-pushed the refactor-create-community branch from 454011d to bc0510b Compare March 8, 2024 04:57

leblowl marked this pull request as ready for review March 8, 2024 04:57

leblowl requested review from EmiM and siepra March 8, 2024 05:19

leblowl force-pushed the develop branch from 76752cc to 5a5e0c5 Compare March 8, 2024 19:39

EmiM approved these changes Mar 13, 2024

View reviewed changes

Lucas Leblow added 4 commits March 27, 2024 11:55

refactor: Move Community data model to backend

7e94297

Refactor owner cert registration

1e78feb

Refactor misc community updates

315dd50

Fixups

2227744

leblowl force-pushed the refactor-create-community branch from bc0510b to 2227744 Compare March 27, 2024 18:55

Lucas Leblow added 6 commits April 3, 2024 11:39

Revise migration strategy

93c262b

Fixup

d7116b5

Fixup

7930ef4

Merge branch 'develop' of github.com:TryQuiet/quiet into refactor-cre…

7a16539

…ate-community

Merge branch 'refactor-create-community' of github.com:TryQuiet/quiet…

6bebbaa

… into refactor-create-community

Fix test

a11c5db

leblowl merged commit 29c1c1b into develop Apr 4, 2024
26 of 31 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor create community #2305

Refactor create community #2305

leblowl commented Feb 19, 2024 •

edited

Loading

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024 •

edited

Loading

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

leblowl Mar 8, 2024

EmiM Mar 12, 2024

leblowl Mar 20, 2024

EmiM Mar 12, 2024

EmiM Mar 13, 2024

leblowl Mar 20, 2024

EmiM Mar 13, 2024

leblowl Mar 20, 2024

EmiM Mar 13, 2024

leblowl Mar 20, 2024

		await this.tor.destroyHiddenService(hiddenService.onionAddress.split('.')[0])

		// TODO: Do we want to create the PeerId here? It doesn't necessarily have


		jest.setTimeout(20_000)

		describe('General channel', () => {

Refactor create community #2305

Refactor create community #2305

Conversation

leblowl commented Feb 19, 2024 • edited Loading

Pull Request Checklist

(Optional) Mobile checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leblowl Mar 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leblowl commented Feb 19, 2024 •

edited

Loading

leblowl Mar 8, 2024 •

edited

Loading