Realm syncing incomplete data

[njmulsqb]

How frequently does the bug occur?

• select --

Description

Hi, I am using realm on react native expo.

The issue is that when I start the app, there are some subscriptions which loads the data, but the data loaded is incomplete, and when I insert some data into the same collection, realm then syncs all the data including the newly inserted one.

Why isn't it syncing all the data at first place?

Here are some code snippets to help with troubleshooting

## App.tsx (Realm initialization code)

 <AppProvider id={process.env.REALM_APP_ID}>
      <UserProvider fallback={realmLogIn}>
        <RealmProvider
          sync={{
            flexible: true,
            onError: console.error,
            initialSubscriptions: {
              update(subs, realm) {
                subs.add(realm.objects("realmTest"));
                //subs.add(realm.objects("users"));
              },
            },
          }}
        >

## Getting all the data using subscriptions

  useEffect(() => {

    // Subscription for fetching the connections
    realm.subscriptions.update((mutableSubs, realm) => {
      const connectionsQuery = realm
        .objects("connection")
        .filtered("user = $0 OR friend = $0", new Realm.BSON.ObjectID(userId));
      mutableSubs.add(connectionsQuery, { name: "connections" });
    });

    // Subscription for fetching the users
    realm.subscriptions.update((mutableSubs, realm) => {
      const connections = realm.objects("connection");
      const userIds = Array.from(
        new Set(
          connections.reduce((ids, connection) => {
            ids.push(connection.user, connection.friend);
            return ids;
          }, [])
        )
      );

      const usersQuery = realm.objects("users").filtered("_id IN $0", userIds);
      mutableSubs.add(usersQuery, { name: "users" });
    });
    // Subscription to fetch the groups
    realm.subscriptions.update((mutableSubs, realm) => {
      const connections = realm.objects("connection")
      const groupIds = Array.from(
        new Set(
          connections.reduce((ids, connection) => {
            if (connection.groupExp) {
              connection.groupExp.forEach((group) => {
                ids.push(group.groupId);
              });
            }
            return ids;
          }, [])
        )
      );
      const groupsQuery = realm
        .objects("group")
        .filtered("_id IN $0", groupIds);
      mutableSubs.add(groupsQuery, { name: "groups" });
    });

    // Subscription to fetch the expenses
    realm.subscriptions.update((mutableSubs, realm) => {
      const connections = realm.objects("connection");

      const filteredExpenseIds = connections.reduce(
        (filteredIds, connection) => {
          if (connection.nonGroupExp) {
            connection.nonGroupExp.forEach((expense) => {
              filteredIds.push(new Realm.BSON.ObjectID(expense.expenseId));
            });
          }
          return filteredIds;
        },
        []
      );

      const filteredGroupIds = connections.reduce((filteredIds, connection) => {
        if (connection.groupExp) {
          connection.groupExp.forEach((group) => {
            filteredIds.push(new Realm.BSON.ObjectID(group.groupId));
          });
        }
        return filteredIds;
      }, []);

      const expensesQuery = realm
        .objects("expense")
        .filtered(
          "_id IN $0 OR groupId IN $1",
          filteredExpenseIds,
          filteredGroupIds
        );
      mutableSubs.add(expensesQuery, { name: "expenses" });
    });

    // Removing the initial subscription
    realm.subscriptions.update((mutableSubs) => {
      mutableSubs.removeByObjectType("realmTest");
    });

    // realm.subscriptions.update((mutableSubs) => {
    //   mutableSubs.removeAll();
    // });

  });

## Fetching the data in groups.tsx
const groups = realm.objects("group");

## and here's the code that inserts the object
  realm.write(() => {
        let realmObject = realm.create("group", {
          _id: new Realm.BSON.ObjectId(),
          name: values.name,
          members: [
            {
              hasJoined: true,
              isManager: true,
              userId: realmUser?._id,
            },
          ],
          isPinned: false,
          isArchived: false,
          defaultCurrency: currency?.currency_short_form,
          description: "",
          settlementCycle: { repeatAfterDays: 7, startingFrom: date },
          createdAt: new Date(),
          dueDate: dueDate,
          inviteId: inviteId,
          isSettledUp: false,
        });
        setNewGroup(realmObject);
        console.log(newGroup)
      });

Stacktrace & log output

No response

Can you reproduce the bug?

Always

Reproduction Steps

Just do something that wipes the current session i.e. log out, remove realm files from explorer, remove all subscriptions and reinitialize, and you'll be having incomplete data.

Add some data to realm using realm.write and whole collection will be synced then.

Version

Realm 11.9.0, @realm/react 0.5.1

What services are you using?

Both Atlas Device Sync and Atlas App Services

Are you using encryption?

No

Platform OS and version(s)

React Native Expo 48.0.0

Build environment

Which debugger for React Native: ..

Cocoapods version

No response

[takameyer]

@njmulsqb Thanks for submitting this issue and sorry you are facing issues. Can you provide an example of the data that isn't showing up? What specific subscription is failing in the provided case?
It would be helpful to see which data value is causing the issue against which subscription.

[njmulsqb]

@takameyer Thanks for the response,

groups subscription is failing to sync properly here. Here's the schema of the group susbcription

import Realm from "realm";

class Group extends Realm.Object {
  static schema = {
    name: "group",
    properties: {
      _id: "objectId",
      defaultCurrency: "string?",
      description: "string?",
      dueDate: "date?",
      createdAt: "date?",
      inviteId: "string?",
      isArchived: "bool?",
      isPinned: "bool?",
      isSettledUp: "bool?",
      members: "group_members[]",
      name: "string",
      settlementCycle: "group_settlementCycle",
    },
    primaryKey: "_id",
  };
}
export default Group

The first three entries are only getting synced initially

and when I add some entry through realm.write all the rest of the data get synced then, one example of the rest of the data is as follows:

[takameyer]

@njmulsqb Can you verify that the groupIds in this code:

      const groupsQuery = realm
        .objects("group")
        .filtered("_id IN $0", groupIds);

reflects the issue you are having? Are there missing groupIds?

[njmulsqb]

Ah,
I think in the beginning I am missing groupIds, because it is being fetched from connections table where that ID is not present, but the question is, how are those groups being fetched after the creation then?

P.s. does realm has any official slack channel or discord server to discuss problems? The documentation is not so great and realm community is still maturing so it becomes challenging to solve realm related issues.

[takameyer]

@njmulsqb I think it's more of a concurrency issue, since the connections are being subscribed to and read from in the same callback. What I would recommend is to break up your useEffect into multiple useEffect hooks and create explicit dependencies which will update the subscriptions.
I would not advise changing the subscriptions too often, rather to setup a longer living subscription of more data and filtering the local data.

P.s. does realm has any official slack channel or discord server to discuss problems? The documentation is not so great and realm community is still maturing so it becomes challenging to solve realm related issues.

GitHub is our primary. We do also have a community forum which is also being monitored by us and was designed with discussions in mind.

[njmulsqb]

Thanks a bunch mate @takameyer

[takameyer]

@njmulsqb Cheers and happy coding!