Collections and Snapshot modes

[rivella50]

Hi there,

great idea, thanks for adding the project.
I have some questions:

• if i've read the documentation correctly, this also works for streaming Firestore collections?
• comparing to snapshots you currently support the mode QuerySnapshot.docs i'd say. Do you plan to support .docChanges as well?
• which StreamFetchPolicy would i need to choose if i'm interested in the most recent version from the server, and if there is no network available a fallback to the cache version (is that the Nimbostratus or the Firebase cache) should be performed?

I was looking for such a solution and i'm curious if i've found it with your package.

[danReynolds]

Hey thanks for reaching out.

1. Yes it should work for streaming collections, which Firestore API are you thinking of?
2. You're right docChanges isn't supported yet. Currently streaming multiple documents returns an array of DocumentSnapshots not a single QuerySnapshot that tells you which documents have been added, removed or modified. I'd imagine supporting docChanges would require switching the streamDocuments API to return a query snapshot so that it could include the metadata or alternatively there could be a streamDocumentChanges API added that does it.If you'd be interested in contributing feel free to explore a solution, I'd be happy to offer support there. Otherwise I'll be adding this to the feature roadmap and look into it when I can.
3. I think you would want to do a cacheAndServer policy then to receive both server and cache updates. If the server is unreachable, than the data would come from the cache. If it became reachable, Firestore should start serving it from the network again. This is actually the default behavior for Firestore I believe. Have you seen it work offline as described here? https://firebase.google.com/docs/firestore/manage-data/enable-offline is that the intended behavior you're looking for or is there something else you need?

[rivella50]

Hi Dan,
thanks for answering. Here my comments:

1. I'm thinking of the cloud_firestore API
2. Yes you're right, i guess then you'd have to support QuerySnapshot as well
3. The reason why i'm interested in using your package (or parts of it) is this issue i've filed only last Friday.

In short: Certain views in my app need the most recent version of a streamed collection value on init/build and stay with it while shown to the user (e.g. when user input is needed and showing a data update would be not a good idea).
When relistening a StreamProvider (this happens after leaving a view, all providers are disposed, and then i enter the view again) i always get the cached version of the collection (which is normal and ok) and only in a second event shortly afterwards (but already too late for my mentioned views) the most recent list (snapshot.docs) or changes (snapshot.docChanges) from the database, if there were some changes in the database while i was not listening the stream provider.
Since your package seems to offer various modes to choose from in which order or single mode the data is fetched it looks promising to use (or at least some ideas from it).

Unfortunately i'm in the middle of a Flutter project and do not have spare time to contribute at the moment.
Perhaps at a later time in the year, let's see.
Therefore yes, please add the features you think that are missing and look suitable for this package to your roadmap.

Regarding the mentioned Firebase issue i've also thought about creating an additional layer on top of my Firestore requests, where it would then be possible to define per (re)listening start of a stream from which source the first data event has to come (cache or server) including a fallback solution (e.g. if network is not available or if the database doesn't have any newer data than our cache contains).

[danReynolds]

Got it, if the issue is that your views are first loading the data from the cache and then from the server moments later, the serverFirst stream policy would be the one that you'd want to use here since it'll block on getting that server response.

[rivella50]

Cool. Today i've built something similar with Query.get and using GetOptions. Interestingly with .get they let you choose from 3 different source modes. With Query.snapshots this is not possible.

In your code i see this snippet in streamDocuments:

case StreamFetchPolicy.serverFirst:
      case StreamFetchPolicy.cacheOnly:
      case StreamFetchPolicy.cacheFirst:
        return Stream.fromFuture(
          getDocuments(
            docQuery,
            fetchPolicy: convertStreamFetchPolicyToGetFetchPolicy(fetchPolicy),
            fromFirestore: fromFirestore,
          ),
        ).switchMap((snapshots) {
          if (snapshots.isEmpty) {
            return Stream.value([]);
          }

          final streams = snapshots.map((snapshot) => snapshot.stream).toList();
          return CombineLatestStream.list(streams);
        });

Is this returned stream capable of receiving further updates from Firestore after the first event, or is it limited to the return value of getDocuments?

[danReynolds]

It is currently limited to updates to the documents that were initially returned, as the goal of serverFirst is to read the server once and then only read data from the cache and there is not a way to know when to requery the cache if for example a document that satisfies the query that was not in the initial server response is written at another time.

The cacheAndSever policy alternatively will continue listening to both the cache and server so if a new document satisfies the query on the server then that query will automatically update.

[rivella50]

Ah now i understand.
But aren't you missing a case here? With cacheAndServer you start with cacheOnly and merge then with the serverStream. Since the cached version is delivered faster this one will be the first data event the provider gets.
What if i need the same continuosly delivering mechanism but the first version must come from the server to be sure that the cached data isn't outdated? This applies to the case of relistening a stream provider where cached data is still around and what i try to be enhanced in the mentioned Firebase issue.

[danReynolds]

@rivella50 that makes sense to me. Would this change be what you're looking for then? #4

[rivella50]

I would say yes - need to test it in my own project though (you remember that i actually work with the snapshots variant and quite often with docChanges there).
Thanks a lot for your quick response, i'm looking forward to the new release.

[danReynolds]

Great, moving forward with that change then. Merged in and available under release 0.1.0.

[rivella50]

Very nice, thank you.
I will try it tomorrow in a demo app and let you know.

[rivella50]

Ok i was too curious and tried it right now with cacheAndServerFirst.
I get 2 notifications from the server part (either on first provider listen or on relistening after performing some manual changes in the database). Both notifications contain the exact same and correct data, but i wonder if it couldn't be reduced to just one event?
In your code i see that you combine the 2 modes serverOnly and cacheAndServer. It makes sense with that solution to receive 2 notifications because also with cacheAndServer it detects that the server version differs from the cached version, but it's still overkill. Therefore i would say that the cacheAndServer mode in the second place is the wrong mode here.

[danReynolds]

Hey there! I assume you're referring to this block of code: https://github.com/danReynolds/nimbostratus/blob/master/lib/nimbostratus.dart#L602. It's not actually the serverOnly/cacheAndServer combo that is emitting two identical events, since the result from the serverOnly is mapped over the cacheAndServer one, it is not emitted. The double event is instead coming from the fact that the cacheAndServer policy emits two events, first from the cache and then again from the server. This will only occur the first time we get the server result, when we switch from the serverFirst policy to the cacheAndServer one.

After thinking it through though I do agree that we can move away from cacheAndFirst for this new policy and have changed the code as shown here:

#5.

I've pushed that version up as 0.1.2 so feel free to give it a try.

[rivella50]

That looks much better, very nice!
Next milestone would then be to support docQuery.snapshots as well ;-)

[rivella50]

Although: Does cacheAndServerFirst mode has anything to do with cache in version 0.1.2?
The caching aspect would be interesting as a fallback if e.g. the server is not available.

Regarding implementing QuerySnaphots: I've seen that you already go for snapshots in mode serverOnly, therefore in theory supporting the variants docs and docChanges isn't that far away i would say.

[danReynolds]

Although: Does cacheAndServerFirst mode has anything to do with cache in version 0.1.2? The caching aspect would be interesting as a fallback if e.g. the server is not available.

Regarding implementing QuerySnaphots: I've seen that you already go for snapshots in mode serverOnly, therefore in theory supporting the variants docs and docChanges isn't that far away i would say.

cacheAndServerFirst will provide updates via the cache after the initial server update. Based on how I believe firestore works in offline mode, currently cacheAndServerFirst would not deliver data from the cache if you're offline because it is blocking on getting that event from the server. I haven't worked much with it offline though so I'd have to look into it more.

[rivella50]

Hi Dan,
fyi: i've opened a feature request today at Firebase support which asks for adding the GetOptions parameter in Query.snapshots and DocReference.snaphots calls in the native Firebase SDK.
If they approve and implement it, the parameter could be added FlutterFire's cloud_firestore package as well.