Real Time Collaboration (RTC) rollout plan for Spring 2023 and later (starting with Stat 159)

[fperez]

Summary

With JupyterLab 3.6.1, we can now evaluate again putting RTC in production. I started the process with #4217 (thanks @ryanlovett for the fixes!!).

Background: we tried this back in spring 2022, but we found severe data loss problems that led to backing it out (#3287). See also 2i2c-org/infrastructure#3027.

Also - the 2i2c team incidentally started this same conversation today!. I posted there and we can share as much info/experience as possible across the teams.

Here's my current thinking - I figured I'd post it here and we can discuss how viable this looks to the team. These are the steps as I see them:

1. Now: I plan on testing it this week with @facusapienza21 and @isaacdsloan.
2. Next (Sp23): If things look fine, we'll begin using it with the Stat159 students. The key question is obviously robustness and no data loss, but furthermore we'll be exploring what the educational/collaboration experience is like, filing issues as appropriate, etc.
3. Summer 23: Assuming this Spring'23 stat159 experiment goes well, then we could test (assuming a summer instructor is up for it) with the summer session of Data 100 on campus, which has typically a few hundred students (I have ~70 in stat159).
4. Fall 23: If that goes well, then I'm willing to run a larger test in the fall with Data 100 (~1,200 people).
5. Spring 24: At that point, we can then consider a potential campus-wide rollout for spring'24, which would reach O(10,000) people.

I see this as a reasonable staggered testing path/rollout, but ultimately that would need to be discussed with the Berkeley team in charge. For now consider this only as my personal ideas, that don't commit anyone to anything :)

User Stories

Real-time collaborative editing of notebooks has been long requested, and is now almost expected by students and instructors used to Google Docs.

Acceptance criteria

• No data loss.
• Satisfactory instructor experience.

Tasks to complete

• Test 3.6.1 in isolation for any user-facing problems.
• Activate RTC facilities and test for data loss.
• Explore usage in small class setting (Stat 159/Sp23) and reach clarity on robustness and educational value.
• Possible test in Summer'23 with Data100 or similar.
• Possible test in Fall'23 with Data100 (@fperez likely instructor).
• Possible deployment to other courses/hubs in Sp24. This would potentially require documentation and guidance for instructors on use of the feature.

[balajialg]

@fperez I believe that it is important to include some of the security considerations outlined in this issue as part of the acceptance criteria for rolling out RTC across other classes. With the current implementation in JupyterLab 3.6.1, I can generate a shareable link, open it in incognito mode without authenticating using bcourses and delete files that were present in my home directory. I see this as a potential issue for rolling it out in other classes and see the need to include security considerations as part of the acceptance criteria. Please correct me if I am missing something here.

Security Considerations for adopting RTC

• There is a share menu that generates a link with a token that can be sent to other users
• User shares a link to the next user, that user can then see and modify all the files in the user's home directory for this particular hub - including other assignments and homework
• This permission goes away when the user server stops - either explicitly from the control panel, or after non-activity for 60 minutes
• There is no ability to revoke the link, but you can stop the server and the link becomes invalid
• Anyone can forward the link to the next person - there is no link-level user control. Users would have to be careful where they share links while that server is active since anyone on the internet could get control.
• Currently, the Data 8 hub has its own file directory, only that class is on it - the main Datahub has all of the user's classes (eg connectors and modules on it)
• User needs to leave the session to go to work on their own files

[fperez]

Ah, absolutely!! I didn't realize that the shared link would bypass the main authentication!! I was under the (evidently mistaken!) belief that it would only work after the user had done the CalNet authentication.

This is most certainly a problem, and would prevent adoption in many other contexts.

We were just discussing this on Friday with @minrk and @rcthomas in the context of another deployment. It's my understanding that there, it works as I described. Folks - could you comment on this, and where my understanding went wrong?

[minrk]

Short answer: you need a convenient UI to share links to the current server without authentication. Right now, copying the browser URL is the only place that's currently available. If you don't want to grant users permission to impersonate you, don't install jupyterlab-link-share in your user env, since that's the main thing it facilitates.

I opened jupyterlab-contrib/jupyterlab-link-share#54 to possibly address this in jupyterlab-link-share

Longer answer:

There's two ways to share links - JupyterLab's built-in 'create shareable link' works correctly and will not include a token, but does point to /user-redirect/ to send you to your own server, not the current server. The jupyterlab-link-share extension, on the other hand, explicitly creates a link with your authentication token, granting anyone with the link permission to become you on that server without any authentication other than having that link. I believe this is done because the point is to be able to grant folks access to your server without needing to go through JupyterHub permissions, which can't (yet) be modified by users to grant others access to their own servers. It includes the appropriate warning:

but doesn't add JupyterHub-specific context that logging won't identify the user. Any time a link is enough to do anything (as opposed to link + login), that's going to be the case.

RTC doesn't change anything about this, though - it's purely a jupyterlab-link-share issue. You do need a nice way to create the links, though!

The built-in shareable link patch here in jupyterlab if patched back to self.base_url would produce the correct link.

[fperez]

Many thanks @minrk for the clarification and opening of that issue, we can follow up further in there.

[balajialg]

Thanks @minrk for the clarification. That's helpful!

[aculich]

@balajialg we here at D-Lab (cc: @pssachdeva @tomvannuenen ) are interested in testing out RTC features right now.

According to Data Science Curriculum Guide > Using Real Time Collaboration feature in Datahub

this experimental functionality is enabled in a few hubs (Data 8 and D-Lab at the time of write-up)

However we don't see it fully enabled, so can you please let us know what we need to do and/or fully enable it for us? Thanks!

During this Spring 2023 semester, we're planning to just experiment with it.

However, over Summer 2023 we hope it's ready for prime time for our Data Science for Social Justice Workshop June 12 – August 4 with a cohort of ~20-30 people.

Thanks!

[ryanlovett]

@aculich I spotted that in the docs a little while ago and mentioned to @balajialg that it is not accurate. RTC was enabled for some hubs last year but it caused data corruption at the time and was reverted. The docs were not updated afterwards.

Those issues were fixed, and RTC is now being tested in stat159 hub for Spring '23. A few new issues have emerged and are being worked on. It is the bleeding edge, requiring the latest versions of several major components, and @fperez is comfortable with managing the course like that, but it may not be the right configuration for your user community. If it were enabled for dlab.datahub, your users may run into stability or security issues, some related to RTC and some due to just running with very new components. For example in some circumstances, lab does not render correctly -- users just see a blank app with no cells displayed. (We haven't identified the cause, and it may be completely unrelated.)

Early on it was felt that if RTC worked well enough for Stat 159 for Spring, it might be suitable for Data 8 in summer. That may still be true, but we'll know more by the end of the term. It is not my place to gatekeep on what dlab wants to enable, but just FYI: there be dragons here. :) So if you do want to enable it, it will require a lot more user support in the short term and the infrastructure staff is not in a position to handle that. I'm happy to elaborate further, chat, etc.

[balajialg]

Thanks @ryanlovett for the detailed inputs. Appreciate it. Sorry @aculich for not updating the RTC docs with the latest information. It is on my ever growing to-do list and will probably update the latest information immediately so that it is relevant.

I hope you are tracking the open issues highlighted in this thread and the linked threads. I agree with @ryanlovett that it is better to wait till the semester ends before making a final call on enabling RTC in D Lab hub (taking inputs from @fperez's Stat 159 course). Let us know if that is a timeline you are comfortable with.

[aculich]

@balajialg actually, for our use case in D-Lab we have no graded student coursework so we're an ideal candidate given that we are workshop-based and won't explicitly be requiring our workshops use it. Instead we'll use it for internal testing and for piloting certain things.

And given the nature of our workshops, they are not long-duration. At most a 4-part series runs over the course of 1-2 weeks, so data loss is not a big issue in our context.

I think D-Lab can be a really great partner being on the leading/bleeding edge of your implementations so we can help test things in the wild on a smaller scale (5-100 people) before they go into wider production on campus. Think of us as a very willing testbed to support the work you're doing.

So, if we could enable it ASAP, I know that @pssachdeva @tomvannuenen are really eager to test it out and they have the technical background to be able give good informative feedback and debug reports.

[fperez]

FYI @aculich, if you and your team plan on testing the RTC machinery, a few things to keep in mind:

• Don't keep more than one tab open from the same browser simultaneously logged in. This is particularly problematic in the event of a server restart, as it is the main trigger of the document duplication bug.
• We're not quite sure what triggers the non-empty notebook appears blank bug.
• Keep an eye out for JupyterLab 3.6.2, which hopefully will fixes for at least some of the above. See relevant PRs #lab/collab 108 and #lab 14128 for more details.

Thanks for any info you can help us gather on this machinery!! Happy to chat about our approach to deploying it, BTW :)

cc @facusapienza21 for reference.

[balajialg]

@aculich Sounds great. Thanks for your interest to pilot RTC despite the limitations. As a next step, Created a PR to enable RTC again in the D-Lab hub - #4371. Once it gets merged to staging, will reach out to you for testing it.

Thanks, @fperez for your input. Updated the RTC docs with your inputs - https://ds-modules.github.io/curriculum-guide/workflow/use-realtimecollaboration.html?highlight=real