Interface for remote execution in Jupyter Notebook

[Mecoli1219]

Tracking issue or PR

#2579

Why are the changes needed?

Enable this usage:

init_remote(Config.auto(), default_project="flytesnacks", default_domain="development")

@task
def hello(name: str) -> str:
    return f"Hello {name}"

@task
def world(pre: str) -> str:
    return f"{pre} Welcome to the world!"

@workflow
def wf(name: str) -> str:
    return world(pre=hello(name=name))

future = wf.remote(name="Flyte says...")
out = future.get()

What changes were proposed in this pull request?

• a remote() function for workflow and task.
• a init_remote() function to store the remote configuration.
• FlyteFuture class to store the remote execution data.
• a new integration unit-test testing the Jupyter Notebook environment.
• Default option, currently set as another variable, should merge into FlyteRemote in the future
• Doc
• get_deck
• Add translator unit-test if interactive_mode_enabled => check pickle & no dest-dir
• update unit-test to remove version
• Remove fast_register_file_uploader

How was this patch tested?

All the tests were tested locally.

1. I created a ipynb file similar to the one provided in [wip] Jupyter notebooks support - run tasks on remote cluster! #2579 (https://www.loom.com/share/1369e80571e8427fbfc8e2c6ad127d67?sid=658c01ce-da5c-4bff-b50d-57c61107fba6) with the new interface.
   • My Demo
2. I tested the PyTorch example in Flytesnack.
3. I tested the new integration unit test with the docker image built from Dockerfile.dev.

Screenshots

1. get_deck()

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

#2579

Docs link

[kumare3]

OHH WOW this is amazing stuff. I am really excited about this. The only problem I see is if we simply convert this into a script it will fail, but we can solve that problem as a separate PR.

[kumare3]

@Mecoli1219 this is great, how about few more unit tests for translator, tracker and entrypoint.
I would love to have unit tests for entrypoint specifically. Otherwise it will be subtly break.

Also if you could test if this thing works with some large scale examples, like maybe when we have some state in jupyter notebook etc. Basically a few more real world tests. Also if ray/spark work?

[pingsutw]

if it always returns the "", why do we need it here?

[codecov]

Codecov Report

Attention: Patch coverage is 79.87013% with 31 lines in your changes missing coverage. Please review.

Project coverage is 49.94%. Comparing base (ae9c6f8) to head (76db07d).
Report is 57 commits behind head on master.

Files with missing lines	Patch %	Lines
flytekit/core/future.py	70.58%	13 Missing and 2 partials ⚠️
flytekit/tools/translator.py	80.00%	4 Missing and 4 partials ⚠️
flytekit/core/tracker.py	0.00%	2 Missing and 1 partial ⚠️
flytekit/remote/remote.py	88.00%	3 Missing ⚠️
flytekit/core/promise.py	75.00%	1 Missing ⚠️
flytekit/remote/executions.py	0.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (ae9c6f8) and HEAD (76db07d). Click for more details.

HEAD has 14 uploads less than BASE

Flag	BASE (ae9c6f8)	HEAD (76db07d)
	15	1

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2618       +/-   ##
===========================================
- Coverage   90.87%   49.94%   -40.93%     
===========================================
  Files          74      196      +122     
  Lines        3331    19824    +16493     
  Branches        0     2867     +2867     
===========================================
+ Hits         3027     9902     +6875     
- Misses        304     9399     +9095     
- Partials        0      523      +523     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Mecoli1219]

@kumare3 Default options and interactive_mode_enabled kwargs are added, and all the existing and new unit tests pass. Could you take a look at it? Thanks!
BTW, we can have a 1-on-1 chat if needed

[pingsutw]

Could we move the log (Jupyter notebook..) to init_remote?
Also, I think we can remove Uploading Pickled representation of Task to remote storage...

[pingsutw]

This code is the same as line 869~879; could we add a common function for it?

[thomasjpfan]

Done in #2733. We can open a new PR if we want to revisit the interface.