Documentation Request: vscode jupyter integration

[dlashua]

Like many programmers and Home Assistant users, VSCode is my editor of choice. VSCode has Jupyter Notebook functionality builtin. It even supports multiple kernels. However, I get errors when attempting to use it "locally".

Unsupported KernelSpec file. args must be [<pythonPath>, '-m', <moduleName>, arg1, arg2, ..]. Provied /usr/bin/python3 --ip=127.0.0.1 --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"89b5054b-983d-4ac9-b2ee-d8fe74606e74" --shell=9002 --transport="tcp" --iopub=9004 --f=/tmp/tmp-3332Qc6ejHA53Qtk.json

If I, however, run jupyter notebook on the command line and then paste the URL it provides in to VSCode as the remote server, it works fine.

Do you use VSCode when working through Jupyter, or are you using the browser-based application? Do you have any tips to make this work?

[craigbarratt]

I've never used VSCode; I just use Jupyter notebook directly when interacting with HASS.

It looks like VSCode is providing all the necessary information to launch the kernel, but not in a form that the current kernelspec shim expects. Currently it expects just a json file with all the parameters.

What's in the /tmp/tmp-3332Qc6ejHA53Qtk.json file?

I suspect we need to either update the current kernel shim to allow the VSCode-style of arguments, or make a new shim for VSCode.

I guess I should start by trying out VSCode. Where in VSCode do you select a Jupyter kernel?

[craigbarratt]

I installed VSCode and I got the same error as you did. I searched the vscode-python source and couldn't find where that error message is coming from.

Unfortunately there's almost no documentation for Jupyter kernels. The pyscript one works with Jupyter notebook, console and lab. Someone who knows VSCode is going to have to figure out how it tries to launch the kernel, since it isn't the same way Jupyter does.

Unfortunately, pasting the Jupyter URL doesn't work for me. That would be a decent workaround; not sure why it doesn't work. When I paste the URL, hitting "Enter" does nothing - the dialog stays and there is no visible error or effect.

Also, in looking at the VSCode python setup options, I see there is a setting for python.dataScience.jupyterCommandLineArguments, but I can't find any documentation for it or where it is used in the source code.

[dlashua]

Ooooh. I had that same problem at first (pasting the URL doing nothing). Trying to remember how I resolved it....

[dlashua]

I think the "URL does nothing" issue was the result of my URL not changing. If the kernel breaks and you want to retry with the same URL, even though it looks like it's changing (because the box contains something else to begin with) if it's not different from the last time you configured it, it does nothing. I "fixed" this by entering some invalid URL, allowing it to reload the window and fail, and then setting it back to the correct URL.

VSCode's Jupyter support seems to be "just enough" to consider it supported, but not very complete. Once it's working, it works well. But getting it there is a pain. Which is a shame because, VSCode is otherwise quite wonderful on every platform I've used it on (MacOS, Ubuntu, and Windows 10).

I think I have an idea on why it won't accept the pyscript kernel. I think it expects argv to be in the ["python", "-m", "modulename", "-f", "{connection_file}"] format and fails because of that.

This is only a guess based on using other modules and seeing what the error messages are.

Adding the "-f" option to the kernel is trivial. But, I've never written a python module so I'm not sure what's involved with that. Though, in general, even without VSCode, the install process for the pyscript kernel would be easier if one could "pip install" it instead of copying files manually.

[craigbarratt]

I'm going to refactor hass-pyscript-jupyter and turn it into a PyPi-installable package. It will need to include some helper script entry point that installs the pyscript kernelspec, and the kernelspec arguments will need to be very similar to the IPython ones, so VSCode hopefully stops complaining. The user will still have to edit the pyscript kernel config file to add the HASS server URL and token.

[craigbarratt]

Ok, I've done a first pass at the major refactoring so that it can be a PyPi installable package. I haven't pushed the package to PyPi yet, but you can create the package locally (see the README file).

It seems to work ok with Jupyter, but still doesn't work with VSCode. It does get past the earlier error about the arguments, but it doesn't connect and gets a timeout.

I did make progress in finding the JavaScript code in VSCode that does the Jupyter connection, but I haven't look at it yet.

[craigbarratt]

I got as far as confirming the hass_pyscript_kernel package doesn't get executed by VSCode, even though the kernel.json file should be to its liking:

{
  "argv": [
    "python",
    "-m",
    "hass_pyscript_kernel",
    "{connection_file}"
  ],
  "display_name": "hass pyscript",
  "language": "python"
}

I did find the code in .vscode/extensions/ms-python.python-2020.10.332292344/out/client/extension.js. After pretty-printing it using the Chrome debugger, around line 133336 it has:

        async launch(e, t, n, r) {
            const [i,s] = await Promise.all([this.daemonPool.get(e, n, r), o.pathExists(t)])
              , a = n.argv.slice()
              , c = a.findIndex(e=>"-m" === e);
            if (-1 === c)
                throw new Error("Unsupported KernelSpec file. args must be [<pythonPath>, '-m', <moduleName>, arg1, arg2, ..]. Provied " + a.join(" "));
            const l = a[c + 1]
              , u = a.slice(c + 2)
              , d = n.env && Object.keys(n.env).length > 0 ? n.env : void 0;
            if ("start"in i) {
                const e = await i.start(l, u, {
                    env: d,
                    cwd: t
                });
                return e.proc && this.processesToDispose.push(e.proc),
                {
                    observableOutput: e,
                    daemon: i
                }
            }
            return {
                observableOutput: i.execModuleObservable(l, u, {
                    env: d,
                    cwd: s ? t : process.cwd()
                }),
                daemon: void 0
            }
        }

but I don't know how to navigate to this code and set breakpoints from inside VSCode. I can get the Chrome debugging window up from inside VSCode (Help -> Toggle Developer Tools), and some core JavaScript is visible, but not this file.

I don't know JavaScript nor the tools to debug it, so it's probably not worthwhile for me to pursue this further.

[dlashua]

Thank you for taking it this far! I know Javascript fairly well (a little less than I know Python) so I'll see where I can take it from here.

[craigbarratt]

One more discovery: around line 132345, there is this:

async updateConnectionArgs() {
            const e = d.findIndexOfConnectionFile(this.launchKernelSpec);
            if (-1 === e)
                throw new Error("Connection file not found in kernelspec json args, " + this.launchKernelSpec.argv.join(" "));
            if (this.isPythonKernel && 0 === e && "-f" !== this.launchKernelSpec.argv[e - 1])
                throw new Error("Connection file not found in kernelspec json args, " + this.launchKernelSpec.argv.join(" "));
            if (this.isPythonKernel)
                this.launchKernelSpec.argv.splice(e - 1, 2),
                this.launchKernelSpec.argv.push(...this.addPythonConnectionArgs());

So VSCode requires a -f flag before the {connection_file} argument in kernel.json (as well as requiring a -m flag too). It then replaces the -f and {connection_file} with a whole bunch of arguments that duplicate what's in the connection file, which it supplies with a --f option, eg:

python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9028 --control=9026 --hb=9025 --Session.signature_scheme="hmac-sha256" --Session.key=b"fd129d18-12ae-47bd-3d50-1b994819fcde" --shell=9027 --transport="tcp" --iopub=9029 --f=/var/folders/3d/05ystg8j4qb6ww6q46br8m041200gn/T/tmp-48441kLTNac6Uhiwu.json

So the arguments in kernel.json need a very specific form (-m, -f , {connection_file}) and VSCode replaces the last two with a bunch of other arguments.

Currently my kernel shim doesn't take a -f option. I'll go ahead and add that, and support the other arguments VSCode is supplying. Hopefully it will get further with those changes.

[craigbarratt]

Progress... things aren't quite right (for some reason VSCode starts two kernels; not sure why, and log output doesn't display). But basic statements, autocomplete and exceptions work. Here's the first signs of life...

[dlashua]

I was able to install the module manually and Jupyter is functioning within VSCode for me now.

I only see one Kernel running. However, as you indicated, log.info() does not display in VSCode as it does in Notebook over the web.

[craigbarratt]

Thanks for testing it! I pushed this change to pyscript master about an hour ago, which should fix log.info() output in VSCode. Are you running with that fix?

Background log output (eg, from a trigger function) doesn't get displayed in VSCode, unlike Jupyter notebook. So you'll have to look at the HASS log file for that.

[dlashua]

I am not. I am currently running my PR to "detect old attributes". So I need to pull your master into that. I will do so shortly.

[craigbarratt]

The change is just in jupyter_kernel.py, so it shouldn't overlap your changes.

[dlashua]

As expected, this shows a log in VSCode:

log.info('hi')

This does not...

@time_trigger('startup')
def startup():
    log.info('hello')

Thank you for fixing VSCode. So much easier to stick in the same IDE than jumping to the browser.

[tms320]

I don't see pyscript kernel in VSCode. What am I doing wrong?
See my issue #23