BUG: Weird interaction with other modules

[larsoner]

This bit of code is adapted from two pylsl examples and just starts one thread to send samples and receives them in the main thread, works on Linux (Ubuntu 20.10, pip) as it correctly starts printing a bunch of random samples:

import time
from multiprocessing import Process
from random import random as rand

from pylsl import StreamInfo, StreamOutlet, local_clock, resolve_stream, StreamInlet
import scipy.fft


def _send():
    info = StreamInfo('BioSemi', 'EEG', 8, 100, 'float32', 'myuid34234')
    outlet = StreamOutlet(info)
    start_time = local_clock()
    sent_samples = 0
    while True:
        elapsed_time = local_clock() - start_time
        required_samples = int(100 * elapsed_time) - sent_samples
        for sample_ix in range(required_samples):
            # make a new random n_channels sample; this is converted into a
            # pylsl.vectorf (the data type that is expected by push_sample)
            mysample = [rand() for _ in range(8)]
            # now send it
            outlet.push_sample(mysample)
        sent_samples += required_samples
        # now send it and wait for a bit before trying again.
        time.sleep(0.01)


print('Starting serving process')
process = Process(target=_send, daemon=True)
process.start()

print("looking for an EEG stream...")
streams = resolve_stream('type', 'EEG')
inlet = StreamInlet(streams[0])
while True:
    sample, timestamp = inlet.pull_sample()
    print(timestamp, sample)

However, if I move the import scipy.fft to before the pylsl import, things break. The stream is never found and it just hangs on the resolve_stream step. I narrowed the from scipy import fft issue down to it being due to an import of scipy.fft._pocketfft.pypocketfft, which is a pybind11-compiled module. (I can also experience it when other modules are imported in that same spot in the script, so I don't think it's specific to SciPy, though.)

This is on Linux, where I noticed there is a known issue that multiple threads are not supported. Is this what is meant by "not supported" in that it might work in some situations but not others depending on the import order? If so, is it expected that the conda-forge version will not have this problem, or is it some fundamental limitation? Also I'm not sure if this is related to #6 / #5 or not.

The above example is a minimized version of how we unit test and build docs for MNE-Realtime, hence my interest in it being fixed if at all possible.

[larsoner]

Okay looks like it's not a threading problem -- doing this with the built-in examples works fine:

$ python -u SendData.py & python -u ReceiveData.py 
[1] 132179
looking for an EEG stream...
2021-02-08 15:27:11.392 (   0.000s) [        2CC42740]             common.cpp:50    INFO| v1.14.0
now sending data...
2021-02-08 15:27:11.891 (   0.500s) [        7E6CC740]             common.cpp:50    INFO| v1.14.0
1612816032.1011267 [0.32223236560821533, 0.4601549804210663, 0.6337152719497681, 0.37596964836120605, 0.6952264308929443, 0.7275350689888, 0.5970755815505981, 0.5940881967544556]
1612816032.1011553 [0.5168118476867676, 0.18368901312351227, 0.8172319531440735, 0.6709344387054443, 0.7501394152641296, 0.33982735872268677, 0.8810415863990784, 0.16764724254608154]
...

But if I add a import scipy.fft to the ReceiveData.py script, it just hangs:

$ python -u SendData.py & python -u ReceiveData.py 
[2] 131766
2021-02-08 15:25:36.148 (   0.000s) [        2FBBB740]             common.cpp:50    INFO| v1.14.0
now sending data...
looking for an EEG stream...

So something about the pylsl and scipy.fft (and other modules) seems not to work.

I've tried this on latest scipy master built from source, plus SciPy 1.6.0 from PyPi. And CircleCI using pip also has a variant of this problem on our CIs.

[larsoner]

... and it looks like it works okay with pylsl-1.12.2 but not pylsl-1.13.1 based on pip install-based version testing. So something about the code and/or lib change over that span seems to have broken things.

If it matters, this is all Python 3.9 on Ubuntu 20.10 x86_64.

[cboulay]

What Linux distro are you on? I only have Ubuntu 20.04 in easy reach.
Edit: Sorry I saw the Linux distro after.

It would help narrow down the problem tremendously if you could please try the following.

pip install pylsl to get the latest version. Then git clone https://github.com/sccn/liblsl.git, build that, and copy the liblsl.so it outputs to the pylsl folder (replacing the old liblsl.so). If this works then we know it's not a liblsl version problem, it's just how liblsl is built for the linux pylsl packages on pypi. (Aside: there seems to be a problem with the MacOS packages too for some versions)

We're hoping the next version of liblsl will have a different distribution process. See sccn/liblsl#110
Then we don't need to build liblsl for manylinux, so hopefully we can avoid this problem altogether. Getting liblsl into Debian or NeuroDebian might take longer than you'd like, so maybe we can start with a custom conda channel.

is it expected that the conda-forge version will not have this problem?

Sorry what are you referring to here? Did you test something installed from conda and it didn't have the bug?
I'm guessing not pylsl because that's not on conda-forge.

[larsoner]

I 1) git cloned this repo, 2) pip install -ve .ed it to use latest master, 3) git cloned liblsl.git, and then:

$ cmake -S . -B build -G Ninja
$ cd build
$ ninja -j 4
$ ln -s $PWD/liblsl.so ../../liblsl-Python/pylsl/lib/

And everything works! And when I replace liblsl-Python/pylsl/lib/liblsl.so with:

• https://github.com/sccn/liblsl/releases/download/1.13.1/liblsl-1.13.1-manylinux1_x64.so - works
• https://github.com/sccn/liblsl/releases/download/v1.14.0/liblsl-1.14.0-manylinux1_x64.so - works
• https://github.com/sccn/liblsl/releases/download/1.13.1/liblsl-1.13.1-manylinux2010_x64.so - fails
• https://github.com/sccn/liblsl/releases/download/v1.14.0/liblsl-1.14.0-manylinux2010_x64.so - fails
• ~/.local/lib/python3.9/site-packages/pylsl/lib/liblsl.so from the 1.14.0 installation by pip - fails

And then I verified that the 1.14.0 manylinux2010 library has the same MD5 (1793cbd2a45621572a14ff9090c57f32) as the 1.14.0 from PyPi / ~/.local/.... So it seems like the manylinux2010 build is for some reason problematic.

[cboulay]

Thank you so much! This was tremendously helpful.
I deleted all of the manylinux2010 releases from pypi. I hope the manylinux1 release is adequate for most users until we start shipping liblsl through package managers.

[larsoner]

Indeed I just did pip uninstall pylsl; pip install --no-cache pylsl and things appear to be working, thanks!