How does pybullet check client ids for subprocesses?

[cadop]

Following some of the past issue discussions on multiprocessing, it seems the examples show how to use DIRECT with a pool of processes that are spawned at the same time to calculate some data and return results to a main process. I was trying a slightly different approach, which a main process would be able to run the server, do some single-core calculations, then 'distribute' some work to multiple cores, aggregating back and continuing some single-core simulation.

At first I thought the SHARED_MEMORY mode would make sense, from a python class generating n number of servers with a specific key, then in each subprocess passing it the key name to attach to. I couldn't get this to work, but also limited examples show how to set a key and start a server inside a running file, so if anyone has samples of this type of work it would be great to see.

So here I tried starting a server and simulating something, then spawning a bunch of processes that would do some calculation.

Run file

import multiprocessing as mp
import mp_tester as sim

def run():

    s = sim.Simulator()
    s.setup_simulator()
    s.run(n=3)
    
if __name__ == '__main__':
    mp.freeze_support()
    run()

the imported class

import os
import multiprocessing as mp
import pybullet as p
from pybullet_utils import bullet_client as bc

class Simulator():

    hz = 120    
    gravity = (0,0,-9.8)

    def __init__(self):
        self.p_client = None # Physics client for this instance 
        self.data_list = range(100)

    def setup_simulator(self, main=False, freq=hz):
        self.p_client = bc.BulletClient(connection_mode=p.DIRECT)
        print(f'Process client: {self.p_client._client}')
        self.p_client.setGravity(*self.gravity)
        self.p_client.setPhysicsEngineParameter(fixedTimeStep=1.0 / freq)

    def run(self, n=1):
        jobs = []
        results = mp.Queue()
        for i in range(n):
            print(f'Starting Job:{i}')
            proc = mp.Process(target=self.init_mp_sim, args=(i,results,))
            jobs.append(proc)
            proc.start()

        print('Getting results')
        res = [results.get() for i in range(n)]
        for pr in jobs: pr.join()
        print(f'Returned data: {res}')

    def init_mp_sim(self, i=None, q=None):
        self.setup_simulator()
        self.p_client.stepSimulation()
        print(f'Loaded and set gravity for process {i}')
        q.put([i])

This results in an error pybullet.error: Not connected to physics server from the subprocess in setup_simulator() when trying to set gravity. What I found strange was the first line, calling BulletClient(), does give an id as it prints Process client: 0, then immediately errors saying not connected. I would have expected the _client attribute to be -1 then. (perhaps this is just an interface issue with the bullet_client.py, I am not sure it is really checking if it is connected or not, it just assumes the instance has a connection so it returns that it is? )

After some testing, I noticed the subprocesses do work when I never create a bullet instance in the main process. From that I found I just had to change the variable name of the main process to get it to work.

Changing the internal variable name for the main process only, solves the issue of subprocesses not connecting to the server.

    def setup_simulator(self, main=False, freq=hz):
        if main:
            self.p_client_main = bc.BulletClient(connection_mode=p.DIRECT)
            print(f'Process client: {self.p_client_main._client}')
            self.p_client_main.setGravity(*self.gravity)
            self.p_client_main.setPhysicsEngineParameter(fixedTimeStep=1.0 / freq)
        else:
            self.p_client = bc.BulletClient(connection_mode=p.DIRECT)
            print(f'Process client: {self.p_client._client}')
            self.p_client.setGravity(*self.gravity)
            self.p_client.setPhysicsEngineParameter(fixedTimeStep=1.0 / freq)

When doing thing, all of the subprocesses have a client id of 0, but the code runs and they are able to connect to a server.

So when trying to figure out the client id situation, I went back and in the main script duplicated the creation of the client:

    s.setup_simulator()
    s.setup_simulator()

And this worked, I assume because the _client was set to 1 before the subprocess is created. Is that why, or is something else happening?

Perhaps this is an obvious python-related programming issue that is expected, but hope to get some confirmation/clarity from some people with experience on this. Or would it make sense for pybullet to catch this error and return a new connection and id?

[cadop]

So I do think it is an issue of the bullet_client util trying to 'cleanup' stuff that is not there, and since it really only knows about an id (which is an incremental int), and not bullet in memory, it disconnects something it shouldn't.

I tested this by adding some print statements to the bullet_client in the __del__ method. Using the first code setup that gets a physics connection error, I found:

• Main process creates a variable p_client that is assigned as an instance of the BulletClient(), assigning it an _client of 0.

• When the subprocess is spawned, it copies the entire class and scope (as expected from python).

• The subprocess then reassigns p_client to a new instance of BulletClient(), as that subprocess needs its own access to bullet (using DIRECT method). This new instance has a _client attribute of 0.

• Now, because the object p_client is reassigned, the old object (which was an instance copied over from the main process), is dereferenced and then has the __del__ method called.

• Because that instance was the first one created in the main process, it has a _client attribute of 0, and now that the subprocess has just created (and reassigned p_client) a BulletClient() instance with a _client attr of 0, the pybullet.disconnect() is passed 0, and disconnects from the subprocesses connection, not the main process.

• The reason duplicating the line worked in the example above was that it was simply making p_client=1, so when the subprocess is called and p_client is reassigned, the objects __del__ tries to disconnect id 1 (instead of 0), which doesn't matter.

@erwincoumans am I missing something or misusing the module? If this is not expected behavior I have a fix I could propose.

[erwincoumans]

provide a small repro python file please (that we can run) by attaching a zip file

[cadop]

mp_test.py is the first script I showed, 'mp_tester.py' is the second code block I gave. Run mp_test.py and it should error that server is not connected. Editing mp_test.py the way I explained solves it.

mp_failing.zip

[erwincoumans]

Please check this example:
https://github.com/bulletphysics/bullet3/blob/master/examples/pybullet/gym/pybullet_robots/panda/batchsim3.py
This example should work fine, and runs simulations in separate processes.

Can you modify it and see if the problem persists?

[cadop]

that example doesn't use a class, so there is no instance that would cause this error. it also has hard-coded the entire process in a main, so it seems to be the same as running completely separate files that don't share any common start.

I think the example zip I just gave is as simple as can be to show the example. if you still want to see that file edited I can, but it will be completely different because I need to wrap it in a class and make local variables.

[erwincoumans]

Looks like something specific to how Python and multiprocessing passes data around, not specific to PyBullet.
Thanks for pointing this out. If you can't find a solution, just go use https://github.com/bulletphysics/bullet3/blob/master/examples/pybullet/gym/pybullet_robots/panda/batchsim3.py as a starting point, that should work.

Unfortunately, I don't have time to further into this.

[cadop]

its a problem with pybullet_utils, not the pybullet.c interface. I can do a PR for a fix for bullet_client.py. The issue is with the __del__ method doing something it shouldn't in a subprocess. Are there python tests being run in CI (I saw a post awhile ago about trying to use gitlab), so I would know if it changes anything else?

[erwincoumans]

Are there python tests being run in CI

Yes, the docker container runs some tests:
https://github.com/bulletphysics/bullet3/blob/5ae9a15ecac7bc7e71f1ec1b544a55135d7d7e32/.ci/script.sh

This disconnect in __del__ is important, otherwise connections made by the bullet_client are not always disconnected, causing hard-to-debug issues.

def __del__(self):
    """Clean up connection if not already done."""
    if self._client>=0:
      try:
        pybullet.disconnect(physicsClientId=self._client)
        self._client = -1
      except pybullet.error:
        pass

[cadop]

The solution was provided in #3819