Gazebo freezes when YARP_CLOCK is set

[randaz81]

If I set the env variable:
YARP_CLOCK=/clock
and the I start gazebo with:
gazebo -s libgazebo_yarp_clock.so
then the system immediately freezes (the gazebo gui does not appear).

So, currently, the only way to work with simulated time is to have all process with YARP_CLOCK=/clock, with the only exception of gazebo that must have this env variable unset.
Thus the environment variable YARP_CLOCK cannot be put into a bash_rc file.

[traversaro]

Back in time (2015/2016) this used not to be the case, but at some point it started happening, probably due to some yarp::os::Time::delay that is being called in some callback of the Gazebo's Physics update, that is causing a deadlock, as yarp::os::Time::delay will be waiting for the Gazebo time to pass to return, but the Gazebo's Physics time will not pass if the physics update does not complete because it is blocked by yarp::os::Time::delay.

Is this happening in the case of empty world?

[randaz81]

Yes, it freezes just with:
gazebo -s libgazebo_yarp_clock.so
So the world is empty and the only plugin loaded is the clock

[traversaro]

Then it is possible that there is something going on with the Network constructors or the checkNetwork method?

[drdanz]

I don't remember changing anything...
A bisect is probably the best way to find where this issue was introduced

[traversaro]

I don't remember changing anything...
A bisect is probably the best way to find where this issue was introduced

I am afraid that bisecting will not help as gazebo-yarp-plugins will not compile against of a YARP older then 2018/2019 . Probably it is easier to check where the program is blocked after is launched via gazebo -s libgazebo_yarp_clock.so .

[xEnVrE]

To add some context, when launching Gazebo with that configuration on the terminal one reads

[INFO] |yarp.os.Port| Port /iiticublap202/gzserver/44642/clock:i active at tcp://192.168.1.5:10005/
[INFO] |yarp.os.Network| Success: port-to-port persistent connection added.
[ERROR] |yarp.os.NetworkClock| Cannot find time port "/clock" or a time topic "/clock@"
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
[INFO] |yarp.os.Time| Waiting for clock server to start broadcasting data ...
(...)

It seems that this is happening in

Time::useNetworkClock(const std::string& clock, const std::string& localPortName)

and is part of the initialization of the object yarp::os::Network in

gazebo-yarp-plugins/plugins/clock/src/Clock.cc

Line 71 in 6936859

m_network = new yarp::os::Network();

The yarp::os::Network object will try to read from a network clock that will be never available because the configuration of the clock plugin itself gets stuck in GazeboYarpClock::Load

The sequence of calls should be the following:

yarp::os::Network::Network()
yarp::os::Network::init()
yarp::os::Network::init(yarp::os::YARP_CLOCK_DEFAULT)
yarp::os::NetworkBase::initMinimum(yarp::os::YARP_CLOCK_DEFAULT)
yarp::os::NetworkBase::yarpClockInit(yarp::os::YARP_CLOCK_DEFAULT) (where the env variable YARP_CLOCK is detected)
Time::useNetworkClock("YARP_CLOCK")

[traversaro]

Thanks @xEnVrE , after your comment I am able to connect the dots. Probably the regression was caused by robotology/yarp#1277, that introduced the early clock initialization. Before that, the Network clock was lazy inizialized, so the plugin was able to load correctly even if YARP_CLOCK was set to /clock .

[traversaro]

A possible solution is to switch to use the system clock after yarp::sig::Network was initialized, and switch back to use the default clock once /clock has been created by calling yarp::os::NetworkBase::yarpClockInit(YARP_CLOCK_DEFAULT). This should work fine unless another Gazebo plugin called one of those methods to perturb the process-global YARP clock, but I think that is ok.

[traversaro]

A fix for this issue is provided in #537 .