-
Notifications
You must be signed in to change notification settings - Fork 2
Home
Underworlds3 (Uwds3)
In the previous version of Underworlds, the perception pipeline was separated from higher level reasoning, after experimenting with the original version it became clear that higher level reasoning need strong interaction with the perception layer to not loose information in the abstraction.
Uwds3 is a framework to design perception pipelines with high level reasoning capabilities like physical reasoning, perspective taking, human beliefs estimation, social signal processing or language models for human-robot interaction.
Uwds3 represented the environment by a set of entities called scene nodes that can be either a camera, meshes(or primitive shapes) or an abstract entity that do not have consistency, like the environment frame.
The timeline is in the same way a set of events or relations that give an extra layer of knowledge on top of this scene graph. Each event can be spatially located in the scene allowing the robot to look at this point for example.
One on the particularity of Uwds3 is that the [real-time simulation] is one of the core of the software. It allow the robot to have physical sense of his body at run-time but also to correct physical inconsistency during tabletop scenario by applying gravity and reasoning about collisions which allow the robot to infer the behavior of not visible objects.
Other particularity is the presence of a tensor based triplet store that store beliefs about agents for long term reasoning, while the timeline store only recent events. The formulation of this beliefs base allow to compute a divergence metric during a collaboration between the robot and a person to trigger repair actions.
More information in the dedicated section.
In underworlds, the pinhole camera model is a key component as it is used to project into a plane the 3D scene. If you are not familiar with this concept read this opencv lesson. The scene nodes, represent at the same time the 3D entity in the scene and the 2d bounding box represented by the camera that observe it. For example, one scene node can have multiple and distinct representation depending on how many camera is looking at it.
Uwds3 have it's own vector library that additionally implement for each one a stable version by using a linear kalman model in order to tradeoff accuracy and stability, two important parameters when dealing with physical simulation.
Each node that compose the scene graph can have multiple shapes associated
Each node can have one camera associated
The boundingbox represent an aligned bounding box in a 2D image plane
The detection represent the output of a detector it is composed by a bounding box and a mask