The simulator is open-source and free to use. It is aimed for, but not limited to, academic research. We welcome forking of this repository, pull requests, and any contributions in the spirit of open science and open-source code 😍😄 For enquiries about collaboration, you may contact [email protected].
If you use the coupled sim for academic work please cite the following paper:
Bazilinskyy, P., Kooijman, L., Dodou, D., & De Winter, J. C. F. (2020). Coupled simulator for research on the interaction between pedestrians and (automated) vehicles. 19th Driving Simulation Conference (DSC). Antibes, France.
- Get out of the way! Examining eHMIs in critical driver-pedestrian encounters in a coupled simulator - repo, paper.
- Stopping by looking: A driver-pedestrian interaction study in a coupled simulator using head-mounted displays with eye-tracking - paper.
📺 These days, a video is worth more than a million words. The image below points to a YouTube video of the recording of a demo of the simulator with 3 agents:
The coupled simulator supports both day and night-time settings. Figure above shows a view of the night mode. Figures below shows the top view of the environment. It is a model of a US-like city containing:
- Network of 2-lane roads.
- Loop of 4-lane road (partially surrounded by buildings).
- Loop of 6-lane road (partially surrounded by buildings).
- Half-clover interchange for the motorway.
- 10 intersections with traffic lights that can be turned on and off before the experiment or programmatically in real-time.
- 34 zebra crossings.
- Static objects (buildings, parked cars, trees).
- small (similar to Smart Fortwo) - DrivableSmartCommon object.
- medium (similar to Ford Focus 2011) - DrivableHatchbackCommon object.
- medium (similar to Pontiac GTO) - DrivableFireGTOCommon object.
- large (similar to Nissan Datsun) - DrivableNissanCommon object.
Cars that are not controlled by the human participants can be instructed to follow a defined trajectory or can be programmed to respond at runtime to other road users. Adding new vehicles can be done by new importing assets (easy).
The coupled simulator supports a keyboard and a gaming steering wheel as input sources for the driver of the manual car, a keyboard for the passenger of the AV to control the external human-machine interface, and a motion suit or a keyboard for the pedestrian. At the moment, supported motion suit is Xsens Motion Suit.
The simulator supports giving output to both a computer screen and a head-mounted display (HMD). It has been tested with Oculus Rift CV1, Varjo VR-2 Pro.
The current number of human participants supported by the coupled simulator is four (host and three clients). However, this number can be expanded up to the number of agents supported by the network. Synchronization in a local network is handled by a custom-made network manager designed to support the exchange of information between agents with low latency and real-time data logging at 50 Hz for variables from the Unity environment and up to 700Hz from the motion suit. The data that are logged include the three-dimensional position and rotation of the manual car and the AV, the use of blinkers, high-beam, stop light, and 150 position and angular variables from the motion suit. The data are stored in binary format, and the coupled simulator contains a function to convert the saved data into a CSV file (Convert log to csv button in Play Mode).
Besides logging data to the binary file, the same set of frame data (in a very similar binary format) is being sent with requested intervals (that can be set with NetworkingManger.RealtimeLogInterval property) during the simulation to UDP port 40131 on localhost. The data can be used to monitor the simulation with external tools at runtime. Tools/log_receiver.py file contains an example python script that consumes binary data and assembles it into a structure that is easy to interact with. It should be started before starting the simulation with python log_receiver.py (with Python 3) from within the Tools folder.
The structure of the frame object (with example data) is as follows:
{
"timestamp":1.711700439453125,
"roundtrip":0.0,
"Driver [car index]":{ # single integer-indexed entry for a player controlled car avatar
"position":(-94.58521270751953, -0.056745946407318115, 64.8435287475586),
"rotation":(0.5252280235290527, 90.8692626953125, 359.9905700683594),
"blinker":0,
"front":false,
"stop":false,
"rigidbody":(3.1246094703674316, -0.0026992361526936293, -0.046936508268117905)
},
"Driver [car index]":{ # multiple integer-indexed entries for each AI-controlled car avatars
"position":(-112.08060455322266, -0.056995689868927, 21.383098602294922),
"rotation":(0.5083497762680054, 157.48184204101562, -0.0032447741832584143),
"blinker":0,
"front":false,
"stop":false,
"rigidbody":(0.5082536935806274, -0.0017757670721039176, -1.2219140529632568),
"speed":5.904001235961914,
"braking":false,
"stopped":false,
"takeoff":false,
"eyecontact":false
},
"Pedestrian [pedestrian index]":{ # multiple integer-indexed entries for each player or AI-controlled pedestrian avatars
"bone [bone index]":{ # multiple integer-indexed entries for each bone of pedestrian rig
"position":(-94.58521270751953, -0.056745946407318115, 64.8435287475586),
"rotation":(0.5252280235290527, 90.8692626953125, 359.9905700683594)
}
},
"CarLight [car traffic light index]":{ # multiple integer-indexed entries for each car's traffic light
"state":"b""\\x00"
},
"PedestrianLight [pedestrian traffic light index]":{ # multiple integer-indexed entries for each pedestrain's traffic light
"state":"b""\\x02"
},
}
The host agent may use Visual syncing button to display a red bar across all clients and the host for 1 s to allow for visual synchronization in case screen capture software is used.
The simulator was tested on Windows 11 and macOS Ventura 13.5. All functionality is supported by both platforms. However, support for input and output devices was tested only on Windows 10.
After checking out this project, launch Unity Hub to run the simulator with the correct version of Unity (currently 2022.3.5f1).
When opening this project for the first time, click Add and select your entire downloaded folder containing all the project files. In case you have already used this project before, select the project from the Unity Hub projects list. Wait until the project loads in.
Once the project is loaded into the Unity editor open StartScene scene. You do this by going to the project tab, and clicking on the StartScene icon in the Scenes folder. Now, we will explain three different ways to run scenes, the locations for the first three steps for running as a host or client correlate with the numbers in the picture below.
- Make sure that all three checkboxes (
Hide GUI,Run Trial Sequence Automatically,Record Videos) inNetworkingManagercomponent onManagersgame object are unchecked. - Press the Play button to run enter
Play Mode. - Once in
Play Mode, pressStart Hostbutton. - If needed, wait for clients to join.
- Once all clients have connected to the host or in case the host is handling the only participant, select one of the experiments listed under
Experimentsection. - Assign roles to participants in
Role bindingsection. If no role is selected, the participant will run a "headless" mode. - Select one of control modes listed under
Modesection. - Start an experiment with the
Initialise experimentbutton - all clients will load selected experiment. - Once all connected clients are ready - the experiment scene is fully loaded on each client, press
Start simulationbutton.
- Make sure that all three checkboxes -
Hide GUI,Run trial sequence automatically,Record videos(Managers(game object) ➡️NetworkingManager(component)), are unchecked. - Press the Play button to run enter
Play mode. - Once in
Play Mode, pressStart clientbutton. - Enter the host IP address.
- Press
Connect. - Once connected, select one of control modes listed under
Modesection. - Wait until host starts the simulation.
- Run host agent inside of Unity (as described above).
- Note the IP address in the console of Unity.
- Run other agents after building them
File➡️Build And Run. Connect using the IP address from the (1).
Providing input for multiple agents on one machine is not supported. Input is provided only to the agent, the window of which is selected. This mode is intended for testing/debugging.
If the simulation only has one participant that is controlled on a host machine, simulation trials can be set up beforehand and run automatically. It is especially useful when using simulator to record videos for trials which is described in next section. Most of a times user should have any GUI disabled, which can be done by checking Hide GUI checkbox. GUI can be enabled at runtime by pressing Tab button on the keyboard.
To run simulation trials automatically, both Run Trial Sequence Automatically has to be checked and trial sequence has to be set up. Once it is done, press Play button to enter Play Mode - first trial in the sequence should start automatically. To finish current trial and either start next one or exit simulator (if currently played trial the last one), press Escape button on the keyboard.
In order to set up trial sequence, user has to define entries on the Trials list (StartScene (scene) ➡️ Managers (game object) ➡️ NetworkingManager (component) ➡️ Trials (click arrow to open the field)). Each entry consists of the following fields:
ExperimentIndex: int variable which indicates a zero-based index of a selected experiment inExperimentslist (NetworkingManager(component) ➡️Experiments(field)).RoleIndex: int variable which indicates a zero-based index of a selected role inRoleslist (ExperimentDefinition(component) ➡️Roles(field)) of a selected experiment prefab.InputMode: enum variable, that sets participants display/controller pair for the trial. Available values are:Flat: use a flat-screen to display simulation and mouse&keyboard/gamepad/steering wheel to control it.VR: use virtual reality headset to display simulation and mouse&keyboard/gamepad/steering wheel to control it.Suite: use virtual reality headset to display simulation and XSense suite to control it (only pedestrian avatar).
Additional experiment parameters can be defined for each trial that would modify baseline scenario implemented in experiment. Parameters take form of name-value pair defined in ExperimentParameters list. Those parameters are consumed by any enabled scripts contained in experiment prefab that implement IExperimentModifier interface right after experiment is loaded and before simulation has started. An example of such a script is EnableAVLabel.
Simulator is able to record trial videos for "offline" use. Most of the setup is the same as for running trials automatically as described above. There are three additional steps that need to be taken to set up automatic trials recording.
Record Videoscheckbox has to be checked.- Each trial in
Trialslist has to haveRecording Start TimeandRecording Durationdefined.Recording Start Timedefines at which second after the trial has started video recording should start.Recording Durationdefines how long the recording will last. After recording is finished simulator will proceed to the next trial in the sequence. - Following video output parameters can be set up in
Managers(game object) ➡️Recorder(component): --Directory- output directory relative to Application.dataPath (when running from Unity Editor it isAssetsfolder). --Resolution- videos output resolution --Framerate- videos output frame rate
Filenames of recorded videos conform following naming scheme:
{trial index}\`{ExperimentDefinition.ShortName}\`roleIdx-{role index}\`{multiple "\`" separated " paremeter name-value pairs}\`{date and time in "yy-MM-dd\`hh-mm" format}
The central point for simulators configuration are two major components on Managers game object from the StartScene scene:
PlayerSystem: gathering references to player avatar prefabs,NetworkingManager: gathering references to experiment definitions and elements spawned during networked experiment runtime (currently only waypoint-tracking cars -AICar).
The experiment is defined solely with prefab containing the ExperimentDefinition component in the root object.
To make newly created experiment selectable you have to add its prefab to Experiments list on NetworkingManager component.
To edit the experiment definition, double click the experiment prefab in the Project window.
Prefab will be opened in edit mode along with the currently defined Regular Prefab Editing Environment. When defining the experiment it is worth setting Regular Prefab Editing Environment variable to the Unity scene defined in the experiment (Edit ➡️ Project Settings ➡️ Editor ➡️ Prefab Editing Environments ➡️ Regular Environment).
ExperimentDefinition component defines the following fields:
ShortName: the short name of the experiment used in video recording output filenamesName: the name of the experimentScene: Unity scene name to be loaded as an experiment environmentRoles: list defining roles that can be taken during an experiment by participantsPoints of Interest: static points that are logged in experiment logs to be used in the log processing and analysisCar Spawners: references to game objects spawning AI-controlled carsAI Pedestrians: defines AI-controlled pedestrians behaviour with a list ofPedestrianDescstructures containing a pair of anAIPedestrian(the game object that defines an AI-controlled pedestrian avatar) andWaypointCircuit(defining a path of waypoints for the linked avatar)
Points of interest is a list of Transform references.
CarSpawners list references game objects containing component inheriting from CarSpawnerBase. It defines, with overridden IEnumerator SpawnCoroutine() method, spawn sequence (see BaseSyncedCarSpawner for reference implementation). Car prefabs spawned by the coroutine with AICar Spawn(CarSpawnParams parameters, bool yielding) method must be one of the referenced prefabs in AvatarPrefabDriver list on NetworkManager component.
Base.prefab from ExperimentDefinitions folder is an example experiment definition showcasing most of simulator features.
Roles field is a list of ExperimentRoleDefinition structures defining experiment roles with the following base data fields:
Name: short name/description of the roleSpawnPoint.Point: defines where player avatar will be spawnedSpawnPoint.Type: a type of player avatar. It may be: --PlayerControlledPedestrian, --PlayerControlingCar, --PlayerInAIControlledCar.
To add a new agent either increase the size of Roles array or duplicate existing role by right-clicking on the role name and selecting Duplicate from the context menu.
To remove the agent right click on the role name and select Delete from the context menu or decrease the list size removing end entries that don't fit the resized list.
Add a new game object to the prefab and set its position and rotation.
Drag the newly created game object object to the SpawnPoint.Point field in role definition.
Avatar prefab can have multiple cameras defined (PlayerAvatar (component) ➡️ Cameras (field)), for example for a driver and passenger position. Camera that will displaying world for the avatar is defined by providing SpawnPoint.CameraIndex.
Additionally to the location, additional camera settings can be provided for a spawned agent with CameraSetup component. The component can be added to the SpawnPoint.Point game object.
The component exposes two fields:
FieldOfView: value which is set at spawn-time toCamera.fieldOfViewproperty.Rotation: value which is set at spawn-time toTransform.localRotationof a game object hostingCameracomponent.
No additional configuration is needed for pedestrian type agents.
Following additional fields has to be defined:
CarIdx- indicates car prefab that will be spawned for this role. Selected prefab is the one on the indicated index onAvatarPrefabDriverlist (field onPlayerSystemcomponent)VehicleType- indicates how spawned car will be configured. It may be eitherAVorMDV. This parameter corresponds toAVandMDVsections inPlayerAvatarcomponent.
Following additional fields has to be defined:
TopHMI,WindshieldHMI(see image below),HoodHMIfields - defines which eHMI prefab to spawn on corresponding spots. Spots are defined in player avatar prefabs (PlayerAvatar(component) ➡️HMI Slots(field)).AutonomusPath- references game object defining waypoints for the autonomous car viaWaypointCirciutcomponent
Paths that can be followed both by non-playable pedestrians and vehicles are defined with the WaypointCircuit component.
To add waypoint - press + sign at the bottom of the list and drag waypoint Transform into the newly added field.
To remove waypoint - select waypoint on the Items list and press + sign at the bottom of the list.
To reorder waypoint - drag selected list item to the new position on the list.
To change position of a waypoint - select waypoint transform (by double clicking on the reference from the Items list) and move it do the desired position.
Additionally, for pedestrians, PedestrianWaypoint along with trigger BoxCollider component might be used to further configure agents behaviour on a tracked path.
PedestrianWaypoint component allows to change walking speed when pedestrian avatar enters BoxCollider with following parameters:
targetSpeed- controls movement speedtargetBlendFactor- controls animation speed, by blending between idle and full speed walk
Additionally, for vehicles, SpeedSettings along with trigger BoxCollider component might be used to further configure agents behaviour on a tracked path.
SpeedSettingscomponent allows to change car behaviour when car avatar entersBoxCollider.Type- Indicates what kind of waypoint is it -- InitialSetSpeed - Waypoint placed at the spawn point that sets up initial speed tospeedor to 0 ifcauseToYieldis set to true. -- SetSpeedTarget - Regular waypoint that changes car behaviour according to rest of parameters. -- Delete - Destroys car avatar.speed- Indicates target speed that car will be trying to reach.acceleration- Indicates acceleration that will be used to reach target speed (If target speed is lower than current speed, value has to be negative.)BlinkerState- Indicates whether car should have any of turn indicator blinking.causeToYield- If checked, car will decelerate withbreakingAcceleration, waityieldTimeand resume driving by accelerating withaccelerationuntil car reachesspeed.yieldTime- Indicates how long car should stay in place after making a full stop.brakingAcceleration- IfcauseToYieldis true, this value needs to be negative. Eye contact related parameters are described in "Configuration of driver's eye-contact behaviour" section.
Waypoint can trigger custom behaviour, on AICar with linked CustomBehaviour derived component, by providing ScriptableObject deriving form CustomBehaviourData (See reference implementation in BlinkWithFrontLights and BlinkPatternData). CustomBehaviour components should be linked to AICar at spawn time (See reference implementation in BaseSyncedCarSpawner).
WaypointCirciut can be serialized into CSV format (semicolon separated) with an Export to file button.
CSV file can be modified in any external editor and then imported with an Import from file button. Importing files will remove all current waypoint objects and replace them with newly created ones according to the data in the imported CSV file. Following parameters are serialized to and de serialized from CSV files:
GameObjectproperties
| Property name | CSV column | Type | Description |
|---|---|---|---|
| name | name | string | name |
| tag | tag | string | tag |
| layer | layer | integer | unity layer index |
Transformproperties
| Property name | CSV columns | Units | Description |
|---|---|---|---|
| position | x; y; z | meters | position |
| eulerAngles | rotX; rotY; rotZ | degrees | rotation in euler angles |
| localScale | scaleX; scaleY; scaleZ | - | local scale |
SpeedSettingsproperties
| Property name | CSV column | Type | CSV values |
|---|---|---|---|
| Type | waypointType | WaypointType enum | 0 for InitialSetSpeed, 1 for SetSpeedTarget, 2 for Delete |
| speed | speed | km/h | - |
| acceleration | acceleration | m/s2 | - |
| BlinkerState | blinkerState | BlinkerState enum | (0 for None, 1 for Left, 2 for Right) |
| causeToYield | causeToYield | boolean | True/False |
| EyeContactWhileYielding | lookAtPlayerWhileYielding | boolean | True/False |
| EyeContactAfterYielding | lookAtPlayerAfterYielding | boolean | True/False |
| yieldTime | yieldTime | seconds | - |
| brakingAcceleration | brakingAcceleration | m/s2 | - |
| YieldingEyeContactSince | lookAtPedFromSeconds | seconds | - |
| YieldingEyeContactUntil | lookAtPedToSeconds | seconds | - |
| customBehaviourData | customBehaviourDataString | any class derived from CustomBehaviourData | multiple entries of % separated {scriptable object name}#{scriptable object instance id} pairs |
BoxColliderproperties
| Property name | CSV column(s) | CSV values | Description |
|---|---|---|---|
| enabled | collider`enabled | (True/False) | is component enabled |
| isTrigger | isTrigger | (True/False) | is collider a trigger |
| center | centerX; centerY; centerZ | - | box collider center |
| size | sizeX; sizeY; sizeZ | - | box collider size |
Initial eye contact tracking state and base tracking parameters are defined with fields in the EyeContact component.
EyeContactTracking defines the initial (and current at runtime) driver's eye contact behaviour while the car is not fully stopped.
MinTrackingDistanceandMaxTrackingDistancedefine (in meters) the range of distances at which eye contact tracking is possible. Distance is measured between the driver's head position and the pedestrian's root position (ignoring a distance on a vertical axis).MaxHeadRotation(in degrees) limits head movement on a vertical axis.EyeContact, if tracking is enabled, selects as the target the closest game object tagged with a"Pedestrian"tag that is within the distance range, if it meets rotation constraint (this constrain is checked when the closest object is already selected).
EyeContactRigControl is a component that consumes tracking target provided by EyeContact component and animates drivers head movement.
Eye contact behaviour tracking state can be changed when the car reaches the waypoint. Behaviour change is defined by the SpeedSettings - the component embedded on waypoint objects. The following four fields control those changes:
EyeContactWhileYielding: defines how the driver will behave while the car is fully stoppedEyeContactAfterYielding: defines how the driver will behave when the car resumes driving after a full stop. This value simply overwrites the current value ofEyeContact.EyeContactTrackingif the car has fully stopped.YieldingEyeContactSince: defines how many seconds need to pass before the driver will make eye contact (starting from the moment the car has fully stopped)YieldingEyeContactUntil: defines how many seconds need to pass before the driver ceases to maintain eye contact (starting from the moment the car has fully stopped)
DayNightControl component helps to define different experiment daylight conditions. It gathers lighting-related objects and allows defining two lightings presets - Day and Night, that can be quickly switched for a scene. This component is intended to be used at the experiment definition setup stage. When the development of the environment is complete, it is advised, to save the environment into two separate scenes (with different light setups) and bake lightmaps.
Creating a traffic street lights system is best started with creating an instance of ExampleStreetLightCrossSection prefab and adjusting it.
TrafficLightsManager component manages state of CarSection and PedestrainSection objects that group respectively CarTrafficLight and PedestrianTrafficLight instances that share common behaviour.
Traffic light initial state is defined with list of TrafficLightEvent structures stored in initialStreetLightSetup field. This events are triggered once before simulation has started.
Traffic light initial state is defined with list of TrafficLightEvent structures stored in streetLightEvents field. This events are triggered sequentially in a loop.
Each event is defined with the following fields:
Name: descriptive name of an eventDelta Time: relative time that has to pass since previous event to activate the eventCarSections: cars traffic light group that the event applies toPedestrianSections: pedestrian traffic light group that the event applies toState: state to be set on the lights specified by sections
CurrentIndex and CurrentTimer allow to skip certain parts of sequence.
CurrentIndexselectsTrafficLightEventthat the sequence will start from.CurrentTimersets time offset from the start of the selectedTrafficLightEvent.
Speedometer component controls a speed indicator.
To use digital display, set the Speedometer Text field in order to do that.
To use analog display, set the following fields:
Pivot- the pivot of an arrowPivotMinSpeedAngle- the inclination of an arrow for 0 speedPivotMaxSpeedAngle- the inclination of an arrow for max speedMaxSpeed- max speed displayed on the analog display
Connect the ASUS Router with the USB Adapter to the PC and plug the Ethernet in a yellow port of the router. Let it boot up for some time while you prepare the software and the suit. Connect the Oculus Rift HDMI and USB 3.0 to the computer.
Plug in the black MVN dongle in a USB 3.0 port of the computer Run the latest version of MVN Analyze. Fill out the anthropometric data and participant name by creating a new recording (1).
Check under Options ➡️ Preferences ➡️ Miscellaneous ➡️ Network Streamer that the data streamer is set to Unity3D.
Note: if you find out later on that somehow the avatar looks buggy in Unity, play around with the down sampling skip factor in the Network Streamer overview to improve rendering.
Turn on the Body Pack with all the sensors connected. This will sound like one beep. Wait for 30 seconds and press the WPS button on the router. When the Body Pack of the suit beeps again, continuously press the button for 2 seconds until you hear two sequential beeps. Look if the light of the button becomes a strobe light. If this does not happen within a minute or 5, you’ve got a problem named Windows.
Delete the following software from the pc and re-install the latest version of MVN Analyze. This will re-install also the bonjour print services.
If you have an avatar in MVN Analyze and all the sensors are working, boot Unity for the simulation. See figure below: press play to launch the simulation, use the dropdown menu to select a participant and press the trial button to launch a trial.
Note. If you want to match the orientation of the Oculus to the orientation of the avatar’s head, make sure you have left-clicked the game screen in Unity and press the R-key on your keyboard. Pressing the R-key to match the visuals with the head orientation is an iterative process which requires feedback from the participants.
If you want to start a new trial, click play again at the top of the Unity screen to end the simulation. Also match the head orientations again with the R-key loop.
Always run Oculus Home software when using Oculus Rift. Otherwise you will encounter black screens. Make sure your graphics driver and USB 3.0 connections are up to date. If Oculus gives a critical hardware error, disconnect Rift and set Oculus Home software to Beta (Public Test Channel, 1st figure below) and check if Oculus Home setting is set to allow Unknown apps (2nd figure below).
The agent PCs need to be connected via a local network. If you cannot reach the host machine, try to ping it.
Inbound rules: Set correct Unity Editor version to allow all connections.
Check if supporting software is installed (e.g., Logitech gaming software G27 is used in our case). In Unity, you can check which button corresponds to your specific wheel. You can find this out by using the following asset in an empty project: https://assetstore.unity.com/packages/tools/input-management/controller-tester-43621.
Then make sure you assign the correct inputs in Unity under Edit ➡️ Project Settings ➡️ Input (see figure below).
We have used the following free assets:
| Name | Developer | License |
|---|---|---|
| ACP | Saarg | MIT |
| Textures | Various | Creative Commons CC0 |
| Oculus Integration | Oculus | Oculus SDK License |
| Simple modular street Kit | Jacek Jankowski | Unity asset |
| Realistic Tree 10 | Rakshi Games | Unity asset |
| Small Town America - Streets | MultiFlagStudios | Unity asset |
| Cars Free - Muscle Car Pack | Super Icon LTD | Depricated, Unity asset |
| Mini Cargo Truck | Marcobian Games | Unity asset |
| Street Bench | Rakshi Games | Unity asset |
| Waste bin | Lowpoly`Master | Unity asset |
| Smart fortwo | Filippo Citati | MIT |
| Vehicle - Essentials | Nox Sound | Unity asset |
| Road System | Maxi Barmetler | Unity asset |
| Kajaman's Roads | Kajaman | Unity asset |
| HD Low Poly Racing Car No.1201 | Azerilo | Unity asset (free at time of download) |