Added GerRL tool

README.md

@@ -1,61 +1,40 @@
-# Cyber-Physical Systems Testing Competition #
-
-The [SBST Workshop](https://sbst22.github.io/) offers a challenge for software testers who want to work with self-driving cars in the context of the usual [tool competition](https://sbst22.github.io/tools/).
-
-## Important Dates
-
-The deadline to submit your tool is: **January 21st 2022**
-
-The results of the evaluation will be communicated to participants on: **February 25th 2022**
-
-The camera-ready paper describing your tool is due to: **Sunday March 18th 2020**
-
-## Goal ##
-The competitors should generate virtual roads to test a lane keeping assist system using the provided code_pipeline.
-
-The generated roads are evaluated in the [**BeamNG.tech**](https://www.beamng.tech/) driving simulator.
-This simulator is ideal for researchers due to its state-of-the-art soft-body physics simulation, ease of access to sensory data, and a Python API to control the simulation.
-
-[![Video by BeamNg GmbH](https://github.com/BeamNG/BeamNGpy/raw/master/media/steering.gif)](https://github.com/BeamNG/BeamNGpy/raw/master/media/steering.gif)
+# GenRL at the Cyber-Physical Systems Testing Competition #
 
->Note: BeamNG GmbH, the company developing the simulator, kindly offers it for free for researcher purposes upon registration (see [Installation](documentation/INSTALL.md)).
+GenRL is a tool that **Gen**erates effective test cases for a lane-keeping system in a simulated
+environment using **R**einforcement **L**earning (RL).
 
-## Comparing the Test Generators ##
+This repository is a fork of [tool-competition-av](https://github.com/se2p/tool-competition-av), in which we implemented our RL-based approach.
 
-Deciding which test generator is the best is far from trivial and, currently, remains an open challenge. In this competition, we rank test generators by considering various metrics of effectiveness and efficiency that characterize the generated tests but also the process of generating them, i.e., test generation. We believe that our approach to compare test generators is objective and fair, and it can provide a compact metric to rank them.
-
-### Ranking Formula
+Install additional dependencies for GenRL with
+```
+pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio===0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
 
-The formula to rank test generators is the following weighted sum:
+pip install -r additional-requirements.txt 
+```
 
+To start the test generator using the tool competition pipeline, run `competition.py` with the following command line parameters:
 ```
-rank = a * OOB_Coverage + b * test_generation_efficiency + c *  test_generation_effectiveness
+--module-name genrl_sbst2022.genrl_test_generator --class-name GenrlTestGenerator 
 ```
 
-where:
-
-- `OOB_Coverage` captures the effectiveness of the generated tests that must expose as many failures as possible (i.e., Out Of Bound episodes) but also as many different failures as possible. We compute this metric by extending the approach adopted in the previous edition of the competition with our recent work on [Illumination Search](https://dl.acm.org/doi/10.1145/3460319.3464811). As an example, our novel approach has been already adopted for the generation of relevant test cases from existing maps (see [SALVO](https://ieeexplore.ieee.org/document/9564107)). Therefore, we identify tests' portion relevant to the OOBs, extract their structural and behavioral features, and populate feature maps of a predefined size (i.e., 25x25 cells). Finally, we define `OOB_Coverage` by counting the cells in the map covered by the exposed OOBs. **Larger values of `OOB_Coverage` identify better test generators.**
-
-- `test_generation_efficiency` captures the efficiency in generating, but not executing, the tests. We measure it as the inverse of the average time it takes for the generators to create the tests normalized using the following (standard) formula: 
-
-    ``` norm(x) = (x - min) / (max - min)```
-
-    Where `min` and `max` are values empirically found during the benchmarking as the minimum and maximum average times for generating test across all the competitors.
+# Cyber-Physical Systems Testing Competition #
 
-- `test_generation_effectiveness` captures the ability of the test generator to create valid tests; therefore, we compute it as the ratio of valid tests over all the generated tests.
+The [SBST Workshop](https://sbst22.github.io/) offers a challenge for software testers who want to work with self-driving cars in the context of the usual [tool competition](https://sbst22.github.io/tools/).
 
+## Important Dates
 
-### Setting the Weights
+The deadline to submit your tool is: **January 14th 2022**
 
-We set the values of the in the ranking formula's weights (i.e., `a`, `b`, and `c`) to rank higher the test generators that trigger many and different failures; test generation efficiency and effectiveness are given equal but secondary importance. The motivation behind this choice is that test generators' main goal is to trigger failures, while being efficient and effective in generating the tests is of second order importance.
+The results of the evaluation will be communicated to participants on: **February 25th 2022**
 
-The following table summarizes the proposed weight assignment:
+The camera-ready paper describing your tool is due to: **Sunday March 18th 2020**
 
-| a | b | c |
-|---|---|---|
-|0.6|0.2|0.2|
+## Goal ##
+The competitors should generate virtual roads to test a lane keeping assist system using the provided code_pipeline.
 
+The generated roads are evaluated in a driving simulator. We partnered with BeamNG GmbH which offers a version of their simulators for researchers, named [BeamNG.tech](https://www.beamng.tech/). This simulator is ideal for researchers due to its state-of-the-art soft-body physics simulation, ease of access to sensory data, and a Python API to control the simulation.
 
+[![Video by BeamNg GmbH](https://github.com/BeamNG/BeamNGpy/raw/master/media/steering.gif)](https://github.com/BeamNG/BeamNGpy/raw/master/media/steering.gif)
 
 ## Implement Your Test Generator ##
 We make available a [code pipeline](code_pipeline) that will integrate your test generator with the simulator by validating, executing and evaluating your test cases. Moreover, we offer some [sample test generators](sample_test_generators/README.md) to show how to use our code pipeline.

genrl_sbst2022/README.md

@@ -0,0 +1,18 @@
+# GenRL at the Cyber-Physical Systems Testing Competition #
+
+GenRL is a tool that **Gen**erates effective test cases for a lane-keeping system in a simulated
+environment using **R**einforcement **L**earning (RL).
+
+This repository is a fork of [tool-competition-av](https://github.com/se2p/tool-competition-av), in which we implemented our RL-based approach.
+
+Install additional dependencies for GenRL with
+```
+pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio===0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
+
+pip install -r additional-requirements.txt 
+```
+
+To start the test generator using the tool competition pipeline, run `competition.py` with the following command line parameters:
+```
+--module-name genrl_sbst2022.genrl_test_generator --class-name GenrlTestGenerator 
+```

genrl_sbst2022/genrl_test_generator.py

@@ -0,0 +1,38 @@
+import logging as log
+
+from stable_baselines3 import PPO
+
+from genrl_sbst2022.road_generation_env_transform import RoadGenerationTransformationEnv
+
+class GenrlTestGenerator:
+    """
+        Generates tests using a RL-based approach
+    """
+
+    def __init__(self, executor=None, map_size=None):
+        self.executor = executor
+        self.map_size = map_size
+
+    def start(self):
+        log.info("Starting CaRL test generator")
+
+        # Instantiate the environment
+        # env = RoadGenerationContinuousEnv(test_executor, max_number_of_points=20)
+        # env = RoadGenerationDiscreteEnv(test_executor, max_number_of_points=8)
+        env = RoadGenerationTransformationEnv(self.executor, max_number_of_points=4)
+
+        # Instantiate the agent
+        model = PPO('MlpPolicy', env, verbose=1)
+
+        # Start training the agent
+        log.info("Starting training")
+        model.learn(total_timesteps=int(1e2))
+
+        # If training is done and we still have time left, we generate new tests using the trained policy until the
+        # given time budget is up.
+        log.info("Generating tests with the trained agent")
+        while not self.executor.time_budget.is_over():
+            obs = env.reset()
+            while not done:
+                action = model.predict(observation=obs)
+                obs, reward, done, info = env.step(action)

genrl_sbst2022/levels_template/tig/art/road/materials.cs

@@ -0,0 +1,73 @@
+
+singleton Material(tig_road_rubber_sticky)
+{
+    mapTo = "tig_road_rubber_sticky";
+    diffuseMap[0] = "levels/tig/art/road/road_asphalt_2lane_d.dds";
+    doubleSided = "0";
+    translucentBlendOp = "LerpAlpha";
+    normalMap[0] = "levels/tig/art/road/road_asphalt_2lane_n.dds";
+    specularPower[0] = "1";
+    useAnisotropic[0] = "1";
+    materialTag0 = "RoadAndPath";
+    materialTag1 = "beamng";
+    specularMap[0] = "levels/tig/art/road/road_asphalt_2lane_s.dds";
+    reflectivityMap[0] = "levels/tig/art/road/road_rubber_sticky_d.dds";
+    cubemap = "global_cubemap_metalblurred";
+    translucent = "1";
+    translucentZWrite = "1";
+    alphaTest = "0";
+    alphaRef = "255";
+    castShadows = "0";
+    specularStrength[0] = "0";
+};
+
+
+
+
+singleton Material(tig_line_white)
+{
+    mapTo = "tig_line_white";
+    doubleSided = "0";
+    translucentBlendOp = "LerpAlpha";
+    normalMap[0] = "levels/tig/art/road/line_white_n.dds";
+    specularPower[0] = "1";
+    useAnisotropic[0] = "1";
+    materialTag0 = "RoadAndPath";
+    materialTag1 = "beamng";
+    //cubemap = "cubemap_road_sky_reflection";
+    //specularMap[0] = "levels/tig/art/road/line_white_s.dds";
+    translucent = "1";
+    translucentZWrite = "1";
+    alphaTest = "0";
+    alphaRef = "255";
+    castShadows = "0";
+    specularStrength[0] = "0";
+   colorMap[0] = "levels/tig/art/road/line_white_d.dds";
+   annotation = "SOLID_LINE";
+   specularStrength0 = "0";
+   specularColor0 = "1 1 1 1";
+   materialTag2 = "driver_training";
+};
+
+singleton Material(tig_line_yellow)
+{
+    mapTo = "tig_line_yellow";
+    doubleSided = "0";
+    translucentBlendOp = "LerpAlpha";
+    normalMap[0] = "levels/tig/art/road/line_white_n.dds";
+    specularPower[0] = "1";
+    useAnisotropic[0] = "1";
+    materialTag0 = "RoadAndPath";
+    materialTag1 = "beamng";
+    //cubemap = "cubemap_road_sky_reflection";
+    //specularMap[0] = "levels/tig/art/road/line_yellowblack_s.dds";
+    translucent = "1";
+    translucentZWrite = "1";
+    alphaTest = "0";
+    alphaRef = "255";
+    castShadows = "0";
+    specularStrength[0] = "0";
+    annotation = "SOLID_LINE";
+   colorMap[0] = "levels/tig/art/road/line_yellow_d.dds";
+   specularStrength0 = "0";
+};

genrl_sbst2022/levels_template/tig/art/terrains/materials.cs

@@ -0,0 +1,17 @@
+
+new TerrainMaterial()
+{
+    diffuseMap = "levels/tig/art/terrains/Overlay_Grass-01";
+    detailMap = "levels/tig/art/terrains/Grass-01-D";
+    internalName = "groundmodel_asphalt1";
+    diffuseSize = "150";
+    detailDistance = "50";
+    normalMap = "levels/tig/art/terrains/Grass-01-N";
+    macroSize = "100";
+    macroStrength = "0.4";
+    macroDistance = "450";
+    macroMap = "levels/tig/art/terrains/Macro_grass";
+    detailSize = "3";
+    detailStrength = "0.7";
+    annotation = "GRASS_THAT_BEHAVES_LIKE_ASPHALT";
+};

genrl_sbst2022/levels_template/tig/info.json

@@ -0,0 +1,7 @@
+{
+	"title": "tig",
+	"description": "Precrime base level",
+	"previews": ["template_preview.jpg"],
+	"size": [2048, 2048],
+	"authors": "Precrime team"
+}

genrl_sbst2022/levels_template/tig/main.decals.json

@@ -0,0 +1,10 @@
+{
+ "header":{
+  "name":"DecalData File",
+  "comments":"// Instances format: rectIdx, size, renderPriority, position.x, position.y, position.z, normal.x, normal.y, normal.z, tangent.x, tangent.y, tangent.z",
+  "version":1}
+  ,
+ "instances":{
+  }
+  }
+

genrl_sbst2022/levels_template/tig/main/MissionGroup/CameraBookmarks/items.level.json

@@ -0,0 +1 @@
+{"name":"overviewbookmark","internalName":"NewCamera_0","class":"CameraBookmark","persistentId":"87700263-bfbd-48dd-a941-750c49025e6f","__parent":"CameraBookmarks","position":[17.95940017700195,-19.97909927368164,8.283820152282715],"datablock":"CameraBookmarkMarker","isAIControlled":"0","rotationMatrix":[0.6946083307266235,0.7193881273269653,-1.639127731323242e-07,-0.6582382917404175,0.6355648040771484,-0.4034596681594849,-0.2902439832687378,0.2802465558052063,0.9149974584579468]}

genrl_sbst2022/levels_template/tig/main/MissionGroup/generated/items.level.json-readme.txt

@@ -0,0 +1,3 @@
+This is an example of file that describe the customized objects into the 3d level
+
+{"name": "street_1", "class": "DecalRoad", "breakAngle": 180, "distanceFade": [1000, 1000], "drivability": 1, "material": "tig_road_rubber_sticky", "overObjects": true, "persistentId": "d62cdb76-3384-4620-9d3d-134b21968f32", "__parent": "generated", "position": [0, 0.0, -28, 8], "textureLength": 2.5, "nodes": [[0, 0.0, -28, 8], [5, 4.207354924039483, -28, 8], [10, 4.546487134128409, -28, 8], [15, 0.7056000402993361, -28, 8], [20, -3.7840124765396412, -28, 8], [25, -4.794621373315692, -28, 8], [30, -1.3970774909946293, -28, 8], [35, 3.2849329935939453, -28, 8], [40, 4.946791233116909, -28, 8], [45, 2.060592426208783, -28, 8], [50, -2.7201055544468487, -28, 8], [55, -4.9999510327535175, -28, 8], [60, -2.6828645900021746, -28, 8], [65, 2.1008351841332047, -28, 8], [70, 4.953036778474352, -28, 8], [75, 3.251439200785584, -28, 8], [80, -1.4395165833253265, -28, 8], [85, -4.806987459397784, -28, 8], [90, -3.7549362338583805, -28, 8], [95, 0.7493860483147617, -28, 8], [100, 4.564726253638138, -28, 8], [105, 4.18327819268028, -28, 8], [110, -0.04425654645201938, -28, 8], [115, -4.231102020875853, -28, 8], [120, -4.527891810033119, -28, 8], [125, -0.6617587504888651, -28, 8], [130, 3.8127922523980136, -28, 8], [135, 4.781879642022515, -28, 8], [140, 1.3545289415393453, -28, 8], [145, -3.318169421064838, -28, 8], [150, -4.940158120464309, -28, 8], [155, -2.0201882266153253, -28, 8], [160, 2.757133406208453, -28, 8], [165, 4.999559300536336, -28, 8], [170, 2.645413430600119, -28, 8], [175, -2.140913347480755, -28, 8], [180, -4.9588942672155785, -28, 8], [185, -3.2176906667849976, -28, 8], [190, 1.4818428935469266, -28, 8], [195, 4.818976931420439, -28, 8], [200, 3.725565802396744, -28, 8], [205, -0.7931133440235449, -28, 8], [210, -4.582607739578169, -28, 8], [215, -4.158873713142992, -28, 8], [220, 0.08850962552706788, -28, 8], [225, 4.254517622670592, -28, 8], [230, 4.508941738244046, -28, 8], [235, 0.61786561372612, -28, 8], [240, -3.841273306618334, -28, 8], [245, -4.768763263797359, -28, 8], [250, -1.3118742685196438, -28, 8], [255, 3.3511458792168733, -28, 8], [260, 4.933137960202426, -28, 8], [265, 1.9796257509091708, -28, 8], [270, -2.793945244258081, -28, 8], [275, -4.9987758667931, -28, 8], [280, -2.607755010434559, -28, 8], [285, 2.1808237762391247, -28, 8], [290, 4.964363240422686, -28, 8], [295, 3.1836900356956894, -28, 8], [300, -1.5240531055110833, -28, 8], [305, -4.8305888500419645, -28, 8], [310, -3.695903483246114, -28, 8], [315, 0.8367785015140345, -28, 8], [320, 4.600130190983953, -28, 8], [325, 4.134143397450517, -28, 8], [330, -0.13275577011983397, -28, 8], [335, -4.277599894876611, -28, 8], [340, -4.489638403446456, -28, 8], [345, -0.5739240689159362, -28, 8], [350, 3.8694534077894454, -28, 8], [355, 4.755273266271873, -28, 8], [360, 1.2691168138101814, -28, 8], [365, -3.383859784436538, -28, 8], [370, -4.925731302341237, -28, 8], [375, -1.9389081770471521, -28, 8], [380, 2.8305381844909014, -28, 8], [385, 4.997600792903657, -28, 8], [390, 2.569892279937676, -28, 8], [395, -2.220563343537542, -28, 8], [400, -4.969443269616876, -28, 8], [405, -3.1494399713722694, -28, 8], [410, 1.5661439121654257, -28, 8], [415, 4.841822305500926, -28, 8], [420, 3.665951600366461, -28, 8], [425, -0.8803780997429355, -28, 8], [430, -4.617292235020299, -28, 8], [435, -4.109089183154112, -28, 8], [440, 0.1769915136683034, -28, 8], [445, 4.300347029062267, -28, 8], [450, 4.4699833180027895, -28, 8], [455, 0.5299375587557843, -28, 8], [460, -3.8973303480790236, -28, 8], [465, -4.741410706349736, -28, 8], [470, -1.2262599273382717, -28, 8], [475, 3.4163085736806047, -28, 8], [480, 4.917938727171725, -28, 8], [485, 1.8980386951376085, -28, 8], [490, -2.8669093599521145, -28, 8], [495, -4.996034170931768, -28, 8]]}

genrl_sbst2022/levels_template/tig/main/MissionGroup/items.level.json

@@ -0,0 +1,7 @@
+{"name":"sky_and_sun","class":"SimGroup","persistentId":"43465d7d-0971-4933-89de-c4833b623141","__parent":"MissionGroup"}
+{"name":"CameraBookmarks","class":"SimGroup","persistentId":"10c228e4-8d9c-40fb-bdb6-d20febb48221","__parent":"MissionGroup"}
+{"name":"vegetation","class":"SimGroup","persistentId":"4f71ae35-2df2-4faf-9297-383b1a62642a","__parent":"MissionGroup"}
+{"name":"Water","class":"SimGroup","persistentId":"f0bb0c96-1127-4a30-bce3-cc64d2f856ec","__parent":"MissionGroup"}
+{"name":"PlayerDropPoints","class":"SimGroup","persistentId":"ebe56c16-ada2-45a5-8b62-07095074d620","__parent":"MissionGroup","Enabled":"1"}
+{"name":"terrain1","class":"TerrainBlock","persistentId":"4e2586a5-7955-49f0-bf35-73613f8cb368","__parent":"MissionGroup","position":[-1023.514038085938,-1114.167846679688,-539.9841918945313],"terrainFile":"levels/tig/terrain.ter"}
+{"name":"generated","class":"SimGroup","persistentId":"c545370a-d2b9-4f42-b6fa-578dda84bd81","__parent":"MissionGroup"}

genrl_sbst2022/levels_template/tig/main/MissionGroup/items.level.readme.txt

@@ -0,0 +1 @@
+{"class":"GroundPlane","persistentId":"8538a723-a048-47be-8a82-0d46c1159338","__parent":"MissionGroup","position":"0 0 0","material":"WarningMaterial","squareSize":54}