-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathbeast.yaml
266 lines (263 loc) · 10.8 KB
/
beast.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
---
user_name: beast
scenario:
name: "BEAST: operating wheeled devices"
description: "The robot under test has to precisely and safely follow a
trajectory through an irregular environment while pushing a shopping
trolley/walker"
image: undef
protocol:
- id: 21
name: "beast"
# todo: two variants: how to handle this?
name_readable: "BEAST: operating wheeled devices."
definition: "This protocol has two variants, employing two different active
devices (trolley and walker).
In its trolley version, the protocol evaluates the capability of the robot
to safely and accurately operate a self-supporting wheeled transport
device (the trolley) while performing navigation tasks.
In its walker version, aimed at scenarios where the main difficulty lies
in the walking process itself, the protocol evaluates the capability of
the robot to minimise its reliance on an external stability-supporting
device (the walker)."
image: https://raw.githubusercontent.com/madrob-beast/docs/master/img/beast_arena_checkpoints.png
keywords: [shopping trolley, walker, handle, wheel, swiveling wheel]
bipedal_system: [humanoid, prosthesis, exoskeleton, human]
pi_algo: beast
duration: "10 minutes per run"
controlled_variables:
- name: wheeled device
definition: wheeled device that the robot interacts with
type: string
range: [trolley, walker]
unit: none
- name: start already gripping
definition: Whether the robot starts the benchmark with its end effector
already positioned on the trolley/walker’s handle or the robot has to
grip the handle by itself
type: bool
range: [true, false]
unit: none
- name: load
definition: Amount of mass added to the trolley/walker
type: enum
range: [0, 9000]
unit: grams
- name: disturbance type
definition: The profile of breaking action applied to the door panel
based on its angle
type: enum
range: [no force, sudden force]
unit: none
testbed:
- item: "test environment"
type: other
file_sheet: undef
- item: "instrumented shopping trolley"
type: combined
file_sheet: undef
- item: "instrumented walker"
type: combined
file_sheet: undef
step_description: "Push a shopping trolley/walker through the environment
along a prescribed trajectory, compensating for possible disturbances to
its motion"
step_list:
- description: Manually set robot in start pose
detail: "human-executed; not part of the benchmark"
- description: Start the benchmark
detail: "human-executed; not part of the benchmark"
- description: Grasp trolley/walker handle(s)
detail: "Optional: robot can start with end effector(s) already
positioned on handle(s)"
- description: Navigate to Checkpoint 1
- description: Navigate to Checkpoint 2
- description: Navigate to Checkpoint 3
- description: Navigate to Checkpoint 4
- description: Navigate to Checkpoint 5
- description: Wait for benchmark completion
detail: human-executed; not part of the benchmark
# todo: check information provided
has_multiple_run: false
has_multiple_condition: true
has_multiple_subject: false
pi:
- name: time_to_grip_handle
name_readable: time to grip handle
description: "Time elapsed from the start of the benchmark to the first time
the handle is touched by the robot/humanoid.
This PI accounts for the time the robot takes to perceive the
walker/trolley's handle, plan its actions and grasp the handle."
unit: sec
output: scalar
intra_run_aggregation: undef
inter_run_aggregation: [[0, mean], [0, median], [0, std]]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
- name: time_to_checkpoint_1
name_readable: time to checkpoint 1
description: "Time elapsed from when the trolley/walker starts moving to
when the trolley/walker crosses the checkpoint 1."
unit: sec
output: scalar
intra_run_aggregation: undef
inter_run_aggregation: [[0, mean], [0, median], [0, std]]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
- name: time_to_checkpoint_2
name_readable: time to checkpoint 2
description: "Time elapsed from when the trolley/walker starts moving to
when the trolley/walker crosses the checkpoint 2."
unit: sec
output: scalar
intra_run_aggregation: undef
inter_run_aggregation: [[0, mean], [0, median], [0, std]]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
- name: time_to_checkpoint_3
name_readable: time to checkpoint 3
description: "Time elapsed from when the trolley/walker starts moving to
when the trolley/walker crosses the checkpoint 3."
unit: sec
output: scalar
intra_run_aggregation: undef
inter_run_aggregation: [[0, mean], [0, median], [0, std]]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
- name: time_to_checkpoint_4
name_readable: time to checkpoint 4
description: "Time elapsed from when the trolley/walker starts moving to
when the trolley/walker crosses the checkpoint 4."
unit: sec
output: scalar
intra_run_aggregation: undef
inter_run_aggregation: [[0, mean], [0, median], [0, std]]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
- name: time_to_checkpoint_5
name_readable: time to checkpoint 5
description: "Time elapsed from when the trolley/walker starts moving to
when the trolley/walker crosses the checkpoint 5."
unit: sec
output: scalar
intra_run_aggregation: undef
inter_run_aggregation: [[0, mean], [0, median], [0, std]]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
- name: roughness_of_actuation
name_readable: roughness of actuation
description: "Maximum force applied to the handle."
unit: N
output: scalar
intra_run_aggregation: undef
inter_run_aggregation: [[0, mean], [0, median], [0, std]]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
- name: safety_of_navigation
name_readable: "Safety of navigation"
description: "This PI is a measurement of the safety of the navigation.
It is obtained as the minimum distance of the walker/trolley from the
obstacles."
unit: meter
output: scalar
intra_run_aggregation: undef
inter_run_aggregation:
- [0, mean]
- [0, median]
- [0, std]
- [0, min]
- [0, max]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
- name: capability_level
name_readable: "Capability level"
description: "Number of steps of the benchmark actually completed by the
robot.
Each step is considered completed only after all the steps preceding it
have been completed as well."
unit: none
output: scalar
intra_run_aggregation: undef
inter_run_aggregation: [[0, median], [0, mean], [0, max]]
visu: Bar plot
inter_cond_aggregation: "Average of score for all conditions should be fine.
Although ideally the number of runs with different conditions should be
balanced."
inter_subject_aggregation: "Average of score for all subjects should be
fine.
The score may be very dependant on the subject, so we advise to run the
experiments with the same subjects."
pi_algo:
- name: beast
description: Compute all metrics related to beast protocol
url_paper: undef
url_code: https://github.com/madrob-beast/madrob_beast_pi
docker_host: eurobenchtest/pi_madrob_beast:v2.0.0
docker_name: eurobenchtest/pi_madrob_beast:v2.0.0
pi:
- capability_level
- safety_of_navigation
- roughness_of_actuation
- time_to_grip_handle
- time_to_checkpoint_1
- time_to_checkpoint_2
- time_to_checkpoint_3
- time_to_checkpoint_4
- time_to_checkpoint_5
input_files: [event.csv, distance.csv, wrench.csv, condition.yaml]
input_command:
- run_beast
- event.csv
- distance.csv
- wrench.csv
- condition.yaml
language: python
implemented: true