Verify performance of baseline

[emilio-cartoni]

Keep an eye on how the baseline performs during integration.

[emilio-cartoni]

On commit: 3cbe8d3

NOPLAN
100001steps [16:56, 1982.66steps /s]{'score_REAL2020': 0.09163811733270416

BASELINE
1500001steps 19:39, 2216.19steps /s 0.10718097341998632,
15000001steps 2:20:49, 2269.06steps /s 0.17869694419642482

[emilio-cartoni]

On commit: a68a9ca

BASELINE
with experience file filtered_transitions_15000000_12486_1_4170_466197_60x128_pos_noq_4_xy_push.npy
Extrinsic Phase: 100%|█| 50/50 [16:54<00:00, 20.59s/trials , score_REAL2020=0.09{'score_REAL2020': 0.09163811733270416

without:
Intrinsic Phase: 15000001steps [2:37:03, 2127.19steps /s]{'score_REAL2020': 0.1628960306426154,
Intrinsic Phase: 15000001steps [2:40:07, 2080.36steps /s]{'score_REAL2020': 0.12885043929342968,
Intrinsic Phase: 15000001steps [2:40:13, 1938.87steps /s] {'score_REAL2020': 0.15607540637037715,

NOPLAN
Extrinsic Phase: 100%|█| 50/50 [06:27<00:00, 6.91s/trials , score_REAL2020=0.04{'score_REAL2020': 0.0401959478225
Extrinsic Phase: 100%|█| 50/50 [06:11<00:00, 7.28s/trials , score_REAL2020=0.04{'score_REAL2020': 0.040195947822584996
Extrinsic Phase: 100%█| 50/50 [05:46<00:00, 6.61s/trials , score_REAL2020=0.040{'score_REAL2020': 0.040195947822584996,

Extrinsic Phase: 100%|█| 50/50 [05:21<00:00, 6.01s/trials , score_REAL2020=0.04{'score_REAL2020': 0.040195947822584996
Intrinsic Phase: 100001steps [06:06, 1935.13steps /s]{'score_REAL2020': 0.040195947822584996

[DavideMontella]

14 tests done with an average of 0.173

[emilio-cartoni]

14 tests done with an average of 0.173
With a 15M timestep intrinsic phase?

Let's keep this issue open until we release the Starting Kit.

[DavideMontella]

Yes, with 15M

[emilio-cartoni]

15M, with master ba121b3 and real_robots REAL2020 436f174
0.141
0.167
0.149
0.186

[emilio-cartoni]

Starter kit: 5a95163
real_robots: 0036124

NOPLAN, 1 object - 0.009
score_3D': 0.0065647186052345107, 'score_total': 0.0093648237362823957, 'score_2.5D': 0.0042436305504342259, 'score_2D': 0.013557581700210451
array({'2D': [0.0019238853468477371, 0.014646116762107234, 0.048329430719743632, 2.5206305740121675e-05, 0.037496311650508624, 0.0052332436507228977, 0.020380452399946389, 0.059569587310970271, 0.00018916903745471261, 0.027199084161152232, 0.00013083856632348473, 0.00021494142303393095, 0.00018921974232565845, 0.00026355219019182904, 9.6823605526141426e-05, 0.00010775576218115716, 0.0082696999888048437, 0.017498710185933061, 5.1160499418949119e-05, 0.047631148890843966, 1.0097519259201269e-05, 0.0086437200647780325, 0.0088970949253597779, 0.031326907584173717, 0.00061538421191363027], '3D': [0.011075257460490333, 0.01813792883665712, 1.127737925253152e-05, 0.002186923870105011, 0.0043489824021749647, 0.0068826776417246546, 0.0005238670337095303, 5.8904818376775313e-05, 0.0093045396734613792, 0.013116826936392812], '2.5D': [2.0418761466028717e-05, 5.5672870882952706e-05, 1.2744765160688819e-05, 0.0082813724892642215, 0.00025559948778319992, 0.0027008181263109081, 0.0018580968837133597, 0.0061901565422679749, 0.0050607456863446185, 0.015931357834323388, 0.0023816306365511955, 0.019474455390238496, 0.00052397305214397041, 0.00016237765792980274, 0.00074503807213260008]}

NOPLAN, 2 objects - 0.024
{'score_2D': 0.024226881473714795, 'score_2.5D': 0.016296566911933392, 'score_3D': 0.035530497474211886, 'score_total': 0.024108510305279798}
Extrinsic Phase: 100%|████████████████████████████████████████████████████████████████████████| 50/50 [03:44<00:00, 4.85s/trials , score_2D=0.0242, score_2.5D=0.0163, score_3D=0.0355, score_total=0.0241]{'score_2D': 0.024226881473714795, 'score_2.5D': 0.016296566911933392, 'score_3D': 0.035530497474211886, 'score_total': 0.024108510305279798}
{'2D': [0.04458546856689659, 0.05246327490573012, 0.002295162957392198, 0.011582211060486238, 0.06499581044790674, 0.0022169912438870856, 0.03536981266533803, 0.006013415339186251, 0.0019519270054632037, 0.013626183798269045, 0.028359984499848338, 0.04129984093848884, 0.004407287278455697, 0.007074204490545617, 0.03927022132766552, 0.04483449291349705, 0.0014029232885688107, 0.0027401458938103975, 0.026705049077939143, 0.04298895698912243, 0.03617689778436479, 0.002891152177613947, 0.05803325151931604, 0.030651966858386726, 0.003735403814691079], '2.5D': [0.05215592364978039, 0.0074181760122556525, 0.034489967643338955, 0.0035723360067399035, 0.026273182886867412, 0.001724153946449026, 0.0005874009228253276, 0.05010126750226719, 0.039925112194027695, 6.598735483097037e-05, 0.0035658941640465073, 0.008000714040828185, 0.007544086446980664, 0.00816672415604296, 0.0008575767517200522], '3D': [0.08368598908699233, 0.0014508271592979475, 0.0012270512967539368, 0.04189886900227845, 0.06732884570142209, 0.00577885972551587, 0.06961957360965254, 0.027153361437618997, 0.037816011408767385, 0.01934558631381934]}

NOPLAN, 3 objects - 0.032
{'score_2D': 0.03051578470205347, 'score_2.5D': 0.025829095121291883, 'score_3D': 0.044327031915719795, 'score_total': 0.031872027270558254}
Extrinsic Phase: 100%|█| 50/50 [04:09<00:00, 4.80s/trials , score_2D=0.0305, sc{'score_2D': 0.03051578470205347, 'score_2.5D': 0.025829095121291883, 'score_3D': 0.044327031915719795, 'score_total': 0.031872027270558254}
{'2D': [0.023407293743322385, 0.009036193117741823, 0.007650961482459576, 0.042759588832695414, 0.035179160612573104, 0.0651086841112424, 0.025996322135482287, 0.11957366830714725, 0.04627149998643088, 0.005769387505534589, 0.030040779020443747, 0.03247219049392795, 0.011275983111967536, 0.004955345708258985, 0.00233862474616395, 0.012993308139938467, 0.032381644175570726, 0.013654299989028625, 0.004969002424046994, 0.03704709211113489, 0.05262638189922506, 0.06495127496239983, 0.05462342895434255, 0.02327771445287137, 0.0045347875273863], '2.5D': [0.004944019771538153, 0.02209958920744585, 0.01885953302519892, 0.06425402535085971, 0.024458757733159434, 0.0370570225578504, 0.023645182439674096, 0.08186653809662953, 0.019330573353242675, 0.009067143684545797, 0.019197877821694215, 0.005652205992977563, 0.04177477892279089, 0.010182815239401531, 0.005046363622369453], '3D': [0.042532479418119995, 0.009111523146351275, 0.04610965053003911, 0.031354290541203436, 0.11846204819099314, 0.010562747726175289, 0.03885998336239692, 0.014964287752484338, 0.01896246643937177, 0.11235084205006265]}

RANDOM -- in progress

BASELINE, 1 object (0.201 avg)
8 runs:
0.189
0.168
0.224
0.205
0.184
0.190
0.259
0.189

[emilio-cartoni]

Starter kit: 5a95163
real_robots: 0036124

	RND 1 obj	RND 2 obj	RND 3 obj	BASE 1 obj	BASE 2 obj	BASE 3 obj
N. Sim	108	108	224	110	100	40
Avg	0.022	0.060	0.101	0.211	0.095	0.127
Max	0.044	0.128	0.179	0.289	0.160	0.184
Min	0.008	0.029	0.055	0.143	0.052	0.088

[emilio-cartoni]

With Fixes on planner and environment (double action):
REAL2020: 56e4b3d
real_robots: e09c89e

	BASE 1 obj with fix
N. Sim	21
Avg	0.211
Max	0.278
Min	0.143

[emilio-cartoni]

Note on memory usage:
3 simulations on the server running the extrinsinc phase after 60M intrinsic timesteps are using 30Gb RAM each.
So a raw estimate is 0.5 GB per 1M timesteps.

[emilio-cartoni]

f1bba9c	macro_action
N. Sim	30
Avg	0.223
Max	0.319
Min	0.149