RL training not working

[charlo1998]

Hi @srivatsankrishnan , i'm having a similar issue to #1 : i'm able to connect, but the training won't start and I get a NotImplementedError. am I supposed to implement the step, reset etc. methods myself?

(workspace) C:\Users\charl\workspace\airlearning\airlearning-rl\run_time>python collect_data.py
Using TensorFlow backend.
C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:522: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\charl\workspace\lib\site-packages\tensorflow\python\framework\dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tornado.general:Connect error on fd 4344: WSAECONNREFUSED
Connected!
Client Ver:1 (Min Req: 1), Server Ver:1 (Min Req: 1)

2022-05-13 10:06:15.588240: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2022-05-13 10:06:15.674282: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties:
name: NVIDIA GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 5.02GiB
2022-05-13 10:06:15.674451: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2022-05-13 10:06:16.075145: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-05-13 10:06:16.075245: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0
2022-05-13 10:06:16.076013: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N
2022-05-13 10:06:16.076385: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3686 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
Connected!
Client Ver:1 (Min Req: 1), Server Ver:1 (Min Req: 1)

2022-05-13 10:06:16.299001: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2022-05-13 10:06:16.299178: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-05-13 10:06:16.300228: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0
2022-05-13 10:06:16.302869: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N
2022-05-13 10:06:16.303480: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3686 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
Traceback (most recent call last):
File "collect_data.py", line 104, in
main()
File "collect_data.py", line 100, in main
runTask(task_el)
File "collect_data.py", line 52, in runTask
train_class.train(train_obj, env)
File "C:\Users\charl\workspace\airlearning\airlearning-rl/algorithms/discrete/dqn\dqn_baselines.py", line 40, in train
agent.learn(total_timesteps=settings.training_steps_cap)
File "C:\Users\charl\workspace\airlearning\airlearning-rl\stable_baselines\deepq\dqn.py", line 170, in learn
obs = self.env.reset()
File "C:\Users\charl\workspace\airlearning\airlearning-rl\stable_baselines\common\base_class.py", line 523, in reset
return self.venv.reset()[0]
File "C:\Users\charl\workspace\airlearning\airlearning-rl\stable_baselines\common\vec_env\dummy_vec_env.py", line 45, in reset
obs = self.envs[env_idx].reset()
File "C:\Users\charl\workspace\lib\site-packages\gym\wrappers\order_enforcing.py", line 16, in reset
return self.env.reset(**kwargs)
File "C:\Users\charl\workspace\lib\site-packages\gym\core.py", line 74, in reset
raise NotImplementedError
NotImplementedError

[limshoonkit]

Hi @charlo1998 , @srivatsankrishnan
I am encountering the same issue. Is there a fix for this?

Traceback (most recent call last):
File "C:\opt\AirLearning\airlearning\airlearning-rl\run_time\collect_data.py", line 104, in
main()
File "C:\opt\AirLearning\airlearning\airlearning-rl\run_time\collect_data.py", line 100, in main
runTask(task_el)
File "C:\opt\AirLearning\airlearning\airlearning-rl\run_time\collect_data.py", line 52, in runTask
train_class.train(train_obj, env)
File "C:\opt\AirLearning\airlearning\airlearning-rl/algorithms/discrete/dqn\dqn_baselines.py", line 41, in train
agent.learn(total_timesteps=settings.training_steps_cap)
File "C:\opt\AirLearning\airlearning\airlearning-rl\stable_baselines\deepq\dqn.py", line 202, in learn
new_obs, rew, done, _ = self.env.step(env_action)
File "C:\opt\AirLearning\airlearning\airlearning-rl\stable_baselines\common\vec_env\base_vec_env.py", line 94, in step
return self.step_wait()
File "C:\opt\AirLearning\airlearning\airlearning-rl\stable_baselines\common\base_class.py", line 529, in step_wait
actions, values, states, information = self.venv.step_wait()
File "C:\opt\AirLearning\airlearning\airlearning-rl\stable_baselines\common\vec_env\dummy_vec_env.py", line 36, in step_wait
self.envs[env_idx].step(self.actions[env_idx])
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\gym\wrappers\order_enforcing.py", line 37, in step
return self.env.step(action)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\gym\wrappers\env_checker.py", line 37, in step
return env_step_passive_checker(self.env, action)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\gym\utils\passive_env_checker.py", line 214, in env_step_passive_checker
result = env.step(action)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\gym\core.py", line 115, in step
raise NotImplementedError
NotImplementedError

[charlo1998]

Hi @limshoonkit, i was able to find a fix for this, although i don't know if this was the intended way to do it. If you search a bit the RL files, you'll see that the step method is declared like this: _step():

if i understood correctly, importing with "from gym_airsim.envs.airlearningclient import *" ignores the underscored methods, so i changed the declaration to step(): and it worked.

hope this helps

[limshoonkit]

Hi @charlo1998,

Thanks for the hint. I noticed you forked the project and made some changes. Did you have any luck on migrating the codebase from Tensorflow 1 to Tensorflow 2?

[charlo1998]

Hi @charlo1998,

Thanks for the hint. I noticed you forked the project and made some changes. Did you have any luck on migrating the codebase from Tensorflow 1 to Tensorflow 2?

unfortunately no, i stuck with tensorflow 1.8 and changed my CUDA and Cudnn drivers for compatibility

[qinglong0276]

Hello, how did you get to this point? I encountered an error "AirLearning could be compiled. Try Rebuilding from source manually" while trying to install the AirSim plugin for Air Learning in the first step of the Air Learning Environment Generator. Now, I don't know what to do for my next step. Can you teach me?