Implement a DCBO agent compatible with Stable Baselines3

[ChampiB]

Problem description.
I did not find a simple way to benchmark the DCBO agent against reinforcement learning baselines.

The desired solution.
I would like to be able to run the DCBO agent in the same way that I run a Stable Baselines3 agent, i.e.,

from yawning_titan.integrations.dcbo.dcbo_agent import DCBOAgent as DCBO
from stable_baselines3.common.evaluation import evaluate_policy

agent = DCBO(args...)
agent.learn(total_timesteps=1000)
evaluate_policy(agent, env, n_eval_episodes=100)

I have done the following.
I tried to locate the code related to DCBO, and found the following directory:
https://github.com/dstl/YAWNING-TITAN/tree/main/yawning_titan/integrations/dcbo

I read through part of the code, and if I am not mistaken the DCBO agent is learning offline and then its performance can be estimated.

However, when looking at the class DCBOAgent, I could not find anything related to training the agent.

@ChrisMcCarthyDev @dstl-admin Do you know how to implement what I described above? Any pointers would be appreciated.

[ChampiB]

I just found the following: https://github.com/alan-turing-institute/causal-cyber-defence

The above repository seems to contains the code for a DCBO agent running on the Yawning Titan environment.

[ChrisMcCarthyDev]

Hi @ChampiB

Thank you for your post! This is an interesting one, so thank you for bringing this to our attention.

Since taking on the YT repo, we haven't touched on Agent and agent profiles too much yet. Our focus thus far has been on the engineering aspects of the package to make it more stable, useable, and extensible. We do have several tickets on our backlog around Agent classes and Agent profile which are schedule to be picked up over the course of our current sprint and the following two sprints.

Having looked at the Example.ipynb notebook in alan-turing-institute/causal-cyber-defence, it looks like there's a dependency on the neildhir/DCBO repo too. There's a few actions we'll take away from this and include in the sprints where we look to developing the agent classes and agent profiles further:

1. Regarding the current yawning_titan.integrations.dcbo sub-package, while it appears there's work gone into it to have a dynamic causal Bayesian optimisation blue Agent trained in the Yawning-Titan environment, it's not immediately clear how to implement this.
2. The neildhir/DCBO package from @neildhir looks like it can be used in making this possible.
3. The recent updates to Yawning-Titan around how the game mode config and network configs are captured and passed into the NetworkInterface has placed breaking changes into the alan-turing-institute/causal-cyber-defence. As this is dependent on Yawning-Titan, we will put some time aside to submit a pull request alan-turing-institute/causal-cyber-defence with an update the the notebook using the most recent version of Yawning-Titan. Or at least have it pull Yawning-Titan v0.0.1 so that it still works.

Many thanks,

@ChrisMcCarthyDev & the Yawning-Titan dev team!

[ChampiB]

Lots of good information here! As I mentioned in the other response, I may try to implement an online DCBO agent at some point but I need to catch up with the literature first.

[neildhir]

Hi both,

Neil here, one of the authors of DCBO.

@ChampiB DCBO is trained offline and all it is doing is estimating the parameters for a set of Gaussian processes which we use as function approximations for the causal diagram. In the original paper this was done offline but in the cyber-application (as you linked above) we also enabled it to learn online (where the GPs are updated with streaming data coming from the system). Certainly, that paper uses the DCBO packages (as you can see with the added path at the top of the Example notebook).

Best,

N

[ChampiB]

Good to know! I guess the version of DCBO required for this task is the online version. I will probably read the code at some point, but I need to catch up on the literature first.